[MUSIC PLAYING]
MARTIN SPLITT: Good to
be back on this stage.
And I hope that you all
have some energy left
from two fantastic days
of lots of content.
And content is what we're mostly
going to be talking about as
well, because I would like to
talk to you about what we have
done to bring your content
out more in Google Search
and make it shine.
Cool!
All right.
So I think one of
the things that I
have learned over the last one
and a half years-ish of talking
with you all and also bringing
my perspective as a web
developer to the
Google Search team
is that sometimes
we underestimate
the power of Google Search and
what it does to our website.
Because, fundamentally,
why do we build websites?
I mean, sure, OK, we
also build websites
because we are kind of
excited about the stuff
that we can build, like any new
capabilities and all that stuff
that we brought to
the web platform.
They're exciting.
But, fundamentally, we build
our websites for users.
And all our work towards
performance, and user
experience, and safety,
and privacy, and all that
are basically
mirroring that as well.
But, fundamentally, we want
users to find our content
because the web
apps that we create,
the content that
we put on the web,
we think is very, very valuable.
Hopefully you think that
your content is valuable.
If not, we should have
a chat, I believe.
But, OK, let's assume that
you think that your content is
really good, and you have
done the necessary steps
to make it good.
How much does Google
Search give you?
How much does it help you?
Well, it turns out,
by a recent study
that was done, that
approximately 53%
of the traffic that
websites are seeing
are actually coming
through organic search.
What is organic search?
That's your day-to-day
search engine
of choice, whatever it is.
That can be Google
Search, for instance.
So what we are doing
is we are trying
to bring free traffic to you, to
bring your content to the users
to make sure that your content
can actually shine and prosper
on the web and that you,
your company, your business
can actually be
successful as well.
So that's one way
to look at Search.
And I think, for us developers,
it's an important venue
to understand where are the
users coming from who we hope
to be on the site,
and how can we
make sure that they are
seeing the content that they
need to see, because it's
our content after all.
And today, I'd like
to look back at what
we have done this year to make
Google Search better for you.
So we are making things
better without you
having to do the work.
Isn't that amazing?
It's like free, good stuff.
It's like a free cookie.
I don't know.
It's not as good as
a cookie, though.
I still think, though,
cookies, the entire thing
with baking cookies
is a weird one anyway.
I think you have
good cookie dough,
and then you put it in the
oven and it comes out as good?
But no, anyway.
Might just be me.
So one of the biggest things
that has happened in this year
is probably our
announcement at Google I/O
that Googlebot is now evergreen.
Right?
That's a quantum leap.
We have been using a crawler
or a rendering engine that
was a little behind, and
we have jumped forward
to making that evergreen, to
keep up with Google Chrome
and actually give you all the
features that Google brings
to the web platform, also in
rendering so that we can index
the content that relies
on these new technologies,
and possibilities, and
capabilities of the web
platform.
But what does that really mean?
Well, first things first.
This is a huge leap, and
has been basically worked
on by many teams, many people.
A lot of the work is now
also visible in other venues.
Puppeteer, for instance,
is a side effect
from what we have done
in Google Search as well.
And it brings over a
thousand new features.
ES6, we have proper Web
Component version 1 support.
We have-- well,
no we don't have--
we have a bunch of new stuff
that we didn't have before.
You can now rely on that to
not be an SEO problem, to not
get in the way of
your content actually
being accessible
to Google Search
and actually being
visible in Google Search.
We have also made sure
that we are not just
doing one update one time.
But within a relatively
short timeframe
of every new stable release
that comes to Chrome,
we are actually updating our
Googlebot rendering engine
as well, so that what
you see in your Chrome
is also what you see
in Google Search.
And not just that.
We also made sure
that the testing
tools we have out there--
the rich results
test, the M-test,
the mobile-friendly test, the
URL inspection tool within
Search Console--
all these tools, they are
also up-to-date with whatever
version of Chrome
we are using when
Googlebot renders your pages.
So you can actually
verify that things
are working in Google Search.
The nice thing is that,
with this new change,
we are all still catching up to
what this means when Microsoft
announced that Edge is
now using Chromium as well
as the rendering engine.
We were really, really excited
to see that Bing has also went
forward and went like, OK, yes.
We are also running with
an evergreen Bingbot,
and we want to render JavaScript
using state-of-the-art
rendering as well.
So this means that now, across
the industry in search engines,
we are seeing signals
of actually getting up
to speed with the modern web
in search engine indexing,
and that's really,
really exciting to me.
Also, basically, Google
Search got a lot more powerful
because we are using the
browser that you are probably
using as well when we
are rendering your pages,
and making sure that
we work and continue
to work on making sure that
what we see when we index
your content to show
it to users on Search
later, that we are seeing
what you're seeing as well.
Right?
But there is one thing where
we still have some work to do.
This is the user
agent that you see
in your server logs
if we are accessing
your content with Googlebot.
And the keen eye might spot
that something's off here.
Right?
And I get asked this
question, but you said
it's an evergreen Googlebot.
But then I look at my
server logs and I see this.
What's happening here?
The simple answer
to that is that we
are working to maximize the
compatibility between Google
Search and websites,
and we can't just
make changes like this.
We have to test
things very carefully.
And we make sure that we are
supporting everyone's content
in Google Search.
So we had to basically
be very, very careful
moving forward with
the user agent.
But we have announced
multiple times now
that this is about to change.
We are about to update the
user agent in Google rendering.
So please, please,
please, if you
are having some
sort of code that
is doing UA sniffing
for Googlebot,
do not match to the exact
string that you saw earlier.
Match, basically, very
roughly about Googlebot.
If you want to know if it's
a real Googlebot or not,
that is perfectly possible.
We have documentation on that.
You can do a reverse
DNS lookup from the IP
that we tell you
we are coming from.
If that resolves towards
Google IP addresses,
then you know it's
an actual Googlebot.
Because the user agent
string does not really
give you that benefit.
So please make sure that, if you
have workarounds for Googlebot
and others, that you are not
too strictly matching the user
agent in the future
going forward,
because this change is
hopefully happening in December.
That's what we are aiming for.
If you want to
learn more, you can
go to webmasters.googleblog.com
and learn
more about this change as well.
Cool.
Another thing that I
oftentimes get asked--
and it's a very,
very fair question--
is what about JavaScript being
so much slower in indexing
and rendering in Google Search
than non-JavaScript websites.
Is that still a problem?
Well, last year, Tom
and I were on this stage
and telling you well, you
know, it can take up to a week.
We are very sorry for this.
Forget this, OK?
Because the new numbers
look a lot better.
So we actually went
over the numbers
and found that it turns out that
at median, the time we spend
between crawling and actually
having the rendered results,
on-median, it's five seconds.
[APPLAUSE]
Right?
A few keen eyes here
might be like, oh,
but what's the 90th percentile?
To which I can say,
don't worry about that.
That's within minutes.
So you don't have to worry about
week-long delays or something,
or anything like that.
However, be prepared that
you might still hear that.
And the reason for
that is the way
that the entire
pipeline really works.
So the pipeline that
Googlebot runs through
is actually quite
a complicated one.
This is a gross
oversimplification.
It's much, much more
complex than that.
A bunch of stuff might
happen at the same time.
A few steps might
run multiple times.
It is more
complicated than this.
However, most of it
is transparent to you
because it's an
implementation detail
that you shouldn't
be worried about
and you don't have
to be worried about.
But effectively, that means
that the only moments when
you actually see if
something's happening
are actually when we cross.
When we make an HTTP
request to your service,
we leave a mark in
your server logs.
You see, a-ha, Googlebot
is fetching this request.
But you don't really see
when we are processing
because there's not much
actionable things that
happen in processing.
It's just like us
dealing with the response
and with the robots.txt.
And all that kind of stuff
happens and in crawling
and in processing.
You don't really see that.
Rendering itself,
you don't really
see that because we are just
working through the resources,
executing your JavaScript,
producing content, bringing
that back to processing.
There's not much you can see.
What you can see, though,
is at the very end
of this entire process, you
see when the content shows up
in search results.
Right?
So there is a whole lot
of stuff happening in
between these two points
that you can actually
measure and see.
There's canonicalization, which
means removing duplicates.
There's error handling.
What happens if there's
a network timeout?
What happens if your server
was down for a moment?
These kind of things
we handle for you.
You don't have to
worry about them.
But that means
that if we are only
coming around crawling
every now and then,
and then certain things
happen on our side
that you don't have
to worry about,
it might take a while
until you actually
see things in the index.
But that doesn't mean that
JavaScript held them back.
Generally speaking, as we saw,
we are really, really fast
in rendering, so that's
not really an issue
that you need to be
worried about anymore.
OK.
So now, I have basically
addressed the two biggest
questions that I kept getting
since I/O, when we announced
the evergreen Googlebot.
The user update is
going to change,
and rendering delays are
not really a big problem
that you have to deal with.
You can be pretty relaxed about
using JavaScript in your pages
for SEO reasons.
But now let's take
a step back and see
what is actually the complexity
of this entire thing.
Why did it take
so long and what's
involved in making a renderer?
Or how do I make a
rendering engine?
How hard can this be?
Just give me 10 minutes and
I'll have a renderer, right?
So what if we take
Puppeteer, for instance,
which is effectively a
programmatically-controlled
Chromium instance, which
is what our rendering is
as well, right?
And we just build our
own rendering engine.
OK, cool.
Well, renderer for
search purposes.
So I think the easiest
thing when we use Puppeteer
is to just get started.
You just open a new page
in your Puppeteer instance,
and you set the viewport.
Except what do you
set your viewport to?
Mobile dimensions.
OK, but what are
mobile dimensions?
And what if the
content is very long?
Hmm.
This is already a little weird.
OK.
Let's just assume we somehow
figured that one out.
We set our viewport dimensions--
easygoing, whatever.
Next step is we just have
to fetch the actual page.
So we make a network request.
We get all the HTML in,
that loads all the images
and all the JavaScript.
So we have to load
these resources as well.
And we wait until
the page is done.
But what does "done" mean?
Do we wait until the
DOM content loaded?
Do we wait for page load?
Do we wait for when the CPU
doesn't do anything anymore?
But what if the CPU keeps
cycling through JavaScript
because there's some sort of
periodic thing that happens?
What if it takes too long?
What if there's an infinite
loop in your website?
Does that mean that
this instance is now
completely blocked forever?
Do we wait five seconds?
10 seconds?
This is what Rendertron does.
Rendertron just goes
like, until we do not
have anything on
the network anymore
or a certain threshold
is over, then
we're just going to kill this.
But this is a simplification.
We are having a
lot of work to do
to make sure that we actually
wait until your website is
ready and do not fall
in the trap of waiting
too long either.
Right?
So that's a problem.
And last but not least,
now we have actually
gotten this website to render.
What do we do next?
Well, we need to figure out
what is the content here.
So can we just pull the HTML
from the entire document
like this?
But what about Shadow DOM?
Hm.
That's a tricky one.
How do I deal with
Shadow DOM being there?
Because I might not get
Shadow DOM through this,
because the Shadow DOM border
is going to hide that Shadow DOM
content.
Ah.
This is tricky.
You see, the point here is that
it isn't as straightforward
as you think to actually
build a renderer.
Our teams have worked years
to make this as perfect
as possible.
And we are continuously
working on improving.
And we're continuously
testing and monitoring
what happens in Search to
make sure that your website is
rendered properly.
There are things you can
do to help us out here.
Make your websites fast.
Make your websites reliable.
Make sure that they're not
taking too much CPU resources.
They are not
killing the network.
Because these are effects
that users will also feel.
So we are working to make
these complexities go away,
but you should not worry
about these complexities.
This is the nice thing about it.
It's an API that is relatively
easy and straightforward
to deal with.
You just make good
websites for the user.
The rest is my problem.
It's nice, isn't it?
It's like, it's not my problem.
It's Martin's.
How can I ever sleep again?
I don't know.
We'll see.
This has been quite a journey.
I mean, this sounds
quite not large and big,
but the evergreen Googlebot
turned search results indexing
from websites that
showed up when Googlebot
was accessing them
like this, into getting
the proper content.
So we are finally
ready to use whatever
you are throwing at
us, as long as it
is reasonable for the
users and the web in total.
And I think this is a
very important moment.
So these are complexities that
we are taking away from you,
and you can focus on building
better things for the user.
And I think that's the key here.
And we are pretty happy that
this is not just us being like,
well done!
No, it's actually being
seen in the industry,
especially also from SEOs.
So Bartosz-- from Onely, which
is a technical SEO agency--
published a case study and an
article of the JavaScript SEO
state in 2019.
And he compared that
to 2018 and said
this is an amazing improvement,
and we have been doing so much
better in these terms.
And I like this
specific quote where
he said, "If you look
at National Geographic,
they're a very good example.
100% of their JavaScript
content is indexed quickly."
So that makes us
very, very happy.
And this should be your signal
to bring this information back
to your SEO experts and SEO
advisors, and tell them, look,
JavaScript is an
important part of the web.
And modern web applications can
be successful in Google Search.
That is absolutely possible.
Now I would like to quickly
walk through a few things
and a few patterns that I'd
like to quickly discuss,
because I think they are
great patterns to use.
And they are also
very, very helpful
for indexing in Google Search
or rendering in Google Search.
One is lazy loading.
I think we're all very
happy that we finally
have the native lazy loading
APIs for images and iframes.
These work in Googlebot.
As Googlebot is
evergreen, we are
supporting these attributes.
And they are not hurting you.
So if you are not
using them yet,
you should definitely
consider using them.
It's such a low-hanging
fruit to do.
If you need more complex
lazy loading, or lazy loading
on elements or
content that does not
fall into the two categories
where native lazy loading
exists, I highly
recommend checking out
the Intersection Observer.
Because who would have thought?
This is also now
supported in Googlebot.
And we are testing this
quite frequently as well.
So this allows you to
have more custom lazy
loading if you need to.
As I said, Web
Components are supported.
But as I also said, when
I said let's build a naive
renderer using
Puppeteer, Shadow Dom
is an interesting thing, right?
So I got questions
about how does that--
huh?
Is it supported?
Does it work?
Can we actually index
content inside Shadow DOMs?
And the answer is yes!
Yes, we do.
This is absolutely,
perfectly fine.
We have the Shadow DOM content.
In this case, is this indexed?
Yes, it is.
What about this?
And then we use some
mechanism to get
whatever is in the Light DOM.
In this case, can you render me?
We pull this into
the Shadow DOM,
and that gets flattened
into the tree.
And then we can
index both of these.
So all of the content here
will be successfully indexed
in Google Search.
So that means we have finally
full, proper version 1 Web
Component support
in Google Search.
[APPLAUSE]
I cried a little when
we had that moment.
I'm like, finally!
Such a beautiful moment.
Another thing that I'd
like to just bring up--
because that's something that
doesn't just like magically we
fix it, because we
think it's not broken--
is if you have images.
Images are a fantastic way of
engaging with your audience
and making sure that you
surface your content.
If you have your
original image content,
you should put that
in Image Search,
because it's a fantastic
venue for people
to discover your articles
around these images as well.
If you want us to index
your images, totally cool.
We got you.
The HTML image element is
there and is perfectly fine.
This is what we do.
We search for image
elements in your content,
and then we index
them, unless you
can prevent us from doing so by
setting no index or something.
That's not great.
We're not going to index that.
Don't.
OK?
Sweet.
Speaking of images, we
are also basically working
towards making Image Search
more useful for users.
We have a bunch of different
venues of doing so.
So we are having better query
and understanding, as well as
we are trying to be more
useful for figuring out
the context of the
image in the article.
So we are featuring
images differently.
And soon you will be able to use
Search Console to also tell us
if you want to use
structured data to give us
high-resolution images.
These high-resolution--
oh, sorry.
[COUGHS] I shouldn't have sung
that many songs yesterday.
Anyway.
Stage.
Right.
So if you want your
high-resolution images
to not hurt performance,
you can tell us
in structured data in
the future that you
have a high-resolution
version, and we
can surface this in other places
like Google Homes or Google
Assistant, which brings
more traffic to these images
if you're lucky.
So we highly
recommend checking out
the articles that are
coming up on the Google blog
in the near future,
and also using
Search Console to see if
there's new stuff coming.
Speaking of structured
data, we have more
structured data to make sure
that your content really
can shine.
We have how-to markups.
We have QA pages and all
that kind of lovely stuff
if you want to check that out.
I highly recommend
using that link
to find what kind of
verticals we support.
We support all
sorts of verticals.
We support events, like
this, for instance.
The website has
structured data so
that we can highlight
that in Assistant
as well as in the Search
results, like here.
We have recipes.
We have articles.
We have books.
We have movies, I
think, and TV series,
and all that kind of stuff.
And this gallery is
a fantastic place
to check out what's
happening there.
And, definitely, it gets you--
this is very, very much
an eye catcher, right?
You get more screen
estate in Google Search,
which means your content
is more likely to catch
the eye of the user.
Recipes are one example.
Sweet.
Last but not least, I
say we have made sure
that our tools are
always up-to-date with
the actual rendering engine.
So when you are using
mobile-friendly tests,
rich results tests,
all of these tests,
you are actually rendering
through the real Googlebot
rendering process.
And this updates as
Googlebot updates now,
so we are no longer having
a discrepancy between what
we do in the testing
tools and what
we do in the actual Googlebot.
We also made sure
that we gave you
a better flow of working
with things in Google Search
Console.
The new Google
Search Console allows
you to monitor how your
entire site is doing
in terms of how is it
performing in Search,
but also are there pages
that are having problems,
is there structural
data that is broken,
and also all sorts
of other things.
You can easily debug these
things within Google Search
Console and use other
tools on top of it
as well to make sure that
you are on top of what's
happening on your page.
And you can use the Validate
Fix flow, where you can
signal to us as Google Search.
I think we fix
this and then we'll
double-check if that is true.
If it is fixed, we'll let
you know and also re-index
the page.
You can validate if everything
that you have been doing
is right.
And then last but not least,
the performance report also
helps you figure out
where you can do better,
where can you grow
your search traffic.
So besides debugging
any sort of issues
that you might
have on your site,
we have a few new,
really cool things.
So this is the URL
inspection tool.
That has been there for a
while, but it's worthwhile
checking it out.
It tells you how Google
Search was accessing your page
the last time.
What was the result?
Can we index this
content or not?
Why can we not index it
if we can't index it?
What can I do about it?
And there's also the Validate
Fix flow in the screen.
But also, we basically give
you JavaScript debugging
capabilities in
Search Console, too.
So you get the stack trace.
You get the error messages
that we are seeing when
we are rendering your website.
So you no longer
have to just guess.
You can actually
run this and see
what we are stumbling
over, if we are stumbling
over some JavaScript problems.
A very nice one is the speed
report that launched last week.
So we are now having this report
that gives you an idea of,
based on the Chrome
User Experience Report
from real user data,
where are pages
that we might want to optimize.
So it takes them
into three buckets.
It tells you these pages.
This is a sample of
pages that are fast.
These are pages that
are OK, moderately good.
And then these pages are slow.
And you can work
with these pages
and basically integrate
with other tools,
like Lighthouse or
PageSpeed Insights,
to figure out where you need
to spend a little bit of time
to make them faster.
Imagine you have a website
with a million pages.
Do you want to run
Lighthouse on all of them?
Like, OK, cool.
I'll be gone for a week.
I'm Lighthousing again.
Lighthouse is a
fantastic tool once you
know where these problems are.
The speed report is your
way, your lens, into that.
You see where can
I spend some time
to make my website
significantly faster.
What are the low-hanging fruit?
Where can I really--
where should I
be drilling in and digging
in to make sure my website is
fast and great for users?
So we highly recommend
you check out this report,
as it gives you a
wonderful sitewide
overview of what's happening.
Cool.
As you can see, it lets you
drill into the different kinds
of issues.
And you can also,
then, see specifically
what the pages need.
I see people taking pictures.
That's good.
We also have a
fantastic new document.
Well, it's not that new,
but it's new to many people
here, probably.
It's the troubleshooter guide.
If you have JavaScript
problems, we
have a guide that
tries to identify
the individual problems
and then guides you
through potential
solutions to them.
This guide is relatively new.
And we have worked
with the community
to make sure that
we are covering
the most frequent
problems, but we would love
to hear your feedback on this.
If you see problems that were
not tackled in the report
or in the document,
then please let us know.
We are happily extending the
document with your feedback
as well.
Sweet.
Last but not least, I'd like
you to take a few things away
and also to bring this
back to the SEO experts
that you're working with.
If you are an SEO expert
in the crowd here-- woo--
then please also bring this back
to your developers, who might
not be watching this stream.
I think one of the
things that shocked me
the most over the last
one and a half years
was that, oftentimes, SEOs
and developers are not
working hand-in-hand together.
Because I think we are
very powerful allies
if we work together.
And they can help you not deal
with all the monitoring effort
and making sure that your
content strategy is right.
They are really good at that.
But you, as developers,
you can help
them understand better what do
these performance metrics mean.
What does the slow performance
mean on a specific page?
What does the soft
404 really do?
And basically help them
prioritize the things.
And if we work
together, we'll be
able to bring better
content to all our users
much, much more
efficiently, and we
will be able to unlock
these additional users that
might come over organic traffic
if we tick all the boxes.
So we can be working together.
We should be allies.
We should not be working
against each other.
OK?
So last but not least,
I'd like to sum it up
by saying Google Search is here
and will be here to help you
and your users to
find each other.
We want to bring relevant
content to the users
that you care about, that you
might not even have met yet.
And with that, I'd
like to say thank
you very much for your time.
Enjoy the rest of the evening.
Bye-bye.
[MUSIC PLAYING]
