[MUSIC PLAYING]
TOM SLATER: Hi, everyone.
I'm Tom.
I'm one of the technical
leads over on our VR and AR
efforts at Google.
My team specializes in
developer productivity tools.
I'm really excited,
just one week
after announcing AR Core,
to stand here today and talk
to you all about the
AR Core SDK preview
and how you can really
easily add AR capabilities
to your Android apps.
So just a quick run
through of what we're
going to talk about today.
So we're going to talk about
AR and its really closely
related cousin VR.
We're going to talk about the
concepts that underpin AR Core.
We're going to have a
walk through the API.
We're going to talk
through all the options
that we have to
build our AR content.
And we're going to talk about
all the options for building
your code as well.
So before we get into
the technical details,
I really want to start with
what immersive computing means,
where we've come from, and
what can it do for you.
So I'm always asked
why AR and VR?
We have these mobile
devices, and it's all great.
Why should we go and
do these other things?
So with Cardboard,
Daydream, and Tango,
we've been investing in this
space for quite some time.
And we really believe
that interacting
with your data in a more
natural way is the future.
Immersive computing removes
a lot of those abstractions
between you and your computing.
So I want to take a minute just
discussing these technologies
before we jump into some code.
So we think of AR
and VR as being
points on a spectrum
of immersive computing.
On the far left, you have
reality, which is the world
as we know it, and
we're sitting in today.
If we start to add digital
content into our reality,
then we start to augment
it, and that is, hence,
augmented reality.
And if we completely replace
that reality with the virtual,
we have virtual reality.
And so AR can bring any
of your digital content
to you and in your world, such
as this guy posing with a dog.
So some of the benefits of AR--
you can see objects at real
size and scale in your world.
Imagine being able
to buy furniture
and being able to see if it
actually fits in your house
before you buy it.
You can also see
things in context.
Again, imagine buying that
furniture and making sure
that it matches with all of
your other decor in your house.
And being able
to-- imagine being
able to annotate the
real world with post-it
notes without actually
causing a real physical mess.
And it also adds the ability
for natural input for 3D scenes.
So if ever you've used digital
content creation or modeling
tools, it's actually
really hard to control
that camera in your scene.
With augmented reality,
you just hold your phone,
and you just look at it,
and anyone can do it.
And then if you replace
everything in your world
with the virtual,
with digital content,
you've got virtual reality.
It allows you to go to places
and visit worlds in an instant.
And some benefits of this are
you get complete immersion.
You really feel like you've been
transported to another place.
And there's some really, really
huge innovations in input
that really lets you work
naturally in your world.
Both VR and AR enable us
to experience computing
more like we experience
the real world,
and both take advantage of a
lot of the same technologies.
So back in 2014, we
started with the idea
that your devices should
be able to understand
more of the world.
So with dedicated
hardware, Tango
allowed us to
understand the depth
and allowed us to build some
really great applications,
from being able to measure
your world, being able to map
and share your house,
and to be able to play
some games on your
tabletop with your friends.
We built two consumer
devices with Asus and Lenovo,
and developers create more than
100 applications for Tango.
We learned a lot, and now,
we can do more with software
than we could three years ago.
And we don't have to rely on
custom sensors, which leads us
today and last week's
announcement with AR Core.
So last week, we
announced AR Core.
It's a preview SDK that
allows you to get up
and running with building
AR applications right now.
AR Core takes everything
we learned from Tango,
and it makes AR available on
the phone you have today--
no depth cameras or custom
special sensors required.
AR Core is currently
in preview, and we're
looking to developers like
you to give us feedback on how
to make this SDK work for you.
And with two billion
devices out there,
we have a huge potential
audience for this technology.
With AR Core so far running
on the Pixel, Pixel XL,
and the Galaxy S8, we currently
run on millions of devices
already.
And we're working
with manufacturers
like Samsung, Huawei,
LG, Asus, and others
so that at the end
of the preview,
we anticipate that we're going
to run on 100 million devices.
And we're working with
these hardware vendors
to make this possible with
a really consistent bar
for quality and high
performance, in the same way
that we did with Daydream.
And so our before we
get into the code,
I think it's really,
really helpful
to understand some of the
fundamental concepts behind AR,
just so you know how AR Core
is working under the hood.
So there are three main
concepts to think about.
One is motion tracking, two is
environmental understanding,
and three is light estimation.
So let's go through them in
a little bit of detail now.
So to render AR content, you
need a virtual camera that
matches your physical camera.
You render the virtual scene,
you composite with your camera,
and you're done.
This sounds simple.
It's actually
really, really hard.
Whilst your phone gyroscope
is really, really great
for rotation, it
can drift over time.
And whilst your
accelerometer is great
for those instantaneous
inputs, it's
not so great to figure
out an actual position.
So the really hard part
of getting AR right
is to figure out this
translation and rotation
of your device in
real time so you
can render digital objects
with the same virtual camera
as your physical camera.
If you get this wrong,
objects in the world
will be misaligned with
their virtual equivalents.
They will swim and jump,
and they won't appear
properly rooted in the world.
And you can see
how effective we've
done this with AR Core
because of this scarecrow
who looks just like everyone
else queuing for tacos.
So to do this, AR Core
uses the device's camera
and inertial measuring
unit to track
exactly where your device it
in the world using a process
called Concurrent Odometry and
Mapping, also known as COM.
It looks for visually
distinct features
that can track over
successive frames
and builds a point cloud so it
can localize against that point
cloud.
This, combined with the
high frequency IMU data,
gives you rotation
and translation
in the world so you can render
virtual content in exactly
the right place.
This is over and above other AR
experiences you may have tried,
which only uses the gyroscope
to get a rotation, which
has the problem of content
sliding around your world,
and you can't move in
closer for a better look.
This is really, really key for
anchoring your digital content
over the real world.
So this illustration
is an example
of the device tracking
feature points in your world
and creating a point cloud,
but when it does it for real,
has actually a lot more
points than just four.
So on top of motion tracking,
which is really important,
environment understanding
is also super important.
Rendering content isn't actually
that interesting by itself.
You need to be able to interact
with your world as well.
So AR Core is looking for
clusters of those feature
points that appear to lie on
common horizontal surfaces,
and it makes these surfaces
available to your app
as planes.
Since planes are
mathematically infinite,
AR Core also provides the bounds
of these surfaces as a polygon.
And you can use this
information to place objects
in your world like
this Android guy here.
So planes are detected
on horizontal surfaces,
such as the floor, tables,
kitchen countertops, benches,
chairs, you name it.
However, because AR
Core uses feature points
to detect these
surfaces, flat surfaces
without any texture or
highly reflective surfaces
might not be detected properly.
And then finally,
light estimation.
So AR Core is able
to detect information
about the current
environment's lighting
and gives you a
value representing
the average intensity
of a given camera image.
This information lets
you light your scene
and your virtual objects
under the same conditions
as the environment,
which increases
that sense of realism.
If you don't do this,
your digital objects
will stand out and
not appear to be
a part of that world,
which is really, really
key to realistic rendering.
And it lets you do some
really fun effects,
like have this lion
who gets scared
when you turn out the lights.
And it might seem frivolous, but
it's actually really important.
If you've ever taken your
camera and just looked
at a light source or
pointed it out the window,
the auto exposure of
your camera actually
changes that range
pretty hugely,
so it can be really not so
great if you don't take this
into account.
So now, we've gone through
the main concepts of AR Core.
Let's jump into
some code and see
what it takes to build an
AR application in Java.
So this is how I
think of the API.
You create a session, which
represents the AR session
that you're running.
Once you've got this,
you update the session,
and it gives you a frame.
Once you have a frame,
that represents your camera
and all the metadata that goes
alongside that, you get planes.
Once you have planes, you're
able to create anchors.
With anchors, you're
able to place content
in your real world.
And our Hello AR app exercises
all of those parts on the API,
and it is on GitHub,
and is great to get up
and running with this.
So going through the code
it's really, really easy
to start an AR Core session.
You just start a session.
We provide a default
configuration file,
which basically turns on
every part of the API--
motion tracking, plane
finding, light estimation.
Simple way to check if
your device does not
support AR Core.
And when we come to render our
application, first thing we do
is clear the frame, and then
we simply call session.update.
One of the key concepts
to understand here
is, when you create
your session,
one of the things you might want
to change within your config
is whether it's using blocking
or latest camera image.
If it uses blocking,
it basically gives you
a frame at the rate at
which your camera runs,
which kind of makes sense.
You don't want to render
any faster than you have to.
You want to make sure that every
pose aligns with every frame.
But for some of
your applications,
where you want a really smooth
update, you have animations,
you might want to just render
as fast as you possibly
can and at the expense
of power and performance.
So once you have a frame, we
have this helper function,
Hit Test, which helps us to
cast a ray into the world based
on a touch location
and see if you have
touched one of these planes.
If it collided, and the
tap was within the bounds
of that detected plane,
then we create an anchor.
And we'll go into anchors in a
little bit more detail later.
So we also have to see if the
frame was actually tracking.
If you have put your hand
in front of the camera
or something, and
you're not tracking,
then you want to make sure that
your intersections are correct.
And then you query the
frame for all the data
that you need to
render your objects.
And don't worry if
you haven't used
projection matrices before, and
you don't know a view matrix
is.
That's OK.
As mentioned earlier, to
render this AR content,
you have to match the field
of view of your virtual camera
with the field of view
of the real camera.
The projection matrix contains
all of those properties
that you need,
and you just query
AR Core to-- the AR Core
session to gain access to those.
And in this example, we
set the range of objects
that you will render from 10
centimeters to 100 meters.
And then the view
matrix is what contains
all of the information
for motion tracking.
That actually contains
the pose where
the camera is in the world.
And then finally,
just getting access
to the lighting estimation--
it's just a simple
accessor call.
Once you've got that
value, you can either
use it for some
logic, or you can
use it to affect your
rendering or your lighting.
And then finally, we loop
over all of our anchors
that we've placed in the world.
And if they're being tracked,
we render those objects.
So if the projection
matrix contained the camera
properties, and the view match
matrix contained the camera
location, the model matrix
contains the location
of that anchor within the world.
With that combined model
view projection matrix,
you have everything
you need to put
the pixels on the right
place on the screen,
properly overlaid on
top of your camera feed.
So talking about anchors,
why do we need anchors?
What is this concept
of an anchor?
So you might think, well,
it's three dimensional space.
Why don't we just
call where you start
the application the origin
and place your objects
relative to that?
So it turns out there's actually
some error in the poses you
get back from motion tracking.
Motion tracking is
constantly updating
its understanding of the world.
And if you use anchors, as
this understanding of the world
updates, the pose of your
models will update as well.
A good example where
you want to use anchors
is imagine walking around
a building in a loop.
When you get back to
where you started,
that drift will have
accumulated over time,
and you really want to
use anchors to make sure
that that all stays correct.
And so if you place an
anchor in the world,
you should also
make sure that you
place the digital object on
top of that anchor as well.
So any time there is
error in that rotation,
then the further away that your
object is from that anchor,
you'll end up with this lever
arm effect, where you're
rotating around a pivot that
is in the center of the object,
and it translates off
from where it wants to be.
And then really think about
these anchors are there
to root your digital objects,
your physical objects
in the world.
If I was going to place
an object on a chair,
I want to create one
anchor for the chair
and place the
object on the chair.
If I want to create-- put
10 objects on my desk,
I don't need to
create 10 anchors.
I should just create one and
place them all relative to it.
And then just again, avoid
using those global coordinate
systems.
You'll have a bad
time if you use those.
So we kind of skipped over this.
Apart from cleaning the
screen, we didn't really
talk about rendering 3D.
Rendering 3D is actually
a really big topic
that I couldn't possibly cover
in a 30-minute talk here today.
But to give you
something to look at,
there are a few options.
You can use Open GLES
directly, and we actually
have some great tutorials on
how to get started with this
on our developer.android.com.
Or you can use frameworks
such as [INAUDIBLE],, which
does a lot of that
heavy lifting for you.
Or you can take a look
our Hello AR sample,
which actually contains some
model loading code and model
rendering code as
well, if you just
want to place some
objects in your world.
So not only does AR
Core work with Java,
it also works with
Unity and Unreal,
and we've done a lot
of work to make sure
that it integrates really well.
Common game engines
like these, they
remove a lot of the
complexity from managing
a complex 3D scene, so that
you can focus on actually
building your application.
So we've ported
Hello AR into Unity.
You can see it there.
It exercises all
of those same APIs,
and we've designed
it so you can easily
get up to speed, use
those scripts and prefabs
in your own application.
And it comes as part of the
SDK when you download it,
and similarly, we have the
same for Unreal as well.
We've done everything
to make building your AR
applications really
easy using the tools
that you're most familiar with.
So another option is WebXR.
So real web standards
for AR don't exist yet,
but these prototypes
allow web developers
to start building augmented
reality web experiences today.
Their experiments will teach
us all what AR on the web
could look like,
which will hopefully
make the real web
standards arrive faster
and be better designed.
These capabilities are
built on top of WebVR,
so if you're familiar
with that, it's
really easy to get up
and running with WebXR.
And this demo that
you can see here also
works in experimental
versions of Chrome for AR kit
as well on iOS.
And so we've talked a lot
about building AR applications,
but the really hard part
is how do I get content?
We all know how to
create content for 2D.
We know how to get images.
We know how to get
text, fonts, videos.
We know how to
create them, and we
know how to use them
in our applications.
So Google have been working in
AR and VR for quite some time,
and we've made creating
this content really,
really easy, which is
why we created Blocks.
So Blocks lets you build 3D
content in VR really quickly.
And even if you don't
have a VR headset,
take a look at our library.
Any content tagged as remixable
is available for download.
So let's take a quick look
at some of the content
that our creators
have built for blocks.
So if you want this guy, a
ramen chef, in your kitchen,
that power is now open to you.
And if you want this
guy to watch over you
at night on your bedside
table, you can live that dream.
It's really, really
impressive, what
people have made using Blocks,
and it's a super easy way
to get 3D content
into your application.
So I encourage you all to
head to our Blocks website,
download it, and give it a play.
It's really, really fun
building this content in VR.
So before we finish,
I'd love to give you
a taste of some of the things
that we've built in AR Core.
[MUSIC PLAYING]
So this has been a whirlwind
tour of the capabilities of AR
Core, and all the information
you need to get started
is there today on our website.
So we've talked about
the fundamentals of AR.
We've talked about the options
you have to build your AR
applications, and we've talked
about how to build content
for those applications.
We're actively seeking
feedback from developers
such as yourselves through
this preview phase,
and we'll be monitoring
our GitHub issues
and other channels
closely so that we're
building a platform that works
for what you want to build.
So you can see that AR marks
the next big shift in what's
possible with mobile devices.
Get started with the
AR Core preview today.
We really can't wait
to see what you build.
[APPLAUSE]
[MUSIC PLAYING]
