[MUSIC PLAYING]
AUSTIN MCCASLAND: Thank
you so much for coming
to our session on
AR as a feature.
And for everyone watching
on the live stream,
thank you for tuning in.
We're really excited
to talk to you guys
about how you can supercharge
your projects and products
using augmented reality.
But before we dive
too deep into that,
let us introduce ourselves.
My name is Austin McCasland.
I am a UX designer and
prototyper at Google Daydream.
And before I joined Google, I've
written a couple courses on AR
and VR product design
and development.
I've designed and developed
some award-winning AR apps,
and I just am really passionate
about spatial and immersive
computing.
DIANE WANG: Cool.
And my name is Diane Wang.
It's nice to meet you all.
I'm a lead user
experience designer.
I work on Austin's team, and I'm
really passionate about working
at that intersection
between digital and physical
interfaces.
Before I was
working at Daydream,
I was designing
products at Nest,
so really thinking about
hardware interfaces,
spatial and physical
computing, and everything
that's in between.
And before that, my background's
in computer science.
But in general,
my day-to-day now
is working with Austin on
rapid prototyping all the time.
So we're super excited to jump
in and share with you what
we have today.
So today, we're here to talk
about augmented reality.
We're excited because AR
can give apps superpowers.
But the key message today
is that you don't have
to make an AR app to use it.
AR has progressed
to a point where
it can be reliably
used as a technology
that any app can leverage.
The ability to spatially
understand and output content
into the world opens
up new possibilities
across every vertical.
So today, we're going
to start by talking
about AR as a technology.
We'll talk about
where it was and has
been in the past few years, and
where it has gotten to today,
and why we have this
message for you all today.
We'll spend some time talking
about how you can evaluate
your app for its AR potential.
And this is going to draw a
lot from our team's experience
working with a lot of
apps, working with AR.
And we'll talk
about different ways
you can think about your
app and where we found
that AR brings a lot of value.
And then we'll spend
the bulk of our time
today talking about different
approaches and processes
so that you can develop
your own AR features.
And now, I'll hand it
over to Austin to jump in.
AUSTIN MCCASLAND: All right.
So I want to start
off just by level
setting on what we mean when we
say AR, or augmented reality.
So, to us, there's
really two key components
that makes AR this awesome
thing that can up-level any app.
And the first of
those is the ability
to understand the world
through the camera.
So when we're talking about
understanding the world,
it could be anything from
little feature points
that are on objects around.
It could be detecting
that a plane is somewhere.
It could also be more semantic
understanding of the world.
So if you lens a banana, we
know it's a banana, right?
The ability to reach out and
have the computer understand
space is super powerful, and
that's one of those two spokes.
The second is the
ability to reach back out
and put spatial content
believably seated in the world.
So what you're
looking at there is
when you put Andy the
Android on a table,
and he's moving
around, and there's
lighting estimation
happening, and there's
tracking that helps
keep it all in place.
And when you combine the
ability to understand the world
through the camera and to
put content back into it,
all these really
interesting things
start to happen in terms of
use cases and user journeys
that you're able to
enable in your product.
So before we dive into
the nuts and bolts,
I wanted to set
context for why we
are talking about
AR as a feature
today, because AR
has not always been
ready to be a feature in all
different kinds of products.
So we're not going to do an
exact historical run-through
of dates and times, but I
do want to bucket eras of AR
from a product perspective.
So the first one, at
the very beginning,
AR was pretty much relegated
to like lab environments,
or you would see art
installations that had AR.
And we even consider some of the
early projection mapping work--
so projection mapping is
using a projector to light up
particular shapes and stuff--
as being these early
first steps of AR.
But if you're a
product person, it's
not that easy to see how you're
going to create a product out
of it, unless you're doing
experiential marketing,
or immersive theater,
or something like that.
And so if you have to get
people to go to a place
to experience AR, it's
kind of hard to build
a scalable product
around it in that sense.
So there's this next
phase, which was--
I'm sure many of you
are familiar with it.
If you're not,
it's sort of where
we're using images
tracked on mobile devices
to pin content in the world.
And the way that worked is
you would have your phone,
and there would be some
sort of image or QR code.
And by using that as
a reference point,
we could put this
digital content
overlaid into the world.
And it was awesome.
It actually represented
this big jump
because, now, you could
deliver AR experiences
on mobile devices, right?
I could put
something on a store.
Someone could download
it, and then they
could have this cool experience.
But there were a
couple of shortcomings.
The tracking wasn't as
good as what we have now.
And if you lost sight
of the image marker,
it would kind of go haywire.
And most importantly, you
had to have this image
marker in space, right?
And that's a really
tricky proposition.
It's hard enough to get
someone to download an app.
But to get them
to download an app
and also print something
out can be challenging.
And so we did see some
adoption of this tech
in marketing campaigns, where
there's physical collateral,
or if there were
instruction booklets,
things like that, where you
would know that your app could
have those markers available.
But the game really changed
with modern AR platforms,
like ARCore.
And this was basically,
from a product perspective,
like a quantum jump.
I remember when I first
saw these new AR platforms.
I was working with the
image maker stuff, right?
And there was always a struggle.
And I saw some
videos, and I just
was like, I don't believe that.
That's not real.
So I went out, I bought
devices, and I tried it.
And I remember sitting
there and thinking,
man, this can actually
be a thing now.
I can release production quality
user experiences with AR.
And the really important
thing about this
is there's no need for markers.
That is a feature
that you can access,
but you can send anyone a
link to download your app.
They can download it, and no
additional effort is required.
They can have the
experience you intended.
And so lowering that
barrier to entry,
combined with a really
consistent output of quality,
has just really changed the
game for AR in a product sense.
And that brings us to today,
which is AR as a feature.
So when these platforms
first hit, everyone went out
and we were making AR stuff.
That's when I made an app
you can paint in the air.
It's really cool, really fun.
But these were these AR apps.
And what our team
has discovered,
because we sort of go around
inside of Google and say,
how can AR unlock new
potential use cases for people,
is that your app doesn't
have to be an AR app.
And in fact, some of
the most powerful ways
we can execute on AR
is including it just
as a feature in an existing app.
And today is the perfect
time to be doing it,
because these frameworks
make it super easy.
When you see AR, it can look
kind of like black magic,
like it's super hard.
You need to be a mega genius
to get AR stuff happening.
But, in reality, ARCore
makes it super easy,
and there are tons
of helpful tools.
So today is the
time when, I think,
any product person
can be evaluating
their product for this stuff.
So AR as a feature essentially
means that the whole app
isn't about AR.
So what you see here
is Google Maps, right?
And Google Maps
is not an AR app.
It helps you get
where you're going.
It helps you search
for awesome stuff,
but it's not all about AR.
However, the team discovered
that, for certain areas like,
in this case,
walking navigation,
it can be really powerful and
enhance the app experience
to use AR in those moments.
And same in your I/O app.
If you pull it out
and use Signpost,
you can get where you're going.
But it's not it's not relegated
just to navigational use cases,
as we'll see later.
So I'm going to kick
it over to Diane,
who's going to tell you
about how you can assess
the stuff you're working
on to see if maybe AR could
do something awesome with it.
DIANE WANG: Cool.
Thank You, Austin.
So let's bring it
over to all of you
and start talking about
how you can determine
if AR is right for your app.
Here are several
principles and questions
that you can ask
yourself as you decide on
whether AR has the potential to
add a lot of value to your app.
And this is all going to
be drawn from our team's
experience, but we'll just kick
it off with the big question.
So we always start off by asking
ourselves the simple question,
why do we want to use AR?
And that really helps
ground us in user value
and thinking about
the specific value
that this suite of
technologies brings to us.
And going a little bit
deeper here, on our team,
we think that there are two
broad categories of experiences
that AR can make shine.
Use these as guidelines,
as starting points,
to answer this question.
So here are the two dimensions.
You may have heard this
in our developer keynote.
The two dimensions here are
helpfulness and creativity.
So these are two
broad categories,
and that's one really
easy way to get
started in thinking about,
hey, does my app fit in here?
Does my feature fit in here?
And just thinking about these
from a broad perspective--
helpfulness is a really great
place for AR to help shine.
Because AR understands so
much of the spatial context
around a user that,
if you're trying
to have an app that helps the
user accomplish something,
AR can do that in a lot
more quicker of a way.
And when you think
about creativity,
AR has a lot of
opportunity here to help
make features shine as well.
You can see that AR allows you
to be a lot more expressive
and engaging when you're
thinking about a creativity
feature.
You can directly overlay
things and output content
into the world and also
have suggestive content
that leverages our
understanding of the context.
But beyond these two
buckets, our team
also has some other
insights that we found,
just by working across
a lot of different apps.
So here's one to start with.
One area that we found
has high AR potential
is when there's
information from the world
that we'd like to
have in our app.
Often, if users need
to look something up
from the phone to
complete a flow in an app,
AR has opportunities to enhance
and streamline that experience.
AR has the ability to capture
information from the world
and make user journeys a lot
easier and more streamlined.
So that was from the
helpfulness dimension.
But when you're thinking
about creativity apps as well,
AR can suggest
expressive content
that's relevant to
the scene and context.
And another category
of experiences
that has high opportunity
for AR features
is when someone needs to
interact with the real world
to use our app.
So if someone needs to
do something in real life
to complete a flow, these
are really great signs
that AR may be useful.
And one example of that
we illustrated here
is, you can imagine if someone
has some specific dietary needs
or preferences, and
they're at a grocery store,
and they really need
to go and fiddle
with the products
in front of them
to understand the products
that are right for them,
AR has the opportunity
to understand
that at a much faster rate.
And we can even output
that in a way that's
really, really natural.
So this illustration shows
that, for example, maybe we
could just highlight that
area that would have taken you
a while to get to.
So I've just covered two
broad categories, but also
two other insights.
But is that all?
Definitely not.
These are not absolute truisms.
And there are often entirely
different AR opportunities
in an app that don't
fit in these buckets.
So you're all going to have
your own special scenarios.
So we encourage you to use
these as starting points,
as ways to start
thinking and having
discussions about your app,
and to try out the experiences.
And so we're going
to go over this next.
But one of the fastest
ways to really understand
where AR can add a lot of
value is to just jump right
in and start making things.
And one last note is
to remember that AR
is a tool and a
suite of technologies
and will add a lot of value
to some problems, but not all.
But our guiding
principle remains.
Always ask yourself, why
is this better with AR?
And now, I'll hand
it over to Austin
to talk about how we
get these to happen.
AUSTIN MCCASLAND: Sweet.
Thank you, Diane.
So I call this
part the fun part.
It's the fun part, for me.
So when we talk about
building AR features,
how do we go about
approaching that?
And our team does
this all the time--
over, and over, and over again.
It's great.
So I'm going to run you
through these three principles.
And these principles, they're
just these lighthouse concepts
that we try to keep
top of mind when
we're developing AR features.
And there's some unique
things in here that sort
of differentiate an AR feature
from a more traditional feature
in an app.
And these that our
prototypes are our sketches.
We consider design
and development
to be intrinsically linked.
And that it's OK to be cool,
but we need to strike a balance.
So I'm going to dive a little
bit into each of these,
starting out with
prototypes are our sketches.
So if anyone has been in a
room with designers, or product
people, or anyone
anywhere, we know
that sketching is this awesome
tool for communication.
And for the purposes
of this piece,
let's think about sketches as
a way that we can assess ideas.
So I might make a sketch,
and I give it to Diane.
And we use that to figure
out, is this idea good?
What do we like about it?
What don't we like about it?
And those things run
through product teams.
Product management,
engineering, design-- everyone
lays eyes on these.
But we've found that, with AR,
understanding whether or not
something is good
or worth pursuing,
it's really hard to
capture from a sketch.
There's all sorts of things
that look great on paper,
and then when you try to go to
execute on them, they stumble.
And it's really tricky.
And so we try to get
to these prototypes
as quickly as possible
because that's really
the baseline for
our AR features,
for us to be able to
say, is this working?
Do we want to continue
on this and refine it?
The next principle is that we
consider design and development
to be intrinsically linked.
And I know, they are in all
normal app development cycles
as well.
However, the relationship
between design and engineering
in AR is incredibly tight.
And in fact, every time
we start a project,
we start design and
development at the same time.
We don't come up with
designs, and then
show them to engineering.
We don't even run like
tight back and forth cycles.
We sit next to each other.
And we'll have a computer
with a development environment
happening, and a computer with
a design environment happening.
And that's because,
right now, there's
all sorts of
opportunities for design
to be really helpful for some of
the harder technology problems.
So an example that
I like to give
is, let's say it's
pitch black, right?
That's a really hard
situation for AR.
We lose tracking.
And so from an
engineering perspective,
that's really hard.
You're like, ah, do
we need depth cameras?
Do we need all that stuff?
But if the design
team is right there,
they can say, actually, hold on.
As long as you can
tell it's dark,
we'll just ask the user
to go turn the lights on.
And we find that there's
all these opportunities
to design around really
hard technical problems,
and that the technology
often informs what's
even possible in a design.
And you really have to
try it and the hand feel.
It's super important.
So the coupling of these in AR--
super critical.
And the last
principle I'll go over
is that it's OK to be
cool, but strike a balance.
So in the design world, we often
call these moments of delight,
moments where we do something
that's really exceptionally
powerful or useful for a user
that was surprising to them,
or these little moments
that you see in animations.
Or just go to dribble,
if you're looking
for what I'm talking about.
But in AR, these
moments exist as well.
And they're actually
often much more powerful.
They're these sort
of moments of awe.
Like if you watched the keynote,
you saw that shark go on stage.
And that's just one
of those moments
that it's not just
delight, it's this.
Whoa!
And we have found in
our testing that that
is really powerful for users,
because it does two things.
One is that it captures
their attention.
And when they're more
focused on our app,
they more successfully
complete tasks inside of them.
And so it actually serves
a functional purpose.
And two is that
we've noticed when
we were working on features
where we actually really focus
on these awe moments
and consider them worth
spending the time on, they
tend to go hand-in-hand
with users being
successful, overall,
even besides attention.
And it really helps
them be memorable
against the competition.
So the other part of this
is, strike a balance.
It can be really
easy to go overboard,
once you have the
AR stuff running
and turn your app into
a game experience.
And maybe you want that,
but you want to be on brand.
And it's possible to maintain
the mood and attitude
of your application
and have AR that's
impressive and exciting,
without distracting
from the core needs
of your users.
And now, I'm going to dive
into just a couple of methods
and techniques we use.
A lot of these will probably
look very familiar to you,
if you've done product
design type stuff.
But I'll give you
some examples of how
we use them in our AR practice.
So they are sketching
and whiteboarding,
rapid prototyping,
and quick user tests.
And I know I just said that
prototypes are our sketches,
but it does not mean
that we do not sketch.
We do a lot of that.
Early in the cycle of
an idea, having a shared
visual for these
spatial computing
concepts is super important.
There have been
multiple times where
I've had like a 30, 45-minute
conversation with the PM,
a fellow designer, or engineer.
And we're both vibing.
We're on the same page,
and we're super excited.
And then I go to draw
what I was talking about,
and it turns out that we
were not on the same page
and that it was very different.
The sooner that you can get
some sort of shared visual
to frame your thinking
around, the better,
because the vocabulary for
describing spatial problems
and the way that users
interact with space
is just really challenging.
And so these visual
supports are super helpful.
We'll also do
storyboards where we
show how users are
going-- we anticipate
them moving through space
with our experience.
And we'll do flow
charts, you name it.
And this is really
just a tool for us
to iterate super fast, before
we even touch those prototypes.
Next up is rapid prototyping.
And I would have to say, if
our team has a superpower,
it's this.
We make prototypes fast.
And you too can
make prototypes fast
because, one, they
don't always need
to be technology prototypes.
As Diane will show you later,
you can do physical prototyping
in space.
And often, that's
enough to assess an idea
and at least get
it off the ground.
But also, there's
amazing tools nowadays.
Like we use Unity
a ton on our team
to just bust out these quick and
dirty prototypes, just to get
the hand feel.
And if our prototypes
are our sketches,
then we do a lot of
sketching, right?
So we try to just get to the
quickest and dirtiest way
to feel it, and try
it out, and assess it.
And if it passes muster,
then we really refine it.
And one of the ways
that we do this refining
is through quick user testing.
And I don't mean doing a really
quick SurveyMonkey user test,
although that could be fine.
It's more like I'll go to Diane,
and I'll tap her shoulder,
and I'll say, hey, try this.
I'll just constantly thrust
new builds of the prototype
in her hand, because it's
really difficult to see
the forest for the trees in AR.
And part of that
is because there's
this physicality to augmented
reality as a medium.
And you actually
develop a muscle memory
from making your app
work the right way.
So you're like, oh,
this is how you do it.
You hand it to someone else,
and then they're like, whoop.
You know?
They're off in space, and
you didn't realize that, oh,
I need to have some
affordance to guide them back
to what they need to be doing.
So lots of quick user
testing and feedback.
And then eventually,
once you think
you've landed on it, yeah,
run some actual user tests.
Validate a feature.
I'm going to kick
it over to Diane,
who's going to run
you through what
it might look like for our team
to go through this process.
And it's a process you
might be able to scalp
for your own
product development.
DIANE WANG: Cool.
Thank You.
So yeah, we're going to go
through the whole process
and take a look at what
it might look like,
end-to-end, for you to start to
prototype your own AR feature.
And so I'll talk through
the steps first right now,
and then we'll walk
through them one by one.
And these are incorporating a
lot of what Austin just said.
There's principles, methods.
And these are very
special to AR,
because these features
are so closely
tied to the physical
and world space
that there's new things
that classic methodologies
in the field might
not really address.
So we generally follow
these steps in our process,
although we always adjust
steps based on what
is relevant for the feature.
And we recommend
that you do the same.
Kind of dig into what I'm saying
and the underlying purpose
behind these methodologies
and pull the steps
that really work for you.
And so, from a high
level, we always
start our process by
jumping into a brainstorm
and opening up the
AR opportunities.
This is where you start to
bring in a lot of people,
get a lot of diverse
perspectives,
bring out the
creativity in the group.
And then we start deciding.
And this is where my
previous conversation
about how you can evaluate
different AR features and apps
comes in handy.
Then after we decide on our
ideas, we start sketching.
And this is where we get the
broad strokes of all our ideas.
Austin talked about
this before, but there's
different types of sketching.
You can work with any
low-fidelity materials
you have around you.
But then we gradually push
it toward physical testing.
And if it's really
applicable, you
might have some apps that
are really closely tied
to the physical world
space and physical elements
that are all around you.
So the fastest way to get a
sense of those interactions
is just to try those out.
And then after that, we kind
of understand our concepts,
so we jump into design and
development simultaneously.
And I'll talk more about
how we do that later.
And another underlying step
that we consider more horizontal
is, we're always
testing early and often.
And that's with the
people around us,
but it's also great
to try to incorporate
your users as soon as you can.
And we always wrap
up our process
by taking the feedback that
we get from all of our tests,
and we prioritize the stuff
that we want to incorporate.
We think about
supporting functionality,
and then we walk out with
a really cohesive concept.
But jumping right in,
let's just start talking
about the brainstorm phase.
And so this is where we open
up all the opportunities.
We have a really great, robust
conversation with our group.
And we really try
to pull together
people of diverse
perspectives, so people
from different functions,
of different backgrounds,
especially from
backgrounds of people
who will be using your feature.
So if people can empathize
with those groups,
try to pull them in.
And if you get a
diverse group, you also
get really creative ideas
on the board as well.
And this photo shows you
what one of our brainstorms
might look like.
Over here, I'm just
putting ideas on the board,
but I'm using a
design methodology,
called affinity
mapping, where we're
grouping a lot of the
Post-its together by theme.
And that really lets us
take a step back and talk
about the themes that are really
exciting, before we get too
caught up into details.
And once we have our
ideas on the board,
we have discussions
about the ideas
that we are most excited about
and have the most AR potential.
And this is where our previous
sections on evaluating AR apps
and features comes in handy.
And this photo shows
what one group of ideas
may look like at the
end of our brainstorm.
We're doing something
called dot voting
where we give each person
a certain number of votes.
And these are dot stickers.
And we use this as a way to
gauge the general interest
and ones that we're most
excited about, as a group.
And we use this as a way to
facilitate our conversations,
and not to decide our ideas.
So try this out yourself.
And one other tip is that
our team has done this a lot.
So we find that giving each
person around three to five
dot stickers is a
really great balance
between trying to get a
nice breadth of votes,
but not spending way
too much time trying
to get those votes done.
And now once we've
decided on our ideas,
we jump into sketching.
So we sketch to get a first-pass
glance at the solutions
that we're excited about.
In this photo, I'm
storyboarding a complete journey
of an AR feature using
whiteboard and marker.
And this is where
you can see that AR
starts to have a difference in
classic design methodologies
as well.
This is not a sketch
that you would typically
have for a 2D app,
because you can
see that I'm sketching
a user in world space
with elements that are
pinned in world space.
So you can see that,
even at this stage,
we're starting to think
a little bit differently.
And you might have to pull
out your perspective drawing
skills.
And so we encourage
anyone to use
any low-fidelity and scrappy
materials that are available.
You can do pen and paper,
whiteboard and marker
to just try out the
entire user journey
and take a look at the
feature at a first glance.
So one unique part
about working with AR
is that UI elements interact
with physical elements
in the real world,
so it's a great idea
to try out your designs
through physical prototyping.
So here's a really
great photo of something
that our team tried out.
We have these UI
elements that we wanted
to have pinned to these fruits.
And so in order to understand
a few factors, like scale,
legibility, and how
they look like and feel
like in world space, we just
created and started prototyping
with physical elements.
So paper is great for this.
We've worked with LEGOs.
We've worked with clay.
Those are really great
low-fidelity, physical things
to start working with.
And once we've gotten a good
look at initial designs,
we jump into design and
development simultaneously.
And Austin talked about
this a bit before,
but our team really
enjoys working
with Sketch for 2D UI
designs, and Unity,
to develop our
experiences quickly.
And at this stage,
we work through a lot
of iterations of our concepts
as questions show up.
We usually have designers
and developers partnering up.
So you have that really
nice collaboration and just
back and forth of
questions along the way.
And this step is
incorporated into some
of the previous steps, as well.
But testing early and often is
really key to a successful AR
feature, since there are so
many edge cases and interactions
that we've never encountered
before in 2D designs.
And here's a photo
of Austin testing out
a prototype in a lightweight
manner with real world
objects--
AUSTIN MCCASLAND: Beautiful.
DIANE WANG: --in our office.
Yeah, just try to do it
really early, really often.
And continue testing
with users all the way
through development.
I know we're talking a lot
about rapid prototyping
for early concepting of things
that are just kicking off.
But something that's
really important
is work with your
stakeholders, or whoever
is involved in pushing
your feature to launch.
Work with them along the
way to share your insights,
but also push them to also
continue testing along the way.
And here's our last
step for today.
Once we have a lot of
feedback from your colleagues,
from users, we bring
this feedback back
for our final polish.
And this is where we walk
through all of the remaining
pieces for a cohesive feature.
And some examples of what
we'll be working through
in this phase include,
notifications,
edge cases in
different contexts,
offline scenarios,
and scalability.
But now, I'll hand
it back to Austin
to bring everything together.
AUSTIN MCCASLAND: Cool.
Thank you.
So upgrading your app with ARj--
I hope you guys do
it, because it's rad.
Let's pull together everything
we talked about today.
So if you are a product person,
or a designer, or an engineer,
or anyone who cares
about the space--
maybe you have an
idea for a product--
really start thinking about
AR and what you're working on.
And you should really
just evaluate your own app
for AR potential.
Think about, are
there ways that I
might be able to make this
thing easier to do for my users,
or more exciting
for my users to do?
Or is there something
that we never
could have done before
that we can show them
in the real world?
And it's not right
for every problem,
but if you can
answer the question,
why is this feature
better in AR,
then you should do it in AR.
The next is that you
won't know if it's
good or not, until you try it.
So try stuff out.
And I'm not kidding.
The suite of tools that are
available for creating quick
features that you can
test out, it's amazing.
So I'm a designer,
and I do some coding.
And I make these
all day, every day.
If you're an engineer,
go on Unity's website
and check it out.
Or do native stuff.
It's a lot less
intimidating than you think
to build these AR experiences.
Just get your feet wet.
I bet it's way more fun
and easy than you'd expect.
And ARCore makes a lot of
the hard stuff really easy.
And finally, keep on
testing, and testing,
and testing, until you're
confident it's awesome.
And deploy it.
Thank you, very much, for
coming to our session.
I'm Austin McCasland.
DIANE WANG: I'm Diane Wang.
AUSTIN MCCASLAND: And we
hope you guys enjoyed it.
And build some super
powered apps with AR.
Cheers, you guys.
DIANE WANG: Thank you.
[APPLAUSE]
[MUSIC PLAYING]
