Jeffrey says I've got
a very strange job.
I don't work at Google at all.
I barely know what's
going on over there.
I work at GV, Google
Ventures, as he says,
which is an independent
venture capital
arm that's funded by Google.
So we invest over $400
million a year into startups--
at this point about
300 startups-- and I
get to work with those
companies across a wide range
of industries.
We're talking about the
companies you'd expect,
consumer internet companies
like Medium or Slack,
as well as all kinds of
companies you might not
expect, a lot of enterprise
businesses, security companies,
and a lot of life
sciences businesses.
Last year we did a considerable
amount of our investing
in life sciences companies.
I get to work with companies
like Flatiron Health--
it's a big data oncology
company in New York--
or Quartet Health, which
works on mental health.
It means I get a really nice
broad view of the design
and entrepreneurial
community, where
business and design intersect.
And every week, I try to leave
one day open for office hours.
So on Thursdays, I'm generally
meeting with five or six
companies in a
single day, and I get
to hear about their
biggest challenges.
And what typically happens--
say it's a new company that
I've never met before--
usually, we'll get together,
they'll come into my office,
and they look nervous.
It's really surprising.
Right?
I get to work with
entrepreneurs, heads
of product, CEOs, and these
are people who are generally
very confident in life.
They walk into the room, and
they know I'm a design partner.
They don't know a ton about
design so they walk in,
and you can tell.
I do a lot of research
studies, and so you can
tell when people are nervous.
And they've got that look.
They're sitting
back in their chair
away from me, arms
across the chest,
looking a little bit skeptical
like they've gotten invited
to the principal's office.
And usually, the way that
the meeting typically starts
is I'm like, OK, so
what's on your mind.
And they're like, OK, so
we've got these design issues.
And it's usually stuff
like user interfaces--
say their onboarding
flow is inefficient,
and they think they
can make it better--
or brand questions like,
oh hey, been in business
for a couple of
years, and now we're
really thinking about that
brand that my cousin made.
And maybe we should
do something better.
They talk about these things
that are fairly surface level,
and we get into them.
And it's really interesting.
And then about halfway
through the meeting,
I usually ask this
magic question.
I'm like, oh OK, that
was really interesting.
Those are great challenges,
but what keeps you up at night?
And it's really, really
funny because there really
is a pattern to this.
They look kind of astonished.
A designer just asked them
what keeps them up at night.
And then they relax.
Arms come down off the
chest, they lean forward,
arms on the table, and they're
like, oh, you wouldn't believe.
It's all this stuff.
And they start talking
about hiring and retaining
tech talent.
They start talking about
building new features
and whether or not
it's going to piss off
their current customers or
attract the customers they're
hoping it will attract.
They talk about moving
into new markets,
especially the--
we work in Europe,
and the European
companies, when they're
thinking of coming to
America are like, this
is popular in Europe.
I just don't know if this
is going to translate
to the American market.
And they talk about things
like funding rounds.
So these things that are
really keeping them up at night
are very, very different
from the things
that they perceive
as design problems.
And they're shocked
that a designer even
asked them about them.
So I'm genuinely concerned--
and I really mean this--
I'm genuinely concerned that
designers are off on the side
doing design while everyone
else in the company
is doing the business and that
we're working on things that we
perceive as important,
design stuff, and the rest
the business is sweating
bullets about a whole bunch
of other issues.
And if we're not
working on them,
we're really leaving
ourselves off
on the side fairly powerless
within an organization.
So the way I think
about this a lot
is like that magic question.
I'll repeat it again, the
what keeps you up at night.
You wouldn't believe
how good that question
is when you're talking
to people in other parts
of your business, whether
it's your HR person,
your CFO, your head recruiter.
Go ask your recruiter what's
keeping them up at night,
and you'll be shocked what
they're talking about.
But then you stop
and think about it,
and you're like, oh,
hiring engineers is totally
a systems design problem.
What perception do engineers
have about our business?
Where did they learn about us?
What do they think of us
compared to our competitors?
How do we approach them?
What's the thing we say to them?
What does the job description
say, and in what order
does it say it?
When someone's comparing
us to businesses like ours,
why are they choosing the
competitor instead of us?
Why are engineers
going to a Facebook
instead of to our large startup?
This is a systems
design problem,
and almost no designers
are working on it.
We're off working on marketing
websites and apps and all
these things that we
perceive as important.
But if your number one issue in
your business is hiring talent,
it just makes sense
that design should
be going and working
on that problem
if we've got something to offer.
So I've been
thinking lately a lot
about where I've seen
design being really, really
effective in businesses.
And design done right can be the
scientific method for business.
All those ideas that are
happening in companies,
all those things that
are keeping people up
at night, design can help bring
more certainty to those ideas.
I think a lot about
ideas as inklings.
They're floating all
around a business.
Every time you go
around a business,
and you're talking to
people, everybody's
got ideas about how
to make things better.
Right?
HR's got ideas on how we
could be hiring engineers.
Engineers have ideas about how
we should be hiring engineers.
Everybody has ideas on how
we can improve our product.
But they're really
just inklings.
They're these little
rough-edged kind of loose ideas
that we all argue about
in meetings all the time.
And there's very, very
little certainty about it.
It's really hard to pick the
good ideas from the bad ideas.
And what happens
a lot of the time
is when you describe
an inkling verbally,
what comes out of your mouth and
what I'm picturing in my head
are wildly different things.
All those times we've
been in stupid meetings
where everyone's arguing about
what something will look like
and how it will work and
how customers react to it,
we're all just
bullshitting ourselves
because we're describing
things in a way
that we can't
understand each other.
We're talking right
past each other.
And I think design has
this superpower that
lets us bring those inklings
down, give them bones, give
them some shape, prototype
them, and have them better
understood.
And now, we can
start evaluating them
in a much more rational way.
So the ways we typically
talk about hypotheses
is we argue about them.
So we have this problem
where we talk to each other
and we're describing
the merits of something.
And the worst of all
is we're guessing
what customers are going to do.
I've been designing for
almost, what, 21 years.
The only thing I've learned is
to be wrong faster because I
have no idea what the fuck
people are going to do
when I put something out there.
And then, typically, in this
new environment we have,
we try to launch and
measure very, very quickly.
The ship early, ship often
kind of method, the move fast
and break things
in Silicon Valley.
And the risk with
this is that what
happens is we typically
make our best guess,
we build as quickly as
we can, we launch it,
and then we measure it.
This is how the whole iterative
process is supposed to work.
But the big risk-- and
I see this all the time
in businesses--
is that you're really
testing one idea.
So we've had four or five ideas
when we were sitting around
in a meeting room together,
but we somehow intellectualized
with ourselves and
argued quite a bit.
And we're like, OK, this
is what we're going to try.
And usually, it's not
the most extreme option.
Because the really crazy idea,
everyone argued about it,
and we're like,
no, that will never
work so we don't even try it.
It's always a safe
option because if we're
going to launch out in the wild,
even to a small group of users,
we don't want to
fall on our face.
And so we're often not really
taking a swing for the fences.
Then it always takes longer
than we think to make it,
unless your engineers are way
better than the engineers I've
ever worked with,
and I've worked
with some great engineers.
We're never very
good at scheduling
how much time it will take.
And even if we build something
in two or three weeks,
that's a long time and
very, very expensive.
Once we actually started
building the thing,
even if we're starting to guess
that it might be a shitty idea,
we go and launch it anyway
because we feel committed.
We've got some cost.
We hope to get
really clear data,
but it's almost
always inconclusive.
It's great if you've
got a home run
or if you completely strike
out, but we're usually
right in the middle somewhere.
And we're arguing
about, oh, it would
be better if we only
improved x, and we
argue past each other about
what the data actually means.
And by the time
you've launched it,
it's almost impossible
to bring it back.
Even if 90% of people
don't use the feature
that we just designed,
10% of people
will come with
pitchforks for you.
And then we're
supposed to iterate,
and we almost never
do because we move
on to the next shiny object.
So what I think of
when I say design
is the scientific method is
what we really want to do
is shortcut that process.
If we can test
several ideas in a lab
and actually get some
measurement around them,
we'll at least get directional
feedback about which
ideas seem to hold merit.
And so I think of this as
if design can basically
recreate the
benefits of a lab, we
can be very, very effective
within a business.
I'm going to walk you
through three levels of labs.
There's one version, the
industrial grade lab,
which requires a lot of
effort and a lot of people.
I'm under no illusions.
Even the companies I work
with, which are all fancy pants
venture-backed
businesses, even there,
it's a big ask for a designer
just to walk in the door
and be like, I want to make
serious process change.
And everyone's like, f you.
We're busy here.
We're doing stuff.
So it's going to be
really hard to get by
and to do a big heavy process.
So I'm going to show you some
really lightweight ways of what
I think of as lab type
design that you can build up
to be very, very effective in
a more process-driven approach.
So the basement lab,
most basic thing.
What you want to do is help
teams see their own inklings.
All those times
people are arguing
with each other in meetings,
if you can give some shape
to those ideas so we're actually
talking about the same thing,
you're being very,
very effective.
There's a really great quote
from one of the Kelley brothers
who founded IDEO.
He said, "If a picture is
worth a thousand words,
a prototype is worth
a thousand meetings."
Somebody really
appreciates that.
Nice.
I love this quote.
I think it's very, very true.
And the nice thing is
all you need is you.
You don't need to get in
cahoots with anyone else
to actually make this effective.
And you need really basic
tools, and you don't need
to ask anyone's permission.
So I'll give you one example.
This is a little app I actually
made with a friend of mine,
Kevin Rose.
It's a little
intermittent fasting
app for people, health nuts.
What happened is-- and
this is so typical.
So the CEO came to
me, and he's like, oh,
I've got all these plans, and
my engineer doesn't really
understand what I'm doing.
And I think it's a good
idea, but I can't tell.
And he brought, basically, a PRD
to me, a product requirements
document.
And he kind of sketched
out some crappy interface
that didn't make any sense,
and he had a really hard time
showing this to
anybody and getting
buy in, even his own engineers.
And they were all confused like,
well, this will never work.
I don't understand it.
What do you even mean?
And instead of
sitting in a meeting
and arguing with him about it
for hours, I listened to him,
we talked about it
for half an hour,
and I went back to my desk.
And I was like, hey, Kevin,
just give me an hour or two.
I just went back to my desk.
I took what was
basically just text.
Oftentimes, when we get a
PRD, it's just somebody either
had just talked out loud about
what they think their idea is
or they had sketched down some
really crappy rough version
of it.
But as a designer, you've
got this sleight of hand,
this superpower in design
that you can make something
appear real very, very rapidly.
And so I went back to my
desk, and I opened up Sketch.
And I sketched out what a real
interface might look like.
And I opened InVision.
You can use Marvel,
use Flinto, use Framer,
whatever tools you like.
But the trick is to get it
from this kind of inkling,
this really rough-edged
idea into what
seems like a real interface,
get it on a device,
and then hand it
back to the person.
So all those times you're in
meetings and someone's spouting
off about an idea, if you
can just shut your trap,
open up your laptop,
and start prototyping,
you come back to that person--
your CEO, your product manager,
whoever you sit near--
and be like, oh
hey Joe, I really
loved when you were
talking in the meeting.
This is what I
think your idea is.
And all of a sudden,
the two of you
are collaborating
about a real thing.
All of a sudden, you can
take this around and show it
to engineers.
You could show it
to the salespeople
in your organization.
You could actually
start getting feedback,
and you're all talking
about the same thing.
Don't get me wrong.
This isn't a great interface.
This isn't what we
ended up shipping,
but this is the start
of the conversation.
This is like we ran a little
experiment in the lab,
and now we're all talking
about the same thing.
I think of this a lot.
Your designers talk a lot
about paper prototyping.
And then you hear
designers, especially really
technical designers--
I suspect like many of
yourselves-- who really, really
want to get into
code and start making
something that's very real.
But prototypes are often
best in this Goldilocks area,
where they're real enough that
it feels like the real thing,
but you didn't invest so much
time in it that you're not
willing to throw it out.
I think of prototypes very
much as throwaway interfaces,
something that's good
enough that we can all
be on the same page, evaluated,
and have the humility
to be like, oh, that
was a terrible idea
and go throw it in the trash.
Or oh, that's interesting.
Let's change it in
significant ways.
So I really think
about the basement lab
as just spending a few
hours on something.
Really try to time
limit yourself.
The more time you give
yourself, the more room
you have to hang yourself.
And then two weeks
of work later,
once we all agreed
on the basic ideas,
we made a real interface.
This fasting app is now used
by 120,000 people a week.
It's crazy.
There's really neat--
totally an aside,
but it was downloaded
a zillion times
right before Ramadan, something
we never really expected.
So people are using it
for religious fasting too,
which is really interesting.
The slightly more
advanced version of this,
what I'm going to call the
high school science lab.
What you really want to do now
is make a prototype like this,
but go test it with
realistic customers.
In the basement prototype,
we're still just guessing
whether or not customers would
actually use it, whether people
would function with this thing.
Research to me--
I work with a ton of
companies every year,
and one of my colleagues
is a research partner.
All he does is user research
with portfolio companies.
And honestly, it's
like our secret weapon.
All these companies
are running around
like chickens with
their heads cut off,
guessing what customers
are going to do.
And if you can give them
some directional certainty,
it's very, very powerful.
And so if you can
get your prototype
in front of customers,
you're much more effective.
In this case, you need you
and a product person, usually.
Here's an example of this
high school lab in action.
I work with a company
called Managed By Q, just
a New York-based startup.
What Q does is they
do all those things
that people in denim
shirts do in your office.
They will clean offices.
They will hang
pictures on the walls.
When you need to install a
television in a meeting room,
they'll send
someone in to do it.
Hanging light fixtures.
All these basic things that
keep offices up and running.
And huge enterprises have
teams of people that do this.
They do this for all the small
and medium sized businesses
that don't yet
have a giant staff.
And when I was
talking to JT White--
so JT, other than having a
fancy pants profile photo,
JT's their head of design.
And he was working with
their head of product,
this guy named Chris.
And Chris was like, listen JT.
If you're going to
become our VP of design
and really belong in the C-suite
in our business, what I really
need you to do is to help
me see around corners.
And I was like, yes, this
is how I think design can
be so effective in a business.
And an example of this is
Chris had this challenge
on the product team.
They were thinking that
office managers who
are their prime
customer, so the person
sitting at the desk at
the front of the office
is usually the one
ordering services in.
They thought office
managers would
order more services
if prices were
more transparent and concrete.
So you could imagine
I want to put
a light fixture into my office.
The old system at Q
worked like, OK, tell me
about where the
light's going to go.
Do you already own the fixture?
Where do you need it installed?
And you'd say, answer
these questions.
You'd submit it.
And then it would go off
and say, in 24 hours,
we'll come back to you
with a price quote.
If any of you guys
have ever worked
with a general contractor,
that's basically how it works.
You're like, oh, here's the
parameters of the project.
You go back and do some math.
And then come back
to me with a price.
That's a really
frustrating process.
It's not like going and
buying a widget at Target.
You can't just go and
buy the damn thing.
I'm busy in my life.
I just want to buy it.
And they were like, I wonder if
we could start guessing, based
on those parameters, within a
tight enough pricing scheme,
we might be able to
just give you the price.
It would be a lot of
work, but maybe we
could sell a lot more stuff.
And so JT took that idea,
and he prototyped it.
This is literally
one day of work.
JT and his design team went
and they're like, well,
let's just imagine what
this would be like.
And they sketch
out this interface.
And it's about eight
or nine screens.
It's like you log into
the app, and then you
go through a few questions about
installing the light fixture.
You take a picture
of where it'll
go so they can know what type
of material's in the ceiling.
And then they give you
a guaranteed price.
This is all kind of magic
waving your hands product work.
They're assuming that they
can do a bunch of things.
But the way Chris was looking
at it, the product manager,
he was thinking, well if
even this magic version
doesn't work with office
managers, why the fuck should
we waste months trying to figure
out if we can logistically
even make this function.
So he wanted to
measure twice before he
spent all the time
actually cutting and making
the whole system.
So it's a huge amount
of engineering effort.
It's a huge amount of
operational effort.
It was one of
those inklings that
was keeping him up at night.
He's like, shit, this seems
like such a big opportunity,
but it's such a huge investment
of time and resources
to actually pursue it.
Well, let's pretend it exists
and see if even if it did
exist, if it will be successful.
So they made this thing,
and the really cool part
is JT arranged for five office
managers to come in on Friday.
So Monday, Chris gave
him the challenge.
He prototyped it in a day.
And on Friday, five people
came in, but not just anybody.
This isn't like
coffee shop testing.
If you read stuff
like The Lean Startup,
they'll encourage you to go
out and just go to Starbucks
and get some people
using your software.
That's OK.
It's better than sit in
your office pontificating.
But the best thing
to do is actually
go get realistic
people, and so they
brought in actual
office managers who
actually understood Q already--
because that was at least
one place to start--
and were talking to them,
basically, about whether or not
this system worked for them.
They did one-on-one
user studies.
So this isn't rocket science.
Pretty simple studies.
This is Michael, my
research partner,
with a study participant.
No fancy stuff.
No one-way mirrors.
Any of you guys ever
used one-way mirrors?
Seriously, everybody
watches cop shows.
See, yes, somebody's
got their hand up.
Everyone watches cop shows.
They knows someone's back there.
It's freaking creepy.
[LAUGHTER]
We just do this
in a meeting room
with a little bit of basic
equipment, usually a device.
If we're doing a mobile thing,
we use a document scanner
over the top of the device.
But if you're
using a laptop, you
don't even need those things.
And we use GoToMeeting
and a laptop.
We keep it really simple.
In this case, it turned out
that office managers were like,
oh, finally I can
order this way.
I've been meaning to get a bunch
of stuff done in my office,
but I wasn't doing
it because it's
too much of a pain in the butt.
And if I can't get
a guaranteed price,
I can't get my CFO to sign off
on actually doing the work.
And they're like, oh OK, cool.
So they invested tons of time
in actually making this work
and drove up their
business significantly.
It's a great investment
of their time.
So the industrial
grade science lab
is both of these things I was
just talking about on steroids.
So what you do is create
the prototype through group
collaboration, so actually
working with a bunch of people,
not you off in the
corner doing design work.
Real design work at
real organizations
gets done in mixed groups.
It's almost never some genius
designers off on the side.
And then going and
testing with customers.
We wrote a whole bunch
of articles about this
at gv.com/sprint, which
are all out there for free.
We call this a design sprint.
And we even wrote
a book about it
if you want to pick
up a hardcover book.
I'm going to give you an
example of a sprint we
did with a company
called Savioke, which
is a company in our portfolio.
So Savioke is a company
that makes autonomous robots
for the service industry.
They looked at this issue.
So this is a hotel lobby.
Whenever you've
been in a hotel, I
bet it never looked like this.
This is what a hotel lobby
looks like at 3 o'clock
in the afternoon
when no one's there.
In the morning and
in the evening,
everything's crazy down there.
When I was checking
in last night,
I waited for 20
minutes just to be
able to get the key
card to my room.
And at that time of day,
it's very hard for the hotel
to offer quality service in
the rest of the building.
So you've just checked in, and
you forgot your toothbrush.
And you're like, I just got
off an overnight flight,
I'm up in my room, I
forgot my toothbrush.
You call down to the desk, and
they're off their feet busy.
It takes them 45 minutes
to get you a toothbrush,
and you're sitting there
with furry mouth for an hour.
And this was an issue
that this company
thought they could solve.
And of course, like any
good [INAUDIBLE] company,
they were like, do you
want to solve that, robots.
So they made this robot
called Relay, and Relay has
a compartment in the top
that you can put stuff in.
It's about waist height, and
it can operate autonomously
around a hotel.
So it can just wander
its way around the hotel
and come to your room
and deliver something
like a towel or a toothbrush.
And when Savioke
came to my team,
they were about a
month and a half
from deployment at
their first hotel,
the Aloft Hotel in Cupertino.
And they were like,
guys, we really
want to do a design sprint.
There's a bunch of stuff
that's keeping us up at night.
And we're like, cool,
I'll work with robots.
That sounds awesome.
So they came into our office,
and the first thing we did
is we gathered a team.
So this is the
biggest difference
between a design sprint
and the simpler lab systems
I was talking about earlier.
You have the people
you'd expect.
You've got designers,
one of the guys
had an audio background,
audio technician background.
You have a roboticist
and an engineer.
You also have these
people you wouldn't always
expect to be involved in
ideation and prototyping.
You had ambassador.
So Zumi, the woman
on the left, she
spent all of her time basically
doing customer service
with hotels, and she
was one interfacing
with the actual
people in the hotels
who'd be operating the
systems and teaching them
how it'd work.
We had Steve Cousins,
who's the CEO of Savioke,
longtime roboticist.
And really importantly, we
had their operating officer
who was really worried about
the business of how they would
actually succeed as a company.
And we asked them, OK,
what are the hypotheses
we want to test, high risk
or high reward things.
The things that
are down the middle
that you're probably
going to do anyway,
we don't need to
test them so much.
You'll be able to get
that data in the wild.
What we really
want to do is test
the things that are really
keeping you up at night.
And in their case,
it was they were
about to launch into hotels.
And they were sure that
they could actually
get the thing out there and
that it would operate around
the hotel.
So in their own office,
they could test how
it would work with elevators.
They could test would it
run over people's feet
and they could
make a safe device.
They could get it to
open and close the door.
So it could do all of
the basic functions.
But what they were
really worried about
is how people would
actually react to the thing.
Seeing a robot just
wandering around in the wild
is not something
we're used to yet.
And so what we did is on
Monday, before we really
even knew what we were going to
do, we created time pressure.
And what we did is we scheduled
five customer interviews.
So it's on Monday.
We have no idea really
what problem we're solving.
We have really no idea how
we're going to solve it.
We're already scheduling five
people to come in on Friday.
It scares the pants off you.
Part of the reason people ship
early and ship often and do
agile methodology is to
light a fire under your butt
to get stuff done
fast and to force you
into making quick decisions.
And we like recreating
that pressure in a sprint.
So in this case, we were
very particular about who
we want to recruit.
We didn't want just
anybody off the street.
If you're a homebody who
never stays in a hotel,
you have no idea what you would
do if you forgot something
when you went to the hotel.
We also don't want
road warriors.
Road warriors never
forget anything.
They leave a bag
packed all the time.
I travel just enough
that I basically
have a suitcase that's
just constantly packed
with toiletries and my basics.
I just pack that
stuff up and head off
to a conference like
this, and I'm fine.
We wanted somebody just
in the middle, somebody
who travels at least five or six
times a year mostly on vacation
and will forget
stuff all the time.
They're right in our sweet spot.
So in this case--
and this what we often do--
we list it on Craigslist.
Then we said, hey,
we're doing a study.
If you'd like to make $100 or
$150 and do a study with us,
that'd be great.
And then we get them
to fill out a form.
This is a really important part.
There's a bunch of questions
buried in the form.
We use this as a screener
to make sure that we get
the people we're looking for.
So we ask these sideways
questions about travel
and a whole bunch of
things so they can't
know what we're hunting for.
And then we're selecting
out customers who
travel the right amount for us.
Then we find the
right challenge.
So we sketched out
the entire map.
So what happens?
What are all the
times people actually
interact with the robot?
If your biggest thing that's
keeping you up at night
is how people will react when
a robot just randomly comes up
to them, there's a whole bunch
of places that could happen.
You walk into the lobby of
the hotel for the first time,
and there's a robot
just running an errand,
and your just like,
whoa, what just happened.
That's one opportunity.
One other big risks was when it
gets in the elevator with you.
So it's actually
riding in the elevator
with humans, which is, all
right, another little aside,
but it's kind of cool.
An elevator is a bit
like a basic robot.
And in every room
like in this building,
the elevators have a server
room off to the side.
There's literally
a rack of servers
like you'd expect in
a normal server room.
And what Savioke does
is they put another rack
into the server, and the Savioke
robot talks to that server.
That server talks
to the elevator.
And so while the Savioke
robot gets its instructions
and it's like, oh I'm 20
seconds away from the elevator,
it's calling over to the
elevator being like, come
down for me.
I need to go to five.
Elevator comes down.
It slides in to the
back of the elevator.
It goes up to five,
and it drives out.
But we have all
these human motions
that we indicate that we
need to get off an elevator.
We can tap somebody on the
shoulder, we can make a noise,
we can cough, we can shuffle.
We all do these things.
You kind of jiggle your luggage.
A robot can't do
any of these things,
and it certainly
can't touch you.
We're very, very worried
about hurting people.
So that was one risk
that we could've tackled.
But in the end, what we
decided to focus in on
was that moment it
shows up to your room.
What does someone think?
Because the biggest
function for us
was to know how satisfied
somebody is with the service.
And the really, really
interesting part
is that when we had everybody
in the room on that Monday
and we were asking what
their biggest challenges are,
their COO looked at us and he
goes, listen guys, if we're
going to be successful as
a business, what we really
need to do is improve
Guest Satisfaction Index.
And we're all like, what.
What's a GSI, man?
Turns out, Guest
Satisfaction Index
is the main factor that hotels
use to benchmark themselves
against each other.
How happy are the
guests in our hotel?
And they are tracking
this on a weekly basis
and making sure that
within their own hotel,
that they're growing the Guest
Satisfaction Index and that
between hotels within a chain,
that they're tracking well.
And that even among
their peers, that they're
within a high percentile on
the Guest Satisfaction Index.
So if that's your most
important number as a hotelier,
if you're building robots
for the hotel industry,
you're golden standard is to
increase the Guest Satisfaction
Index.
If you increase the
Guest Satisfaction Index,
you can drive up deployment
of your robot significantly.
Super useful information
we never would have
got if it was just a bunch
of designers and engineers
sitting around trying
to come up with ideas.
And we were like, OK, the
moment we could really drive up
the Guest Satisfaction
Index is at that moment
that you got service.
Right?
And we're like, oh, if we could
make that delight not just
functional--
they knew they could
actually create
the function of delivering the
towel or the toothbrush to you,
but if we could drive that up
and actually be delightful,
we might have a real piece
of gold on our hands.
So the big question
then, is if we
are going to improve the
Guest Satisfaction Index,
how should the robot
actually behave.
How can we actually delight
people with a robot?
The problem is the Savioke
robot, what you call Relay,
isn't Wall-E. It's not C3PO.
It's not are R2D2.
One of the big issues in
robotics in the real world
is that people who have
never experienced a robot
have strong expectations
of what a robot can do.
We've all watched
science fiction,
and robots are amazing.
Wall-E looks like a
simple industrial robot.
Wall-E is wildly complicated.
Articulating arms,
expressive eyes,
can understand English language,
it's very, very sophisticated.
The Savioke robot,
on the other hand,
doesn't even have a microphone.
It's not listening to you,
let alone understanding you.
All those times you've yelled at
your Siri or your Google Home,
it's because you expect
that it can understand you.
But it just can't.
We're not very good
at that stuff yet.
So we thought, can
we delight people
without setting their
expectations too high of what
the robot can actually
do, without creating
a cognitive dissonance
with the robot.
So we brought in the
robot into the office.
We snuck it in under a blanket
because they didn't want Google
to know anything about it.
Drove it into our
office, and of course,
we took a selfie
with it right away.
[LAUGHTER]
We're very excited.
And so one of the big risks
that had been keeping them
up at night-- and I know this
might sound a little silly--
is whether or not to give
the robot personality.
They had debated this.
This is that thing I was talking
about where in meetings, they
had debated this and debated
this and gone back and forth
and tried to imagine what
it would mean for the robot
to have personality.
And when they brought
it to our office a month
and a half from
launching, the robot
just had a text interface
that was very functional.
It would say exactly what
the robot wanted you to do.
But they thought if we
could give it personality,
you might be able to drive up
that Guest Satisfaction Index.
But if we give it
personality, we
might have people
abuse the robot.
It's really a thing.
I swear to God, when
you talk to roboticists,
robot abuse is a problem.
They're worried you'd
talk to the robot
and be like, thanks robot.
And the robot would be
like, just staring at you.
And you'd be like, fucking
robot doesn't listen to me,
and then you'd knock it over.
[LAUGHTER]
Maybe not that extreme, but that
you'd be frustrated with it.
And it wouldn't
actually be delightful.
All right.
So what we did is we found
more than one solution.
On Tuesday, we spend the whole
day sketching down ideas.
We do almost no
group brainstorming.
Group brainstorms
are a waste of time.
The strongest voice always wins.
And the biggest problem
is, in your company,
think about how many
times you actually
had three hours to sit down
and articulate an idea.
Almost always
we're in situations
where we're shouting
out ideas or that we're
running between meetings.
And we never have time to be
like, oh, if I was presented
with this problem, say,
giving a robot personality
and creating a delightful
experience at the door, what
would I do.
You never have time to really
think it through and articulate
a fully formed idea.
So we go through a
bunch of exercises
where we start
very low fidelity.
And then we try and go
wide and then go narrow.
All this is described
in those free articles
online if you want to read them.
But the nice thing is it's
like a library atmosphere.
We're all sketching down.
And by the end of
the day, we all
have made little
sketches like this.
It's like a little cartoon.
This happens, and then
it results in this.
And the final result is x.
Interfaces are almost
never one screen at a time.
In this case in
particular, it was
a lot about how the robot would
actually act rather than what
the screens would show.
In the end, we end up with
10 solutions at least.
And we put them all
up on the walls.
The next thing is we try to
make good decisions about what
we're actually going to try.
Again, no group voting.
We're not like,
oh hey, what does
everyone think of this idea.
Everybody, how
genius is my concept?
One of the real
dangers of designers
is we're really good at
talking about our design
and convincing people that
a shitty idea is actually
a good idea.
All the ideas have to
stand on their own.
So people walk
around quietly again.
There's no talking.
We really discourage
people from debating ideas
and evaluate the concepts
all on their own.
Because the person had
enough time the day before,
the concerts all have
real text to them.
There's no lorem ipsum.
There's no, here's a picture.
It's a box with an x in it.
Draw a picture.
I don't care if
you're Picasso or not.
Draw is this mountains?
Is this an informational graphic
of a human doing something?
Draw a stick figure.
Make it make sense.
It means that we do
the silly thing that
looks like a bunch
of insolvency crap
where we put little stickers
next to all the good ideas.
And this really helps
people quietly come together
on the best ideas,
and you see people
gravitate towards things.
And then we let the CEO
or the head of product,
the most powerful
person in the room,
do super voting on the ideas
that they want to see built.
We're trying to replicate
the power dynamic
of a real organization.
We don't want a fake the
organization to some democracy
that it's not.
I was working with Slack.
The CEO of Slack, who's
a good friend of mine,
he is a very
strong-headed person.
He's very intelligent,
very decisive.
He's excellent.
If he was the only
person in the room who
thought an idea was good, he
would still come back to it
and want to see it made.
So in the end, when we
were working with Slack,
he voted on one idea
that he was really
passion about as
well as the idea
that everybody
else had voted on.
And we ended up
prototyping both.
You don't want to leave
ideas on the floor
that powerful people in
the room think are good.
In the end, we tried a
bunch of risky ideas.
So these are the
ideas, the kinds
of things that have been
keeping them up at night.
How do we knock on the door
and make that experience great?
This device, the robot, doesn't
have an articulating arm.
It could bump up
against the door.
It could call into
your room on the phone.
We could have a human call
your room on the phone.
There's a whole bunch of ways
we could potentially get in.
We decided to give it sounds.
They're a month and a
half from launching,
and they had never
had the device
actually make noises before.
But somebody came
up with the idea.
I think it was Adrian,
their audio engineer.
He was like, listen, the actual
thing has a speaker on it.
We've never tried
to use it before.
I wonder if that
would be delightful.
One of the big risks,
we gave it a face.
We're like, OK, we are
going to try a face.
We're going to see how
people react to it.
Then we gave it a
bunch of dialogue.
So the face had next to
it, things it was saying.
And then we tried one really
zany idea, is we made it dance.
We were like, one person had
this crazy idea during the day.
And they were like, listen, I
think if we do a great delivery
and it's successful, the thing
has three wheels under it
so it can turn on the spot.
What if it just said, weee,
and spun around in a circle?
This is the kind of thing--
I know it sounds silly, but
is the kind of thing the teams
debate a lot because you'd
be like, I don't know.
It's so simple and basic.
Nobody would actually
find that delightful.
Or, I don't know.
What if you're a
business traveler
and it just did a spin
thing when it brought you
a power converter
for your laptop,
are you just going to think it's
stupid and not serious enough?
You debate these things,
but we could actually
go test this with real
customers and get some feedback
about how do they actually
react to the thing.
All right.
So we have one day
left to actually make
all of those things
real on a robot.
It doesn't sound
like that much time.
It sounds like we
should be panicking.
But the trick is we work at
the right level of fidelity.
We've already started from
these really crappy sketches
to an interface that
looks real, even if it's
on paper, that has real
text that we're really
going to put on the interface.
And what we're
really doing is just
raising the level of
fidelity on this thing.
In this case, we
just used Keynote.
It was easy because
then the PMs could even
help us do the prototyping.
You didn't need any
skills in design software.
So choose whatever
tools work for you.
Sometimes, we even use
things like Keynotopia.
We're not reinventing buttons.
I don't care if the buttons
look production-ready.
They just got to like buttons
so customers think it's real.
Adrian did the sound effects
because he was into that.
I designed all the interactions
with my colleague John.
For the robot
choreography, for stuff
like the dancing and
the autonomous driving,
we actually just drove
it from around the corner
using a PS3 controller.
So instead of spending
all the time programming
to get it to do new behaviors,
we just thought we'd fake them.
As long as the
customer can't see us,
as far as they're
aware, it's autonomous.
We're not going invest
the time in actually
programming the thing until we
know it's a good idea or not.
We prototyped
fairly long stories,
so it was multi-screens.
We wanted to see if the
customer would actually read
the interface as it's talking.
Did you get
everything you wanted?
We tested timing,
so how long does
the message have to be on
the screen for us to know.
We tested the screen
they'd never had before,
which is how did I do because
this was the trigger for it
to do the dance.
But it's also the way that
we measure longitudinally
whether or not they're driving
up the Guest Satisfaction
Index.
You know every time you
order something in a hotel,
they'll ask you, can I get you
anything else, how did I do.
They're secretly collecting
the Guest Satisfaction Index
from you.
We were like, oh, the
robot can do that too.
That would be great,
and this is data
we can report back to
hotels and sell more robots.
And then we did
the dance, yippee.
Sorry, I'm skipping
through this.
A fairly complex
interface for one day.
We tried a bunch of face styles,
a dopey look, a fun look.
We were trying to drive
down people's expectations
so they wouldn't think too much.
We even tried a dog.
That seemed too dopey
to us, so we ended up
with kind of a
Pokemon interface.
And on Friday, we now tested
this with real customers.
So all of this, Monday to
Thursday is fairly useful.
But in the end, if we weren't
testing with real customers,
we would still be
BS-ing ourselves about
whether or not any of
these ideas were any good.
Now, we've got a better
sense from looking at them,
but let's get this in
front of real customers
and see what happens.
So remember, we had--
oh, I'm sorry.
This is Michael.
This is our researcher.
So he actually did
the studies this time.
And as I said, we normally
keep it really simple,
basic devices, usually.
In this case, we were
testing a robot in a hotel
and can get a little
bit more complicated.
But we even did this
in the same five days.
So we went to the Aloft.
We wanted to test
it in a real room.
Took a lot of gear with
us, a whole bunch of gear.
Michael's a professional.
He brings his gum.
We set it all up.
We taped up some cameras.
Had one over the
door so we could
experience the actual
interaction at the door.
And at 9:00 AM, our very
first participant showed up.
Keep in mind, they had
been on Craigslist.
They'd seen an ad that
says, please come in.
Michael made sure to
include a line that says,
I realize it's unusual to attend
interviews in a hotel room.
[LAUGHTER]
So they read Craigslist.
They come in on a Friday
morning to a hotel.
There's a strange man waiting
for them in the lobby.
[LAUGHTER]
He asked them to come with him
in the elevator to his hotel
room.
[LAUGHTER]
But Michael's a professional.
He wore his Google
badge and made
sure to show it
to them the minute
they came in the front door.
He maintained a respectful
distance from the participants.
He's good at this stuff.
Meanwhile, a bunch of us
are back at the office
watching on the cameras,
and we're taking notes.
And what we're really
looking for here
is patterns of behavior.
Having one exciting person
come in and do something really
unusual is really fun, and it's
cool to tell stories about it
later.
But what you're really looking
for is patterns of three
or four people doing the
same thing because that's
the kind of stuff
that will give you
directional feedback
for what might actually
happen when you deploy this
thing in the real world.
And what happened
is people loved it.
It was really great.
A whole bunch of the
risky ideas worked out.
We had even one guy--
the way the behavior actually
happened is Michael asked them
some questions about what stay
in a hotel is normally like.
And he's like, oh,
and what happens
when you're in the hotel
and you forget something
like a toothbrush.
And they described
the normal process.
And he goes, well, why
don't you try it here.
And they're like,
what do you mean.
He goes, well, just do
what you'd normally do.
And they call down
to the front lobby,
but it's really one
of us on the phone.
And we're like, oh, you
forgot a toothbrush.
No problem sir.
The robot will be up right away.
And we'd hang up, and the
person looked really quizzical
every time.
And Michael's like, well,
what's going to happen.
And he goes, well I
guess a robot's coming.
[LAUGHTER]
And it was so useful
for us because now we're
asking them, well, what's going
to happen when it gets here.
And there were like,
well, I guess it'll knock,
but maybe it'll just call.
And we're like, great.
That is how we were going
to knock on the door.
But then we knew what customers
expectations were like.
And we had one guy--
I swear to God-- one guy got off
the phone, and they were like,
OK, the robot will
be right there, sir.
And he hangs up the phone,
walks over, picks up a remote,
turns on the TV to ESPN.
And Michael's like, so
what's going to happen.
And he goes, robot's
bringing my toothbrush.
[LAUGHTER]
And we're like, only
in Cupertino would that
be totally normal.
[LAUGHTER]
But it turns out when we
were recording things that
worked, unusually, we
nailed a lot of things.
A whole bunch of the
risky ideas worked.
The delivery and
handoff, which they
had struggled with in
the past, everybody
got their toothbrush out of
the device really easily.
They actually thought things
were exciting and delightful
about the robot.
They didn't expect the robot
to be able to listen to them.
None of them tried
to talk to the robot.
None of them wished the robot
came across the threshold
into the room.
They were all happy
with it being out there.
They were happy that it
had a personality, in fact,
wanted to know what to call it.
And so we ended up putting
a name badge on the robot.
That's a really
interesting challenge--
we worked with them on this--
is you can't call it Bob or
give it a human name or even
a Pat or something that's
an androgynous name.
You have to give it
a name like Relay,
which is what we
end up calling it,
kind of these abstract names.
Because if you call it Pat,
you walk into the lobby
and there's five
of them, everyone's
creeped out by five identical
humanoid-ish things.
So we gave them
a non-human name.
And the really
interesting part is
we can get qualitative
feedback from people.
We were talking
about whether or not
this was better than having a
person bring it to their room,
and people really
struggled with it.
They were like, well,
some parts are better.
It's nice not to have
to tip the person.
But it's also nice that it
feels like a shitty task
to ask a human to come
all the way up to my room
just to bring me a toothbrush.
I feel like I put somebody out.
And so they felt
really good about that.
But they were worried about
robots taking people's jobs,
and it was a really
healthy debate with them.
And we understood how
customers might perceive
a robot in this environment.
Super, super helpful.
Not every sprint has
green all the way
down the chart like this.
Oftentimes, we fail completely.
So a sprint isn't
to be successful,
a sprint is to learn
as quickly as you can.
So even if it's all
red down the chart
and we failed a
lot of stuff, phew,
we just saved ourselves
weeks of engineering effort
and operational
effort to actually do
something that likely
wasn't going to work anyway.
But it is fun when
everything works.
So this is just one example
and kind of an unusual one.
We do this with a
lot of other teams.
As I was saying, we
did one with Slack.
I've done sprints with Medium.
We've done sprints
with a lot of--
oh, Slack, we did
the out of box flow.
So this is really interesting.
So I could ramble
up here all day.
Slack, we were working with
them on the onboarding flow
because it's such a strange
onboarding process that you
sign up for a chat room
product by yourself
and you end up in a chat
room all by yourself.
And it's really hard
to know whether or not
it's any good or not.
But it's really a big ask
to ask somebody else to come
into the chat room with you,
let alone your workmates.
And so we were
trying to figure out
how to get over that threshold
as quickly as we could.
We work with expert
users all the time,
people who are harder to
recruit than Craigslist,
developers, vacationers,
geneticists.
Oh, sorry.
We even did one
with truckers once,
and we had to go to a truck
stop and talk to truckers.
All these things are possible.
You just have to get
creative, and it's
a little bit more difficult.
But the big things
and the reason
I'm really bullish
about sprints is
you gather a great team
of people together.
Don't isolate yourself
as a designer.
Go work with engineers
and PMs, people
like your operations people,
people like customer service.
Go talk to your customer
service team as much as you can.
You'll learn a great
deal from them.
Give yourself time pressure.
Don't overbuild the
prototype because you'll
fall in love with it and have a
hard time receiving humiliating
feedback or being humble in
the face of feedback that
might be humiliating.
And then the real
gold standard is
to do quick, credible research.
I meet a lot of teams
that do sprints,
and they cut off the
research at the end
because it was too hard
or too time consuming
or they didn't have somebody
who had research as their core
expertise.
As I was saying
before, the only thing
I've learned in a lot
of years of designing
is how to be wrong faster.
And the best way
to be wrong really
fast is to get your product
out in front of real customers
and get credible evidence.
As I was saying, a whole
bunch of free articles
at gv.com/sprint about how to
do each element of this as well
as the book if you want.
Go [INAUDIBLE]
forth and prototype.
I hope you're all
successful with it.
Thank you so much.
[APPLAUSE]
