CAROLINE O'CONNOR: Hi there.
I am Caroline O'Connor,
and I met Jake and John
a couple of years ago when I
was a designer in residence
at Google Ventures.
I was coming over from the
Stanford Design School,
where I was a faculty
member, and I had a chance
to see the sprint process
that these guys were
running with startups up close,
and I was really blown away.
It was a process that we
were teaching at Stanford,
but I will say these
guys were getting
amazing results with what
they had honed and refined.
And so I'm really, really
excited to introduce them today
to talk about the book
"Sprint" that they've written,
which really lays out
a blueprint for running
these kinds of sprints for
your teams and how it's useful
and how it can help
in the bigger picture.
[APPLAUSE]
JAKE KNAPP: Thank you, Caroline.
Thanks, you guys.
So before we have a
little discussion,
I wanted to talk to you
guys about where the sprint
process came from and tell
you a little story about it.
So it all begins back
in the year 2003,
and it was that year that
my first son was born.
And that's a picture of him.
And I have to tell you
that when my son was born,
I kind of freaked
out a little bit,
because I had this
realization that he
was going to be growing up.
He was going to be a baby and
then a toddler and then a kid,
and this whole sort of life
was going to be going on.
And when I went
back to the office,
I realized every day
I'm at the office,
I'm kind of like missing that.
I'm missing out on
a piece of his life.
And so what I do at the
office should really matter.
I should be making the most
of those hours and days
that I spend away.
And so I thought,
well, I'm going
to take a look at
what I'm doing.
And so I thought
about my work week,
and I looked at my
weekly schedule.
And it looks kind of like this.
It was meetings.
People would schedule
me for meetings.
I'd say yes.
I'd schedule other
people for meetings.
They'd say yes.
And if I ever wanted to
get anything important
accomplished, I
had to kind of wind
my way through this like
obstacle course of meetings.
And I realized I
wasn't really doing
sort of the best use
of my time by winding
through those meetings and
checking email and just
kind of churning a lot of the
time, so what I decided to do
is get productive.
And I read every book I
could find on the topic.
I read "The Four-Hour Work
Week" and "Getting Things Done"
and "Brain Rules,"
and I experimented
with different kinds
of to-do lists,
like five over the
course of a couple years.
Pretty crazy.
And over time, I
did get productive.
In those little windows when
I had a gap between meetings,
I would make the most
use of it possible.
And a few years later, 2007, I
was at Microsoft in those days,
and I left and came to Google.
And when I came to
Google, I thought, man, I
can't wait to see what the days
are going to be like at Google,
because Google is this
like crazy new place.
What's it going to be like?
And it turns out, as
you guys might know,
it's kind of the same.
There are a lot of meetings.
But I had honed these
sort of productive skills
for those gaps between
meetings, and I
knew how to get things
done, and I also
found that at Google,
there was a real spirit
of experimentation, and there
was a spirit of experimentation
in the products but also
in the way people worked
and the way teams worked.
People were willing
to experiment
with how they conducted
their execution on things,
and I've started to do a
new kind of a quest to make
the team process better.
And I imagined, what if
I could just sort of wipe
the week clean, wipe
the calendar free
of all those meetings,
and start from scratch?
And so I started doing
day-long workshops
with teams and then a couple
of day-long workshops,
and eventually ended up
with this week-long process
that I call a sprint.
And that's what we're
going to talk about today.
And I took this
sprint process in 2012
to Google Ventures,
which is now called GV,
because since the
alphabet thing,
we've lost all of our letters.
So we're just GV,
and at GV we have
this interesting
challenge, because we're
investing in
startups, and we want
them to be as
successful as possible.
If you run a startup, you've
got all kinds of ideas,
but it's hard, because
you don't know which
ideas you should focus on.
It's hard to know.
Some of these might be big hits.
They might make
you a lot of money.
They might be really successful.
But a lot of them are worth
pennies on the dollar,
and some of them are
downright dangerous.
But up front, you don't know.
They're just a bunch
of question marks.
They're just a bunch of ideas.
And so this idea in startup
land, in Silicon Valley,
but really around
the world has caught
on that you should take an
idea, make your best hunch,
and just get data about
it as fast as you can.
The trouble is that getting data
requires you to usually build
something and launch it.
And as most folks who have
gone through this cycle know,
building almost always
takes longer than you
think it's going to take.
And so that knowledge
that the build process
will probably take-- if we
think it's going to take weeks,
it'll probably take
months, and if we
think it's going
to take months, it
might take years--
tends to make us
more cautious about
the ideas that we try.
And we argue more about
which hunch we should follow.
So the sprint gives
you a chance in a week
to collect data
quickly, and we found
that that was a really valuable
technique for our startups.
They could try riskier ideas
and try them a lot faster.
So that's what I'll tell
you guys about today.
And the process is
broken down day by day
into five big steps with a
lot of little steps inside.
And rather than just tell
you about how it works,
I'm going to tell you a story.
We've done over 100 of these
with the different startups
in the GV portfolio,
and many of them
are ones that you
might have heard of,
but I'm going to tell you
a story today about one
you may not have heard of.
It's a company called
Savioke, and Savioke,
when it was first-- well, this
is their mission statement,
but when it was first described
to me by my colleague, John,
who you'll meet
in just a moment,
he was like, these
guys make robots.
And we've totally got
to try to find a way
to work with these guys.
And luckily as it
turns out, they
had a really pressing
question that they
wanted answered right away.
And so it was a perfect
time to do a sprint.
So this is the
founder of Savioke,
this guy named Steve Cousins,
and Steve is a robotics expert.
He worked at a place called
Willow Garage for years
and years, and he
left Willow Garage
with the idea of building
robots for the service industry,
robots that would be helping us
in our sort of everyday lives.
He wanted to take robots out of
this kind of abstracted world
of these really high
tech expensive robots
and make ones that we'd
would have in everyday life.
So he built this team of
roboticists, engineers,
designers, and they decided
to build their first robot
for hotels, which might seem
like kind of a funny choice,
but if you think about
working at a hotel front desk,
your day might look
something like this.
This is a diagram that
I just sort of made up,
but I think it's
probably fairly accurate,
because if you imagine the
morning times when people are
checking out and unpacking,
moving around, the evenings
with people checking in,
there's a lot of activity
during those windows at the
front desk with people who
are there physically,
and then also people
are calling from the rooms
and asking for room service,
asking for a toothbrush,
a towel, some extra thing.
And so it's really hard
to staff those bikes.
So Savioke thought, If
we built a robot that
could make some of
those deliveries,
we could let the folks
at the front desk
stay there, be focused there,
do a really good job there,
and then we could also make
sure that people in the rooms
got stuff they needed.
So for the first five
months of their existence,
Savioke builds these
progressively more
sophisticated
prototypes until they've
created this thing, which
is called the Relay robot.
The Relay is a robot about
the size of a trash can.
It is self-driving, so it
can sort of autonomously
navigate the hallways.
It's got a little hatch on
the top with a locker in it,
so at the front desk, you could
put in a towel or a toothbrush
or whatever, close it up,
type in the room number,
and then the robot would drive.
You can call the elevator,
ride the elevator,
and make the delivery
right at the room door.
So all that was great,
and they had worked out
this pilot program with the
Aloft Hotel in Cupertino.
And they were going to take
their first-- they just
had one robot that
worked, and they
were going to take that one
robot and put it into service,
and for a fledgling
startup, this
is like a great opportunity,
but also a really big risk.
They wanted to make
sure they got it right.
And it was going to start making
deliveries in just a few weeks.
But they still had one really
big question left unanswered.
They weren't sure exactly
how the robot should behave.
It's a really good question,
because if you guys think
about, say, Isaac Asimov--
I don't know if anyone here
is familiar, if there's
any science fiction
nerds in the audience, but
I'll confess to being a science
fiction nerd.
There's this really
sophisticated idea
about how robots
should behave, and it's
deeply ingrained
into the science
fiction that has come ever
since Asimov wrote these down.
But as Steve told
us, look, this robot
is not that sophisticated.
It's not going to be
thinking about much.
It's just going to
be delivering things,
and it can't have a
conversation with you.
He said, we've really been
spoiled by Wally and by C-3PO,
and we think that robots have
sort of thoughts and plans
and hopes and dreams,
and the reality
is that this robot
won't be able to have
any kind of a
dialogue with people,
and they were really
concerned that people might
get frustrated or
disappointed if they
tried to interact
with the robot,
ask the robot to do
something, tell it something,
and got no response.
So in order to play it safe
with this launch coming up,
the safest thing to
do is just make sure
that the engineering worked
well-- they had done that--
and not give it a
personality, not
do anything that might ruin
all that other hard work.
But they still kind of
wondered, because it
seemed like this
interesting opportunity,
and they had a bunch of ideas,
but they didn't have a good way
to test them.
So we did a sprint
together with them.
We got to be there in the room
while these guys hashed this
out over the course of a week,
and what happened is on Monday,
they brought the robot into our
building here in Mountain View,
so we could do the sprint there.
They rolled it in.
Here, it's covered
up in a sheet as
though a ghost might be less
conspicuous on the Google
campus than a robot.
And brought it in, and as you
can see here, it kind of looked
like a printer when they
first took the blanket off,
and we were like,
well, OK, you know.
But then we saw
it driving around,
and there's something kind
of amazing that happens,
because this thing can-- it sees
you, and it kind of cautiously
moves around you.
And they had really engineered
that motion exquisitely.
It seemed very dog-like almost,
and so we fell in love with it.
And this is possibly the
first robot selfie ever taken.
But we just thought
like, this robot is-- OK,
let's try the personality.
We got excited.
So on Monday, the first job
in the sprint of the team
is to share all the
information that they've got
and make a map.
And then on that map,
pick the best spot
to focus for the sprint.
So we decided that
the moment of delivery
was both the biggest risk
and the biggest opportunity
for personality, because
up to that moment,
it's possible that you
never saw the robot.
You didn't even know
that the robot existed.
You asked for something to be
delivered, and lo and behold,
you open your door.
You're wearing god knows
what, and there's a robot
at the door of your hotel room.
So what we did then on
Tuesday was to come up
with a bunch of solutions.
How might the personality work?
And when we do this, instead
of having a big group
brainstorm where everyone's
shouting out ideas,
we instead have everybody
sketch on their own.
And I mean everybody,
so in this instance,
it's not a bunch
of just designers
or product folks working on it.
It's everybody on
the team, and when
I say we're working
quietly, it looks like this.
It's kind of boring.
Everybody is in great detail
sketching out their solutions.
So then by Wednesday,
we've got all
these different competing ways
for how the robot might behave,
and we decide from
all of this-- I
think we had 10, 15
different approaches--
to focus on three big ideas.
So first, we've got giving the
robot a face, which obviously
carries some risk, because it's
going to promise personality.
We thought about a lot of
different kinds of faces,
and we ended up going with
this kind of sleepy robot
that sort of matched that
dog-like impression we got
from the way the robot
moved, and we thought
might suggest that you
can't really talk to me.
I'm nice, but you can't
really talk to me.
We wondered if people would like
to play games with the robot,
and we have this idea
of the robot doing
sort of a happy dance after
the delivery was completed.
So on Thursday, we had
to build a prototype,
and we'd spent the
first three days
kind of getting all
these ideas on the table,
making all the choices, and now
we had to build a prototype.
And we think it's really
important that the prototype be
realistic so that when
we test it with customers
on Friday-- that's
how the sprint ends--
we'll be able to
trust their reactions.
So we put together
sound effects,
sort of a sound effect library,
and we divided up the work.
So here's somebody from Savioke
working on that sound effect
library.
Here's the face,
designing the face,
and then just putting it
on an iPad Mini on Keynote,
and we replaced the panel
on the front of the robot
with an iPad
temporarily, because it
would look realistic.
And finally, the
robot choreography.
So normally, the robot is
running completely on its own.
It's all done with sensors.
It's all programmed out,
but as you can see here,
Tessa's got a remote control
like a PlayStation remote.
And temporarily, we could do
sort of a Wizard of Oz thing
with the choreography.
It only needed to work
for five deliveries for us
to be able to get
some data on it.
So finally, it's Friday.
It's time for the test.
We're going to find
out what happens
when you expose this robot
personality to people.
And what we did, and
what we always do
is to do these
one-on-one interviews.
So in this case, our
partner, Michael Margolis,
came down to do the
tests, and here's
a photo of him in
front of the hotel.
And he comes in at 7:00
in the morning and starts
sort of rigging the
room up for the test.
So he's got a suitcase,
and inside a suitcase,
he's just got all kinds of gear.
He's got computers.
He's got tripods.
He's got duct tape.
I don't know how he gets
through airport security,
but he does consistently.
And with the help
of the Savioke team,
they're making sort of
a makeshift research
lab out of the hotel room.
So they're wiring
up these drop cams
so that we'll be able to see
as the robot kind of moves
into position.
And then at 9:00 AM, it's
time for the first test,
the first one-on-one interview.
So I want you just for
a moment to imagine,
put yourself in the shoes
of this first customer who
shows up, and
earlier in the week,
you responded to
an ad on Craigslist
for a usability interview.
You've filled out this
survey, and you get this email
from Michael, and
it says, hey, this
is going to be in a
hotel room, and you're
like-- this is like
Wednesday to you.
You're like, OK, I
guess I'll check it out.
And then on Friday, like
actually, you're like, oh, god,
kind of cursing yourself.
You're there in the lobby,
and Michael shows up,
and he's like, OK,
so come with me.
We're going to go
up to my hotel room,
and you're in the hotel room,
and there's all these cameras,
and you're just like-- you're
probably a little weirded out.
But I mean this just
goes to show that even
under the most potentially
awkward of circumstances,
you actually can get good
honest reactions from customers,
because what Michael does is
he puts the customer at ease.
So he's wearing his work
badge in the hotel, which
doesn't totally make
sense, but it does show
that he is who he says he was.
He's got a clipboard.
His body language is
even very careful,
and he's asking questions
of the customer.
He's saying, tell me
about your hotel routines
when you check in.
Where would you
set your suitcase?
And if you found that you had
forgotten your toothbrush, what
would you do?
And they say, well, I guess
I'd call the front desk
and ask where to get one.
And he says, OK, I want
you to go ahead and call
the front desk.
Here's the number.
So this first person,
she calls the front desk,
and this is actually the phone
number of Allison from Savioke,
who says, oh, sure, I'll
send a toothbrush right up.
So meanwhile we're back
watching over video,
and we can see on
all these, it's
kind of like this
like FBI setup.
He's got all these cameras.
And we know that the robot
is coming into positions
from inside the room
she doesn't know.
And the robot drives
into position.
She opens the door, and
we can watch the reaction
to the personality.
We can watch her take
the toothbrush and see,
does she talk to it?
What happens?
And so we get to watch this
five times throughout the course
of Friday.
We get to see five totally
cold reactions to this robot,
and we see what happens.
And it turns out that
five is enough to give you
these big patterns of things
working and not working.
So it turns out
that nobody wanted
to play games with
the robot, which
is good, because that would have
been a lot of engineering work.
We were able to sort of
cut that right off the bat.
But the face worked.
No one also had any
inclination to have
a conversation with the
robot, to ask the robot
to do something verbally.
That didn't happen.
So the face was a big success.
And the dance even worked.
They even found the dance, which
when described in the abstract,
didn't sound that
great, honestly,
but it was quite
delightful in real life.
This is a little video of what
the robot looks like now out
in the wild, and
Savioke launched
with this sort of
robot personality,
and they now have more
orders than they can fulfill.
And as you can see, it
has very simple eyes.
The dance is really
just kind of rotating,
but it all gives this
kind of delightful feeling
to the robot.
And it turned out
to be something
that guests absolutely loved.
So the idea with the
sprint is, of course,
to not make us be so cautious
about taking these big bets,
to not water down our
ideas or play it too safe.
It might sound a
little bit corny.
You can almost be
true to your vision,
because you know that
you're not betting
the whole farm on everything.
At worst, you'll
embarrass yourself
in front of five customers,
which is nice to know.
That's the idea behind the book.
There are stories about Savioke
and many other companies
in there, and it's also
kind of a DIY guide.
And yeah, let's talk about it.
You guys want to come on up?
CAROLINE O'CONNOR: So
tell us about why does GV
have a design team?
Like what's the goal of having
a design team on a venture
capital?
JAKE KNAPP: We're trying to
slip by as long as possible.
Well, the big idea with GV
is to invest in companies
that we're excited about.
We're excited about the
businesses they're building.
We're excited about
the technology
that they're building, and
we want them to succeed,
but we want them
to succeed in a way
where we make a return
on our investment.
And so we actually see design
as kind of a strategic advantage
for us.
So if you think about a
company that's starting out,
and they're starting to get
some traction on their business,
they've got this really
big idea about where they
can go with the technology.
They've got to, in order
to get there, build
this kind of bridge
between their ideas
and the reality of the real
world, the reality of what
their customers will want
to use, be able to use,
fit into their lives, and
design, as we kind of saw
in this story, offers this way
to really quickly prototype
ideas, understand
how they'll actually
work in customers' hands.
And we find that it's kind
of our secret advantage
over if those companies
were just sort
of doing things on their own.
CAROLINE O'CONNOR: Yeah.
So given you guys have
hundreds of companies
in the GV portfolio.
JAKE KNAPP: Over 300.
CAROLINE O'CONNOR: So I
imagine there's a ton of things
that you could do to
try and help them out.
How did you come to sprints
as kind of the primary way
to help the portfolio?
JOHN ZERATSKY: Well, we
kept investing in companies,
and we were like, oh, we can
help all these companies.
It's not quite like
that, although it's not
that far from the truth.
But I was at GV before Jake
joined us, and had a couple
people on the
design team, and we
had worked as
designers and writers
at other startups in
other parts of Google.
And so we kind of
had this approach
where we would go from
company to company,
and we were sort of the experts.
Or they thought we were experts,
and we thought we were experts,
and we kind of felt this
pressure and this anxiety
of needing to have the answers,
to go into a company and say,
this is what we
think you should do.
But it became clear that
given the breadth of companies
we were investing in and
the different challenges
that those companies faced,
that there was no way we
could have all the answers.
Nobody could have
all the answers,
and so we began looking for,
instead of the answers, a way
to find the answers,
and that was
really where the sprint
process became, we thought,
really powerful.
So we didn't have to say,
well, when I want YouTube,
this is what I did.
And maybe it'll work for you.
We could say, hey,
let's work together,
and let's figure out the
answer to this big question
that you're facing.
CAROLINE O'CONNOR: Awesome.
So you guys have had a
chance to apply this process
with a lot of different
kinds of companies,
small startups, big
startups, but also you've
done a lot of work here at
Google, very big organization.
What have you guys seen
in terms of pitfalls
that are maybe common
to teams or things that
can be blockers
for them generally,
and how this solves it,
or what can block them
from doing a sprint well?
JAKE KNAPP: I think
one of the struggles
is having discussions
in the abstract.
John calls this getting
stuck in abstractopia, where
you know when you're going
to have to build something
that it's a big decision.
And so we debate.
We wave our hands
in conference rooms
and try to anticipate how people
will react in the real world.
And that's tough.
That's an energy
drain, and it often
doesn't yield the
best decisions.
So part of the
reason for the sprint
was to add a little
bit of structure
to those conversations
and to use
some of the things that actually
work really well in design.
Design is a technique
and a set of skills
that can enable people to
make things real really fast.
There's a whole idea
of critique and design.
And design is usually thought
of as this kind of kooky,
creative art that people don't--
if they're not designers,
they kind of walk quickly past
the design room and think like,
it's like those guys are playing
D and D in there or something.
But in the sprint, we've tried
to make those things very
practical, very accessible
to everyone on the team,
and then use them to
short circuit a lot
of those pitfalls.
CAROLINE O'CONNOR: Awesome.
And how do sprints fit
with what everybody thinks
of as like regular work?
Like the calendar
that you showed
that we're all too familiar
with here at Google-- is it
something teams should
be doing all the time
if they're working well, or
when do they bring this process
to bear?
JOHN ZERATSKY:
Yeah, a lot of times
we think that that
sprints make sense
as sort of the kickoff or
kind of this initial burst
for a new project
or a new initiative.
So if a company wants to
reach a new kind of customer,
or they want to introduce a
new feature, a new product,
sprints are a really good
way to sort of kick that off.
But they're not meant
to be sort of the way
that you work all the time.
At the same time,
there's a lot of ideas
that are part of the sprint
that we think are really
valuable to use sort
of in day-to-day work
and day-to-day life.
One of the examples is
sort of about the way
that time and activities are
so structured in the sprint.
For example, I do
all of my meetings
on Thursdays and Fridays, so
a lot of the work that I do,
I'm writing, I'm designing, I
need uninterrupted work time.
So I actually leave
Monday through Wednesday
open for those things, and
then Thursday and Friday
are the days for meetings,
so being very structured
in that Jake and I are very
sort of aggressive about--
JAKE KNAPP: It's sort of weird.
JOHN ZERATSKY: We're sort
of weird about limiting
distractions, so Jake
introduced me to this idea
that he came up with of the
distraction-free iPhone.
So uninstalling Twitter and
removing your email account
from your phone, so you
can't check your email,
and even-- it sounds
nuts-- but even
like disabling Safari
in the restrictions
and the settings for the iPhone
so that you don't have access
to this sort of unlimited
pool of potentially very
interesting but ultimately
distracting information
that exists in the world.
JAKE KNAPP: You
can see the picture
that's being painted here.
Fundamentally, John and I
have very poor self-control,
and if left to our own devices
in a typical work week,
we would be checking
Twitter and our email just
continually over and over again.
And the sprints and then some
of these other kinds of methods
are ways for us
to put rails on it
so that it's easy to do
the right thing, the best
thing with our time.
JOHN ZERATSKY: But
I think usually
what happens with
the teams that we
work with is that
they'll use sprints
as sort of the kickoff
for some new thing
that they're working on.
And a lot of times they'll do
two or three sprints in a row,
so the first sprint is five days
the way that Jake described,
and the second sprint
is usually shorter.
Usually the prototype
that you built and tested,
there may be some
problems with it,
some things you want to
fix, and then there's
some other things
that went really well,
that worked out really well,
and so the team will then
do a shorter sprint where
they tweak that prototype
and test it again, and
then maybe the next week,
they do another
sprint, where instead
of tweaking that
existing prototype,
they create a different
kind of prototype, something
that they can test
with live traffic
or do a different sort of more
quantitative kind of test.
CAROLINE O'CONNOR: Awesome.
Can you talk a little
bit about user testing
as a forcing function for teams?
You guys have that
set on Friday,
but talk a little
bit about how that
can help with the distractions.
JOHN ZERATSKY: Well, the
sprint is actually just
an elaborate scheme to get more
companies to do user research.
JAKE KNAPP: Well,
it's also-- I mean
it does get back to that idea
of me being a procrastinator.
And I did have this
realization that when
I had a deadline, like
many, many procrastinators,
I suddenly got really productive
right before the deadline.
So if you decide you're
going to run a sprint,
and then on Monday
you say, these
are the customers
we're bringing in.
You start recruiting them.
You know that five strangers
are coming in on Friday,
on Thursday, you'll
be like, you don't
want to embarrass yourself.
So you will get
really productive.
And there's all
these good reasons
for bringing in customers.
It makes you focus on those
customers in a very concrete
way throughout the
week, so you're not just
waving your hands,
but you know these are
the people who are coming in.
And it also gives
you data right away.
It's the very
fastest way that you
can get some data on a
complex idea like the ones
that we put into a prototype.
But that forcing function is a
very real, very powerful part
of having people show up.
JOHN ZERATSKY: Part
of our motivation
for sort of building the sprint
around customer interviews
specifically is
that the startups
that we work with and
imagine a lot of the teams
that you guys are on here,
your center of gravity
or your sweet spot is writing
code and shipping software.
That's what we all
know how to do.
And so there's a
tendency for that
to be the thing
you do when you're
trying to figure out something.
And you see this in sort
of like the lean startup
and different approaches like
that like I'll create an MVP,
create some basic
version of the product
that you can get out
there and you can get data
as quickly as
possible, but we think
there's this amazing
shortcut, which is creating
this realistic prototype.
And then instead of
having to launch it
and having to analyze the
data, just watching people
react to it.
We think it's not a
replacement for sort
of quantitative
testing and launching
something in the
wild, but it gets
you a different kind of data.
It gets you this really rich
qualitative understanding
of which things are
working and which aren't.
And the best thing is you can
do it in a few days instead
of weeks or months.
JAKE KNAPP: You can try a
sprint once, do that experiment
with your team, try
working in a different way,
and it all comes
with the sprint.
And then you can
see how it works.
CAROLINE O'CONNOR: So for
partners at a venture capital
firm, you guys have pretty
unusual backgrounds.
No top hats or monocles
that I've ever seen.
JAKE KNAPP: We left
it at the door.
CAROLINE O'CONNOR:
Can you tell us
a little bit about how you
came to be partners at GV?
JAKE KNAPP: Well, that's
a very long story, Carol,
but I'll begin at the beginning.
I won't begin all the
way at the beginning,
but I studied art and
painting in college,
which was, generally speaking,
not helpful for any of the work
I did after that.
But I've always been interested
in computers and making things,
and when I was a kid, I would
make games on the computers
and test them on my
unwilling friends
not knowing that
that was basically
what I'd end up doing sort of
forever, only not with games.
But for me, the process
that led me to GV
was, ironically,
not about having
a ton of startup expertise.
I've learned a
lot about startups
and built that by being at
GV, but actually just having
this sort of interest
in figuring out
how teams can use
design and how teams can
make better use of their time.
So it was the
opportunity at Google
to really test that out and
experiment over and over again
and make the process
better that gave me
the experience
that I needed to be
able to start doing that here.
But you have another very
different road to GV.
JOHN ZERATSKY: Yeah.
I got my start in journalism.
I was actually
introduced to design
by working at a newspaper,
and so I sort of accidentally
found my way into this
really interesting kind
of design work.
So every day I would come in.
It was a daily newspaper.
I would come in, and
the editors would say,
OK, here are the stories that
we have for today's paper,
for tomorrow's paper, I guess.
Here are the photos
that we have, and here's
sort of the priority.
These are the most
important stories.
These are the least important.
And so I'd kind
of do this puzzle
of putting the paper
together and figuring out
how everything fit.
And then it was a paper
that was distributed
on a college campus, the
University of Wisconsin.
So the paper would
be printed, and then
I would go to
class the next day,
and I would actually watch
people read the paper.
So I was like doing usability
testing without really knowing
it, and in a lot of ways, that
experience kind of crystallized
my approach to work-- trying
to get in a lot of reps,
trying to create environments
or seek out environments where
I could get in a lot of
these little loops of trying
something and then seeing
how it worked with customers,
and then fixing it and making
it better the next time around.
So after that, I worked
at a startup called
Feedburner as a designer.
We were acquired by Google.
Worked at Google in
Chicago, worked at YouTube
doing sort of product design,
UX design work all along,
and I was attracted
to GV for a lot
of the same reasons as Jake.
I wanted this
opportunity to work
with so many companies working
on so many different things.
I saw it as being sort of like
being at the newspaper again.
I knew that I was going
to have these loops.
I was going to have
this opportunity
to try things, make them
better, and try them again.
CAROLINE O'CONNOR: Awesome.
So the question is,
how do you make sure
that the small sample of
users that you're getting
relates to the larger
sample that you're
going to be working with?
JAKE KNAPP: Well, one
of the great things that
happens in the course of setting
up the customer interviews
is early in the week,
you talk about all
of the different customer
types that you have.
And you figure out, OK, we're
going to focus on this group,
and what happens when you make
that decision is you start
to get very real about
what defines that group,
because if we want
people to come
in who look like that, who
use that kind of software,
have that kind of job, whatever
it is, then you realize, well,
in order to get
them, we're going
to need to post an ad here or
contact our network of folks
here, and we're going to
need to screen people out.
We're going to need to
recruit a lot of people
and put together a survey,
where we ask them questions,
and we want the results of
those questions to tell us,
is this our customer or not?
And it's that survey,
that sort of screener
that we use that helps
us make sure we're
getting the right people.
But that exercise,
having to do that
is something that we find many
teams at all kinds of companies
small and large defer.
They talk in sort of broad brush
strokes about their customers,
but the sprint makes
you get very specific,
because you want to make sure
you get the right people in.
JOHN ZERATSKY: Yeah,
and that exercise
on Monday of creating the
map is also really helpful,
because you avoid the temptation
to think in terms of personas
and what kind of person it
is, and you think more about
what situation is the person
in, and what are they doing?
What's the task they're
trying to complete?
And that's sort of that--
Jake showed that big map
and then zooming in on that
one point, so in that case,
it's a traveler who is
checking into a hotel
and realizes that they
forgot their toothbrush.
So you can look for
people who would likely
be in that very
specific situation.
JAKE KNAPP: To give you
a concrete example, when
I worked with Slack, who's a
sort of messaging software,
they were interested
in finding better ways
to explain how Slack
works to companies
who aren't tech companies.
Slack's grown tremendously
with tech companies,
and they're figuring
out how to explain it
to other kinds of teams.
And so they knew that
they wanted to expand
to other kinds of teams.
In the sprint, they had to get
really specific about which
kinds of teams, which
are the best example
of representative teams?
Is it a team who's in media?
Is it a team who's
in health care?
What should we look for?
What questions should
we put in the survey
to find those people?
CAROLINE O'CONNOR:
So the question
is we have so much data.
Do you want to be
looking at the data
that you've got and digging into
that before you do a sprint,
or do you do the sprint earlier?
How do you think about that?
JAKE KNAPP: Yeah.
So there's maybe three
things I'll remind myself,
there's three things
I want to mention.
The first one
actually is that you
should be really careful
of three-day sprints
or anything short of five,
because usually the first thing
people cut when they make
the sprint shorter is
that realistic
prototype and the test.
So I do know there are some
folks who will compress all
the steps into three
days, and then you
have to look out
for like dehydration
of like passing out,
because it's pretty intense.
But if you actually are
doing it at the normal pace,
you just want to
be really sure you
build that realistic
prototype and test it.
Otherwise, you might not know
if you have the right idea.
JOHN ZERATSKY: Yeah,
you can cut out days,
but you can't cut out steps.
Or you shouldn't cut out steps.
JAKE KNAPP: Yeah.
That's a nice concise
way of putting it.
But in terms of when
you have a lot of data,
knowing what to do
with it, I think
John and I both came from
working at Google where
there was tons of data.
John worked on YouTube.
I worked on Gmail,
and obviously, those
are products where
you launch something,
and you get a lot of data
about what's going on.
But even when you
have a ton of data,
it can be very hard to know
why something is working or not
working.
And so we certainly
have the experience
of working for a long
time on a new feature,
launching it, and seeing that
people were using it or not
using it, but not
knowing why still.
And that's part of
the thing that you'll
get with the kind of
qualitative research
that you do in a sprint
when you're doing interviews
is that you'll know why.
You'll be able to see
people actually doing it,
and you don't have to just
guess what the numbers mean.
In terms of when
to do the sprint,
I think what we really
look for is a big question.
And so Savioke is honestly
among all of our sprints,
it's more unusual for us
to work with a startup
when they're so close to launch.
It's not crazy or
unheard of, but it's
more common to do
it at the beginning
when you have that big question.
You're starting off on
a totally new product
or a big new feature,
or maybe it's
the start of a new
quarter, and you
know that you're going to be
putting a lot of effort in.
You just want to check
where things are.
But it's that feeling of, gosh,
we're making big decisions,
but we don't know for sure.
We're arguing or we're
scratching our heads.
You can satisfy
that with a sprint.
JOHN ZERATSKY: Yeah, I would
say that to your question
about sort of bringing
data into the sprint
and how to incorporate
that, having a lot of data
to work with when the sprint
starts is actually a luxury.
It's actually a great thing.
And the sprints that I
can think back to that
were the most successful
started with a lot of data.
Either we did a round
of customer interviews,
or we had a lot of
quantitative data
about how people were
using the product
or how effective
the marketing was.
It does take some
work to distill that
and analyze it and present
it in a way that makes sense,
but that's always a challenge.
So what we'll typically do is
we'll invite sort of someone
from a data team or a product
manager or, in our case,
a lot of times, it might
even be the CEO or something,
because we're working with
these really small companies.
And they'll give us on that
first day of the sprint
a half an hour or an
hour sort of run-through
of who's overseeing.
Here is maybe the conversion
funnel, or here are the usage
patterns in the product, and
this is what we know so far,
and this is what we're
trying to figure out.
CAROLINE O'CONNOR: So
is five days really
the number for a sprint?
Could you do it in one day?
And how do you think about sort
of cutting it down or playing
with time?
JOHN ZERATSKY: Well,
we've experimented a lot,
so that's kind of
the first thing
is we're pretty confident that
it's the right number of days.
But I like your question about--
you asked, what is the essence?
Like if you were going to sort
of slim down the sprint, what
would be like the absolute
essentials to not get rid of?
And I want to hear
Jake's answer.
JAKE KNAPP: Well,
you definitely don't
want to get rid of that
realistic prototype
and the test,
because what happens
at the end of the sprint
is you run that test,
and then you're seeing
how customers react,
and then that's surprising.
Ideas that seem so
brilliant on Wednesday
turned out to be duds, and
then something that maybe you
just took a risk on, and one
thing we didn't talk about so
much is that we'll sometimes
make two or three competing
prototypes and see how
they do head to head.
And maybe it's that riskier
idea that turns out to work,
but you don't know that until
you've shown it to customers.
And the other
things you get when
you have a realistic
prototype is for the team,
it's this concrete--
it's like you fast
forward it into the future.
What if we were done
already with the product,
and it looked like this?
And that's really helpful.
It's helpful for the team to
decide if the product seems
to fail with customers,
do we have belief
that this artifact is
something we can make better?
And we're just going to
try to make it better
in another sprint,
or are we off track?
This wasn't as great
as we imagined.
JOHN ZERATSKY: Yeah, I think
a lot of the short sprints
that we've seen or
that we've heard about,
for whatever reason, they
tend to be oriented or sort
of weighted toward the
early parts of the sprint.
So they tend to be more
about coming up with ideas,
and I actually think
that coming up with ideas
is usually not the challenge.
People are constantly
thinking of ideas,
and in fact, we found that
the ideas that people come up
with in the sprint are often
not as good as the ideas
that people have
had kind of rattling
around their brains for
the last months or years.
And I think it's because
the ideas in the sprint
are sort of abstract,
they're unrefined,
and they're new, whereas
the existing ideas have
been through the
wringer, so to speak,
of kind of thinking about
them, working on them,
considering them.
JAKE KNAPP: Just to
be clear, we make
sure people put those old
ideas into [INAUDIBLE].
JOHN ZERATSKY: Exactly.
Yeah.
JAKE KNAPP: They're
on the table.
But I think then you can
take-- the essence of it
also is to-- we're sort of
geeky about human interactions.
John and I are a
little-- I don't know.
We're sort of pod people.
But we think that if you
can put some structure
around the things that we often
do at work with no structure,
that it helps a
lot, and so there
are a bunch of little
pieces in a sprint that
can be helpful at any time.
It can be really
helpful when you're
having and you're starting
to talk and have evolved
into a brainstorm in a meeting
to say, wait, hold on a second.
Let's all quietly
write down our ideas.
All of a sudden, you give
the introvert or the person
who's not so good at
pitching their ideas,
you give them a chance to
have their idea be on a level
playing field with
everyone else's.
Sometimes it's helpful to
just put the ideas on the wall
and vote, the kind of thing that
we do in a sprint all the time.
It helps you shortcut
sometimes a discussion that
isn't necessary.
And the idea of
talking to customers--
putting what you have
in front of customers
is something you can do.
Even if you're not
running a sprint,
you can test your product
and put it in front of people
and start to answer
those why questions.
CAROLINE O'CONNOR:
So the question
is, how do you go from 17
ideas on paper to the few
that you're going to test, and
how teams navigate that well?
JOHN ZERATSKY: That's
probably the most
like robotic, scripted
part of the whole sprint.
And for good reason, because
it is very challenging.
I guess we sort of talked
through the specifics,
but I'll lay out the
high level, which
is that we have a very
particular set of voting
exercises that we do starting
with what we call a heat
map, which is where everybody
just sort of looks at the ideas
and has these small
colored sticky dots.
They get effectively an
unlimited number of them,
so if they spend them all,
they can get more, and just
put a dot on anything
that seems interesting.
Then we do what we
call a straw poll
sort of round of voting,
where people get larger dots,
and they get a limited number.
But it's non-binding,
so you're sort
of going around the room
looking at the ideas
and voting on the ones
that seem most promising.
And then we do a
really fast critique,
so the group together talks
about which ideas they thought
were the best and which ones
maybe are problematic or not as
interesting.
And then one of the more
sort of unusual steps
is that we then rely on the
decision-maker in the sprint
to actually make the final call.
So the leader, the
executive, the CEO,
whoever it is-- that person
gets to sort of absorb
all the work and all the ideas
and all the opinions that
have been shared in
the room, and make
the final call about
which prototype
or which prototypes
to build and test.
JAKE KNAPP: One of
the ways that we
think about this is if you
think about your phone.
Like every morning, if you
charged your phone overnight,
you wake up, the battery's full,
and then throughout the day,
you do stuff, and the
battery wears down,
unless you have a
cooler phone than I do.
But that's what happens to me.
Our ability to make decisions
is kind of like that,
so we wake up in the morning.
We've got like a full battery
of decision-making ability,
and then as the
day goes on, if you
have a lot of intense
conversations,
if they're 17 ideas,
and you're trying
to narrow them down to three
by like arguing the whole day
or like pitching one
versus the other,
like your battery is just
going to go [SHRINKING NOISE].
And it's not going to work.
And so in the structure that
John described, what we're
trying to do is make
those decisions as
effective as possible
so that every time you
burn a little battery,
you're making progress.
And to do that, we
cut out sales pitches.
We make a lot of the
evaluation silent.
We make the sketches
anonymous, so you're not
evaluating who made it.
And then ultimately,
when it starts
to be the hardest
part of the decision,
we turn it over to that
one person who we know
will make an opinionated call.
So we don't have to
worry about group think.
We don't have to worry about
watering down and getting
to a consensus.
We're just saying like,
OK, look, decider,
you make the call, and
then we'll find out.
Right away we'll get data,
so you can take a chance.
CAROLINE O'CONNOR:
So the question
is, how do you modify
this for situations
where humans are not
that end consumer?
JOHN ZERATSKY: Well, I think
that is a really interesting
question, and I don't think
we've done a sprint like that.
JAKE KNAPP: I'll
give you an answer
that-- I'll start
with a bad answer
to that question, which
is that for most things--
and you may be talking
about an exception
to this-- for most things,
there is some point at which
the product does touch a human.
And it might be downstream.
It might be that you're building
a back end service that,
in turn, supplies something
to another service,
and in turn, the place
where it touches the human
is somewhere downstream.
But fundamentally,
humans are the problem.
Whenever anything,
you make something,
and it doesn't work well,
or there's a problem,
or people don't like it,
it's that surface area
where the human touches it
that's usually the unknown.
And so we often
think, well, where
is that human touch point?
What is it?
It might not be like
a software interface.
It might be a
sales conversation.
You might be building
back end that's
going to supply functionality
with the expectation
that we'll be able to
sell this to a third party
or that it'll enable us to
do some new kind of query
that people will want,
searches that they'll
want to run or something.
I'm just making this
up, but that idea often
helps us get to the root
of the question, which
is about what will people do?
People are the ones who
are hard to predict.
Now if that doesn't
work, the answer
that's slightly better, but
I'm waving my hands a lot more,
is that you try to prototype
what you would be supplying
if all of the coding was done.
And you say, during our day when
we're prototyping on Thursday,
we're going to be
faking that part.
And then we're going to be
somehow manually supplying
that to whatever's on the other
side and seeing what comes out.
I'm not a back end
developer, so you're probably
just laughing at me, but I
think that might be the idea.
OK
JOHN ZERATSKY: I
was going to suggest
that she change the test to
try to come up with-- the test
that we do, our
customer interviews,
but there might be
some other kind of test
that you could think up.
I'm also waving my
hands, but there
might be a different way
of sort of validating
whether an idea or an
approach is any good or not.
JAKE KNAPP: A big
part of the magic
is that constraint
of five days, and we
know that the
steps in the sprint
will provide us rails
to evaluate ideas,
make decisions, and then
very quickly, try something,
and it might not be
right, but for me, there's
so much frustration
when you spend
weeks or months doing
that sort of discussion
and trying to decide.
And so yeah, it might be just
a different kind of test,
and you commit to
doing the sprint
to have something
at the end, so you
can see what it might be like.
CAROLINE O'CONNOR:
I have one more
I wanted to throw at you guys.
You talk a lot about the value
of having the whole team watch
the user testing and really
see it for themselves
and be connected to it.
I found that to be really
challenging at Google
given the calendar situation.
And so then you end
up with challenges
to a data set of
[INAUDIBLE] of five,
that that can't
possibly be valid,
and especially for folks who
haven't sort of seen the user
test.
Do you guys run into that?
And if you do, how do
you deal with that?
JAKE KNAPP: Well,
that idea of having
everyone watch is
really important,
and one of the reasons
why it's important
is that if everybody
can't watch together,
then it often falls on
one person's shoulders
to conduct the interviews
and then communicate
what he or she saw in those
interviews to everyone else.
JOHN ZERATSKY: And that person
has to be really convincing.
JAKE KNAPP: Yeah.
Yeah, incredible, and
it's a lot of work,
and there's a time delay to
put that information together.
So if you get
everybody in the room
together on a
Friday watching it,
this magical thing happens where
there is no argument about what
people saw, and there's
also-- we've never
had a sprint where after
watching five interviews,
people, no matter how much
hard core data nerds they were,
didn't say like, oh, yeah.
It's clear what the
big findings were,
what the big patterns were,
because by that fifth person,
you're just starting
to see the same things,
things that you saw
earlier happen again.
There's nothing new.
And I don't know if you
want to talk about Y5.
JOHN ZERATSKY: Well, I was just
going to add something and talk
about one of the
mechanisms that we
use to sort of make sure
that people stick around
for the research and actually
watch the research is by making
sure that the
people in the sprint
all participate in the
creation and the selection
of those ideas so that
everybody is sketching,
everybody's involved in the
decision-making process,
everybody's involved
in the prototyping,
and the reason
that's important is
that then those people
really want to see what's
going to happen in the test.
It's like people
are then into it.
They're in suspense.
They want to find out if their
ideas are going to work out
or not.
So by making sure that
the entire team is going
to participate, even if that
means a smaller team, a smaller
core team who's going
to be really focused,
it creates an incentive
for those people
to stick around for the end.
JAKE KNAPP: There's a
study done by Jacob Nielsen
in the '90s where he
evaluated hundreds
of these kinds of interviews
to see at what point
the sort of learnings
trailed off,
and it turns out that
whether you interview
five people or 30, by the
time you've talked to five,
you'll see 85% of
what you'll ever see.
And so you're better off--
that's the point of diminishing
returns-- you're better off
then turning your efforts
to doing a new prototype and
fixing it or changing it.
And so we've also
seen that anecdotally
in our own experience that,
again, by that fifth interview,
it starts to be
repeated information.
JOHN ZERATSKY: We both heard
a great metaphor the other day
for this.
Imagine if there was
a piece of carpeting
here that was kind
of flipped up,
and people were coming up to
ask us questions or something,
and 20 people were going to
come up and ask a question.
And the first two people tripped
on the piece of carpeting.
Would we need to watch
the other 18 people
all trip on the carpeting?
We wouldn't need to.
So there are some
things like that
that just become so obvious
that need to be fixed
or things that are
working well after even
a couple of customers
that five really
is a magic number for
customer research.
JAKE KNAPP: In the sense
of being a complement
to large scale data.
Either later on or before,
you have that large scale data
that can tell you something
different about it,
but five works well.
And because it's so important
to have the team there,
to have everyone
see that research,
we really think that
you're better off,
if you don't have time to
do that realistic prototype
that fifth day when you test,
you're better off not doing
a sprint at all, because it
might be a sign that what
you're working on is not
important enough to the team.
And so in that
case, you might want
to wait until it is
the important thing,
and you're willing
to do it right,
because you'll get so much
better results by going
all the way through.
CAROLINE O'CONNOR: How important
is it to test three ideas
and evaluate them against each
other as opposed to maybe just
testing one idea?
JAKE KNAPP: We don't
always test three,
but we often do, and the
reason why it's valuable
is because-- actually,
there's a couple reasons.
One is that if you know
you have two or three,
you can take bigger risks.
So you're less likely to try
to make this really tough call
between two promising
directions or to try
to sort of water them down
or merge them into one idea.
The other thing is
that frankly, we've
done this enough times that
we know how often we're wrong.
We know how often founders,
anybody who's a decision-maker
is wrong.
It just turns out that
humans are unpredictable,
and when you show
them this new thing,
it's really hard to tell how
they're going to respond.
And so you have just
increased your hit rate.
If you've got two or
three different things,
it turns out that the
chances that one of those
is the right one is
much, much higher.
JOHN ZERATSKY: In
the Savioke story,
there were three
different ideas,
but they were all packaged
into one prototype.
So a lot of times,
that's what we'll do.
We thought it would
be weird if there
were like multiple robots making
deliveries in your hotel room.
But if it's something--
JAKE KNAPP: Here's
your toothbrush.
JOHN ZERATSKY: If it's a more
conventional type of product,
if we're talking about like
when we worked with Blue Bottle
Coffee, so we were
helping them figure out
how to expand their
business online,
how to bring the
experience that they
created for their customers
in their stores to the web.
We had a lot of different
ideas for how we might do that,
and what we did in that case was
we took those different ideas,
and we created three
separate prototypes,
and what it looked
like to the customers
we tested with were three
different websites where
you could buy coffee.
JAKE KNAPP: Three totally
different coffee companies.
JOHN ZERATSKY: They even
had different brands.
We made up fake logos.
They had different
color schemes.
And so that's a
really powerful way
to test when you have ideas
that are completely in conflict.
With Savioke, the face and
the games and the dance
were not in conflict, so
we could put them together.
But if you have ideas
that are in conflict,
these multiple competing
prototypes with fake brands,
they create that
illusion for the customer
that they're looking
at real stuff,
and they're just sort of
reacting and thinking out loud
and telling you, I
really like this one,
because it seems I don't know
if I can trust it or whatever.
You're sort of
watching them shop
between these different options.
CAROLINE O'CONNOR: Thank
you, guys, so much.
This was really,
really interesting.
JOHN ZERATSKY: Thanks,
everyone, for coming.
Thank you.
[APPLAUSE]
