[MUSIC PLAYING]
DAVID BARRY: Today
at Google we're
delighted to welcome
Rebecca Kantar.
Rebecca is the
founder of Imbellus,
which is a company that
is reinventing the way we
think about human potential.
Rebecca attended
Harvard College,
but dropped out at 19
to pursue her career,
and has been especially
passionate about thinking
about education in
the US and how it
pertains to the future of work.
So without any
further ado, we're
very excited to have Rebecca.
She's working on huge
projects at such a young age,
and we are delighted
to have her.
So please join me in welcoming
to Google Rebecca Kantar.
REBECCA KANTAR: Thank you.
[APPLAUSE]
DAVID BARRY: Thank
you for coming.
REBECCA KANTAR: Thanks.
DAVID BARRY: So I'd love it
if you could start by just
telling us about Imbellus.
Start with a high level.
REBECCA KANTAR: So
at a high level,
I have been on the
quest to figure out
how we can move beyond
measuring how humans think
based on multiple choice tests.
So if you think about
the education system,
change has always flowed
from the top down.
Our colleges set standards
that become college admissions
standards.
Many of you are
familiar with tests
like the SAT, the ACT, the SAT
subject tests, the AP test.
There's a huge litany
of these exams,
and they're only able
to look at our ability
to answer really
set questions that
have very clear
mandates and very
clear answers, and both the
right and wrong answers.
So when you think about that
as kind of a constraining
medium for how you can get at
people's processing, how they
develop ideas, how
they make decisions,
it's a really limiting factor.
Imbellus is working on building
a new age of assessments that
are able to look
at how you think,
instead of just what you know.
So moving beyond multiple
choice and rote memorization,
content based
understanding of assessment
towards an era of can we observe
every piece of your cognition,
every step in your processing,
not just that end choice.
DAVID BARRY: Gotcha.
And so what was it
that made you decide
to focus on standardized tests.
REBECCA KANTAR: So thinking
about the education system,
if you kind of dive into the
education reform movement,
there's a lot of
change in K through 7,
new elementary schools
or middle schools.
There's a lot less
change in high school,
and it's really where the
rubber meets the road.
If people are orienting their
high schools towards getting
kids into good colleges,
and those high schools
don't actually prepare kids
for college admissions tests,
it makes it very hard
for those high schools
to be well-regarded long term.
So when you think about how
you impact change, particularly
at the high school
level, you've got
to take on high
school curriculum.
And in our country, the current
set up of K through 12 schools
makes it such that our
state standardized tests
and then our college
admissions tests
kind of compound these
problems of inequity
throughout the system.
So if you're in a poor
district when you start out
in kindergarten
through fifth grade,
you continue on to a mediocre
or poor middle school,
and then a mediocre
or poor high school,
your chances of getting
into a great college
are slim to none at that point.
And I think we have
in this country
an idea that college
is universally
something that lifts people
up, that provides mobility.
And in fact, it's not,
and we can kind of
get into this throughout
our discussion
as we look through the
data, but there's really
a difference between going to
a top 20 or a top 100 college
and going to a
bottom 5,000 college,
and standardized tests
make all the difference
in terms of which
levels of access
students from
different backgrounds
are able to attain as they
seek college admission.
So thinking about these
tests as a key lever--
until you fix this kind
of one line of code
of how we measure
how people think,
instead of measuring how
much stuff they've learned--
really important to alleviating
the pressure on high schools
to teach in a broad
content mastery based way--
as in, how many
subjects can you take,
and how well can you do
in all those subjects?
To instead focus
on how many skills,
and abilities, and core
concepts can you internalize?
And how can you use those across
a broad variety of applications
that are relevant not just to
school, not just to college,
but to work and to life?
And I think increasingly
we have kind of
separated the two-- of
our education system
and our employment system.
And our aspiration of focusing
on assessments with input
from both employers
and from colleges
is to reunite education in
assessment and employment
systems as kind of one
giant ecosystem that
determines how you start out
in your early adult years.
DAVID BARRY: So how can
Imbellus play a role
in changing the education?
REBECCA KANTAR: Sure, so maybe
we can put the slides up here,
but we have a focus on
simulation based assessments
that evaluate how people think,
instead of what they know.
And just to make that kind
of tangible for folks,
say you have an idea.
Since our product
is very visual,
we have this natural
world ecosystem.
It's a 3D virtual
environment where
people deal with
scenarios that look
totally different-- perhaps a
group of animals who are sick
with a disease, as opposed to
something you do on the job,
like writing an email
to your product manager.
But underlying that skin,
that's very different.
The structure of
the scenarios we
ask people to deal
with in our environment
are really analogous to those
that you face in real work
or in life.
So our idea is, by understanding
how people make decisions
in this environment,
how they develop ideas,
how they process
information, we can translate
every click, every almost
click, every time stamp
into inferences that have much
better and more significant
implications in
the future of work
and where things are going,
as opposed to just relying
on what we've done
for the last 100 years
or so in assessment,
which is saying, hey,
you've known more
about this subject,
and therefore critical
thinking must kind of sort of
be going on behind the scenes.
We actually see that
critical thinking happening
in a very raw, unfiltered way
just with different subject
matter material,
but that difference
is what we call far
transfer, meaning
if we see you do it in
this environment that's
really abstract, the
idea is you should
be able to apply those skills
across many other different
environments.
And the benefit to kind of
getting at skills in that way
is you allow high
schools to maybe,
if they're based
in Tennessee, focus
on teaching subject
matter that's
very different than high schools
that are based in Pasadena
or based in Boston.
So you're able to
move away from just,
can you plow through
180 days of a textbook?
And towards project
based learning,
or application based learning,
or flipped classrooms.
There's a whole variety
of different solutions
that kind of try and
inculcate deeper skills,
as opposed to more content,
but to date, none of our tests
have reflected that
curricular progress as such.
So Imbellus you could think
of as this one line of code
that, if we can change how
people are tested primarily
between high school and college,
and college and employment,
perhaps we can change the
way the high school operating
system runs.
DAVID BARRY: So let's contrast
that then with the old model.
So how did standardized
testing become what it is?
REBECCA KANTAR: So most
people aren't super familiar
with kind of the history
of standardized testing.
So just to back up, the SAT,
which many of us know and love
or know and hate, depending
on your experience--
the SAT used to be called the
standardized aptitude test,
and it was actually a
derivative of the IQ
test, which came about in
the early 1900s, around 1906.
And the IQ test then was
the basis for the Army Alpha
test used in World War I.
So we went from measuring mental
retardation with the IQ test
to measuring soldiers for
fitness in World War I,
to then the same general
group of psychometricians
and learning scientists working
on this first instantiation
of the SAT at the bequest of
some of our elite colleges.
So the elite colleges in the US
at the turn of the 20th century
had created something
called the committee of 10,
and the committee
of 10 was focused
on standardizing high
school curriculum so that,
no matter where
you grew up, you'd
have a relatively similar path
of preparation for entering
the job market, usually as a
factory assembly line worker,
or, in some cases, as a
manager, or as a judge,
or as a doctor, any of the
careers that at that time
really were served by
the elite colleges that
were available for a tiny
fraction of the population.
So the SAT kind of
originated from that ground
for the elite colleges
to be able to suss out
who was fit for that
aristocratic lifestyle
and who was fit to be
an assembly line worker.
And from there, throughout
the 1930s and 1950s,
you had this huge swell, more
and more students wanting
to go to college, more families
expecting it as an attainable
end goal for their children.
And the SAT had
to kind of adapt,
so the College Board
released the SAT, and then
the SAT subject tests, many of
which are still around today.
People usually take
two or three if they're
applying to a top college.
And then in the 1950s,
really the major change
that kind of started
the trickle down effect
that we experienced today
was the ACT and the AP exams.
So the ACT came on
the scene in 1959
with a promise of
measuring information
taught in high
school, as opposed
to critical thinking
or aptitude.
I want to be careful of
this word of aptitude.
People think of it as
inherent or intrinsic skills
and abilities, as
opposed to what it really
stood for at the
time, which I believe
to be much more analogous
with what was your ceiling
and trying to figure out how
much theoretically could you
process, could you understand.
So the ACT shifted the focus
away from any questions
around aptitude or
critical thinking
to just measuring how
well did kids absorb
what their schools taught.
Really important change
in the trajectory
of standardized testing,
because the APs then
came along to kind of
reinforce that trend and say,
well, maybe we can get much
more predictive results
about who will do well
in college if we focus
not just on measuring
math or English,
but on measuring every
other subject, too.
So today we have over 39 AP
tests, 20 SAT subject tests,
and those tests kind
of became the tail
wagging the dog of our
entire education system.
So instead of
curriculum being shaped
by employers or by the
government, increasingly
at high school we
have curriculum
being shaped by the
tests that are required
for college admissions.
So that's kind of how the
history has played out
until now.
DAVID BARRY: So
how do you know--
say Imbellus shifts to a
model that's more about how
you think than what you know.
How do you know
that that's really
a better way of testing
than current methods?
REBECCA KANTAR: So as we think
about what should an assessment
be like, there are
a few requirements
for this day and age
that are table stakes.
One is we need to try and undo
the compounding inequality
that this litany of exams that
we've had for the last 100
years have installed
in our system.
The reason these exams tend
to perpetuate inequality--
as we all know, if you take
an SAT or ACT prep course,
if you take a set
of AP prep courses,
if your school
has more resources
to throw at teaching to higher
level, more rigorous courses
like APs, it tends to
follow that students
in those schools or in
those test prep courses
do better on the exams.
And to the College
Board and ACT's credit,
they've done a lot
to democratize access
to exam preparation,
but you can't
undo 12 years of
bad public education
with 12 months of exam prep.
So the first requirement for
a new assessment should be,
how do you make something
that's not gameable?
That can't be easily cheated?
That can't be prepared
for in 12 hours?
Because really the point of
introducing new assessments
should be to shape 12
years of public education.
So when you think about that as
kind of the first requirement,
from there, the
question becomes, well,
how do you make something
that's more generalizable,
that gets at these skills?
Instead of getting
at specific stuff,
that might be more
sensitive to the inequities
of one school versus
another, one child's
background versus another.
So we focus on, how do we build
environments that are, again,
abstract such that
the nature of skills
can shine through,
even if kids learned
how to use those skills
through a broad variety of life
experiences or
school experiences?
The next question is, if you're
going to deploy something
in a computer based environment,
how do you prevent people
from cheating and hacking?
Especially if you're going to
deploy things remotely, which
right now part of the
limiting factor of an SAT,
or ACT, or SAT II, or AP
type experience is you
have to physically come
into an exam location.
Your parent has to be
able to drive you there.
You have to apply
for a fee waiver
if you can't afford
the fees online.
There's a lot of friction
to coming and doing an exam.
Ideally, we get
to a place where,
even if it's monitored
remotely, we're
able to deploy at a
library, or at a study hall,
or maybe even some
day at your home.
And that necessitates your
exam being different enough
for my exam such that
you can't be messaging me
after about what you
did, and what works,
and what didn't work.
So we think about, how do
you achieve remote deployment
by having unique versions
of your assessment
for every person every time?
And then finally, there's
an underlying question
for assessment of this
century, which is really
the core of why work with
employers at this point for us,
which is what
skills are relevant?
What should we be testing?
If we were to redo the
SAT from first principles,
what content should be in there?
What skills do
people really need?
So we stepped back and said,
OK, if you kind of go along
the general thesis of what the
education system has believed
to date of problem
solving, critical thinking,
analytical reasoning
skills being important,
the question becomes, how
do you define those skills
for the modern work life
environment of this century?
And so we've said, maybe we want
to go out and observe skills
that are happening at companies
where they're at the bleeding
edge of kind of current
work environment,
bleeding into the future
of work environment,
somewhere like Google,
as it might pertain
to a skill like creativity,
and really get a chance
to observe how people at that
place leverage that skill.
And you're not just looking
at what do they make
or what do they do?
You're looking at
how do they think,
and then trying to translate
that to an assessment.
So when we think about
our efficacy long term,
our goal is not just to predict
your first year GPA in college,
because there's a disconnect
for many, many colleges
between what success in college
might look like compared
to what success in
life might look like,
but instead our goal is
to be able to show that,
if we say someone is
capable of critical thinking
and creativity, if they then
enter a career that involves
critical thinking
and creativity,
can they remain
gainfully employed?
Can they deal with
daily life situations
that involve those
skills and how
do the choices they make impact,
over the long term, their life
status and their
ability to succeed?
So those are kind of the
longer term metrics that we're
going after to decide if
our assessment is truly
better than what's there now.
DAVID BARRY: So you've worked
specifically with McKinsey
so far, right?
REBECCA KANTAR: Correct.
DAVID BARRY: So could
you talk a little bit
about that and the
types of things
you've been doing with them?
REBECCA KANTAR: So we
started with McKinsey largely
to focus on problem solving.
They're world known
for solving problems.
That's why they exist.
It's how they make money.
So we said, perhaps
if we could understand
how problem solving happens in
an abstract sense at a place
like McKinsey, maybe
we could work backwards
to define what problem solving
should look like if it's
done well, but also to
understand problem solving
across a variety of
industries, across a variety
of nuanced contacts, perhaps
some sort of problem solving
in emergent markets versus
really well developed markets,
problem solving
in scenarios where
there's a lot of ambiguity
versus very little ambiguity.
So there's kind of
a really nice canvas
for observing how
problem solving manifests
at a place like McKinsey.
And our goal together is to
develop this problem solving
assessment that can kind of live
at the top of their recruiting
funnel, and to then
bring all of the data
that we get from working
with employers like McKinsey
back to top colleges in
the US who have forever
controlled the standards that
we set for college admissions.
So if we can make the case
to a place like Harvard
that we've been
testing 60% or 70%
of their graduating
seniors, now you
have a powerful body of data.
And that's really
the only way I think
to ever challenge the
traditional status quo
players and assessment.
DAVID BARRY: So
what other skills--
so it sounds like problem
solving is a big thing,
looking at the way people think.
What are other skills that
Imbellus seeks to measure?
REBECCA KANTAR: So
I like this quote
just because it kind
of brings it to life,
but Bill Deresiewicz--
he's a famous author--
has a book called "Excellent
Sheep-- the Miseducation
of American Elite."
And I like to think
about it-- it says,
what does it take
to produce more
than just mere mortal sheep?
And if we think about this
idea that everyone's training
for jobs that we
know are soon to be
obsolete due to
automation, how do
we think about the
universe of human skills
that really differentiate
humans from machines?
And for us, those are
skills like problem solving,
like critical thinking,
like abstract connections,
everything that we've kind
of talked about so far,
but there are also
skills like integrity
and like collaboration,
which obviously comes up
in every workplace now.
So we're focused on
how do we broaden
beyond just the traditional
cognitive skills.
How do we go beyond something
like what the AP tests measure
and really get at the
layers of human intelligence
that continue to differentiate
us from machines?
And just to kind
of underscore why
that's so important,
even if you're
a fan of content mastery--
and don't get me wrong.
I think we need both,
kind of understanding
that everyone can do basic
math, that everyone can read,
that everyone can write.
But even if you're
a fan, in theory,
of teaching to so many
subjects, the problem
is our kids attain
horrible results.
So if you actually
take a look at-- this
is a nice mapping from
the College Board's data
on AP performance.
Most students score
between a 2 and a 3
out of 5, which doesn't sound
so bad until you realize
that, in order to
get a 5, you're
only talking about getting 50%
to 70% of the possible points.
So a 2 or 3, if we're generous
with our scaling of the College
Board's rating system, might
be somewhere like 20% to 40%
of the possible points.
So we're spending so
much time over indexing
on skills like working memory,
and like processing speed,
and basic critical thinking
that are absolutely
necessary to the world
of employment now,
and we're absolutely
building assessments
to kind of get at the core
of those challenges, as well,
but they're in no way
defining a new ceiling.
They're just setting a
floor that ultimately is
too low for too many students.
And long term kind of when
you think about the evolution
of tests over the last 100 years
or so-- this is just a snapshot
since the 1990s--
you can see that AP
is exactly testing
this kind of rote
processing memorization,
basic critical thinking.
These have exploded
at a faster rate
than any of our other
college admissions tests.
So our goal is to
think about, OK,
if we're coming in to replace
this ecosystem of SAT, ACT,
AP exams, how do we go beyond
just the cognitive skills
that everyone knows that
we've already been testing
in schools, and
have tried to test
for a long time to the skills
that purely differentiate
human intelligence?
I think creativity and
imagination are kind of
amongst the most important
and earliest of those skills
that we're going to take
on that are markedly
different than what
traditional assessments used
in the educational
context try and get at.
DAVID BARRY: And why do you
think the traditional testing
methodologies have
discounted these areas
or evolved in a way where they
didn't incorporate the picture?
REBECCA KANTAR: I think
the main reason you've
seen the traditional players of
really the College Board, ETS,
and the ACT stay
around for a while
and have such a strong
hold on this market
is, one, because
it's really hard
to disrupt their distribution.
So as soon as schools,
and parents, and colleges
all coalesce around one market
of AP, SAT, ACT, very hard
to change those standards, but
that's true in any other market
where standards
are set, as well.
But beyond that, turns out
that cognitive assessment
and assessment of anything
that's a latent trait that I
can't directly observe you
doing and say, oh, here
is how good your coding results
were, for example-- that would
be much less latent, much
less difficult to get at,
but I can't open up your
brain and say, oh, there
was problem solving.
I saw it right there.
I have to infer based
on some evidence stream
that you're either capable at
a high level of problem solving
for some environment or context
in which you'd want to use it,
or not.
And doing that is
just really hard.
We haven't had the technology
until quite recently
to be able to pull apart
in non-black box ML models
this idea of how your
every click, or time stamp,
or mouse movement in
our environment maps
to one of these high level
skills like critical thinking.
We haven't been able to
diagnose all of those behaviors
and then trace them in a really
understandable, explainable,
and defensible way,
which is important
when you talk about assessment.
And we also haven't
been able to untangle
when a solution like a
simulation based or a game
based assessment isn't
working, when you don't
see great distributions,
when everyone
fails or everyone passes on
some of your questions or tasks.
We haven't been able to figure
out where things go wrong.
So unpacking was it a UI
problem, was it an art problem,
was it a language
problem, was it
an underlying assessment
problem, also really hard.
So I think that
keeps the barrier
to entry in this space of
trying to move beyond pen
and paper, multiple choice--
keeps the barrier
to entry very high.
And when you think about
the added complexity
of regulatory
environments, and some
of what the
government tries to do
around coupling
huge initiatives,
such as Race to the Top,
or No Child Left Behind,
or the Common Core standards,
with assessments that
try and do those major
policy initiatives justice,
you kind of witness
the inability
of the modern
players in the market
to really meet the demands.
Because how do you
test life skills
in multiple choice tests?
You can't.
I could ask you a million kind
of self-reflective questions
about yourself, but
those aren't always
going to be really
indicative of what you're
truly like in a
high stakes context,
because something is
on the line, which
means you have
incentive to kind of say
something perhaps a little bit
different than what's true.
So when you think about this
ecosystem of so many schools,
so much distribution,
coupled with it's
really hard to
move to technology
because you never know
when things go wrong,
or how they're going
wrong, and it's really hard
to deal with all the data that
comes out, finally that it's
really expensive, you
have a lot of inertia
in the system keeping
it the way it is
and very little reason
to try and pursue
building new assessments
from first principles.
DAVID BARRY: And that's the key
reason why there's sort of--
so when we're talking
about the shortcomings
of the current model, I think
it's something that I've known.
You're kind of articulating
it in a better way,
but what's really
preventing people
from trying to fix it in the
way sort of like you're doing?
Is it mostly just the entrenched
system is so in place,
and it's hard to crack
and change things?
REBECCA KANTAR: Yeah,
I'd say all the points
I've just made around
government regulation
coupled with expense, coupled
with market dynamics--
I mean, all of these things
coalesce to make it hard,
but past that there's this
broken link in infrastructure
in our system where
folks haven't really
done a bottom up
analysis of what
jobs now and jobs in
the future are going
to require in terms of skills.
So one of the hardest
questions to answer
is, what should
you be measuring?
And how do you know
what good looks like?
And we haven't done this kind
of generalized analysis--
not under the lens of chemistry
and what problem solving looks
like in chemistry, or under
the lens of what good math
looks like.
We haven't done this
kind of analysis of,
what does critical
thinking mean?
What kinds of critical
thinking do people
need to be capable of?
What does creativity mean?
What kinds of creativity do
people need to be capable of?
No one has done this
linking of employment
back to education for 100 years.
So when you go and
try and research,
OK, let's build a creativity
assessment, well, good luck.
How are you defining it?
Is it just simply the number
of ideas people come up with
and the nature of how
divergent those ideas are?
That's usually the
barometer folks
use in the cognitive
space for creativity,
but that's not that practical.
That doesn't tell me about
the nuanced differences
between what it takes
to be creative at IDEO,
what it takes to be
creative at Google,
and what it takes to
be creative at Pixar.
All three places require
creativity, but how do we know?
How do we describe the
differences in those places?
And what do their distributions
look like in terms of the data?
So I think the final barrier
to really preventing folks
from going after cognitive
assessment in this high stakes
environment is you have to be
predictive of whatever you're
trying to assess.
So if you have no
correlation between what
you say people are like and
then how they do on a job
or in school, your
tests aren't going
to be around for
a long time, which
means you have to do this hard
work of figuring out what's
relevant to that job or to that
person's performance in school
or whatever context.
And that's a huge
undertaking, as well.
It's a lot of chugging along,
doing observation, doing
research, experimenting, and
spending years just patiently
getting to the answer of what
better assessment requires.
DAVID BARRY: And that's
what you guys are
working on doing [INAUDIBLE]
REBECCA KANTAR: Yeah,
as long as it takes.
DAVID BARRY: Whenever I think of
an online platform for testing,
I'm worried about maybe hacking
or cheating in some way.
How do you protect against that?
REBECCA KANTAR: So
thinking about if you're
building an assessment,
whether it's
around collaborative problem
solving or creativity,
and you have something
that's game-like,
people always have an incentive
to try and beat the game.
I mean, this is a known problem
for all sorts of massive multi
online player worlds and games.
The key difference
between an MMO game
and somewhere like Imbellus
is it's fine for humans
to create all the variance
between one version of "World
of Warcraft" when you
sit down to it one day
and the next
version the next day
because you're not trying
to hold some threshold
understanding of what
everyone started as when they
entered "World of Warcraft."
You're just trying
to understand how
they do in that point of time
and keep them in a flow state.
But for us, it's
critically important
that I understand the starting
state of your assessment
and the starting state
of my assessment.
That means yours can't
be wildly harder,
and mine can't be wildly
easier, or it wouldn't be fair.
It also means that, since I have
no understanding of what you're
like cognitively until you
come into the assessment
and start performing--
and I don't
have an understanding of what
anyone else is like cognitively
before they come
into the assessment
and start performing-- we've got
to start you at the same place
so that we can watch your
progress in an apples
to apples way.
So not only from a
fairness perspective,
but from a measurement
perspective,
it's key that the
assessments be similar enough
that I can call them comparable,
but then different enough--
not caused by humans, because
if it's all caused by humans,
your state could end up
being way harder than mine,
and we could lose
control of the system--
but controlled by
our system itself.
So when you think about
cheating, and hacking,
and gaming, the challenge
clearly becomes,
how do you keep content fresh?
How do you keep everything new
every time for every person,
but also predictable
on our side?
Because if it's not
predictable on our side,
we don't know if it was you
who actually stumbled and had
a hard time with a
problem, or if it
was someone else who created
that variance for you.
So we can't rely
on lots of people
playing at once to cause any
differences in our platform.
Instead we use AI to
basically generate
new versions of our scenarios
that are similar enough
that they are considered
equivalent as assessments,
but different enough such that,
if you went online and wrote
down on some post everything
you thought you did
and why you thought
it was right or wrong,
someone else took those notes
and then tried to use them
in their scenario,
everything has
to look different and
feel different enough
that it's of no utility.
So to make this practical,
let's think about a scenario
where a group of animals
is sick with a disease,
and the disease is spreading.
It's your job, as the
user in our assessment,
to figure out how the disease is
spreading, which disease it is,
which animals are
being impacted,
how the group dynamics
or the herd dynamics
are feeding into that equation,
and to then develop a solution,
try it out, see
how well it works,
adapt as the simulation
gives you feedback.
Now, yours could be happening
in the desert with camels.
Mine could be happening in the
mountains with mountain goats.
All of the properties
driving which disease,
how it's spreading--
everything has to be emergent.
So your environment is totally
different than my environment,
and how things
unfold are therefore
going to be different,
meaning our optimal solutions
are going to be different.
Now, beyond just
the solution state,
we're also looking at every
click, every time stamp,
how you do the processing
in that environment.
So you could think
of it like chess.
Maybe someone memorizes
some openings on chess,
and they execute those, but
that's a little different
than the person who
sees 12 moves ahead
and is executing a careful
strategy of distracting you
in one region of the
board so that they
can take advantage of
your dislocated attention
in another region of the board.
So when you think about
that as an analogy,
our goal is to
build an assessment
that, even if you've seen
some YouTube videos online,
doesn't meaningfully impact
your odds at actually doing well
in the environment.
And just to back
up with one thing
that I don't think
we've hit, the reason
we choose the natural
world is largely
because we have
such a rich context
for creating this kind of
anti-cheating variance.
If I were to use an office
setting, because there are so
many strict ties of reality
and so many expectations
that, if you grew up
in one country or one
city versus another, you might
hold, it's very hard for us
to create the same
level of variance
in a procedural or
an AI based solution
if we're using
people writing memos
to their product managers.
Suddenly this possibility space
becomes really constrained,
and it's very difficult
to kind of generate
this rich set of property
or data driven environments
that can constantly
unfold in new ways.
So that's why you'll notice
a lot of our assessments
have plants, animals, the
weather as the primary actors,
and it's your job
to cope with all
of the emergent consequences
from those drivers.
DAVID BARRY: Do you anticipate
that people who in the past
have done on the ACT, SAT will
do well on Imbellus's tests?
REBECCA KANTAR: So I think
one of the ugly truths,
unfortunately, of the
American education system
is it's not an
accident if you do well
or don't do well in college
admissions assessments.
It's more so a verdict of
the schooling you've had
and the lifestyle you've
had until that point
than your particular
skills or abilities.
Certainly, skills and abilities
of the individual, general G
intelligence people
like to talk about--
sure all of those
things are factors,
just like in every other
part of your life in terms
of why you succeed or why
you don't at task you try.
But in the education
system, disproportionately,
if you start out
disadvantaged, you're
likely to end up disadvantaged,
and if you start out
very privileged, you're likely
to end up very privileged.
And that's because,
at a state level,
when we introduce
something like Common Core,
the tests for Common Core rely
on how much teachers have been
successful in teaching
to students over a year,
and that's reflected in an
English test or a math test
starting in, say,
kindergarten through third
through fifth grade.
And that cycle of testing
how much knowledge kids have
acquired continues on
through high school.
So if your high school is
good at teaching AP courses,
and you take AP
biology and AP history,
and your teacher has it down--
they've been doing it for 10
years--
your odds of doing
better on those tests
are so much greater than someone
whose school is introducing
APs for the first time.
And the challenge that our
neediest and poorest schools
face beyond just that they
perhaps have lower teacher
retention-- they have
younger teachers.
All these things are true--
but beyond that,
the main challenge
is just they have needier
students and fewer resources
to deal with those needs.
So if you have more kids who
are English language learners,
maybe you have more families who
need holistic kind of community
supports.
It makes it really hard to focus
on teaching to AP biology or AP
history when you're still
working on literacy.
What are you going to do with
those kids in that environment
when you have so much
information to plow through?
So I would not expect kids
who have been disadvantaged
their entire lives
to suddenly correct
for that in a new assessment
between high school
and college.
I would expect our tests to
make it a little bit less awful,
and that's a terrible starting
goal, but a practical reality
of where we are.
On the flip side, if
students do really
well in the SAT, SAT
IIs, APs, I don't
think they're going to
tank our assessments.
There's no reason that
that should be the case,
but they're not going
to be as easy to fake
by just studying, and
preparing over and over,
and kind of memorizing
the format of questions,
and the format of answers, and
what's probabilistically there.
And that's important,
because if we're all
serious about wanting
a meritocratic system,
we should want to alleviate the
challenges of being born poor
versus the privilege
of being born rich,
and we should want
to get at, what
is this individual's potential?
So I think about the role of
our assessments on a macro level
as hopefully raising the floor.
We tend to focus a lot, when
we think about assessment,
on the individual,
which is logical,
because it's an assessment
for each person,
and then we talk about their
fate after that assessment.
But I take a different
view on assessment,
which is assessment
drives the system.
That's our theory-- is what you
test determines what you teach,
and how you test
determines how you teach.
So if you see it
that way, Imbellus
is focused really on
raising the floor for what
is the bare minimum
set of skills
that all people need when they
graduate from high school.
Forget college, because
for a lot of people,
the reality is it's
not in the cards,
or they're part of
the 50% of Americans
who start out in college,
don't graduate in six years,
and have $40,000
or more in debt.
When you think about
those odds, you really
need to optimize for high school
being the last public point
of intervention
in someone's life
to give them the skills
they need as an adult.
And for us, that's kind
of the thrust of what
a good assessment
should be and how
you should judge performance--
is how do we raise the floor?
Not necessarily does the kid
from Harvard-Westlake in LA
do better on our
tests than the SAT,
and are they therefore able to
get into Princeton and Harvard,
not just Columbia and Duke?
I mean, they're going
to be fine regardless.
So thinking instead
about how do we make sure
that the kids in the Bronx
who right now have teachers
scrambling to teach
to the AP bio test
are instead learning
content that's
appropriate for their
context, but most importantly,
just developing
the skills they're
going to need behind whatever
content is best for them
to engage with.
DAVID BARRY: So I remember
when I was applying to college
the SAT was kind of you had
to get above a certain score
in order to be eligible
for the college,
but after that, whether
you got a perfect score it
didn't matter that much.
Do you foresee Imbellus
maybe having more weight
in the admissions process so
that the person from the Bronx
might be able to beat the
person from Harvard-Westlake
who has all these
extracurriculars and everything
because their problem solving
skills are that much greater?
So like rather than, whereas
the SAT, like you said,
is just a floor,
this has a better way
of sort of measuring
potential of how they
might perform once they arrive.
REBECCA KANTAR: Yeah,
it's absolutely our hope,
because look, I think it's a
hard case for someone to make
that, if an individual
tests like employees
at Google, or Facebook,
or Amazon, or Goldman,
or wherever hopefully
someday we're
privileged enough to
partner with in the way
we have with McKinsey to date--
it's hard to say
that that kid is
unfit to have a
successful beginning
of their career, at least.
Who knows how they're
going to be long term?
And perhaps they're going to
have other life challenges that
come as a circumstance of
poverty, but at the very least,
they're showing
potential in terms
of how they're thinking
that's emblematic of how
lots of people at lots of other
high performing places think.
So yes, I think
you're going to be
able to find kind of diamonds
in the rough, if you will,
who right now perhaps might not
be recognized because school
is tainting their
ability to demonstrate
their raw potential.
And I don't mean raw, again,
in the inherent sense,
but perhaps they
learned a set of skills
after school watching a
sibling or working a job.
Wherever they came from, our
job should be to recognize them.
I think you bring up
an important point when
you talk about
extracurriculars, because I
would posit that over
the next 20 years
or so you're going to notice the
current lag effect of education
under serving the elites and
upper middle class in a way
that really they're
not noticing right now.
So we don't think about
that the upper middle class
families', the elite families'
kids are going to soccer
practice, and band,
and summer camp,
and all of the interactions
they're having after school
and between school years is
defining a lot of the key
skills that they need
in work and in life,
while poor kids aren't
necessarily having the same
after school experience.
They're going an hour
to and from school.
They're baby sitting
a younger sibling.
Maybe their parents
are working three jobs
and can't do
flashcards over dinner.
Whatever the circumstance,
hopefully our assessments are
able to recognize the underlying
abilities and kind of put
a little bit less emphasis on
the learned skills that might
only come from activities that
rich kids might have access
to, while poor kids don't.
But I think there's
going to be kind
of a groundswell of parents
in the next 20 years--
probably of the millennial
generation-- saying,
look, even the after school
activities aren't cutting it.
Our kids are still
woefully under prepared.
An MGI, McKinsey
Global Institute,
report that I found
valuable talks
about a survey they did
with employers where
60% of employers in
major economy countries
said that graduates were
under prepared for work when
they started on the job.
And it's like, how is
that possible after 16
years of education in either
the US, or the UK, or somewhere
similar?
How are our kids who have been
through the tried and true path
of all of public
education, plus college--
how are they unable to work
in the modern workplace now?
And how is that not going
to get exponentially worse
over the next 30 years?
That's where I
think there's going
to be a rise in public support
for a solution like Imbellus
that hopefully can cut through
whatever stuff kids are
learning that's less relevant.
But what actually
matters is, are they able
to think in a variety of ways?
And are they dexterous enough
to cope with constantly changing
content, but really have their
underlying abilities shine
through?
DAVID BARRY: So you
pointed to the William--
how do you pronounce his last?
REBECCA KANTAR: Deresiewicz.
DAVID BARRY: Deresiewicz,
the professor.
REBECCA KANTAR: Try spelling it.
DAVID BARRY: So in
his book, he talks
about how there's sort
of a double edged sword
with the Harvard-Westlake kids
who pad their resume with all
these extracurriculars.
They sort of get in
this mindset of I
need to be doing
all these things.
I need to be checking
all these boxes.
I need to be jumping
through all these hoops,
and that actually
hurts them later
on when it comes time to
sort of navigate their life,
because they just want the
most prestigious thing.
It's not necessarily
what they should be doing
or what they naturally
gravitate to.
So what I'm envisioning is
for Imbellus to have a greater
weight in the process so that
that's not reinforced so much
and that people will end
up at these institutions
where they have opportunity,
who can then do something
like you've done, where you
graduate and you sort of carve
your own path, as
opposed to just having
these super saturated
markets of people trying
to go into investment
banking, or consulting,
or private equity, or what
have you that have the most
prestige and money in them.
So I want to frame
this as a question.
So I think it's a
difficult thing to change.
It's very entrenched, and
it has a lot of social--
it has to do with a lot of
social things and values.
So I think it's very
interesting that you're
tackling it from the
testing point of view,
but I'm curious--
it seems like it's
the social value
system that sort
of has contributed
to make things this way.
So how do you think about
that and approaching that?
REBECCA KANTAR: Absolutely.
I think education in our
country exists primarily now
to serve social status.
So undoubtedly, it
would not be acceptable
for you to go to Harvard,
graduate, become a plumber,
and make $200,000 a year.
It's a ridiculous
idea because it's not
seen as a prestigious
career track when
you could go to Goldman or
Bain and do the same thing.
So thinking about the
seismic shift that
would have to happen
in this country
to embrace that,
in fact, college
isn't the only endpoint
for education--
and there's been a growing
movement towards community
college or vocational
schools, but unfortunately,
this is kind of Race to
the Top data from Obama's
balanced scorecard initiative.
When you think about
the litany of schools
that exist to serve, perhaps,
things different than the four
year liberal arts education,
a lot of them are total scams.
The average salary coming out
of those schools is $27,000.
So when you think about, again,
12 years of public education,
two or four years
of education on top
of that at a post-secondary
level, it's like,
what's the point?
That's not a living wage--
in most parts of the country,
anyway.
So when I think about
unraveling that social system,
I don't think it's practical, at
least in a short term horizon.
I think the US has embraced
the idea that it's not just
a meritocracy.
It's also hereditary
wealth that drives
how we have odds at
advancing through society
throughout our children to
adolescent to adult lives.
But what I do think
is more practical
is to actually use that
inertia to our advantage.
So I think it's becoming
less one size fits all
if you're not in
the top 10% in terms
of what you might do after
graduating from high school.
Is going to Trinity
College a great value
if you have $80,000
worth of debt?
It depends.
It really depends on what
you're going to study,
and where you're
from, and where you're
going to go to seek
employment after school,
but I think people are starting
to ask those questions.
I think the broader
implication is, how do you
stop this kind of predatory
for-profit tiny colleges,
community colleges
serving 10 or 20 students
doing digital
photography as their two
years of post-secondary
training,
and then making $5,000, or
$0, or whatever it is year,
and having ballooning debt?
So for us, the
focus should be on,
for those kids, in
the case that they
don't choose to go to college--
as regulatory environments
hopefully kind of crack down
on these types of schools that
really cannot demonstrate any
efficacy--
can we make sure that what
they did in high school
prepared them for
that lifestyle?
And I think your
point of social status
also rings true when you think
about Harvard, and Princeton,
and Yale as hedge funds that
have schools attached to them.
There's so much money,
and there's so much
power across these endowments
of major research universities
that it's unrealistic
to think that you're
going to get them to
change their mantra,
change how they sort.
People have been advocating
for a portfolio based approach
to admissions, for
example, for a long time
to try and relieve
some of the inequities.
But unless these schools see
that they have a better shot
at finding talented people
who are likely to reinforce
their social capital,
their fiscal capital,
and their intellectual capital--
meaning win Nobel prizes,
go on to graduate schools,
become professors,
continue not just to be
elite in terms of earning,
but to be elite in terms of
prestige in their intellectual
pursuits throughout
their careers--
it's unlikely that
schools are going
to embrace a new
standard, because what
they have now does the job
of sorting that they need.
The problem for
all of us is that
the downward negative
externalities
that it exerts on the
rest of the system
means that it's not just
sorting that's happening.
It's shaping of all
of our curriculum,
and that's where I
think as a society we
need to say there's
more than one end
goal than the elite
liberal arts college
or the non-elite liberal
arts college, the 300
or so derivatives of Harvard.
And we need to
embrace, instead, kids
learning what they need
to learn in high school
to prepare them
to become adults,
and then pursuing
post-secondary opportunities
that further more of
the same in a place
where there's actually market
demand for the sets of skills
they're acquiring.
DAVID BARRY: Do you
think the government
should play a role in
educational testing
or admissions testing?
REBECCA KANTAR: So
government already
does play a huge role in terms
of state standardized tests.
Lots of parents have been up
in arms about the Common Core
for years.
I think the Common
Core was a nice idea
and was really
challenging to implement.
There are very few
assessment providers
who are going to, as we talked
about a little bit earlier,
be able to measure skills
through multiple choice.
I haven't seen it yet.
I would definitely be
impressed if someone
manages to do that in a high
stakes and predictive capacity.
But thinking about
the assessment players
who are out there, the
challenge for the government
is, every time there's a new
set of Common Core standards,
or Race to the Top, or
No Child Left Behind,
or ESSA, or any of the
policy initiatives,
finding a way to
measure how well are
states doing or districts doing
on forwarding those policy
goals--
comes to student assessment,
because what else are you
going to use?
You don't want to wait
20 years necessarily
to see how socioeconomic
data plays out for a district
once people graduate from
elementary, middle, high
school, and then
start employment.
So people are
usually implementing
shorter term feedback
cycles through state
standardized tests.
And in that case, government
has provided a foundation, where
no one else has, around
cognitive assessment
and the bar for little kids,
but in the high school market,
it's much more detached.
It's much more so
the College Board
and ACT who have set the
standards both for what
we test, but also
for what we teach--
in the College Board's
case, with APs in the most
literal way possible.
There's an AP packet
and course that's
given for every single
subject, and it's teachers' job
to stick to the script.
So I think for us the hopeful
long term role of government
is to help unite lots of
players who are already
working on curricular
reform in what I see
as the right direction,
basically moving away
from that rote
content based teaching
to embrace a new
era of assessments
at at least the high
school level, maybe
some day for younger students,
as well, that go beyond just
measuring how well teachers
did at teaching content
to students, and instead look at
how well are students being fit
and prepared for
the nature of work
that they're going to encounter
in school and in life.
DAVID BARRY: It's funny.
At most high schools the course
itself is titled AP English.
So what about taking a look
at-- so I'm from Massachusetts,
and we had I think it was
called the MCAS or something,
the state test.
REBECCA KANTAR: I'm
from Massachusetts, too.
DAVID BARRY: You are, as well?
REBECCA KANTAR: So it
is indeed the MCAS.
DAVID BARRY: And if
you did well enough,
I think you could go to
UMass Amherst or something
for free or something, right?
REBECCA KANTAR: Yep.
DAVID BARRY: So I
remember that from
elementary and middle school.
You are focusing right now--
well, I guess right now you're
focusing on the post-college,
but really ultimately your goal
is to focus on high school.
What about those
important formative years,
the elementary,
middle school years?
Do you have a plan?
REBECCA KANTAR: Yeah,
so I don't think
we'll address those
anytime really soon,
and the reason why is I think
in the K through 7 level you do
need to teach some
basis of information
that everyone needs to function.
They've got to be able to read.
Kids have to be able to write.
They have to be able
to do basic math.
They should have some idea
of the country we live in,
the world we live in, the
history of those things.
So our ambition is
not to wipe out all
of the traditional system,
either in kind of the K
through 12 realm in
terms of basic subject
mastery, but also in
the college realm.
We're not advocating for
getting rid of liberal arts.
The goal, I think, long term
should be in those cases of K
through 7, how do you move away
from so much summative testing?
Because most parents
and students, I think,
feel that it's absolutely
suffocating as it exists now.
Maybe we'll start to play in
that space in 10 or 15 years,
but for right now, we want
to remain squarely focused
on college to employment.
What do people look like there?
So that we can reverse engineer
an understanding of high school
to college and what
should good look like
across the dimensions
of thinking
that really matter
in that layer.
DAVID BARRY: Great.
Well, I think that does
it for my questions.
So I think we could take
some audience questions.
There's a mic in the
back there, and we
can form a line back there.
Is there anything else,
I guess, while we're
waiting that you'd
like to mention,
or that we may have missed
in talking about Imbellus
that you'd like to plug?
REBECCA KANTAR: No, I think
that we're in good shape.
DAVID BARRY: We'll give
it another few minutes.
REBECCA KANTAR: Sure.
AUDIENCE: So you mentioned
collaborative problem solving.
Does that mean
students are going
to take the assessment at the
same time with other people?
REBECCA KANTAR:
Yeah, good question.
AUDIENCE: Or how does that work?
REBECCA KANTAR: So this
is one of the newer skills
that we're working
on, so I'd be lying
if I said we had
it all figured out,
but I can tell you it's kind of
a graveyard of assessment space
right now.
So our vision is, look, it's not
practical to think about only
using a chatbot to measure
a collaboration, which
is what most folks to
date have tried to do,
because it's not emblematic
of the kind of collaboration
that you actually do at work.
So we're thinking, how do
you integrate, for example,
a several course meal,
where, in one course,
we're looking at
your personality,
in another course
maybe we're having you
do a questionnaire that you
self-report on what you think
of how you did in various
situations, what you learned
in different environments,
what matters to you?
Then maybe you're
doing a video call
and actually going through
one of our assessments
with other people talking
about what's right,
what's wrong, how you're making
your way through processing
the information.
And then we might put you
in your own environment
where you're kind of playing
one on one with our game based
environment in our
virtual world and track
how you incorporated
everything you
learned before
everything you discussed.
So that's just emblematic
of a totally different form
of assessment.
Can we use lots of
different vantage points
to build a mosaic
of what you're like?
But it's still very,
very early days
for us to try and
figure out, how
do you have multiple people
in an environment interacting?
And how do you score that
without having humans?
Because we all
know that, once you
have humans scoring millions and
millions of kids' assessments,
you have bias creeping in.
You have people getting
tired at Friday at 5:00 PM
when it's the last
assessment they're scoring,
and that could have
an impact on scores.
So how do we stick to
our core principles
of, hey, this thing needs
to be remotely deployable,
it needs to measure
what matters,
and it needs to be somewhat
engaging and interesting
if you're going to have a
full conversation that's
at all emblematic
of what you'd have
on the job around the
assessment itself?
So very early days,
but in general,
that's kind of where we're
heading with collaboration.
AUDIENCE: Thank.
DAVID BARRY: I have one
final question, I think.
REBECCA KANTAR: Sure.
DAVID BARRY: Or do
you have one, as well?
AUDIENCE: [INAUDIBLE].
DAVID BARRY: Well, in the
meantime, I'll ask this one.
REBECCA KANTAR: Sure.
DAVID BARRY: So I
think about people
who prepare for, say,
the McKinsey interview,
where it's a problem
solving question.
If you do enough of those
brainteaser type questions,
you start to get
an knack for it.
So are you looking
at how people can
be conditioned to the Imbellus
test in trying to control for--
basically make it so that
they cannot be conditioned
and improve?
Or is it more of a situation
where the time, and energy,
and effort it takes to
get from the nothing state
to the prepared
state is less, so
that people who don't have
the leisure time to prepare,
like they do now
for the SAT, aren't
at some disproportionate
advantage?
REBECCA KANTAR: So
I would actually
flip it to say the goal
should be that it takes
many, many years to prepare.
As in, you can't
do it last minute.
That is definitely the ambition.
And in the many
years of preparation,
it's not that you need
to sit in our world
and play through the same task.
Just as you could get
better at something
like chess not just
by playing chess,
but by learning to do
abstract theoretical reasoning
and to memorize different
patterns, same idea here.
So what we're trying
to do is force schools
to embrace teaching to the
underlying sets of problems
that involve a lot of
ambiguity, where you have
to define the problem statement
and pursue different solutions,
and you maybe don't know if
a solution is perfect or not.
You have to respond to
feedback that you get.
We want you to practice
those kinds of environments
and getting comfortable
with, say, problem solving
in those environments,
but not practice directly
in our assessment.
So we will publish a
few of our tasks online,
and people can get
used to the UI,
get used to the
game mechanics, get
used to how the world works, but
not such that you can sit there
and play for 50 hours.
Not the goal.
Because I don't
think anyone really
defends that the
system should work
that way, that you shouldn't
be able to prepare overnight.
That's not the
intention of tests
that are supposed to
reflect how well you've
done in high school or your
aptitude for college and life,
depending on which way
you see these tests
kind of playing out.
So for us, that's the underlying
driver of, can we make things
challenging enough such that
maybe you learn broad brush
strokes, the kinds of context
you're going to encounter,
but you can't
memorize specifics.
And spending more
time trying to do
that versus focusing on just
developing a set of experiences
and skills across your life
and your high school years
would be futile.
Think about us as
not necessarily being
industry specific or
firm specific in terms
of assessments, but
being skill specific.
So if we were deploying an
assessment at Disney Pixar,
for example, and we wanted
to measure creativity,
because that doesn't
have any overlap--
or maybe it has some,
but limited overlap
with some of the skills that
we've been focused on to date
around problem solving--
we would go out
and study what does
it take to be successful in
those types of environments,
and what kinds of
skills are critical?
And then build assessments
that are best fit for getting
at those skills.
But right now, we are still
in a phase of building
the overall library of all
the assessments we would ever
want to have, as opposed to just
deploying the same ones over
and over, but there's
definitely a degree-- just
like any other
assessments company,
there is a degree
of generalizability
and reusability of both the
assessments and the underlying
skills.
So problem solving might
be similar at McKinsey,
as it would be in an
environment like medical school.
We have to just
go ahead and look
at what are the roles,
and what are the jobs,
and what are the tasks
that people are doing.
That should inform
the reusability
of how we deploy
various assessments
in different contexts.
DAVID BARRY: And
then ultimately, say,
looking at colleges,
would you say,
if a college has X number of
English majors versus physics
majors, is it going to be the
same sort of tailored approach
where, in a physics
major, we're looking
for these criteria or
this sort of [INAUDIBLE]
REBECCA KANTAR: It's
a good question.
I think it will somewhat
depend on how colleges want
to work with us in the future.
My guess is there's
always going to be demand
for a very one size fits all
generic standardized test that
goes between high
school and college
simply because of the
imbalance between the number
of applicants--
hundreds of thousands--
versus the number of spots
at any of the elite colleges,
and certainly en masse,
all of them together.
So I don't think you're
ever going to move away
from wanting to test
critical thinking,
standardized in some sense
that everyone agrees,
yeah, these questions or
tests are emblematic of that.
But perhaps there add ons
that one university has
more of an engineering
focus, say,
somewhere like
Olin College, where
people are doing very
application based learning.
Maybe what they're
looking for is slightly
different from a place
like Wash U, which
is slightly different from
University of Florida.
So starting to learn about what
their students are like, what
it takes to be successful
there, and tailoring
I think might happen
in the future,
but probably after there's
a more standardized unit
of problem solving, critical
thinking, creativity,
collaboration, whatever
we can jam in there
and kind of convince people
are necessary for the future
of work.
DAVID BARRY: Great.
Well, thank you, Rebecca.
Thank you, everyone, for coming.
REBECCA KANTAR: Thank you.
DAVID BARRY: Take care.
One more round of applause.
[APPLAUSE]
