- Good evening and welcome
to the John F. Kennedy Jr. Forum.
My name is Ryan Davis, and I am a junior
studying economics here at the college.
I'm also the chair of the
JFK Jr. Forum Committee
here at the Institute of Politics.
Before we begin, please
note the exit doors,
which are located on both the park side
and the JFK Street side of the forum.
In the event of an emergency,
please exit, walk to
the exit closest to you
and congregate in the JFK Park.
Please also take a moment now to silence
your cellphones.
You can join the
conversation tonight online
by tweeting with the
hashtag AIPolicyForum,
which is also listed in your program.
Please take your seats now and join me
in welcoming our guests, Dr. Jason Matheny
and tonight's moderator, Eric Rosenbach.
(audience applauds)
- Hello, everyone. Good evening.
Eric Rosenbach, as you heard.
This is Jason Matheny, the director
of IARPA, which is the
Intelligence Advanced Research Activity.
- That's close enough, absolutely.
- We're gonna cut right to the chase here.
Tonight we're gonna talk
about AI and public policy,
which, as you all know,
is a big topic not only
at the Kennedy School,
but in the real world, and also in the
intelligence community.
We're very lucky to have Jason here.
This is a smart guy.
You can read his bio and see that even
in the hallowed halls of Harvard,
this is someone who can
probably out-degree,
degree for degree, almost anyone here.
He also is a thoughtful guy.
There's not always a strong correlation
between people who have lots of degrees
being actually thoughtful.
In this case, that's true.
Jason, also, he's a
good guy, so that's what
we look for most when we're trying
to bring a good guest, is people who
are smart, they're
thoughtful and they're also
good people.
What we'd like to do, because Jason,
in his position at IARPA,
has a lot of exposure to AI,
machine learning and the nexus of that
in national security, have
a conversation with him.
I'm gonna start in, ask
him a couple questions.
We'll do that for about the first 30
or 35 minutes.
Then I'll go out to y'all in the audience
and let you ask some questions, too.
For starters, Jason,
since I can't remember
exactly what the acronym
for IARPA stands for,
tell us a little bit about that.
What is IARPA?
It's not a DARPA.
What is it and what are
you doing on a daily basis
as the director?
- We do rhyme. (Eric laughs)
We've got that down.
First, it's really such
a pleasure being here.
I think the work that you all
are doing here makes us
smarter within government.
When you're critical of
the kinds of policy choices that are made,
when you're thoughtful about what we could
be doing that we're not already doing,
you make us smarter and wiser.
We're grateful for that.
I really do deeply value
these kinds of engagements.
It's also just a personal privilege
to be up here with a national hero
and somebody who really has set an example
not only for national service,
but for being a good human being.
That's important to us, too.
- [Eric] You already got your forum spot,
so you don't have to. (men laugh)
- But I wanna be invited back.
IARPA is a organization that funds
advanced research for
national intelligence.
We fund basic and applied research at over
500 universities, colleges,
companies, national labs.
Most of that work is done
unclassified, in the open,
as a level playing field competitions.
We sort of run research tournaments.
We fund work as diverse
as applied research in mathematics,
computer science, physics,
chemistry, biology, neuroscience.
We fund a large amount of
work in social sciences,
understanding how human analysts can make
errors in judgment, how
they can avoid those errors.
We fund work in political science
and sociology, even anthropology.
We have one of the most
diverse portfolios, I think,
of any federal R and D organization.
But that work only
succeeds because we fund
places like Harvard to actually solve
our hardest problems for us.
We don't have any researchers in house.
In fact, if you come and visits IARPA,
it's sort of a letdown
'cause you sort of imagine
that you're visiting Q branch from the
James Bond movies, and that we'll have-
- [Eric] You're not doing
the weaponized watches there?
- No weaponized watches.
We outsource all of that
to Harvard. (laughs)
- [Eric] It's in a deep
corner of the Belfer Center.
- That's right.
They're all Timexes from circa 1983
to the big calculator watch versions.
- [Eric] Ash Carter has one.
(Jeremy laughs)
- The work that we do is
really focused on trying to understand
how we can make better
intelligence judgments
about a very complex world faster
and in a way that's more accurate.
That means, then, that we need to have
better methods of collecting data,
better methods of analyzing data,
better methods of assessing
the voracity of data,
which has led to fairly large investments
in AI, the topic for tonight.
- One of the things we wanna do
in the forum is also educate people
and have them actually understand what
some things mean.
When you hear about AI
and machine learning,
sometimes hard to understand
what that actually is.
In my class, I like to make students
break things down by
their component parts.
Break down AI and machine learning.
What does that actually mean?
Maybe what's the
difference and what are the
component parts?
- AI is a broader category
than machine learning.
AI really is a set of techniques
that could be used to allow a machine
to mimic or replicate some aspects
of human cognition or animal cognition.
The ability to reason, plan,
act autonomously, make inferences,
make judgments.
AI can be separated into things like
expert systems, which are rule-based,
if this, then do that, or robotics.
But then there's a subset of AI
called machine learning, which is really
focused on developing systems that learn
based on available data or the experience
of the system.
There are a few different components
to being able to do that kind of learning.
First, you need to have
an algorithm or a model.
You need to have data from which to learn,
and then you need to have computing.
With those three elements, you're able
to achieve a number of fairly powerful
performance
milestones.
Just recently in the last several years,
there's been a lot of low-hanging fruit
in machine learning that's been plucked,
really owing to techniques
that were developed
about 25 years ago.
A whole family of machine
learning techniques
called deep learning or deep neural nets
has proven remarkably
powerful when you have
large volumes of data, when you have
plentiful and cheap computing and when
you have some algorithms that really
haven't changed all that much in the last
25 years, but with some modifications,
have proven quite powerful.
A number of very hard problems in
machine learning have been solved over
a short period of time, just within, say,
the last six years.
I think, really, the major milestone
was image classifiers being applied
to something called ImageNet,
and proving that these
deep learning techniques
could achieve human levels of performance
very quickly.
- If you follow one line of a
philosophical debate, it would be whether
it's the algorithm that matters more
and having a great algorithm,
or having great data.
What you just said about the idea
of the algorithms themselves
not actually evolving
that much over the last 25 years,
maybe some of the same models that
some of the Kennedy School students use.
Is it that the data and the availability
of data has become more abundant,
or that the algorithms are getting better?
Which really matters more if you have
to make a final call on it?
- I think computer
scientists would probably
be irritated by my saying that we haven't
made so many fundamental advances
in algorithms over the last two decades.
There are some important
classes of new work
in generative adversarial networks
and reinforcement learning that really
are fundamental, but at the same time,
I think we've seen the greatest gains
of improvement in the availability
of high-quality label data
and in computing.
Computing, especially.
We've seen what's sometimes
called Moore's law,
this improvement in performance
per unit cost that tends to double
every one to two years,
and has historically
for quite a long time.
But I think if one were to break down
in the most significant applications
of machine learning, say,
in image classifications,
speech recognition, machine translation,
navigation, it's a pretty even trade-off
between data and compute.
- You decided not to give a clear answer.
(Jason laughs) That's okay.
- Is that a timeliner tradition?
- Exactly.
If you were my student, I would
nail you down any further, but we'll
get down to some of the
hard policy questions here.
When you're the director of IARPA,
who's your boss?
How do you decide what investments
you're gonna make?
Then AI in particular, do you follow
the lead of the Director
of National Intelligence
or the Secretary of Defense?
How do you decide what you're gonna do?
- We're given quite a lot of latitude.
- Specifically in AI.
- We're given quite a lot of latitude
to be thinking over the horizon about
challenges that may not yet be a
national intelligence priority,
that may not be a crisis yet,
but would represent a deep challenge,
say, in 10 years to national intelligence
or an opportunity in 10 years
because research that we fund
takes at least five years to pay off,
and very often as much as 10 years.
For example, we're funding a lot of work
in quantum computing,
for which we're probably
the largest funder of academic research.
We fund a lot of work in
superconducting computing,
which has applications
that will really pay off
probably in five to 10 years.
I think many of the investments that
we're making in machine learning
will really be paying off on the
10 year time scale.
For instance, we have a
program called Microns
that's aiming to understand
how animal brains
actually learn from data,
which is very different
from the way that machines currently using
machine learning learn from data.
We know that there are
these repeated circuits
within the brains of animals, within the
neocortex, that learn from very small
datasets, which is very different from
the way that most machine learning
is used today, where you require thousands
of examples.
Yet, most toddlers do not need to see
a thousand examples of
a chair to recognize
what a chair is.
How is it that they're
able to be so efficient?
That's not a crisis.
We're not suffering from a problem
of achieving toddler
parity in machine learning,
but it's a fundamental problem that
for us to make progress
in machine learning,
we really need to take
some more inspiration
from how animals learn.
- It's not an
operational-type organization
where you may get a call one day
from the White House
National Security staff
and they say, "We need
an answer in a month
"on how we can use AI to conduct
"better surveillance of video cameras
"across the United States."
- No, we're the place where you would call
if you wanna understand what is the
research landscape look
like, what's the state
of technology in the laboratory.
We help on that side
so that decision makers
can get wiser about
what's actually happening
in science and technology.
But our programs take so long to run
that if somebody wants
a solution next month
to a problem, it's gonna take us a month
just to award the grant
to Harvard to start
working on that problem.
- Although we're the fastest, I'm sure.
- Oh, yeah, I'm sure, definitely.
(Eric laughs)
- There's always a question about
when you're doing work on AI,
and in your case in particular,
whether you're trying to find something
that would replace a human
or augment a human in making a decision.
In the intelligence context, this is a
pretty big question,
too, because I remember
even when I was an Army
intelligence officer,
and this was a long time
ago, I was in charge
of a Signals Intelligence Unit.
I collected telephone calls and email.
We were both doing it for Yugoslavia,
but also trying to collect
on bin Laden back then.
It's interesting 'cause the linguists
were led to believe that within two years,
they would no longer have a job
because there were some AI mechanism
that would go through
all these voice cuts,
transcribe it and immediately
spit out a report.
That obviously didn't happen 'cause
that was almost 20 years ago now.
But talk to me, Jason, about
the focus now in the intel community,
whether analysts, linguist, data,
and is it the objective to replace
intelligence analysts and
operators in some way,
or augment it?
- The focus is definitely on augmenting.
I think we're a long way from replacing
human analysts.
Really, the goal is to reduce
the amount of data that an analyst needs
to look at in order to make a judgment.
One of those steps to data reduction
is to be constantly scouring
the data environment
for signs that might be important,
and then perform analytic triage
for the analysts so that you present
the things that are sort of balanced.
This could be important or it might not
be important, tee it up.
To give one example, we have a program
called Open Source Indicators
that looked at many different kinds
of unclassified data for indicators
of disease outbreaks.
One of the real challenges
for detecting outbreaks
overseas is medical
reporting isn't very good.
It's spotty.
We were looking for ways
of detecting earlier
whether there was a
hemorrhagic fever outbreak,
like Ebola.
Some of the early indicators are people
staying at home rather than
going to work and back,
so absenteeism.
One way to detect absenteeism is if
mobile phones stay at home rather than
going to work and back.
You don't need personal-level data,
you just need population-level data
of phones staying in one
place during a workday.
But you could also look at whether there
was an increase in crowding
at certain pharmacies
or health clinics from
commercial overhead imagery.
You could look at whether
there was increased
web search queries for symptoms for people
trying to understand what they might
be suffering from.
Or from social media messages,
they were posting their symptoms.
The volume of data that was involved
in making those kinds of judgments
was trillions of bits,
too much for any human analyst to inspect.
We trained systems that
could automatically
look through these data.
We trained them on
historical disease outbreaks
in order to identify those patterns
of behavior that indicated whether
an outbreak was taking place.
We could then get outbreak detection
that happened weeks
faster than traditional
medical surveillance.
That's the kind of problem that I think
machine learning is really well suited for
is taking the ocean of
data, and bringing it
down into something about
the size of a kiddie pool
an analyst can make sense of.
Actually really dig down into in order
to understand whether there's
a real event happening
or whether this is an
artifact from noisy data.
- What are the areas
that the intel community
itself is most interested in gaining
the leverage and the
assistance of AI-enabled tools?
- I think imagery.
We have a program called Finder
that takes an image that might be from,
say, a cellphone camera or from a
ground-based camera,
and even if that image
is not natively geotagged
by a GPS with a latitude or a longitude,
the system automatically
detects where on Earth
the picture was taken, just by looking
at features like the skyline,
the architectural features,
geologic features,
botanical features.
Automated geolocation in a way that would
be impossible for human analysts to look
at a picture of a random spot
in Afghanistan, and figure out where
that was taken.
That's led to some really important
operational successes.
Another example is machine translation
and speech recognition in any language.
Most of the big internet
companies that invest
in machine translation or speech-to-text
are really interesting in languages where
there are lots of consumers 'cause
that's where the market is.
In intelligence, we have to worry about
all those other languages, too.
We have a program called Babel and one
called Material that work
on speech recognition
and machine translation for any language,
and bringing training time down to a week,
from probably about nine to 12 months
was the norm before this program.
Getting about a two order
of magnitude improvement
in performance.
That's because there are these
highly concerted features
of language that we
can use to train models.
A fourth one I would say is being able
to infer from imagery what the function
of a building is, from satellites.
Can you tell whether a particular building
is a bomb factory or an automotive plant?
Can you label the features of the world
in ways that are useful
to intelligence analysts?
Impossible to do manually just because
there are so many buildings on Earth.
Being able to monitor
those patterns of activity
that can tell you whether something
is a weapons plant or not is too subtle,
really, for a human analyst to keep
eyeballs on all the time.
- How good is the intel community at using
these tools now?
All the examples you're giving right now,
most are in use, I expect, right now.
You're not divulging classified
info, so it's not that,
but how good are they?
How much does it actually help?
How many false positives?
Then,
I'm gonna follow up on some of the
public policy issues, of all that data
that you just talked about,
and ask a little bit about that.
- The biggest challenge
is in picking a target
for a research program
that's sufficiently hard that it would
be valuable, and not so hard that it
would be impossible.
What we've found is that usually
we underestimate just how much the
performance gains can be.
That is, we pick a number that we think
is just on the threshold
of being impossible,
and then we exceed it with
some of these techniques.
For example,
some of the speech recognition work,
you're looking at a
hundred-fold improvement
over the state-of-the-art in existing
speech recognition systems.
In our geolocation work, you're looking at
about a 100,000-fold improvement over
comparable techniques.
Doing those head-to-head comparisons,
you sort of have a
randomized control trial,
and you test your methods against
either manual practice or against what the
preexisting state-of-the-art is so that
you really do have a level
playing field tournament.
I think that's really
key to getting analysts
to see the value of these tools.
The other challenge,
though, is not so much
technical, but cultural.
How do you make analysts comfortable
in using these tools?
For that, we found one of the biggest
difficulties is baking into
machine learning systems
the need for explainability.
'Cause it's not enough
just to spit out a result.
You actually need to say, "Here's the
"evidence for this result that's come out
"of an algorithm."
We have a program called Mercury,
which is a program that
looks at raw SIGINT,
so raw intercepted communications,
for indicators of military mobilization
or terrorism activity.
These are really high consequence events,
high consequence intelligence judgments.
The analysts will not trust the outputs
unless they see what did the system find
that led to an alert,
that led to a forecast
of this activity.
We do two things.
One is we actually run
a forecasting tournament
in which the research teams are predicting
real events before they occur,
which is a lot harder
than predicting history,
we found.
We actually started these
forecasting tournaments
'cause we kept getting
these business pitches,
basically, of people
with PowerPoint slides
saying, "We could have predicted 9/11,
"trust us.
"Here's the PowerPoint slide to prove it.
"We ran our model backwards,
"and here's a big blip on 9/11."
We said that's not gonna
work 'cause we actually
need to tell whether these systems work
against events that aren't
already in our dataset.
We run a forecasting tournament to see
can these systems actually predict real
military mobilization,
real terrorism events
from intercepted communications.
They can.
For analysts to trust those forecasts,
then we need them to be
able to see the audit
or the evidence trail.
I think that's just as critical to making
accurate forecasts is to
be able to explain them.
- I think it's fair to say, having spent
almost every day for two 1/2 years
with the Secretary of Defense that
any intel analyst who
would come to Ash Carter
and say, "My machine just told me that
"the Russians are mobilizing
on the western border
"of Poland, or going into the Crimea,"
without being able to explain the data,
why they think that,
where it all came from,
would be a very uncomfortable situation
for that person.
- That's right.
- You can see it helping, but you still
would want the human, I think, to be able
to explain why and make sure it was there.
So let's talk about
another aspect of this.
Back when I was an intel officer and when
I was doing oversight at NSA and Cybercom,
the amount of data collected is enormous,
but in those organizations, there are very
strict rules on the collection of US
person data, for certain,
and even ally data,
and a lot of training that goes into that.
When you talk about what you just did,
in terms of the, I think you said
a thousand-fold increase or even higher
for geolocational data and doing terrain
or voice recognition, what's baked into
the algorithm to try to protect against
collection on a US person?
How do you deal with that, even when
you're talking about research assignments
for someone up in
Cambridge, Massachusetts,
and they're trying to design that?
- Maybe the best example of this is in
we had a program that looked at a lot
of social media.
We didn't wanna inadvertently
include social media
from US persons.
First, we removed any social media
that was in English.
Then we removed any social media that
was geolocated in the United States.
We lost some performance, probably,
because of that, because there were some
foreign social media messages
that were in English,
but we erred on the side of
removing it, rather than including it.
With our research, we do a civil liberties
and privacy protection review for any
research program that we fund,
in part to avoid things that really
don't have a place in
research to begin with,
that ultimately is going to be based
on foreign intelligence.
The other thing that we do is we invest
a lot in privacy-enhancing technologies.
We take privacy really seriously,
not only as a policy issue, but as a
technology issue.
We have a program called HECTOR,
which is focused on a
new form of encryption,
homomorphic encryption,
which can allow you
to run queries that are encrypted against
encrypted databases so that you wouldn't
have to give up personal details
when all you need to
know is how many people
in this list have the
flu or how many people
in this list have encephalitis.
You don't need to know
the names of the people.
You just want the number of cases.
This is a technology
that could allows us to,
as a society, balance privacy with a lot
of other security and public policy goals.
The other thing that I think is an example
is we had a program called ALADDIN,
which goes through posted video.
It could be videos on
people creating IEDs, how-to videos,
or martyrdom videos.
It turns out that
when people post these kinds of videos
as part of a terrorist activity,
they don't tag their videos.
They don't say this is a martyrdom video.
They don't tag it saying this
is an IED how-to video.
Instead, they post the video, they send
the URL out to their colleagues.
We wanted to be able to detect the posting
of these videos automatically.
There are not enough
human eyeballs on Earth
to monitor all of the posted video.
We had to develop tools
that can automatically
characterize that video and tell
whether this is a video that's about
a terrorist activity.
As part of that, though,
we wanted to ensure
that we weren't inadvertently collecting
on US persons.
We made sure, again, that the origin
of these videos was overseas,
that there wasn't personal
content in the videos.
Very often, that's costly to do that
kind of scrubbing, but it's worth it.
- Could
a skeptic look at the
research you're doing,
in particular on the algorithm, if,
like we talked about in the beginning,
there's the algorithm, the data and the
computing power running it and say, well,
you've got the algorithm, you've got
the computing power.
What you chose to control was the dataset.
Someone not as nice as you could take
those two other things, put it up against
a dataset that's running
on the United States,
and do something more nefarious.
Do you worry about that?
- I do.
I think all of us who work in research
recognize that
these technologies that we create
are all double-edged swords.
One of the questions that we ask ourselves
before we fund any new research program
is how could this technology
be used against us,
be used maliciously?
And what can we do to prevent misuse?
Can we insert intrinsic safeguards?
Can we insert security
measures that prevent
theft or reverse engineering?
Then we ask the question
under what conditions
would you regret having
created this technology?
I think this is something
we in the research areas
need to be thinking about,
especially in things like
AI and machine learning,
also in the biosciences
because there's risks.
How can you make
investments that are safe,
that are reliable?
Are there investments that you can make
that are more defensive, that they enjoy
an asymmetric advantage
in being used defensively
rather than offensively, such that you can
steer us towards a state
in which defense has the upper hand?
We really try to align
our research portfolio
with that goal.
- Surveillance and AI is one
tough public policy issues,
maintaining privacy,
Fourth Amendment rights,
all of those things
that are a core part of the United States.
But when I was in the
Department of Defense,
we started to get a lotta questions about
whether or not we were
gonna use AI-enabled tools
to allow the use of lethal force,
literally, to take out a terrorist
in a combat zone.
What are your thoughts about that?
First, in terms of the
state of the technology
that would enable that,
and then philosophically,
what should Americans and democracies
all around the world,
what should they think
about a question like that?
- It's an important public policy debate.
Some of the advocates
for introducing more autonomy into
weapon systems note that humans,
particularly under stress,
can make bad decisions about the use
of lethal force.
Opponents of autonomy or greater autonomy
in weapon systems note, though,
that there's a need
for meaningful human control
over weapon systems.
I would say particularly given the state
of machine learning, that is a real need.
The tools that exist are generally
not robust to various kinds
of spoofing,
to misinformation.
There's a class of
tricks that can be played
against machine learning systems
called adversarial examples.
It's now a kind of favorite parlor trick
among computer science undergrads,
with how little effort can I fool
this state-of-the-art image classifier
into thinking that this picture of a tank
is actually a picture of a school bus?
It doesn't take much work.
I think we're a long ways off from where
I, at least, would be comfortable
having a significant
degree of automation in,
say, targeting or weapons decisions.
I think our defense department has been
pretty wise, then, about establishing
policies that prevent the automation
within that targeting
decision.
I think much of that owes to you
and during the period
that you were helping
to formulate policy.
If you don't mind, I'll
turn around the question
just a bit, and ask how do you
think about this and where do you think
the world is likely to
converge on an equilibrium,
if it does, with respect
to lethal autonomy?
- That's the classic Socratic method,
you turn it back on the prof when
there's a hard question.
When we were working on
this in the Pentagon,
believe it or not, to me, this was not
a hard question at all.
Maybe this will sound overly simplistic,
but I can barely get Siri to work
to tell me where the
closest hamburger joint is,
more or less rely on
some fancy AI algorithm
to make an informed decision about
when to use lethal force.
In particular, because
when you're at the top of the pyramid
in the Pentagon and you're making
really sensitive
decisions about when to do
a strike, and the times
that it would matter most,
the process for that is so rigorous,
it's only approved by Secretary Carter.
In some cases, only by the president.
That's after a series of
four different briefings
in which you ask a hundred hard questions,
and only then you would give the approval.
The idea that there's gonna be a machine
and data and a fancy
algorithm that does that,
based on facial recognition, geolocation
and terrain, and say
this is the right person
and this is the right time,
and the collateral damage
estimate is correct, too, I'm gonna fire
without approval seems very far fetched,
for the United States, anyway.
I can see how others would do it.
I think for us, it was an easy decision.
We put that in policy more than anything
just to put at east the people who
were very concerned
that we were programming
drones with AI algorithms to start
going out and bombing people.
But let's look at it
from another perspective.
We're at the Kennedy School, and this
is a very international place.
Do you think all other nations would have
that same approach?
You see a lot about what
others are doing, too.
What are things that concern you from
what you see the rest of the world doing?
- I'm worried.
I think that other countries
haven't demonstrated the same degree
of reservation
about inserting autonomy into some very
high-stakes military decisions.
One public example is the perimeter system
that's used by Russia for
nuclear command and control,
which involves a high
degree of automation.
If there's ever a place where you would
not want automation,
it's in nuclear command and control.
The notion that you
could have a nuclear war
that's started
due to a series of errors,
of computer errors,
of sensor errors,
is, I think, one that
makes me anxious.
- Can you explain that a little bit more,
just that system and the way it works?
This is the unclassified stuff, again.
He's far too wise to disclose
classified information.
From what we know in reading in the
unclass world, what is it that this,
how does this work?
- Right, so I'll just quote Wikipedia.
- Perfect. (men laugh)
- There's a whole book, by the way,
on the perimeter system,
called The Dead Hand
by David Hoffman.
The notion originally for developing a
autonomous nuclear
command and control system
within Russia was that if decision makers
felt like they had a very short time
in which to make a decision about
a second strike, a retaliatory strike,
if they felt as though
an attack were underway,
they might make really bad decisions.
They might make decisions
under uncertainty,
under time pressure then, in fact,
were based on bad information.
The original intent was let's relieve
the time pressure such
that even if the Kremlin
were destroyed in a nuclear attack,
there would be a retaliation that would
be certain to occur.
Through a set of ground sensors to detect
a nuclear attack in Russia,
there would then be a
automated retaliatory strike.
Well, there's any number of ways that one
could imagine that going wrong,
whether it's a terrorist
nuclear detonation
or sensor error or computer error.
All of this is the fodder for 1/2
of the science fiction movies I've seen.
Those movies usually don't end well.
I think the notion that we would have
autonomy in such critical
systems seems like
one that's worth more attention.
I'm actually surprised
by the degree to which
this doesn't come up more frequently.
It's not just thinking about what would we
wish that other countries would avoid,
including into their lethal
autonomous weapons systems,
but also what are the stakes of those
lethal autonomous weapons systems?
There's certainly a continuum.
I would say nuclear command and control
is at the rightmost part of that continuum
where the most attention is needed.
- This fall, in connection
with the Russian angle,
President Putin said,
"Artificial intelligence
"is the future not only of Russia,
"but of all mankind.
"Whoever becomes the leader in this sphere
"will become the ruler of the world."
To which Elon Musk responded,
"Competition for AI superiority
at the national level
"is the most likely
cause for World War III."
He said, "IMO."
He was tweeting,
in his modest opinion, I think.
To which Mark Zuckerberg responded,
"Musk's AI doomsday rhetoric
is pretty irresponsible."
What do you think about all that?
Who's right? (audience laughs)
- Thanks for the softball.
- Elon Musk or Vladimir Putin?
- Is there a fourth option?
- Nope. (men laugh)
When you run the chair,
you get a whole bunch
of bad options to make
pick from. (Jason laughs)
Is that a framework to think about this?
What's your perspective on all this?
- I think that there are reasons
to be concerned about the vulnerabilities
of machine learning systems.
Without getting too exotic, I worry
less about Skynet and Terminator
than I do just about digital flubber.
Just machines that are programmed
to do something,
and they're poorly specified because
the programmers made errors or because
the sensor data going into
the system was errorful.
I think
including autonomy in fairly
high-stakes systems
is something we should
be very careful about,
and not just in the
area of weapon systems.
Also in financial systems,
also in power systems.
We have examples of algorithmic traders
run amok in our financial system.
We have instances
of power grids run amok
due to automated forms of control.
We do, as, I think, a society,
need to become a bit wiser about
what are the vulnerabilities
of these systems
before we deploy them into networks
that have high stakes.
I think that the long-term trajectory
of these technologies
offers enormous promise.
Benefits to healthcare and being able
to do more accurate and
earlier diagnosis of disease.
Benefits to accelerating the rate
of scientific innovation by automatically
generating hypotheses and testing them
to see which have more
explanatory power over data,
data volumes that human scientists
may not be able to fully analyze.
Improvements in material science,
in biochemistry,
in autonomous transport.
All of this has enormous upside potential
for humanity,
but in order to navigate the various
speed bumps along the way, I think
being mindful of some of the safety
and vulnerability and
reliability questions
is something that is
worth paying attention to.
The thing that really
actually kind of occupies
a large part of where IARPA is investing
in machine learning reliability
is making sure that our systems are robust
to human error,
to intentional attack, so feeding
misinformation to a machine
learning classifier,
a poisoning attack that
tries to compromise
the data that goes into a
machine learning system.
Then the various kinds of challenges
in cybersecurity, where
you have different forms
of automation that are responding
to cyber attacks.
We have a program called
CAUSE, which automatically
detects and even forecasts cyber attacks
based on chatter in hacker forums,
the market prices of
malware that are traded
on the black market,
patterns of help desk tickets across
an enterprise when your
computer's acting wonky.
Looking at those kinds of trends,
you want to be able to make the best
possible judgment, but
there is an arms race.
The cyber actors will be trying
to mistrain your classifiers by throwing
a whole bunch of malware
that has certain kinds of features
so that your classifiers start to train
on those features, and
then send a different
kind of malware that
really has the payload
that doesn't have that feature.
It's an extraordinary
machine deception arms race
that we're in.
I think this convergence of policy,
of cybersecurity and machine learning
computer science research will get
very busy over the next decade.
- That's a great way
to transition to going
to audience questions here in a second.
Just that area of cyber and AI
is a very rich field in and of itself.
Hopefully some students here tonight
get motivated by what
you just talked about,
and the idea of deception for AI,
which is the old, classic military feint
so that you then go in
with the real cyber attack.
I can see a whole thesis and line of books
just written on that in a
cyber strategy perspective.
But we do have a lot of really bright,
motivated,
hardworking Kennedy School students here,
and students from around Cambridge, too.
What's your advice to them about how
to get into this field?
Is it in the national security space?
In the private sector?
Where do you think it's most interesting
and where are the most interesting
public policy questions?
- Well, my main advice
is to try to find opportunities to work
at the intersection of
policy and machine learning because
right now, there is nothing
close to the pipeline
of expertise at that intersection
that we'll need in the coming decades.
If you're looking for
a future in policy and you're interested
in machine learning, I think
you'll have lots of opportunities,
whether it's in government and places
like the Office of Science
and Technology policy,
or in Congress or in OMB,
or it's in industry, helping industry
to navigate some of these
very challenging policy issues.
I think I'm really encouraged
that there are companies that
have realized that this is a priority
and have started to hire people
who are at that intersection
of computer science
and policy studies.
Then, lastly, in academia,
for think tanks, like Belfer,
to really start building that pipeline,
that generation of expertise that
we'll need to make us smarter
and navigate some of the policy challenges
that we'll have in the future.
Whether you're a lawyer or a
public policy scholar
or a computer scientist,
or you're all three, we need you.
- That's good.
That should get a lot of
you motivated out there.
Why don't we do this?
Now, I'd like to turn to questions
from the audience.
For those of you who've been in the forum,
you know how this works.
If you could start by lining up
at the microphones.
I'll call on questions
as people come there.
If you could, please,
state your name, kinda tell us who you are
very briefly, just so we know.
Remember, what we're looking for here
is a question, not a speech.
A question is an interrogative that ends
with a question mark, and
usually is one sentence.
If you need to set up with one sentence,
it's okay, but we're just trying to get
as much interaction here as possible.
I'm looking around.
Yes, sir, we'll go to you first.
- Hi, I'm Elliot. I'm a math student here.
You made a point earlier about how
there's a danger involving AI running
nuclear systems because
of possible radar errors
or any other mistakes leading to an
unnecessary strike.
How is that much different
from the status quo,
where, if I'm not mistaken,
there have been such
close calls in the past?
- Yeah, there are an alarming number
of close calls, strategic miscalculation,
nuclear accidents.
I think if you read a book like,
say, The Limits of Safety by Scott Sagan
that sort of goes through them,
the one thing that saved us every time
was a human being looking at the data
and saying, "This doesn't make sense.
"This is not real."
That's what I wanna ensure is always
in the loop is a human being.
- I would just say
something very quickly, too.
When I was assistant
secretary, one of the things
I was responsible for was
nuclear command and control.
It is not easy to launch a nuclear weapon.
It is not just the president, no matter
who it is, has the idea that he or she
can launch a nuke.
It happens there are
hundreds of people in a chain
that make something like that happen,
which is very different, I think,
than an AI-enabled decision
making process, for sure.
Should make some of
y'all feel better, too.
Yes, sir, go ahead.
- Hi, my name is Alex.
I'm a software engineer.
You mentioned earlier the idea of some
defensive technologies, technologies that
are easier to use to
protect than to do harm.
I was wondering if you could elaborate
on what sort of things those are
and where one can find them.
- Great question.
We're really interested in the idea
of safety engineering
or intrinsic security
within machine learning systems.
I think one really interesting area
for research is machine learning
on encrypted data.
Microsoft had a piece on crypto nets,
which is on the archive.
The idea if you could train a classifier
on encrypted data, not only would that
allow it to be privacy preserving,
but also, it would prevent these kinds
of data poisoning attacks
that we're concerned about,
or the ability to reverse
engineer a dataset
from a model.
These are sometimes called
model inversion attacks,
which is another challenge.
If you're training a classifier on some
classified data, the classifier
then has features of that data,
which then really means
that the classifier itself
is classified.
That's awkward.
That's awkward, if you wanna get
maximum use out of these tools.
Secure machine learning,
I think in general,
is a really exciting area for a lot
of technical work.
- Thank you. Yes, sir.
- Hi, my name is Sikandar.
I'm coming from Pakistan.
I have a question with regard
to advising China in this
sphere of innovation.
According to the latest data by
the National Science Foundation,
it said that China's in five, 10 years,
going to become the superpower in
key areas in science and innovation.
How do you view the future
US-China relations in that regard?
What is your opinion on latest issue
with regard to the Huawei and the
Silicon Valley, where Chinese companies
are trying to invest?
- That's a great question, thank you.
- China
has a well-organized plan
for AI, in particular.
It has an AI development plan that spans
multiple years.
It has specific milestones.
In fact, the most recent policy guidance
from China includes things like
we want a speech recognition system
that's 98% accurate and we want a
machine translation
system that's 95% accurate
and image classifiers
that are 99% accurate.
It almost reads like an
IARPA or DARPA program.
There are these specific milestones,
specific schedules.
I think China has the ability to mobilize
industry, government, academia
all in the same direction.
Rather than imitating that, I think,
in the words of Eric Schmidt, we should
be more like us.
The United States leads in AI
in part because we have
a heterogeneous community of folks who
are exploring lots of things that we
would not centralize.
We wouldn't be giving orders
to universities to all do the same thing.
We wouldn't be giving orders to all
of our companies doing the same thing.
We have an extraordinary
innovation ecosystem,
and it's largely founded on universities.
We have the world's best universities
in machine learning and computer science.
We should be leveraging that.
But we also have extraordinarily
innovative companies
that have decided to put a large part
of their money in basic
and fundamental research,
which I think is another advantage
that we have.
We are getting outpublished
in machine learning
by China, but if you quality control
that publication, if you remove out the
self-citations, the US leads.
I think in order to maintain that lead,
though, it means we need
to continue to invest
in what makes us so unique globally,
which are the universities as a basis
for fundamental research.
- [Eric] Great, thank you,
that was a good question.
Yes, sir.
- Hi, Gene Freuder. I'm an AI researcher.
About a year ago, Harvard Business Review
published an article entitled
The Obama Administration's
Roadmap for AI Policy,
which discussed some reports that
President Obama's
executive office published
that laid out his plans
for the future of AI.
My question is do those documents,
will they continue to have any influence
on public policy, or are they dead because
President Obama's name
is attached to them?
- They continue to have an influence.
As one of the co-authors of that report,
I'm happy to see the influence.
The national AI R and D strategic plan
continues to inform investment decisions
at the agency level.
I was also happy to see
that the recent policy budget guidance
from the White House has a whole section
on AI and machine learning.
The national security strategy, I think
for the first time,
describes the importance
of machine learning.
I think there is a commitment in the
Office of Science and Technology policy,
in particular at the White House,
to really lead in machine learning
and to push investments
where they matter most.
Michael Kratsios
at the OSTP within the White House
just had a great interview with
The New York Times talking about
the importance of fundamental research.
I think if you look at the NSF investments
and IARPA and DARPA and NIST,
we're increasing the level of basic
and applied research.
- I'd say both when I was in the
Department of Defense and
I see on the trajectory,
that's only increasing.
Some of that, of course, is for purely
military-type research, but as in
a lot of things, like GPS, for example,
voice recognition technology,
maybe the internet, there can be off-spins
of that that I think can be helpful, too.
Yes, sir, please.
- Hi, I'm Charlie Freifeld,
associated with The Graduate
School of Arts and Sciences.
I work in quantitative management,
investment management, at the moment.
I wanna follow up on some of the things
that were already said in question.
Go into the future of,
say, 10 or 15 years,
and imagine that we have developed
artificial intelligence
to a tremendous extent,
way beyond what we have today.
Now you're in the Defense Department.
You have a computer that has
just analyzed the situation,
the military situation.
It's taken into account
all of the knowledge
that everybody has, including, let's say,
the Chinese, and it tells you that
there's a 73% probability that China
is gonna attack in the next week.
Right now, the best thing to do
is attack China first, right now.
When you query it and
say, "Well, wait a minute,
"I wanna find out how
you got that result,"
it says, "Well, I did a
hundred trillion simulations.
"That's the net result."
You don't have any ability to do
a hundred trillion simulations
and go through that.
It's an old question that
Norbert Wiener raised
70 years ago.
What happens when the machine is so smart
you have no idea how it
came to its conclusion?
Do you rely on its conclusion?
What do you do then?
- Thank you.
I think we've actually sort of had
that dilemma for the last several decades.
I, at least, can't understand
all of the parameters that, say,
a linear regression is simplifying for me.
We have had systems that were designed
to try to provide
forecasts of military tension,
based on regression models.
Yet, we didn't abdicate
our decision making
and say, okay, we should
leave it to the model.
I think it's very unlikely.
I think it's hard for me, at
least, to imagine a period
in which human decision makers
and, say, nuclear command and control
within the United States,
would defer to a computer model
in making a decision about, say,
the most consequential military actions
that they have to contemplate.
What do you think, Eric?
- I would totally agree, even in the
hypothetical fact pattern
that you gave there,
which is pretty compelling.
Unlikely, but compelling.
I can't imagine any Secretary of Defense
would ever make a decision based on that,
or it would be more than one
of 100 data points that
that person would take.
There's some things about this that
are inherently protective against
AI-enabled decisions.
First of all, there's widespread mistrust
of intelligence reporting in general
in the Pentagon and among policymakers.
That's no offense, but it is what it is
because very often,
you'll get an intel report
in the president's daily brief,
and they can't even explain the sourcing
for something that came from a human.
You may not rely on it, more or less,
again, or a fancy algorithm.
It seems very unlikely
that someone would make
a decision as consequential
as starting a war
with China based on a 73% algorithm.
- You hope the Chinese
don't go ahead and do it
as a result of their program.
- I can't speak that well for the Chinese,
but I bet even the Chinese or the Russians
would think twice about starting a war
based on an algorithm.
Yes, sir, go ahead.
- Hi, my name is Oliver Wrigley.
I'm a biomedical engineer
in the pharma space.
My question is if you're aware of any
research into defending
against social engineering
attacks augmented by
AI or machine learning.
I think we saw in the last election that
our democracy's kind
of vulnerable to these
asymmetrical information warfare tactics,
like targeting swing
voters with Facebook ads.
I'm wondering how we were planning on
addressing that?
- Thank you.
We've funded research on some aspects
of this topic.
One, for example, was looking at
how disinformation can propagate within
social media networks.
Can you predict when something
is going to take off?
Another project that we funded looked at
detecting disinformation edits
within Wikipedia.
Could you tell when somebody was trying
to manipulate a Wikipedia page?
Then we also do some work understanding
censorship patterns or
manipulation patterns
in news reporting.
DARPA has funded a program called
Social Media and Strategic Communication
looking at detection of
chat bots in social media.
There's a fair amount of research
at the basic research
level of just understanding
can you detect when somebody has a
disinformation campaign
that's centralized?
But I think much more
research is gonna be needed.
This will be a moving target because the
disinformation campaigns will get
increasingly sophisticated.
- That's a great question.
Thank you very much.
Yes, sir.
- Hi there, my name's Alex.
I'm a recent alum of MIT's
Technology and Policy Program.
My question is very similar to the
gentleman's down there,
maybe a bit more generalized.
You talked about how,
the explainability of
machine learning needs
to be key in the miliary context,
but at the same time,
deep nets are generally
not that explainable.
How do you, in the military context,
balance this trade-off
between explainability
and perhaps performance because deep nets
are currently the state-of-the-art in both
the context of funding research
and deciding what to
build in terms of models?
- At IARPA, we've had several programs
where we require explainability,
and we're willing to suffer the decrease
in performance in order to get it
'cause a tool that's
not used by an analyst
is non-performing.
Even if the tool performs well
in a technical sense, if it's not trusted
by an analyst, it's not gonna get used.
In those cases, maybe you suffer, then,
the 10% decrease in performance,
but if it's explainable, it's going
to be much more valuable to the analyst.
I do think there are some
really good research efforts now aimed
at explaining the behavior
of deep neural nets.
For example, the explainable AI program
by our colleagues in DARPA.
Then some work at IARPA that's looking at
explaining different classifier outputs,
in part, to detect manipulation.
- Thank you.
Go ahead, please.
- Hi, my name's Dan.
I'm an undergrad at Tufts.
I study Russian and
computational linguistics,
and I also work on some
IARPA-related projects.
My question relates to education
of artificial intelligence
because I see that
as being a very large roadblock for people
like me and other undergrads
who are hoping to,
like you said, eventually
get into this intersection
of public policy and
artificial intelligence.
Right now, at least as far as things seem,
there's been a huge push
of people at universities
migrating towards the private sector,
where there are just larger
financial incentives.
There's a lot of talk
about how there's a very
high demand for deep learning
and artificial intelligence researchers.
However, the supply is fairly minimal,
given that, like you
said, things have only
started up six years ago.
Maybe this is more of an education
policy-oriented question,
but how do we incentivize
people to, I guess, stick around
in the public policy sector
so that we can continue to foster
this artificial intelligence education
for future generations?
What sort of role
does IARPA play in that?
- It can often be hard
to compete on salary.
We try to appeal
to social service.
The ability to influence where
humanity's long-term
welfare can be improved,
the ability to shape technology such that
we're safer and healthier and happier,
as opposed to improving ad hits by 1%.
Not to put down industry jobs.
I think there's lots of important work
that can be done within industry to ensure
that the technologies are used
to the good of humanity.
But there's nothing like
working either within NGOs
or government agencies
or think tanks that help advise the public
and government about these important
technology issues.
The satisfaction that you gain from that,
I think, beats the salaries.
- That's a great question, thank you.
I'd just say,
having seen the Department
of Defense and NSA
struggle to recruit really
high-end cyber talent,
the market dynamics are very similar.
Pay far outstrips the government.
But in that field, when you know you can
legally go and hack the Iranians
all day long, or, in this context,
you can take on all of
these hard questions
that we've been grappling with here,
you only do that in government.
Come on, y'all.
You can do the right thing.
You can contribute to the country,
to the world, too.
You can go work for Facebook later,
but make a different now.
That was your little pep talk.
Go ahead, yes, ma'am, could you go ahead?
This'll be our last
question 'cause I think
I'm getting the hook.
We'll just go right here to you.
- So much pressure. (laughs)
Hi.
I work here.
I do research primarily on how cities
are using AI and predictive analytics.
One of the things that
we're really seeing,
in terms of policy challenge in the field,
is issues around intellectual property.
I was really curious, so
basically private companies
owning algorithms that
are deployed by cities.
Then sometimes the public advocating
for algorithmic transparency.
My question for you, I guess, is
what does the economy of this look like
in terms of national security
and in terms of intellectual property?
- We fund mostly open source software
just because it's easier to defend.
With more eyes, all your bugs are shallow,
that kind of idea.
I do think that
most of the companies that are attracting
really the world-class
talent in machine learning
have also realized that to attract them,
they typically need to free up
the models and the algorithms.
The companies treat their data
as being proprietary and the secret sauce,
but are publishing on the algorithm work.
I think that's something that,
in general, we also follow.
The intelligence community
data is the secret sauce.
But we would like for more people
to be reviewing the
algorithms and improving
those algorithms so that
they're less errorful.
- Before we wrap up,
there's one other person
I wanna recognize.
This is Charles Carithers.
Stand up here, Charles.
This is a real, live Kennedy School grad
who also works in IARPA.
He's the associate
deputy director at IARPA,
helping out Jason here, too.
We gotta give him a round
of applause 'cause he,
(audience applauds)
just like the question posed, he could
be making a lotta money someplace else,
but he's doing public service.
We appreciate that.
Thank you very much.
- He's done you all proud.
Every time he goes to a meeting and people
find out that he's from
the Kennedy School,
they're like, okay, I'm gonna go there.
(Eric laughs)
- Thank you all very much for coming,
giving the time.
Jason, thank you so much.
- Thank you.
- It was a really good talk.
I really appreciate it.
- Thank you.
(audience applauds)
