I love that I can unlock
my phone with my face,
and that Google can
predict what I’m thinking.
And that Amazon knows
exactly what I need.
It’s great that I don’t
have to hail a cab
or go to the grocery store.
Actually, I hope I never have
to drive again or navigate
or use cash or clean or
cook or work or learn.
But what if all
this technology
was trying to kill me?
The same technology that
is making your life easier
is being weaponized.
That feature that unlocks
your phone with your face,
here it is attached to a
self-learning machine gun.
It’s manufacturer, Kalashnikov,
made this video to show the
gun using object-recognition
software to identify targets.
They say it gets more
accurate the more you use it.
That drone advertised to get
awesome snowboarding shots,
here’s one that doesn't
require a pilot.
This ad shows it with a
high-explosive warhead.
It hangs out in
the sky, until it
finds an enemy
radar system, then
crashes headfirst into it.
Oh, and that driverless car
you thought was so cool,
well, here it is in tank
form at a Russian arms fair.
It’s called the T-14.
Dmitry, here, says
he sells them
to the Russian government.
That contract is part of
a trend that’s changing
the way wars are waged.
Like all good
stories, this one
starts at a Russian arms fair.
We’re a few hours
outside of Moscow.
Everyone from government
officials to gun enthusiasts
have come here to see
the latest weapons.
It’s a family affair.
Buyers want to know how
the 21st-century technology
boom can give their armies
a strategic advantage.
They want to know: Can
technology make war safer?
But some fear giving
weapons too much power
because it brings us closer
to machines that could
go out and kill on their own.
They say, we might not be
able to control weapons
like these, weapons loaded
with artificial intelligence.
“So artificial intelligence
is a study
of how to make machines
behave intelligently,
which means acting
in a way that
will achieve the objectives
that they’ve been given.
And recently, I’ve become
concerned about the use of A.I.
to kill people.”
Stuart Russell.
He was an early pioneer in
artificial intelligence.
He’s also been warning people
about its potential danger
for years.
“So a killer robot is something
that locates, selects and
attacks human targets.”
Stuart isn’t so worried
about robots like this.
We’re still pretty far
from the “Terminator.”
But Stuart says we’re
not as far from something
like this bee-sized drone.
He imagined one, and made
a movie that he hopes
will freak you out.
In Stuart’s movie,
we see swarms
of them armed with explosives
set loose on their targets.
“The main issue is you’re
creating a class of weapons
of mass destruction, which can
kill millions of people, just
like a nuclear weapon.
But in fact, it’s
much easier to build,
much cheaper, much
more scalable,
in that you can use 1 or 10
or 100 or 1,000 or 10,000.
Whereas with a nuclear weapon,
it’s sort of all or nothing.
It doesn’t destroy the
city and the country
that you’re attacking.
It just kills all the
people you want to kill,
all males between 12 and 60
or all males wearing
a yarmulke in Israel.”
The weapon Stuart is
describing is terrifying,
if it works perfectly.
With the current
state of tech,
many experts say it
wouldn’t, but that
could be even scarier.
“The way we think about
A.I. is we build a machine
and we put the objective
into the machine.
And the machine
pursues the objective.
So you put in the objective
of ‘find a cure for cancer
as quickly as possible.’
Sounds great, right?
O.K. Well, probably the
fastest way to do that
is to induce tumors in the
entire human population,
and then try millions
of different
treatments simultaneously.
Then, that’s the quickest
way to find a cure.
That’s not what you meant,
but that’s what you asked for.
So we call this the
King Midas Problem.
King Midas said,
‘I want everything
I touch to turn to gold.’
And he got his wish.
And the, his food
turned to gold,
and his drink turned to
gold and his family turned
to gold.
He died in
misery and starvation.
You know, this is
a very old story.
We are unable to correctly
specify the objective.”
Machines will always
be limited by the minds
of those who made them.
We aren’t perfect.
And neither is our A.I.
Facial recognition
software has had trouble
with dark skin.
Self-driving vehicles still
need good weather and calm
streets to work safely.
We don’t know how long it
will take for researchers
to create weapons with
that kind of flexibility.
But behind closed
doors, defense labs
are working on it and
they’re not working alone.
“Militaries don’t
have to invent A.I.
It’s already being
built — it’s being
driven by major
tech companies out
in the commercial sector.”
Paul Scharre, here, led
a Department of Defense
working group that
helped establish
D.O.D. policies on A.I.
and weapons systems
for the U.S. military.
“The reality is all
of the technology
to put this together,
to build weapons that
can go out on the road, make
their own decisions to kill
human beings, exists today.”
But it’s one thing to assemble
a weapon in a lab, and another
to have it work in
any environment.
And war is messy.
“Machines are not
really at a point
today where they’re
capable of flexibly
adapting to novel situations.
And that’s a major
vulnerability in war.”
Governments around the world
see potential advantages
in these weapons.
After all, human soldiers —
they get tired, emotional.
They miss targets.
Humans get traumatized.
Machines do not.
They can react
at machine speed.
If a missile was
coming at you,
how quickly would
you want to know?
Autonomous weapons
could save lives.
“The same technology that will
help self-driving cars avoid
pedestrians could be used
to target civilians or avoid
them, intentionally.”
The problem is we’ve
gotten this wrong before.
“To really understand the
growing trends of automation
in weapons that have
been growing for decades,
you have to go all the way
back to the American Civil War,
to the Gatling Gun.
How do I describe
a Gatling Gun?
Do I have to describe it?
Could you guys show
a picture of it?
Richard Gatling
was looking at all
of the horrors that
were coming back
from the Civil War.
And he wanted to find a way
to make war more humane,
to reduce the number
of people that are
needed on the battlefield.
Wouldn’t that be amazing?”
Four people operating
Gatling’s gun
could fire the equivalent
of 100 soldiers.
Far less people would be
needed on the battlefield.
It was the precursor
to the machine gun.
And it was born with the
intention to save lives,
at least for the army
that had the gun.
Of course —
“The reality was
far, far different.
Gatling’s invention had
the very opposite effect
of what he intended.
And then it magnified the
killing and destruction
on the battlefield, by
orders of magnitude.”
Gatling was wrong.
Automating weapons
didn’t save lives.
And Dmitry, here, is saying
something eerily familiar
over 150 years later.
And it wasn’t just Gatling.
Revolutions of warfare have
typically not gone well.
“Before we ever developed
usable biological weapons,
the biologists said,
stop doing this.”
“All civilized countries
today have given up
chemical weapons as
tools of warfare,
but we see that they are still
used by some rogue nations.”
And then, there are
nuclear weapons.
Even with multiple treaties
in place to police their use,
the threat of
nuclear obliteration
remains a global anxiety.
“Now, I am become death,
a destroyer of worlds.”
“Early in the war
in Afghanistan,
I was part of a
Ranger sniper team
that was sent out to the
Afghanistan-Pakistan border
to watch infiltration routes
for foreign fighters coming
across the border.
We drove all night,
and then began
to hike up a steep
rocky mountain
under cover of darkness.
From our position
on the ridgeline,
we could see for dozens of
miles in every direction.
And by the time
the sun came up,
we looked down at this
compound beneath us.
We were basically in
someone’s backyard.
We were certain that
people would be coming
to take a closer look at us.
What I didn’t anticipate was
that they sent a little girl
to scout out our position.
She wasn’t particularly
sneaky, to be honest.
She was reporting
back our position,
and probably how many
of us there were.
We watched her and
she watched us.
And then, she left.
And pretty soon after,
the Taliban fighters came.
The gunfight that
ensued brought out
the whole village.
And we knew that many,
many more fighters
would be coming before long.
So we had to leave
that position
as we were compromised.
Later on in the day,
we talked about
what would we do in a
similar situation to that?
You know, one of the
things that never came up
was the idea of shooting
this little girl.
But here’s the thing: She
was a valid enemy combatant,
and killing her
would’ve been lawful.
So if someone deployed
an autonomous weapon,
a robot that was designed
to perfectly follow
the laws of war, it would’ve
killed this little girl
in that situation.
Now, I think that would’ve
been wrong, maybe not
legally, but morally.
But how would a robot know
the difference between what’s
legal and what’s right?”
With so much at
stake, you’d think
a debate would be happening.
Well, there is.
It’s just that technology
moves at a different pace
than diplomacy.
“We will continue our
discussion on Agenda Item 6A,
characterisation
of the systems
under consideration
in order to promote
a common understanding on
concepts and characteristics
relevant to the objectives and
purposes of the convention.”
“One of the things I
learned very quickly
was that the official
proceedings at the
United Nations appear to be
completely meaningless.”
"Thank you,
Mr. Chairperson —”
“Support continued
deliberations —”
“We need a
normative framework —”
“Difference in interpretation —”
“The importance of a
multi-disciplinary —”
“Down the rabbit hole
of endless discussions
on a subject of —”
“Thank you, Mr. President.
We are not in a position to
make a declaration right now.”
“Good morning.”
“How are you?”
“I’m good.
How are you feeling?”
“Oh, I’m fine, except for
the governments, you know,
their do-nothing attitude.”
“We’d like to
hear about that.”
Jody Williams, here, won
a Nobel Peace Prize
for her work
banning land mines.
Now, she’s
part of the
Campaign to Stop Killer Robots.
“Academics attacked
the campaign
in the beginning years,
you know, saying robotics
and A.I. are inevitable.
Maybe they are,
but applying them
to killing human
beings on their own
is not inevitable,
unless you do nothing.
And we refuse to do nothing.”
Today, the
Campaign to Stop Killer Robots
is staging a protest
outside of the U.N.
The group is made up of
activists, nonprofits,
and civil society
organizations.
The campaign’s goal?
A ban on all weapons that can
target and kill on their own.
So far, 30 countries
have joined them
in supporting a
ban, as well as 100
nongovernmental organizations,
the European Parliament,
21 Nobel laureates, and
leading scientists,
like Stephen Hawking,
Noam Chomsky and Elon Musk,
as well as Stuart Russell,
and more than 4,500 other
A.I. researchers.
Protester: “Everyone, you
can get up now.”
"Yay.”
[cheering]
Jody’s here with
Mary Wareham.
“So this is the sixth
time that governments
have come together
since 2014 to talk
about what they call lethal
autonomous weapons systems.”
We’re going to
apologize in advance
for the obtuse use of acronyms
in this portion of the video.
“We’re not trying
to prohibit the use
of artificial intelligence.
You know, it can be
beneficial to humanity.
We’re pro-robots.
We’re just anti-killer robots,
anti-fully autonomous weapons.”
“The C.C.W., the forum of the
Convention for Conventional
Weapons, — which actually
has a name this long,
and I can never remember
it — operates by consensus.
Which means you either
negotiate the lowest
common denominator,
which means
doing pretty much nothing,
or if the entire room
of diplomats wants to move
forward with a treaty,
for example, and one
state says no, then it
goes nowhere.
And that’s really a
dictatorship by one.”
“Once a bullet leaves
a gun, the rifleman
ceases to have control
over that bullet.
Autonomy is a way of extending
human control beyond the time
a munition is deployed.”
That’s the United States arguing
that A.I. will save lives.
And remember, without
their support,
any kind of regulation
can’t move forward.
“Using algorithm and software
to determine and engage
target reduces
people to objects.”
“In the U.S. perspective,
there is nothing
intrinsically valuable about
manually operating a weapon
system, as opposed
to operating it
with an autonomous function.”
The United States isn’t alone.
The countries working hardest
to build autonomous weapons
insist we can’t regulate
what doesn’t exist yet.
And at the same time,
their militaries
are developing these
weapons right now.
“The line between a
semi-autonomous weapon
that has a human in control,
and a fully autonomous weapon
could simply be a
software patch.”
“Indeed, some may say
it is similar to trying
to discuss the
internet in the ’80s,
’70s, ’60s at this stage.”
“It is not necessary or
desirable at this time,
to define laws.”
“This so-called
difficulty of definitions
continues to be
willful obfuscation.”
The truth is, whether they
exist or not just depends
on how you define them.
We don’t have weapons
with artificial general
intelligence or A.I. that’s
as smart as humans.
But we do already
have weapons that
can use A.I. to search,
select and engage targets
in specific situations.
And the technology is
only getting better.
“So it could easily
take another 10 years
before they even agree
on a definition of what
an autonomous weapon is.
And by that time,
it will be too late.
I think for some countries,
that’s the point.”
In the ongoing race between
technology and diplomacy,
technology is winning
because in this race,
the dual-use nature
of technology means
software being designed to
make your life easier clearly
has military applications.
“The A.I. community,
myself included,
we were sort of asleep at
the wheel for a long time.
And we weren’t really
thinking about the ways
that it could be misused.”
Whether we like
it or not, we’ve
entered the age
of the algorithm.
And A.I. is changing our
place on the battlefield.
Is it possible the next
generation of soldiers
won’t have to kill?
“Look, it’s an appealing
idea that, someday, robots
will just fight other robots
and no one will get hurt.
I don’t think
that’s realistic.”
“Unfortunately, if
it worked like that,
we could just say,
‘Well, why don’t we
just play baseball and decide
who wins or Tiddlywinks?’
No country is
going to surrender
until the costs that they’ve
incurred are unbearable.
So even if your
robots are defeated,
the next stage is
that their robots will
start killing your people.”
“Because the unfortunate
reality is that wars
will only end
when people die.”
