It's really good to be here.
First a quick disclaimer, the views in this
talk are mine alone and not necessarily those
of the Global Catastrophic Risk Institute.
Now the work that I'll be presenting today
comes from a broader theme in my research,
especially these three papers, which is that
essentially for us to make a productive change
on the issues that we care about, we need
to really understand the perspectives and
opportunities from those people who are really
positioned to take important actions on these.
We can't just think about what sorts of actions
we would ideally like to see on these issues.
We need to see what people could actually
do in their positions.
In order to do this we need to understand
their perspectives, how they think.
They don't necessarily think like we do.
They don't have the same beliefs, the same
values necessarily, and even if they did,
based on where they work, the institutional
context, they might not be able to take certain
types of actions.
We really would do well to kind of internalize
that and adjust our own strategies accordingly.
A good place to start is in understanding
how they think and talk about these sorts
of ideas.
So in international security, for example,
specifically in intelligence analysis, they
have a concept called "mirror imaging" or
"mirroring".
Now mirroring is the mistake that analysts
often make when they assume that the people
who they are analyzing think and act like
they do.
It's essentially like "If I was in that position,
this is what I would do".
Well, you're not in that position, it's someone
else and intelligence analysis is typically
somebody from a different country, with a
different cultural and political background,
different institutional context that they
work in, and it's a mistake to think that
they would do what you would do in that circumstance.
That works well for them and it's also an
important message for us.
A pretty classic example of this, in global
catastrophic risk, from the early 1980s and
the early studies of nuclear winter, this
is Carl Sagan and colleagues.
They were really concerned about nuclear winter
for essentially the same reasons that we might
be concerned about global catastrophic risk
and existential risk, and things like that,
which is because a nuclear winter would be
a very severe event, potentially including
loss of future generations.
They were concerned about this and concluded
therefore, we should reduce the size of nuclear
arsenals down to a level that would be too
small to cause nuclear winter.
They tried promoting this idea in international
security policy circles and met with this
response, which was that no actually, to the
extent that nuclear winter is a concern, it
is a concern that only reinforces our existing
nuclear weapons policy, it doesn't suggest
any changes.
Because the policy and particularly the nuclear
deterrence, the idea is basically nuclear
war would be so catastrophic that the only
sensible policy is to avoid war in the first
place.
Well nuclear winter just makes nuclear war
even more catastrophic than it already was,
reinforcing the deterrence policy.
This is really where the debate over nuclear
winter ended.
It was a stalemate, didn't really get much
farther, didn't change the actual nuclear
weapons policy that we had.
It was because they didn't really reconcile
with this reaction from the nuclear weapons
policy community.
More recent example, a high profile video
on lethal autonomous weapons, the term "slaughterbots"
was put out recently, and you can agree or
disagree with what the video itself was actually
saying.
I personally am somewhat skeptical, but it's
still the case that the video itself was fairly
poorly received by a lot of important people
in the international security policy communities,
and because of that it has made it more difficult
for the people behind the video to get their
message out there to these very important
audiences.
Then finally, an example with my own work,
I am not above making this mistake myself.
A few years back I published some research
on a concept I called "winter-safe deterrents".
Now this was actually an attempt to reconcile
concerns about the risk of nuclear winter
with the belief in nuclear deterrence as a
policy construct, and I said, "Okay if we
are going to reduce our nuclear arsenals down
to a level that would not cause nuclear winter,
are there other things we can do using other
types of weapons to achieve a successful deterrence
policy, so that we can have kind of the best
of both worlds."
For this I looked at a number of different
types of weapons, including biological weapons,
and it was the attention to biological weapons
in this research that prompted these responses
that you see here.
This is actually just a small portion of the
very negative reaction that this part of the
work generated.
I was not intending to do this, I was not
trying to be contrarian or anything like that.
This was a mistake.
I really did not appreciate how strongly the
international security community would come
out against even the suggestion of a positive
role for biological weapons.
I mean how I thought about it myself, in ethical
terms, thinking basically as a consequentialist,
I would look at weapons not for every type
of weapon, but for a lot of different type
of weapons, kind of like that, where for small
amounts of the weapon that are carefully used
in very select circumstances, there could
be some benefits to this society, and then
you get more problems as the arsenal size
increases.
Well what I failed to realize is that a lot
of people think about it more like that, where
there is a very large benefit to not having
any of the weapon in the first place.
Might even be larger than what's shown here,
in that they just think that the weapon itself
as a class of weapon should not be there.
This is why we see bans of a number of different
types of weapons: biological weapons, chemical
weapons, land mines, and several others.
Okay, some of that I disagree with.
Part of the argument that is made for banning
weapons is, in ethics terms, deontological.
They will argue that certain types of weapons
are fundamentally immoral.
Personally I disagree with that.
Like if I'm going to be killed in the battlefield
I don't actually care what type of weapon
it is that kills me, I'm dead anyway, and
likewise if I'm going to be hurt, I don't
care which type of weapon it is, I care about
the pain that I have to suffer.
But at the same time there are a lot of people
who think this way, and that's something that
I need to internalize and factor in.
But there's more to it than that.
Banning a class of weapons is just a simpler
approach to policy.
I mean, "ban biological weapons" - that's
three words.
You could put it on a bumper sticker, versus
like "craft a carefully optimized small quantity
of biological weapons to be used in specific
circumstances."
That's a lot more complicated, it's harder
to rally political support behind it and in
consequentialist terms that matters, and so
this is something that I have been, since
this, trying to grapple with and internalize,
and factor into my own approach to weapons
policy.
Now there are still some parts of it where
I really do think the international security
community gets it wrong.
In particular in the last few years with the
really massive international attention paid
to a fairly small handful of chemical weapons
attacks in Syria, in comparison to the much
larger number of attacks using guns and conventional
explosives, these types of weapons that just
aren't quite as salient, for whatever reason.
This just seems wrong to me.
Where we make such a big deal out of the people
who are hurt and killed by chemical weapons,
but for this much larger number of people
who just have the misfortune of being hurt
and killed by the wrong weapons, we don't
actually care about that so much.
There are some people who do care about it,
like this organization Action on Armed Violence.
They are focused specifically on this issue,
and superficially it seems like they're doing
good work.
I haven't looked into them closely, so I can't
make an endorsement of them, but it looks
like they do pretty good work on this, and
for good reason.
I mean, I'm actually myself thinking of getting
involved in this, to really help drive home
the point that there are all these lives that
are being harmed and killed on something that
we're essentially neglecting.
I think this is a good way to put in more
quantitative reasoning into international
security conversations.
That could be important for just overall.
I mean essentially it kind of looks like this.
You have your high frequency, low severity
types of weapons like conventional explosives,
then on the opposite end you have our low
frequency, high severity weapons like nuclear
weapons, which is where most of my work is,
and then you have chemical weapons, which
are in this kind of sad bottom left corner,
where we don't use them that often, when we
do the magnitude's not that large.
Okay, sure, you can cause a lot of harm with
chemical weapons, but aside from like World
War I trench warfare, 100 years ago, for the
most part that hasn't really been the case.
Now with respect to nuclear weapons we should
be careful in talking about the international
security community perspective, because there's
not just one perspective.
There are a lot of different perspectives
and we should recognize that.
This chart here summarizes, kind of what I
feel like are sort of the main perspectives
on this, where starting from the top you have
a small minority of people who actually argue
for more nuclear proliferation.
These are people who really believe in nuclear
deterrence, and there is a certain logic to
this.
If nuclear deterrence brings peace, peace
is good, so we should spread nuclear weapons
around to more countries, so that they can
have peace.
I might not believe in the effectiveness of
nuclear deterrence quite as much as they do,
but there is at least a logic to it.
That is a small minority though.
A more common view among the hawkish folks
are to essentially maintain the status quo.
We will keep our weapons because we see benefits
from them, but we don't want another countries
to have them, especially the other countries
that we don't trust so much.
Another prominent view is to have gradual
disarmament on timescales of decades or so,
as the international conditions permit.
This is like Obama, people with similar views
as him, the NGO Global Zero is big in this
space.
And then finally another very prominent view
is for a much more rapid nuclear disarmament.
ICAN is the international coalition for the
abolition of nuclear weapons.
They just won the Nobel peace prize for this.
And NWS, Non-Nuclear Weapons States, the countries
that don't have the nuclear weapons, quite
a lot of them also support this view on nuclear
disarmament.
Now I've been studying the risk of nuclear
war for many years now.
And I've gotta be honest, I'm actually not
sure which of these camps I support.
It's tricky because okay, nuclear weapons
obviously make war more severe.
That much is pretty clear, but with nuclear
deterrence they have some potential to reduce
the frequency of major war, such that if you
switched over to conventional deterrence it's
possible that you would have major wars that
occur more frequently.
And so you have this trade off between the
frequency and severity of war.
And it's actually not really obvious to me
how that trade off is resolved.
And so because of that, I have tended to shy
away from advocating specific policy positions
on nuclear disarmament - it's not quite clear
which policy position we should be pushing
for.
Instead at least a few years ago, what I would
do is push for a different type of policy,
which is to improve relations between the
different countries, especially the nuclear
weapons countries that don't get along with
each other, starting with the United States
and Russia, which have the two major nuclear
arsenals.
This is something that would clearly reduce
the risk of all types of war, and it's something
that people across all the different disarmament
camps would agree with.
Everybody agrees that if countries get along
with each other better, they're less likely
to have wars.
And that makes the world safer.
And, there are things that could be done on
this.
Something that I like to point to is a really
great project by Dorothie and Martin Hellman.
Martin Hellman, a professor at Stanford, did
early work on cryptography, and then some
work on the risk of nuclear war, that my own
risk analysis is built off of.
This is a really amazing project that the
two of them analyzed their own marriage, and
what they did to overcome their marital struggles,
and used that as a starting point for thinking
about how to improve relationships between
different countries around the world.
Really brilliant concept.
And this is the sort of thing that I would
point to as having good potential to make
progress on this front.
However, within the last year or two, things
have changed.
We now have, as I'm sure you're all aware,
a fairly different political environment,
such that if you are to talk about trying
to improve the relationship between the United
States and Russia, right now it's more complicated.
I think that's a fair way of putting it.
And in this political environment, I actually
don't have great ideas.
As far as what the best way forward, because
so much of the issues now are within the space
of domestic politics, especially within the
US.
That's really kind of emerged as a big thing.
It used to be that there as a lot of consensus
within the US about say, how to approach the
relationship with Russia, now it's just gotten
a lot more complicated.
And the important actors for domestic politics
are not technical policy analysts like myself,
it's more like journalists or grassroots political
organizers, campaign managers, of course the
politicians themselves.
And so I have not felt like I've had really
good opportunities to make a positive difference
on this front instead.
My own strategy has actually mostly been to
lay low, do some research and some quiet networking
with policy communities and sharing ideas
with them.
Which doesn't get to the root issues going
on but is still something I feel like we can
make constructive progress on.
What is clear though, is for those of us working
on international security issues, we need
to understand this and internalize this and
factor it into our plans and our strategies
because this really matters.
And who knows how this story is gonna play
out.
It's too soon to tell.
And that goes for nuclear weapons, and a lot
of other issues which as of recently includes
artificial intelligence.
This is just a small portion of the attention
that major national governments have paid
to AI as a security issue within the last
year or two.
And I would expect that there will be more
and more of this as time goes on.
Now, most of this is for narrow near-term
AI.
Not the more dramatic transformative AI that
we could have in the future.
However, when you look at that, so shown here
this is a map from a survey of artificial
general intelligence projects that I've published
last year.
And what you see is they're all over the world.
And maybe half of them are based in the United
States, but there's a lot in Europe, a lot
in China, a few in Russia and other countries
around the world.
Now most of these are private, they're either
at private companies or in academia, but some
of them have government ties and the governments
that the countries that these projects are
based in, may be paying attention to them
anyway.
I think we will expect to see more and more
of that as this field progresses.
And so for this as well, it's important for
us to understand these international security
dynamics, and also the perspectives of the
people who are making the actions on that.
So that we can customize our own efforts on
these issues accordingly.
I'll stop there, thank you.
All right, awesome.
So we do have some time for Q&A, feel free
to submit questions via the app or the website.
sf.ea.global.org/polls.
What does it take to create a nuclear winter,
what is that threshold?
There's thousand and thousands of these missiles,
how many have to go off?
That's a good question because if you have
just one nuclear weapon it's probably not
going to cause anything close to a nuclear
winter.
We had two used in WWII, that didn't really
change things that much.
What I did in my survey was to take a cautious
approach.
Which is to say that well, we don't know exactly
what it would take, let's err on the safe
side, aim low for it.
And so the number that I proposed in this
study was 50.
This was based on other research that had
studied nuclear war scenarios involving 100
nuclear weapons and they found substantial
effect.
And so I thought, okay, let's take a number
that's lower than that, but I have to say
that this 50 number was somewhat arbitrary.
It's not something that we really have a precise
understanding of.
That would be enough though, to certainly
destroy any...
I mean I guess you have to still factor in
how many of them will actually hit targets
and go through defenses and what not, but
there's no rival that can survive 50 nuclear
strikes.
So what is the argument that that's not enough?
I don't have a great intuition for that.
Sure.
And as far as what the number of nuclear weapons
that a country needs to have successful deterrence,
opinions vary quite widely.
I've seen anything from as few as 5 to as
many as like, 1500.
And the high numbers come from this idea that,
we may need weapons for a variety of contexts,
different adversaries that we may need to
simultaneously deter as well as defenses and
so on that these adversaries may have.
And so in order to have a reliable deterrent
we might need that many nuclear weapons.
I'm skeptical that we would need so many,
but the important to understand is that ultimately
deterrence is psychological.
Deterrence succeeds when the other side makes
a decision to not attack, and so ultimately
it's not about the number of weapons, it's
about the human reaction to that.
And we have, the United States has 7000 or
something?
It's in the thousands, depends on how you
count it.
And there is some interest in reducing the
size of our arsenal, certainly in the Obama
administration there was interest in that.
My sense is that the main factor for progress
on nuclear disarmament right now is the relationship
between the United States and Russia.
I think we are unlikely to unilaterally make
significant cuts to our arsenal, that we would
only do that in tandem with cuts from Russia.
What do you think the trajectory is that we're
on right now?
It seems like, well obviously we have a summit
coming up in two days, it seems like things
are not going that great.
But how do you assess it?
It's not clear that where we are right now
can be readily described as a trajectory.
Yeah.
I think that's what I would sum it.
It's kind of a random walk.
It's a volatile situation.
I live in New York, that's one of the target
cities, San Francisco is another target city.
Sometimes I have my fingers crossed.
But that's, I guess we'll see.
Have there been any historic...
I know there have been major high level arms
reduction agreements in the past, has there
been any breakthrough with a situation even
remotely like the North Korea situation?
I'm kind of struggling to come up with one.
People would maybe say Libya but now that's
like the warning to anybody else.
The Libya example is a good one because a
key attribute of the situation with North
Korea is they are a much smaller country than
we are.
So it's a very uneven dynamic in contrast
with Russia which Russia and China the US
considers "near peer" adversaries.
They might consider themselves peer adversaries.
That's the term that gets used.
North Korea I think, from North Korea's perspective
it makes sense that they would recall the
Libya example.
The United States has not been trustworthy
on these matters in the past and I would expect
North Korea to understand that and factor
that into its plans.
I would be surprised if North Korea were to
agree to give up its nuclear weapons.
I would be very surprised to be honest.
There is progress that can nonetheless be
made in a talk like this, but that gets into
more subtle technical details of the nuclear
weapons enterprise.
We'll see, we'll see.
Okay.
A number of questions have come in through
the app.
So thank you everybody in the audience.
You spoke for a minute about kind of branding
mistakes that you and others have made as
you engage with this content.
So one person just kind of asking if you could
offer any tips about how to, or advice I guess
about how to kind of engage without projecting
Snowden-style risk, if somebody really wants
to get involved.
That's a good question and the really the
answer is pretty simple.
It's just to understand their perspectives.
So for example in my own paper that I did
on this, I probably could have done a better
job of sharing early drafts with people from
this, I mean there were international security
people who saw the drafts.
The paper went through peer review, in an
international security journal and the readers
actually did not flag the thing that ended
up getting all the attention, so you never
know.
That's the sort of thing I could have done
more of and that I think applies more generally.
The more that we are kind of, of that world
and of that community then the better we can
predict how they would react to things and
get people's reactions to test ideas out before
putting them out more widely.
Okay.
We'll go through a few here rapid fire.
It is officially time for break but we've
got a chance to just run a couple minutes
long so here's a handful in quick succession.
How much of what goes on in this space of
kind of managing these international catastrophic
risks happens behind the scenes and is sort
of invisible to the public, versus kind of
gets discussed in the open?
A lot of it is behind the scenes.
There is no question about that - it's a mix.
It's a mix, the public context is important.
National leaders especially in democracies
do need to care about the domestic reaction
to what they're doing.
Not just in democracies.
In China for example there is a sense that
since Xi was made like president for life,
or whatever the term is, that he actually
now has more flexibility to pursue international
initiatives because he is in a more comfortable
place domestically.
So it goes both ways, there's always a bit
of a negotiation in two levels, internationally
and then also nationally.
The old public and private position as somebody
once famously put it.
Here's an interesting one.
Why are there so few women in the audience
right now for this talk?
Question possibly from a woman in the audience.
Do you notice that this is sort of a gendered
topic in some way?
To some extent.
It varies so, nuclear weapons more so than
say biological weapons.
In the biological weapons space the gender
breakdown is more balanced.
I, as far as why, I don't think I have any
particularly clever insightful answers.
What I can say is that there are some people
in international security who are aware of
this sort of imbalance and try to address
it.
Especially I think like when there is discrimination
against women, which probably happens.
That definitely should be recognized and countered
because we lose talent, and that's a problem.
I'm glad that there are at least some women
in the audience and hopefully more can be
in this conversation.
There is actually some sense that women bring
different perspectives that add something
very important to these conversations.
Like, for example, we talked about the marital
struggle thing, that sort of contribution
is kind of a traditionally female thing though
of course men can do it, such as Martin Hellman,
who was an equal partner on that project.
I think there's an important role there and
it's something good to pay attention to.
Okay.
A few more, we can see how many we can do.
A clarification question on this map.
What do these things kind of represent?
Are they technical projects, trying to crack
AGI?
Are they kind of ethical think tank type stuff?
Yes, that's a good question.
So these are R&D projects, these are projects
that are trying build AGI, and you see it
says lead and partner.
The lead is the country the project is based
in and the partner is when there are projects
that have other collaborators from other countries.
Okay.
What possible benefit, if you can tell a story
about the possible virtues of a mad man style
of managing such situations.
Do you see any benefit to that?
So the story is that if the other side thinks
you're crazy they're more likely to yield.
So the classic analogy in the literature on
deterrence theory is that like imagine you
and someone who you really don't like are
fighting each other while tied to each other
by rope and right by the edge of a cliff.
So what you could do is you could try to just
beat up the other person and win, or you could
do something that's kind of crazy like start
dancing and shimmying closer to the cliff
where they start to think maybe this guy is
a little crazy, maybe he actually wants to
go over the cliff or she wants to actually
go over the cliff, and maybe that will induce
them to yield.
That's kind of the basic concept.
Does it actually work in practice?
Maybe it does, maybe it doesn't.
Maybe it depends on the context but that's
the idea.
Okay.
Last question for now.
What do you...
what is your kind of outlook for the future
especially as the AGI dynamics kind of begin
to develop perhaps along similar, perhaps
along different lines to the nuclear dynamics.
Do you think we have prospect for kind of
new age of international cooperation or are
we in kind of a new dark age maybe?
There's prospect for it but there's a lot
of work to be done.
This will all play out in the context of the
international relations that we're going to
be having anyway because of all this other
stuff going on.
So a lot of it depends on what is going on
with the rest of the world.
Now with all types of AI, it's important to
recognize that it's not just international
in the sense of between one nation and another
because so much of the work is being done
privately.
So a lot of it might look less like say nuclear
weapons and more like say climate change where
with climate change it's mostly the private
sector that are the important actors.
The fossil fuel companies the energy industry
all of that, so we can learn both from the
nuclear weapons experience as far as how to
get certain types of cooperation, especially
on really high stakes threats and then also
from climate change and similar types of issues
as far as how to handle the international
cooperation on issues that are driven by the
private sector.
Well unfortunately that is all the time that
we have.
Thank you to the audience for a bunch of great
questions.
How about another round of applause for Dr.
Seth Baum.
Thank you.
