He is here from the global priorities project, which he leads as part of the Centre for Effective Altruism,
he has a PhD in mathematics from the University of Oxford
he was a founding member of Giving What we can,
and he is a James Martin Research Fellow at the Future of Humanity Institute.
He has written papers in mathematics, economics, philosophy, and his research currently focuses
on how we can prioritize between pressing problems, under conditions of uncertainty.
Owen is here today to talk about what AI means, and also what it does not mean, for the EA community.
Please join me in giving a warm welcome to Owen Cotton-Barratt.
Some of you may have noticed that a bunch
of people in this community seem to think
that AI is a big deal.
I was going to talk about that a little bit.
I think that there are a few different ideas
which feed into what we should be paying a
lot of attention to.
One is that from a moral perspective, the
biggest impacts of our actions - and perhaps
even overwhelmingly so - are the effects of
our actions today on what happens in the long
term future.
Then there's some pretty empirical ideas.
One is that artificial intelligence might
be the most radically transformative technology
that has ever been developed.
Then actually artificial intelligence is something
that we may be able to influence the development
of.
Influencing that could be a major lever over
the future.
If we think that our actions over the long
term future are important, this could be one
of the important mechanisms.
Then as well, that artificial intelligence
and the type of radically transformative artificial
intelligence could plausibly be developed
in the next few decades.
I don't know what you think of all of these
claims.
I tend to think that they're actually pretty
plausible.
For the rest of this talk, I'm going to be
treating these as assumptions, and I want
to explore the question: if we take these
seriously, where does that get us?
If you already roughly agree with these, then
you can just have a like sit back and see
how much you agree with the analysis, and
maybe that's relevant for you.
If you don't agree with one of those claims,
then you can treat this as an exercise in
understanding how other bits of the community
might think.
Maybe some of the ideas will actually be usefully
transferrable.
Either way, if there are some of these that
you haven't thought much about before, I encourage
you to go and think about them - take some
time afterwards.
Because it seems to me at least that these
are, each of these ideas is something which
potentially has large implications for how
we should be engaging in the world in this
project of trying to help it.
It seems like it's therefore the kind of thing
which is worth having an opinion on.
Okay, so I'm going to be exploring where this
gets us.
I think a cartoon view people sometimes hold
is if you believe in these ideas, then you
think everybody should quit what they're working
on, and drop everything, and go and work on
the problem of AI safety.
I think this is wrong.
I think there are some related ideas in that
vicinity where there's some truth.
But it's a much more nuanced picture.
I think for most people, it is not correct
to just quit what they're doing, to work on
something safety related instead.
But I think it's worth understanding in what
kind of circumstances it might be correct
to do that, and also how the different pieces
of the AI safety puzzle fit together.
I think that thinking about timelines is important
for AI.
It is very hard to have any high level of
confidence in when AI might have different
capabilities.
Predicting technology is hard, so it's appropriate
to have uncertainty.
In fact, here's a graph.
You can see the bunch of faint lines showing
individual estimates of people working in
machine learning research of when they expect
high level AI to be developed.
Then this bold red thing is the median of
those.
That's quite a lot of uncertainty.
If you take almost any individual's view,
and certainly this aggregate view, that represents
very significant uncertainty over when transformative
AI might occur.
So we should be thinking about that.
Really our uncertainty should follow some
kind of smooth distribution.
For this talk, I'm gonna talk about four different
scenarios.
I think that the advantage of anchoring the
possibilities as particular scenarios and
treating them as discrete rather than continuous
is that it becomes easier to communicate about,
and it becomes easier to visualize, and think,
"Okay, well what would you actually do if
the timeline has this type of length?"
The first scenario represents imminent AI,
maybe something on the scale of 0 to 10 years
away.
In this case, it's more likely that we actually
know or can make educated guesses already
about who the important actors will be around
the development of AI.
I want to explore a little bit about what
strategies we might pursue based on each of
these different timelines.
If you assume this first one, then there's
no time for long processes.
If your idea was, "Well, I'll do a degree
in CS, and then I'll go and get a PhD in machine
learning, and then I'll go into research,"
you're too late.
On the other hand, if you are already in a
position where you might be able to do something
in the short term, then it could be worth
paying attention to.
But I feel for a lot of people, even if you
think there is some small chance of this first
scenario happening (which in general you want
to pay attention to) it may be that there
isn't a meaningful way to engage.
The next possible scenario would be maybe
between 10 and 25 years out.
This is a timescale in which people can naturally
build careers.
They can go and they can learn things.
They can develop networks.
They can build institutions.
They can also build academic fields.
You can ask questions, get people motivated,
and get them interested in the framing of
the question that you think is important.
You can also have time for some synthesis
and development of relevant ideas.
I think that building networks where we persuade
other people who maybe aren't yet in a direct
position of influence, but might be later,
can be a good idea.
If we look a bit further to another possible
scenario, maybe between 25 to 60 years out,
that's a timescale at which people who are
in the important fields today may be retiring.
Paradigms in academic fields might have shifted
multiple times.
It becomes hard to take a zoomed in view of
what it is that we need, but this means that
it's more important and build things right
rather than quickly.
We want to build solid foundations for whatever
the important fields are.
When I say the important fields here, I'm
thinking significantly about technical fields
of how we build systems which do what we actually
want them to do.
I'm also thinking about the kind of governance,
policy, and processes in our society around
AI.
Who should develop AI?
How should that be structured?
Who is going to end up with control over the
things which are produced?
These scenrios are all cartoons.
I'm presenting a couple of stylized facts
about each kind of timeline.
There will be a bit of overlap of these strategies,
but just to give an idea of how actually the
ideal strategy changes.
Okay.
The very distant maybe more than 60 years
out, anything, maybe it's even hundreds of
years, at this level predictability gets extremely
low.
If it takes us this long to develop radically
transformative AI, it is quite likely that
something else radically transformative will
have happened to our society in the meanwhile.
We're less likely to predict what the relevant
problems will be.
Instead, it makes sense to think of a strategy
of building broad institutions, which are
going to equip the people of that time to
better deal with the challenges that they're
facing then.
I think actually it's plausible that the effective
altruism community, and the set of ideas around
that community, might be one broad, useful
institution for people of the far future.
If we can empower people with tools to work
out what is actually correct, and the motivation
and support to act on their results, then
I'd be like, "Yep, I think we can trust those
future people to do that."
The very long term is the timescale at which
other very transformative things occurring
in our society are more likely to happen.
This can happen on the shorter timescales
as well.
But if you think on a very long timescale,
there is much more reason to put more resources
toward other big potential transitions, rather
than just AI.
I think that AI could be a big deal, but it's
definitely not the only thing that could be
a big deal.
Okay.
I've just like talked us through different
timelines.
I think that most reasonable people I know
put at least some nontrivial probability on
each of these possible scenarios.
I've also just outlined how we probably want
to do different things for the different scenarios.
Given all of this, what should we actually
be doing?
One approach is to say, "Well, let's not take
these on the timelines.
Let's just do things that we think are kind
of generically good for all of the different
timelines."
I think that that's a bad strategy because
I think it may miss the best opportunities.
There may be some things which you only notice
are good if you're thinking of something more
concrete rather than just an abstract, "Oh,
there's gonna be AI at some point in the future."
Perhaps for the shorter timelines, that might
involve going and talking to people who might
be in a position to have any effect in the
short term, and working out, "Can I help you
with anything?"
Okay.
The next kind of obvious thing to consider
is, well, let's work out which of these scenarios
is the most likely.
But if you do this, I think you're missing
something very important, which is that we
might have different degrees of leverage over
these different scenarios.
The community might have different leverage
available for each scenario.
It can also vary by individual.
For the short timelines, probably leverage
is much more heterogeneous between different
people.
Some people might be in a position to have
influence, in that case it might be that they
have the highest leverage there.
By leverage, I mean roughly, conditional on
that scenario actually pertaining, how much
does you putting in a year of work, trying
your best, have an effect on the outcome?
Something like that.
Okay.
Maybe we should just be going for the highest
likelihood multiplied by leverage.
This of course is like the place where we
have the most expected impact.
I think there's something to that.
I think that if everybody properly does that
analysis for themselves and updates as people
go and take more actions in the world, then
in theory that should get you to the right
things.
But the leverage of different opportunities
varies both as people take more opportunities
and also even just for an individual.
I've known people who think that they've had
different opportunities they can take to help
short timelines and then a bunch of other
opportunities to help with long timelines.
This is a reason not to naively go for highest
likelihood multiplied by leverage.
Okay.
What else?
Well, can we think about what portfolio of
things we could do?
I was really happy about the theme of this
event because thinking about the portfolio
and acting under uncertainty is something
I've been researching for the past two or
three years.
On this approach, I think we want to collectively
discuss the probabilities of different scenarios,
the amount of leverage we might have for each,
and the diminishing returns that we have on
work aimed at each.
Then also we should discuss about what that
ideal portfolio should look like.
I say collectively because this is all information
where when we work things out for ourselves,
we can help inform others about it as well,
and we can probably do better using collective
epistemology than we can individually.
Then we can individually consider, "Okay,
how do I think in fact the community is deviating
from the ideal portfolio?
What can I do to correct that?"
Also, "What is my comparative advantage here?"
Okay.
I want to say a couple of words about comparative
advantage.
I think you know the basic idea.
Here's the cartoon I think of it in terms
of:
AI in EA 5
You've got Harry, Hermione, and Ron, they
have three tasks to do, and they've gotta
do one task each.
Hermione is best at everything, but you can't
just get Hermione to do all the things.
You have to allocate them one to one.
So it's a question of how do you line the
people up to the things so that you have everyone
doing something that they're pretty good at
it, and overall you get all of the important
things done?
I think that this is something that we can
think of at the level of individuals choosing,
"What am I going to work on?
Well, I've got this kind of skillset."
It's something that we can think of at the
level of groups as well.
We can ask, "What is my little local community
going to work on?" or "What is this organization
going to do, and how do we split up responsibility
between different organizations?"
Comparative advantage is also a concept you
can think of applied over time.
This is a little bit different because people's
actions in the past are fixed, so we can't
affect those.
But you can think there's things that might
want to be done and we can do some of these.
People in the past did some of them.
People in the future might do some of them
and there's a coordination question of what
we have a comparative advantage at relative
to people in the future.
This is why when I was looking at longer scenarios,
the next generation in the distant cases,
I was often thinking it was better to let
people in the future solve the concrete problems.
They're gonna be able to see more clearly
what is actually to be solved.
Meanwhile, we have a comparative advantage
at building the processes, the communities,
the institutions which compound over time,
and where getting in early is really helpful.
If you're taking something like this portfolio
approach, I think that most projects should
normally have at least a main scenario in
mind.
This forces you to be a little bit more concrete
and to check that the things you're thinking
of doing actually line up well with the things
which are needed in some possible world.
I also think you want to be a bit careful
about checking that you're not doing anything
which would be bad for other scenarios.
There's always an opportunity cost.
If you're doing something where you're thinking,
"I want to help with this short timeline scenario,"
then you're not doing something else you could've
done to help with the next generation in a
longer timeline scenario.
You could also have situations where maybe
I would think that if AI is imminent, the
right thing to do is to run around and say,
"Everybody panic.
AI is coming in five years.
It's definitely coming in five years."
If it definitely were coming in five years,
maybe that would be the right thing to do.
I actually don't think it is.
Even if it were, I think that would be a terrible
idea because if you did that, then people,
if it didn't occur in five years and we were
actually in a world where radically transformative
AI was coming in 25 years, then in 15 years,
a lot of people are gonna go, "We've heard
that before," and not want to pay attention
to it.
This is a reason to have an idea of paying
some attention to the whole idea of the portfolio
that as a community we want to be paying attention
to even if individually, most projects should
have a main scenario in mind.
Maybe as an individual, your whole body of
work has a main scenario in mind.
It's still worth having an awareness of where
other people are coming from, and what they're
working on, and what we're doing collectively
then.
I've mostly talked about timelines here.
I think that there are some other significant
uncertainties about AI.
For instance, how much is it that we should
be focusing on trying to reduce the chances
of catastrophic accidents from powerful AI?
Or how much of the risk is coming from people
abusing powerful technologies?
We hypothesized it was gonna be a radically
transformative technology with influence over
the future.
How much of that influence actually comes
through things which are fairly tightly linked
to the AI development process?
Or how much influence appears after AI is
developed?
If most of the influence comes from what people
want in the world after an AI is developed,
it might makes sense to try to affect people's
wants at that point.
In both of these cases, I think we might do
something similar to portfolio thinking.
We might say, "Well, we've put some weight
on each of these possibilities," and then
we think about our leverage again.
Maybe for some of them, we shouldn't be split.
Some of them we might do.
We can't do this with all of the uncertainties.
There are a lot of uncertainties about AI.
Here's a slide from another talk.
It just lists a lot of questions.
A lot of them about how AI might develop.
We can all have nuanced views about each of
these questions.
That's fine.
We need to do some picking and choosing here.
But I do think that we should strive for nuance.
I think the reason is that there's a lot of
uncertainty, and we could potentially have
extremely nuanced views about a lot of different
things.
The world is complicated, and we have a moderately
limited understanding of it.
One of the things which may make us better
equipped for the future is trying to reduce
our limits on our understanding.
What can individuals do?
I think consider personal comparative advantage.
You can ask yourself, "Could I seriously be
a professional researcher in this?"
Check with others as well.
I think people vary in their levels of self-confidence,
so I actually think that others' opinions
often can be more grounding than our own opinion
for this.
It's a pretty specialized skillset that I
think is useful for doing technical safety
research.
Most people in the community are not gonna
end up with that skillset and that's fine.
They should not be quitting their jobs, and
going to try and work on safety research.
They could be saying, "Well, I want to give
money to support this," or they could be aiming
at other parts of this portfolio.
They could say, "Well, I want to help develop
our institutions to build something where
we're gonna be better placed to deal with
some of the longer timeline scenarios."
You could also diversify around those original
assumptions that I made.
I think that each of them is pretty likely
to be true.
But I don't think we should assume that they
are all definitely true.
We can check whether in fact there are worlds
where they're not true that we want to be
putting some significant weight onto.
I think also just helping promote good community
epistemics is something that we can all play
a part in.
By this I mean pay attention to why we believe
things and communicate our real reasons to
people.
Sometimes you believe a thing because of a
reason like: "Well, I read this in a blog
post by Carl Shulman, and he's really smart."
He might provide some reasons in that blog
post, and I might be able to pallet the reasons
a little bit.
But if the reason I really believe it is I
read that, that's useful to communicate to
other people because then they know where
the truth is grounded in the statements I'm
making, and it may help them to be able to
better see things for themselves, and work
things out.
I also think we do want to often pay attention
to trying to see the underlying truth for
ourself.
Good community epistemics is one of these
institutions which I think are helpful for
the longer timelines, but I think they're
also helpful for our community over shorter
periods.
If we want to have a portfolio, we are going
to have to coordinate and exchange views on
what the important truths are.
What does AI mean for effective altruism?
My view is that it isn't the one thing that
everyone has to pay attention to, but it is
very plausibly a big part of this uncertain
world stretching out in front of us.
I think that we collectively should be paying
attention to that and working out what we
can do, so we can help increase the likelihood
of good outcomes for the long term future.
