Thank you.
As I said before in my talk a little earlier,
with respect to the EA community, from the
bottom of my heart, thank you for existing.
Of course, I get a little jaded since we work
together with CEA in the same office space,
but I do try to keep some... try to remember
how wonderful it is.
Okay, this talk is going to be a bit condensed
and crowded.
I think I may be pushing the limits of that
a little bit, I'm not sure.
I haven't given this talk before.
But I will plow forward at some brisk pace,
and this is a marvelous group of people and...
who, I trust, keep up wonderfully, yes.
Okay.
So Paretotopian Goal Alignment, a key... this
worked a moment ago.
There we go.
Key concept, Pareto-preferred futures, meaning
futures that would be strongly approximately
preferred by more or less everyone.
If one can have futures like that, that are
part of the agenda, that are being seriously
discussed, that people are planning for, perhaps
you get there.
Strong goal alignment can make a lot of outcomes
happen that would not work in the middle of
conflict.
So, goal alignment, when it matters.
Changing perceptions.
There's the concept of an Overton window,
the range of what can be discussed within
a given community at a given time, and what
can be discussed and taken seriously and regarded
as reasonable changes over time.
It also changes with the community.
We have... the Overton window for discourse
in the EA community is different from that
in... oh, I don't know, the public sphere
in Country X.
AI seems likely to play a pivotal role, and
I would like to have a couple slides here
that overlap with ones I used earlier, in
a talk earlier.
Pivotal role.
Today we can ask questions we couldn't ask
before when AI was a very abstract concept.
We can ask questions like, "Where do AI systems
come from?"
Because they're now being developed.
They come from research and development processes.
What do they do?
Broadly speaking, they provide services, they
perform tasks in bounded time with bounded
resources.
What will they be able to do?
Well, if you take AI seriously, you expect
AI systems to be able to automate more or
less any human task.
And more than that.
So now we ask, what is research and development?
Well, it's a bunch of tasks to automate.
Increasingly, we see AI research and development
being automated, using software and AI tools.
And where that leads is toward what one can
call recursive technology improvement.
There's a classic view of AI systems building
better AI systems.
That's been associated with agents.
What we see emerging is that happening in
association with a technology base, ongoing
inputs of human insights, but human insights
being leveraged more and more, being higher
and higher level, and perhaps being less and
less necessary.
So at some point, AI development with AI speed.
Where that leads is toward what I describe
here as comprehensive AI services.
Expanding the range of services, increasing
their level toward this asymptotic notion
of "comprehensive."
What does "comprehensive" mean?
Well, it includes the service of developing
new services and that's where generality comes
from.
So, I just note that the C in CAIS does the
work of G in AGI.
And so, I... watch out now, you see, if you
ask, "But in the CAIS model, can you do X?
Do you need AGI to do X?
Because... well, what about..."
I say, "What part of 'comprehensive' do you
not understand?"
Okay, so beware.
I won't be quite that rude but I will pretend
that I will .. I will now pretend that I will
be that rude and say that, yes, if you ask,
the answer is "well, it's comprehensive now.
What is it you want to do?
Let's talk about that."
Okay, so, for this talk, I think there is
some key considerations for forward-looking
EA strategy.
They're anchored in AI, in an important sense,
this set of considerations.
Some day, I think it's reasonable to expect
that AI will be visibly, to a bunch of of
relevant communities, poised to be on the
verge of explosive growth.
That it will be sliding into the Overton window
of powerful decision-makers.
Not today, but increasingly and very strongly
at some point downstream, when it's sort of
happening before their eyes and more and more
experts are saying, "Look what's happening."
As a consequence of that, we will be on the
verge of enormous expansion in productive
capacity.
It's one of the applications of AI, is fast,
highly effective automation.
Also, this is a harder story to tell, but
it follows.
Beyond the verge, potentially, if the right
groundwork has been laid, of having systems...
security systems, military systems, domestic
security systems, et cetera... that are benign
in a strong sense, as viewed by almost all
parties, and effective with respect to x-risk,
military conflict and so on.
So, fourth key consideration is that these
facts are outside the Overton window of policy
discourse.
One cannot have serious policy discussions
based on these assumptions.
And the first set of facts make possible an
approximately strongly Pareto-preferred world.
And fact four constrains strategies by which
we might actually move in that direction and
get there.
And that conflict is essential to the latter
part of the talk, but first I would like to
talk about resource competition, 'cause that's
often seen as kind of the "hard question."
Resources are bounded at any particular time,
people compete over them.
Isn't that a reason why things look like a
zero-sum game?
And that does not align goals, that makes
goals oppose each other.
So, here's a graph called "quantity of stuff
that party A has," vertically, "quantity of
stuff that B has," horizontally.
There's a constraint, there's one unit of
"stuff," and so the trade-off curve here is
a straight line, and changes are on that line,
and goals are opposed.
Zero-sum game.
In fact, resources increase over time, but
the notion of increasing by a moderate number
of percent per year is what people have in
mind, and the time horizon in which you have
a 50% increase is considered very large.
But even a 50% increase, shown here, is something
that... okay, the gray somewhat shows in this,
this display... so here we have "current holdings,"
this is B taking all the gains, equal share,
which is where I started arbitrarily here.
Here B is taking away from A, A is actually
worse off, and B is taking 90% of the total.
So those are some illustrative possible steps
during a 50% expansion in resources.
But, ordinarily when we're thinking about
utility, we don't regard utility as linear
in resources, but as something more like the
logarithm of resources.
Common mathematical model, we'll adopt that
for illustrative purposes here.
If we plot the same curves on a log scale,
the lines become curved... so there's the
same unit constraint.
And here's the 50% expansion.
Qualitatively, looks rather similar.
Well, the topological relationships are the
same, it's just re-plotting the same lines
on a log scale.
But on a log scale, we can now represent large
expansions, and have the geometry reflect
utility in a direct visual sense.
So here's the same diagram, current holdings
and 50% expansion.
Here's what a thousandfold expansion looks
like.
Taking all the gains and taking 90% of the
total have now switched position.
Someone could actually take a larger share
of resources and everyone is still way ahead.
What matters is that there be some reasonable
division of gains.
That's a different situation.
And key point here is that that is small.
The difference between taking everything and
having a reasonable share is not a very large
difference in utility.
Very different situation from the standard
zero-sum game over resources.
Again, the reason for this is that we're looking
forward to a decision time horizon that spans
this large change, which is historically not
what we have seen.
So, let's consider the case for when some
party tries to take everything, or tries to
take 90%, or like that.
How far do you get?
Well, greed brings risk.
This is going to create conflict that is not
created by attempting to do that.
So the argument here is that not only is there
a small increment of gain if you succeed,
but, allowing for risk, the gains from attempting
to grab everything are negative.
Risk-adjusted utility is bad.
Your optimum is in fact to aim for some outcome
that looks at least reasonably fair to all
of the other parties who are in this game,
in this process of mutually adjusting policies.
And, so, this region, labeled "Paretotopia"...
this is a region of outcomes... now, just
looking at resources, there are many other
considerations, in which all parties see very
large gains.
So, that's a different kind of future to aim
at.
It's a strongly goal-aligning future, if one
can make various other considerations work.
Problem is, of course, that this is not inside
the window of discussion that one can have
in the serious world today.
The... well, first, about what one can do
with resources plus strong AI.
It can eliminate poverty while preserving
relative wealth.
The billionaires remain on top, they build
starships.
Poor and rural Africa remain on the bottom.
They only have orbital spacecraft.
And I'm serious about that, if you have good
productive capability.
They expand total wealth while rolling back
environmental harms.
That's something one can work through, just
start looking at the engineering and... what
one can do with expanded productive capabilities.
More challenging are preserving relative status
positions while mitigating oppression.
Why do we object to others having a whole
lot of resources and security?
Because those tend to be used at our expense.
One can describe situations in which oppression
is mitigated in a stable way.
Structure transparency is a concept I will
not delve into here, but is related to being
able to have inherently defensive systems
that circumvent the security dilemma, "security
dilemma" being "I'm developing military systems
purely defensively."
Oh, but they have multiple uses.
They look dangerous to you, so you develop
military systems "defensively," but those
look dangerous to me, and so you have an arms
race.
So, so-called "security dilemma"... if one
were able to build truly effective, genuinely
defensive systems, it would provide an exit
door from that arms race process.
Again, these opportunities are outside the
Overton window of current policy discourse.
So, where are we today?
Well, technological perceptions... on the
left we have "credible technologies," on the
right, "realistic technologies," given what
engineering and physics tell us is possible.
And the problem is that these are... do not
overlap.
"Credible" and "realistic" are disjoint sets.
It's a little hard to plan for the future
and get people aligned toward the future in
that situation.
So, that's a problem.
How can one attempt to address it?
Well, first we note that at present we have
what are called "track one policies," "business-as-usual
policies"... what is realistic is not even
in the sphere of what is discussed.
Now, I would argue that we, in this community,
are in a position to discuss realistic possibilities
more.
We are, in fact, taking seriously advanced
AI.
People take seriously the concept of the "cosmic
endowment."
So, we're willing to look at this.
How do we try to bridge... how can we make
progress in bridging between the world of
what is credible, in "track one," and what's
realistic?
I think, by finding technologies that are
plausible, that are sort of within the Overton
window in the sense that discussing contingencies,
possible futures like that is considered reasonable.
The concepts are not exotic, they're simply
beyond what we're familiar with, maybe in
directions that people are starting to expect
because of AI.
And so if this plausible range of technologies
corresponds to realistic technologies, the
same kinds of opportunities, the same kinds
of risks, therefore the same kinds of policies,
and also corresponds to what is within the
sphere of discourse today... like expanding
automation, high production... well, that's
known to be a problem and an opportunity today.
And so on.
Then, perhaps, one can have a discussion that
amounts to what's called "track two," where
we have a community that is discussing exploring
potential goals and policies, with an eye
on what's realistic.
Explicit discussion of policies that apply
both... that are both in the "plausible" range
and the "realistic" range.
Having the plausible policies, the plausible
preconditions, be forward in discussion.
So, now you have some toehold in the world
of what the serious people are willing to
consider.
And increasingly move these kinds of policies,
which will tend to be aligned policies that
we're exploring, into the range of contingency
planning, for nations, for institutions, where
people will say, "Well, we're focusing of
course on the real world and what we expect,
but if this crazy stuff happens, who knows."
AI... people are thinking AI might be a big
deal, you folks are telling us that AI will
do... expand resources, will make possible
change in the security environment, and so
on.
Well, that's nice.
You think about that.
And if it happens, maybe we'll take your advice.
We'll see.
So, in this endeavor, one has to work on assumptions
and policies that are both plausible and would,
if implemented, be broadly attractive.
So, that's a bunch of intellectual work.
And the strategic context is one in which
we will go to in the next slide.
I'll spend a couple moments on this one first.
So, realistic: superintelligent-level AI services.
Credible: extensive applications of high-end
AI.
People are talking about that.
Physics-limited production.
Truly science fiction in quality.
Well, a lot of the same issues arise from
strong scalable automation, of the sort that
people are already worried about in the context
of jobs.
Solar system scale energy, 10^26 watts.
Well, how about having... breaking constraints
on terrestrial energy problems by having really
inexpensive solar energy?
It can expand power output, decrease environmental
footprint, and actually do direct carbon capture,
if you have that amount of energy.
Solar system scale resources, kind of off
the table, but people are beginning to talk
about asteroid mining.
Resource efficiency and... one can argue that
resources are not binding on economic growth
in the near term, and that's enough to break
out of some of the zero-sum mentality.
Absolute defensive stability is realistic
but not something that is credible, but moving
toward defensive stability is.
So... and note, it's okay to be here.
You don't necessarily, here in this room,
have to take seriously superintelligent-level
AI services, alert systems, scale resources
and so on, to be playing the game of working
within the range of what is plausible in the
more general community, and working through
questions that are of what would constitute
"Paretotopian goal-aligning policies" in that
framework.
So the argument is that eventually, reality
will give a hard shove.
Business-as-usual scenarios, at least the
assumptions behind them, will be discredited
and, if we've done our work properly, the
policies that are based on those assumptions.
The policies that lead to the idea that maybe
we should be fighting over resources in the
South China Sea just looks absurd because
everyone knows that in a future of great abundance,
that's the... you don't want to have a fight
over that, because fighting over something,
it's worthless.
So, if the right intellectual groundwork has
been done, then, when there's a hard shove
from reality that is toward a future that
has Paretotopian potential, there will be
a coherent policy picture that is coherent
across many different institutions, with everyone
knowing that everyone else knows that it would
be good to move in this direction.
Draft agreements worked out in track two diplomacy,
scenario planning that suggests it would be
really stupid to pursue business as usual
in arms races.
That kind of work in place, that... with a
hard shove from reality, perhaps, in fact,
we see a shift... track one policies are discredited,
the Paretotopian policies... people are asking,
"What should we do?
What do we do?
The world is changing."
Well, this.
Looks good.
Fight over it, you probably lose.
And if you fight over it, you don't get much
if you win, so why not go along with what
has been thought through in some depth and
looks attractive?
So that is the basic Paretotopian strategic
idea.
We look at these great advances, back off
to plausible assumptions that can be discussed
in that framework, work through interactions
with many, many different groups, reflecting
diverse concerns that, in many cases, will
seem opposed but can be reconciled given greater
resources and the ability to make agreements
that couldn't be made in the absence of, for
example, strong AI implementation abilities,
and end up in a different world.
Now, it says "robust," well, robust against
what?
All of the capabilities that are not within
the range of discussion or that are simply
surprising.
"Compatible."
Well, Paretotopian policies aren't about having
one pattern on the world, it really means
many different policies that are compatible
in the sense that the outcomes are stable
and attractive.
And with that, the task at hand, at least
in one of the many directions that the EA
community can push, and a set of considerations
that I think are useful background and context
for many other EA activities, is formulating
and pursuing Paretotopian meta-strategies,
the framework for thinking about strategies.
That means understanding realistic and credible
capabilities and bridging the two.
Bunch of work on both understanding what's
realistic and what is credible and relationships.
Understanding and accommodating diverse concerns.
One would like to have policies that seem
institutionally acceptable to the U.S. military,
and the Communist Party of China, and to billionaires,
and also make the rural poor well-off, and
so on, and have those be compatible goals.
And to get really understanding the concerns
in the language of these communities, their
conceptual language and actual literal idiom
is a key direction to pursue.
And that means deepening and expanding the
circle of discourse that I'm outlining.
And so, this is a lot of work, lot of hard
intellectual work and, downstream, increasing
organizational work.
I think that pretty much everything one might
want to pursue in the world that is good fits
broadly in this framework and can perhaps
be better oriented with some attention to
this meta-strategic framework for thinking
about goal alignment.
And so thank you.
