Alright.
So Jaan, thank you for being here.
Very excited to have you.
Thanks for inviting me.
A lot of stuff to cover.
We may also be able to take some questions
through the app or through the website, so
sf.eaglobal.org/polls, I won't promise that
I'll get to them, but we might have some time
to do that.
I think that I first met you 8 years ago.
At the time you have much more recently, certainly
relative to now, wrapped up your involvement
with Skype and you gave a talk at an old Singularity
Summit event and you said "I find myself in
the kind of unusual position of trying to
figure out what to do, knowing that I've just
wrapped up what is probably going to be the
most successful project that I'll ever do".
That is a unique challenge, but you've really
made the most of that opportunity and have
engaged in a lot of really interesting things
over the last 8 years that have spanned AI
and other existential risks, some work on
global coordination, some interest in a whole
bunch of different companies.
So, I want to give you the chance to talk
about all of that sort of stuff.
But let's start off with a little bit of a
retrospective on the last 8 years since you
first got involved.
How have things gone from your perspective?
How have they deviated from what you thought
might happen?
I'm not sure I remember how much I thought
what will happen, but yeah, things have gone
really well with one exception.
Things that have gone well is that the EA
movement has scaled.
Probably the most important thing is that
the so-called Overton Window - things that
are now acceptable to talk about - now firmly
includes at least some version of AI safety.
So you can actually publicly talk about without
being ostracized.
Then, much more people, much more talent,
much more resources, money available in existential
risk ecosystem in general, more organizations,
etc.
Things are really growing really, really nicely.
I think the main exception is that we still
have a lot of uncertainty how much time we
have.
It's great progress against an unknown deadline.
From the very beginning you were really worried
about the Yudkowsky-style scenario of fast
takeoff or things getting out of control quickly.
Have your views on that changed over the last
eight years?
I mean, a little bit.
I guess they have diversified or gotten more
uncertain over time as more total people have
entered the AI safety space.
There are basically more hypotheses, more
views about what might happen and what might
be important.
My general approach is to basically look at
when people are making total arguments and
see whether there's... where I can find a
bug in them.
If I can't find a bug, I just say, "Okay,
here's some probability mass for you."
Therefore, yes, people have made these plausible
arguments.
I have increased my uncertainty for a process.
For example, still I'm missing the days when
it was just one fairly crisp idea what might
go wrong.
Just recently, I funded a project in Berkeley
that is trying to reconciliate different ideas,
different hypotheses of how things might develop
so we could have a better, hopefully more
compact idea, of what's going on, what's going
to happen.
I think one way you have always stood out
is in your epistemic modesty.
I mean, here going back to 2010, you're this
very successful person, you've had this very
visible company that's made this impact on
the world and when I first encountered you,
you were willing to spend time with anyone
who had an interesting idea on AI and it did
not seem to matter to you what credentials
they had or didn't have, but then at the same
time you've used your own social capital to
take the best of that and try to take it to
other places where maybe those idea authors
wouldn't be welcome.
So tell us a little bit about your strategy
for leveraging your own social capital, but
being so humble at the same time.
Yeah so, one thing that I sometimes say that
there's really nice benefits that you get
from doing programming full-time and one of
them is that you get an intuitive sense of
what it means to have your thoughts fully
specified and that you can use it on your
thoughts, you can use it in other arguments,
like are they using just metaphors to conflate
things, or are they actually doing something
that potentially a machine could understand.
This is useful when I wanted to use it for
epistemics.
If I wanted to evaluate arguments by a bunch
of weird people in the Singularity Institute
that really helped.
One thing I think that programming gives you
is indeed, epistemic humility.
Because, like programs, you're so confident
that you know what's going to happen now,
and then you press f5, and nope.
Then, you look at it and it's like, "Yeah,
okay.
Clearly, I made this mistake."
You fix it.
Nope, that that still wasn't the case.
So, yeah.
It creates epistemic humility.
And when I entered a young ecosystem I immediately
can look like, "Okay, so what are things that
I could contribute?"
After trying to debug arguments and finding
that okay they seem to be solid.
And indeed, I found that I do have this side
effect of my programming career, I picked
up this brand, that I can then use by basically
touching the brand and my reputation to arguments
that I believe that were true, but people
needed some kind of brand to be attached to
those in order to actually take them seriously,
and that seemed to work pretty well.
For example, Hugh Price, my co-founder at
the Centre for the Study of Existential Risk in
Cambridge, he has this written this New York
Times piece.
It was My Road to Existential Risk, Taxis
and Copenhagen, or something like that where
he explicitly says that the reason why he
started taking it seriously, even though he
had heard those arguments many times, was
that somebody who really has a reputation
of being practical in software actually takes
those ideas seriously.
Were there some very early fundamental arguments
that persuaded you most and do those still
hold a lot of weight in your mind, today?
Oh, absolutely.
The main realization is the idea of recursive
self-improvement.
The idea, I think initially formulated by I.J.
Good in 1965, but I read it from what was
Overcoming Bias when Eliezer Yudkowsky was
writing there.
He was making this argument in a very clear
manner that I just bought into, and I still
think that it's one possible way how things
could go massively wrong in the world.
We just have an accident in an AI lab, where
we're suddenly pushing meta-learning or whatnot.
And suddenly, the meta-learning system starts
to meta-learn and you have a system that you
can no longer turn off.
So how has that seed view evolved?
I mean, you've talked about more different
ways that things can go wrong, so now, you
have a richer view of that.
Yeah.
So tell us a little bit about the more nuanced
view that you have today.
One competing theory is that the current deep
learning framework might actually take us
to the point where we have systems that are
roughly on human level or superhuman, but
they still cannot understand themselves, that
they are still fairly opaque to themselves.
Therefore there's no immediate danger of fast
takeoff in a lab, but you still have a situation
where there's global instability because suddenly
humans are no longer the smartest planners
on this planet, for example.
So yeah.
I think it's a more complicated, messy, less
clearly defined scenario, but I do think it's
plausible and so I'm assigning some probably
mass to that.
And you're involved now with a number of different
organizations as an adviser, in some cases
as a funder.
Give us a little sense of your portfolio of
activity across the AI space with the different
organizations and sort of strategies that
you're involved with.
Yeah so, my strategy over the last ten years
or so has been to cultivate diversity in the
ecosystem, in the sense of, if I basically
see that there are some smart, and what's
now called aligned, people and they want to
contribute to existential risk reduction,
what I've done is: "Okay, here's some seed
money."
And then, if they stay around and start growing
and becoming productive, then I have just
increased the amount of funding that I support
them with.
Yeah, over the 10 years or so, I've been contributing
to something between 10 and 20 different organizations
and projects in this space and now recently,
starting last year, I thought that I should
really start scaling up.
One half-joking way to say it is that I assign
very significant probability to human economy
not surviving the introduction of human-level
AI, meaning there is no point in accumulating...
maximizing the amount of money that you have
at the point when money becomes meaningless.
So I need to look at how much time do I have
to spend this.
I mean obviously again that it's still possible
that the human economy or something that resembles
the human economy will continue after that,
but I don't see why.
I might be wrong, which means that indeed
I have to then scale up my giving.
So I've been working with BERI - existence.org,
with Critch, who is sitting right there and
seeing how can we institutionalize me for
some value of making.
Critch put it yesterday in a way nicer, that
BERI was originally conceived to help various
x-risk institutions.
Then, I approached BERI with the thought,
"Okay.
Perhaps, you can help me."
Yesterday, Critch was like, "Oh yeah, Jaan
is like an institution, okay."
I'm qualified.
You've put a lot of energy, highly intentionally,
into moving the Overton Window, as we were
talking about it, around what kind of thoughts
or worries are okay to have.
Going back eight to ten years, it was just
too weird for anybody who was a respectable,
tenured computer science professor, even in
the AI space, to have worries about things
going wrong.
That seems to have shifted in a big way.
Where do you see the Overton Window being
today, and do you think that work is done
has it shifted, or is there more to do?
No.
It seems to me that Overton Window right now
caps out at technological unemployment, so
the weirdest thing that you can talk about
when it comes to AI, in a "respectable" setting
is a societal effect.
This is, of course, not a respectable setting.
No, it's not.
Like respectable setting, I mean people who
are mostly concerned about looking respectable
and yeah, thinking about what... optimizing
the opinion of others.
I would really like to push the Overton Window
higher by having the realization... the topic
of AI that might have effect beyond social
stuff entered the Overton Window.
So I've been pushing it in a way that I've
been saying that AI would be an environmental
risk.
Because first of all, I think there's an environmental
risk eventually.
The nice thing about environmental risks is
that these are things that unify humans.
We have pretty... we are very picky about
our environment.
Change it by a hundred degrees, which is tiny
compared to astronomical this way or that
way, and we go extinct in a matter of minutes.
An AI doesn't care about the environment.
Why should it preserve the parameters?
The nice thing about it is that indeed, if
there was a realization, "Oh, wait a minute.
We need to control AI in order to not lose
the environment", then this is a much more
tangible thing to have a global discussion
about, whereas if you just talk about social
issues, for example, in Asilomar Principles
that we did last year, there is this very
contentious principle about human rights,
which actually can automatically preclude
the Chinese to join the principles.
However, if AI was about long-term environmental
effects, there would be no problem of bringing
everybody on board.
That's fascinating.
So you're going around giving a lot of talks
on the subject, kind of trying to find the
right way to deliver the message so that it
can be heard and that you seem just the right
level of weird.
You've mentioned China, and you said you were
recently in China and give it a series of
talks there.
So what's the report from China?
Obviously, it's something...
AI is something that Chinese government, Chinese
society is really working on.
Yeah, I had a few updates.
I mean, one update was that, I mean, I grew
up in a totalitarian regime and basically,
I knew sort of what to expect, but I hadn't
been to China for a decade or two.
Mainland China, actually.
So one update, was that actually it seems like a much freer place than the place I remember
behind the Iron Curtain.
So that was nice, I guess.
Then another update was that the AI researchers,
at least the ones that I talked to, they were
very heads-down, optimizing classifiers and
robotics doing robotics, things like that.
I tell you, they are feeling, like, proud
to say that "we are very practical, here".
Now, on the other hand there was a really
interesting appetite about talking long-term,
philosophical manifestations of AI, which
was surprising even to the level that I don't
see much in the West, or people are more careful
about saying in West.
So that was interesting, yeah.
Half-jokingly, like the world that I cannot...
In my talks, I pushed, nudged a little.
Because I don't know if that's a good idea
or not, but I think it might be an interesting
idea to have China as the grown-up in the
room, because they really take pride in saying
that they're an old civilization who thinks
long-term as opposed to those silly democratic
countries who just think four years at a time.
There are pros and cons, obviously, to that,
but let's just exploit those pros and have
them think five or six years into the future
and perhaps be a couple of years ahead.
I think, maybe, the best piece that I've seen
in the American mainstream media about A.I.
and the future of society was from a Chinese
legal scholar who said, "With a powerful AI,
you might actually be able to have a planned
economy that is effective, and you don't necessarily
have to rely on market mechanisms, and you
don't necessarily have to fall into a scenario
where a few people sort of own all the data
in society, and we can actually collectivize
that and all share the benefits of it."
Were these things you heard there?
I was aware of that.
I did like a lot of preparation work when
I went to China, so I heard those things,
but I heard during the preparation work.
But it's funny, actually.
I've seen communism fail and the reason why
it failed was because people don't like to
work.
If you have AI doing all the work you don't
have that problem for communism at least.
You might have other problems, but, at least,
this particular, very crucial problem, you
eliminate.
In terms of your... we've covered a lot of
your personal and reputational efforts to
move the window and get people thinking in
new ways, and you've also been supporting
a lot of organizations.
What kinds of organizations are you supporting?
Do you think about investing in for-profits,
nonprofits, a mix, do you care?
What kind of organizations are you looking
to support?
Almost entirely nonprofits.
I think that well, a basic computer science
realization, when you have two optimization
targets, then you are not going to be great
on either one.
So if you want to optimize for the good of
the world and for profits, then you have to
trade off one of those.
I think the maximum effectiveness would be
from, yeah, effective altruist organizations
or nonprofits that don't have these constraints
in profit.
So interestingly, for example, the same applies
to...
Okay, think about if there's a startup, whose
business plan is to do a bunch of fundamental
physics and then to use those fundamental
results from fundamental physics research
to gain a commercial advantage it would just
sound silly.
However, that is a very typical pitch from
AGI companies, that they're going to do this
fundamental AI research and then use those
results to get a commercial advantage, which
almost never works.
When you start doing that, you immediately
get this tension, and so I think the most
successful AI companies capabilities wise
therefore have been either in academia, or
in teams in academia, or just they're just
nicely cordoned off sections of big companies
that have lots of cash.
So, yeah.
How about what's going on in your own home
country of Estonia?
I mean, I'm hearing more about great companies
being built there.
Obviously, the Estonian government is known
for being a technology leader.
Do you think that Estonia has a role to play
in this kind of AI future, given its unique
position as a digital society?
Possibly, but I think it's... if, then mostly
about the near-term issues about how do you
integrate increasingly sophisticated technology,
but I think there's a fairly stark phase change
when you go from subhuman systems that are
just really smart in certain domains to super
human systems that might actually do their
own technology development.
It's not clear that the work that has been
spent on the shorter term problems immediately
scales.
Some of that might scale to the longer term,
but it's not obvious.
Changing gears a little bit, you've also spoken
quite a bit about global coordination as a
challenge of interest to you.
Just give us a little overview of your interest
in that topic and how you think about-
Well that will be like-
In two minutes.
Yeah.
That'll be like two hours.
Maybe even just pointers to other places where
people can go find more would be good as well.
Yeah, I mean, I've been thinking about it
a little bit over many years now.
So it's very diluted thinking about things
like upcoming technologies that might make
the global coordination easier.
Examples there are, the constant, we're going
to get a lot of data about what's happening
on the planet and people already are using
that to weed out or find bad players.
For example, in deforestation, it's kind of
apparent that it's harder to deforest the
planet now than it was before the ubiquitous
satellites.
And blockchain is a particular interest of
mine.
Turns out, there's a few things that you can
do.
One way we are putting it is that for last
seven, eight years, we have had a regime on
the planet where it's possible to globally
agree about a piece of data without trusting
any central authority to maintain that data.
Can we use that interesting property that
the world has, to come up with coordination
regimes where in a situation where you have
participants that don't necessarily trust
each other?
Google it for more.
There's a lot more out there.
A few questions coming in from the audience
that are, I think, very interesting.
How much, and that you're uniquely positioned
to answer, how much do leaders of huge tech
companies actually talk to each other?
I'm not actually sure.
Fair.
Yeah, I don't know.
I always appreciate your willingness to say,
"I don't know."
What would you advise people to do who are
interested in either earning to give to philanthropically
support AI safety, or possibly trying to get
more involved directly if they can?
But let's just start with the money.
Where is the best value per dollar right now
in AI?
I'm not sure if Critch appreciates me saying
that, but I think it's that, the current situation
we have in BERI, existence.org is that, I
easily can't give more to them because that
would be basically put me on... the fraction
that I'm giving to them on a too big proportion.
If I could have other people join me, then
that actually might be helpful.
So that's one thing.
But yeah, I think just going to 80,000 hours
and following there.
I mean, they have done a lot of thinking about
like how people can be useful, so I can think
anything that I can say is going to be strictly
inferior to what they have been saying.
How afraid do you find famous people to be
about talking about issues that might be perceived
as weird, even if they are important?
Really kind of depends on people and how they
have... what type of reputation they have
to protect.
If they're politicians, obvious really careful
about saying things.
So actually, my co-founder at the Center for
Study of Existential Risk, Lord Martin Rees,
he's a politician and it's fascinating to
see how he really cleverly balances the "craziness"
of the message that he's giving with an optimized
perception.
So that is interesting.
Not sure if this one is right up your alley
or not, but what do you think is the main
bottleneck to increasing cryonics adoption?
That was the first question.
Main bottleneck.
I might be wrong, but I think the main bottleneck
is still that it's not sure if it works.
I do have an investment in Nectome, which
is competing with cryonics.
They're doing destructive brain preservation
and they say like, "Look."
I mean, they have their bias to say that,
but they said, "Look.
If you just look at the scanning what they
have, post freezing brains, doesn't seem that
there's much information there.
But, I mean, I don't have a lot of information,
but indeed if there would be much more higher
confidence that this thing works, then hopefully
it would become more of a popular thing to
do, but obviously there are many weird social
blocks that also need to be overcome.
When you say destructive brain preservation
is that like slicing and scanning?
Yeah, basically what they do.
First, they pump you full of cyanide, and
other highs, and then like heavy metals to
make a brain high contrast.
Then, they freeze you, so there's no hope-
You're not coming back.
You're not coming back.
Not in the same form.
Nothing that I...
Yeah, because they preserve the information,
not your brain.
We are very much running out of time, but
maybe one last question.
Beyond AI, which is obviously your number
one focus, are there other x-risks that you
see rising to near that AI level in terms
of urgency?
What are they and-
I mean, synthetic bio has always been in a
very close race with AI, because it seems
much easier to use for destructive purposes.
Much cheaper.
On the other hand, it has this nice property
that we can use intelligence to control it,
whereas it's not clear if we can use that
in the AI case.
Jaan Tallinn, a philanthropist for our modern
times.
Thank you very much.
Thank you.
Round of applause.
Thank you very much.
