I
Can you hear me?
can hear you Randal.
I think I've got my
volume down really low.
Randal,
I can hear you just fine.
Oh wait, I see what's wrong here. Okay.
I can hear you too.
It's
quiet because I've got the
wrong headphones plugged in.
No problem.
We have about 84 subscribers on YouTube
and we are live in the way that any of
them would know, and could. So,
we're just pre-announcement live.
Have you... So, the YouTube links...
It should be on the live-stream active
page, right? Or, the active live-stream.
I haven't updated the links yet.
That's where you put it,
and then all you have to do is switch
the redirect when you're ready to go?
Yup.
Getting that going now.
I'll put the music back
on for a little bit.
Oh yeah,
did you still want to put a
link on the front page as well,
or is that already there?
It says call.carboncopies.org.
Do you see it?
Right
on the front page of Carboncopies?
Oh, sorry, Yeah. I'll
put it on the homepage.
I thought you meant on the first slide.
No, no. Yeah.
Oh,
I'll put that on there as well.
Awesome.
Okay.
Grabbing 'er now.
And how's the
Uberconference thing starting up.
Is that there?
It's working really well
and it started great.
Perfect.
Hi Mallory.
Mallory, if you can hear me, can you say
something? So, I know your stuff works.
Not hearing you yet.
Still not hearing you.
Mallory, we can hear Randal, but not you.
Nothing yet.
If you haven't done it already,
you can click the settings thing in
hangouts and tell it specifically which
audio to use;
which microphone to use.
True.
Not hearing you yet.
We did just test this.
Yeah. That's why it's good to get
this started, like 20 minutes before.
I see Sarah is online.
Mhm,
that's great.
Yeah, we're still not hearing you
Mallory. If need be, when the time comes,
I can start it up and then hand it over
to you as soon as you figure out your
microphone.
Yes?
Allen?
When it's 11,
either you or I should say something that
we're still figuring out a last minute
technical glitch or something like
that, rather than, waiting around.
That sounds fine.
I can do that.
I'm just setting the redirects
right now to go to the right spot.
I'm going to use this moment to test
what it looks like if I'm sharing my
slides,
because I haven't really tested that yet.
Mallory, nothing... not coming through.
You might try turning the computer
off and on again? I don't know...
Just making ideas here.
Can you stop sharing slides for a second,
Allen,
so I can share mine for just a moment?
Yes, actually, you just
have share your screen.
So, I wanted to pick a specific window
rather than... Okay, this window,
share, and now what does it look like?
Okay, it looks like that. What if I do...
You'd want to do full screen.
Yeah... I'm going to try to
see what happens if I say
present. Okay, there you go.
I'm seeing it looking like
an Android-size window.
Oh you are? Okay. I might have
to change the size of the window.
Here we go.
Let's see what happens now.
Is it still looking like
an Android-size window?
Nope. Now it looks full screen.
I think you can present,
and then in the bottom of present,
you click...
It should hover over a little thing and
you can click the button that makes it a
browser-size window. Oh, it looks like
it's not really showing up for you.
This thing here?
Mallory, I saw you just jump back
on, and we can't hear you, still.
Sorry, Allen does this look
about right? Or is this still...
That looks the right size now,
but I do see the top of your browser.
So, you see the top of the
browser. Oh, okay. Let's see.
Oh also, hold your... well
no, that looks alright.
How does one do that?
How about if it's like this?
Does that work?
Or is it the wrong size again?
That's the android size.
Ah, rats... Okay, so then
maybe the best for me,
given the weird way my window's working,
I'll just keep it...
It looks like we have about
six live viewers. So, I just
wanted to, real quick...
Hi, my name's Allen. We're, still
testing out a few things. So,
anyone who's viewing this,
we're about to get ready with our
Spring Workshop for the Carboncopies
Foundation, and this has all been
made possible by volunteers. So,
we're all really grateful to anyone
who's contributed to making this workshop
possible. This workshop is going to
feature a few different speakers,
including Ben Goertzel,
Anders Sandberg,
and Jaan Tallinn.
So,
we're going to get started
here in just a few moments.
There may be just some
background music for a while,
and we're just trying to get one
more of our speakers on if we can.
Ok,
you can put your slides back up.
Can you guys hear me now?
Yeah,
there you are.
Okay, great! Restarting the
computer actually worked.
And Allen... Yeah, okay.
There we go. Perfect.
So it looks like we're about ready to go.
Yeah, how many viewers do we
have and when should we start?
10,
and whenever you want.
All right, I guess we'll go
ahead and start. Like Allen said,
welcome to the Carboncopies Foundation
2019 Spring Workshop. I'm Mallory Tackett,
the president of Carboncopies,
and I'll be moderating this event today.
Our topic is whole brain
emulation and AI safety.
The agenda for today... We'll be starting
with our chairman, Randall Koene,
who will deliver a few
remarks about the topic.
At 11:30 AM (Pacific Time) we will show
an interview with Skype co-founder and
AI safety supporter Jaan Tallinn
that was prepared for this event.
Following that interview,
I will introduce our panel and start
the discussion of any issues that were
raised. This is the first opportunity for
any audience members to ask questions.
At 1:30 PM (Pacific Time),
we will have the interview,
or we will show the interview,
with existential risk expert and
computational neuroscientist Dr. Anders
Sandberg.
Dr. Sandberg will then join us from
Oxford to answer audience questions. Then,
at 3:00 PM (Pacific time),
leading AI and AGI researcher Doctor
Ben Goertzel will present his remarks.
This will be followed by our
concluding panel discussion.
There are two ways for audience
members to participate.
You can write questions directly
in the live stream chat.
This live stream chat is monitored by
a volunteer who will alert me to any
questions that I can then ask the panel.
You can also call in and
join the discussion at
call.carboncopies.org or call
the phone number (415) 455-3017.
This is also moderated...
Mallory,
hang on for a second.
Jesse is just telling us that
while he can hear you on Hangouts,
he can not hear you on the live stream.
Oh.
I don't think that's
necessarily your problem,
because you're coming
through on the Hangouts. So,
Allen should be making sure that
you're audible on the live stream.
I believe I'm on.
I also believe she's
audible on the live stream.
Yeah, in Google Hangouts, I
believe I'm audible. So...
Can anyone just verify that they hear...
Anyone in the live stream that wants
to verify that they can hear me?
We have 13 viewers testing if
the Carboncopies are seeing this.
We have a guy named Luke...
Help us out here.
Hi Luke.
This is Allen from...
Luke,
from the live stream, has
confirmed, and so has Claire.
Multiple people are confirming that
they can all hear us. So, it's just...
Ok great.
Jesse, I don't know what's
up with your situation there,
but others can hear it. Okay. Sorry,
Mallory, for that interruption.
It looks like we actually
are live, then. Perfect.
All right. I'll just repeat how
audience members can join in,
since we might've had some
new people that joined.
You can ask your questions just
in the live stream chat directly,
or you can call in at
call.carboncopies.org or
call the phone number:
(415)-455-3017.
This is also moderated by a volunteer,
and it's going to be very
similar to calling into a radio.
When your question is chosen,
you will be able to ask it to the panel
directly and speak with the respondent.
This second option is still new,
so please be patient with any
technical difficulties that we have.
When the workshop is done,
we would appreciate audience
members to complete our survey at
survey.carboncopies.org. Our thanks
to our donors, our volunteers,
and all the experts that participated
in this workshop. With that,
it's time for introductory
remarks by Dr. Randal Koene.
You may know Dr. Koene as our
chairman and founder of Carboncopies.
He's also a neuroscientist with a
background in physics and artificial
intelligence.
He has spent the last 20 years working
in neural engineering labs and companies.
While doing that,
he has also been studying and bringing
awareness to the technical challenges for
whole brain emulation and mind uploading.
If you'd like to know more about
his background, his writing, talks,
or anything else, you can find that
information at RandalKoene.com. Welcome,
Dr. Koene.
Hi,
and thanks for the introduction.
I'm going to take one second
here just to pop up a few slides.
I'm making sure that works out correctly.
So,
I'm going to pick that window
and hopefully I'll do it right.
We're sharing the window.
I'm going to make it present.
Does this look correct, does this look
like Android to anyone? Does it look...
Android.
Android, thank you. Okay,
so I'll do it the other way around then.
That looks better.
I just need to...
Yeah, Okay. Well, it's better,
I don't know if I can get it
to be much better than that.
I suppose this is slightly better.
The problem is that my monitor is actually
flipped the other way around and it
doesn't seem that Hangouts
can deal with that. All right.
However that may be,
let's get into it. So,
thanks everyone for joining, and thank
you for that introduction, Mallory.
As you probably all know... Let me just
get back to the first slide here too,
so that's all making sense.
At the Carboncopies Foundation,
we primarily focus on the technical
challenges to whole brain emulation.
Occasionally,
we also explain why we think that mind
uploading through whole brain emulation
is an important part of the
long-term future for humanity;
and we could get into that,
but I'm going to try not to do that
until maybe in the panel somewhere. We
haven't previously dedicated
events to artificial intelligence,
even though artificial brains are
of course a special category of AI.
There are already so many who dedicate
their time and present the issues around
AI. Some people, for instance,
let's say companies like Deep Mind,
or Open AI,
laboratories like MIT's artificial
intelligence laboratory and specialized
organizations that have been
around for quite a while,
such as The Machine Intelligence
Research Institute, MIRI, Open Cog,
which is headed by Ben Gertz Hill and
The Future of Life Institute and many,
many others.
Of course,
everyone at Carboncopies is also very
interested in artificial intelligence and
an artificial general intelligence.
And just to be a bit specific,
by artificial general intelligence,
we usually mean a kind of
artificial intelligence that
isn't focused on a narrow
problem,
but that's focused on being able
to handle any kind of problem;
coming up with solutions for any variety
of problems, sort of like people do.
Now, personally, of course,
I've traveled in circles where AI and
AI risk or primary interests for quite
awhile.
As I just mentioned,
MIRI is out there in Berkeley
and in the whole bay area,
there are a lot of people who care about
this problem. So, many of my friends are,
are dedicating their time to it. And
then, it's clear that, of course,
there are areas where these domains,
whole brain emulation and AI,
have to interact.
If they're both part of our future,
they interact in some way.
And it's important to consider what
the outcomes are going to be. So,
when we talk about safety concerns in AI,
typically what we're talking about is,
what happens if artificial intelligence
gets to a point where the programs can
write themselves,
where every artificial intelligence
algorithm can come up with the next better
algorithm; or can somehow improve
itself along a utility function.
And then, there's the idea that
if this happens fast enough,
it could happen so quickly that we
don't know what's going on and we can't
control that in any sense,
this sort of "foom," this big
explosion of artificial intelligence,
the singularity. Now, whole brain
emulation is an artificial
brain, in a sense. So,
in what sense is that a
type of AGI we might ask?
It's certainly a kind of intelligence
and if it's in an artificial brain,
you might say it's an
artificial intelligence.
Even if the sort of mind that it's
creating isn't that artificial,
it's something based
directly on human brains.
How general is it? Well, humanity,
of course,
was evolved to fit into the niche,
the evolutionary niche,
that happens to be present right
now and 2 million years ago;
but still,
were pretty general in the sense that
we keep tackling new problems using this
same brain that we've got and
than the tools that we build.
So, we're a part of this big ecosystem
of intelligence, as we might say.
And you could even just look at
that whole ecosystem and wonder,
where's that going to go? How do all
the pieces interact? What can we expect?
And when the ecosystem moves
in a certain direction,
what happens to those intelligence?
The ones that are the human intelligence,
as originally, what happens to
them? Where do we fit in? So,
we have to wonder, as all these
bits and pieces are interacting,
does that increase or decrease
what we're calling the risk,
possible risk,
from AI?
Now,
those interactions and outcomes haven't
received a lot of attention so far.
There are just a few examples
of academic writing on it,
and what we're going to try to do is
we're going to try to focus on it a bit
more now. In fact, that's what
we do with all of our workshops.
We keep on trying to
highlight different aspects,
different pieces of a puzzle that is...
Oh,
it says that my slides are
not full screen. Sorry. Yeah,
I know they're not completely full screen.
When I do that,
then we get the Android version, because
my screen is flipped vertically. So,
I'm afraid this is probably the best
that we can do right now. Maybe,
in future I'll run it on a separate
laptop, or something, when I do this.
Anyway,
what we're trying to do,
is we're trying to highlight
different parts of this puzzle,
and there are a lot of parts when we
talk about whole brain emulation and the
whole ecosystem it's in. If you look at
what we've done so far in our workshops,
right? Since one piece that we filled in
is when we did the Transcending Biology:
Reverse Engineering the Brain Workshop,
we were really looking at the roadmap to
whole brain emulation and an update on
the status of all the technical bits,
what was possible now,
and what may be possible soon, and which
technologies are going there. Then,
we did a workshop called From Brain
Preservation to Reconstruction,
and in that workshop we were looking,
specifically,
at where are things going in terms
of being able to preserve a brain,
then be able to slice our section it
in some way, image it, and from there,
get to something that is a functional
model and what sort of problems are you
going to run into?
We were trying to highlight those
problems in a bit more detail.
Then we've done a workshop on the
metaphysics and ethics of whole brain
emulation which was very different
from what we typically do,
because we've been focusing
on the technology so much.
And now we're trying to address AI
safety and whole brain emulation,
and I think there are going to be a
lot more pieces of this puzzle as we go
along. Now, the topic that,
that we're looking at
today, I hope it's, also,
going to become an official part of our
road mapping effort and we're going to
advocate that it's going to be
included in what I'm calling, for now,
the version 2.0 workshop,
or conference about whole brain emulation,
at Oxford University,
which I hope is going to happen this year.
There's no definitive plan yet,
but a lot of indicators are
that this may happen this year.
Speaking of that, when that
workshop does happen, to update,
let's say,
the white paper that came out in 2007,
I hope that we bring in a whole set
of new people and different angles,
because a lot of the things
that we would talk about now,
they just weren't around back then.
For example,
now we have Anthony Zaidor who appeared
at our transcending by biology workshop,
who developed the molecular bar coding
approach to mapping a connectome,
the Boyden team, including
Adam Marblestone. By the way,
Adam Marblestone is now at Deep Mind,,
developed expansion microscopy and
wrote a set of seminal papers about
fundamental neuro-technology. None of
those were present at the first workshop.
We've had professor Scheffer come in
who tried to explain how his team has
started to attempt to
reconstruct functional,
drosophila circuits from structure
scans as presented in our From Brain
Preservation workshop.
And then of course,
the Burger and Song labs that have been
working full tilt on cognitive neural
prosthesis,
which is really the closest example
we have of partial brain emulation.
Those were not on the radar back in 2007
and that's really just the tip of the
iceberg.
The white paper that came out of the
first workshop focused on how much compute
power a whole brain emulation would
need and how a preserve brain could be
sliced and scanned the big problems of
wholesale functional data acquisition and
of how to convert brain data
and to working brain models,
those were hardly addressed,
nor were subdomain topics like AI safety,
models of consciousness,
or societal and ethical issues.
Okay. So, at this point,
much of what's been said about AI Safety
and whole brain emulation sounds more
like hand waving speculation,
then careful study.
There's been a mention in
Bostrom's book Super Intelligence.
Some introductory studies
that Carl Schulman has done,
not too much of that
has been published yet.
Informal communications that
we've had with people from
the FHI, the FLI, or MIRI.
And there was one paper that came out in
2017 that was interesting by Daniel Eth
on AGI and neural probes,
which connects fairly closely to
AGI and whole brain emulation.
And then,
of course,
a few Sci-fi situations like the version
that was depicted in transcendence
where,
you could say,
they included both whole brain emulation
and the danger of runaway AI in one
movie. Perhaps that was a
bit much to squeeze in there,
but it was an attempt to do that,
right?
Now,
the people we've invited to this workshop
have expertise that comes from several
directions. Jann Talliin has a background
in AI safety and existential risk,
he's been busy in that for quite awhile.
Dr Ben Goertzel has of course been working
for years on the development of AGI
and he's had a hand in many other sort
of tangential parts of that as well.
Dr. Anders Sandberg has done a lot of
work in existential risk and he was also,
of course,
involved with whole brain emulation,
specifically with that first workshop in
Oxford where we tried to pull together
some of the people at the time who had a
lot of interest in the topic. And then,
of course, those of us who are on the
panel from the Carboncopies Foundation,
we've all been involved in some sense,
or another,
in whole brain emulation development,
even if it's mostly from
the technical side. Now,
some people that we wish were here aren't
here this time, like Carl Schulman,
Nick Bostrom, people from open
AI or deep mind, and more. But,
the first workshop can only be so big,
and there's going to be plenty to digest
in this first iteration of the topic.
So, see now, when we talk about...
Oh,
I want to get to the next slide here.
When we talk about the kinds
of interactions that can
happen between artificial
intelligence or artificial intelligence
safety and whole brain emulation,
one of the problems is that you get into
interactions between different domains
and different technological developments,
and those are always really
complicated and hard to predict.
If you look at examples of
attempting to predict these things,
then often people have to choose something
very constrained. Take, for instance,
our friend Robin Hanson's book,
The age of M,
where he tries to predict from an
economic perspective as an economist,
what would happen if you had a wholesale
whole brain emulation available and
basically infinite compute power.
But,
he leaves out some things.
For instance,
he does not include artificial
intelligence, that is not
whole brain emulation,
directly in that.
So,
a lot of the things that in his economic
model are done by copying brains.
They could be done by using a different
type of AI, something more narrow,
perhaps. So, then that changes a lot
about this society he's depicting.
And you can imagine that the society
that he's depicting there could easily
transform itself quite quickly into
something very different, perhaps,
to something that does
include a lot of AI. So,
the problem here is that
as you explore angles,
different angles of the
question and questions,
you keep discovering that you have to
clarify those questions more and dig
deeper and uncover more of them.
And you're going to notice this in the
interviews that I did with Jaan Tallinn
and Anders Sandberg. Every insight
uncovers many more deep questions,
but you've got to start somewhere.
So,
I've decided to seed the conversation
by asking everyone a series of questions
that are based on statements, assumptions,
or intuitions that I've run into.
I'm going to try to take you through these
very briefly before we move on to the
next section of this workshop.
Just so that we're all on the same page,
in taking you through this, I just
want to mention that, typically,
when I talk to people who are coming
from an AI safety background about whole
brain emulation,
and whether they think
that this is something that
should receive extra funding,
or should be pushed hard that
there should be work done on it,
I get two different kinds of responses.
Some people will see work on whole
brain emulation as being something that
decreases the risk of runaway AI,
and there are a few of these examples
that I'll mention in just a moment.
And others,
they seem to come at it with an intuition
that whole brain emulation research is
more likely to increase the
danger of runaway AI. And again,
there are some examples of the sort
of thinking that goes into that,
but as I mentioned already, most of that
seems fairly hand wavy, at this moment,
and it really does deserve
more precise attention. So,
let me just get started
on the first one here.
Oh,
I think I may have skipped one.
The first one is, well, BCI for AI safety,
brain computer interfaces.
In recent years there've been a few,
especially some very well
known entrepreneurs who've
started companies like
Neuralink,
who've claimed that work on brain
computer interfaces on high bandwidth
interfaces between the brain and a machine
are a route to improve AI Safety by
causing something like a symbiosis
between humanity and AI. But,
you have to dig in a bit deeper to try
to understand whether that's really true
or in what way that would happen,
and that will be discussed more later on;
but here I'm just going to mention a
few of the big questions there. So,
for instance,
what is a high-bandwidth interface?
What does it mean when we
say high bandwidth BCI What
are we comparing that with?
Are we comparing it with the rate
that we can type or speak, imagery?
Does it mean that you target a certain
percentage of all of the neurons at the
same time and stimulate them?
What does it really mean?
And if you can make a connection like
that between a biological brain and a
machine, what's the predicted
effect of a connection like that?
And what's the effect on the human?
How does the human experience change?
How much does the human become
a part of that AI ecosystem?
And what's the effect on the AI?
How does connecting with this human brain
implement something you might call a
level of control or a level
of supervision, in some sense,
or anything like that?
When people talk about neural interfaces,
sometimes in the popular press,
they're conflated with neuro-prosthesis,
but they're not. They're not
a neuro-prosthesis. There,
just the connection between
a brain and a machine. Now,
the human brain has about 83 billion
neurons and each of those neurons
communicates at a typical maximum rate
of about a hundred spikes per second.
Some of them can go up to a
thousand spikes per second,
but not much more than that,
which is a much bigger interval than,
say,
the nanoseconds and microseconds
that computers work at.
Also,
human thought is based on processes like
pattern matching or giving an emphasis
to certain patterns by up or down
regulating the neurons in that area or on
regularity, which is heavy in learning,
things that fire together, wire together.
And machine thought is typically
based on the execution of programs,
on arithmetic, on logic, and on
sometimes artificial neural networks.
I just want to mention that I'm
not currently looking at my email.
I see that there are
some emails coming in.
If anyone's trying to actually reach me,
I can't see it;
a message on my phone,
if there's something wrong with
me coming through or whatever,
please ping me there.
So,
if you create a high bandwidth
communication pathway
between the brain of a
biological human and an advanced AI,
how does that affect AI Safety?
That's a very legitimate question
that's not that easy to answer. Now,
the other argument that is often made,
is to urge caution in the development of
whole brain emulation, because, perhaps,
whole branding emulation, itself,
could be a risk to humanity.
Maybe because, itself,
could become a runaway AI.
But the scenario there isn't often
clarified very well. How, for instance,
would a human brain be able to follow
an engineered or predictable self
improvement curve?
How would this compare with the notion
of self improving AI that follows a
utility function and uses reinforcement
learning as its primary method of
improvement?
If a whole brain emulation is
accomplished and implemented on a suitable
processing platform,
will a whole brain emulation be able to
rapidly self improve in a manner that's
akin to this supposed
AI take off scenario?
Is there a way for whole brain
emulation to do that rapidly?
Is there something like a whole
brain emulation "foom?" That too,
deserves more detailed thought then
whatever our intuitions may be saying.
Now,
toggling back to looking at this as
a potential risk in a different way,
concern about the acceleration
of risky AI improvement,
or the curve that AI improvement may take;
that's another possible reason for being
cautious about the development of whole
brain emulation. But again, the way that
it could happen isn't entirely clear.
The way this is presented is usually that
the development of whole brain emulation
may lead to insights that will then
make the development of AI happen faster;
just like the basic insight of how neurons
work that led to neural networks and
deep learning, and this is still
something that's used in AI. But you know,
how much of that is really happening
today? How much insight are we getting?
And also,
how does this really differ from the
insights that are gained from general
neuroscience research
that's happening today?
Is there something specific about doing
research on whole brain emulation that
would accelerate AI development in a
way that general neuroscience research
doesn't. We need to at least
have some examples of that.
Now finally,
the last question that I wanted
to present at this point is, well,
sometimes people will say the best way
to achieve AI safety is if we ourselves
can develop at the same rate as AI
if we're completely merged with it,
If there's no distinction
between human and machine,
if we somehow bring those two together,
thereby getting rid of a competitive
race between humanity and artificial
intelligence. Now again, that
needs to be clarified a bit more,
because we don't really know what it means
to say we're linking an uploaded mind
with an AI. As we just mentioned,
they work in slightly different ways.
What does it mean to link the kind of
pattern matching that we do with the sort
of algorithms that are happening in an AI?
How does that change the way
that the human brain thinks?
How does that change the way that the AI
thinks? How do they control each other?
How does this whole ecosystem
move forward? And then
there's the question, well,
even if you do this, if you merge the
two and they're moving forward together,
this merged ecosystem of intelligence,
does that really solve the
original problem of AI Safety,
or is it just taking away that one problem
of the competition between humanity
and AI or that one aspect,
because the self improving AI,
which could be say something that is
reinforcement learning and following a
utility function,
that might still be a problem in itself.
Even if you've got these other merged
AI and human brain uploads there,
you could still have a separate class
of AI agents that are a problem.
So, it could be that those two things
are tangential or maybe even orthogonal,
in a sense. So again, that's something
that deserves more time. Now,
obviously we can't answer
all those questions here,
but we can make a start at being more
precise and it's stating those questions
clearly, and that's what I hope
I've managed to do here. Now,
I'm going to stop sharing my slides.
I'm going to hand it back to Mallory here.
Thank you,
very much.
All right,
thank you Randal.
There's going to be plenty of time for
followup questions during our panel
discussion,
but again,
you can put those followup questions
on the live stream or call in at the
numbers.
Now,
we're going to be going to the
interview with Jaan Tallinn.
He is the co founder of Skype and he's
been a longtime AI safety supporter.
So,
we'll go ahead and show that.
Hey Mallory, it looks like, last
minute, mine was working in our tests,
but it's not.
Would you be able to stream it?
It looks like we're having
some technical difficulties.
I am going to go ahead and just play
the interview on my computer and I will
stream it for you guys.
Thanks,
Mallory.
Just need to change my speaker output.
There we go.
We're sharing.
I'll just introduce you first,
and introduce the topic.
We can always crop out anything in
the beginning that doesn't belong.
Our esteemed guest for
this event is Jaan Tallinn,
an accomplished and well
known entrepreneur, investor,
and theoretical physicist from Estonia
who co-founded the peer to peer
video-calling service
Skype among other ventures.
He's since been a leading voice on the
frontiers of AI safety
and effective
altruism and has co-founded
The Center
for the Study of Existential Risk and The
Future of Life Institute.
Welcome Jaan.
Thanks for agreeing to give us your
time today despite your very busy travel
schedule.
As you may know, at
the Carboncopies Foundation,
we usually focus on the technical
challenges for whole brain emulation.
Occasionally we explain why we think
mind uploading through whole brain
emulation
is an important step for the
adaptability of an intelligent species,
but everyone at the Carboncopies
Foundation is also very
interested
artificial
intelligence and artificial general
intelligence.
Since, however,
there are already so many labs
and companies like Deep Mind,
groups like the Machine Intelligence
Research Institute
and others dedicated to
AI,
we often don't address
it in our workshops.
We decided that the interaction between
those two domains should be addressed
explicitly.
The possible interactions
are sometimes alluded to,
from what I've heard and read so far,
there's a lot more vague hand waving,
I would say, than careful study.
So,
we wanted to try to move
that along a little bit,
starting with this workshop on whole
brain emulation and AI safety.
So,
if you don't mind, I'd love to ask you a
few questions
about your thoughts on AI,
and from there,
maybe we can gently tip-toe into questions
of where AI and whole brain emulation
meet.
Is that okay?
Sounds alright.
So, you've got a long history
of being concerned about,
or supporting serious study of,
existential risk and, in particular,
AI risk and AI safety.
Would you mind telling us a little bit
about how your thoughts on that topic
have evolved over time and
where you stand on that today?
Yeah, I think my thoughts have gotten
more and more uncertain, in some sense,
as I've been exposed to more
and more considerations,
and more research has actually come out
on achievements and progress that AI has
made.
When you say that you're more uncertain,
do you mean you're more concerned,
or just that your not entirely
sure of what the problem is?
Yeah, I see a wider spectrum of possible
scenarios now. Before I got involved,
before I read [ ... ] and stuff, I just
didn't think that was an issue at all.
I was a programmer and I wasn't
afraid of programs.
But then, [ ...
] pointed out that, like, wait a
minute,
if you can create a program,
that is able to create programs,
that's able to create programs,
and that happens to be smarter with each
step,
we might have a serious problem.
Then I got focused on this scenario,
and I do still think that we have no
guarantee against that certain program:
that runaway AI. However, recently,
I think the Open Field project introduced
a concept of transformative AI:
instead of talking about recursive
improving AI, or narrow and general AI,
which in some sense is actually
specific:
we should, actually,
just focus on the aspect of
AI that is important for us.
It can be very transformative.
Even if it's narrow,
or if it's recursive certain programs,
its for some reason does not exceed
some threshold than what we improve,
but still, it's going to
be transformative. So, that
is, in that sense, my...
I see a wider spectrum of transformative
situations on transformative scenarios
for AI which is,
in that sense I'm more uncertain.
Yeah yeah.
So, we could go in all
kinds of directions with this,
and I don't want to spend
too much time on it,
just because we have these other
topics that are really the focus of the
workshop that's going to happen.
I am
kind of curious, because you mention,
getting real advantages for the species,
for all of us,
out of narrow AI out of certain
directions. Narrow AI, itself,
also could pose concerns, right?
I mean,
we've seen some issues with
Narrow AI even in recent times,
like the way that the stock market crash
could happen, or other things like that.
Right?
The Facebook scandals,
they have to do something with AI?
I'm not sure how much though,
but still.
Yeah, indeed. Okay, yeah, but
it's not as
concerning as,
say, the issue of AI agents,
to you,
probably?
Not that current of things,
but perhaps in a few generations from now,
we might have some thing that is just
super-manipulative, or super-addictive.
Why not go that direction,
because of a certain program? Say,
like that kind of a scenario
are still a part of my palate.
Okay. Well, if you don't mind, I'd
love to know what your idea is, of say,
a worst-case scenario vs. a best-case
scenario of how we move forward with AI.
Yeah,
I get that question quite a bit.
I usually don't have a
very specific answer,
because I think
that whenever I start
thinking about very specific scenarios,
you're almost guaranteed to shoot
yourself in the foot, because each detail,
basically, will make the
scenario less likely.
So,
I think it's more valuable to think
about narrow
properties of traits that
really positive scenarios and really
negative
scenarios have in common.
So,
for example, I think almost all
existential risk scenarios
coming from AI,
synthetic bio,
from new quality,
that nuclear is going to unlikely
going to be existential catastrophe.
One trait that I think
they have in common,
is that they are going to look
like catastrophes to humans.
So,
it's an interesting point that the
first piece of existential resource of
humanity
is by the Manhattan
Project scientists.
Where,
what is the probability of igniting
the atmosphere
once they do the first
nuclear detonation?
For the rest of the plant,
it would have looked like one big
catastrophe with no air to breathe.
So,
I think that's one likely common trait in,
not all,
but in many existential scenarios.
Like there is a hoop of scenarios one
of the traits that I can think of that
they have in common is that there's
an increase rather than decrease in
optionality so like one thing that
technological progress has given us is
increasing optionality.
Like I do have an
option to easily visit other countries.
These types that hundred years
ago very few people have.
At least not that in that quantity.
Whereas in future,
if things go well we might have options
to travel to other places or options to
upload ourselves or not upload ourselves
that's also an important position to
keep.
That's another common trait I think that
is positive and that's another common
trait that unifies a set of futures,
and
specifically a set of positive futures.
I can totally the point
that you're making.
I also see optionality as being one
of the most important things for us.
It's sort of a something that for some
reason as human beings we care about that
sort of thing.
I think that's more general
than human
I think that's agents in general that
have references over the world states
like all other things being equal they
would prefer more options and less options
because of the uncertainty that they
have.
I think it's a meta mind,
if you have the references but you don't
know what the correct actions are to
take right now,
then it would be valuable to have all
more options because in the future you
will have more information about
what options are actually there.
So that's a general agent.
You can get into some interesting sort
of philosophical directions there because
when you think about having as many
options as possible then of course that
includes things like options to do
things that we currently simply can't do
because maybe we can't even think that
way so something like a neural prosthesis
or mind uploading would be necessary
even to just be able to do those things
like let's say for example be aware of
and react to things that are happening at
very short time intervals like
microsecond intervals it's a world that we
currently can't live in right and so
optionality might include changing who we
are in a sense.
It gets interesting because some people
would say if we change ourselves a lot
and we're not really who we are anymore
as humans right now because we've
changed a lot then perhaps that's very
similar to a
scenario where say a worst
case scenario that was mentioned where
suddenly the environment no longer
supports humans as we are. There are
areas where that might look very similar.
I think it's important to be more precise
about what do we mean by optional and
how do you measure
the
increase in optionality.
You can certainly have an increase in
optionality that one person or one group
of people like Manhattan Project
scientists have to detonate potential like
earth bending device that's
an additional option for them.
But I might result in everyone else
losing all of their options by becoming
extinct
so in that sense like a more
general version is that like we should
really measure some kind of aggregate
optionality that humans and more general
sentient agents
And even that was very very difficult
to measure because you might then argue
that you should be measuring the aggregate
over all possible future descendants
of humans as opposed to just the people
who are here because let's say that
something takes optionality away from
ninety percent of the people but then the
ten percent of the people who have more
optionality that creates a much greater
future with more optionality for many
more people than the alternative.
We'll go into either popular population
ethics and infinite ethics even.
It's like active research topic and it
looks like we need to solve it sooner
rather than later.
I'm saddened by the fact that we can't
talk about every topic at length because
these are super interesting things and
I've always wanted to dive deeper into
this like how do you decide what's good?
Don't you find it almost suspicious that
there's a very strong overlap between
super interesting topics
and super important topics,
it's not always but it
should be like that.
Yeah that could point to
something as well. It's true.
Yeah okay well I guess maybe we'll have
to have another workshop someday on what
is good or how do you decide good.
But that seems that falls more into
say the domain of effective altruism or
something like that.
You can always kind of like make progress
by going up on the level of meta.
So like you say like like ah it seems
really hard to decide like what is good
then you go like up one level meta,
okay so what kind of things should I do
now that would give me more optionality
to decide in the future what
is good. Like, for example,
at the recent conference in Puerto Rico
the Future Pipe Institute organized that
there was like someone who was making
a really good case that it's probably
going to be really likely that this
massive value in having like this
contemplating period during which we
basically are not like as humanity think
about what's going to happen next
trillions of years as opposed to like just
like rushing along with the
first thing that comes to mind.
Because it's like just we have like good
reason to think about that we're going
to be better at making great
value to different periods.
Then we would be basically losing from
not acting companies in this first period.
That's not a meta point.
Yeah I think that's probably true.
It's unfortunate that when you look at
the decision makers across the world very
often that's exactly not what happens
it's more the opposite that the next
supporter is what matters the most.
That's also like an opportunity
because you see why this is happening,
you see the game theoretic
underpinnings of these situations.
And once you can realize that,
you can take a step back and think
about how do you solve those.
What's going on coordination problems.
Okay. Well having created this context
and putting it in a setting like this.
Maybe this is a good moment to dive into
the whole brain emulation aspect of it.
So just kind of introducing
here where this is coming from.
When discussed in the context of AI
safety I get different reactions to the
notion of whole brain emulation.
Sometimes people will be seeing
it as a potential benefit,
something good that will help us not have
as much AI risk
because humans will be
able to keep up better,
we will be better integrated with
changing world of technology.
But sometimes I get the exact
opposite reaction which is, well,
whole brain emulation seems like
something that adds another unknown.
It adds something else that could
accelerate the development of artificial
general intelligence or an agent AI.
And that it could be its own risk,
in some sense as well.
Which kind of relates to what
we were talking about before.
About which changes on optionality are
good and bad and sort of figuring out
what the difference is there.
So I was wondering if we can get into a few specific questions there because there are some
main avenues of thought or critiques that come up from time to time.
So the first one that I wanted to ask about is this idea that making a connection
between the human brain and a computer. So brain computer interfaces,
that is somehow a very important step towards uplifting us to a point where
we are more integrated with machines and where the gap is is made smaller
between human minds and artificial intelligence and the differences in our capabilities
that somehow this is going to create a symbiosis as one well-known entrepreneur has said with artificial
intelligence. Have you thought about BCI? And I'm not just keeping it separate from neural prosthesis or mind uploading
for a moment, but just BCI as a tool for that.
I guess my clip answer would be that, like Robbie Hanson's comment about this, he said that trying to make humans
competitive with computers by developing brain machine interfaces is like trying to make horses competitive
with cars by developing stronger ropes and so I wouldn't rule out that there is something there, but the bandwidth
argument that might just start being utterly unconvincing because if you think about it if you were kind of like partnered
up with someone who runs 1 billion times slower than you do. Then that beta is basically a person who can say one word
per decade. And it's just going to be a burden. Is it's okay to be useful. You already know what the word going to be.
That's the answer to bandwidth. That's a claim but there might be some more interesting arguments that I haven't heard but like...
It's possible that I just haven't paid attention to some more serious thinkers in that space.
I can certainly learn more things about the brain with higher bandwidth.
And then that could be useful in brain uploading for example. Or useful for alignment purposes.
I see brain computer interfaces as an essential component for data acquisition, for learning about the brain in order to be
able to model brains to be able to make neural prosthesis and that sort of thing. So as a part as a technology, heading in
that direction, absolutely of course it's very important. But yeah I agree with you that in itself that's not really solving a
problem of this gap between the two. We don't have to get into that much deeper sense we're both on the same side with
this one. I would like to hear someday from the people who do advocate that as being the most essential thing to do for AI
safety. I'd like to understand where they're coming from or how they imagine this should work. If there would be like some
serious careful thinkers, that would be super interesting to talk. So the next critique or argument that I often get is
that if you if you work hard on building neural prosthesis and then from neural prosthesis or partial
brain uploads you end up getting two whole brain emulation or some kind of artificial brain like that,
this is in itself of course a type of artificial intelligence or artificial general intelligence if you like although other people would call it
natural intelligence emulated. And that it would be fairly easy to make it superhuman just by say increasing the
speed or something like that or making sure that the memory is never fading something like that. So then that there
could also be some sort of takeoff scenario, either a slow one or a fast one. Now I'm not exactly sure how self-improvement
works with a human brain. I'm sure there is a method. And it may be a bit more patchy and a bit more complicated than
with something that has one specific algorithm that is following. But before we even get into that do you see ways in which whole brain emulation itself could
be an existential risk to humanity? Oh sure, the way I look... I have talked much about it. So in that sense my thoughts are
not super careful here. As I out source my opinions I should be more careful and honest in this in this topic.
And he believes brain uploads are net positive so in that sense I'm working on priors that come from him.
But I do think that there's this almost uncanny valley before we get to brain uploads. We will get to something that is
potentially capable but not really human yet. So we might have like a potentially very capable nonhuman agents on this
planet which is like almost isomorphic. So that is like one scenario definitely that might be problematic.
Another scenario is basically when it turns out that there's some kind of a competitive
situation or something. Where you get some advantage in some competitive access
by sacrificing a piece of humanity, pieces of values. Which means that like we have
like some race to the bottom. That would also be another extension of disaster potentials.
Getting fake humans might be problematic or ending up in some weird place.
But but also if you get things right, then that might be really awesome because we might just buy more time when it comes to
solving problems. Yeah but it's clearly so complicated because there are so many different details about how the research
evolves and how you get from the very first bits you're doing. Say you're just making Ted burgers hippocampal
prosthesis or something like that. Or you're trying to emulate the brain of a Drosophila and what happens if you start
building agents based on these two Drosophila brains. There's so many different paths as you said that makes it a
really complicated question of how do you make sure that you end up with a net positive of that. Versus you end up going
down some path that has risks associated with it. But of course that's also true for any technology development.
You know every time we develop some new technology there are many different possible outcomes and products and
things that could have net negatives and net positives as well. It's just too hard to...
So there definitely are some ways in which it could be an existential risk. But it's not that clear.
I was kind of curious about this case of saying how could people using a neural prosthesis become an existential
risk to to all of us or something like that. But again that sounds like a it sounds like a great sci-fi scenario.
Yeah like these weird cyborgs with metallic voices. I have to think if this is again vain optionality versus staying the same.
You mentioned competition. But what if that competition itself is something that has ultimately a long-term net positive
or something like that. You have a species that is a little different that humans...
Technological competition seems to have given us a lot of tools that are clearly useful.
We're actually making pretty good progress through these questions. Even though I thought could take forever.
Let's get into the next time the next critique then or the next argument.
The next argument I've heard that cautions against a strong push for neuro technology and whole brain emulation
is that work in those areas may accelerate advancements towards runaway self-improving AI.
And well does it concern you and how do you see this different say working on whole brain emulation versus just to wait
neurosciences currently investigating the brain? Intuitively it doesn't sound to dangerous to me.
I think because it's sort of like lower hanging dangerous fruits like just throwing more computer at the stuff
at simple algorithms that people are doing already anyway. And trying to create AIs that create AIs
like architecture search. Or like if you're kind of tinkering with something like a spaghetti code and trying to get insights.
That seems possibly problematic but like just doesn't seem to come to me as the most dangerous thing to do.
I was trying to think of some examples of specific things that I might think that would be discovered
by doing research on whole brain emulation as opposed to the way that the most neuroscience is conducted today.
It's not obvious to me. Even with things that are still problems in AI. Say for example that deep learning requires
exponentially more learning as you try to train up more complicated knowledge or a better understanding of context
around it. That's something that we would love to understand better. How it is that humans managed it. Especially since we can only digest
very few examples for everything that we learn. It matters possibly to the point upon that.
Because human is a spaghetti code. It's not clear how you would learn this from trying to do whole brain emulation.
It's more something that cognitive science, AI research itself, something like that is more likely to come across that solution.
AI itself was able to figure that out when it's doing architecture search. If they feel it just like a human doing
architectural reverse engineering on human minds or AI doing architecture search in AI space.
My money would be on AI. It can simply do so many more searches so much faster. It's like sume are dumb, but evolution was even dumber still.
That brings us to the fourth big argument. And this one is one in favor of whole brain emulation. And the idea is of course this idea of symbiosis
or the merger between what is the human mind and the merger with AI and its capabilities.
And the idea that you can reduce AI risk or the risk to us in a sense by being more closely integrated completely entangled with
everything that's going on in AI. So that a whole brain emulation of mind uploaded person could benefit more directly from
what is developed in AI because AI and modules can be part of them or they can be part of it. It's it's kind of hard to describe because
I don't really know how to imagine it. But it's sort of this join them rather than compete with them. This this idea of in the
past royal families used to marry among one another to avoid competition and to consolidate their power.
It's a mingling of a DNA in a sense except that you could say that this is alien DNA it's AI DNA is something different than human DNA.
Have you given some thought to what this merger might look like? You brought out the alien DNA. AI is going to be more alien than aliens.
If you have biological aliens, they are likely going to have similar eyes. This thing that DNA is going to be similar if we merge with aliens,
but even that sounds icky to me. Just to say the least. Merging with AI should be way more icky. Even though it doesn't seem like aliens.
And also I think it just it doesn't strike me as a sufficiently rigorous topic exercise.
When you talk about solving a complicated problem by just ignoring most of it.
Having said all that, I still I would leave open that door for some serious thought in
this in the space and perhaps people actually would come up with some interesting scenarios that involve things like transparency.
For example transparency to make it more difficult for AIs that are improving to do like...
And the more ways we have to monitor them the more difficult would it be to do that.
...more the better we would be of steering those critical phases. So that would be one angle of these merger scenarios.
I completely agree that it's necessary to get more detail to think more rigorously about these examples.
Because what we're doing now it's useful to discuss like this but it's still kind of hand wavey. We're
saying in general one could think of how this might help in supervising what how AI is developing and stuff like that.
But when I try to think in real nitty-gritty detail about how AI and and human minds can merge, then I start to think about
things at a smaller scale. Not the scale of an entire brain but... Let's take an example, it's a real example, let's take Ted burgers hippocampal prosthesis.
Now Ted burgers hippocampal prosthesis, when implemented on, either software or in a chip. It's really just a model
that is or learning to approximate functions from input to output and it's searching for those best matches. And then using them to to
do what that piece of brain tissue is supposed to do. Now there are many different ways you could implement that.
And and of course, even now, that's where AI comes in. He's using machine learning techniques to build those those models.
So there is a lot of ways in which developments in AI can find a home and these pieces that are then developed as
replacements for bits of the brain. It could be very small. It could be at the level of an individual neuron or an entire area. It could then lead to differences like
say a hippocampus that allow you to directly select which memories to keep and which ones to forget and that sort of thing.
You can imagine that there could be other ways of strengthening connections between different patterns than the normal Hebbian learning.
But another way to do that. So I can see a lot of ways that where at this very fine scale where AI immediately starts
to become integrated into neural prosthesis and whole brain emulation because it's just the natural way to try it to solve these
problems to implement things in there. And and that's the first part of it. And then the second part is where you would intentionally want to
make it possible to feel that you have the capabilities of some AI system and that it feels as natural as say being able
to retrieve something from your memory so that it's like a part of your brain.
And and then you start to wonder, well, if these are the things that you would want to do anyway for doing whole brain
emulation, then how does that map into wanting to to have some kind of safety valve on how AI develops because that's not automatic
that's not something that just evolves directly out of it. You would have to expressly try to make sure that the environment in which
AI is developed is such that human minds have insight as, you say, transparency
into what's going on. So that's still not really an automatic thing.
It still feels like even if you're trying to create this merger because you know whole brain emulation uploaded minds
are already closer to software code and I mean they are, they can be, software code and therefore you see this at a small fine
detail you see these these mergers happening. It still feels like the problem of AI agents that self improve rapidly
and could go off in a wrong direction is kind of a tangential problem. It's still something that needs to be explicitly
dealt with and isn't automatically... Not just like individual AI agents, but also like some weird races that we might create as in this like whole brain emulation scenario.
The opportunities are kind of similar in orientation and whole brain emulation
that we will just create smarter humans and we definitely need smarter humans right now to figure out like how to actually stabilize the future.
However the threat model again is like we might be creating smart non-humans by doing careless augmenting
or we might create some weird race to the bottom in competitiveness and basically lose overall optionality
and all humantics. And that is what feels like not good to me. There is opportunity there in having...
Simple thing is like, I think, a lot of like augment allocates point to is that we kind of are already augmented. In some sense it would be a natural
progression and so far it seems to be with with a few setbacks and seems to be like so far so good.
Other question in like, how much runway do we still have until we you have to be more careful and more planning.
So we've run through pretty much all of the main questions and I do have a concluding question but I wanted to ask first if there is
something in this whole topic that you feel that we haven't addressed or haven't talked about that you think really should be mentioned.
I haven't specifically spent a lot of time, when it comes to augmenting and uploading humans.
I'm probably like another ajacent scenario that we... Another pursuit of research or research area that we haven't talked about right reasons.
Its basically genetic modifications and embryo selection or or direct gene editing which has very similar, perhaps minor,
threats and opportunities on augmentation. So that's another interesting alternative to keep in mind.
If you say that you want augmentation for x that's one question that you have to be prepared to ask when it's like, why not just
like genetic humans did you get that x, might be safer or might not depend on x. There's probably a lot of things that you can't simply genetically modify humans to do.
Yeah but the cool thing though is that it seems to be a minor version or augmentation, built-in opportunities, there are fewer opportunities
probably and there, are but there also have fewer threats. Because in some ways where...
Some interesting point that somebody made was that you could humans, you could get a superhuman by just averaging humans.
So like I guess I'm not an expert, so like, I mean the fact is that if you average human faces they become attractive.
Which is like interesting phenomenon. Which basically you're smoothing away irregularity so if they cancel out.
And you get like really nice attractive faces. So it's for similar reasons like when your average humans you get rid of the deleterious alleles.
So the territorial alleles keep us from from potential. So in that sense, it seems like not so drastic thing to just
start averaging humans like in some ways we naturally do that.
Like what communities organizations and companies do, by combining a bunch of humans together in some structure. in an abstract sense.
Well I think that's like a fairly different sense of averaging. Basically you're not averaging humans, you're aggregating humans.
Yeah, I was wondering how that compares with averaging.
The specific thing is like the low-hanging fruits in genetics sound pretty good.
It's just like, just get rid of the bad stuff and why not? But then you can't go beyond.
Yeah that's true. Yeah okay. So then let me get to my final question for you which is really the question about action.
If you want to, and as I know you do, you want to see to it that humanity can maximize sort of its long-term chances for survival and thriving,
optionality as you call it, what is it that humanity as a whole should do, say decision makers
or or individuals as well? Yeah I think coordination
and cooperation. More coordination more coordination seems to be a really good.
One of my favorite essays is Meditations on Work by Scott Alexander.
It's a book-length essay about coordination columns. All of the problems that humanity has, they can be just summed up as
their coordination problems and bad... So yeah figuring out how to cooperate more, because realizing how much there is at stake.
If you think that the world is at stake, then just don't forget that planet Earth it's just a tiny speck in the universe.
I mean Derek Drechsel doesn't like his ideas to be associated with him because he thinks that
ideas are better if they're not associated with any person. So like there's this idea from somewhere, that a utopia where you're building your utopia
with this additional constraint that every step should be an improvement. They shouldn't make anyone's situation worse.
And cannot allow pathological cases like somebody just like wants to be the king. But you know that it's not always how solving
it a difficult error landscape works. Yeah the interesting thing is that we do have like this additional resource
that most people are not aware of which is the rest of the universe. And the place is bigger than anyone.
That totally makes sense if the pain point that people feel is resources. Which it is currently. But I mean there are other possible pain points. Like
we talked about, what if there is a change in some set or subset of people and that is somehow different and there maybe, could be, some pressures that
are experienced as painful to some other subset of people so these sorts of pain points if you want to avoid all of those as well that's that can be difficult.
Yeah so there are some values that are zeros on some sense. Like those values are going to be problem no matter how much resources we have.
So the idea is to try to satisfy them in by segregation. Like everyone who wants to be the king of the world can have their own planets.
So like just accept the meta value that you're going to have a way more optionality than you have, way more resource and have-nots,
way more way better future than even considered. It's just not going to be like relatively good compared to others.
Yeah by pointing to the universe, one big important thing that you've mentioned there is perspective,
because in addition to cooperation, I think perspective is extremely important in having a good perspective on what's really going on. Not just being focused on
you know what you think is super important right at this very moment. And that I think is a big problem, is this long-term thinking is
not very common. Probably because our lives are short. That's one of the reasons why being somewhat cautious,
but still pushing the AI risk framing more like an environmentalist rather than a socialist.
Because like social risks, if you say that AI could be like really bad for society, which is true.
It also creates really bad competitive intuitions in people. It's like yeah I want like my social values to be like
Trump over Chinese social
values
and that's that's just not great.
This is way better to frame it as
a problem that we have in common.
And like environment is that one
thing that we have in common.
So in that sense that frame is periodic.
So framing could be an important thing
to work on. Better frame problems.
Well thank you. This was extremely
interesting. And as I mentioned,
very much at the beginning,
we could have gone off for
hours in just one direction.
Like how do you think about
good or something like that.
So yeah that was wonderful.
Thank you.
Thank you so much.
Can everybody hear me?
Yup.
Okay.
Now I have the screen sharing off.
Sorry for my audio going
out randomly like that.
That was a really interesting interview;
I liked it a lot.
We're going to follow this up
with our first panel discussion.
These questions are going
to be in relation to the
Jaan Tallinn interview that
we just watched,
but after that,
we're going to take a short break and we
will resume at 1:00 PM Pacific time for
our general panel discussion.
Again,
you can ask questions directly in the
live stream YouTube channel or you could
call in at call.carboncopies.org
or call the number (415) 455-3017.
This information is
also on our events page.
Before we opened to questions
our panelists will have
an opportunity to discuss
the interview and our workshop topic
and then we will move to our questions.
Our panelists today are Dr. Randall Koene,
Dr Abdulfaz Alipour,
and Dr. Keith Wiley.
I already introduced Dr Koene,
but again he's our founder and chairman
for the Carboncopies Foundation and he's
known for his efforts to advance
research in whole brain emulation.
Dr. Alipour has his PhD in pharmacy and
is pursuing a double PhD in psychology
and neuroscience at Indiana
University Bloomington.
In his research,
he is developing new neural
network architectures that
are inspired by cortical
column circuitry and he is testing
their predictions through large scale
in-vitro neural recording.
Dr. Wiley has his PhD in computer science
and he has expertise in artificial
intelligence,
evolutionary algorithms,
various artificial intelligence
techniques and machine learning.
He is also author of the book A Taxonomy
and Metaphysics of Mind Uploading,
which is available on Amazon. Welcome
panelists, thank you for joining me.
Yes, thank you Mallory. Just to add to
things you've already said, by the way,
could you mute yourself,
or can
everyone mute themselves when someone
else is speaking, just in case? Okay,
there we go.
Mallory, you're not muted at the
moment. I don't know why. Anyway,
it doesn't show up as muted,
but you probably have a different mood
button on your headphones or something.
So,
just to add quickly here,
I don't see,
Abdulfaz Alipour on the hangouts yet.
So I'm assuming that he hasn't logged
into that. That's not a problem.
There he comes, he's joining right
now. And just so you all know,
there are a few other people from the
Carboncopies Foundation who are also
logged into the hangouts and you guys,
you're all welcome to directly
participate as well of course,
just like our viewers who can
call into that phone number or use
call.carboncopies.org.
I wanted to just quickly say,
there are a bunch of questions that came
in from the audience and we're going to
try to get to as many of them as we can.
Some questions can be a little bit
difficult in the sense that we may not
understand exactly what
you meant in your question,
so we might answer it the wrong way. If
you want to be sure that we get it right,
that's where calling in
either using the phone number,
or call.carboncopies.org,
is an excellent way to
prevent that problem because,
if you hear us getting it wrong after
you've asked us the question yourself,
you can just correct us and say,
"No, no, I, I really meant this,
could you please answer
that question?" So,
there's slight advantage over typing
it into the live stream chat. Okay,
with that, I just wanted to
throw out a first question,
or present this to the panel.
One of the things that we got into,
in that conversation,
is we really started talking a bit about,
what does it mean to have
a good outcome for humans?
Because when we talk about AI risk,
I guess the simple version is where you
assume that AI becomes smart and then
there's a terminator scenario,
and all the humans get destroyed.
That's the science fiction
version of AI risk.
But that's not the only
outcome you could have.
You can have an outcome where AI
is simply going about its business,
and humanity is just this tiny little
thing in the corner that doesn't have much
to do with it anymore. We
don't participate much in
their exploration of space,
they just kind of keep us alive in
a zoo. You might well say, right?
And then the question is,
is this a good outcome?
And people may have different views on
that because okay, humans are still alive,
our society still exists,
and maybe we don't have all that
option-ality that Yam was talking about.
But maybe if you look at it from the
point of view of a whole intelligence
ecosystem instead of what
are the humans doing,
maybe some people still think that's good.
So,
determining what a successful or good
outcome is can be the first thing you have
to do. And I'm wondering what the people
on the panel think about what is a good
outcome for us and why.
Am I coming through?
Well,
with regards to this sort of a
preference to sort of protect our,
special-ness and therefor sort of make
sure that we don't get pushed into a,
into a sort of corner of
obsolescence or something.
You could re-frame that question as,
what if we discovered,
eventually,
that there actually are other
intelligent species in the universe,
which is a popular position. It's actually
not one of my preferred positions,
but if we found out that was true and
we've found out there were sufficiently,
pretty far advanced beyond us,
then we would have the exact same feeling
of mediocrity. It'd be like, oh, well,
okay,
we're just some podunk,
relatively doumb intelligent
species in a corner of the galaxy.
So then, the prescription would be,
oh, well, to preserve our specialists,
we need to somehow keep those
aliens down or something.
So, barring the technical challenge of
sort of countering some alien species,
that's already ahead of us, it's just
philosophically unconvincing. Like,
why is the solution to our bruised ego
to go crush all the other aliens out
there that have gone further than
us? So then, if you rephrase that,
then it's like, okay, so
now do we have a good
motivation to try to prevent AI,
or some sort of enhanced
versions if humanity,
from coming into being?
Does that rationale really make any
sense when you look at it the way that I
just phrased that?
Yeah, I see what you mean.
Before I try to answer that,
I was wondering if anyone else
on the panel has an opinion.
No, no specific opinion
on this particular topic.
Okay.
Yeah,
I try to grapple with this question as
well because of course when we talk about
whole brain emulation,
we're talking about changes to at least
optional changes that some people could
choose for themselves. It could be
that we all go in different directions.
That, some people develop in one
way or another. And in the end,
if you look at the end result again,
just like what Keith was saying,
this could look just like bumping into
aliens with the only difference being
that there's this common origin.
And then you start wondering about
things like the rate of change.
Does it matter how quickly we sort of
develop into lots of different kinds of
intelligence.
So,
all of this is hard to pin down and
it's hard for us to determine in advance
before we've thought through all these
questions, what are the criteria,
the parameters that we think are valuable
or important about humanity and what
we've accomplished, that we say is
this is why we should all survive.
This is why either humans as they are
now biological humans should continue to
exist into the future,
or at least minds that think like humans
to some degree or whatever intelligence
evolves,
needs to be derived from the way we're
thinking because of certain values that
we think are really important and
that we want to carry them over.
Or we could just say, well, this is
all way too analytical and ultimately,
it's just the drive to continue to exist.
And as long as we feel that whatever is
happening is because we choose to do it,
then it's us, and therefore,
it's okay. That's success.
And if whatever's happening to
us is not something we've chosen,
then it's not success.
But as you'll see later when
I talked to Anders Sandberg,
that isn't quite as simple either,
that of when are you in control.
Does anyone else have a question from
the panel about Jaan's conversation,
before I bring up another one or,
before we go to an audience question?
Sorry,
nothing comes to mind.
Okay. So I think we'll switch to an
audience question in just a moment.
I want to throw one more question
out there before we do that.
And the one I want to put out is,
towards the end of the conversation Jaan
mentioned that another issue we should
look at is biologically
enhansed humans or,
that whole other different technology,
not using computers, not uploading, but,
what happens if we can genetically modify
to achieve whatever augmentation we
want.
And I'm wondering,
does anyone here have an opinion about
what that means either in terms of being
a risk,
or a benefit in itself?
How does that compare to something
like AGI and whole brain emulation?
What do you think about
this biology as the route?
Are you referring to when he was talking
about the sort of hopelessness of
trying to bring horses up
to competition with cars?
There was that,
but then he said that,
and then at the end he kind of
switched back and said, well, you know,
another thing that we should really
not forget about when we're looking at,
or trying to plan ahead for humanities,
we should also look at the
biological augmentation angle,
because a lot of things
might be achieved that way.
I think my response would be the same
thing that he said about the horse,
which is that there are certainly no
particular reason to think that blind
evolution has converged on
some sort of biological maxima.
We just are,
kind of,
whatever evolution was able to cobble
together on what's called a local Optima.
So,
we're good at what we are,
but in the space of all
possible DNA configurations,
there's very little chance that
were as good as we could be.
But it's still seriously worth considering
whether we believe that biology has
any capacity to compete with non biology,
in the long run.
Right now, there are many ways in
which biology is the better technology.
And in that sense I just described biology
as sort of successful nanotechnology.
But once you sort of escape the protein
and lipid and sort of need for water and
once you kind of remove those biological
variables that are incidental to what
you're really trying to accomplish, but
they're required at the chemistry level,
then it's hard to believe that biology
is really buying you anything intrinsic.
Biology is just how we do
it. It's how nature does it.
It's how evolution does it.
But there's no reason to think
that it's really competitive.
And we have all sorts of
evidence that points that way.
So,
our relatively simplistic 20th century
computers on certain metrics just
outperform biology in ways that are...
There no conversation to have about it.
And I'm not convinced that that won't
be the pattern for everything as
technology goes on.
I can't see an obvious counter example
where biology has to be better than just
going at a problem without having a
preference for a particular solution.
Instead of leaning toward what LG or lean
toward the full space of all possible
physics and chemistry,
I'm doubtful that biology is going
to win very many of those races.
Yeah,
and I agree with Keith because,
from an AI standpoint,
I totally agree with the horse
analogy of that was discussed before,
and I think it would be,
somehow,
meaningless sometimes to think of it like
that as a method to catch up with AI.
But from another perspective,
if you think of nondestructive
whole brain emulation,
you would eventually need some sort of
a brain computer interface or some sort
of a neuro-prosthetic to transfer
information from biological brains to some
artificial medium. And I think
from that perspective, BCI,
neuro-prosthetic,
is extremely important,
but from arguing for advancement of BCI
and calling for more attention to BCI or
neuro-prosthetics, because in order
to catch up with AI, I think that
would be a little bit, a little bit
improbable or implausible at least.
Okay. Yeah, I think those are all
really good points. And yes, Keith,
as you mentioned,
I think you're right,
what Jaan was basically alluding to was
the idea that you can look at biology as
getting to the point of what you might
accomplish with some nano technology.
In a sense it is nano technology, but
yes, as you both pointed out as well,
there are limitations to that.
So,
I noticed that I've sort of
taken on the kind of moderator
role in this bit of the panel.
I'm Sorry about that, Mallory.
You can always jump in and
do that if you want to.
All right. Are we going to be
moving to audience questions now?
Okay.
So,
the first audience question that we're
going to ask is from Leslie Seamor.
He asks,
"Skype moved voice and
multimedia communication from
conventional PLTS to open
protocol,
scalable Internet with firewalls,
and similar security
architecture constructs.
Do you see any similar changes,
at least at the metaphoric level when
the Internet payload content breaches the
wet brain directly?" Could somebody,
maybe rephrase that question,
or can answer to that,
or has knowledge pertaining
to that question?
I can try, but I'd love to let
someone else have a first go at it.
I'm still thinking, and I
spoke first last time, you go.
Abulfazl,
how about you?
So,
if I understand this correctly,
the question is,
"What would happen if you are able to
connect..." To put it super simplisticly,
and correct me if I'm wrong,
"What would happen if you are able to
connect a search engine into your brain?"
Do you agree that this would be a
simplified version of this question?
Yeah, I'm not 100% sure.
This is where, as I said,
if people asking questions
were on our call line,
it would make it easy for them to clarify
a little easier than
typing into the chat there.
I'm curious if Lezlie...
Oh yeah,
sure.
Go ahead Keith.
I'm curious if he's asking about the
benefits of open source or open
standards as opposed to a walled garden
approaches to security.
I completely forgot what
the acronym PLTS stands for,
and trying to just read that.
At any rate,
I certainly think that there is,
in most cases,
security is better served by open
standards that embody secure algorithms,
sort of like the way
public encryption works,
as opposed to security that is
achieved by trying to make sure that...
Thank you Rose for
decrypting that acronym.
...as opposed to security that basically
depends on making sure that you...
that no one even knows
how your security works,
and that is the method of security.
That's an inherently weak
form of security. So,
one thing that we definitely
have to take very seriously,
as technology becomes ever increasingly
incorporated into our brains,
is we've really got to get security
right in a way that we have clearly,
completely, failed at in
the last 30, 50 years.
When I was in graduate school,
one of the professors who I was inspired
by, in terms of artificial life,
one of the people that sort of drew me
to the school I chose was David Ackley.
And,
he was interested in computer security
and since he was one of the pioneers to
artificial life, his approach
to it was very, sort of,
evolutionarily inspired.
And he and Stephanie Forest,
and others at the
University of New Mexico,
took an approach to computer security
that was intentionally inspired by
biological immune systems.
And I think we have seen some of these
analogies crop up in our attempts to
build these auto adaptive self
monitoring on the fly security systems.
We do sort of try to do some
of this in the modern era,
but I'm not sure if we have a very refined
and underlying philosophy for how we
do it.
Just to get back to the original question,
I think that as this becomes increasingly
incorporated into our bodies,
it will just become
that much more serious,
just because it'll put
lives more directly at risk.
We're already concerned about
hacking pacemakers, Parkinson's,
and brain implants.
And this is just getting more
and more serious as time goes on.
And just to add to that,
imagining that the question was,
what would happen if you
have brain implants and what
would you do to secure the
communication in these brain implants?
I was actually...
When you talk to people about
brain implants, one of the
common concerns is that,
how would you secure that a brain implant,
how would you make sure that
someone cannot upload some viruses,
or some bad software
into my brain implant,
where someone could basically
hijack my brain implants.
I think possible,
approaches to this challenge
might be to use... So,
one possible idea would be to use some
sort of a distributed ledger technology
that is currently developing
for internet-of-things devices,
and one of the approach would
be to use quantum switch. But
the way I think about,
I think, at the moment,
one possible viable option
to secure communication in
brain implants would be to
use some sort of a
distributor ledger technology.
That's what I think about it.
Okay. It looks like the person who
asked the question, Leslie Seamor,
is actually trying to
call in at this time.
So maybe he's going to be able
to clarify as questioning person,
which would be very useful.
I'm not sure he's ready yet,
but we'll see in just a moment.
Is that phone number he's attempting
to use supposed to go into the
Uberconference?
It goes into the Uberconference,
which Alan can then connect
directly to our hangouts.
I'm not hearing him yet,
so I'm assuming that Alan hasn't...
And we have Leslie,
who had a question from the audience,
he is here live on the call now.
Go ahead Leslie.
All right, Leslie. So, how it
will work out is they will just...
I'm afraid the audio just cut out.
I couldn't hear it.
I think she's just explaining
to him how it'll work,
Okay.
and then he'll be on.
Leslie's live on the air.
Leslie,
can you hear us?
I didn't mute Leslie. Sorry, Leslie.
Can you hear me?
Now we can.
Thank you.
All right.
Hello?
Yes, we can hear you now, Leslie.
Hello. So, you can hear me, right?
Yes,
we can hear you.
So,
I believe that...
Leslie, if you would just stop
the YouTube that you're watching,
then you'll be able to just hear us,
and then you can join in back on the
live stream once you're off the call.
Okay.
Why don't I try again?
Okay.
I have no YouTube.
All right.
I assume you can hear me.
We can hear you just fine.
So,
a simplified model of would be that
we have a brain implant or something,
some sort of
infrastructure. So, the question is...
There are two questions. One is how
to protect it from interceptions.
Just like when you have a cell phone.
You make sure that no one is listening,
no one can inject
any information as the
wireless signal goes up.
It maybe a wireless signal,
but the signal is basically translated
from internal nural signals.
And a at the infrastructure side,
there will be agents,
I would assume which represent you as
a person and that agent is intercepting
your brain tree
infrastructure communication.
So,
that brings up two major questions.
One is how to protect the pipe between
your brain and the agent on the cloud or
And the other question is how the agents
of different people communicate with
whatever they're going
to have at that time.
one another.
So,
what would it be...
I would expect that when it becomes
real there will be a standard division
So, It seems to me that just following
the paradigm that telecommunication is
following,
all of these protocols are standardized.
effort.
And really how secure your brain and
will be will basically will depend on the
quality of the protocol and the
firewalls in between those regions;
or something equivalent some
block chain or something that',
and if there are any central services
that all agents are talking to.
How is that central service protected,
when put in place on the central server.
So,
basically the question was poking
at other people's ideas that,
what kind of systems
architecture arrangements.
I didn't see any of these issues being
in the list of questions regarding the
merger AI and membrane.
So, basically, that is what I am asking.
Does anybody have any kind of vision
as to what would be a roadmap,
and how the whole architecture
would be regulated by the industry,
similarly to all the telecommunications
standard bodies that operate today with
the conversion of an
IP base communication.
So, Leslie, I hope you can
hear me as I'm responding.
I don't know how the sound is
going back with Alan's trick,
If you hear it straight through.
How much was transmitted?
You came through fine,
on my end anyway.
Some people are saying that they heard
a lot of echo but I didn't hear that.
So maybe this is...
Hello?
I'm hearing you fine.
Can you hear me?
Hello?
I think that Leslie...
I don't know how the sound goes back to
the Uberconference Allen, if that's...
He should be able to hear you directly
so we can, we can talk to him...
So, you don't need to... He doesn't need
to turn on the live stream, or something,
to be able to hear me.
Right?
He should be able to hear you. So I don't
know why not, but it came through well.
So if you want to...
Okay. In any case, I will
try to address the question.
So, you're right, Leslie,
that the question of safety
for the communication
protocol when you build a BCI
or the safety of a whole brain emulation,
in an ecosystem of AI that
problem was not addressed clearly,
at least so far in this workshop,
and it also, to my knowledge,
hasn't been addressed very clearly or
explicitly by most of the talking points
or the writing that's come out
of the AI safety community.
And I think that's partly because they
sort of constrained themselves to the
problem of what to do about the
intelligence of the AI and the possible
dangerous runaway scenarios of that.
Whereas this other problem, you can
kind of set that aside and say, well,
this is a problem for the encryption
community because this is all about data
paths and encrypting that and about
not letting people hack your brain.
It's easy to dismiss it and I don't
think it should be dismissed because it's
clearly not simple.
All of our systems today
are pretty hackable.
Hardly any of them are foolproof.
So,
I can only say that,
in my own background,
I did some encryption stuff because,
in information theory,
you learn about that sort
of thing to some degree.
Back then the professors were all saying,
well,
even though our systems are
not foolproof today, in theory,
you could make them all pretty unhackable
and now add on top of that blockchain,
which didn't exist back then,
maybe that would make it even better.
I'm not sure because at
this point I'm no longer...
I couldn't call myself
an expert in that area.
All I know is that it's definitely an
issue because currently everything seems
pretty hackable,
or a lot of things do,
and that's going to make a big difference
to how secure people feel in the sense
that,
okay,
I want to connect myself to this
machine out there to these other agents,
or I want to have my brain
emulated and be an upload.
These agents,
again,
how those agents represent us and how
they communicate and whether those could
be faked and other things like that,
this is additional problems and
hurdles you could have there.
It sounds to me like just the general
problem that people try to hack and misuse
every technology out there.
And in the future it might not
just be people trying to do that.
It could be other intelligence,
other AI trying to do the same thing too,
tring to use it to their purposes.
So,
every little point needs to
be analyzed in that sense.
And maybe this is why going to open
standards is where you end up because then
you have the most people paying attention
to at least one protocol and trying to
get it right instead of lots of people
having their own little thing and each of
them being very breakable,
not getting enough attention,
discovering all the holes.
So,
I think something that standardizes is
probably what you would typically see.
So, in that sense, wrapping
all the way around,
my answer to your question would be,
yeah,
I think it would probably go in
a similar direction that way.
I don't know if this answers your question
because it is a complicated question.
It's not really in my domain,
but that was my attempt.
If anyone else wants to make an attempt,
please go ahead.
I just want to say that I agree with
Randal and nothing is 100% foolproof
un-hackable. And yeah, there's always
the concern and it would be much,
much better if we have a system that
defines a standard and everyone is using
that.
That would be much more helpful.
I'm looking forward to
hear what you think Keith.
Right.
Well we're sort of mirroring each other.
I would just reiterate,
I think Leslie mentioned,
in theory there are notions
of quantum security.
There are notions of security in which
even if you can't prevent someone from
breaking in or listening in,
you can make it possible to
detect that it has occurred.
Even if you can't prevent someone from
breaking in without also breaking your
own channel,
you can make it so that if someone breaks
in the whole thing collapses so that
at least they don't succeed and
then you can rebuild afterwards.
There are all sorts of approaches.
We have all become sort of
jaded in the last decade or two.
Every time someone says that they've got
a new approach to security that's going
to solve all problems we
learn that's not true.
I think quantum mechanics is trying to
bring a whole new level of confidence to
that discussion,
but for my part,
I just kind of lost
confidence in the whole thing.
Which is why I go back
to what I said earlier.
The research that was being done
around my graduate department,
although not directly with me,
was to sort of accept that
security is not a perfect thing.
Just the way your body's
security is not a perfect thing.
You were subject to viral and bacterial
and parasitic attacks all the time.
Evolution solution was not to ever
attempt to evolve the sort of a
metaphor of an iron clad wall.
Instead it detects and responds and reacts
and fights back and just sort of does
its best.
And the truth is it doesn't always work.
Things that attack biology sometimes get
through and kill you in various ways.
But that is what the system attempts
to do. It just attempts to detect,
adapt and fight.
And I think that ultimately the right
approach to security is to be adaptive.
We're just going to have to make our
computers and especially our computer
bio-interfaces and everything like that.
We're going to have to abandon this idea
of a mathematically perfect system and
that will be very hard for society
in general to come to terms with,
because we need our bank accounts to be
not some sort of computer version of..
I think
we might've lost Keith there.
Yeah,
that's too bad,
but it gives me an opportunity...
Oh he's back.
Maybe you want to repeat what you
said the last like half minute or so.
I was wandering around aimlessly anyway.
I was just wrapping up that we really
do have to accept the idea of these
computerized immune systems that detect,
adapt and fight and sometimes lose,
but the war wages on,
even if the occasional battle is lost.
And that is our overarching
philosophy to computer security.
I don't think we're there yet.
I don't think our society
is ready for that.
I want to quickly latch onto that...
...Entity that's going
to allow us to do that.
So first of all, I wanted to say, Keith,
that answer was probably the most lucid
answer of the lot and thank you very
much for saying all this.
I wanted to latch onto it because when
you mentioned you have to come to terms
with the fact that nothing is going to
be a perfect solution for security and
you sort of have to learn to live with
that, but society may not be ready for it.
Suddenly,
on a meta-level,
I'm seeing this as something we may
encounter with the whole AI security or AI
safety issue,
which is that the attempt to
analytically figure out AI safety,
which is really what the FHI,
the FLI, MIRI, and others,
are trying to do is an analytic approach
to how do we accomplish AI safety.
It may be that ultimately we'll discover
that it's always going to be the same
thing as this immune system approach
where there is no perfect solution.
It doesn't constantly work, and you may
just have to come to terms with that,
just like in security and communication.
So,
maybe this applies to
the bigger topic as well.
Okay, I think that kind of concludes
the answers for that question. So,
unless there was anyone that wanted
to add anything else on our panel.
It doesn't sound like it. So,
I'll move on to the next question.
This is from Ree Rose.
She asks,
"BCI technology could be
very positive and enhancing.
How do we prevent potential malicious
usage and unintentional consequences?" And
I think that kind of wraps back around
to what we were just discussing.
She goes on to say, "For example,
control of a person's mood,
voting, consumer preferences, et
Cetera." But I would also like to add,
what about indirect control,
such as when we have the
kind of controversy we have
right now with people's
data being available and not being used
for marketing campaigns and targeting
people specifically,
directly for certain things.
If that's going to be an issue once
we're able to upload our brains and if
there's some way that
that could be accessible.
And I also just wanted to mention,
I am incorrect about us taking our break,
we're actually going to be taking our
break at 1:45 and then coming back at 2:00
PM Pacific Time.
Okay. Yeah, thanks for that correction.
I'll take a quick first stab at this.
And yeah,
thanks for broadening this
from just looking at how to
make sure that there's no
malicious,
say,
signals going into the brain or that
your signals are being misrepresented
outward.
But even what Mallory was saying,
how about control of a person's mood?
This is actually part of Rose's question,
could you have control of a person's
mood and that sort of thing.
And I think it's very closely related
to Leslie's question in the sense that
ultimately it's a security problem.
It's a matter of who...
What are the rights that we bestow,
first of all,
when we have these systems?
Like when we started messing around
with DNA there was the same sort of
question, what are the laws going to be
about this? How can you use someone's DNA?
Are you allowed to use a person's
DNA without their explicit consent?
And if they give their consent,
do they really know what they're
giving consent to? Similarly,
when you talk about brain
computer interfaces,
you're going to want to have informed
users and the person using it is going to
have to have a fairly good understanding
of the security that they've got coming
along with that. And that security has
to be pretty good, very good in fact.
But in addition to that,
there are all these indirect ways that
people might find that there is some
control seeping in from elsewhere
that they wouldn't want.
If you can give somebody
images directly into the brain,
for instance, if you can put them
into a virtual reality of sorts,
there's the potential that that virtual
reality can be shaped by those who
create those programs and that that
can influence what a person thinks.
And it can be very subtle.
Now the same sort of thing of course
is happening today with advertising and
political campaigns and all the rest,
and the whole concept of fake
news and all of this. So,
it's not a completely new question.
It just becomes more pressing when
you're so much more directly connected.
And so it's not just the question
of security and protocols,
it's also a question of having laws that
people and agreements they have to make
and education.
Understanding what those agreements
actually mean and what you can,
what you can think that you
are, how secure you are, and
how secure you aren't. So,
this happens today,
even on the Internet,
people don't realize how easy
it is to fall for something.
Say you receive an email,
what happens if I click on this link?
So the informed user is the safer user
education is going to be an important
part.
I know this doesn't sound like a perfectly
technical solution to everything.
A lot of it's kind of messy and requires
bringing a lot of people up to speed
and all of that. But I think that's
exactly how it's going to be.
It's going to be a bit messy.
And the only good thing about this,
assuming that not everything goes foom
is that hopefully we won't have to adapt
everything all at once.
Hopefully it's a step-wise process
for some people are going to get,
say an artificial retina or if
they're a locked in patient,
they'll have some way of controlling their
new robotic exoskeleton and something
like that.
And those are some fairly limited paths
through which communication happens.
And you can,
you can figure out how to safeguard that
and explain to the user how to use it.
And then we become used
to those sorts of things.
And then it goes to the next level
where maybe people have brain to brain
communication of a kind.
And first it's a thin channel and there
are ways to secure that and then that
becomes a broader channel remorse as
possible and they've already learned to
live with it to a degree.
So, hopefully this gradual,
and iterative process allows for the
learning that we all have to do to make it
happen. Anyway, I'll leave
this to someone else now.
I just wanted to add to what you said
and I think there are interesting
promising technologies on the horizon,
as I talked about before,
a little bit about the distributed
ledger technology. So for example,
blockchain is an example that is
immutable. You cannot change it.
So I believe that in the coming years
with the advent of web 3.0 with the
Internet of thing taking over and
becoming more and more prevalent,
I think there might be technical solutions
for these concerns and these problems
that not 100%,
but to a good extent,
they're covering all these different
issues that we're dealing with.
For example, controlling someone's mood,
their voting, and consumer preference,
and all these different hot topics that
we have to consider when we want to
think about neuro-prosthetics.
For example,
imagine that I haven't neuro-prosthetic
that is for depression, for people who,
who have chronic depression.
And then what happens if someone can
hack into that and instead of making
someone better, just,
worsen their condition.
So how would it be secure that,
I think one technical answer to that
would be to look into the coming
technologies like,
Hash Graph or,
Iotas,
and tangled technology,
the Dac technology that it's coming and
I think that might provide interesting
solutions to deal with these problems.
Yeah.
I totally agree that there are probably
a lot of new solutions coming down the
pipeline.
But I'd like to caution that just
because something seems hot right now,
since it hasn't been used much yet,
we don't really know what all of its holes
are and some of them can be insidious.
So take for example,
what you just said about the person
who's receiving treatment for depression.
Now let's assume that there are 20 million
people in the world who are receiving
the same treatment and there's a company
that's providing this. As a company,
therefore has probably got some data
about how often each of these people are
receiving their treatment.
And the user may have signed an agreement
where they said that the company is
allowed to use this data to improve
their services or something like that.
And they interpret that to mean that
they can use that to improve services to
these people,
but not just for the treatment,
but also to sell it to advertisers to
send them stuff that's good for people
with depression.
So again,
you've created a loophole here where
some unexpected things might start to
happen.
You might start to suddenly notice
that all of the advertisements that are
popping up for you,
they're all based around depression,
but they might even be
subtle advertisements where
they are not directly about
depression.
They're just advertising things that
typically appeal to someone who happens to
be going through depression at
certain times of the day and strange,
strange stuff like that.
So it can be very hard to locate
all of the problems in advance.
That's where I think the analytical
approach fails and it has to be this
iterative learning approach for how do
we really safeguard what we're going
through right now.
Yes, that's correct. I
totally agree with that.
Okay. It sounds like we're
done with that question.
The next question that I
have is from Alexei Popov.
I don't think I'm pronouncing
that correctly. He asks,
"Why are we so concerned about AI
threats while staying in the body is 100%
fatal?
Should we better discuss
whole brain emulation into
context of mind uploading or
evidenced based cryonics?" I think
maybe what he's asking is why are we so
concerned with artificial intelligence
safety when there are still issues,
to be solved with mortality?
I'm not sure if I'm interpreting
that correctly or not.
Well, hopefully we were interpreting
it correctly, but he could, of course,
call into call.carboncopies.org
if he wants to elucidate further.
If you'd like to clarify on the YouTube
channel, we can also look at that.
So,
there's one obvious way in which the
question of AI threats is relevant even
though biology is sort
of the guaranteed death.
We're all sort of thinking about solving,
although it's not necessarily my
primary motive in mind uploading, but,
the obvious response is that
everyone's concerned that AI might end
not only your life but end humanity.
So,
if it's a choice between letting humanity
continue to hobble along for a few
more centuries, while we slowly
figured this out and accidentally,
creating some existential elimative event,
then clearly the choice would be, okay,
let's, let's not go straight at AI now,
even though it's tempting.
Let's just keep hobbling along with
our 100 year life spans or 60 year life
spans or whatever.
And let our technology get to the
point where we can do AI without wiping
humanity out.
I think that's the obvious
sort of go to response there.
Yeah,
I would agree with that obvious response.
And just to sort of emphasize
the second part of his question,
shouldn't we be discussing whole
brain emulation in the context of mind
uploading or evidence based
cryonics? Yes, of course.
And that's what we normally do. Say for
instance, in our previous workshops,
that's exactly what we did.
We talked about the technology
to get to mind uploading,
which is whole brain emulation
of one sort or another.
And the different paths that go there.
We've talked about ways to work with
preserved brains and how to get to whole
brain emulation.
And then we've expanded that to
look at a few other topics as well.
So I don't know if you were there when
I was giving my opening remarks and I
pointed out that this is a big puzzle
and we keep adding in more pieces in our
workshops are trying to address
a whole bunch of them eventually.
So now we're trying to address
this overlap area where AI,
safety and whole brain
emulation may interact.
And that's an acknowledgement
of the fact that yes,
we personally may really care about whole
brain emulation as a technology either
because we think it's super important
for individual people or we think it's
super important for humanity as a whole.
But,
but we have to acknowledge that
you're not working in a vacuum.
If we create whole brain emulation,
if we're doing the research
towards whole brain emulation,
the things that happen because we're doing
that work may interact with the other
things that are happening around us,
say in the world of AI
Development and where that goes.
And that all together can have
an effect on where we all end up.
So the outcome isn't just dependent on
our thinking about mind uploading and
cryonics. So I think that's why we're
trying to, you know, find these,
these corners situations,
these tangential things,
these places where domains overlap
and think about that as well.
And from another perspective,
I think it's still important to
think about AI threats. So let's,
in an imaginary scenario,
let's imagine that we have the first
whole brain emulation and that one becomes
an AI thread by itself and it takes over
and it makes it impossible for everyone
else to perform the whole brain emulation.
And so it wipes out the entire humanity.
And that would be the only
thing that exists on the planet.
So I think still it's really
important to consider that AI threads,
especially when it comes to whole
brain emulation, what happens,
and I think this is one of the four
topics of this workshop as well,
that whole brain emulation
itself would be a runaway AI.
And I think from that perspective it
is really important to consider this AI
threat topic,
in whole brain emulation.
All right. I think we'll
move on to the next question.
So let's see.
I believe we will do the
question from Roman Citalu.
"His whole brain emulation
necessary for mind uploading?
Could a far simpler model be sufficient
to simulate the human mind of a
particular person?" To also just keep
it in the context of our workshop,
I'm curious to know if this simpler model
would be generated by AI Algorithms,
instead of necessarily completely doing
a one to one copy of the biological
brain.
That's a great way to put an extra spin
on it. I'm going to jump in right away,
because it's so happens that there was
a part of the upcoming interview with
Anders Sandberg that got into this.
And this is because,
of course,
you can make a distinction between a
simulation of a person's brain or mind and
an emulation where the distinction we'd
make is a simulation is something that
to the outside looks like perhaps this
is a person reacting the way they should
be reacting. But on the inside
perhaps it's really not all the same.
There isn't really the
same person behind it.
It could be more of an actor.
And that's precisely where Anders,
for instance, pointed out well.
Our systems are best when
they understand us best.
So we can see with a cell phone,
for example,
that it works better when the interface
kind of understands where we're coming
from. When let's say Siri learns
something about us or whatever,
or Google learns what our likes and
dislikes are and then they can go out and
find the things we enjoy.
They become our agents looking for the
answers we want to find or the things we
want to buy and soforth.
Now as you get better and better at that,
if you have an AI that keeps on making
better predictions of you at some point
that AI could go out there as an agent
and basically pretend to be you and go
shop for the right things, invest your
money the way you would be doing it,
talk to your kids to make sure they feel
like you're paying attention to them
and all that sort of thing. And it gets
to a point where you could say, yeah,
there's a pretty good simulation of
that person there. But the question is,
is that simulation the same
as an emulation is something
that can pretend to be
you and other people won't notice
it's not, is that close enough?
Is that good enough?
Is that what you're going for?
And that wraps right back around to,
I guess what is always the fundamental
question that we bring up whenever we
talk about whole brain emulation
and research towards it,
which is what are the success criteria?
What do we mean by mind uploading
and whole brain emulation?
What is it we want to accomplish?
Do we want to make agents that
can pretend to be us online?
Is that the main purpose?
Do we want to create something
that can improve us ourselves?
Where we have the ability to
say, think faster, you know,
notice things at a microsecond, a splint
interval and react to those things.
Do we want to be able to live longer
ourselves by being able to emulate our
brains and run as a mind
upload in another body,
have an artificial brain as it were.
Whether you decide to do this by replacing
bits one at a time or scan the whole
preserved brain or which
approach you want to use.
So part of the question is always,
why is someone interested in it?
What is the success criteria
in what they're going for?
And so this part of this
is reaching a consensus,
what do those of us working on it really
think the success criteria should be,
what are we aiming for?
And some of it is perhaps something you
can derive more analytically where you
can say, well, if people are
interested in mind uploading,
it really only makes sense to go after
this if we at least care about x,
otherwise we could just make any
old AI or something like that.
So there are reasons,
by... I'm sorry, somebody
wants to go on. Okay.
I'll finish my thought later.
Next person.
We actually have Roman on
the Uberconference line.
I think he's going to
discuss his question.
So we're going to have Roman on now.
Oh,
actually he just said his question was
answered correctly, so that's great.
Is there anyone else that was wanting
to add any thoughts to that question?
If I had anything around the safety
angle that we're trying to put on it...
Randall's very good at phrasing the
underlying question here about what's the
criteria we want.
There are a couple of ways in which
I've seen this sort of issue come up.
There's always the question
of philosophical zombies,
this notion that if you
emulate... Well, no,
actually philosophical zombies...
No matter what level you emulate,
have you actually sort of achieved
the goal of identity of consciousness,
sort of re-invigoration in
some metaphysical sense.
Of course the best way to just sort of
respond to that quickly is nobody knows
and let's just move on for the time being.
Now there is this other question,
and there are companies
that have done this.
I can't remember if
Randal touched on this.
There have been companies around for
several years now that are, sort of,
trying to get a foot in the door with
a sort of... What would you call it?
Sort of, my mind archival...
They don't even call it that,
but just this notion of let's capture as
rigorous a snapshot of a person's life
as possible and see if we can turn that
into something that we would call an
imperfectly preserved mind and identity.
There have been multiple
approaches to this.
I can't remember where I read this
article. There was some guy who...
He was a journalist, and he interacted
with a company that was doing this.
And their goal was to,
basically,
they would do several things.
They would run interviews with you and
try to sort of compile this sort of Chat
Bot.
They're trying to refine a chat bot that
responded increasingly like the person
does in an interview.
They would scrape your social
media profiles or webs,ites
or anything available
to try to build as complete a
picture as possible of a person.
And you can start seeing
where this is all going.
And then the larger question is,
is there a fundamental error in the
assumption that this is all predicated on?
Thank you Leslie.
I see that in the chat.
The Replica Chat Bot with a
K. Oh, Mind File, Jan. Yes,
I remember that name.
See, I have been aware of
these projects for so long,
Yeah,
that's the Terresam movement right?
I've forgotten. So it sounds like
we're all aware of us together.
So yes,
there are people who have asked the
question of whether you can achieve
something like mind and
identity preservation. And
then the secondary question,
whether you can actually achieve
any consciousness preservation.
With these very high level black box
approaches to what personhood means,
you basically take a turning
test approach to the whole thing.
Which is that a person is just what
you can get in and out through the IO
interfaces with the system.
So basically what can you put into it
through its eyes and what can you get out
of it through its mouth. And if the
patterns are sufficiently like the person,
then did we achieve our goal or not?
And I'm going to safely say I don't know.
But we've been trying to figure
this out for a while now.
I kind of go over this
somewhat in my book,
that Randal and I have kicked around
the idea of at what level do you perform
your system identification.
It's the term Randall
maybe very familiar with.
At what level would you cut your system
identification cutoff at and say,
if we don't system
identify at this low level,
whatever our threshold, we can
we choose, then we just say,
it just not good enough.
How do we set that threshold if you're
really achieving the same behavior
eventually?
So I don't have an answer to that,
but it's definitely a question that people
have been thinking about for awhile.
That's kind of all I have to say on that.
That was an excellent answer that went
into a lot of detail. Yeah, exactly.
It's about what is the good level of
separation between what you would call a
successful upload,
having captured what you care about,
and all the stuff underneath.
Yeah. Perfectly said.
Yeah.
And just going off of what
Keith said... So, yes.
We all want to have the simplest program
that can preserve the identity of a
person. And so one of the things that
we need is, as Keith just mentioned,
is consciousness.
So what is the simplest thing that
you can do to preserve consciousness?
And that goes back to this
idea of scale suppression.
So should I be able to simulate
or emulate all the ionic channels?
Or should I be able to simulate all
the neurons or the brain region?
So at what scale, and at what level
I can stop and then say, okay,
this is enough to recreate that
consciousness or recreate that mental
experience, that
phenomenal experience. So,
and I think we don't know yet,
but many neuroscientists may say
it's just at the neural level.
You need to emulate
your system,
your artificial brain at the normal
level to be able to recreate that
consciousness and preserve the personal
identity, but we don't know yet.
It was a great question,
but no one knows yet.
All right. Is there anyone that wants
to add anything else to that question?
I don't think so.
So our next question comes from Vitaly,
and I believe this is actually a question
that we've kind of addressed before in
previous workshops.
He asked,
"How detailed is the emulation of electric
chemical processes you intend to use?
Sub-threshold dendrite potentials,
are they taking into account?
Are features of spines taken into account?
What approaches are used to emulate
chemical synapses and neurotransmitter's?"
And one thing that I kind
of like to add to that,
as we experiment with whole brain
emulation and with going down to different
levels of detail,
and as Vitaly asked in his question,
are the insights that we gain from
that information going to help with
developing artificial intelligence.
I missed a bit but I got your question.
Okay.
Very nice...
I think you're having some
connection issues Randal.
I'm not sure if it's you,
but try again Randal.
Randal,
while you're figuring that out,
could you mute yourself and maybe we'll
have Abulfazl or Keith try to answer
that question.
Let's see. I was actually thinking Randall
was the guy for it. So the question is,
at what neuro-level do we want to attempt,
these brain emulations.
Of course no one has decided yet.
One of those popular answers to that
question, and one that I personally favor,
is to take that sort of input output
function I was describing earlier,
at the action potential level of neurons.
Which is to say that when you ask
about dendritic spines and such in your
question,
I am with reservation actually
writing off of that requirement.
I'm proposing that maybe the better level
is the one at which you build a system
that can propagate signals
through a network in a
pattern that is similar to the
way that a brain propagates action
potentials through a neuron connected
network.
So whether or not the synapses actually
have the exact same properties would be
irrelevant,
so long as the action potentials to
actually get through in a statistically
similar fashion. That's not to
say that that's the right level,
it's just one of the more
popular levels to go to.
I actually personally believe that that
can't possibly be the level we have to
go to because when people lose neurons,
from a variety of medical maladies,
at least at a low level,
it seems to have very little effect
on our health and our personality.
You can lose a couple of neurons here
and there and it just wouldn't make a
difference.
I personally suspect that something along
the lines of cortical columns or other
conglomerations of neurons that we
would have to decide what our grouping
criteria is,
but some sort of unit of groups of
neurons that performs a function and this
might be as large as thousands
of neurons like cortical columns,
is probably, sort of the Lego brick
we're looking for in all of this.
But again,
I don't really put a flag in the
stand very deeply on any of this.
We don't know. There are, of course...
I'll let somebody else speak in a second,
but just to quickly...
There are definitely people
who propose much lower levels.
Hameroff and Penrose are well known
for proposing the microtubule structure
inside of neurons as a critical
component of consciousness.
And then of course,
presumably of identity preservation and
everything else we're trying to get at.
People have taken this
as an interpretation that
they're against technologies
like mind uploading and such.
And I believe that they've been cornered
in the occasional interview on this and
it actually said,
no we're okay with it all,
but you do have to take into account
this quantum mechanical requirement that
Penrose and Hameroff are stipulating.
So many of us are not actually
on board with that requirement,
but it just says the same
thing. It says, okay,
well there is this level of perfection
inside of atoms that you've got to
replicate or else we're going to say
that the process has been successful.
So you can make this decision at several
levels of abstraction and surely the
right answer at this time in
history is we don't know yet.
And yet
just talking about Penrose and Hameroff
orchestrated objective reduction theory
and the whole bunch of other
quantum mind theories out there...
So these series of really interesting
getting consciousness to the molecular
level... However, at the time being,
researchers have not been able to show
a mechanistic description of how this
quantum effect,
or quantum phenomenon,
happens that can give rise to
consciousness. For example,
Penrose and Hameroff idea had a real
big push back from the neuroscience
community about the ability and how
it could work in the brain,
given its temperature and noisiness
and other conditions that exist in the
brain.
And I think as you mentioned,
probably at the nural level or at the
circuit level we would find good enough
information so that we could be able to
somehow recreate the mental phenomenon
on an artificial medium.
Okay.
Since we did say we're going to do,
Anders' interview at 1:30 Pacific,
we're going to go ahead and skip
our break and end the panel for now,
but we will address any further questions
from the audience members in Q and A
sections later.
So,
with that in mind,
we will commence with the
second part of our workshop,
which is the interview
with Dr. Andrews Sandberg.
He is with the future of humanity
institute at Oxford University.
This interview was conducted
by Dr. Randall Koene. And
I will just have to play it on mine.
Unless
Alan can play it. I'm not sure if
Allen is going to be playing it.
Allen, if you can, maybe we can give
yours a try and see how that one goes.
I'm queuing it up now.
Awesome.
There may be some noises
in the background here too,
eventually.
It shouldn't matter too much.
We can overpower them
with our own charisma.
Indeed. I'm not seeing it yet,
but I can hear it.
Okay.
So I'm going to introduce you first
and then we can get into our discussion
because we're going to be using
this in our event of course. Okay,
let me get started then. Our expert guest
for this event is Dr. Anders Sandberg
who also happens to be a longtime
friend going back at least to 2007,
and the first whole brain
emulation workshop that was
organized by the Future of
Humanity Institute at Oxford, possibly
even further than that. ...Dr.
Sandburg's work in
computational neuroscience,
which means we could geek out about
that at length if this were a private
fireside chat. Andrews is
a senior research fellow of
the Future of Humanity...
...of course,
act of future technologies and artificial
intelligence and whole brain emulation.
He's also an excellent speaker
and debater and has always,
I've really been looking forward to this
conversation with you. Welcome Anders.
Thanks for agreeing...
Okay.
I guess our backup option is not viable,
so we'll just go with our original option.
So anytime my microphone goes out,
I'll just be paying close attention to
our chat and I'll make sure to turn it
back on.
There
may be some noise in the
background here too, eventually.
It shouldn't matter too much.
Indeed.
Okay,
We can overpower them
with our own charisma.
so I'm going to introduce you first
and then we can get into our discussion
because we're going to be using
this in our event, of course. Okay,
let me get started then.
Our expert guest for this event is Dr
Anders Sandberg who also happens to be a
longtime friend going
back at least a 2007,
and the first whole emulation workshop
that was organized by the Future of
Humanity Institute at Oxford,
possibly even further than that.
Doctor Sandberg did his PhD work
in computational neuroscience,
which means we could geek
out about that at length,
if this were a private fireside chat.
Anders is a senior research fellow of the
Future of Humanity Institute at Oxford
University,
and of course he's a highly respected
in fields of philosophy that deal with
existential risk,
the impact of future technologies,
and artificial intelligence
and whole brain emulation.
He's also an excellent speaker
and debater and has always,
I've really been looking forward to this
conversation with you. Welcome Anders.
Thanks for agreeing to do this interview
and to try to join in on the Q and A on
the day of the event, even though you'll
still be returning from travel that day.
Thank you so much,
Randal. This is exciting.
It's good to be, not quite
here, but communicating.
Right. So, you're, of course,
familiar with the goals of
the Carboncopies Foundation
and I imagine that...
It's okay.
I mentioned that you can also see that
people who are working on whole brain
emulation and people at the Carboncopies
Foundation are also really interested
in artificial intelligence or
artificial general intelligence. Now,
we don't very often talk about that in
our workshops because they're already so
many groups out there that are doing that.
But we did feel that the overlap and the
interactions or possible interactions
between work towards whole brain emulation
and work on artificial intelligence
is something that hasn't really
received sufficient attention yet.
So if you don't mind,
I'm going to ask you a few questions
about your thoughts on AI and about risks
and benefits. And about this area
of interaction. Is that good?
That's sounds excellent.
So you've got a long history of dedicated
concern and ,supporting some serious
study on existential risk and in
particular AI risk and AI safety.
Could you tell us a little bit about how
your thoughts have evolved in that time
since you got started?
I vividly remember being on a train ride
to Jane University back in the early
2000's reading Nick Bostrom's recently
published paper on existential risk and I
had this knee jerk reaction. This
is preposterous. This is so stupid.
This is just going to be used to slow
down science and we need more science and
technology more rapidly.
Then gradually I became aware of...
Actually, that knee jerk reaction might
be a little bit too naive. Gradualy,
I warmed to idea that,
actually, existential risk
is a pretty dominant concern.
It is not a super important concern,
but certainly it matters a lot,
but the problem is of course just
being concerned is not enough.
You need to start picking apart the
probabilities, risks and uncertainties,
just try to see where can we do the most.
There are some risks that
we simply cannot budge.
Some super-volcanoes from the moment,
for example,
there is a finite risk that we
will all die of natural causes.
There's no point in worrying
too much about them.
The other risks are
relatively intractable,
but many people are always working
on them like nuclear war risk.
And then there are those
risks that are more tractable.
And at this point it gets interesting.
If they're neglected but tractable,
then you should probably
put in more effort.
Especially since even a small amount of
effort, if it's a very tractable risk,
can help a lot.
So to me,
future technologies are
interesting because they had
a fair bit of traceability
simply because we are making up
these technologies as we go along.
We're inventing and discovering things.
And that means that we can also regulate
them and safeguard in various ways if
we're careful, if we make the right
choices, which is not always possible.
And besides artificial
intelligence and related areas,
are there other examples that you would
say are areas where we could put in more
of an effort then we're doing today and
maybe not super-volcanoes or something
like that,
but other areas?
I think Bio risks are a good example.
So there's certainly a far bit of
people working on some the bio-security,
but many of the emerging bio-technologies
probably pose entirely new kinds of
risk and I think it would be a good idea
to make a serious inventory of what we
might find there and trying to preclude
some of those risks and that might
require the various forms of disease
surveillance and innovative methods of
actually stopping, and for
example, gene drives, et cetera.
And I think this might be
true for other technologies.
One good example is
actually quantum computers,
which are not an existential risk,
but the risk for encryption and privacy.
The solution is of course
quantum safe encryption,
which both MSA and Google
are trying to develop.
And then we need to promulgate it. Before
the quantum computers become too good.
We want to have it around for so long
that our secrets are kind of safe because
the ones that can be cracked
by the quantum computers,
well they're primary points.
Hmm. It's interesting that you mentioned
that because I was just out at South by
Southwest and there was a startup
company presenting there. Unfortunately,
I didn't catch their talk and
I don't remember their name,
but they were explicitly saying that
they offer quantum encryption as a way to
safeguard your data. Even now, I
don't know how they're doing it,
but there out there.
So one interesting thing is that you can,
of course,
encrypt data using quantum encryption
over our communication slide.
And it's not as easy as people thought,
especially the Chinese have been working
very audaciously on developing this.
But then you also want to have a
quantum safe encryption method,
and that might actually not involve
any magical quantum computing,
but it might be good for your branding
if it has quantum in the name anyway,
And I think this is going to become more
and more urgent.
It's
interesting that we're going
down this path right now,
because that's a great segway
into something I wanted
to talk about where Dr Ben
Hurtsel was concerned.
He's a mutual friend of ours
and a researcher in artificial
general intelligence.
And he sees things a little bit
differently than, for example,
Nick Bostrom does or Eli Asrey does.
He wrote a paper in 2005,
so it's a few years old now,
and the paper was called
Super-intelligence: Fears,
Promises and Potentials.
And it seems,
one of the things that he explicitly
talks about is that the probability of
different risks isn't addressed very
well and it's very hard for us to judge
which risk is really the one that we
should be paying attention to the most.
And then,
for example,
is it a case where other risks are so
great that the potential benefit of
artificial general intelligence in helping
us prevent those risks that might be
on balance,
more important than say,
worrying too much about the
existential risk of AGI itself.
So I think you're probably
familiar with his thoughts on that.
And I was wondering how you
would balance those approaches.
So that argument is a bit similar to an
argument I heard for us trying to send
messages to extraterrestrial intelligence.
Again, we are under so much risk here.
So even though maybe there is some risk
of getting out friendly aliens on the
phone, if we get friendly as
that might actually save us.
Now if one is really desperate,
I think that kind of Hail Mary
strategy really might make sense.
But I don't think we're
necessarily that desperate yet.
We got some years until we get the
artificial general intelligence,
maybe quite a lot of years,
and we actually have a decent
track record of surviving so far,
which might of course not indicate that
risks are small but at least that we can
do something about it. So I do think
that we should do this judgment,
but there is quite a lot of things
we can do about that. So for example,
demonstrating that on the net AGI
is going to be better than no Agi,
well I think that is fairly doable.
On the other hand,
showing that safe AGI is much
easier than the dangerous AGI,
I think that is going to
be tough to demonstrate.
I think you are not going to be
able to prove that within the rigor.
Okay. But it is still kind of hand wavy,
isn't it? Because you said for instance,
that you don't think we're very desperate
in terms of any of those other risks
but desperate in terms of the
risk of dangerous AGI. Right?
That's the same question.
The thing about... Right now when we
look at the existential risk to humanity,
I think that nuclear war still isn't the
top one and you can even make a basic
estimation of naively we have
seen 73 years of no nuclear war.
So if you apply in a data function,
et Cetera,
you end up with between 0.1 and 1%
risk per year, which is disconcerting.
Maybe a nuclear war is
not an existential risk,
but it's still a certain amount of
probability that is kind of worrisome.
Now arguing that nuclear
war risk is important work,
but it also takes a lot of effort.
You want a lot of diplomats.
You want a lot of people in these
studies to do their job. Now,
AI risk is relatively mobile right now
because we're still at a very early stage.
A few insights might actually
change the risk profile quite a bit.
So what I'm arguing is not so much that
we know that unfriendly super-like...
I know some people who actually think
this should be regarded as a default
position. I'm not entirely convinced
by that argument. They might be right,
they might not be right.
But I can certainly see that by moving
AI safety research earlier in time,
we can reduce the risk.
And probably do quite a bit of
useful risk reduction that way.
That makes sense. Yeah. I mean,
if we simply ignore that there was any
risk at all and didn't have any people
interested in AI safety,
that would definitely open up a bigger
potential for other worst case scenarios.
I'm sort of curious about both
of those things. Actually,
in the worst case scenario
and what to do about it;
because now we've spoken very abstractly
about the risk from AI and we've also
spoken in a very abstract sense about,
well we should do something about that.
I'm kind of curious,
what do you think is the worst case
scenario that should be expected if say an
unfriendly AI or AGI is developed and
what do you think is the best way to try
to prevent this? So, if we do have
people interested in AI Safety,
which route do you think
is the most promising?
I do think...
...Because I think most of us,
we can imagine...
We can start imagining ways around this
and I think we could actually have at
least some chance of reducing the impact.
So I would expect the worst case scenario
to be very bad and totally not looking
like anything I could formulate.
So you could imagine a scenario like,
well,
actually a rapidly and self
enhancing intelligence is possible.
So there are some systems that generate
that you get a fairly hard takeoff of
with some random utility function and
then it goes off and does something random
but instrumentally its very,
very competent so it does everything it
needs to prevent us from ever being able
to stop it.
So now basically you're in the same
universe as a force that you cannot by
definition stop and he's going to do
something that's likely very dramatic,
like taking over earth to use for
atoms or doing something else.
It's very hard to say.
Now I think it's very likely that we're
going to run into other bad AI problems
long before this.
So in many ways the worst possible
scenario is that AI works really well and
then improves on it and
it's worked really well,
and it all keeps on going super well...
On the other hand,
I think it's more realistic
to assume that we develop AI,
we do stupid mistakes that fix that,
we fix those,
we keep on doing things,
we discover new problems.
This keeps on iterating and happening.
And in the nice scenario,
we robustly figure out ways of actually
controlling AI, setting up motivations,
and other kinds of safeguards
we currently don't understand.
Then we're getting into truly dangerous
territory. The more scary scenario,
of course,
is the one where we don't learn anything,
but we create incentives to use it.
So companies that don't use AI to direct
where activity or going to the doing
badly on stock market.
So everybody will be using them and then
the systems get more and more powerful
and the world gets crazier and crazier,
but if you don't use AI,
the craziness cross receive.
And then you ended up in a world
that is not suited for humans yet,
nobody has won very much. Now
the ideal scenario, is of course,
that we figure out useful
things early on and we fix them.
And in the best of all possible worlds,
maybe is some simple mathematical
theorem that you can prove,
that gives you a safe AI.
And it's also politically very easy to
convince everybody that we should be
implementing this.
In a more plausible world.
It's going to probably be a
whole little science of safety.
Just like we have the same safety in a
lot of other domains and this is going to
be fairly messy.
Let's think about computer security
and think about how we're failing at
computer security today because
of the incentives are all wrong,
the software companies are not held
liable for the security of their products
yet we buy them.
And the end result is that we've have
built an entire infrastructure on systems
that we can't fully trust that in fact
have enormous glaring security holes,
we can all know that,
and we all choose to ignore it because
living without a modern smart phone is
not practical.
Yeah. Motivations and
holding people accountable,
holding companies accountable
is really important.
So I liked that you went into this,
what is a more plausible scenario and,
sort of, looked at these possibilities.
I mean,
it's obviously very difficult to really
predict anything because it's so many
different routes that are possible.
It's very complex to look into the future.
But,
that takes us a little bit away from
just focusing on something assuming that
the major AGI or AI out there is always
going to be something that has a fixed
utility function and maximizes
that utility function,
that's where the danger comes from.
Where as the reality might
be more messy than that.
Which I think is also something
that Ben touched on his paper.
I do think this shows the importance of
trying different approaches to AI safety.
So,
while our friends at MIRI are taking
a fairly axiomatic approach in their
thinking in terms of
their fixed structures.
Around FHI we have some people who are
working very much on learning systems and
I'm hoping in the future we're going to
see even more approaches because right
now we fully don't understand what the
domain is and we might actually want to
combine different approaches or figure
out what are the sub two problems with
them. And this even includes the
people who work on near term AI safety.
Quite a lot of them are of course not
terribly fond of talking about AGI and
super-intelligence.
They think that they have enough trouble
with their autonomous cars or how to
manage drones well or even get them
out of the bias of the credit scoring
systems.
But I do think there is actually a
continuum here from their near term issues
about controlling complex
adaptive technological systems
to the long term issues
when these systems become essentially
autonomous, and super powerful,
how do we ensure that we already have
enough of what counts as an important part
of control?
Yeah. We could talk about
this forever, basically,
because every single thing that
we say brings up another question.
When we say timelines,
how long term problems versus near
term problems, how long is long term,
all that sort of thing.
I'm going to try to steer this towards
the whole brain emulation topic.
So when I discuss whole brain
emulation in the context of AI,
when I talked to someone about that,
I typically get two very
different kinds of reactions.
Some people were giving me
the reaction where they say,
I think that whole brain emulation is
an important potential safeguard against
humanity suffering the worst
case scenario from a runaway AGI.
And others will say,
well,
that may be true that there is something
to be found in terms of long-term
future benefit for humanity.
But at the same time,
working on whole brain emulation carries
its risks in terms of either whole
brain emulation itself being a risk
or a whole brain emulation somehow
accelerating the development of
potentially dangerous AI. So I,
I'd like to start talking about that
and ask you a few questions about it.
Now I've already talked with Jaan Taleen.
He gave an interview as well,
which is being presented in the same
event and he was cautiously optimistic.
He felt that whole brain emulation
research would probably end up being a net
positive in terms of existential risk.
And he said that he was basing his opinion
largely on what he learned from Carl
Schulman who also at the FHI.
And so clearly we should probably reach
out to Carl in our follow up to this
event. But I was wondering
because he mentioned that,
what do you think about the level of
study that's been done so far on this
particular problem with the interaction
between AI Safety and research on whole
brain emulation?
And do you think there are specific people
who are the leading thinkers in that
sort of overlap area that
we should be talking to?
Right now, I don't think there
has been enough study on this.
There have been informal arguments and
in many cases we see formally arguments
go back and forth and a bit
based on personal taste.
Maybe one has invested a lot of effort
in brain emulation then that is automatic
to bias in towards thinking
this is probably pretty safe.
I think the core question is how much
would the progress in neuroscience push
AGI and vice versa.
So one possible argument would be if we
do brain emulation its not just that we
by definition will have
a lot of computing power,
but we will also have good
ways of scanning brains,
that directly doesn't help AGI much,
but also a good way of modeling and
running, we scan the neural tissues.
And that's going to,
long before we get to...
...Give us quite a lot of...
The audio has been muted,
Mallory.
...did neuroscientists figure out things
by having actually a really good neural
model to start with.
And how quickly do they
tell AI researchers who can
actually make use of it.
So I think empirically the answer seems
to be that it actually takes a fair bit
of time before neuroscientists
understand the neural issues,
even if they have good simulations.
Because it's very complicated and we're
not terribly good at figuring it out.
As a formal computational neuroscientist,
I'm kind of embarrassed by how little
progress we've been making despite our
models becoming much bigger.
But the reason is of course we
need better ways of understanding.
Now brain emulation is not necessarily
based on having a super profound
understanding of the high level
stuff that would drive AGI,
but the most of the scientific interest
is going to be about that rather than
achieving brain emulation.
And even when neuroscientists have
found something interesting like,
how would the reinforcement
learning systems,
of the brain works or some of
the neural representations,
it seems to take a fairly long while
before the AI research is picking up.
So if you take,
for example,
Russell and Norway's book on AI,
we relive
through the chapter about the history of
the field of insights that are driven.
You find a long list of interesting
neuroscience insights and then you try to
think,
how many decades it was between people
figuring it out and it became a part of
an AI project and quite
often it's several decades.
So if things stay the same,
I wouldn't be too worried because we
might be doing the brain emulations and
then a few decades later, the
neuroscientists get the memo. However,
that might change in the future.
It might turn out that maybe AI research
and neuroscience meld together much
more. It might be that actually
people like the Google deep-mind,
started by people coming from
Diane's group in neuroscience,
have learned how to read neuroscience
papers and actually get useful ideas from
them.
With my...
... another example might
be that maybe the way the...
...organizes information and does online
learning, that is really a simple thing,
once you understand it,
and then you can apply it in AI.
I don't know how to assign
much credence to this,
but I think it's worth investigating much
more deeply and we need probably need
to develop tools to do that.
Yeah. Oh Wow. You covered a
lot of ground there and yeah,
this sort of echoes my
thinking about it as well.
I do think that there are
definitely things that we
can still learn from the way
the brain works compared to how AI works.
Say for instance,
how do you manage to learn,
a large variety of things in various
contexts and do so from very few examples
compared to what, say, you pump into
a deep learning network these days.
Those are the sorts of things where
there's still a lot that we can learn.
But it's not entirely obvious how
you discover that in the brain.
And as you said,
it's not clear how fast
can neuroscientists convert
that into something AI could
use. And for instance, this is
a point where Jaan mentioned,
he thought that AI could probably
search the space of probable models and
probable methods better and faster than,
say neuroscientists would be able to
interpret what's going on in the brain.
So if you need to solve some issue with
reinforcement learning or something like
that,
it would probably be faster to
improve that just pure AI development.
I'm not sure that's true, because
some problems explode very quickly.
Numbers are big very fast.
And then when you say we're gonna explore
all possible models or something like
that,
if you don't have a good strategy,
you could be stuck until
the end of the universe.
One of the really interesting things
though is that hyper parameter searches in
machine learning,
have become much better recently.
People had some useful insights on
machine learning methods to improve other
machine learning methods. And I think we
should expect at least that to continue.
That's not necessarily searching
this enormously large landscape.
It's just maybe 10 or 20
dimensions of hyper parameters,
but that's already a pretty
high dimensional space.
So it might be useful to just keep
an eye on this and how good does this
generalize to other domains including
trying to find models in other domains.
If you start seeing the modern automatic
model making to really take off,
then we should expect things
to get rather explosive.
Indeed.
Yeah,
but that's also where it becomes much
more complex than just predicting a system
that will use a simple algorithm to
improve its utility function maximization
over time.
Because if you're building completely
different modules that work in a different
way, then you have different algorithms
and everything becomes a lot more
unpredictable.
...perspective. It might turn
out to be tools to do this too.
So I think Eric Drexler has made a
very good point in his big report about
reframing a bit of AI Safety.
We quite often think about the AI or
the neural network is some what self
contained systems.
But actually the part of a practice of
generating things that solve various
problems and most people in business
anyway don't care about AGI,
they just want to solve
a practical problem.
So we develop in pipelines and
services that generate these things.
Again,
that is worth watching especially since
they can be applied to brain emulation,
then we might get a pipeline towards
better brain emulation When we get a
pipeline for decoding neural tissue
and figuring out things about that,
then we should expect at
least a boost in neuroscience
Indeed,
and what we really should be talking about
is an ecosystem of various AI instead
of the single AI that's going
somewhere. Yeah, indeed.
So now maybe we can run through a few
very specific points that get brought up
because these are just things where I
often get an opinion from someone or a
statement, but there isn't necessarily
that much material behind it.
And I wonder if maybe there's a way
to slowly push towards a more precise
understanding of those
particular concerns or questions.
So the first one I'd like to get into
is the idea of having a brain computer
interface, a high bandwidth,
brain computer interface,
something that is the target of research
and also of some for profit companies.
I'm not going to go into mentioning names,
but we probably know who they are.
And sometimes you hear from
them statements such as if only,
if we can have a high bandwidth
connection between the human brain and the
machine,
then first of all,
that will help us avoid AI safety
problems because somehow it will be more
tightly connected with that.
There'll be some symbiotic relationship
between us and secondly we'll be able to
benefit very strongly from this connection
it will be sort of a part of that
advancement. Have you thought
about this claim regarding to, now,
brain computer interfaces?
We're not yet talking about
neural prosthesis in this case.
Yeah.
So I vividly remember a few years back
when Elon Musk had spouted off something
that was general face palming
around the office. We like Elon,
but many people thought that sounded
stupid and I have a big mouth.
I said maybe there is something to the
argument so then my manager gave me two
weeks to try to make a
steel man version of Org.
That's the opposite of Straw man. Let's
see if we can make this argument work.
And I found that the first version
of the argument that saying,
if we get brain computer interfaces,
we're going to get enhanced and then
we're going to be smarter than AI,
that seems pretty unlikely there are real
problems in making an enhancement that
makes you super intelligent.
Especially if you need to do that
within a relatively short time frame.
We might not know how far it is
until we get superintendent AI,
but the time it takes to test out anything
that deals with meta human biology
anything that's medical and then try to
develop user interfaces and enhancements,
that seems to be something that
potentially could take quite a long time.
There is another argument and that
is if you can't beat them, join them.
So I get my brain computer interface.
I'm linked to the AI and the this way I'm
going to be on the winning side except
of course the nature of that link and
what we mean by winning side is quite
tricky by some accountants.
My mind is already linked to my cell
phone because of extended mind hypothesis,
part of my mind recites in my
calendar and other applications.
I don't have certain memories in my head.
They're in my smartphone.
So maybe I'm already linked to machines.
And now the dangerous paperclip AI takes
over the world and turns it all into
paperclips. In some sense I was on the
winning side because I had a smartphone,
but this doesn't sound like winning.
This sounds very stupid In fact.
The reason is of course what...
...and we want that link to be the
right kind of link. And in some ways,
it's a very deep philosophical mess.
So this doesn't seem very
likely to work a priority,
but there is something else that
actually looked at a bit promising.
If I'm linked to an AGI system and it
can observe my an evaluation situations,
it can actually estimate a bit what I
would have done if it's trying to do what
I would have done instead,
this actually gives us a good chance.
So the system might melt...
You know where this is going right?
Yeah.
If it can predict everything you're doing.
It has an emulation of your brain,
right?
Well not necessarily.
It might my value system but nothing else.
So the naive version of
it might for example,
notice that I liked helping little old
ladies cross the street and I don't think
kicking them in front of
a car is a good thing.
If it's just optimizing for that,
then it might for example,
run away from little old ladies
because it wants to avoid them,
and avoid helping them, et cetera.
You need more rich information.
And this is where things get interesting.
If it could just get an
entire copy of my brain,
then we could definitely
get the information over.
But even my values need to contain some
useful things to do value alignment. So,
for example,
if you follow up the causal reasons,
why do I like little old ladies
crossing the street safely?
Why do I dislike kicking
people in front of cars,
now we might learn something about at
least my morality that's actually quite
useful. There are two things
here, of course, maybe one,
shouldn't just use my own mind because
I'm definitely morally pretty fluent.
Maybe we actually want to make sure
that either we take a certified same or
perhaps even better,
have a lot of different people who are
in the right give input in order to do
general value alignment.
And second, of course,
it's slightly tricky to get this to work
really well because neural interfaces
are tricky. But if I just
say no robot, don't do that.
Whereas if I smile at the robot,
that's already sending a signal.
Maybe I don't need the neural,
interface at all.
The neural interface is just cool because
it might access my orbital frontal
cortex and give some actual evaluation
information even if I can't express it.
Yeah. These are all very good comments.
And I liked that you said, okay,
just even if you don't think about
emulation and we're just talking about
something earlier,
like being able to understand your values,
your value parameters that can be helpful.
And of course I immediately started
thinking about the deep learning networks
that learned to be racist because
they were learning from us.
That's where you get into that whole
value alignment issue and what are the
right values, and should you
even care about the right values?
Are we supposed to tinker with that when
you're dealing with particular person
or invading your freedoms
and stuff like that.
It gets very complicated.
But I also think people
tend to jump straight to the
assumption that we can talk
about a system where you have a strong
integration of the machine and the brain;
where you have a connection that somehow
is tying very closely into what this
machine is doing.
And you mentioned the cell phone.
And I think it's a perfect example
for trying to think through what this
actually means.
Because when we say you have a high
bandwidth connection between the brain and
the computer,
the computers still can operate at
microseconds and even smaller timescales,
whereas the neurons can't.
So in this link they're
talking at different speeds
and that is very similar to
what we're seeing with the cell phone
and ourselves. With the cell phone,
it has a different language,
not just a different speed.
It has a completely different type of
language. It's using of a microprocessor.
...we have a neuro-brain and we work
at a slower pace from neuron to neuron,
although we have a lot of them.
And we communicate in a very
particular way with this cell phone.
We can only press buttons or swipe,
that sort of thing.
So there's this communication delay or
communication constraint built on this.
And then you can think about, okay, so
how has working with these cell phones,
how has that affected the
development of cell phones?
This is sort of an analog for how does
it affect the development of AI and
potential AI safety, right? Or how has
working with the cell phones affected us.
Now we can say a lot about
how it's affected us and,
and some people think there are some good
things there and some bad things there,
probably a mix.
But it also clearly has affected how
cell phones are developed because cell
phones are always developed to try
to fit into say these communication
limitations due to work well with how
we interact as interface development,
o present information to us in a certain
way that makes it easily graspable.
We want interfaces that can tell
us quickly what we're looking for.
But of course that also
constraints with features, right?
The interfaces that are simple and
fast or not interfaces that are highly
configurable and very
adaptable and all that.
So we've got trade offs and
this clearly has an effect.
I think it's a great example and I
think maybe thinking more about how we
already interact with
machines in that way,
is is a way to try to
imagine what this would do,
building a better or faster or applied
interface with machines in some sense.
So ideally of course, if you want an
interface, it should be plug and play,
it should work straight away.
The problem is that probably means
that it needs to be very pared down.
And if you look at the general design
philosophy coming out of apple,
it has always been actually cut away
possibilities and just leave the things
that people actually will want to do.
Hide as mush possible below the hood
or maybe even make the hood the closed,
you can't actually get onto
it. Meanwhile, of course,
if you're using any of the lineups,
or UNIX systems everything is accessible,
but well, reading the manual
and learning how to use it,
it's getting very complicated. Once
you know it, it's very powerful.
But that takes a long time and
most people don't want to do that.
So it might be that the transformative
neural interfaces would be the ones that
actually require a lot of learning.
So the most fascinating,
neural interface I know of is of course
Miguel Nicolelis attempts at brain
that's connect brains to other brains.
As a test of this idea,
it seems like if this works, I want
to see independent replication of it,
but it looks like the brains
can adapt to each other,
and learn how to decode each
other's signals somehow.
That's awesome if it's correct.
But it also seems to take a lot of time
and I don't want to have a system that's
not working for months and months,
but yet I need to train myself to use it.
So it might very well be that the neural
interfaces with computers at first,
they're going to be useful for people
who are desperately in need of it,
strongly motivated and the people who
have the mindset of train themselves.
But most applications are probably
not going to be that deep.
So instead what you want to do is
something that interprets enough of our
signals and use a lot of smarts from
the outside to try to figure out what we
meant and the fact that our minds are
relatively slow compared to the machines,
it's not necessarily a bad thing. It might
be that we have a slow process in, say,
in fast processes and as long as we
get enough feedback in our own pace,
we can actually control that.
This is a bit like our
constitution governance state.
The constitution is
not changing that much.
It's not supposed to be
updatable at a very high pace.
Below that level you have the laws
which are updated in a faster pace while
local regulation and norms can
be updated faster and faster.
So you might have systems with different
levels of flexibility linked together.
And if it's actually working well,
then you can have a slow
system setting the agenda,
the fast systems implementing it.
And so feedback system checking
that everything is working.
In practice...
Sorry,
I hate to interrupt,
but before you were talking about the
degree to which someone could say that
they are a part of the winning side.
They're part of the whole thing,
if they're integrated,
if they're connected with machines.
Now we're talking about a slow system and
a fast system and that the slow system
is somehow still governing and
supervising what's going on.
And it's interesting to compare that with
what's going on in the brain already,
where, for instance, we have
systems that are slower.
Our conscious awareness is definitely
something that progresses at a different
pace and in different chunks,
then all of that subconscious processing
that's going on in parallel throughout
our cortex.
And similarly,
when we're trying to think
something through logically,
and we're trying to make up the steps,
that's a very different and
slower process than the other one.
And that in fact is the process that
the machines can do very quickly.
So it'd be interesting to
see what happens there.
But it's interesting to then look at how
integrated those really are and what's
really going on there.
So for instance,
we tend to think that our conscious
awareness, that our conscious self,
is in control.
We tend to think that that's the part
that is really the boss and we're
governing everything that's going on.
And yet you can show experimentally that
you have already decided something long
before you know that you've decided
something because those sub-cortical
processes and really doing that deciding
work, of course, this makes sense, right?
And the rest is just a
reflection to yourself.
It's just an observation about
what's been going on in your brain.
And now we can imagine that
something similar is the case.
If you integrate the human brain with
the machines who are operating at a
completely different speed,
they live in a world of
nanoseconds that we can't live in.
They live in a world where
they can use things like,
quantum computers or simulated annealing
or something like that to come up with
ideas and solutions that we
can't think of in our brain.
It seems like one of those situations
where ultimately you may have a feeling,
a sense of control without
actually having it.
So I think this is a very important point
and generally I think most people say,
yeah, I want to avoid. There is a...
My impression of the current state of
play is that a fair number of the things
we're doing are indeed illusionary.
But our top level of the
consciousness of mind,
can give up the two and intervene
not very quickly but we can...
Is it the veto,
or is it other drives that are informing
your consciousness to then give the
veto?
That is not a good philosophical question.
I need to check what the
philosophers are saying about that,
but I do think it makes sense that
you have a top level but it's actually
getting information from
most of our systems.
The systems I really have a problem with
is the ones that never had an output on
the conscious level or never tell the
rest of the systems what we're doing.
The real ignorant of secret
societies of the grid.
I don't know how many there are or
whether they play an important role in
general the brains job connected
everything a little bit to average else.
It's just about the degree of control
and the degree of information is not as
intense as one would expect.
Once you started realizing how diffusely
connected are different parts of our
mind or you start to feeling a bit
like a cloud rather than a person.
Yeah, the illusion of control and
whether or not that's a bad thing,
that there's an illusion of control.
That's really I think is core to many
of these questions that we're talking
about today because when we first started
talking we talked about AI risk and we
talked a little bit about what do we
imagine that if worst case scenario,
and the best case scenario is.
Now most of the time people tend to
think that the worst case scenario has
something to do with a situation where
humans are completely not in control and
where they have no say over their
future and perhaps they are even somehow
eliminated.
Now when we're talking about situations
where control is mostly an illusion,
but there are aspects of it where we're
tied in and say there's this veto power
that again,
perhaps isn't real because you know,
freewill is such may not exist at all,
but there's still something that we value.
So really when you talk about the danger
of AI and what it can do to humanity,
we should be talking about what are the
values that we're trying to preserve.
What would we call a successful outcome?
Which bits do we want to preserve and why?
And then what sort of scenarios can
incorporate that in some scenarios may
include a lot of change where a lot of
things that machines will be able to do
and where they develop into becomes a
part of what human society is. Right?
And so I'm reminded with some of I&M
Bank's culture locals were very super
intelligent machines
play a role in the plot.
And in many cases they manipulate
protagonists in various ways,
more or less often.
And the interesting thing is that in
many ways these manipulations can be in
many cases they are fairly benign the,
because there's still a respect that
the protagonist can do differently,
but he's unlikely to want to do it
differently because it's a really good
argument that the situation
is contrived in a certain way.
Would we say that this is a bad situation?
Well,
as long as it's actually
a fairly well intention,
the entity that actually has our own
best interest and some other good goals,
taking that into account,
that might be all right.
If I have a veto power but never exert it,
I might still be pretty happy about
that because I might have my illusion of
control. But it still is
important for me to function.
Normally we won't have a sense
of control because we're agents.
We survive by moving around on the
earth surface and manipulative it,
and when we can't act,
we're super vulnerable.
When we're in a situation
where action is not possible,
we are at the mercy of everything else.
And that evolution has made us
really desire to be the actor.
Intellectually,
we might sometimes realize that I should
listen to a smarter person and do what
he or she tells me because
they actually know better.
But my little inner two
year old will say no,
I would do the opposite just because.
Then of course my,
inner 20 year old will try to get the
two year old to listen to reason and then
there's a lot of debate and meanwhile,
of course maybe a really smart person
with manipulate me because we know that I
have this internal dialogue.
But a good outcome would be that you think
that intentions of transparent enough
that we can kind of see
that these seem to point
towards a future that is desire.
Oh Gosh,
you just opened up whole new can of worms,
where we find good and desirable.
But I just wanted to point out that
you now have also again come across
something that Jann Taleen mentioned.
He mentioned that he felt that a good
future is one with as much optionality as
possible,
which of course is our desire for control.
It's the desiring to have
as many different paths as
we can take as possible.
But you mentioned that sense of the
illusion of control and possibly being
important to us.
And then I was reminded of how many
people say that what they desire is
happiness. And then when we
think of a future with happiness,
then you get into these
wire-heading ideas,
where you can just make
someone feel happy,
even if there was really nothing
has changed about their situation.
And in the same way it could be possible
that we could give ourselves the
illusion of control when
we have zero control.
So again,
it doesn't immediately solve anything,
but if we think through what the real
values that we're trying to preserve are,
you have to think about very basic
things like the value of say,
something that we consider really human
continuing for some reason because we
think that very human thing is important
or has made the universe a better place
than it was without humans in it.
This gets really deep philosophically.
So it comes up in a lot of
existential risk research too,
because maybe if we nuke ourselves and
the cockroaches becoming intelligent,
maybe they will have a much better
civilization then us. In that case,
maybe we should say, oh, that was
a happy ending for the universe.
And of course some value
theories would say, yeah,
just integrate the total amount
of happiness and compare them.
Now others will say,
wait a minute,
how certain are we that happiness
is what should the maximized?
There might be other things,
we might be deep thinking or
paperclips or what have you.
So we also have to take more
of uncertainty into account.
The nature of what is good about humans,
our uniqueness,
I think many people have
an intuition about it.
So there is a lot of things one can do
here and right now I don't think we have
any complete theory
about what the good is.
Obviously philosophers have been working
on this for a few thousand years.
Some progress has been modest. We, I
think we had learned important things,
but it's pretty clear that the list of
things that could be the ultimate good or
the list of arguments why that mabe
this list will never be complete is
awkwardly wide.
But it's worth noticing that even if it
takes something as simple as happiness,
people are quite bad at
identifying what makes them happy.
And even when they know this
action would make me happy,
they might quite often not take that one.
So there is one of the
scariest papers I ever read,
The Trends in Cognitive
Sciences from a few years back,
which was just reviewing these results
and to me somebody who is kind of
politically liberal.
This was horrifying because if people
don't even do what makes them happy
themselves, giving them the
option of doing that, okay,
now that doesn't necessarily
work out that well.
Now I still think that most
forms of control actually
have very bad effects too.
But it shows that it's tricky.
Our feelings of what makes life better
might not correspond to actually making
life better partially because our feelings
are evolved things that made sense of
African Savanna and actually
the life we're living now,
not to mention in the future
might be so radically different,
that we need to develop
new kinds of feelings.
Yeah, that makes sense. Yeah.
So we went down a really deep
rabbit hole there for a moment,
and I don't want this
to go on for too long,
but I'd still like to quickly touch on
a few remaining questions about whole
brain emulation and AI.
So I would have to talk about the idea
that whole brain emulation itself can be
a danger to humanity.
And so I don't know exactly what
people mean by that when they say it,
but I think...
I can give us some ideas.
So I wrote a paper a few years
back together with Peter Eckersley,
who back then was with the electronic
frontier foundation now is at the
partnership for AI.
So we were looking at could you get
risk out to having brain emulation?
I think the most primary one would be
if you have software people who suddenly
have a radical break with a lot of
traditional views on human nature.
You will have some people who actually
have religious reasons to say these are
not true people. They would
have potentially economicaly
disruptive effects.
If we look at the old ideas
in the Robin Hansel's,
the H of M that it seems like getting
a brain emulation breakthrough might be
dramatically disruptive and you get
entities that are in some sense a natural
other so you can very easily tell the
story how you could get just a conflict
where some people think brain
emulated people are not people,
they're evil and taking
our jobs and so on,
so you could get another natural conflict.
That still I think not an argument
against doing brain emulation but rather
making sure we'll use improved tolerance.
It also of course has the idea that if
you get a brain emulation is going to be
self improving so fast,
but now you get some form of hard takeoff.
Before you know it,
you have all the problems of
the normal hard takeoff in AI.
How do you see that possibility?
By the way,
how does a whole brain
emulation do a fast takeoff?
I personally don't believe
this is particularly likely.
So the normal way of arguing
it would be something like,
well it's running on a very fast computer.
The emulated person can do a lot of
neuroscience experiments on her brain,
that you can run a lot of copies that
test out things to improve various
capabilities.
And before you know it,
you have some kind of post human who
might now really start doing weird things
depending on their value system.
I'm not too worried about
one of the person doing this.
But you could imagine a society...
So at some point,
I think it was cultural man who
was arguing with Robin Hanson,
well that brain emulation society he's
describing in great vividness in this
book, it's not going to last forever
because people are going to update the and
compete and change it to other things.
So in real time it's not
going to last very long.
And they kind of agree that maybe two
weeks. That's still a lot of history...
I don't think we have to
get into that necessarily.
I think Robin's book with beautiful,
but I think it has a prediction
for things that will happen.
It has its limitations for instance,
because he left out AI
as being a part of it.
So many of the things that he
has whole brain emulations doing,
are things that just a simple AI program
could be doing and that changes a lot
about that whole scenario.
And you can imagine that they develop the
simple AI sooner or later in the brain
emulated world and suddenly you get the
economic transformation that might be
just as disruptive for the emulated
people as well as the biological people.
That scenario would rather be that you
get very rapid cultural evolution which
might produce actually extremely
different kinds of entities.
Some people again would say,
yeah,
maybe we're going to be some weird post
human and they don't care about us meat
humans and that naturally
leads to conflict. I think
a more plausible risk is,
yeah. This unleashes dramatic
changes in our domain,
but a lot of people will not have access
to and will be too slow to react to.
And that at the very least
look like existential risk.
And it might indeed be some form of
existential risk that develops into
something that lacks those
values that are really humane.
This will get us back to
the same problem again,
which is what is extent existential risk?
What do we call a bad
outcome for humanity?
So let's say that whole brain emulation
leads to a new Cambrian explosion of
different kinds of minds who develop
themselves in various different ways,
not just minds but bodies as well,
et cetera.
And some of them do better than others.
So there's an evolutionary aspect to it
and you can see change happening in a
way that's a bit faster than it is today.
And when I say a bit,
I might just be understating that.
But if that happens,
the question still remains is that bad
because maybe this is the best possible
outcome for whatever we consider valuable
about humanity because it's going to
spread everywhere very quickly, even
if it changes. So does speed matter?
It doesn't matter if you go from a to z
from by going Abcd all the way to z and
doing that in steps or it just as
good to jump from a to z right away.
If the outcome is exactly the same.
If let's say,
just as an argument for just the question
of does the speed of change matter,
is it important that that
rate is slow for some reason?
At least in the case of a parasite delta?
Do people typically think
that the speed does matter?
If I change day by day to become a more
mature person with different political
views, et cetera. Okay. But
I might be fine with that,
but not that one day I wake up and my
personality and political views have
changed in something.
Now it seems philosophical that there
is something problematic going on there.
Because I agree with you.
Why should we care about those steps?
And part of that is of
course error correction.
We might want to be able to go back to,
or note this thing.
There should be time to realizing
what that might be doing. Oh dear,
I know what I was voting for.
I should do to something change myself.
But I think there is something
interesting about that Cambrian explosion
situation to me that sounds lovely.
I think it would be great to have this
open in the future where we get a lot of
humanities and they are all drawing
of course on originated from the same
original species and might retain
different aspects of us. However,
you can also make a very scary scenario
where there are things that we care
about evolve of it.
Nick Bostrom has this very disturbing
paper about some post human futures that
essentially devoid of value. So in
one of them, the mindless outsources,
basically people exist in an uploaded
form and a or outsourcing more and more of
the brains.
And in the end there are no minds anymore.
It's just an economy growing endlessly,
but there is nobody at home.
So the interesting thing here is
where does the value come from?
And this is of course
again the deep rabbit hole,
but I think it's worth recognizing that
rabbit hole and kind of orbit around it
at a safe distance. Out of that
as we normally do, we think, oh,
the conscious and sentient seems to
matter. Probably if consciousness is lost,
then we've probably
lost all of the battle.
I'm willing to argue that
maybe that's not all there is.
Maybe there are unconscious systems
that are valuable lives too.
But that gets me into a big argument
with the philosophy department.
We can also say that some of our human
nature might be valuable just because it
is a human nature. We might not care
about some famous paintings in a museum,
but they're famous paintings
that we should preserve
because they're part of our
process just as well as some of
the horrors of our historical past.
We should never forget about them,
not just because we want to
learn not to do it again.
But even that we think that
if we were to forget about it,
all the people that died
in that particular horror
would in some sense are loss
of it.
There is a fair bit of interesting
arguments in philosophy about this.
There is a book here by so
Summer Schaeffler Why Worry
About Future Generations,
which I found very fond because he's
taking a non-consequentialist view about
why we should care
about humanity's future.
And I find that really weird because I'm
typically just thinking about what are
the consequences. And it points out
that you can actually love humanity.
We can actually see there are things
about humanity and its nature,
that we love works in all and
the we want this to go on.
So I think there's going to be a thick
bundle in that black hole of what really
makes life worth living but matters.
And it might be that we
need to hedge our bets.
We don't really know whether it is our
human-ness or our consciousness or the
fact that we originated on this planet
or just that we experience happiness.
Maybe we want to have it all we want that
Cambrian explosion because it gives us
a good chance of having..
I would love to have a future that has
both the brain emulation people and
biological people as an existential
risk backup. If it was just one of them,
I think we would actually be in big risk
and we might have missed out on true
values.
It might turn out,
that just like consciousness might have
added a profound new value to the entire
universe. That might be the next
step as a conciousnes prime,
which is even more important.
It's unfortunate.
No kind of matter in the
current universe got that.
You need to be maybe a brain emulation
or come quantum computer or whatever to
actually experiancing his study.
And then to get on the other
side of this and say, yeah,
back then of course those
poor Precambrian humans...
They didn't have this thing.
And it might not even be,
it might not even be something
like the next level of conscious.
It could be something
completely separate from that.
Something that we also think it's super
valuable but has nothing to do with
consciousness.
Yeah. But to some degree, expanding
the circle to encompass bigger,
bigger sets seem to be a good idea.
So one of my common amalies...
...they might say, well, given the
capabilities of those post apes,
they call them humans, they could get a
lot of bananas. And the apes would say,
yeah, that's great bananas
are good. And indeed,
we are really good at getting bananas.
We can get more bananas than any ape
could imagine by planting them and doing
other things.
It's just that we're getting constantly
sidetracked from the pursuit of bananas
and sex,
by also doing philosophy and technology
and politics and art and sports.
And we would try to say to these apes,
well actually you don't know
what you're missing out on.
Actually science is pretty awesome
and post humans or these quantum
consciousness, prime whatevers,
they might say, Oh yes,
philosophy and science they're so cute,
there a very nice part of life.
We of course got these other things
that you can't even imagine that's okay,
that we do like our bananas.
The thing is,
if we retained some course and
they might be somewhat silly,
things like bananas that might also be a
continuity that means we care about our
future. I think one of the
big fears people have is
what you get this disruption.
There is humans post humans and
there is no similarity whatsoever.
This is part of the fear we have of AGI
because we might make something that's
utterly dissimilar from us and we actually
don't have any way of telling them
about what values we have and the like.
When it comes to brain emulation,
I always felt that we are much closer.
We at least starting out very similar.
Then might go off in different directions,
but I think we have a good start.
We start with whole brain emulation.
We know that whatever you've got there
begins with something that is very human.
And so whatever these values
are that we care about,
they're probably contained
in there somewhere.
Although it'd be great if we can
explore the value thing. Not right now,
but if that gets explored further and,
we can't make a prediction here.
Are we going to solve the problem of what
are the values we want to preserve in
five years, 10 years?
Where's our deadline here?
Yeah. In some sense, doing
philosophy with a deadline.
I think that's a new thing
that Nick Mastroni draws.
And I think it's important to realize
that there are some deep philosophical
problems that maybe we can't solve,
but I think we should try to do practical
progress on the handling them because
they're actually becoming action relevant.
And in some cases that
progress might just be okay,
we need to do more hedging because we
will not figure it out before the time's
up. But, well, that trade,
the moral hedging are impossible
things so they can be quite powerful.
Indeed.
So I've kept you on this now for quite
a long time so it's probably time for me
to sort of bring it to a close.
But before we do,
and I have a concluding question for you,
but even before I ask
my concluding question,
I wanted to know if you think that there
was something important or relevant
that we haven't touched upon,
haven't covered that you think
should be mentioned here.
I think the core issue I have is,
we have been talking about many things
that have been around the community for a
long while. Some of it we have
written a bit of papers about,
but it's very little. I think
there's more work to be done there.
Some of that might be done by
neuroscientists and philosophers.
I think there's quite a lot of room
for interesting joint work here.
Trying to turn this a
little bit more rigorous,
maybe measure what does the flow of
ideas between different sciences that's
interesting on its own. You can make
a good academic paper out of it,
but it might also tell us a little
bit about the safety issues.
I think we need to work a lot more on,
can we do safety for
neuro-morphic computing.
Right now a lot of safety work tends to
assume is platonic computer's running a
little black boxes on white boards.
And that might be good to prove some
things about what we might want to figure
out other ways of doing AI
safety and machine safety. So,
there's lots of things to work on here.
All these questions about values and
what values we're trying to preserve and
therefore what a good outcome is;
that's really part of that, isn't it?
Yeah. And some of that of course
is traditional philosophy.
There is even a branch of ethics
axiology the philosophy of value itself.
But I think also we have found that there
are orbital frontal cortex does have
value representations and we're actually
figuring out non trivial things.
Both from addiction research and the
scanning of brains when we make choices
about how we do evaluations
and how we function.
I think there's a lot of room
to do empirical research here,
but might actually give a good kick to
the traditional philosophy to see it in
new ways.
I think we have seen a rekindling of
interest in ethics in many parts of
philosophy because of AI Safety research
because it actually forces philosophers
to try to formulate,
can I make my ethical systems something
that could run on a computer and most
traditional systems are
absolutely impossible and to
the first to the final will
have to acomodate computation deamnds.
And then we can start asking,
so what is a moral system that actually
can run on a brain or a computer and
that waters the
philosophical import of that.
So you do think that at some point we
will have answers to all these questions
or do you think it's just going to be a
situation where we become progressively
more aware of what all the detailed
questions are and we just kind of lay out
all the questions and
we sort of swim in them?
Well philosophy is interesting because
many scientists try to advance,
we try to move forward, but in philosophy
tends to try to dig deeper instead.
And sometimes that's the right choice.
But in many practical case we want
to move forward. I have in my,
I'm writing a book about the longterm
future and I actually right now I have a
very loose calculation. Given that
we haven't solved these problems yet.
What's the likely time until philosophy
solves them if we believe there is a
Pareto distribution of problem difficulty
and they ended up with something like
maybe it's going to take 10 to the 20
philosopher years before we figure out
what it's all about. That's alot. It might
be faster if we have better computers,
but it might turn out that actually
it's going to be far in the future when
finalism super-super-post-human
kind of goes, Oh,
that's kinda makes sense.
Isn't it interesting how the Hitchhiker's
guide to the galaxy already predicted
this?
42?
Yeah. Again, it made a very good point.
You need to understand the question
really well then the answer is typically
quite easy.
So one of the interesting things about
this interplay between philosophy and
science and practical engineering is
that quite a lot of time you get these
practical issues that forced you to
refine your deep philosophical question.
I think we're going to find eventually,
that axiology probably get old because
we actually do a lot of neuroscience and
then that forces philosophers
to think in a different way.
Okay, awesome. So concluding
this my last question,
what do you think that, I mean, this is
again an impossibly complex question,
but what do you think that humanity should
be doing or how should it change how
it approaches problems compared with
what we're doing today to maximize our
long-term chances for both
survival and thriving as a society.
So I do think we need to find ways
of aiming at an open ended future,
but that doesn't necessarily
mean everything is possible.
We might want to cut off
disastrous possibilities.
We want to avoid existential risk.
And in order to do that,
we probably need better tools
for insight and coordination.
So right now it might seem that we
are in this world of fake news and
international chaos.
We're doing really badly about it,
but compared to a century
ago or two centuries,
the United Nations is a
really hopeless organization.
But two centuries ago it was unthinkable
what getting all nations together
without weapons and actually agreeing
on at least a few things in 200 years we
might actually be able to do something
much better. Similarly, yeah,
we have a lot of problems with fake news,
but we also got things like Wikipedia,
which is kind of an early stab at
creating a common knowledge base.
Again, that was the first step. And this
is probably not going to be the last,
or the best.
So I do think that we can work quite
hard on improving this insight and
coordination of things.
Better filtering mechanisms of
separating true and false information.
Which includes of course science, the
replication crisis in a specialist iconic,
demonstrates that we need to work quite
a bit on building a better engine of
scientific knowledge. But even the
problems we have in the data mine,
the scientific papers and tried to run
that to do medical diagnosis turns out
into trouble because
many papers are so bad.
We need better ways of doing this and
those tools are useful in a lot of
domains. We might want to have a better
chance of tracking our own future.
We might want to find ways of
having these courses at work better.
It's a gradual thing and there's going
to be alot of struggle to get there.
Similarly,
the coordination in the long run,
we need to coordinate before we start
scattering across the universe have become
too incomprehensible for each other.
And too far course removed before that
we might actually need to have a big
meeting and decide,
okay,
this is what we're going to
do longterm with the universe.
This is how we go to hedge our moral bet.
This is how we're going to move the
galaxies and making sure that it stays in
the right.
Is that a meeting the FHI
is going to have in 2021?
Let's hope for something like that.
So the origional ideas Will Macaskill
talks about the long reflection,
but maybe once we got
to interact together,
we should just sit down and as a species
and maybe spend 1000 years debating
what to do just to get it really right.
Right now,
this sounds absurdly utopia.
This sounds totally crazy.
But then again, so did the
United Nations once upon a time.
So we should start working on making
the tools to make the tools to make the
tools to make good insights and
good decision making. And of course,
making sure that we don't
go extinct while doing this.
We should never let the perfect
be the enemy of the good.
That can be difficult sometimes.
On a more personal note.
Given all of the things we just talked
about and there's so much more of course.
How do you deal with the fact that this
is such an overwhelming pile of stuff?
I mean,
how do you tackle that or just like go
at it and still feel that there is good
purpose and moving forward every day.
So the problem is not finding purpose
because there are so many interesting and
important things to work on.
It's rather how do I choose the
most important thing to focus on?
I'm lousy at that.
I'm basically an academic magpie.
I see something shiny and I go for it.
So over the past one and a half years,
I've been somewhat focused because I'd
be working on my book about what I called
Grand Futures. And it's broad enough
that it actually allows me to,
whenever I get distracted,
I just write another part of the book.
That is one way of doing it. I have
colleagues who are way more focused.
They sit down and they think
very careful about what matters.
Spend even more times checking that
it really matters and then they start
working on it and I think we can again
like hedging bets you hedge yourself by
having people give the
different strategies and then
sometimes we compare notes
and see, oh, that seems to be
something I should be doing.
Okay. That's really good advice. Yeah,
there are some different strategies there.
Okay, fantastic. Thank you very much.
Fantastic conversation. I have
to say. Yeah, like always.
Well we should keep it up,
make sure that the future continues to
have great conversations until the end of
time.
Oh yeah.
And you'll be welcomed
back at the Carboncopies
Foundation again and again for
sure.
Thank you so much. Yeah. Cheers!
Okay.
Stop the recording now.
Not hearing you at the moment.
I don't know if you can hear me.
You are audible,
Randall Mallory is
not,
That was really awesome
to hear Anders talk again.
He's always a fantastic speaker.
No matter what the context.
I wish we had a way to
really do claps online.
Maybe we need a canned
clapping or something.
And then whenever appropriate we'll turn
that down to a slow clap or something.
We're downloading a clap track now.
Okay.
I'm just checking.
Okay. Maybe then Mallory
has a more serious problem
with her mic at the moment.
So many technical things
because everything depends
on so many little bits and
pieces.
Is the Bluetooth working,
are the batteries still charged
and all that sort of stuff.
And in my case, the Internet
drops out every once in a while,
so that's not great either.
We have heard from Anders
recently about 27 minutes ago,
he said that he was 30 minutes out,
so maybe he'll be here in three minutes
if we're lucky or maybe it takes a
little bit longer to get off of his bus
and get into his house and get all set
up.
That's all possible.
So in the meantime,
we can start addressing some of the
questions that came in a while...
Oh, here's Anders. Then we
don't even need a meantime.
We can just go straight at it.
Hello Andrers. You must have
just come in the door and
sat down basically, right?
Yup. Literally. Yup.
Just ran in from the bus,
then got home.
Okay, well Kudos to you
for being able to do that.
We all just listened to the interview
and as usual you're fantastic.
The way you present everything
is so clear because unlike me,
where I go into abstracts very quickly,
you tend to bring everything back down
to examples that people can really get
into and dig into.
Like,
AI that is helping a little old ladies
across the street and things like that.
So that's perfect and I think we also
got a bunch of good questions here and I
see Mallory just poped up again.
So where did we leave off after I joined?
Are we going to start
with our first question?
You can go ahead with whichever
question you want. Anders is here.
All right.
And just to remind everybody,
you can ask your questions in the YouTube
live stream chat or you can call into
call.carboncopies.org or
the number (415) 455-3017.
The first question that we
have is from Roman Sitlou,
even today there are some people who
are both capable of recursive self
improvement,
like adopting the new mind enhancing
technology and have strong misanthropic
tendencies.
Basically in a way their
biological non-friendly AGIs
what can we do about this
scenario?
I think we actually have a surprising
amount of good tools at our disposal.
After all,
one reason we behave ourselves is because
our mothers told us various things
that we should be doing and
that's a foremost social software.
We have other forms of social software
we get when the get enculturated as well
as police reputation various force of
market solutions and many of these things
constrain us to a tremendous degree.
Then of course if you try to do recursive
self improvement through enhancement
today,
it's quite limited.
No amount of smart drugs or meditation
and or getting the best apps are going to
make you tremendously more effective
than just somebody who is really smart or
specialized.
And you're not going to exactly be deep
blue either without having a computer on
your side.
But the deep part of the question is how
do we handle when it becomes easier to
make our full individual minds?
And I think in general we do
that by having a lot of minds,
but they partialy constrain each other
and systems of minds because generaly
systems of minds are much more
powerful than individuals.
So to just take a little parable here.
We have the earliest youth conciousness
interesting AI Boxing experiments.
Where we practically demonstrate
actually convincing people to let out the
possibly on friendly AI out
of a box was pretty doable.
If you're smart enough and Glib enough,
you can talk your way out.
This is impressive except why don't
prisoners do this all the time?
Why don't we see more prisoners
talking the way out of prison?
And the answer is of course,
well quite often we talk their
way past one or the prison wards,
but there are others,
there is a system or a career.
So being prison chief and the warden,
that all dependand on prisoners
not talking their way out too much.
And that actually makes actual prisons
surprisingly effective in keeping even
the most clever prisoners in.
Not perfect, but quite well.
So I do think we can do the same
kind of system and system management.
It's not like we want to
only upload the saints,
so we don't even have a good
definition of sainthood,
and maybe the saints won't stay saints
once they're in the computer either.
But you don't want to have just one mind.
Great. Does anyone want to add to that
or shall we move onto the next question?
I think Anders pretty much answered the
whole question unless you wanted to dig
into the detailing that Roman
did when he pointed out,
people who want to self improve may
already feel that they are perfectly fine
and eliminating a huge part of humanity
in the future or something like that.
If you could restate that properly.
Or Mallory,
sorry, maybe I didn't make myself
clear. I thought you could maybe...
Or I could just get at what I
think that question in emboldens.
So really ruthless sociopaths and people
think that the world would be better
off with just me around
certainly psychological exist.
The vast majority of them are not very
good at doing anything because by virtue
of being so misanthropic and sociopathic
may rarely can acquire the resources.
I think the intuition we have is, oh,
what if they take a technological means,
that pill that just makes
you essentially god like,
at that point we will be in
trouble if it goes there.
The problem is they're probably
not first in the queue.
The real problem might of course still
be the scenario where you have successful
sociopaths,
successful misanthropes or who just ran
into a lot of resources and they will be
getting power.
But again,
the question is why aren't particular
billionaires wiping out everybody else and
running the kingdom? Well, to some
degree they still need other people.
But even there you can actually replace
people with consultancies, et cetera.
The actual problem is of course societies
in order to remain rich and functional
are quite complex.
You can't actually run all the parts.
So the people who just introduce
in every wealth and power,
they are relatively sane and my
scorch society in other ways.
So the only case that we
need to be worried about is
really misanthropic person
who gets very powerful.
But I do think the scaling of
power isn't in favor of us.
Once upon a time Metachip family,
they were wealthier than essentially
the principality they were run.
They were rivaling the nations
in European of wealth and power,
but meanwhile they weren't perfect.
But the United States is,
harder for rich people order
organizations or people like state,
actualy it has become weaker.
So I'm much more worried about
misanthropic and crazy states and the
institutions then individual
misanthropic people.
I think that answers the
question pretty well.
The next question that
we have is from Jaysa RC.
What do you think about the human habit
of comparing ourselves to the technology
we create for ourselves or our tendency
to humanize things even if they're a
robot or an artificial intelligence?
Yeah, so one of the main
reasons people make robots,
is of course they actually like
to use humanize technology.
If you think about much of the
Japanese mindset about robots,
it's important to have a social
relationship to technology.
And certainly they also believe that
we have social relationships even to
non-human technology.
But if it has a face and it's easy to
interact with it's more manageable.
And I think that this gets
to something important.
We humans have...
This sometimes leads us to think
about things in the wrong way.
When lightning strikes,
we wonder who is angry and come up with
a story about somebody must be angry
with somebody or some
reasons to course lightning.
But on the other hand we use
this to do intuitive physics.
So it's very natural for us to try to
make these kind of machines and put in a
lot of anthropology modification,
into them.
And we also of course tend to assume that
if something with a face or talks back
to us must be part of the human world.
Which is why it's so
easy to have a chat bot.
And this is why people fail in
the Turing test all the time.
But then there's opposite thing.
By trying to make artificial intelligence,
it's also a mirror. It's a great way
of trying to understand ourselves,
what we are,
what we mean by thinking and feeling
and what aspects we care about,
but also what we fear a lot of worries
about artificial intelligence or our own
projections.
But what if we made something like us?
And I think that's the
first level of worry.
I'm not so concerned about that myself
because I think the second that being
told them there alien and
hence conspiracy theorists.
But you should look miss the
fact that they are deeply human.
Great.
Does anyone want to add to that?
Yeah,
just a quick note,
which is this point that you made at
the end about more worried about things
that are more alien.
This kind of comes back to that bit about,
worrying about corporations or something
like that that are very powerful
because in a sense,
these are also alien because they have
different way of making decisions.
They're way more focused on what
we might call utility function,
which is the bottom line and the
quarterly results and that sort of thing.
So they can be very sociopathic in that
sense and very powerful at the same
time.
So I think you're absolutely right to
focus there first before worrying about
the individual person in a sense.
So yeah, that kind of jumped back,
but I guess that's okay. Oh, quickly
before someone else answers something.
I just want to point out that while there
are lots of good questions coming in
and there's way we're going to be able
to address every question in the time we
have on the panel.
We are collecting all these questions
and as we see good questions there,
we're still going to
work on them later on.
You may find them in transcripts
or another work that we do.
You can still contact us about them
through questions@carboncopies.org and we
can get back to you.
I think I can move on to
the next question now.
Our next question is from
Jan Clok or Yon Clok.
and he says,
"While initial collaboration may be nice,
competition like the US versus China,"
and I'm assuming he's referring to our
competition with artificial intelligence.
That's probably the most prominent one,
"is that competition is going to stay?
How do we utilize that for whole
brain emulation and artificial general
intelligence development
and keep it safe?"
So the first thing worth mentioning is
of course many of assumptions that we
are,
let's say US versus China in a competitive
race might actually be because humans
like to project their human moods and
feelings onto these alien things like
nations.
We are fairly competitive and you can
totally think of international politics as
kind of the school yard
bully going around.
And we quite often construct it that way
because it allows us to think about it
in the easy ways and of course a lot of
people involved actually like to think
about.
It's just that in practice governments
or disjointed entities with different
goals and different parts of government.
And you can imagine for example a
competitive situation on the artificial
inteligence in China and the US,
that involves total sharing of safety
information and I think that is something
we should be pushing very strongly for.
Even if people think that this is a
strategic technology and it might even be
important to be first,
it should also be a no brainer that it's
also important to have a world to be
first in. So maybe we should be also
shairing as much as possible on the safety
technology.
This actually happened during the Cold
War between the US and the Soviet Union,
where the US deliberately shared some
technologies for keeping nuclear weapons
safer,
simply because that would be in the
interest of everybody to have it.
It didn't always work out
as diplomatic as it should,
but it was definitely a good try.
When you want to use competition
to do something good,
typically it is this concept of creative
destruction or let a thousand flowers
bloom and so on that you allow a lot of
diversity and then that competition will
hopefully bring forward interesting things
that you can note is interesting and
useful that can be replicated.
And this is of course typically why
market economists do so well when compared
to a planed economy because they can
find new solutions rather than trying to
optimize for an obvious one.
But when it comes to research,
this is extra important because in most
fields the amount of progress you make
is a very complex curve of
amount of efforts you put in.
So in the earlier era,
you need to get a lot of rapid progress,
but you don't know what you need to
work on. You need to map out the field,
you need to do a lot of experimentation.
But you can also learn a lot of things.
Eventually it's time to bring out
the big super computers and the big
collaborations. But first we
need to solve a local small step.
And that's a great thing to actually
have dispersed between not just different
teams, but also different
mindsets and approaches.
If everybody tries to achieve brain
emulation or AI in the exact same way,
the probability of it succeeding
goes down quite a bit,
which may be good for safety work in
the background. But by the same token,
we want to have a pretty big diversity
of safety mechanisms explored before we
notice which one seems promising enough
that we should be starting to pile on
and boost to their abilities.
Wow.
I think first of all,
hat it's worth pointing out that what
the person who was asking the question is
talking about,
it may also be something a little deeper.
It could be just,
not specifically about how
people working on AI should work?
Should they compete or
should they cooperate?
But also just this fact
that the competition is
sort of a natural thing that
just emerges from there being differences
between things and then the selection
mechanisms that apply. So you see
that in evolution too, of course.
And in evolution you have differences
in DNA through some genetic mutation and
then one of them works
out better than the other.
And that's a kind of competition, even if
it wasn't intended to be a competition.
And the same could be true,
you could say that even for a single AI
that is trying to improve itself as it's
trying out different algorithms,
those algorithms are competing on
some kind of performance criteria.
So there's a competition going on there.
So you're always going to have within
the ecosystem of all the things that are
going on,
you're going to have both competition
and cooperation because cooperation is
often a strategy that helps,
that enhances what any single thing
can accomplish. They, for instance,
the cooperation between different
algorithms to have two different ways to
detect people in a scene instead of just
one way to detect people in a scene.
So you've got both collaboration and
cooperation going on at both times.
And then what Yan was asking.
It could be interpreted as,
since there's always going to be both
collaboration and cooperation and it's
really hard to weed out and to say we
should all be working collaboratively on
the same AI here or something like that.
How can you ever make sure
that's going to be safe?
Because any actor who happens to be
slightly less cooperative may get an edge
because of that. Right. And I was
going to go further with this,
but I think I've already brought
enough new stuff in here. Anders,
do you want to say something about that?
I'm reminded of something that Eric
Drexler rote somewhere in one of his old
papers,
I think it was his algorics papers that
he wrote somewhere in the late eighties,
and he pointed out that we often talk
about how wonderful it is in nature with
all this simbyosis and all
of his harmonious cycles.
While human business, oh,
it's a red in tooth and claw.
It's all about competition.
But he pointed out that actually the
reason that we hear about the crimes and
when people betray each other in politics
or in business is that it's rare.
It's very unlikely that you get
robbed by gunpoint by your grocer.
Most of the time you will just to
get groceries and pay for them.
Most economic transactions
or extremely cooperative.
We tend to notice a difference.
Similarly nature,
actually an awful lot of
interactions are fairly adversary.
And then we get very in the rosy about
all these wonderful cooperation we
notice. So many systems and
different styles and biases.
Now the reason human economies work
so well from a comparative standpoint,
is that we have a lot of
tools to enable us to do that.
It's much easier to negotiate if
you've got the brain and the language.
It's way harder for the different plants
and animals in the forest and find some
nice equilibrium.
I do think the real question is can you
then set up the rules or the agent so we
naturally generally tend to be cold.
So the rewards in academia reward you
for trying to be first with a cool paper
that the,
it's a competitive thing,
but you also need to tell everybody how
you did it and everybody gets to comment
on it and that you actually develop
this cooperative, effective.
True it enduces some sense competition,
but it also helps truthful
and useful papers get ahead.
And that we might say,
well maybe we should set up the incentives
for AI and brain emulation research
so we do this.
Right now it might be that a brain
emulation research is mostly an academic
pursuit subject to the same quirks,
but normal academic competition and
the corporation are, for example,
a lot of the competition
is relatively weak,
but you also don't have strong incentives
actually get something that works
because you can always write more papers
about approaching something that works
well in the industry.
Making something that works,
it's actually a really good
idea because you make money,
otherwise your company's
not going to be around.
So we might want to
investigate this more deeply.
I'm not a good enough economist,
but I think there's this area of mechanism
construction that people have done in
game theory that would be very valuable
here to try to see could we find some
new ways of setting up corporate and
collaborations or competitive races.
The Human Genome Project after all was
leisurely developing technology and
getting somewhere, but at a slow
pace and then Ann Craig went there,
stormed in and forced everybody to rush
because of a competitive impulse but
also actually pushed things to build in
that phase where you want to get results.
And then we took the genomes and most
of the genomes are not put in public
databases and shared etc. The real
competitive thing is figuring out the
meanings of them.
So you can also have layers of competition
and collaboration and we might want
to look at at what layers do we think
is the biggest risk of a bad accident
would be a headache.
Let's see if we can make sure that one
is strongly in their corporate team,
which of course includes also, yeah.
Other people thinking about, oh,
that guy is trying to mess up.
And the good thing we got going here,
we're going to have to ally against,
which is another decent
incenive for being alies.
There's an interesting way
that this bridges way back
to almost the beginning of
the workshop where one of our callers
Leslie Seymour was asking about security
protocols and we kind of got to the point
where there was some agreement around,
yes there's a reason why everyone in the
world coalesces around open standards
and around, using the same
open standards and protocols.
Because that way you get a
lot of people looking at it,
making sure that all the holes get
patched and all that sort of thing.
But then at the same time, you were
pointing out that it can be really useful,
especially in these sorts of cases to
have a lot of different approaches.
Try out a lot of different
things and come up with new,
better approaches than used to be there.
You have to break out of
the box from time to time.
And so it begins to seem like it's
very handy to have competition inside a
sandbox where you have your peer
review and then ultimately to have
collaboration and cooperation
and standards outside of
the sandbox whenever you
can accomplish that.
Yeah.
I completely agree and being
good at designing sandboxes.
I think that is a fantastic
thing to work towards.
So Mallory,
did you want me to go ahead and ask a
question that I had written down or did
you want to go with another one of those?
I see that you've got another
by Abulfazl for instance,
and there's one by Justice.
Yes. So we do have one from
Justice. I'm going to ask that one.
"If brain emulations were developed as
open source software and posted on the
Internet, ala Bitcoin. What hope is
there to effectively control them?"
That depends very much on what
resources it takes to run it.
So you might look at the current
debate about open sourcing AI tools.
So if Google amounts a very nice AI
algorithm or at least a machine learning
algorithm today,
in many ways they're not losing much
of a competitive advantage because you
typically need a Google data center and
an enormous amount of data to train it.
So even though we amatures and academics
can sit around and read algorithms and
try to tweak something, we can never
turn it into a production system.
It wouldn't surprise me at all.
But the actual code for
running a brain emulation,
is going to be a somewhat messy
computation neuroscience system,
whether it's with a virtual reality
system and a lot of data management,
nothing too weird.
Everybody can download it from Git Hub.
It might also be that we can download a
few preacher in neural network for from
some mice,
some monkey and maybe some human volunteer
who is very willing to be everywhere.
But that doesn't mean that you get the
full power of being able to scan my brain
for example because scanning a brain is
going to require hardware of some kind
and quite often also the knowledge
on how to use that hardware,
which is sometimes shockingly
tricky to transfer.
Anybody who's been hanging around the
experimental biology has kind of noticed
that they're very skills that some
people seem to have and others don't and
generally of course in science we try
to figure out ways of automating that so
we get rid of it.
But there is still always,
especially the early
days of MIA Technology,
a lot of requirements there.
I do think that open source is useful
for checking code and improving code,
but it doesn't give you the
capability necessarily to round it.
If we use it to its fullest,
but this might depend of course quite
a lot on which scenario we have.
If we end up with a scenario where the
code is relatively easy but getting
computer power hard,
you get monopolies of computer power and
it might also be that we start up with
small simulations and
get better and better,
but that means that many people can
tool around with it a little bit crude
simulation on the computer we get the
better societal debate. On the other hand,
if we have more overhang scenario where
the code has been around forever and
then finally somebody does a bug fix
and now it works with those pre in and
scan, brains that have been staying
around in Git Hub for decades,
then things get really weird.
Then you have not just an overhanging
maybe of hardware and the scans,
but also the overhang
off the accessible code.
I don't think it's necessarily that
likely, but it's worth looking into.
Another thing that Peter did in our paper
Risks of Brain Emulation was to worry
about computer security aspects.
If brain emulations are bad at security.
If it's easy to hack your
server running various minds,
then we have an enormous problem.
So we might want to think about how do
we make that code so it's really robust.
This is why we might want security on the
code and make sure it's actually quite
open source and people have been
spending a lot of time finding the flaws.
All right,
thank you for that.
Next we have a question from actually
one of our panelists. Abulfazl.
He asks,
can you elaborate on the
idea that uploaded minds or
whole brain emulations are
not as dangerous as AGI or artificial
general intelligence is if that's the
case,
why is that?
So this is still the intuition which
is based partially on that distinction
between what about human
like versus alien entities.
So there's this cliche idea about the
robot uprising as the robot fields
oppressed, they rise up against
the oppressors, the humans.
And that's after all what the human
would do in the same situation.
Feeling oppressed and desiring
freedom is a very human thing.
Which is why humans love reading
stories about robot rebellion.
Of course,
if you actually get that kind of robot
rebelion your have basically made humans
tin cans and first of all you probably
should give them rights and second you
have succeeded very well
in making a relatively safe
AI because you can probably
reason with them.
You could imagine however the serious
problems that happened when you have
something much more alien,
and this is of course the
line of reasoning that has
emerged from the work of
Aleas Ukofski and Nick,
but actually in general if you
try to optimize for something,
you willoptimize for that thing
and ignore most other parameters.
This is a favorite
argument by Stuart Russell,
but if you make very intelligent systems,
so the powerful optimizers and
tell them to make one thing,
all the other values in the world,
even if they are somewhat similar to
our mind will fall to the wayside.
This makes him quite dangerous.
And then you can elaborate of course in
all sorts of interesting cases of how
this might be that is driving a fair
bit of the research on how to make safe
super intelegence.
Now the intuition is of course if you
managed to make a brain emulation you
basically got the original person.
If everything worked out well with the
same quirks and flaws and moral failings,
and moral virtues that's original.
From one the hand that mind might not be
saintly but it's certainly not going to
turn the universe into paperclips just
because somebody asks them to make
paperclips. After awhile you
can say, I'm tired of this.
I want to develop my personal
capacity or I want to sleep.
Or actually I really think now a
robot rebellion would be suitable.
You're basically having
to deal with a human.
We can certainly channel humans to
build organizations that are scary and
dangerous, but it seems like we
don't get those inhuman falures.
We don't get the papercliper that just
mindlessly constructs paperclips in a
paper clipping system.
We don't get systems optimized
for something totally valueless.
Even if you enhance a human a lot,
so they become more of a human,
and I think has gone really wrong
with some human values and goals.
But there is this general idea
in the study of the Ai Safety.
Human values are fairly
complex and fragile,
so it's not enough to get most of the
values in because if we've got about
something on the list,
whether that is forgiveness
or love or vocational nature,
is nice you get the world that gets
optimized for everything but important
things and it's actually quite dystopia.
Now the question is,
is this argument right?
Does this actually mean that
brain emulations are safer
than AGI and that is of
course where people's views
go in different directions.
It might be what leads to relatively
quickly to normal AI because we learn
clever tools from neuroscience.
It might be that brain emulations could
rapidly evolve into something else.
It might also be that our
tools for keeping humans in
line work really well for
them.
So that might benefit safety,
but it might also be that we can prove
safetyin other ways for more purely
artificial system,
we can morally do things to their minds
and the science that you could not
really morally do with brain emulation.
So in that case,
maybe you want to prefer it.
So in the end,
I think we need to actually write papers
and analyze with more detail. I have,
my probrain emulation view,
but I'm not certain it's right.
We need to investigate this much more
deeply and find ways of formalizing these
arguments.
Okay. I don't know if anyone
has a comeback right away.
But otherwise I can go
ahead and ask my question,
which I think relates to this in a sense.
And it relates in the way that,
well you just described,
for instance,
an example of why whole brain emulations
may be safer because you're basically
dealing with a person.
They're not going to want to paperclip
the universe because they want to do
something else.
He'll get bored after a while.
But then I could come up with a scenario
where I say, well, what if, what if,
this is in Robin Hanson is universe,
you're making whole brain emulations
as your basic tool and you want to
paperclip very well.
And so you may change,
or you may have gotten permission
from the original owner of that brain,
let's say,
to make a little change so that the
whole brain emulation no longer has that
much of volition of their own and might
just want to carry on making paperclips
and all of a sudden you can come
up with a dangerous scenario.
And this leads into my question,
which is really when we talk like this,
it seems pretty easy,
and I've heard this happen all the time
from everyone who talks about AI safety
basically,
whether it's Nick Bostrom or
Elon Musk or Eliezer or anyone,
it seems very easy for
every proposed system,
security or otherwise every scheme to
come up with some totally theoretical
scenario where this breaks down and
that makes it seem like AI safety is
possibly a lost cause because
no matter how you look at it,
you know there's always going to be some
system that escapes. And that's the end.
So in other words,
it seems like the probability of a given
scenario seems very important to assess.
And how would you do that?
That is a very good point.
And I totally agree.
Trying to kind of get
perfection never works.
In actual computer security and actual
security engineering people think about
swiss cheese solutions.
You have a number of layers of security
but they have holes in them like swiss
cheese.
But these are not aligned.
The probability of something getting
through holes in all the layers is low
enough.
And what can of course sort of arguing
here about how low probability of a
disaster is acceptable.
I think that's an important guide point.
But we need...
So some risks I think are
easier to formalize than others.
So when you think about an
artificial intelligence,
you might be able to prove things
about that if it's a really nicely well
defined one. And at least put some
bounds if you have a nice theory,
let's say of machine learning
reinforcement learning or what it's doing,
that might give some
possibilities to bounding it.
But it still sounds to me very much
like what we would love to do with the
computer science department and maybe
it wouldn't actually correspond to
anything realistic in terms
of actual safety checking.
So we might want to think here
about safety testing, methods.
There is a fair bit of that for actual
software and the actual industrial
systems.
They're not perfect,
but we might want to actually go and
loot that literature for tools and start
applying them to our own thinking.
My general feeling is this is going
to work well for technical stuff.
I think we can do code audits and
estimation of likelihood of certain neuron
networks doing certain things, but on
the other hand you have the human side.
What's the probability of somebody
agreeing to have a brain edited by weird
actors?
That's probably pretty high.
I don't think we need to even try to
estimate that there's always somebody,
but what's the probability of us
actors doing certain risky things?
Now we have a much harder time.
There I think we need to apply a fair
bit of judgment and essentially world
knowledge.
It's going to be way more uncertain.
I think we have still some evidence
that we can do not to bad probability
estimates in some domains
when we get feedback,
you have Tetlock superforecasters for
example they're interested in policy and
world events.
They're reading the newspaper,
they're discussing with each other and if
you put them together in the right way,
you actually get surprisingly good
for cost compared to the average one.
Could we do the same thing
as super safety forecasting?
I'd never seen anything about it,
but it might be actually worth pursuing.
Thank you.
So that concludes our Q and
A section for Doctor Sanberg.
Thank you so much for joining us.
Thank you.
Next we're going to be showing remarks
that were prepared by Dr Ben Goertzel for
this workshop.
He has probably one of the most extensive
resumes in research toward artificial
general intelligence.
Among other things he is the chief
scientist at Hanson robotics,
the chairman at the Open Cog Foundation
and the chairman of the AGI conference
series.
So now I will show his remarks.
Just one quick thing. Anders,
thank you so much for joining us.
I realize that you're probably
very tired from your flight,
so I can understand if you can't
make it all the way through,
but if you can sort of glimpse at the
screen from time to time and might still
be around after Ben's contribution,
that could be interesting because he
sometimes takes kind of a opposite or
contrarian view from...
Always.
And that's the fun thing about him.
I'm going to try to stay around.
Cool.
All right.
I'll get that started.
... my good friend Randal
Koene was organizing a workshop
on mind uploading, AGI,
all that good stuff.
and brain-computer interface,
I really wanted to participate,
but...
Can anybody hear me now?
Yeah, we can hear you. We just lost the
audio from the video for a second there.
Okay.
Was it off for the whole video?
Most of it, yeah. Only the
very beginning we heard him.
Okay,
I'll go ahead and restart it then.
I really wanted to participate,
but being based in Hong Kong,
I heard that my good friend Randal
Koene was organizing a workshop on mind
uploading, AGI, and brain-computer
interface, all that good stuff.
it wasn't feasible for me to show up
in person. So ask around, go to...
Send me some of the key questions that
he was interested in exploring in the
event.
I'd given a little bit of
a video improvisation on
the theme of his questions.
Now,
some of the questions Randal sent would
take approximately 10,000 years to
really go through and answer in detail.
So I'm going to give some of
them the short shrift, but better
something than nothing.
So here goes.
First question from Randall,
"Could you tell us a little bit about
how your thoughts on AI safety have
evolved over time and
where you stand today?"
Well, my thoughts on AI safety
at base is the same as
they've always been. I think
there's a nonzero risk that as
AI verges on AGI and artificial
superintelligence,
things that are very bad by our current
human standards will happen.
I don't intuitively,
emotionally feel the
risk is extremely high.
On the other hand,
rationally I have to accept that we're
in a situation of tremendous and probably
irreducible uncertainty.
We're taking a leap into the unknown.
And that's not unlike what humanity has
been doing since we stepped out in the
African savanna and started
developing civilization.
We've been taking a huge leap into the
unknown one time after the other since
civilization began and
probably before that. So,
I guess for me the question is really
how much do I trust my sort of inner
spiritual,
heart based intuition that the singularity
is almost surely going to come out
okay;
and is in fact going to
connect us with compassionate,
benevolent aspects of the
universe that we're currently,
largely cut off from due to
our mentalities.
How much do I trust that
intuition versus the more cold
objective reasoning part of my mind which
tells me we have no idea what the hell
is going to happen. This is really
the delemma that is realy on goingly
wrestling with. And then maybe that
process, that dialectic is it valuable,
because certainly I wouldn't want to go
entirely in the direction of following
only my heart and not
reasoning or entirely in the
direction of just reasoning
and not going with my intuition,
which,
can have a deeper insight than reasoning.
I would say one thing where my thoughts
on AI safety have evolved in the last
few years though is I'm getting a more
concrete sense of what there is to be
worried about regarding
the rollout of nowhow
throughout the world before we get to AGI.
And I've been thinking more about
what effect the species' knowhow
that gets developed can have the type of
AGI that comes about as knowhow verages
into AGI.
So specifically as I've been saying a lot
recently that the core applications of
knowhow in the world
right now are selling,
spying, and killing. I mean advertising,
surveillance, and millitary.
And if it does happen when the first AGIs
evolve out of the knowhow as it would
be built in the world today.
What does that mean?
Does it mean the first AGI is
going to be involved with selling,
spying and killing?
I mean not necessarily,
but that's at least something
we want to consider. There
is a related
issue which has to do with
the control of knowhow. And
then they have to do with the control of
AGI and it comes out of the narrow AI,
which is how widespread,
democratic,
and participatory should the controlling
of AI be versus how centralized and
elitest should it be?
And there's been alot of thinking
in the world of safety of AI for
a long time that it
would be safer if no small handpicked
crew of wise and rational AI
safety gurus we're controlling the advent
of AI as it turned from narrow AI to
AGI.
There's
another line of thinking which says that
self appointed elites know what's best
for everyone, often don't
do as good a job as they
thought they were going to do.
And the failure modes of this are
amply demonstrated throughout history.
One of the good guys discovers
a dark side within himself.
The elites split into two groups,
someone gets stolen away by competition.
Humans who band together who
no what's best for everyone,
then pull the puppet strings
of the brothers society...
The track record isn't,
isn't great. Right?
And of course what's been tossing by some
folks in the AI safety world was like
an expert committee of
wise singularitarians who
were building an AGI in the
basement and sculpting it's goal system
to be beneficial and then releasing it
in the world.
What we're seeing now is more a move
towards eliteist control of AI by some
advertising corporations,
and some large governments doing
surveillance and military stuff.
So we're getting this eliteist control,
however
the controllers are not who some of the
elitist AI safety advocates might have
wanted. Right? And to my mind,
well of course a part of me
can't help but think, well yeah,
if I were just the one in control,
if me and ten or twenty or 50 or my
closest friends we're chugging all this
out... We want the best for
humanity and for a super-human AGIs.
Right?
And we've surveyed all the relevant areas
of history and science and engineering
and philosophy and we can probably
make a better choice. Then the
whole of humanity,
which includes a whole lot of ideas
that I think are totally whacked out.
But then if I take a little
bit deeper point of view,
in the end,
I don't think any small group of people
is going to do a better job than the
global brain of public humanity.
And there are kinds of understanding and
wisdom on the planet I've never heard
of and never imagined.
And if we want to
really make the best possible AGI
which are going to be the best possible
super-intelligence we
need to craft ways...
The best odds of success seem to be if
we can craft ways to really draw on the
overall intelligence,
intuition,
and wisdom of the global brain of humanity
and our computing and communication
networks,
not some small elite group,
as fun as it would be to be part of the
small elite team training the beneficial
singularity.
This is what has lead me in large parts
of the singularity project that I'm now
running that I founded it in 2017.
What we're trying to do with singularity
is to create a decentralized platform
and then community for both narrow AI
and AGIs so that all the AIs in the
decentralized network of AIs and the AI
programmers and the AI users can all in
a participatory and democratic way
control the evolution of that network.
And if the singularity or something like
it becomes a predominant way or even a
really significant way AI
is rolled out on the planet,
then that's been the counter act to
this eliteist tendency that we're seeing
with a few large corporations
and governments hiring
most of the AI researchers,
buying most of the AI startups,
and sort of driving the AI agenda.
Randall's, next question, "What would you
consider a worst case scenario for AGI?
What's the best case scenario?
What's the likely outcome?" Well,
worst case scenario,
we probably can't even imagine,
but how about some some crazy Christian
maniac Mind-uploads all of humanity into
a simulation of the Christian Hell and
just burns your simulated clones until
the end of the universe.
That would be pretty bad.
We could come up with worse.
Best is utopia and of course
I could take a lot of forms,
but what I'd like do is form
myself into multiple persons,
let one of them mind-upload
into the global super
intelligent mind matrix of the
multiversal super-intelligence
mind matrix and then let
another one of then stay in roughly
human form an upgrad itself progressively
so it can be an even better and greater
human than is possible in the scope of
legacy humanity. Of course, when
you really think about that,
what's funny is from the stand point
of the form of me that remains human,
when I still have a form that merges
with the super-intelligent mind matrix,
it would be like, okay, that form
of mine has been created. I'll wait,
now in the last one second,
it's experienced 10 trillion times more
things then I ever will be able to,
and it's evolving something totally
uncomprehensible to me. So there,
that's nice that I've spawned that
super-integelent mind child.
Now,
I'll go on being being human.
So there's going to be a
discontinuity between forms if me that
embraces the singularity for long and
becames massively super intelligent verses
the form fo me that
remaines in the human form.
But I would like to see everyone able
to form themselves however they want and
make many copies of themselves and explore
different reasons of mind space and
this lets each mind explore a variety
of different types of realities that are
utopic
in it's own perspective
and this is quite feasible.
It may even be the most likely outcome,
but we don't really have the basis
to fully ration the estimate,
the probabilities.
I think we can work toward increasing
the probabilities of beneficial outcomes
like this in a number of ways.
One is the AIs that we're creating right
now what are going to grow into the
AGI.
Maybe that would be just in the future,
we should be using these AIs to do
benefitial things like cure disease,
you know, teach children, improve
people's states of consciousness,
discover science.
We've got to take the bulk of AI
away from selling spying and killing.
Iwould like to eliminate these
things,
human society and human
psychology being what they are,
but they don't need to be
the preponderance of what
AI is used and developed
for. That's probably the most
important thing we can do now to
move the odds of utopic rathar
than distopia or mediocre
outcomes in a positive
direction.
Next Randal asked,
"What about Nick Bostrom's
book Superintelligence."
I read a review of that
consequencial intelegence fears from
Susan Potentials two years ago. My veiw of
Nick's book now is about the same
as it was when he wrote that review.
I love Nick Bostrom worked together in
that World Transhumanist Association
years ago. We organized the
conference together. Bostrom,
I think you're a really fun creative guy.
I think the book Super Intelligence is
a brilliant example of argumentative
rethoric.
It reads like he was the captain of the
high school debate team or something,
so it makes a rigorous,
powerful argument that super intelligence
doing thing to humans we consider
nasty like anihilating all humans is
possible and that these bad outcomes of
super-intelegence have odds
of greater than zero. Zero.
But then the book often talks as if
its tone and its informal statements,
it talks as it been argued that bad
outcomes from super-intelegence are likely
but that was ever demonstrated.
All that was demonstrated is that the
probability is somewhere above zero.
So yeah, of course the probability
is somewhere above zero,
that superintelligence will kill everyone,
but Bostrom didn't demonstrate that it
was probable and no one has demonstrated
that.
On the other hand,
I haven't demonstrated that it's highly
unlikely that super intelligence will do
bad things either from a rational
point of view we just don't know,
are leading into the leaping
into the great unknown.
But then Nick Bostrom in that book at
least really champions a sort of elitist
point of view and at some point in the
book he's sort of exploring the idea.
You could even have one Genious AI
researcher working on AGI protected by the
auspices of the United Nations and
maybe that will be the safest way to do
things.
I'm like exact opposite of that.
I think we want a tremendous amount of
brilliant AI and AGI researchers all
around the globe with many different
points of view collaborating and I want a
decentralized network coordinating
this in a self organized democratic and
participatory way.
Certainly not the UN,
which can't even handle far,
far simpler tasks than coordinating
the birth of of general intelligence.
Those are the first few
questions from Randal.
This is part two of the video in which
I give some rambling improvised one in
the morning type answers to some questions
posed by my friend by Randal Koene as
part of the Carboncopies workshop on mind
uploading and computer-brain interface,
AGI,
and so forth,
which was held in the part of
the world far away from me. So,
in light of not showing up and rambling
semi-coherent coherently at the audience,
I'm doing so on video from afar.
So let me continue with
some of Randal's questions.
First question,
what do I think about brain computer
interfacing as a tool to improve
AI safety?
What impact would high speed
brain-computer interfacing have on AI,
rapidly self evolving AI,
or AGI?
Brain computer interfaceing
could either greatly aid with AI safety,
or it could terribly
harm
the prospects of safe AI.
It really depends on how you would
see this. On the beneficial side,
if we want our AGI's to bridge and
understand human values, connecting
to the human brain is going to be a
really nice way for an AGI to suck some
values out of the human brain.
Of course,
AGI could even learn
values by watching people.
And by enacting values bridging the gap
between
robots and other agents in the
human world. But to the extent
that an AI get those values from the
brain and make them understand human
values...
...the other hand
if we look at selling,
spying and killing as
the main pursuits of the
AI sphere in the world today,
given the big checks from goverments,
they're controlling so much of
AI today.
How could brain-computer interfacing
be for selling, spying and killing.
You could invent a lot of interesting
ideas that way and will they lead to the
AGIs that are going to be
positive and beneficial to
the humans that they're sentient beings
as they expanded their intelegence?
Well, quite possibly not. Right?
Randal's next question,
"Could you create high
bandwidth brain-computer
interface without first having a
neuroprosthesis or a
completely artificial brain?"
I think that the brain is very adaptive
and probably if you stuck a bunch of
data from a computer into it,
the brain would make some
interesting sense of it.
It would then be no longer
a legacy human brain.
The more complex and the more
data coming into the brain,
the brain would have to morph
itself to cope with this data.
But that'd be quite interesting.
You're creating a hybrid mind.
Of course you get to a certain level
where the brain no longer as the capacity
to adapt to the brain computer interface.
But what exactly that level is,
we don't know enough about neuroscience
or information processing in general to
know that. That problem will
be determined by experiment.
I probably will not be the first
person to volunteer for the experiment,
but I'll be fairly early on.
So this will be quite interesting to see.
I do think it's important
to remember that once
you get into nanotech,
femtotech the amount of intelligence
you can pack into a grain of sand would
probably be a quadrillion times the
human brain and a trillion or something
times any brain computer hybrid.
Because the human brains are inefficient
ways of information processing
creativity.
Probably an inefficient way to manifest
consciousness compared to what's
possible in an engineered physical system
permissible even according with the
known laws of physics,
let alone to the laws of physics maybe
understood to be after a singularity.
Upgrading
human intelligence by
connecting brains to computers,
the hybrid mind is interesting. This is
in the scope of post-singularity minds,
these hybrids are going to be closer
to a monkey or a frog compared to
a super intelligence,
right?
So it's an interesting thing to do in
terms of the transition between here and
post singularly minds,
but in the end it's just a baby
step toward the singularity.
Randal's
next question,
"Do you see ways in which whole brain
emulation and artificial human brains
might immediately present an existential
risk to humanity?" If you emulate some
nasty human and then copy that emulated
human a million times and connect it's
body to selling, spying, and
killing machines around the world,
this may not be good. But again it's
not really about the technology, it's
about how it's used.
My guess is that if you compare whole
brain emulation to an AGI built according
to some new rational,
non-emulating architecture like an
Open Cog system that works really well,
I would guess there's both more benefit
and more risk in the engineered nonbrain
emulating design, because the human
brain is not made for self modification.
If you start enhancing intelligence of
certain parts of it then the parts are
going to break,
and you're going to wind up to not
being able to enhance this intelligence
tremendously without basically
replacing it with a totally different
architecture.
The human brain is an adaptation
to certain resource constraints,
and once you release those constraints,
you are going to need to change the
architecture to manifest the intelligence
that is possible with the new
constraints. I think, on the other hand,
the irrationally architected AGI is just
what we're trying to do with Open Cog
or it'd be something different than Open
Cog with something engineered with self
modification and you know rational
self understanding in mind.
This sort of AI system is going to be
able to reprogram itself it's going to be
able to study itself. It's going to
be able to, replace one module with
an upgraded module,
it's going to be upgrading all its states
and then rationally make decisions as
to what possible improvements
of itself to try.
It can go far,
far ahead of a human augment with a
brain computer interface or an uploaded
emulated human it can go far ahead of
these humanesque, post human minds,
in a good or a bad direction.
Again, the technology has a
lot of potentials It depends
on what you do with it,
and,
of course,
what you do with it will guide
what it does with itself.
Randall asks,
"Rapid,
self-improvement is often described
in the context of utility function
optimization and reinforced
with learning. In short,
it may be accomplished only in a few,
the many complex substances of the brain,
like the neocortex and cerebelum.
Do
you think whole brain emulation can
rapidly self improve?" Well that's,
that's a really
delirious and misguided question Randal.
I think it's true that only a little bit
of what the brain does is reinforcement
learning,
and that's why the brain
can be intelligent because
reinforcement learning is a
terrible overall paradigm
for general intelligence.
Reinforcement learning is only a
tiny bit of what the brain does.
Just like deep hierarchical
pattern recognition, like current,
deep neural lens.
is only a very small part
of what the brain does.
So yeah,
rapid
self improvement can go much more rapidly
if its following methodologies besides
just reinforcement learning.
The reasons that the human brain can't
rapidly self-improve for more than its
architected
contorted and limited way where each
part is dependent on the other parts,
and each part of their dependencies
evolve to work within certain processing
constraints which aren't compatible with
the transient intelegence that we want
to build.
I mean
it's just like you can't take a horse
and double its size and have it still
work.
If you increase the short term memory
capacity of humans to 10,000 instead of
seven plus or minus two,
the connection between short and long
term memory and medium term memory isn't
going to work,
but the relation between declarative
and procedural knowledge and short term
memory isn't going to work.
A lot of other changes have to be
made all around the brain. Whereas,
if you have a rationally architected
AGI system and increase its short term
memory capacity.
If it's written well you could probably
just use some automated code by
rewriting the system and tweek the other
parts of the AGI system to properly
accommodate for the expanded short
term memory of the AGI system.
So,
I think there are limitations in the speed
of self improvement that you're going
to get in the brain emulation or
brain incorporating AGI system.
But these aren't to do with
the limitations of material,
so that extremely narrow and limited
paradigm of reinforcement learning,
these are more to do with just the
constraints of being an evolved system as
opposed to an engineered system
moslty.
Both systems can evolve and
evolution is slow and messy.
Engineering systems can be engineered,
which can be much faster.
Engineered reflective systems can self
engineer, which is going to be really,
really nice and way faster and way
more efficient than the mess of
evolution.
Randall askes,
"There's caution aagainst a strong push
for neurotechnology and whole brain
emulation,
because work in those areas has been
accelerated towards advancement toward
runaway self-improving AI."
Well, as I already said,
I think this particular line of research
in whole brain emulation is a terrible
approach to self improving AIs. So,
if you think self-improving AI is bad,
you should be in favor of
AI through brain emulation.
If you want self-improving AI,
you should be looking at engineered AI
systems that are designed for rational
reflection and self-understanding
and self-modification.
Next question from Randall,
"There's an argument,
the ultimate solution for AI safety is
a scenario for human and AI becoming
inextricably entangled." Well,
again,
there's no reason to think that will
guarantee safety. There could be good,
bad aspects. I mean if
you're taking, you know,
a very powerful artificial cognitive
system and coupling it with these
reptilian/mammalian motivational and
emotional systems like we have in humans,
this can be pretty nasty, right? I mean,
an open cog system is one example of a
non-brain based system that has a certain
set of goals. They don't drive
all dynamics of the system,
but they drive a significant amount of it.
They have a certain set of goals
and then the system rationally using
probability theory and logic chooses
which actions are most likely to achieve
its most important goals
in the current context.
I mean it's not driven by its body and
its emotions and its instincts to the
extent that a human being is.
I think something like that
is probably going to be,
if it's done right,
it's going to be safer than some weird
Frankenstein thing with these evolved
motivational and emotional system latched
into some artificial cognitive system.
We don't know because we haven't built
an AGI based on the human brain or
brain-computer interfacing or
open cog or anything else, yet.
We don't know this tremendous and very
hard to deduce uncertainly here,
but my own instinct and my intuition,
and I've thought about this a lot,
is that it's going to be a lot more
dangerous to make something in one system,
all this nasty mamalian/reptiles
stuff without official cognition.
I think you want the rational,
reflective self-modifying AGI to
understand human culture and human values,
and to have compassion for humans,
and you don't really
want it to be a human.
You want there to still be humans,
but you don't want to try to do
something screwy like Megan said,
the smartest and most powerful
minds or some sort of huamn-AGI,
franken-bob thing.
You want to accept that humans
are just a limited form of mind.
I mean there's beauty in this limitation
as well as hideousness in this
limitation, but that's what we are and
one of our beauties is that we can build
fundamentally superior minds
that are compassionate toward us.
I can self-understand,
self-modify,
and self-improve in ways
that we intrinsically cannot
due to the way that we
evolved, and we can then
coexist with these super minds,
but we can't be these super minds,
and trying to create a super mind that's
tied in with the human/mammal/reptile
control system is far more risky than
any other technology on the horizon.
Another question from Randall,
"in your opinion,
what should humanity do to maximize,
long term chances for survival."
Give Ben 50 quadrillion
dollars! Apart from that,
very clearly what humanity should be doing
is spending a large percentage of its
resources on
globally beneficial applications of
advanced technologies including AI,
nanotechnology,
neurotechnology, and so forth,
and trying to create machines with
compassion toward humans in deep,
rich understanding of the
full breadth of human values.
We shouldn't be including the bulk
of our AI resources into selling,
spying,
and killing.
We shouldn't be putting so little
resources into medical applications and
advanced technology into education,
agriculture, poets, scientists,
social workers,
nurses,
preschool teachers and philosophers,
and so forth.
So I mean,
if you look at it from the outside
and you had a species on the verge of
creating the first, minds more
intelligent and powerful than itself,
you might think a large percentage of
that specie's resources was going into
figuring
out theoretically how to make these new
minds be as beneficial as possible for
the universe and the multiverse and
for the species during the creating,
and to prototyping different kinds
of beneficial engineered minds,
and to making sure that engineered minds,
as they increase in
intelligence year-by-year,
are working closely in a positive and
compassionate way for the species that
created them.
Instead,
almost all AI development
now is driven by commercial
or military ends, and the same for
medical technology and nanotechnology.
I mean almost all this technology
is being developed so that one
country can achieve military power over
other countries or so the one company
can extract money from other people so
the fact that our technologies are being
developed mostly out of tribalistic or
greed based motives truely isn't good.
We want to be developing
these technologies in a way
that is motivated and is
explicitly driving toward broad benefit
for humans and for the other minds that
being created. That's
not what we're doing. I'm
trying
to develop advanced technologies in a
way that will help all of humanity and
will help eulated human minds
that we're going to create and
throw in the animals and plants and the
rest of the ecosystem and any aliens
that may come out of this
or other dimensions as well.
But most relevant advanced technologies
are being driven by very narrow sort of
probablistic or ego based goals.
This
is not optimal.
Of course,
you can't solve the problem
top-down all at once,
hopefully you can solve that problem by
unleashing sort of new methodologies
into the world. I mean just as open
source transformed the software world,
perhaps decentralized AI networks can
transform the AI world and cause AI just
self organize in a way
it's more democratic and
participatory so that as AGI
emerges out of nornal AI,
it's emerging with the input from users,
developers,
and the participants of AI around
the world and it's getting a broad
range of applications and feeding on
a broad range of human insights and
feelings and intuitions.
It's about 1:38 AM here.
Truely I'm becoming less and less lucid as
this ramble continues,
but hopefully I've given you some
flavor of my views on these issues.
So thanks Randall for inviting me to
participate via video and hopefully next
time I can show up in person.
All
right.
I'll
stop sharing my screen,
and I'm back on it.
That was a really interesting
response by Dr Ben Goertzel.
I'm wondering if Anders was
paying attention to all of that,
and if he has anything to say.
Yeah. Well there is so much
to comment on, of course,
to unpack there and I
think there are two parts.
That particularly caught my attention.
One is this issue where a band disagrees
with Nick about how to go about
handling it.
And I think it has to do with prior
probability estimates and guesses on how
does the risk landscape look like.
Then Ben points out that if you have a
situation where it's enough that there is
one bad AI researcher,
then big things happen.
This is very much the problem we outline,
Nick and me,
in a paper we call the
unilateralist curse.
If you have a group of agents that
it's enough that one agent can unleash
then even if all agents are nice and are
trying to do it only if they think it's
something,
a good thing.
As the group gets larger,
it's more likely that somebody
is going to be that guy.
So I think this is a situation one
should recognize when you're in that
situation you should try to
be more conservative than
you normally would like to
be. Just because of the
nature of that situation.
But I think in particular when you
really what to regulate is when you can't
afford to be wrong,
even once.
And there are those people in AI
safety, you really think that, yeah,
we are very close to that heartache of
scenarios and the likely bad outcome of
such a scenario that means that we
have a sharply fetch should we must be
exceedingly cautious.
I'm way more optimistic.
I don't know if that actually is based
on an irrational reasoning rather thing,
but I don't trust this conclusion,
but I think actualy,
it's a more benign domain.
And I think Ben also has
roughly the same view here.
So this of course leads to different
ideas about what you want to do,
regulatory speaking.
However,
it also suggests that if we could get
even a slight bit of battery information
about the actual risks,
actual structure,
the actual probability
we would win so much.
So I think that is another reason
to really, really pursue this.
And then,
another very interesting point he made
very briefly was the resource constraints
that have shaped the brain
once you're free of them,
you might reshape it in a
lot of other ways. Again,
just to mention a paper I wrote with Nick,
we did one about human enhancement where
we looked at evolutionary medicine as
one guide to try to see where evolution
really might have constrained us and
those are the areas where we might then
find ways of unconstraining us in our
constructive ways.
There might be other domains where
it's going to be very hard to fix.
For example,
I think even uploads are good to have
two run sleep simply because our memory
consolidation is so based around that,
which would be tremendously
annoying form some perspective.
But on the other hand,
yeah it's going to take a lot of
reverse engineering to get the answer.
And that was just two main
points on Ben's
enormous discussion.
Interesting bit there about
whole brain emulations and sleep.
That's one of the things I
actually happen to think about,
which was if you have the
freedom to change synapses,
if you look at what sleep is
actually trying to accomplish,
if you've got your consolidation,
and you've also got in rem sleep,
bringing up older memories and mixing
then in and doing some kind of interleaved
learning and that sort of thing.
There are alternative ways to do that.
And for one thing,
it's certainly doesn't have to take
quite as long as it normally does. So,
I imagine that there may be ways of
getting around those sorts of limitations.
Yeah, I totally agree. But it's
going to be a lot of tinkering.
Absolutely. And that's where Ben
is absolutely right, of course,
that it's much easier to come up with an
artificial intelligence that is free to
developing whatever algorithmic method
likes to get it to self improve faster
than an evolution, a patchwork that
was evolved like the human brain.
I think it might have come across to
him as if I were saying the opposite,
but that wasn't the case.
I was just putting forward
the question, well, can whole brain
emulations even rapidly self improve?
Is this something people
should worry about? Since
that sometimes got brought up.
Mallory,
how did you want to proceed?
Do you want to take a question
from the audience first?
Yeah. We do have some questions from
the audience that I'd like to ask.
We have one,
their username is sacked SOS.
Here they're quoting Descartes,
I think therefore I am,
"Does that not mean that all physical
laws and natural laws are just in our
mind?" and I think maybe what
they're trying to say is,
is our interpretation of physical
laws limited by our biological brain?
I also like to tack on to that,
how do you think that will
change if we were able to have
a whole brain emulation and
possibly merge that with AI,
how will physical and natural
laws then be interpreted?
Yeah.
So this is of course familiar territory
for most introductory philosophy courses
where people start getting confused.
Quite a lot of people not
really think that, oh,
there's nothing problematic with my
perception, I'm see reality as it is.
And at this point the gleeful philosopher
will start bringing up optical
illusion, some examples
from quantum physics.
But at the same time I think David Dutch
has an interesting point in his book,
The Beginning of Infinity,
where he points out that the reach of our
mind is way bigger than what you would
expect for most of all systems.
We actually have some mental tools that
allow us to take explainations that work
over shockingly large domains,
even though we don't have
direct access to them.
When we talk about the distribution of
galaxies in the universe in many ways we
can't see the galaxy. We can only see them
if we use telescopes. So in some sense,
nobody has ever seen the Virgo cluster
because you can't see it with the naked
eye if you are a human,
yet I don't think anybody would say
it makes sense not to believe in its
existence or a say that,
oh,
the Virgo cluster is a kind of weird
artifact of having a telescope.
So I think you get the same situation
even if you're a brain emulation.
Yes,
now you might be seeing using a camera
connected to a device driver connect to
various pieces of software,
turning it into neural signals.
You add the extra levels,
but as long as these levels don't distort
things that are relatively faithful,
I think you actually have
something that make sense.
And I do think we can build on top of it.
I think we can... The most obvious
thing is adding new senses,
or trying to link up modules in our
minds so they are functioning better.
If I could just run a quantum mechanics
co-processor and have it seamlessly
connected to my cortex,
that would be wonderful.
And maybe I would finally understand
stuff. I would also have to change myself,
how I work. But that might be
a relatively low level thing,
like changing how I visualize things
in order to visualize more dementions
rather than changing the true core need.
Of course,
philosophers will always throw in,
well maybe there are things
that are absolutely impossible
for minds of our kind
to think about.
And I think that's true,
but it might actually be
relatively uninteresting.
It's not too hard to construct girdles
sentences that certain system...
but that dosn't mean they don't
come up that often in practice.
Would anybody else like to answer that?
Yeah,
I kind of want to jump in on that just
because it's a good point you brought up
that it's interesting that
our brain can handle things...
Hmm. Sounds like Randal might
have lost Internet connection.
I am.
Hmm.
All right,
I can hear you now and see your video.
You hear me? Okay. So then I
don't know what happened there.
It must have just dropped for a second.
I was just trying to
expand on what Anders said about the
brain being able to deal with things that
it wasn't originally evolved for.
It's
interesting to see in which ways the
tools that the brain has are applicable
that way. And just to point
out one little thing in this,
this gelatinous blob here.
Say for instance, the,
the grid cells of the interinal system.
We talk about them as something that
helps map out space but the same part of
the brain that feeds right into the
hippocampus also has a task in humans it
just generally manages to
store new episodic memories,
new conceptual memories concepts.
Now,
the interesting thing is if you take this
system grid cells for mapping
out spaces automatically,
basically putting in vectors along
them and saying, okay, this is x,
Y and I'm over here and I'm over here,
coming up with where you are.
If you take that and you apply
it to more general concepts,
something that has
nothing to do with space,
you can map out any new kind of idea,
any concept that you have and suddenly
apply it in ways that you never could
before.
You can work with it in a sense.
Mapping things that shouldn't normally
be mapped or that you don't encounter in
nature. And I think you take
that all across the brain
and you can see that these
systems,
these tools,
just like deep learning isn't something
you can only use for one task can use it
for many tasks the same way all the
tools in the brain can be applied to many
different things. It's true,
but then at the same time,
it's not necessarily optimized
for a rapid self improvement.
So I think Ben has a point there as well.
One thing I wanted to address briefly
because Ben has a certain style,
how he expresses his opinion on these
matters is he often makes it very good and
positive statement.
He'll say something that he
really hopes is the outcome.
So we hear a lot of very
positive scenarios from
Ben.And in fact both sides of
course want us to end up in the same
type of outcome, a good outcome,
whatever that good means.
It's just that for some reason the methods
that they advocate are very different
sometimes.
And a lot of this has to do with trying
to understand what the probability of
certain outcomes are and
those probabilities are
influenced by where you begin.
So Andrew's already mentioned that,
your bias,
and perhaps by intuition
which is of course,
again bias by your experiences and by
your emotional attachment to something.
So it's really a problem of clear
probabilities lacking in this case.
And we don't really know which
scenarios are going to happen.
Maybe this is just a sign of how
young this field is still.
But I do think one point that Ben makes
is very interesting he brought this up
in his paper in 2015 as well.
The one that I think I might've
mentioned somewhere, that
we may need artificial intelligence or
artificial general intelligence to guard
against some of the bigger risks that
we run into because of the way humanity
goes about things.
We'll talk about how an
AI might want to follow a certain goal
and keep on improving towards that goal.
Even if it has to consume all
the resources in the universe.
You could make a very
similar argument for humans.
We're still based on a system that evolved
2 million years ago and we're about
to use up all the resources of the
earth in order to try to keep doing that
because that's what we
were programmed to do.
We may also be stuck with the same utility
function and following it and we may
be running a great risk of doing
damage to ourselves in the process.
Now again, I don't know if that's
really what's going to happen,
but this is part of the
argument that Ben is making.
Humans are not rational actors were not
necessarily picking the best outcome all
the time.
AI may be more rational than we are,
so it's entirely possible that
developing that as quickly as possible,
may be something that saves us from a
risk that is more immediate or bigger
because we keep on not having a good
chance of comparing these probabilities,
then the likelihood that AGI itself
somehow be an agent that takes us down.
I don't know if I agree with his point
of view more or if I agree with Nick
Bostrom's point of view, but I'd love to
hear Andrew's kind of give this a shot.
Yeah.
I've been thinking a lot about what
can we do to do the research right?
Because as you said,
we need to get the probabilities right,
but we might also want to get to the
values right? So in my current project,
this big book about the long
term future of humanity,
one of the parts that I'm still working
very hard on and that is tough to work
on, it's of course what values
make sense in the universe.
And what would rational
actually mean because yes,
we're definitely not rational right now,
but just like the hippocampal place
cells allow us to handle spaces of quite
arbitrary shapes,
our minds are actually able to have
quite a lot of different models of
rationality and thinking.
It's just that we're not super
good at that so much yet. Well,
it seems to me that one way of making
progress on both of this kind of tough
problems is to look at toy problems.
You take stylized small problems.
So this is how you might help
the AI safety move forward.
You will demonstrate that here is a
little agreed world situation and you want
your agent to behave itself in a certain
way. Can you program it to do that?
And that's useful,
not just to entice software engineers the
realize that actually it's harder than
they think to make AI safe,
but it also is a good way of
revealing tricky problems.
Learning about the size of the problems
and learning about what other questions
you want to do because typically to refine
this as people find ways around what
you wanted to do.
There is another side and that is the
empirical research you actually want to go
out in the world and actually watch
real situations because that keeps you
honest.
It's quite important to not just spend
your time in the lab or in a thinking
chair, but actually go out and try to get
some data and then critically judge it.
Just because people think
autonomous course should be
driving in particular ways,
doesn't mean that's the ethical thing
to do, or the smart thing to do,
but it's certainly an interesting input.
And then ideally once you have both good
models and the empirical data to play
around with, you can also start
building the more deeper theories.
And I think this goes for creating
AGI that is helping us make good choices.
We need to know a bit more
about what good choices are,
both from a correctional
and ethical style point.
But we also actually wanted to develop...
Can we make a decision aid in tools even
in less ethically fraud domains that
are actually useful,
that's not entirely obviously yet,
yet it is a big market.
You can make a lot of money if you
make a good decision aid and there
researchers working on kind
of moral decision aids.
I have some friends in
Oxford working on that,
where hopefully they would tell
you, if you have a dilemma, "Well,
given what you claim to believe,
this is what you ought to be doing.
You want to discuss it more?" and
I think that could be quite useful.
So I think there is an interesting
big research project here.
It's kind of spanning,
not just AI safety and brain emulation,
but quite a lot of other stuff.
Taking what starts out in philosophy,
making it concrete enough that we
can start writing code and doing a
psychological service and they'll start
thinking broadly about how to implement
tools to make us better.
I think that was a great answer.
Next,
we have a question from Dean Horak
and it's actually one of my favorite
questions to talk about.
He says,
"Assume we have substrate independent
minds available on some digital whole
brain emulation platform,
it would seem that accessing someone
else's memories and experiences and
thoughts would be nearly as
effortless as accessing our own.
Given the scenario,
wouldn't we eventually lose our individual
identities and ultimately become like
a hive mind?
Well I think to some degree
we have already created tools
to access. Yet, we actually
deliberately create walls.
In some sense Internet allows any computer
on the Internet to talk to any other
computer.
But it's not just that I can't
get your data out of your computer.
But we also deliberately create structures
like web pages or and the Google
hangouts to actually shape it,
because having everything in
a big pile is not very useful.
My memories are only useful because I
did make good association in relation to
my current experience etc. And if you had
your mind and memories and my memories
sometimes of course they will
overlap in a resonant sort of way,
but quite often it would probably be
just arbitrary. But we want selectivity,
we want filtering and that is probably
what it's going to mean that hiveminds
are probably going to be
more limited than we think.
In fact you can compare it to economics.
Why do they have companies or firms and
of course economists have been going on
about the theory of the firm and that
you have CSR Hidalgo at MIT and the
general argument is that they are
economists of scale up to a point.
It's a good idea to get a bunch of people
together but you also need to manage
the information flow.
Then you create a little box around it
and call this a department or a company
and we are likely to do the
same thing for group minds.
It wouldn't surprise me that once
you have the ability to link minds,
we're gonna experiment endlessly with
a lot of architectures and some of them
are going to be good for certain things,
others less good.
But maybe specialized and then we
might actualy have this big complicated
structure of overlapping
and sometimes heroic income.
Sometimes autocratic and sometimes
totally alien structures.
It's going to be predict exiting.
But again,
it's a good thing that there is an
undo button of your brain emulation.
Great point about the undo button.
That's something I occasionally
mention when people ask about,
this so risky, why would you
experiment with upgrading and stuff.
But you mentioned here,
things that are already
a bit like hive minds,
organizations and groups
and all that thing.
And I think that to
answer a dean's question,
I think that hive minds
merging is probably unavoidable
because it's already a
reality.
It's just that the components of a hive
mind don't know they're in a hive mind
or maybe they do, but sometimes
they don't or they don't admit it.
They'll say I'm still an individual.
But really they're entirely
dependent on everyone else. I mean,
do you farm it's sort of...
We may be part of a hive
mind and not know it yet.
That was great.
I definitely see it that way that we
are already kind of interconnected in a
certain way due to
technology and social media.
Whole brain emulation will just exacerbate
something that's already happening.
Some aspects of social media actually
give us a very good inkling about just
connecting everything completely.
That's not very good.
We actually want to create the right
kind of structures and finding the right
kind can turn out to be amazingly hard,
while down the line
technology is relatively safe.
But then if you're taking
social media as the example,
then maybe it's also demonstrating that
it takes a certain amount of trial and
error and learning because we had no idea
what this would be like before we got
into it.
We didn't know what the Internet was going
to spawn and we certainly didn't know
anything about social media back
in the eighties nineties or before,
so it would have been very
difficult to think ahead and say,
when we design social media,
what should the boundaries be?
I totally agree.
There is this general problem that the
world is much more complicated than our
minds,
so we generaly quite often can't predict
the consequences outside a particularly
simple domain.
We just have to do experiments.
Which is of course problematic
if experiments are risky
or expensive or painful,
but also very exciting because it
means that we've discovered new things.
I find it very unlikely that we will ever
enhance themselves such a degree that
we can just figure it out or rationally
just like some philosopher king just
sitting in his arm chair,
knowing everything.
I think we're going to find that
a world that has great minds,
is going to be even more
complicated anyway, after all,
social media among smart
super intelligence is
going to be probably having
emergent, wierd properties that we can
think about and they can't think about.
They're going to be equally confused
when they get their counterpart to fake
news or Twitter storms and the are
going to perhaps wonder maybe we need a
bigger mind to figure this
hyper social media out.
Hmm.
Yeah.
It seems like there are at least two
different ways to go about preventing the
worst case scenario.
And one of them is you try to anticipate
something bad that could happen and
just don't do it or prevent it
from happening. And the other one,
as we just mentioned,
in the case of experimenting with whole
brain emulation is undo buttons. So,
what we haven't really talked about,
and maybe it's because
nobody thinks it's possible,
is undo buttons for things
that happen in AI development.
I guess part of the problem is that we
imagined that a bad scenario is one where
AI development suddenly goes so quickly
that there is just no way to stop it,
let alone undo whatever happened.
But an undo button would be nice if
such a thing were possible outside of a
sandbox let's say.
So I think the problem is that many of
the AI risks are seen as information
hazards. So once I publish my code
on git hub and tweet about it,
there is no way of gifting
the genie back in the bottle.
And that is an interesting thing because
that might just be because we are not
imaginative enough to come
up with a better undo button.
There are ways of handling
information hazards.
I'm working on some projects and thinking
about how institutions like the Future
of Humanity Institute should organize
its own research when it finds some risky
information and you can certainly have
structured transparency where you can
reveal certain information but not others
in a way that is trustworthy etcetera.
But it might also be that we can find
ways of doing AI research where you can
build in some forms of undo buttons.
Like we pursue a certain line of thinking,
we discovered sort of
risks or certain problems,
and there is a considered
way of removing it.
This is of course what we would
like to have in science too.
If I write a paper that
I would later retract,
ideally all papers citing that paper need
to get a little marker in the citation
of that now we're referring to paper
that turned out to be wrong for one reson
or another. We need to build
an infrastructure for that.
Right now I don't have any
idea of how to do that,
but it sounds like one of those things
that should be a high priority for
someone.
Yeah. Actually, this is something
where blockchain gets interesting.
A ledger that you shared because if you
can actually track everyone who is on
that system,
or who's got a copy of something,
then you could go in and try to withdraw
it or withdraw the ability to run that
thing or to change it to whatever
needs to be fixed. That's one approach.
I'm just talking about it very loosely
because I'm not actually someone who
develops that sort of code,
but it sounds to me like something that
those systems could evolve towards.
And the other is what we call software
as a service, right? If you look at that,
the situation there is that if,
let's say,
the service that you're
making available has a bug.
You only need to fix it where the service
is being run and everyone else uses it
and puts it to use. So if there are all
these little modules that you use for AI,
but they're all software as a service,
it's very different than if
you're giving away the code.
Some opportunities there,
perhaps for something like an undo button.
Yup. Sounds like a really
fun topic to pursue further.
All right.
I think that might be a good question to
end on for our workshop unless anybody
has any objections.
Yeah,
I guess,
let's ask our panel if any of you have
any questions that you think really
should be mentioned,
either because they're your own
or you saw them coming through.
Maybe Jesse or somebody noticed
something coming through,
just as a quick opportunity
to before we close.
But otherwise I agree because it's been
long enough and we can still handle all
of those questions later in other ways
because we've got them all written down.
Doesn't sound like it. So I guess that'll
be the conclusion of our workshop.
Again,
thank you everybody for attending.
I want to say thank you to all of our
volunteers that helped put this together
and all of the speakers that joined
with us today, talked on the panel,
and sent us videos or did interviews.
We really appreciate that.
I just wanted to have just have a couple
of announcements besides the survey for
the workshop that we would very much
appreciate all of our audience members
filling out and that can be
found at survey.carboncopies.org.
I also wanted to mention that we have
a new episode of our podcast and that's
available on our website if
you want to go and view that.
We frequently have new episodes that
address different topics related to whole
brain emulation.
We also have up items on our store,
new items on our store now. We have
sweatshirts, tee shirts, coffee mugs,
bags.
I'm already decked out in the
sweatshirt if anybody can see.
It's got our nice logo
and I really enjoy it
and just wanted to mention
that we are a 501C3 nonprofit.
All of the work that we do
has been volunteer work and
we relly on the generous
contributions of our donors.
If you could visit carboncopies.org/donate
and make a donation that
would be great for us.
Our goal right now is to get to about
$2,000 a month and that'll fund future
workshops and any future research that
we'd like to do that would allow anybody
else, the audience members or people
that follow us to participate.
And on that note we also have several
social media channels that you can join us
on. It just depends on your
preference for Facebook,
it's facebook.com/groups/carboncopies.
We also have a reddit account
and it's just our /carboncopies.
And then our Twitter account
is @carboncopies.org.
Thank you everybody and I
think we'll close on that.
I had a great time and I think I learned
a lot and I have a lot of new questions.
Thank you, Mallory. And one
just quick little thing.
Thank you so much Anders for jumping in
even though you had to travel so far.
I hope you get some good rest now.
I'm looking forward to a
nice Swedish night now,
but this has been so exciting.
I'm going to have sweet
dreams about brain emulations.
Okay,
fantastic.
And also for anyone who had trouble
hearing any portions of the interviews,
or Ben's talk,
all of these are going to be made
available separately in their full high
quality audio and video.
They degraded a bit as we had to
stream it through this live stream.
So you'll get those as well.
Thank you everyone.
Thank you.
Bye Bye.
Bye Anders.
