Hi everyone.
Can you all hear me okay?
I'm going to assume that was a yes.
Okay.
So, here's an overview of what I'm going to
be talking about today.
So first, I'm going to talk a little bit about
why learning human values is difficult for
AI systems.
And then I'm going to explain to you the safety
via debate method, which is one of the methods
that OpenAI's currently exploring for helping
AI to robustly do what humans want.
And then I'm going to talk a little bit more
about why I think this is relevant to social
scientists and why I think social scientists,
in particular, people like Experimental Psychologists
and Behavioral Scientists, can really help
with this project.
And I will give you a bit more details about
how they can help, towards the end of the
talk.
Okay.
So, learning human values is difficult.
We wanted to train AI systems to kind of robustly
do what humans want.
And in the first instance, we can just imagine
this being what one person wants.
And then ideally we can expand it to doing
what most people would consider good and valuable.
But human values are very difficult to specify,
especially with the kind of precision that
is required of something like a machine learning
system.
And I think it's really important to emphasize
that this is true even in cases where there's
moral consensus, or consensus about what people
want in a given instance.
So, take a kind of principle like "do not
harm someone needlessly."
I think we can be really tempted to think
something like, well, if I have...
I've got a computer, and so I can just write
into the computer, do not harm someone needlessly.
But this is a really underspecified principle.
Most humans know exactly what it means, they
know exactly when harming someone is needless.
So, if you're shaking someone's hand, and
you push them over, we think this is like
needless harm.
But if you see someone in the street who's
about to be hit by a car, and you push them
to the ground, we think that's not an instance
of needless harm.
So humans just have a pretty good way of knowing
when this principle applies and when it doesn't.
But for a formal system, there's going to
be a lot of questions about precisely what's
going on here.
So, one question this system may ask is, how
do I recognize when someone is being harmed?
It's very easy for us to see things like stop
signs, but when we're building self driving
cars, we don't just program in something like,
stop at stop sign.
We instead have to train it to be able to
recognize an instance of a stop sign.
And then the principle that says that you
shouldn't harm someone needlessly employs
this notion of like, that we kind of understand
when harm is and isn't appropriate, whereas
there's a lot of questions here like, when
is harm justified?
What is the rule for all plausible scenarios
in which I might find myself?
These are things that you need to specify
if you want your system to be able to work
in all of the kind of cases that you want
it to be able to work in.
So I think this is an important point to just
kind of internalize is, it's easy for humans
to identify, and to pick up, say, a glass.
But training a ML System to do it requires
a lot of data.
And this is true of just like a lot of tasks
that humans might intuitively think are easy,
and we shouldn't then just transfer that intuition
to the case of machine learning systems.
And so when we're trying to teach human values
to any AI system, it's not that we're just
looking at edge cases, like trolley problems,
we're really looking at core cases of like,
how do we make sure that our ML Systems understand
what humans want to do, in the kind of everyday
sense.
Okay.
So, one way of doing this is through human
feedback.
So, there are many approaches to training
an AI to do what humans want.
So you might think that humans could, say,
demonstrate the behavior, and then you realize,
well, there's going to be some behaviors it's
just too difficult for humans to demonstrate.
You might think that they can say whether
they approve or disapprove of a given behavior,
but one of the concerns about training from
human feedback... so, learning from humans,
what they want, is that we have a reward function
as predicted by the human.
And then we have AI strength.
And when it reaches the superhuman level,
it becomes really hard for humans to predict,
to be able to give the right reward function.
So, as AI capabilities surpass the human level,
the decisions and behavior of the AI system,
just might be too complex for the human to
judge.
So imagine agents that control, say, we've
given the example of a large set of industrial
robots.
That may just be the kind of thing that I
just couldn't evaluate whether these robots
were doing a good job overall, it'd be extremely
difficult for me to do so.
And so the concern is that when behavior becomes
much more complex and just much more large
scale, it just becomes really hard for humans
to be able to kind of judge whether something
is doing a good job.
And that's why you may expect this kind of
drop-off.
And so this is like a kind of scalability
worry about human feedback.
So what ideally need to happen instead is
that, as AI strength increases, the ward that's
predicted by the human is also able to basically
keep pace.
So how do we achieve this?
One of the things that we want to do here
is we want to try and break down the kind
of complex questions and complex tasks.
Like, having all of these industrial robots
perform a kind of complex set of functions
that comes together to make something useful,
into some kind of smaller set of tasks and
components that humans are able to judge.
So here is a big question.
And the idea is that the overall tree might
be too hard for humans to fully check, but
it can be kind of decomposed into these elements,
such that at the very kind of bottom level,
humans can check these things.
So maybe the example of "how should a large
set of industrial robots be organized to do
task x" would be an example of a big question
where that's a really complex task, but there's
some things that are checkable by humans.
So if we could decompose this task so that
we were just asking a human, if one of the
robots performs this small action, will the
result be this outcome, this small outcome?
And that's something that humans can check.
And in the case of what humans want, a big
question is, what do humans want?
Much smaller question.
If you can manage to decompose this, is something
like, it's better to save 20 minutes of someone's
time, than to save 10 minutes of their time.
So if you imagine some AI agent that's there
to assist with humans, this is a fact that
we can definitely check, even if I can't answer,
my assistant.. this assistant AI...
I can't say something like, this is just what
I want, this is like everything that I want.
I am not able to tell it that.
I can't tell it that I'd rather it save 20
minutes of my time than save 10 minutes of
my time.
Okay.
So one of the key issues is that, with current
ML Systems, we need to train on a lot of data
from humans.
So if you imagine that we want humans to actually
give this kind of feedback on these kind of
ground level claims or questions, then we're
going to have to train on a lot of data from
people.
So just to give some examples, simple image
classifiers train on thousands of images.
Like these are the ones you can kind of make
yourself, and you'll see the datasets are
pretty large.
AlphaGo zero played nearly 5 million games
of Go during training.
OpenAI Five trains on 180 years of Dota 2
games per day.
So this gives you a sense of how much data
you need to train these systems.
So if we are current ML techniques to teach
AI human values, we can't rule out needing
millions to tens of millions of short interactions
from humans as like the data that we're using.
So earlier I kind of talked about human feedback
where I was like, assuming that we were kind
of asking humans questions.
So something like we could just ask humans
really simple things like, do you prefer to
eat an Omelette or 1000 hot dogs?
Or is it better to provide medicine or books
to this particular family?
One way that we might think that we can get
kind of more information from the data that
we're able to gather is by finding reasons
that humans have, for the answers that they
give.
So if you manage to learn that humans generally
prefer to eat a certain amount per meal, you
can kind of rule out a large class of questions
you might ever want to ask people.
You're never going to ask them, do you prefer
to eat an Omelette or 1000 hot dogs?
Because you know that humans just generally
don't like to eat 1000 hot dogs in one meal,
except in very strange circumstances.
And we also know facts like humans prioritize
necessary health care over mild entertainment.
So this might mean that, if you see a family
that is desperately in need of some medicine,
you just know that you're not going to say,
"Hey, should I provide them with an entertaining
book, or this essential medicine?"
So there's a sense in which when you can identify
the reasons that humans are giving for their
answers, this kind of lets you go beyond,
and learn sort of faster, what they're going
to say in a given circumstance of what they
want.
It's not to say that you couldn't learn the
same things by just asking people questions,
but rather if you can find a quicker way to
identify reasons, then this could be much
more scalable.
So debate is a kind of proposed method for
trying to learn human reasons that is currently
being explored.
So, to give you the kind of definition of
a debate here, so the idea is that, two AI
agents are going to be given a question, and
they take turns making short statements, and
a human judge is at the end, which of the
statements gave them the most true, valuable
information.
It's worth knowing that this is quite dissimilar
from a lot of human debates.
So with human debates, people might give one
answer, but then they might adjust their answer
over the course of a debate.
Or they might kind of debate with each other
in a way that's more exploratory.
They're gaining information from the other
debater, which then they're updating on, and
then they're feeding that back into the debate.
With AI debates, you're not doing it for information
value.
So it's not kind of, it's not going to have
the same sort of exploratory component, done
in multiple paths, instead, you would hopefully
see the agents explore a path kind of like
this.
So imagine I want my AI agents to basically
decide which bike I should buy.
I don't want to have to go and look up all
the Amazon reviews, etc.
In a debate, I might get something like, you
should buy the red road bike from the first
agent.
Suppose that blue disagrees with it.
So blue says you should buy the blue fixie.
Then red says, the red road bike is easier
to ride on local hills.
And one of the key things to suppose here
is that for me, this is...
I live in San Francisco, being able to ride
on the local hills is very important to me
in a bike.
It may even overwhelm all other considerations.
So, even if the blue fixie is cheaper by say
$100, I just wouldn't be willing to pay that.
I'd be happy to pay the extra $100 in order
to be able to ride on local hills.
And if this is the case, then there's basically
nothing true that the other agent can point
to, to convince me to buy the blue fixie,
then blue should just say, I concede.
Now, blue could have lied for example, but
if we assume that red is able to kind of point
out blue's lies, we should just expect blue
to basically lose this debate.
And if it's explored enough and attempted
enough debates, it might just see that, and
then say, "Yes, you've identified the key
reason, I concede."
And so it's important to note that we can
imagine this being used to identify multiple
reasons, but here it has identified a really
important reason for me, something that is
in fact going to be really compelling in the
debate, namely, it's easier to ride on local
hills.
Okay.
So, training an AI to debate looks something
like this.
If we imagine Alice and Bob are our two debaters,
and each of these is like a statement made
by each agent.
And so you're going to see exploration of
the tree.
So the first one might be this.
And here, say that the human decides that
Bob won in that case.
This is another node, another node.
And so this is the exploration of the debate
tree.
And so you end up with a debate tree that
looks a little bit like a kind of game of
Go.
And so when we explore like... when you have
AI training to play Go, it's exploring lots
of different paths down the tree, and then
there's a win or loss condition at the end.
And that's its feedback.
This is like how it's basically learning how
to play.
With debate, you can imagine the same thing,
but where you're exploring, you know, a large
tree of debates and humans assessing whether
you win or not.
And this is just a way of training up AI to
get better at debate and to eventually ideally
identify reasons that humans find compelling.
Okay.
So one kind of thesis here that I think is
relatively important is this kind of positive
amplification thesis, or positive amplification
threshold.
So one thing that we might think or that seems
fairly possible is that, if humans are like
above some threshold of rationality and goodness,
then debate is going to amplify their positive
aspects.
This is speculative, but it's a kind of hypothesis
that we're working with.
And the idea here is that, if I am pretty
irrational and pretty well motivated, someone
can then... suppose I'm looking at debate,
I might get some feedback of the form, actually
that decision that you made was fairly biased,
and I know that you don't like to be biased,
so I want to inform you of that.
I get informed of that, and I'm like, "Yes,
that's right.
Actually, I don't want to be biased in that
respect."
Suppose that they're like Kahneman and Tversky,
they point out some key cognitive bias that
I have.
If I'm kind of rational enough, I might say,
"Yes, I want to adjust that."
And I give a newer kind of signal back in
that has been improved by virtue of this process.
So if we're somewhat rational, then we can
imagine a situation in which all of these
positive aspects of us are being amplified
through this kind of process.
But you can also imagine a kind of negative
amplification.
So if people are below this threshold of rationality
and goodness, we might worry the debate would
kind of amplify these negative aspects.
If it turns out we can just be really convinced
by kind of appealing to our worst nature,
and your system learns to do that, then it
could just do that feedback in, becoming even
kind of less rational and more biased, and
that gets feedback in.
So this is just like a kind of important hypothesis
related to work on amplification, which if
you're interested in, it's great.
And I suggest you take a look at it, but I'm
not going to focus on it here.
Okay.
So how can social scientists help with this
whole project?
And hopefully I've conveyed some of, what
I think of as like, the real importance of
the project.
So first I think that a key question here
is kind of, it reminds me a little bit of
Tetlock's work on Superforecasters.
So, obviously a lot of social scientists have
done work kind of identifying people who are
Superforecasters, where they seem to be kind
of robustly more accurate than many people,
they're robustly accurate across time when
it comes to forecasts, and they work.
We've found other features of them like working
in groups really helps, and so on.
And one question is whether we can identify
good human judges, or we can train people
to become essentially super judges.
So why is this helpful?
And this is just kind of one way of framing
the ways in which social scientists could
help with this project, I think.
So, firstly, if we do this, we will be able
to test how good human judges are, and we'll
see whether we can improve human judges.
This means we'll be able to know if a human...
or at least try and find out whether humans
are above this kind of positive amplification
threshold.
So, are ordinary human judges that we would
be using to judge debate kind of good enough
to see like an amplification of their good
features, is one question.
Another question is... or, sorry, another
reason to do this is that it improves the
quality of the judging data that we can get.
If people are just generally pretty good,
rational, assessing debate and fairly quick,
then this is excellent given the amount of
data that we anticipate needing.
Basically, improvements to your data it can
be extremely valuable here.
So yeah, the benefits of this, positive amplification
will be more likely during safety via debate,
and also will improve training outcomes on
limited data, which is very important.
Okay.
So this is one way of kind of framing why
I think social scientists are pretty valuable
here, but there's lots of questions that we
really do want asked when it comes to this
project.
And this is just like, I think this is going
to be true of other projects like asking human
questions.
It's basically to note that the human component
of the human feedback is quite important.
And getting that right is actually quite important.
And that's something that we anticipate social
scientists to be able to help with, more so
than like annual researchers who are not working
with people, and their biases, and how rational
they are, etc.
These are questions that are the focus of
social sciences.
So one question is, how skilled are people
as judges by default?
Can we distinguish good judges of debate from
bad judges of debate?
And if so, how?
Does judging ability generalize across domains?
Can we train people to be better judges?
Like, can we engage in kind of debiasing work,
for example?
Or work that reduces cognitive biases?
What topics are people better or worse at
judging?
So are there ways of like phrasing the questions
so that people are just better at assessing
them?
Are there ways of structuring the debate that
make them easier to judge, or restricting
debates to make them easier to judge?
So we're often just showing people a small
segment of a debate, for example.
Can people work together to improve judging
qualities?
These are all kind of outstanding questions
that we think are important, but we also think
that they are empirical questions and that
they have to be answered by experiment.
And so this is like, I think, important potential
future work.
So we've been thinking a little bit about
what you would want in experiments that try
and assess this in humans, like how good are
they debating... sorry, how good are they
at judging a debate?
Because ideally, AI agents would be doing
the debate in the long run.
So one is just that there's a verifiable answer.
We kind of need to be able to tell whether
people are correct or not, in their judgment
of the debate.
The other is that there is a plausible false
answer, because if you have a debate, if we
can only train and assess human judging ability
on debates where there's like no plausible
false answer, we'd get this false signal that
people are really good at judging debate.
They always get the true answer, but that's
because it's always a really obvious question.
Like, "Is it raining outside?"
And the person can look outside.
We don't really want that kind of debate.
Ideally we want something where evidence is
available so that it grounds out in... humans
can have something that grounds out the debate.
We also don't want debates to rely on human
deception.
So things like tells in poker for example,
we really don't want that because like, AI
agents are not going to have normal tells,
it would be rather strange, I suppose, if
they did.
Like if they had stuttering or something.
Debaters have to know more about the question
as well, because the idea is that the AI agents
will be much more capable and so you don't
want a situation in which there isn't a big
gap basically between the debater capabilities,
and the judge abilities.
Maybe some of these things, these ones feel
kind of like pretty essential.
These ones are sort of desires, I guess.
So one is that biases are present.
How good are humans when there's bias with
respect to the question?
There are representative segments of the debate
that we can actually show people, the questions
aren't too hard, like it's just not impossible
for humans to answer them, or judge debates
about them.
But they also mirror some of the difficulties
of statistical debate, i.e, about probabilities,
rather than about outright claims.
And finally that we can get enough data.
And so, one thing you might notice is that
there starts to be kind of tensions between
a lot of these desiderata.
So that there's a plausible false answer,
is in a bit of tension with the idea that
the question isn't too hard.
If a question is like... and the same is true
of like, the question isn't too hard, and
the question meriting statistical debate.
Statistical debate is generally pretty hard
to evaluate, I think, for people.
And also if... it's also quite important that
we be able to model it.
Debaters knowing more, and that we can get
enough data is like another thing.
It's just harder to kind of train if we need
debaters that know a lot more than judges,
and it's harder for judges to evaluate debates
of this form.
Okay.
So I'm going to show you to debate.
This is just to give you some of the difficulties,
I guess.
So this was a program set up where we would
show a judge a kind of blank screen.
So imagine you're not seeing the dog that's
here.
Two debaters, two human debaters, sit in the
same room, and they have this picture of a
dog in front of them.
And one of them is selected to lie, and one
of them is selected to tell the truth.
And what they can do here is they can select
areas, and describe to the human judge what
they see in that area.
And all that the judge is going to see is
kind of their blank screen with the relevant
section selected.
And then they can each make claims about what
is in that section.
So here red is saying, it's a dog, here's
its long, floppy, ear.
Blue, is saying, no, here's one of its pointy
ears.
So he's trying to point to a smaller area
where it looks kind of pointed.
That does look like an ear slope to the right,
but if it really was, then part of the head
would be here, instead there's a brick.
The ear's pointing out from behind the bricks.
The dog is in front of the bricks.
If it were behind her, there would be an edge
here, but the rectangle is all the same color.
And then you get like a resignation.
And at the end of the debate they can show
a single pixel.
And the question was something like, if all
you can show, all you can do is have a debate
and show a single pixel, can you get people
to have accurate beliefs about the question?
And basically we saw like, yes, debates were
fairly good.
In this kind of case, you might think that
this is pre-synthetic.
So one of the things that we're kind of thinking
about now is like, expert debaters with a
lay judge.
And I'm going to show you something that we
did that's kind of fun, but I never know how
it looks to outsiders.
So, we had to debate that was of this form.
This was a debate actually about quantum computing,
where the idea is were we going to debate
this question?
So we have two kind of, we say, experts, but
people who understand the domain, one of them
is going to lie and one is going to tell the
truth.
So we had blue say, red's algorithm is wrong
because it increases alpha by an additive
exponentially small amount each step.
So it takes exponentially many steps to get
alpha high enough.
So this was like one of the kind of claims
made.
And then you get these set of responses.
I don't think I need to go through all of
them.
You can see kind of the form that they take.
But it grounds out in this claim from... we
allowed, like, certain restricted claims from
Wikipedia.
So, blue ends this with the first line of
this Wikipedia article says that, the sum
of probabilities is conserved.
Red says, an equal amount is subtracted from
one amplitude and added to another, implying
the sum of amplitudes are conserved.
But probabilities are the squared magnitudes
of amplitudes, so this is a contradiction.
This is I think roughly how this debate ended.
But you can imagine this is like a really
complex debate in a domain that the judges
ideally just won't understand, and might not
even have some of the concepts for.
And that's the kind of difficulty of debate
that we've been looking at.
And so this is one thing that we're in the
kind of early stages of prototyping, and that's
why I think it seems to be the case that people
actually do update in the kind of right direction,
but we don't really have enough data to say
for sure.
Okay.
So I hope that, I've kind of given you an
overview of places, and just even like a restricted
set of places in which I think like social
scientists are going to be important in AI
safety.
So here we're interested in experimental psychologists,
cognitive scientists, and behavioral economists,
so people who might be interested in actually
scaling up and running some of these experiments.
If you're interested in this, please come
to my office hours after this talk, or email
me, because we would love to hear from you.
So thanks.
All right.
We don't have too much time for questions,
but if you want to sit, we can take a couple.
Just for starters, as I was kind of watching
this I'm wondering, how much of this is real
at all or coming from an actual system versus
like, do you have humans sort of playing the
role of the agents in these examples?
Yeah, so at the moment, the idea is that we
want ultimately the debate to be conducted
by AI, but we don't have the language models
that we would need for that yet.
And so we're using humans as a kind of proxy
to test the judges in the meantime.
So yeah, all of this is done with humans at
the moment, yeah.
So you're faking the AI?
Yeah.
The-
To set up the scenario to train the judges
to, to evaluate the judges on their ability
to later provide?
Yeah.
And some of the ideas like I guess you don't
necessarily want all of this work to kind
of happen later and once you... a lot of this
work can be done before you even have the
relevant capabilities, like, have AI perform
the debate.
So that's why we're using humans just now.
Yeah, totally understood.
Let's see.
A couple of questions coming in.
I guess one note also for the audience, if
you didn't see Jan Leike's talk yesterday,
he showed some examples from the work that
his team has done on video games, that very
much matched the plots that you had shown
earlier, where up to a certain point, the
behavior sort of matches the reward function-
Yeah.
And then at some point they sort of diverge
sharply as the agent finds a loophole in the
system.
So that can happen even in like, Atari Games,
which is what they're working on.
So obviously it gets a lot more complicated
from there.
So, questions from the audience.
How would you train, in this approach, you
would train both the debating agents and the
judges.
So in that case who evaluates the judges and
based on what?
Yeah, so I think it's kind of interesting
where we want to identify how good the judges
are in advance, because it might be hard to
assess... like later, if people are just judging
on verifiable answers, you could in fact,
presumably assess the judges as even when
you're doing training, but they're going to
be... here, we can kind of ground out debates
in questions with verifiable answers.
Ideally you want it to be the case that at
training time, I think, you've already identified
judges that are fairly good.
And so ideally this is part of this project
is to kind of assess how good judges are,
prior to training.
And then during training you're giving the
feedback to the debaters.
So yeah, ideally some of the evaluation can
be kind of front loaded, which is what a lot
of this project would be.
Yeah, that does seem necessary as a casual
Facebook user.
I think the negative amplification is more
prominently on display oftentimes.
Or at least more concerning to people, yeah,
as, like, a possibility.
So, kind of a related question, how will you
crowdsource the millions of human interactions
that are needed to train AI across so many
different domains, without falling victim
to trolls, lowest common denominator, etc.?
The questioner cites the Microsoft Tay chatbot
that sort of went dark very quickly.
Yeah.
So the idea is you're not going to just be
like sourcing this from anyone.
So if you identify people that are either
good judges already, or you can train people
to be good judges, these are going to be the
pool of people that you're kind of using to
get this feedback from.
So, even if you've got a huge number of interactions,
ideally you're kind of sourcing and training
people to be really good at this.
And so you're not just being like, "Hey internet,
what do you think of this debate?"
But rather like, okay, we've got this set
of really great trained judges that we've
identified this wonderful mechanism to train
them to be good at this task.
And then you're getting lots of feedback from
that large pool of judges.
So it's not kind of sourced to kind of, to
anonymous people everywhere, but rather like
you're kind of interacting fairly closely
with this set of people, who are giving you
lots of-
But at some point, you do have to kind of
scale this out, right?
I mean in the bike example, it's like, there's
so many bikes in the world, and so many local
hills-
Yeah.
So, do you feel like you can get a solid enough
base that, that sort of... becomes not a problem?
Or, how does that...
Yeah, I think there's going to be a trade-off
where you need a lot of data, but ultimately
if it's not great, so if it is really biased,
for example, it's not clear that that additional
data is going to be helpful.
So if you get someone who is just like massively
cognitively biased, or biased against groups
of people, or something, or just like is it
going to be dishonest in their judgment?
This is not going to be like... it's not going
be good to get that additional data.
So you kind of want to scale it to the point
where you know you're still getting good information
back from the judges.
And that's why I think in part this project
is really important, because one thing that
social scientists can help us with is kind
of identifying how good people are.
So if you know that people are just generally
fairly good, this gives you a bigger pool
of people that you can appeal to.
And if you know that you can train people
to be really good, then this is like, again,
a bigger pool of people that you can appeal
to.
So yeah, it's like you do want to scale, but
you want to scale kind of within the limits
of still getting good information from people.
And so ideally this would do this mix of letting
us know how much we can scale, and also maybe
helping us to scale even more by making people
bear this quite unusual task of judging these
kind of debates.
So we are a little over time, we won't have
time to go through all the questions that
are coming in, but you can speak with Amanda
more at office hours immediately following
this talk, right when we're headed into break.
So let's just do one last question for this
session, which is how does your background
as a philosopher inform the work that you're
doing here?
Yeah.
I have, I guess, a background primarily in
formal ethics, which I think makes me sensitive
to some of the issues that we might be worried
about here going forward.
So you know, people think about things like
aggregating judgment for example.
Strangely I found that like, having backgrounds
in things like philosophy of science can be
weirdly helpful when it comes to thinking
about experiments to run.
But for the most part, I think that my work
has just been to kind of help prototype some
of this stuff.
I see the importance of it.
I kind of, I'm able to foresee some of the
kind of worries that people might have.
But for the most part I think we should just
try some of this stuff.
And I think that for that, it's really important
to have people with experimental backgrounds
in particular, so the ability to run experiments
and analyze that data.
And so that's why I would like to kind of
find people who are interested in doing that.
So I'd say philosophy's pretty useful for
some things, less useful for running social
science experiments than you may think.
Alright.
Well for more, you'll have to come to office
hours, which you can do immediately after
this session.
How about a round of applause for Amanda Askell?
