CHRIS OLAH: In the
last few years,
we've seen neural
nets accomplish a lot
of really remarkable things.
We've seen them learn to
understand images, understand
voices, translate
sentences, and even make
really beautiful images.
But I'd like to take a minute
and step back and ask really,
what is it that they're doing,
and how are they doing it?
What are they?
Sorry.
You'll have to forgive me.
I'm not entirely used to--
maybe I'll do it this way.
So neurons, the sort of neurons
that we deal with-- they're
inspired by biological neurons.
But they're really toys.
They're cartoon versions
of biological neurons.
They're not
biologically plausible.
They're very simplified.
They have a bunch of inputs.
And they have one output.
And they're going to go and
based on those inputs, fire
or not fire.
So some inputs-- we're
going to go and have weights
for all of those inputs.
And those will describe how
the inputs affect the neuron.
Some inputs go and encourage
their neuron to fire,
and then will have
a positive weight.
And some neurons will go and
discourage it from firing,
and then will have
a negative weight.
And we also have a bias because
some neurons should fire
a bit more easily than others.
And so the bias describes that.
And if we want, we can
go in, sort of formulize
that a little bit.
And if we want to compute
whether a neuron's going
to fire or not fire,
we take all our inputs,
we multiply them by their
weights, we add our bias.
And if we're greater than
0, we fire an output of 1.
And if we're less
than 0, we don't fire.
So that's a binary
threshold neuron.
And those binary
threshold neurons
are what we call universal.
You could combine them
together if you wanted
to go and compute any function.
Actually, I missed something at
the beginning, which will say.
So I'm starting
with a very sort of
the standard formal definition
of what neural nets are.
And it might seem a
little bit intimidating,
or it might seem
a little bit dry.
But what I'm hoping to do
is go through this quickly.
And then I want to show you that
there's a visual interpretation
of what neural nets are doing
that we can use to completely
sort of-- a completely different
way of thinking about it.
So let's just go and talk about
this for a few more minutes.
And then I'll tell you that.
So these binary threshold
neurons, it turns out,
are hard to work with.
And so we actually
in practice, we
work with a slightly different
thing-- these sigmoid neurons
or other kinds of
neurons, which they're
the exact same except instead
of just firing or not firing,
we're going to go and
continuously transition
between not firing and firing.
And we usually
think about neurons
not in terms of
neural nets and not
in terms of individual neurons,
but in terms of layers.
So here we have a neural net
layer with three neurons.
And it has three inputs.
And for every neuron,
there's three weights
that go and describe
how it's going
to relate to those inputs.
We multiply the inputs by
those weights, add a bias,
apply our non-linearity,
and we get our output.
And if we want, we can write
that down in equations.
So every neuron, we go and
we multiply by the weights--
multiply the inputs by the
weights-- add the bias,
and apply the non-linearity.
You can write that
down as a vector
if you want and think about
the weights as being a matrix.
So we multiply our input
by the weight matrix,
add our bias factor, and
apply a non-linearity.
So now I'd like to offer a
geometric perspective on this.
So there's sort of
a visual way that we
can think about what
neural nets are doing.
So we can think about the
output of a neural net layer
as a point in a space.
We can think about, well,
if we only have two neurons,
we can think about it as
the neuron 1 as the x-axis.
And we can think about that
one as points on the plane n.
Neuron 1 is the x-axis.
If it fires a lot, we're
on the right-hand side.
If it doesn't fire very much,
we're on the left-hand side.
And neuron 2 will be the y-axis.
So in this first case, neuron
1 didn't fire very much,
but neuron 2 did fire.
And so we're sort
on the left and up.
And in the next layer, one
of the neurons fired a lot.
And so we're on the
right-hand side.
But the other one
didn't fire very much.
So we're sort in the middle.
And based on that
technique, we can now
visualize what it is
neural net layers do.
So what do they do?
They multiply by
a weight matrix,
doing this skewing thing.
They add the bias
vector, translating.
And then they apply
the non-linearity,
stretching the space.
So that's all neural nets do.
They go and they apply
linear transformation,
they translate, and they
apply a non-linearity.
That's all they do.
They apply a linear
transformation,
they translate, and then
they apply a non-linearity.
So let's walk through a
concrete example, just
to understand that
a little bit better.
So here's a little problem.
We have two curves, and
we'd like our neural net
to-- it's going to
go and tell us--
we want it to tell us-- well,
our model to go and tell us--
that points on the
blue curve are blue,
and points on the
red curve are red.
And we could use-- there's
a simple technique that we
use in machine learning
sometimes-- sorry,
I can't see very
well-- where we go
and we use what we
call a linear model.
So it just divides
the space with a line.
And things on one side are blue,
and things on one side are red.
That's a pretty widely used
technique in machine learning.
But it doesn't work here.
And it doesn't work
because there's
no line that can divide
the blue from the red.
But neural nets can solve this.
And the reason they can solve
it is that they bend the space.
They bend to the space.
And with the space,
they bend to the data.
And so even though
they couldn't initially
go and solve the problem,
they transformed the problem.
And that's what neural
nets do that's so
different from other techniques.
They go and they
start with an input.
But they transform it to make
it easier to solve the problem.
And they'll often transform
it multiple times.
Layer by layer,
they'll transform it.
And in the end, even
if they couldn't
solve the problem
initially, they've
learned a transformation.
They've learned a way to
understand the problem that
makes it easy to solve.
And that seems like a very
significant difference
between neural nets
and other techniques
that we've used for
machine learning.
So here's another example.
We have two spirals.
And the neural net
just pulls them apart
and untangles them so that
it can go and separate them.
So every pulse is a
layer of the neural net.
And it's transforming the data.
And so even though it
was initially tangled,
it no longer is.
So it starts with a tangled
version, but it bends the data
and pulls it apart so that
it can go and classify
and can solve the problem.
It's what neural nets do.
So those were toy problems.
They're sort of
two-dimensional problems.
But of course, we use
neural nets for things
like recognizing images
and human voices and things
like that.
So maybe let's walk through
a slightly more real problem.
So this is a data
set called MNIST.
Some people may be
familiar with MNIST
because it's pretty widely
used as sort of a first problem
if you're working
on it on trying
to enter in machine learning.
And so what it
consists of is a bunch
of images of handwritten digits.
So we had a bunch of
people sit in a room
and write down
handwritten digits.
And the goal usually is we go
and we look at these images
and we try to say, aha, that
one's a 5, that one's a 0,
and that's a 4.
That's a 1.
And, well, it probably
seems like an easy problem--
[INAUDIBLE] it's a really
easy problem for you.
Your brain's really,
really good at it.
But it turns out it's actually
kind of a tricky problem.
So we can understand that
image as a bunch of numbers,
as a two-dimensional
grid of numbers.
So when it's white,
we'll put a 0 there.
And when it's black,
we'll put a 1 there.
And if it's gray, we'll put some
number in between-- maybe 0.5.
And if we want, we can
flatten that grid just
into a list of
numbers, into a vector.
So a 28 by 28 image
is 784 numbers.
So we can have a
784-dimensional vector.
And now that presents a
little bit of a problem
because I don't know
about you, but I
have trouble picturing points
in 784-dimensional spaces.
[LAUGHTER]
So thankfully, there
is some techniques
we have called dimensionality
reduction techniques.
So this is one.
Every point here
is a single image.
So this is a 0.
These are all zeroes over here.
All the red points
are different zeroes.
And these green points
over here-- they're threes.
And those orange ones are ones.
Those yellows are twos.
Those blues down
there are sixes.
Those aquas are fives.
And what we can see is that
they're tangled together.
And that makes it kind of hard
to go and solve the problem,
because they're
tangled together.
But if we try to solve this
problem with a neural net,
here's what it does.
It takes that tangled version,
and even with just one layer,
can go and do a pretty good job
of going and pulling it apart.
It stretches them apart
and disentangles them.
And then it goes
and classifies them.
And even-- this is a
very simple neural net.
It's just one layer in the
middle and then one layer
to classify-- one layer
to transform the data,
one layer to classify.
And even that can get 98%
accuracy solving this problem.
And neural nets are
pretty like that.
And again, all they do is
they go and they transform
the problem, often
multiple times,
and then make it easier
to go and classify it
or to go and-- they by
transferring the problem,
they make it easier to solve it.
So neural nets are pretty hard
to understand in some sense.
Even though their sort
of the underlying ideas
are simple-- they're sort of
doing this bending of data--
and you can understand
what's going on locally,
it's often hard to understand
what the neural net as a whole
is doing.
And I guess it's--
I'm not a chemist,
but I imagine it's a
little bit like chemistry,
where we understand
atoms fairly well.
But that doesn't mean we
understand how proteins fold.
And so even when you understand
what's going on locally,
it can be hard to
understand the whole thing.
It's sort of a forest
and the tree situation,
where we can understand the--
maybe we can see the trees,
but it's hard to see the
entire forest as a whole.
So that's a very active
area of research.
How can we understand what's
going on inside neural nets?
And that's kind of what brings
us here today, because it turns
out that going-- and as
we've in the last year
suddenly made a lot of
traction on this problem,
we've created some really
incredibly beautiful images.
And I guess you've seen
them on your way in,
and you can look
at them more later.
But I think it's
also worth thinking
about another kind of beauty
that comes along with this.
In some sense, we don't
know what it means to see.
We don't know what it
means to understand voices.
We don't know what it means
to understand language.
But neural nets are starting
to be able to do these things.
And we're starting to
understand neural nets.
And so it might be that in
the not so distant future,
we'll be able to
use them to come up
with fairly satisfying answers
to what it means to see
and what it means to
understand language.
And that would
tell us something,
I think, really interesting
about the world.
And I think it would also tell
us something really interesting
about ourselves.
Thank you for your time.
[APPLAUSE]
