So now that we understand the concepts of artificial neural networks and Deep Learning let's mess around
with it! It's surprisingly easy to do.
The folks behind TensorFlow at Google have created a nice little web site called playground.tensorflow.org
that lets us experiment with creating our own neural networks
and you don't have to write a line of code to do it,
so it's a great way to get sort of a hands-on intuitive feel of how they work,
so let's dive in. So head over to playground.tensorflow.org and you should see a screen like this,
you can follow along here or just watch me do it, but I definitely encourage you to play around with
it yourself and get sort of intuitive hands-on feel of how Deep Learning works, this is a very powerful
thing if you can understand what's going on in this web page.
So what we're trying to do here is classify a bunch of points just based on their location in this 2D
image.
So this is our training dataset, if you will,
we have a bunch of points here and the ones in the middle are classified as blue and the ones on the
outside are classified as orange,
so our objective is to create a neural network that given no prior knowledge can actually figure out
if a given point should be blue or orange and predicts successfully which classification it should be.
So think of this is our training data,
OK? We know ahead of time what the correct classifications are for each one of these points and we're
going to use this information to train our neural network to hopefully learn that stuff in the middle
should be blue and stuff on the outside should be orange.
Now, here we have a diagram of the neural network itself and we can play around with it,
we can manipulate it, we can add layers to it, take layers out, add more neurons to layer as whatever you
want to do.
Let's review what's going on here.
So first of all we're selecting the dataset that we want to play with, we're starting with this default
one that's called circle,
the inputs are simply the x and y coordinates, the vertical and horizontal position of each data point,
so as our neural network is given a point to classify, all it has to work with are those two values,
it's horizontal position and its vertical position and those start off as equally weighted being horizontal
or vertical, so we can define the position of any one of these points in terms of its horizontal and
vertical position.
For example, this point here would have a horizontal position of negative one and a vertical position
of about negative 5.
And then we feed it into our network,
you can see that these input nodes have connections to each one of these four neurons in our hidden
layer and we can manipulate the weights between each one of these connections to create the learning
that we want,
those in turn feed into two output neurons here that will ultimately decide which classification we
want at the end of the day.
So keep in mind this is a binary classification problem, it's either blue or orange,
so at the end of the day we just need a single signal really
and that's what comes into this output here.
Let's go ahead hit play and see what happens.
What is going to do is start a bunch of iterations where it learns from this training data, so we're
going to keep feeding it input from this training dataset and as it goes, as it iterates through it, it
will start to reinforce the connections that lead to the correct classifications through gradient descent
or some similar mechanism, right?
And if we do that enough times, it should converge to a neural network that is capable of reliably classifying
these things. So let's hit play and just watch it in action.
So keep your eye on that image to the right there.
All right you can see that we've already converged on a solution,
I can go ahead and pause that now.
And pretty cool stuff.
So you can see it has successfully created this pattern where stuff that fits into this middle area
here is classified as blue and stuff on the outside is classified as orange. So we can dive into what
actually happened here,
the thickness of all these connections represent their weights,
so you can see the individual weights that are wired between each one of these neurons,
we start off here,
you can see these are more or less equally weighted,
well, not exactly equal either,
some of these are kind of weak, but what it leads to is this behavior in the middle.
So we start off with equally weighted x and y coordinates, those go to this layer here,
so, for example, this hidden layer here, this neuron is saying "I want to weight things a little bit more
heavily in this corner,"
OK? And things that are like in the lower left hand corner not so much,
and then this other one is picking out stuff on the top and bottom, this one it's a little bit more diagonal
to the bottom right
and this one's even more bottom right heavy
and if you combine these things together, you end up with these output layers that look like this.
OK?
And so we had, we end up with these two blobby things where were sort of giving a boost to things on
the right and giving a boost to things that lie within sort of this more blobby circular area,
and then when we combine those together we end up with our final output that looks like this.
Now this might look different from run to run,
you know, there is some random, some randomness to how this is all initialized.
Do we actually even need a deep neural network to do this though? One optimization thing is to remove
layers to see if we can get away with it, maybe we don't even need Deep Learning,
I mean, really this is kind of a simple thing,
you know, stuff in the middle is blue, stuff on the outside is orange.
Let's go ahead and remove one of these neurons from the output layer,
again all we need is a binary result anyway,
Can it still work?
It does!
In fact it's just as quick,
so do I even need that layer at all?
Let's go ahead and remove that final layer at all.
Still works, right?
So for this very basic problem I don't even need to do Deep Learning,
all I have here is a single layer,
so this is just, it's not even a multi-layer perceptron,
it's just a perceptron.
Do I even need four neurons in there?
Well I think maybe I do, but this one here isn't really doing much, right?
Al is doing is basically doing a pass through and the inputs coming into it have been weighted down to
pretty much nothing,
so I bet you don't even need that one,
let's get rid of it.
It still works.
Isn't that kind of cool?
I mean, think about that,
we only have three artificial neurons and that's all it takes to do this problem,
I mean, compare that to the billions of neurons that exist inside your head.
Now we probably can't get away with less than that,
Let's go ahead and try to do two neurons and see what happens.
Yeah that's just not going to happen,
Right?
So for this particular problem all you need is three neurons, two won't cut it.
Let's play around some more, let's try a more challenging dataset,
OK?
So here's a spiral pattern and you can tell this is gonna be harder because we can't just say stuff
in this corner is going to be this,
this classification, like we need a much more finer grained way of like identifying these individual
spirals,
and again we're going to see if we can just train a neural network to figure that rule out on its own
and well, obviously two neurons will cut it,
let's go back to four,
let's see if that's enough,
I bet it isn't.
You can see it's trying, but it's really struggling.
You can let this run for a while and you can see it's starting to kind of get there, you know, the blue
areas are converging on some blue areas
and it's, it's really trying hard, but it's just not enough neurons to pull this one off.
Let's go ahead and add another layer, let's see if that helps.
You can see it's doing more complicated things now that it has more neurons to work with, but still can't
quite get to where it needs to be.
Let's add a couple more neurons to each layer.
Generally speaking you can either add more neurons to a layer or add more layers,
it's got to produce the same results, but it might affect the speed in which it converges depending on
which approach you take.
It's just fascinating watching this work, isn't it?
This one got stuck, it still can't quite pull it off.
Let's add one more layer.
This is actually a very common pattern you'll see,
you start off with a lot of layers at first and kind of like narrow them down as you go,
OK? So we're going to go to a initial input layer of six neurons to a hidden layer for neurons and then
a layer of two neurons which will ultimately produce a binary output at the end.
I think it's getting there.
Here we go.
Wow! OK! So technically... it's still kind of like refining itself, but it kind of did it, right?
I mean, now this is what we call "overfitting" to some extent, you know, I mean obviously it has these like
tendrils, are kind of cutting through here and that's not really part of the pattern we're looking for,
it's still going though.
Those tendrils are kind of getting weaker and weaker,
so, you know, it still doesn't have quite enough neurons to do exactly the thing that we would do intuitively,
but, I mean, still, I mean, this is a pretty complicated classification problem,
it figured it out and maybe overfitting a little bit, but it figured it out
and all we have is what, 12 neurons here?
I mean, that's insane, right?
Now, another thing I want to talk about here, too, is that it kind of illustrates the fact that once you
get into multiple layers it becomes very hard to intuitively understand what's going on inside the neural
network.
This gets kind of spooky, you know?
I mean, what does this shape really mean?
I mean, once you have enough neurons it's kind of hard to fit inside your own head
what these patterns all really represent, I mean, the first layer is pretty straightforward,
you know, it's basically breaking up the image into different sections,
but as you get into these hidden layers things start to get a little bit weird as they get combined
together.
Let's go ahead and add one more, shall we?
Actually let's add two more to this output layer and add one more layer at the end.
Let's see if that helps things converge a little bit more quickly.
All right, start to struggle a little bit.
See that? Like it's actually got a spiral shape going on here now.
So with those extra neurons he was able to do something more interesting.
We still have this this little spike here that's doing the wrong thing and it can't seem to quite think
its way out of that one.
Give it a few more neurons, though I might be able to figure it out,
these ones are also misclassified, but I find it interesting that it actually created a spiral pattern
here on its own,
so maybe with a few more neurons or one more layer you could actually create an even better solution,
but I will leave that as an exercise for you.
That's, you know, to play around this. I really encourage you to just mess around with it and see what
kind of results you can get.
The spiral pattern is in particular an interesting problem.
Just explain some of the other parameters here, we're doing a classification here that's what we were
doing throughout this section; the Activation function,
we talked about not using a step function and using something else,
some other ones that are popular, ReLU is actually very popular right now;
Regularization function, we haven't talked about yet; the Learning rate is just basically the step size in
the gradient descent that we're doing,
so you can adjust that if you want to as well,
let's see if ReLU actually makes a difference.
I would expect it to just, you know, affect the speed.
Oh my gosh! Look at that!
That's pretty darn close to what we want, right?
I mean, there's a part from this little tiny spike here which isn't really even there,
it's a little bit of overfitting going there, but we have basically created that spiral shape just out
of this handful of neurons.
God! I could do this all day guys and I hope you will too,
you know, just play around, this it's so much fun and it gives you such a concrete understanding of what's
going on under the hood,
I mean, look at this hidden layer here, that's where the spiral shape is starting to emerge and come
together
and when you think about the fact that your brain works in very much the same way it's quite literally
mind blowing. Anyway, mess around with this,
it's a really great exercise and I hope you have some fun with it.
