Hi this is Jeff Heaton, welcome to
Applications of Deep Neural Networks
with Washington University. In this video
we're going to see how to actually
extract the weights from a neural
network created by Keras and put those
weights into an equation so that we can
actually calculate the output from the
neural network and see that this there's
really no magic to this process, that it
is simply weights multiplied against the
inputs that produces a final output. For
the latest on my AI course and projects
click SUBSCRIBE and the BELL next to it
to be notified of every new video. This
is code from Keras and TensorFlow, but
this applies to really just about
anything. I'm going to show you how to
actually extract the weights from the
neural network so that we can then put
them onto a diagram and actually
actually calculate it and come up with
the same number that TensorFlow would
have for this. We're going to use an
exclusive-or neural network. That's the
the XOR function, these are the inputs to
it. [0 0] the usual truth table for any
sort of an and or an or this is the
expected output. The thing to remember
with XOR is if the two inputs are the
same, it's going to be zero. If the two
outputs are different 0 & 1 vs. 1 0, it's
going to be 1. Here we set up a neural
network, it has two inputs and a hidden
layer of two. We're going for the
absolute smallest neural network, just to
show that you don't really need that
much to calculate an exclusive-or. Also,
since we are going to calculate this by
hand, we're just being lazy. We don't, I
don't want to give you a truly deep deep
neural network and then calculate it by
hand. That would not be fine, so we're
going to optimize it with mean square
error and the final output is going to
be one neuron. It's a regression
neural network, you can do XOR is
classification or regression, in this
case I'm doing it as regression. I
basically trained it. Here, I am training
it for a hundred thousand
epochs. It may take longer because
there's so few weights in this. That your
initial random values of your weights
are really going to have a lot of
determination
the success of your training. We could do
this and way fewer than this many epochs,
this is basically grilling a cheese
sandwich with a jet engine, but I am
really not trying to show you how to
tune these things right now, I just want
to get the weights set up so that
they're good for a exclusive-or network.
We're gonna put those on a diagram and
show you that basically you can then
calculate the same output, so you can see
the weights behind what Keras gives you.
I trained it. I predict it this is good
for the for the neural network because
each of these four correspond to these
four up here in the input, this is
scientific notation, so to the -4
that's zero point zero, zero, zero that
number so the these two are effectively
0 and these two in the middle are
effectively 1, so this may look weird if
you're not used to seeing numbers like
this, but this is a great output. The, the
neural network has trained quite well, so
then what we're able to do is, I write, I
wrote this little program here, this
dumps the weights, so you're seeing that
the layer 0 counting starting at 0, the
weight from the bias, to the next layer
layer 1, again we're counting with 0,
neuron one, neuron neuron zero, neuron 1,
these are the biases and these are the
actual weights so there's not a lot of
weights and biases in this neural
network, it's it's a fairly small one so
I'm gonna show draw a diagram here in
just a minute, and we're going to copy
all of these values up onto the diagram,
then we're going to calculate it by hand.
I give you the code down here to
calculate it by hand as well, if you want
to do it you set the input to 1, and 0, so
1 and 0, and an XOR those are different,
so it should be 1, so it outputs a 0.96.
Now since we're calculating this by hand
i truncated quite a few of these numbers
so 1.29 becomes 1.3 and we'll do that
when we diagram it so we lose a little
bit of accuracy, but these are the two
input neurons, we are going to feed this
values 0,
and one so that's what we're calculating
for, you could calculate it really for
just about any other value that you
wanted to, but for the other four that
you would have in the exclusive-or. We're
also going to have the bias neuron, so
like we saw in earlier class videos,
you're essentially calculating a
weighted sum over and over and over,
again for a neural network, so we're
going to calculate the value for hidden
or on 0 ,H0, H0 has inputs from those
previous 3.
These all have weights as we saw from
the output from the program before. So,
I'm just going to copy those in
so this is going to become essentially a
weighted sum, that we're going to use to
calculate what hidden zero is actually,
what its value actually is, so to do that
we're going to multiply the input zero
times its weight which is 1.3,
plus 1 times 1.3,
that one is from the first which is
actually the second input neuron, and
then we have to add the bias, and we
basically perform that calculation, which
ends up being zero, so the value for
hidden in hidden neuron 0 is zero.
Next, we're going to calculate the value
for hidden or on one which is the second
one has a similar sort of thing going on,
it has three weights coming into it. It's
going to copy those values from the
output that we had from the Keras
program so that is simply 1.2,
we are calculating basically, the
weighted sum again the first input I0
row drops out second one contributes
that 1.1, 1.2 to it and
then the intercept, or the bias, for, for
this for the second hidden neuron is
zero, so this whole thing together is
basically equal to 1.2, then
this value we have to pass it through
the through the activation function both
of these needed to go through the
activation function, that is the
rectified linear unit, so that is
basically going to be the max of 0 and
1.2, which is also 1.2, so the output from this hidden
neuron here is 1.2, same sort
of deal up here to apply its activation
function, it's going to be max because
it's also the value in max zero. Zero is
zero,
the activation functions that you can
use, I may give you a different one for
the for the midterm examination on this
one, the only other ones that you will
ever see at least in this class are
sigmoid, hyperbolic tangent there are others
but these are for regression if you're
doing a classification, you might also
see the softmax, but focus primarily on
these now. The output the final layer, so
that we actually get what this this
neural network is providing for us.
This is going to have similar to before.
We do have another bias connection so,
you're going to have essentially three
and in-bound connections again and we're
essentially going to just copy the
weights that we had from the output in
Keras, there is zero and this results
and the final equation that ties this
whole thing together and that is going
to basically be the output from, from,
hidden hidden 0 times 1.6 plus
1.2 times 0.8 and
see the 1.2 times the 0.8 and then plus, the plus the
the bias or the intercept which is zero,
and then this whole thing equals 0.96,
that's your final output.
That is approximately equal to 1.0 which
is what you would have expected from a
exclusive-or of zero and one there you
have it that is the calculation for this
neural network, done using the weights
that Keras is actually trained for us, so
we can see that there is really no magic
there, this simple of a neural network
you could have actually just hand coded
these weights that would have not been
difficult. I'll probably do a video on
that as well so that you can see how to
actually hand code one of these you do
need about this size of neural network.
You definitely need the hidden layer.
Thank you for watching this video, in the
next video we're going to see more
advanced training techniques that we can
use with a neural network to to get
better results, and to measure our error
in a variety of different ways. This
content changes often, so subscribe to
the channel to stay up to date on this
course and other topics and artificial
intelligence.
