the analogy for thinking through this
stuff in machine learning is the professor analogy
you already know all
this stuff
because you've taken exams and if you've
ever set exams, even better!
so let's have a quick look
you're the professor and your student is
going to take a calculus exam tomorrow morning
so the student has written something down from the board to study from
there we go, an equation i said no equations and look at me lying
so the entire job of the student (the machine learning system) is
to go from the inputs the left-hand side
to the outputs on the right-hand side correctly
and so this is what the student's going
to learn from going to learn really
really hard
and find a pattern that works
you can just turn the 8 on its side and there you go
and i want to point out
that in these data
where the student is learning (training)
it works! this is correct
100% accuracy
now at this point
would you go to the exam
if you were the student?
i mean, i've certainly taught some undergraduates
who i think would have gone to the exam
(i've had to grade their papers and cry a little)
but you, my dear enlightened audience, you
wouldn't just go to the exam based on
this, right?
what would you do?
what do i hear?
try another one exactly but there's an
important difference between you and my student
my student also tries another one
they try it on the exam
where it matters
whereas you have the
wisdom
to try it in the safety and comfort of your dorm room
check your understanding while you can
still go back and adjust your model
if you don't get it
so what you are doing and they're not doing is validation
that is what validation in
machine learning is all about
a safe environment to try it out where
you're still
allowed to go back because the stakes
are not high here
a soft interim test
so let's try another
one
apply our model
we get performance and...
turns out that's not how math works
and based on this signal we then can ask
ourselves
fine, should we go to the exam or should we go back to training
then the student sits there and tries
out all these
extra practice problems that you the
professor have given to them
and finally at three o'clock in the
morning you the professor
get an email from the student
"validation passes!!! i did all of the
practice problems correctly and all of
the in-class examples correctly please
give me my A+
i don't need to go to the exam because
i've done it"
and your response is
of course not!
why do you say that?
you say that because your student could
have contorted their understanding
to the in-class examples and the
validation examples and just memorized
the whole lot
maybe even without realizing
that they had memorized them
just formed some recipe some model
that fits what they've studied
it doesn't generalize to anything else
because it's just some crazy
mishmash of the peculiarities of those
examples that they worked with
and you don't know if that's what they
did because you know
they were allowed to retry both the
in-class problems and the practice
problems as many times as they want you
don't know what kind of
mad logic is going to come out of that
memorization is possible there
and so you as a wise professor are going
to say absolutely not
i'm so glad for you student that you
feel like you've gotten yourself to a
state where you're confident
that things look good for you but to
really make sure that i don't
release any really bad students out into the world
you should come to the exam and
we'll check that you know it
now, professors, by thinking through this
exam stuff
you already know the requirements for
the final exam questions
what has to be true about them the most
important thing
what's that
the student is not allowed to have seen
them before
because if the student has seen them
before and knows the answer to them
the student could be memorizing and
your entire goal as a professor in this machine learning space
is to make it so that your system cannot
win by memorization
it cannot just do the look up the old
answers insofar as i remember them
and submit that as my new answer
and the only way you catch memorization
is if you give
truly new stuff for the testing
and that's why we have that separate
test phase
