hello and welcome back to another MATLAB
video my name is patan Barr and today's
topic is semantic segmentation we're
gonna cover two things number one what
is semantic segmentation number two how
do you do it now before we get started I
want to mention that this video is going
to cover higher-level concepts I'm not
gonna be going through a specific code
example but if that's what you're
looking for go ahead and click the link
in the description below that's probably
where you want to be but if you're new
to this you want to learn about the
workflow or just rehash some of the
concepts this videos for you so let's
start with the basics what is semantic
segmentation semantic segmentation is
the process of taking an image and
labeling each pixel in that image with a
certain class let me show you what I
mean here is a picture of me at the
beach now don't be distracted by how
fantastic I look but after going through
the semantic segmentation workflow that
picture would look like this notice how
there's two classes there's person and
there's background the pixels inside of
my Silhouette
are assigned to person and the pixels
outside my Silhouette are assigned to
background now the key to semantic
segmentation is that the computer
actually did the segmenting of that
image automatically in order to do that
we have to take a bunch of images of
people standing in backgrounds and we
have to label where the people are and
label where the background is and then
we use that information to create a
convolutional neural network and we
train that network and when we put in a
new image of a person standing in some
background it'll be able to segment that
image into the two parts it's pretty
cool but now you're probably wondering
what's the practical application of this
let me give you an example let's say
that I owned a car company called
pitambar's cars and this car company
sold cars with automated driving systems
let's talk about that automated driving
system one way I could create it is with
semantic segmentation I could take a
bunch of images of the road and label
where the lanes are
where other cars are where pedestrians
are where trees are where the sky is
where the lamppost says anything I could
just label all of these things and I
could do that for thousands of images
and then I could use all of these images
to create a convolutional neural network
that when given a new image of the road
it can pick out where the lanes are
where are people where are trucks where
are buses etc etc and from there I can
implement safety features like automatic
braking or that thing that stops you
from turning when there's a car right
next to you or a parallel parking system
obviously that requires like more
algorithms but semantic segmentation is
the crux for determining where those
things are okay I hope at this point
that you're convinced on why we're
focusing on the subject now what I want
to do is walk you through the steps at a
very high level so let's jump into
MATLAB and talk about our first step
which is labeling the images I'm going
to open up MATLAB and I'll start by
opening the image label or app there's a
section here called apps and we're gonna
go down and click image labeler we'll
see this new screen pop up the first
thing we'll do is import our images so
I'll go to load and I'll add in these
images this first picture is me as a
freshman in college and the second one
is me as a senior clearly my hair grew
faster than my sense of style now in
order to do semantic segmentation we
first need to define our labels and
there's a section here that says define
new ROI label our oh I stands for region
of interest by the way I'm gonna click
that and we're gonna label eyes nose
ears and mouth I'll create a label that
says eyes and I have two options
rectangle and pixel label I want pixel
label because that will allow me to be
really specific with my selection and
that's what I need to pick to do
semantic segmentation I'll add a
description and click OK I'll create
another ROI label called ears and
another one called nose and finally
mouth and once I've created all of my
ROI labels I'm going
start labeling so let's zoom into my eye
and start drawing a mask around it we'll
finish our selection by clicking where
we started we can see that the eye now
gets highlighted let's pan over and do
the same thing for my second eye now
I'll intentionally mess this one up so I
can show you some of the other tools
oops it looks like I've overshot this
let me use the eraser to erase the part
I don't want Oh No but now I've erased
too much what will I do I know I'll use
the brush and I'll fill in the part I
actually want once I finish with the
eyes I'll click the nose label and
ylabel the nose then I'll do the mouth
and the ears and the same thing for the
second image if I did this correctly
I've turned both pictures of myself into
pictures of clowns
just kidding these are the labels while
I'm doing this process for only two
images you should really be doing this
for way more images depending on your
images you might be able to implement
some kind of algorithm so right here
there's a button that says select
algorithm click it and you can click on
create new algorithm so once you've
finished labeling all of your images you
need to export your labels so I'll go
here and I'll click to workspace and
when I do this and go back to MATLAB I
now have a ground truth object called G
truth this object has all of the
information that I gave my computer when
I labeled the images
alright let's cut back to the video and
I'll teach you how to do the remainder
of the steps at a high level alright
step one was to label our images and we
just did that step two is to create two
data stores in image label data store
and a pixel label data store now this is
very simple to do but I'm not gonna go
into code so again I'm gonna direct you
to the example which you should totally
check out after watching this video step
three is to partition your data stores
into training sets and test sets the
reason we do this is because whenever we
work with convolutional neural networks
we need a set of data to train
our network on and a set of different
data to test our network on and see if
it works
and that brings us to step four which is
to import a pre train model into MATLAB
and modify it to be a segment again I'm
gonna refer you to the code example for
more details we use a model called vgg
16 and we very easily convert that into
a segment finally step 5 train and
evaluate your network and that's it
so with those five steps you've now
learned the crux of semantic
segmentation I'll go over them again and
I'll list them right here as well step 1
label your data or search the internet
for label data step 2 create two data
stores image label data store and a
pixel label data store step 3
partition your data stores into training
sets and test sets step 4 import a pre
train model and modify it to be a
segment and step 5 train and evaluate
your network and there you have it folks
semantic segmentation now thank you guys
for tuning in really appreciate it if
you have any feedback leave it below
otherwise check out the code example and
I'll see you guys in another video
