(audience applauds)
- [Ali] I'm the difficult one
that needed my own
computer because, demos.
So, I'm Ali, I'm a software
engineering instructor.
So, I teach other people how to code,
and today I'm gonna be talking about
augmented reality in JavaScript.
So, little bit of what we're
gonna be talking about,
so we're gonna talk about
what augmented reality is,
and then we're going to play
with a ton of different demos.
After that, we will talk
about the different types
of augmented reality
that there are out there,
look at some of my code,
I wrote Snapchat's filters
in 47 lines of JavaScript.
So, we'll be talking about that
and then we're gonna
talk about the future,
what augmented reality will
look like in a couple years.
So, to define augmented reality,
you're taking the real world
and then you're adding something to it.
You may have heard of virtual reality,
and that's creating a whole
entire new world in 3D.
I think augmented reality
is really powerful
because you can take the real world
and then add something to it
and that's really powerful.
It can mean for any sense,
but the one that you see most often,
at least for now, is visual.
So, you'll have the real world
and have some visual element added to it.
So, I am talking about augmented reality
not because it's something that I do
in my day job every day,
but because I like art and
code and I have a blog,
and so, one of my blog
posts, about two years ago,
was about how to build augmented
reality apps in JavaScript,
and it turns out that a
lot of people thought that
I would be interested in giving
talks about it, which I am,
but I am not somebody, again,
who writes augmented reality apps all day.
So I, at that point didn't feel
like I was an expert at it,
and that is actually the reason
why I'm talking about it.
I was so intimidated by the
this idea of augmented reality,
I thought you would have
to know all this math
and this wild code in
order to implement it
and then I did more research
and it turns out you can
just do it in JavaScript,
and it's basically just DOM manipulation,
is my whole entire application.
I think that's really, really cool.
At the time I was seeing
a lot of apps coming out
that were using Python or C++
and they were just command line scripts
that you could run and
add filter's to your face
based off of that, and that's cool,
but you have to be a programmer, probably,
to run those applications.
You have to download them, you
have to have Python running,
and if you've built something for the web,
anybody can use it just by going to a URL.
Mobile apps are cool as well, but again,
you have to download something,
and the development is a little
bit higher barrier to entry.
So, JavaScript makes
development accessible
and it also makes applications accessible,
and that's why I wanted it use it
to build an augmented reality app.
So, there are so many
different applications
for augmented reality already.
Art, which is one that's
really interesting to me,
and I think augmented
reality is going to be
a really huge forefront there.
There is gaming as well,
you may have seen
something like Pokemon Go,
we'll talk about that a little bit,
and then businesses are
starting to use it as well.
So, it can be a pretty
powerful thing for marketing
or just trying out elements as well there,
and then there's also more too,
so, we'll talk about some of that.
So, one of the most common
questions that I get asked
about augmented reality is,
when will we use augmented
reality every day?
I know for the average
person in the audience,
maybe you don't use
augmented reality every day,
but for a different demographic,
mostly teenage girls and young women,
it is something that
they're using every day,
and my demographic is,
and so I want to talk about that
and how powerful it already is.
There's going to be even
more of a future for it,
but it's already massive,
so let's look at some demos.
So, the first two are
Snapchat and Instagram.
Instagram is so culturally
dominant right now,
not just the app itself,
but the world around it.
Like, I go to events and they
have Instagram installations
so that you take pictures of it
and market the event for them.
It's a huge piece of it,
and then also people
buying ads on Instagram.
So, that one is absolutely
massive, they have the,
in the stories you can put on
filters on your face there,
and then Snapchat,
who they pretty much just
knocked off to do it.
So, Snapchat allows you take images
and put them on top of your face
and alter your appearance that way.
Those are kind of the original ones.
Fun part of this talk is that
you get to see my face a lot,
which is especially fun
because I'm super jet lagged,
but that's fine.
So, one that you may have seen recently,
especially on social
media, is called Face App,
and what it does is it takes your face
and does some machine learning on it
and then spits it out in different ways.
So, the one on that side, the woman one,
is a tweaked version of my face.
So, I don't even know what
they changed about it,
but they made it way prettier
and it's super impressive,
and then the other side is
the male version of my face,
apparently, so I'm glad that
that didn't turn out that way for me.
There's also Facetune,
which is a visual editing
software for your phone,
and you can essentially take what
people made careers off of,
airbrushing people back in the day,
and you can do it in a couple swipes.
Which is really powerful,
but also kind of terrifying.
You can try on makeup, so,
Sephora's website allows you try on
all sorts of different
makeup on their website.
So, I don't own blue lipstick,
but I was wearing some earlier today.
IKEA, allows you to take
furniture, on their app
and try it our in your own living rooms
so you can see what a piece of furniture
would look like dimension wise,
in your own home using this
augmented reality application.
Tesla, they're using augmented
reality for manufacturing.
So, they're using it
to test out their cars
before putting them out into the wild.
They're also experimenting it for
the dashboard on your computer,
trying to have different panels on there
using augmented reality as well.
This is one of my favorite
AR applications from 2010.
So, it was a while ago,
Zach Lieberman is an incredible artist,
and hopefully this will load.
Oh, no, let me try this, sorry,
looks like I got disconnected.
Oh, no!
There we go.
Okay, well, I guess that is not working,
sorry about that.
So, also museums,
museums are doing really,
really cool things with augmented reality.
So, both for extending the
artwork in their museums,
but also for displaying their
content in different ways.
For example, walking tours,
it used to be that you had like, headsets
and would type in numbers
at different pieces of art
and they would tell you about it.
Now, using augmented reality,
you can just walk to places
and based off of your location,
you'll hear different
information about the artwork,
so that's really, really
incredible as well.
The Louvre also made it so that
you could project their artwork,
no matter where you were in the world.
So, in Los Angeles, they had a warehouse
and they converted it into the Louvre
by doing projections there.
So, that's really cool as well.
Pokemon Go, you may have
played this, I didn't really,
but my mom was into it even
though she didn't know Pokemon,
so that was really fun.
You could capture these little animals
that were out in your world
and that game was huge for a while,
and there are other games
like it as well now.
Rock climbing, rock climbing
is one of my big hobbies
and people are using augmented reality
to create climbing routes
and climbing competitions,
which used to be static.
You used to have a route
setter and put tape up
and that's how it usually is now as well,
but they're making it so
that you could have routes
using augmented reality instead,
which allows for much quicker
and dynamic route settings,
so that's really cool as well,
and then it even goes beyond that,
so they're using augmented reality
for some medical purposes as well.
So, in order to increase contrast
for some people who
have visual impairments,
so you can see the different lighting
that may help some people
see certain settings better.
So, it goes beyond this artistic
and marketing world as well.
So, there are a couple different
types of augmented reality,
the first one was marker-based.
So, marker-based augmented
reality was the very first type,
and you would have something
that was almost like a QR code
or this Hiro marker here
and you would project
something based on top of that.
So, that was pretty limited at that time,
but a lot of augmented reality
is still based off of markers.
So, something in reality that
you're adding something to.
Usually your face, but there
are other things as well
that you may add augmented
reality on top of.
Marker-less augmented reality is similar,
but just no markers.
So, you're adding something to the world
without an item that
you're adding it on top of.
So, something like Pokemon Go,
or something that's location based
would be marker-less augmented reality.
There's projection-based
augmented reality,
so something like that climbing route,
where you're actually
projecting something out
into the real world
using augmented reality.
Superimposition-based,
where you have something
that's being added.
So, this is a combination,
usually with marker or marker-less.
So, something like IKEA,
where you're projecting
the couches into the real world,
would be superimposition-based.
So, I wanna talk about the ways
that you can build
augmented reality apps now.
So, one way that you can,
is through something called A-Frame,
and ARjs on top of it.
The really cool thing about it,
is that you can build
augmented reality apps
without even using JavaScript,
you can just write, essentially, HTML code
and have augmented reality apps built out.
You have to use a marker,
like the Hiro marker
that I showed you before.
So, it's a little bit
limited in that respect,
not as fancy as like, the
face-based augmented reality,
but you can build something like that,
where it projects a cube
on top of the Hiro marker
in 10 lines of HTML.
That's the only code you have to write,
it's pretty powerful,
and then I have been using tracking.js.
So, tracking.js is a computer
vision library for JavaScript,
and what it allows you to do
is detect things in the real world
and then you can use the coordinates
of the thing in the real
world and build on top of it.
So, I did face recognition,
but they also have color-based recognition
or parts of your face or your hands
or something along those lines,
and it's pretty lightweight,
it's not a massive, massive
library by any means,
and it's super accessible.
So, they've got great documentation,
and it's not impossible
to write by any means.
You don't have to do a
ton of the math yourself,
and so, we're gonna go ahead
and look at my app and my code.
So, I'm gonna switch over now.
So, hopefully this works on stage,
but it's giving me a flower
crown, and you can see that,
and it does for anybody
that is on the projection.
I can switch it over
to bunny ears as well,
and it gets my face there,
and this whole application
was 47 lines of JavaScript.
So, I'll show you that code now.
So, starting off with the
HTML, it's pretty basic.
The most difficult part
of this whole project
was actually stacking a canvas
on top of a video element,
that turned out to be a little bit tricky,
but the fact that the
CSS was the hardest part
says a lot, I think.
So, again, I have a video element,
which is how my video
is taken from my webcam,
and then I have a canvas on top of that.
The canvas is where the
actual filters are projected on top of.
So, I have those two elements there,
and then my JavaScript.
So, at the beginning,
it's just a bunch of
grabbing things from the DOM
that I'm going to be interacting with,
and then I'm creating a new tracker
for the faces on the screen,
and then I'm grabbing the
flower crown and the button.
I'm taking the image,
so it's gonna be an image object
so that I have all the
information about it.
That's gonna store my actual crown itself
and then the
bunny face as well,
and then I have the XY width and height.
So, I have to scale those in
order to make them fit better.
I have ability to change pictures,
and then I have the ability
to change to the flower crown
and the bunny ears as well.
These coordinates here are
just scaling for the images,
I found two random PNGs on the Internet,
that were the flower
crown and that bunny face,
and so, I had to edit them a little bit
in order to be the proper size.
and so, this is just scaling them.
So, they're X and Y coordinates
and they're width and
height, so that by default,
they fit on somebody's head pretty well.
That was, again, the most time for this
because I was just tweaking that,
but again, not a huge amount of code here,
and then some more tracking code.
So, the first one is set initial scale,
this is how big the box
around your face is.
So, behind the scenes,
tracking.js is making a box
around where your face is,
and so this is saying
that it's going to be,
the initial scale is gonna be four.
If it's 10 instead, then it's a lot bigger
and the crown goes way outside my head,
if it's one, then it's a smaller crown,
so that's essentially
what it's doing there.
The step size is how
often it's going to track
to see faces on the screen.
So,
this one is two,
and so it's pretty fast,
it checks for new faces pretty
frequently on that video,
but you could speed it up,
so it tracks even faster.
It's a little bit less
performant in that way,
or I could slow it down
so it's only looking for faces
every couple seconds or so,
and then the last one is
how picky the model is.
So, there is a model going
on behind the scenes,
so that it's doing all the stats
of whether something's a face or not,
it's a binary classifier,
I'll talk about the
algorithm a little bit later,
and this is just picking
how picky that model is.
So, if I turned this even
further down, so less than 0.1,
then it's going to get
all these false positives
and thinks things that
are not faces are faces.
So, I built this code in
my instructor's lounge,
which has whiteboard walls,
and so there were pictures
on the walls everywhere,
and at first before I
set this edge density,
it kept thinking that all the
drawings behind me were faces,
and they were not.
So, then if I make it way more picky,
then it won't see my face
in some lighting like this
or backstage or something
along those lines.
So, that's just something to play with
to decide how picky you
want the model to be.
I'm making it track the actual
video element on that page.
So, that's how I set that up,
and then here's the actual tracking.
So, it's an event, listens
for the tracking event.
When that happens, it's
going to clear my canvas
that I already have there,
I'm gonna make it blank,
then for every face
that's in that event data,
it is going to draw an image on it.
So, that's all that I had to do.
Most of this isn't even code
that is related to tracking.js,
and I have this fully working AR app.
I think I reformatted it
so it's 51 lines of code,
but it's normally 47, oh,
I added the three comments, that's why.
I didn't even Code Golf this,
if I had Code Golfed it,
anybody use Code Golf?
So, Code Golf is when you write code
that's as few letters as possible
in order to pass some code challenges.
I have fun doing that, but
I didn't Code Golf this,
so I did full variable names
and multiple things on lines
and all that.
So, I could have made it
less than 47 lines of code
if I really wanted it to,
but not a huge amount of
code for a pretty cool app.
So, the algorithm,
so tracking.js uses the Viola Jones
facial detection algorithm.
Facial detection and facial recognition
are two different things,
so facial detection is face, not face.
Facial recognition is whose face is it,
and so, this uses facial detection,
so it's just a binary classifier,
this is a face, this is not a face.
What it looks for is
that people's eyes
region are further back,
so it's darker than the rest of your face,
and the nose and forehead region
and cheekbones are lighter,
because they're closer to you,
and so it's looking for that, essentially.
It sums up the pixel
values in any rectangle
and uses that pixel value
of dark and lightness
in order to check to see if
something is a face or not,
and then...
Yeah,
so that's the Viola Jones
algorithm right there.
It's used by a lot of
facial detection software,
not just tracking.js.
So, that is...
tracking.js, and it has, again,
a lot of other classifiers as well.
So, it has ones for different colors,
it has things for
your eyes or your nose or
something along those lines,
if you wanted to make
your filters even better.
I tried to add the dog
filter at some point,
but that got too difficult
and like the nose and eye gap
was too hard to implement.
So, definitely would recommend
using those pickier models
if you were trying to do
even more complicated facial filters.
There are also a lot of
other libraries out there
as well for doing this,
other JavaScript libraries,
and the reason I didn't go with them
is just because the
documentation was worse,
and so, that's a big call to action
for open source maintainers,
to make sure that your
documentation is good,
but what comes next?
So, I think augmented reality
is going to be huge in the future
for a lot of different purposes.
I think that the artistic
applications are really powerful.
In Washington, D.C., where I used to live,
there's this art museum
called the Art Tech House,
and they've had tons of
augmented reality exhibits there,
and so, definitely a lot
going on in that world,
and then, as far as the code goes,
I think that it's going to
become more and more accessible.
So, Mozilla is coming out
with mixed reality API's,
that are built straight into the browser.
So, you don't need to
download another library
or anything like that, you can
build augmented reality apps
and virtual reality apps
straight with this browser API.
In addition, there's also
the face detection API.
That one's implemented in
some versions of Chrome,
and that detects whether something
is a face or not already,
just in JavaScript, and
just in the browser itself.
So, again, you don't have
to download anything else.
It's really incredible
how fast this is evolving
and how much is already
getting built into the browser,
and it's even gonna go further, I think,
in mobile applications as well.
The thing with websites is
that you're really limited
by webcams and webcams aren't necessarily
the best harnesser of
information in the world,
and so, there's definitely going
to be more with mobile apps
than anything else.
In addition, the marketing context
is gonna be really interesting as well.
Trying on clothes, trying furniture,
trying all different sorts of things
right into your browser.
So, that's all, so these
are more resources as well,
so if you're interested
in reading up about
any more of the heavy math
side or anything like that,
I've got all of these linked.
My slides are online,
alispit.tel/ar-javascript
They will be up online as long
as history goes, essentially,
unless I delete GitHub or
something for some reason.
So, they are online.
You can look through all these resources
on getting started with augmented reality.
There's the Viola Jones
paper, right there,
based off of that mathematical algorithm
that I was talking about
that tracking.js uses
for facial detection,
there's also the
tracking.js implementation
of the Viola Jones algorithm,
my article on using tracking.js
to make the Snapchat filters.
Awesome Code Newbie episode,
if you saw Saron doing
Command Line Heroes earlier,
she has another podcast
and it's a really, really awesome episode
about getting started
with augmented reality.
Zach Lieberman, who is my
favorite augmented reality artist,
I have his Instagram link there,
and then also some of the libraries
that you can use as well,
and keep in touch if you want to reach out
I'll will be at this event tonight
and then tomorrow as well,
but you can also find me online
if you want to talk more
about augmented reality,
and thank you all so much.
(audience applauds)
