SANDEEP GUPTA: Hi.
My name is Sandeep Gupta,
and I'm the Product Manager
of TensorFlow JavaScript.
Today, I'm really excited
to talk to you at the Chrome
Developer Summit about machine
learning in JavaScript.
As a web developer,
you may have felt
that all this buzz and
excitement about machine
learning seems to
require Python,
and how you as a
JavaScript developer
can jump in and use
machine learning.
I want to show you that now
machine learning in JavaScript
is real, powerful, and
useful, and you can
do some amazing things with it.
Earlier this year, Google
released its first ever
AI-powered Google Doodle.
In this, users could
synthesize a Bach-style harmony
using a machine learning
model running directly
in their browser.
So, in only three days,
more than 50 million people
used this Doodle to create,
save, and share their music.
A creative marketing
team in Google
has built a fun, interactive
experience where users
are shown a Zodiac symbol.
And they try to match
the shown animal
with a shadow of
their hand, making
that character come to life.
Other teams are experimenting
with novel ways of making
web interfaces more accessible.
In this example, here,
on the bottom left,
called Create Ability, people
can control web interfaces
with gesture such as playing
a keyboard with their face
motion.
All of these amazing examples
have been made possible
by TensorFlow.js.
TensorFlow.js is an open source
library for machine learning
and JavaScript.
It runs in a web
browser and can also
be used server-side and on
many other platforms that
support JavaScript.
You can use TensorFlow
JavaScript in one
of three different ways.
You can run a pre-trained
machine learning
model in your application.
You can customize
a pre-trained model
by retraining it
on your own data.
Or you can use the
lower-level ops
API to write a new model from
scratch directly in JavaScript.
There are many reasons why
you may want to consider
using TensorFlow JavaScript.
First of all, it allows
web application developers
and JavaScript developers to
use machine learning directly
in their applications,
and it's really easy
to get started and
be up and running.
With only a few lines of
code, you can load our library
and you can load a
pre-trained model,
such as the body-pix model.
And this is a model that allows
you to do person segmentation
on an image.
You create an
instance of the model,
and then you call its
estimatePersonSegmentation
method on your selected image.
And you get back a JSON object,
which contains the segmentation
mask, which can then be directly
used in your application code.
That's all there is to it.
So in just a couple of minutes,
you are up and running,
and you are using ML
in your web application
without having to deal with
concepts such as tensors,
optimizers, losses, et cetera.
TensorFlow.js can
be used client-side
in the browser and other
browser-like platforms.
This has many advantages.
You get all the benefits
of the interactivity of web
as a platform, and
users can use directly
microphones and webcams
and other sensors that
are attached to the browser.
Users don't need to
install anything,
so your application
becomes instantly
deployable and shareable.
Another big benefit is that
user data stays client-site,
and this has very big
implications for privacy,
for applications such as
health care, for example.
And it also helps reduce
latency and server-side costs.
Finally, we support
GPU acceleration
through WebGL, which gives
it excellent performance out
of the box.
Although, in this
talk, we are mainly
talking about
browser use cases, I
should mention that
TensorFlow.js can also
be used with
Node.js server side,
and this is very useful
for enterprise applications
when Node is heavily used.
So a natural question to
ask is, how fast is it?
Well, it turns out
it's really fast.
For example, on mobile
devices running inference
with an image
classification example using
the MobileNet v2 model, you
get real-time performance
with 20 milliseconds
inference time.
And on server side, we
have performance parity
with Python-based TensorFlow.
So to help you get started,
we have a collection
of pre-trained ML models
for common ML problems
that we have
prepackaged for you.
These include use cases such as
image classification and object
detection, detecting
human pose and gesture,
audio models for
recognizing spoken words
and building speech
command interfaces,
and text models for text
classification, sentiment
analysis, toxicity
detection, et cetera.
It's really easy to use any
of these pre-trained models
with our high-level wrapped
APIs from our hosted scripts,
or you could install
from our NPM packages.
Using these models
as building blocks,
applications can be built
for a variety of use cases.
These include things
like accessibility, AR/VR
experiences such
as virtual product
try-ons, conversational
agents, and moderating
user-generated content
on your websites.
So these pre-trained models
are a great starting point.
Often, however,
in real life, you
need to customize these
models on your own data.
So one way to do this
is, again, by using
the power of TensorFlow
running directly in the browser
interactively, and using that to
train a model on your own data.
Let's take a look
at how easy this is.
So in this demo, I
will show you how
to use Teachable Machine
to train a custom model.
So the MobileNet image
classification model is already
running in this
browser session, and we
will modify it to
recognize symbols
for rock, paper, and scissors.
So we're going to output the
word rock for the first class,
we will output the word
paper for the second class,
and we'll output scissors
for the third class.
Now, let's record the training
images for these three classes.
So, first, let's
record images for rock.
So I'll hold up
my hand like this.
COMPUTER: Rock.
SANDEEP GUPTA: And, now, let's
record some images for paper.
COMPUTER: Paper.
Rock.
SANDEEP GUPTA: And, now,
let's record some images
for scissors.
COMPUTER: Paper.
Scissors.
SANDEEP GUPTA: And, now, our
model is ready for inference,
so let's take a look.
COMPUTER: Rock.
Paper.
Scissors.
Paper.
Rock.
Paper.
Scissors.
SANDEEP GUPTA: So that's
all there is to it.
In a few minutes, you
have a custom model
trained in the browser.
So, so far, we have seen the
capabilities of the library.
Now, let's take a look
at some use cases.
Modiface is an Augmented
Reality technology company based
in Canada, and they have used
TensorFlow.js in the WeChat
Mini Program platform to build
a beauty product virtual try-on
application for L'Oréal.
Using TensorFlow.js, they were
able to deploy a custom model
meeting their model size
and speed constraints
for this application.
And you can do the same
thing in browser extensions.
To showcase how you can enable
amazing user experiences
with the power of TensorFlow.js,
we have built an application.
Here, we are running a
state-of-the-art machine
learning model to track
users' lips and face movement,
and we use that to lip sync to a
song and score how well you do.
[MUSIC PLAYING]
You get awesome
real-time performance
that enables this application.
And because there are many
JavaScript visualization
libraries out there,
we can use them to add
on some cool special effects.
So let's take a look.
[MUSIC PLAYING]
(SINGING) Give me one
more minute, I would.
Hey, hey.
Give me one more, one more.
Hey, hey.
Give me one more
minute, I would.
Hey, hey.
Make it now--
SANDEEP GUPTA: Companies
are using TensorFlow.js
for a variety of
purposes, whether it
be for internal tools
for experimentation
and visualization at Uber,
privacy-preserving client-side
application at Airbnb, and also
offline mobile device use case
for field use by IBM Research.
Lastly, I want to highlight
that the success of the library
is due to our amazing
contributor and user community.
The library continues to
grow in users and popularity,
and we are really excited
about the amazing libraries
that the community is building
on top of TensorFlow.js
to extend its functionality
and applications,
so thank you, all.
This was a quick intro
to TensorFlow.js.
Please do check out
our website, where
we have more demos and examples
and getting-started guides
and resources.
And also, from there, you can
join our developer mailing
list.
Thank you so much for watching.
[MUSIC PLAYING]
