Hi, everyone. 
Thank you for being here.
I hope you are all
comfortably seated.
So, my name is Albert Reynaud,
I work in the Android and Play 
Business Development Team,
looking after app developers
across Europe,
and I'm based in London. 
So, today, as you heard this morning,
with being kind of, 
announcing a few things and
we're kind of excited
to do all these announcements
we've been working it really 
hard over the past few months
to kind of bring, keep on 
innovating within the console,
trying to bring new tools,
trying to bring new functionalities.
So you can then come back home 
and also start innovating.
So we're really kind of exciting 
about having you here,
I hope you're enjoying 
your stay in Berlin,
I hope that you're 
enjoying your day so far.
And as you can see 
in this innovation track,
we've got a pretty packed agenda.
we're going to start talking about 
machine learning/assistance topics.
And then we're gonna discuss VR
and have some kind of panel discussions.
and we'll finish with sharing like,
insights and research 
findings about design.
So, let's start with the first presentation,
focus on machine learning, that I will 
be presenting together with
Daniel from Memrise 
and Harsim from EyeEM.
So earlier this morning, Harsim gave you
a presentation by machine learning,
giving you like an overview of the
key principles of machine learning,
as well as why we believe at Google
that this is important.
In this session we'll try to
have a slightly different angle,
and try to give you
insight on the how,
as a developer,
you should be approaching it
and thinking about it. 
And also, how some
of your peers have been approaching it.
recently within the app.
So I know that
machine learning, 
artificial intelligence,
you probably hear a lot about it, 
a lot of people talk about it,
it's probably one of 
the top buzzword of 2017,
probably a lot less people 
are actually really doing it.
But and I'm sure that many of you
are kind of excitingly 
confused about this,
but if you are here, at least that 
means that you are interested.
Who in the room is actually 
doing some kind of machine learning
related project within the app?
Okay, so we got like maybe a good 20%.
And who is considering it for the
next six, twelve months?
Okay, so we got up to 50 percent.
I hope that at the end of this talk,
we're gonna have more than 50%,
but that's great. 
So, any case, at is should be mentioned,
machine learning has become 
the number one priority for Google.
We're moving from a mobile-first world 
towards an AI first world.
and the way I think about it is,
sometimes thinking about five, 
ten years ago,
these companies that we're not really
considering Mobile
seriously in their strategy.
Right?
And actually the success of
many of you
is linked to the fact 
that all traditional
players were not taking
into their strategy mobile seriously.
And we hope that you 're 
not gonna do the same mistake
with machine learning today.
So, machine learning already a reality,
as we saw this morning from kind of
image recognition in Google photos,
content recommendation on YouTube,
over a hundred projects, 
production-ready projects
are actually using neural 
network technology at Google.
But you're probably thinking right now,
"Yes, right this is great but 
how does it apply to me?".
Right so, I'm not Google,
I don't have the resources,
I don't have maybe the skills to 
do it, how should I be thinking about it?
And how should I even get started?
So during this presentation
we're going to try to get
a look at the key aspect you 
should be considering when
thinking about machine learning.
And starting by thinking, okay 
are you machine learning ready?
Okay, so what are 
the key initial conditions,
before taking up a 
machine learning project?
So, first of all do you have a problem?
Sounds a bit silly,
but this is probably
one of the most common mistake 
we observe in industry, like people
are looking at what ML can do 
and try to kind of
randomly apply that 
to their business.
However, you should take the problem
the other way around, right?
you should look at your problems,
and not simple problem, 
like mission-critical problems,
and try to see how machine learning 
can help you solve these problems.
This is important because you 
might end up building a great model,
super successful machine
learning feature, but it's useless
because it's not actually not solving 
a problem within your company.
Once you have a problem, obviously 
you need data, you need a lot of data
you need a lot of quality data,
large volume,
and you need to be able to kind of 
access it and process it at scale.
So that means that you might have to go
through a process of improvement 
of your infrastructure,
you have to clean your data,
and you might have to even think 
about collecting labels for your data.
And finally, you need to identify
the people that are going to execute
this project within your companies.
And if you don't have anyone, please 
don't feel discouraged right away
there are a lot of resources online,
that are available to 
everyone to train your teams.
Coursera, Udacity, YouTube 
and all the kind of GitHub, etc.
There's a lot of resources out 
there that you can leverage.
But ideally, in an ideal world
what we tend to recommend,
is to combine two type of skills.
On one hand what 
we call data scientists,
so people that are able 
to kind of conceptualize
mathematical model that kind of
take into account your 
business requirements.
And on the other side, we call
data engineers that 
translate this into code,
able to train the model and 
to kind of run it on production.
So at the beginning, obviously 
you're gonna be kind of
tapping into whatever 
resources you have.
But the more you grow, the more 
we encourage you to think about
structuring this team into one
centralized team,
that is then going to
diffuse their knowledge
into the different
division and process of the organization.
So now you have a clear problem,
you have skills, you have data,
you need to start thinking about 
choosing your model.
Hirshe this morning mentioned,
kind of describe a few of them
from classic image classification
to reinforcement learning 
within self-driving cars.
You need to choose your model.
And even maybe a combination 
of several model.
And it's often where the magic happens.
One key aspect of it, is the presence 
or not of labels within you data.
So in the example of image recognition,
you might have data that will tell you
the object that is present 
within your image, right?
But unfortunately
often this might not be the case.
In this case, you might have to
consider starting 
to collect these labels,
or even think about choosing
another model that 
will not require these labels.
lot of companies in 
the machine learning field,
believe in what we call 
open research, right?
So including Google, 
including DeepMind,
we believe in open research.
This means that a lot
of the research finding, 
a lot of the libraries,
a lot of the models that are being
developed are being put, like,
published online, available,
open source to everyone.
This means that as a developer, 
you will have to choose between
to what extent you will have, 
you will rely on existing models,
and to what extent you will 
actually develop your own models.
Right? So, obviously at the beginning
we encourage you
to tap as as much as you can into
our existing libraries that are built.
But quickly you might realize
that this doesn't
really fit exactly your needs 
and you might have to either
train your own model with your own data
or even like build your own custom model.
As far as Google is concerned,
we kind of, we have this trade-off 
between simplicity and flexibility
will also kind of determine what 
technology called kind of solution
you're going to be using.
So as far as Google is concerned, 
we can help you
along the whole spectrum with
the kind of ready-to-use side 
of the spectrum,
We got solution like Google Cloud 
machine learning APIs,
which give you cloud vision,
speech and translate
and natural language
and of APIS,
as well as Action Google 
that we're going to be kind of
discussing in the next presentation.
But if this is not enough,
one of the most commonly kind of
used framework 
by developer is TensorFlow.
I hear she mentioned it this morning,
it has become the most common use 
machine learning framework.
You kind of can build 
your own custom model,
you can use it to train your model,
you have a lot of libraries
available out there,
you can kind of run it 
on different processing units,
on different cloud platform, etc. etc.
And as she mentioned
this morning,
we're going to be launching soon
TensorFlow Lite, which will enable
running machine learning models 
directly on device.
With this, I will pass it over to 
memrise and EyeEm, to Daniel and Hasin,
Who are going to share with you 
how they've been approaching
machine learning within their own app.
Thank you very much.
Hello, Hi. 
So, thank you very much Hubbard
and thank you for having us here.
I think it's an amazing conference,
and thank you very much.
So, right, so my name is Daniel.
I'm the CTO of Memrise 
and if you don't know Memrise,
we're one of the leading
language learning apps
in the world,
with over 25 million users and 200
language combinations to learn from
we really strive 
to make language learning
as joyful and effective
as possible.
And very simply we won 
the best app on Google Play,
we're very proud of this achievement.
And today I'd like to talk to you about
how we build product feature 
using machine learning.
So, think I would ask you guys how many
people are interested
or playing around
with using machine learning
to be [unintelligible].
How many of you actually 
have delivered product
using machine learning? 
Right, so not as many.
So it's obvious that 
everybody's really excited
about the possibilities
of machine learning.
But it's not as easy to get from 
having ideas and using the technology,
to get something out there.
So, we're doing Hackathons on a 
regular basis, like every six weeks,
and we really want to give people time
to play around with ideas and technologies
they think might work 
well on the product.
And this particular feature 
started as a hack,
so we try to think, okay, so what 
could we do with object recognition
and after a couple of days with 
developer playing around with it,
they came up with a very neat concept,
So imagine taking your phone,
looking around and we feel on German,
then you look at the screen, you learn 
how to say screen or TV in German,
And that was the idea.
So once we had that in place,
after the hack there was thought,
okay, this is really cool,
how can we take this
and actually make this something
that we can give to our users?
So we started kind of looking at 
the product and try to define it better.
and what we knew we want to have,
we want to make sure 
that it's very accurate
because we teach people languages,
we can teach them the wrong thing,
It has to be in 
the right difficulty level
and it has to be really fast 
and give like,
"Oh, wow, this is amazing".
Like really bringing 
the technology forward.
The interesting thing is that
actually getting 
a machine learning model
or in this particular case, 
object recognition,
was really the easy part.
The more difficult part
was to get it to recognize right things.
So if guys look at that picture,
what do you see? 
Yeah, so a guy, a man.
If I look at that picture,
actually I see James,
who's one of the people
that worked on this project.
Now we took this image and we gave it to
Google vision API
which is an amazing 
piece of technology,
and this is what it came up with.
so hair, facial hair, beard,
well, this picture is definitely much fun,
but I think that's not the first thing
when we look at the picture, we think it.
So I think context is really, 
really important.
It's not about recognizing object,
it's recognizing the right object.
And we spend most of 
the time really focusing on that.
So, once we... after playing 
around with a lot of models,
and there are quite a few out there,
we actually decide
to train our own model.
So we used, the based on was 
using Inception V3 which is a model
which exists, and on top of 
that we trained our own model.
So what we actually did is we got product 
managers to collect a lot of images.
We first define the images 
that we care about.
So we do a lot of user surveys,
we tried to understand how
users use the app.
And we know that
often time, for example,
they use it at their homes.
So we thought about, okay, 
so which objects are available
in their immediate surroundings?
So we try to define that list of objects,
which were in the hundreds,
and then start collecting the images.
Product managers spend a lot of time
collecting images from different sources
and we found out that actually 
we need quite a few example,
and like some objects
needed more than others.
For example, people,
so as you can see here
there is one example,
of [unintelligible] engineer
which is not quite a hair dryer,
maybe he needs one.
But, yeah, so this
was an example of something which was 
a bit difficult and as you can see there
a picture of bicycle
and a picture of glasses,
and they're both recognized as glasses,
Something that we wouldn't imagine, 
that you would never
look at the bicycle
and say, "oh, it's glasses".
But, it's actually problems that emerge
and we had to do a lot of back-and-forth,
collecting more and more images, 
focusing on amateur photography
and also different sizes and shapes of 
the objects that we really care about.
So there are
a lot of interesting examples,
as you can see in the slides
were there were problems,
but by iterating and understand 
what it is that we want to build,
what we care about 
being the final result.
We came up with a really neat product.
So, it's just a few images from 
the finished product that actually
recognized really nice things 
like airplanes and stuff like that.
And now were in the process of finding how
integrate it best
with our product.
So, just a few lessons learned.
I think like I said, 
I think even with a really good model,
different data sets really
equal different products.
So like you see in the Google vision API,
it's not that it was wrong, 
it just gave us,
it didn't give us what 
we wanted it to be
really carried in our product,
we wanted people to look around them
and learn how objects are called.
So we really cared about single objects,
how the user is experiencing it.
And additional to that
I just want to say 
that I think machine learning
I think as Albert said as well,
it's a tool, you want to 
educate people on how to use it,
there's a lot of 
resources out there.
Educate your teams and 
try to encourage them to use it,
and try to find the right problems 
that machine learning can solve.
Thank you very much.
And I hand over to
Harsi from EyeEm.
Good morning.
So today we're going to talk
about how we use 
machine learning at EyeEm,
to help improve our work flow 
for the photographers.
And what is EyeEm?
So EyeEm is a combination 
of our community,
so that these are photographers,
we have around 
20 million photographers,
who upload their content
onto our platform.
And we also have a marketplace where
people can sell their photographs for
for Live Sync, which can be used by
brands for their PR campaign 
or advertisement.
And to connect the right photographer
with the right buyer
and we build the technology.
And the technology we build
is called EyeEm vision.
So the EyeEm vision focuses on
understanding everything 
about a photograph.
So what's inside a photograph?
There is a dog jumping, 
a dog leaping, a stick,
a frozen lake,
atmosphere outdoors,
but we also want to know how 
good the photograph is composed.
How does it, a photograph uploaded
by any person on the web
compare to the photographs 
taken by professional photographers?
And that's what we captured 
through aesthetics.
So, we have a deep learning model,
a machine learning model which can
rate a photograph
a score between zero and hundred.
Hundred means that it is as good as by a
professional photographer.
And so the core of training any 
machine learning model is the problem.
The problem that you want to solve.
And a machine learning model 
needs an objective.
An objective in this case, 
would be given a collection of images
give me the right answer that I expect,
and when you don't give me 
the right answer go correct yourself.
So this is the feedback that the 
machine learning model gets.
And as you can imagine, 
the more complicated your problem,
the more complex your model needs to be.
And that complexity usually
translates into the size of the model
in megabytes or
gigabytes.
and that's a concern 
when you want to go mobile.
So if you want to run a 
machine learning model on mobile
you're constrained, on the size 
of the machine learning model
that a phone can run, on a CPU or a GPU.
So that's additional constraint
that the model has to take into account.
You can actually
take an existing large-scale
big complexity model
and then shrink it
down to the size you want.
There are many methods out there,
quantization and
compression is one.
An analogy would be
changing a PNG image
into a JPEG image, 
for all practical purposes,
their visual quality is similar,
but the more attention 
you pay to the detail,
you start to see the cracks.
And this is all and well, once you have a
machine learning model that
can run on the phone, 
it can produce all the
information we need 
to know about an image.
You can detect the tags, you can detect 
what's inside the photograph.
But you can also get a score, 
a score of how good
the photograph is composed compared 
to professional photographers.
And once we have all the information,
the question is you can 
send all the photographs
to a AWS machine or a Google cloud,
and get all the answers you want.
Why bother with miniaturizing your mobile?
Miniaturizing your 
machine learning model.
What's the advantage?
Why spend so much effort into something 
that is already working for you?
One key aspect is privacy.
So, in our case we are working 
with people's personal collection.
So, it's your personal photos.
It could be photos of your partners,
it could be pictures of your kids or
private moments,
and it's very tedious
to upload your whole collection
into cloud and 
get it analyzed by the algorithm.
And the second key aspect is that
the current functionality 
is that multiple people
upload their photos and 
they go to a single server,
but with mobile phones, 
we have a perfect scenario
we have user and one computing 
machine, the handheld device.
And the third reason is
it's available everywhere.
You don't need
to be connected the internet.
You don't need to have WiFi.
And that makes sense, because taking 
example of machine translation,
when you need the 
translation the most,
when you're traveling abroad, when you 
don't have access to internet or WiFi.
So, you want your machine learning 
model to be able to run everywhere.
So if you are a machine learning engineer
or a data scientist,
that's your part of the problem,
But this is the problem have solved.
Now as the product managers turn 
to turn that into a feature.
And that goes through various cycles.
So once you have a prototype
that could come through a Hackathon,
that could come through 
an internal effort,
you build the first prototype.
And you let the PMs play with it, 
and they can think about how
they can use this machine learning 
model in a feature.
and that's the place where 
either your clients team,
the Android iOS and the designer come 
together to build a specific feature.
The next step is actually 
getting user feedback.
So let real users of the app
or the feature
play with the model 
and get their feedback.
And this is a critical part of actually
retaining your machine learning 
model and collect a new data set.
That new data set should 
reflect the feedback
that you get from real users.
And that the next stage is actually
retraining your machine learning
model to better fit.
With that in mind, we launched 
a feature a few months ago
called EyeEm selects, which runs our 
deep learning models on the phone.
It analyzes all your photo collection, 
you could have thousands of photos
and then it suggests 
what the best pictures
you should upload to the platform.
This makes sense, you don't 
want to upload your private photos.
And also if you are new user to EyeEm,
you see all the good photos 
and good photographers,
and you get intimidated. 
What should I upload?
Am I good enough?
This machine learning model helps us,
help them find the best photos 
they can upload onto our platform.
And this entirely runs on TensorFlow,
we haven't yet moved to TensorFlow lite
but that's gonna come very soon.
If you have a machine learning model,
you can assess its 
performance by various metrics.
There is precision, recall, 
tough and accuracy,
but you also want 
to measure the performance
based on how well 
your target feature is.
And in our case we wanted the users 
to upload not just more photos,
but also better photos. 
And so, when a user uploads
a photo that is recommended
by a machine learning model,
we automatically add a hashtag to it
called EyeEm selects.
And then we assess
how does that
how does that photos with that
hashtag compare
to everything else 
we have on our platform?
And we saw that people were actually
uploading better content over time.
I think that's it.
If you have questions for us...
Please feel free.
