大家好，我是Andrew  Tsai
我是Andrew Tsai
I'm sorry my Chinese is really bad
I'm still a Trish yeah I'm still
learning, so
my name is Andrew Tsai and I'm the CEO and
founder of a company called a vector AI
so I think we have an issue with the
presentations I think I will start to
introduce a bit about myself and what's
the relation with the BitTiger and as
well I will try my best to speak slowly
because a self driving a car is a really
hot area and then if it's really hard to
to develop and as a founder of the
company and an engineer person will
lightly select successfully to build a
self-driving car by myself compared with
other companies like
Karma AI, Drive AI, Plus AI and autonomous
stuff and also Udacity one of example
and I'm really lucky I'm thankful that
Beijing's gives me and Charles gives me
opportunity to talk today with you
Charles I have a quick question so are
you ready with the presentation or okay
so I think I'm going to start over my
name is Andrew Tsai and I'm the CEO and
founder of vector.ai
so what vector.ai does is, we develop a
platform to accelerate the development
of self-driving car companies like
startups to one OEMs they want to have
some sort of different platform to
accelerate the development of
self-driving car so we not only focus in
self-driving car but also we build the
future transportation you try to
accelerate part of the for example the
space the education and also democratize
the technology itself too many cases
in transportation okay so this is bit
about my background I'm a Googler
I'm also worked with the project tango
open project with project X when I'm a
Google Apps and also I also former logic
marking engineer what logic marking dad
is one of the best defense company in
the world and you have a lot of
contracts with DARPA which is that's the
original a of a self-driving car and
also robotics applications and company
like Boston Dynamics so lucky does f-35
and also have a deal with caterpillars
in the right side those are my first
background after several years work in
DoD experience, DoD stands for
Department of Defense and second before
I start my company I work with north of
Brahman, North Brahman is the second best
defense company in the world that
actually doing a lot of autonomous
vehicles for example the Global Hawk
this is one of the wing in the Global
Hawk that actually what Global Hawk says
is civilians airplane that running
autonomously 24/7 on the sky and as well
I've done also several projects with MIT
for example with the their tartar
challenge and also their sales having
car so that's that's a good patient of
mine so before we mow starts quit out
from the Google and also Auto get
bothered by uber during the supplement
track so the original terms of
self-driving paintings technologies in
the old robotics platform is actually
came from CMU and also DARPA and also
depends companies so I start from zero I
start from production car I start to
fill out the Honda and also a CRA's
because he worked with in the Asian
market colors and we do a lot of things
and again our garage essentially myself
and as you can see here there's a GPS
box here and then there is a GPU that we
installed very very
basically that's the server to running
or our neural networks applications and
all the infrastructures and I'm doing a
lot of stuff also in the control systems
control stage in the cars installing the
sensors and on the top of it you can see
there's a VLC 16 which is 16 channels
village angle icons at the top so we
were using that in the first terms of
doing a localization that planning and
the controllers of the car ok so we get
to control quickly so we started in
September 16 with just one person mind
and then what we do in the control is is
we can do can can we dim your lights a
bit oh boo
oh thanks I need to play this one
Oh
should be able to play
yeah the video it's video basically how
how we get the steering angle string
controls of the car or by wire cars
completely get control so we take over
the electronic power steering electronic
throttle and electronic brakes
most of the car to days if if if we know
that okay talk about cars I hope is not
gonna hurt your brain so cars its off
what cars okay
card is like a computers detainee okay
think about you know you have you want
to have a build your own computers you
can customize your car ram you can
customize your hard drive you can
customize your type of CPUs same like in
the cars so cars it is called Intel ECU
right which is electronic control units
or engine control unit design so in the
ECU the question is is just already
controlled by one ECU or two or three
for ECU it's actually not actually have
many ECU's right for example we talked
with a BSA right v equals safety this
isn't all to break your x-ray turn ACC
which stands for adapt Express control
when K this is so this is this is
actually what it's meant to be the car
as a computer right and then also
absolutely if we know I mean like bands
BMW Volkswagen and Audi
so those equipped with the sensors right
there's a millimeter wave sensors
there's also ultrasonic wave sensors
with a different types of Tier one and
you know mobile I perhaps from cameras
you know to do a simple 8 s so a - time
for advanced driving assistant
system sorry but my my writings are
going but this is actually what makes
you the card nowadays today so many
people that come to me like how do you
do it how how do you exactly convert the
car controls by the choice mix for
example or you can have a literally
control of this car we can talk about
this after offline if you want to see
the video of this so so that's that's
more into the very first step to gain
the control of the cars so and then the
car itself right you have also another
things called drivetrain you have a PCM
right we still consider talk each others
with the ECU and then how this car each
other's talks is actually with one
network that exists that in 80's and
then that still happens today with the
different many types of ISOs which is
called a can bus right we're having most
of people that familiar with tuning the
cars and then doing a lot of car
hackings they they know how to play
around with the can which is control
area network in the car you have for
example you have 11 bytes of data that
actually assign specific IP which is
comes first to talk with the ECU right
so you can control your AC you can
control in your door you can control
your ship is that part e drive neutral
neutral or reverse it's all taught to
disk in messaging if your high speed or
low speed and also another thing you
know absolutely you might ask is a tone
can hand you know it's not only can but
there is also Lin luxray right so these
are what the typical communications that
talks to the ECU and instead of protocol
and computers right you have like
Ethernet right you have a wireless right
so those those are the standards in the
car so we easily basically try to
reconstruct our own control strategy
for developing autonomously from now
next what's next after that okay
we know that a gas they work well they
weren't really good they work in heavy
rain they work in heavy Road we have a
role in this in the situation is like
you have pedestrians you have practice
you have a lot of people walking around
we have bicycles right so those are what
happens with a vests but AJ's problem
today is actually you just only can have
that for certain period of time probably
say three minutes probably two minutes
or even 10 seconds right so or you know
for example in a test right imagine this
is rock then did the road right you have
a top view of the road and then there's
you want to this is your vehicle here
and then you straightly try to change
the lane here right this is not
autonomous though is this you steer and
then suddenly you try to make a lot to
the right drifting to the right and then
the 8's weren't you that's that's a
guess right
that's departure warning lane data
cruise control right now we know that
the way we improve it just only not
taking ages completely but we take off
the ADA's completely that's the beauty
of it so
what we do what we do is we take
approach of end to end up for some
driving cars but we fail on that
the reason we fail is it things is like
a black box deep learning is like a
black box right you you have a much
training data and then you have this
supervised learning and then suddenly
you have a model in ways and then boom
it works apparently is not like that way
apparently it's more into you have a
process things we have augmentations you
have a data strategy you have we have a
lot of things that you need to do before
you go into the convolutional later here
right before you slice your data and for
that reason is we we take the approach
of doing end-to-end learning in
different ways which is the first one
agree encoder the encoder takes the
convolutional neural network in one
bundle of network and then after that we
also can train an auto encoder which is
we flip it back and to compose what we
already learn into something meaningful
using long term term memory right in
case of long short-term memory that the
things that more interesting is because
you have the pulley this is interesting
here if you if you know the meaning of
the pooling here and the beauty of long
jump of learning here is actually
whatever you're already done for your
course definitions in the beginning from
your convolutional net works here oh I'm
sorry I don't want I don't want to make
the fee so oops what going on
get it back anything all right thank you
so see so where am i oh yeah of course
so we have chorus here right so one man
of the course you take that first so you
see that the weekend what it does it
means here is actually we have input
image from here coming here we pre
process that we scale it down and then
we feed it into our convolutional neural
network one one line one straight lines
right one one approach to do that and
after that we have at least and
predictive controls three controls right
back on this right
sending intervals of the ECU which is
that steering angle brakes and
accelerations so according from here you
actually already get one rough idea to
how to control the cars but again when
we turned into the driving area is
actually the car did not run well
because they're not about the image not
about the data not about me not about
the the roads or the controls the cars
not doing good are the tissues
none of them those are correct what the
correct things that we think that there
is a problem with the decoder the
problem with the decoder is we're not
having we weren't even got the heavy
strategy to have a different tune to do
the heat map so in chorus ring decoder
what we do is we train with what we call
as a visual attention so for example
this is easier way to understand this
there's two pieces so
we go back on the road again oops we go
back on the road again but this time
imagine that you have a front-facing
camera attached to your road okay so you
use your driving here and there is a car
here and then there is a card here so
with it in the very first step what
we've done is we have a predicting
steering angle a smooth steering angle
and the actual steering angle so these
are the controls visualizations in this
our cell running car similar approach to
what Stanford has and also NVIDIA has
and their deviate Stanford in the family
and junior basically so what happens is
when when it predicts the behavior of
the next card is actually we do the heat
map so we color this the idea taken from
the tech net with the segmentation
network but it's more robust because you
have ability to do the heat map before
you add it in to your lsdm those are way
more powerful then you just have one
vanilla network in end-to-end learning
approach okay it works it works well our
Sigma later and well and then our
gradient descent becomes Whaley much
better
we got really when we train that we got
a really small rmse which is root mean
square error and then it's what way
we're much more happier in the the agent
our car is way much more happier in the
less hectic urban environment so let's
say that in rural area in housing area
space the car can drive autonomous
driving we consider that as a level
three autonomous driving which is we
will talk that about in a second about
definition of the levels of autonomy
now this is the results that we got
after we do all the implementation the
gas this is a video and as you can see
here oh it works really nice this is a
British one didn't work interesting so
we have a previous data so this is data
it's got from the wave one from the GPS
and again we're really happy that in
April this year we finish autonomous lab
completely without no hands on the
sphere no brakes on the up no no feet on
the brakes completely autonomous control
using video using deep learning using
our convolutional neural network so what
happens here what you can see is the
true steering is actually the predicted
data sets that we already have and the
raw is actually the the models and
weights that we frame in within eight
hours we got the output in gradient to
the rostering annual but this is this is
really high standard deviation here and
what's going on okay we'll do the smooth
strategy again back into our the
previous method to do the the strategy
to smooth the steering angle is actually
very Phillips look is only one
difference is here this is the result
that we using we implement a
vectorization
and normalization well in the last class
two years we know in terms of deep
learning any high that um many research
scientists they invented a clause that
into processing Kolob mini section
normalization so you you take whatever
the normalization technique in the very
beginning of pre-processing before you
start to implement that in your neural
networks anything you can be a recurrent
you have convolutional anything so it
makes slide will be
oops so again this is the pictures that
we took the first time we did went to
the tiger office actually in Santa Clara
and this is our first test on the road
and the highway road from San Diego to
Santa Clara to their office it's about
eight hours drives and it's interesting
the car the car ran well the car can can
do complete autonomous driving on the on
the highway really well just we thought
using the mobile eye and we solve the
issue of the the beginning that I
mentioned that only 10 min 10 seconds
driving only 10 minutes driving and then
we extend that period today actually we
came back with another approach with a
tap that giving the deep learning as
more robots backup systems if the camera
failed and we still can't tell drivers
with only using radar and weight points
from the GPS RTK real time kinematic and
I use information IMU is inertial
measurement unit so if you know about
the iPhone stuff the iPhone has a
gyroscope and accelerometer here to know
the positions itself so we have that in
the car really tiny gallium use and then
we use that information to keep
autonomous driving without the camera so
ok back to self-driving again my
experience and self-driving cars with
DARPA robotics you have a relief old
techniques doing slam to British
traditional slams you have a many types
of slams or at slams thanks for
simultaneous localization and neck this
is a coolest thing in fact Sebastian
Thrun the founder of Udacity and Google
X I'm very lucky to get advice like him
and mentor by him that actually he is a
slam guy and he mentioned that the core
technologies of autonomous driving is
also in parts of mapping when I came to
Beijing in 4 days I realized that Wow
Beijing traffic is very very crazy
it's literally like hard to solve but
actually I'm
I'm a big optimist on this I've been
really optimist because white the
research technology and self-driving car
the key is not generally ever so network
ready guys
it's positive reinforcement learning
that's a truth you know about your q
factor in your q network and you have
the divisions of having the slam
combination between robotics and cell
running car with deep learning what we
have today
Beijing traffic is going to be very easy
it there's going to be something that
remarkably beautiful to have this
technology on the road and I think it
can happen so cool with a push of a
button you can go anywhere one big team
so again I mentioned again about the
challenge in robotics and AI platform
itself okay so we have to think here in
in self-driving car matrix you can do
this every car just only play without
using the burning take apart with people
learning approach just your body just do
a product perception planning
localization the car aiming books would
localize very good but it is it
simultaneously got the Velo dine which
is the like a company a really good you
got solid quick sensors for scan right
out from delphi or you got another laser
scanner from IPO or from regal you can
have it but the problem is is still in
development in-house as it could be
where are they how effectively how fast
in cycling through the cycles of you
capturing the data and you process the
data so again there is a rate in
computational process like what Steve
mentioned earlier there is a Nvidia
segmented markers using more into custom
chips that they built for self-driving
car try px Drive see eggs Jetsons those
are Nvidia products
what we need today what we understand is
how we can develop a better backup
system and robust 410 test faster with a
lower cost of technology so that's the
reason there is comprehension in matrix
that you need to have a real-time system
and back into the car back again to this
that's what we do in computer vision
open CV QT OpenGL right we know all this
stuff as a software engineer we we know
that there is a lot of challenges for
example like doing feature extractions
doing the for example with a very very
good example is optics sections like you
want to look at the Lightning's and then
the lights and then also the calibration
itself at the cameras and then also the
course sections on the objects so those
are the talented computer visions where
it lays out in the deep learning part
it's a nuclear forensic pre-processing
within computer visions it's actually
the biggest challenge today that what
happens developing an autonomous driving
and the third is infrastructure when you
get into the car have a nice car right
or even Tesla what's Tesla using is what
is mean a Android a loop Android auto it
can fall into this category as
infrastructure provider operating
systems in development does
infrastructures ok qnn from blackberry
that infrastructures so those those are
the software part of it how about the
hardware part how about the send
revolutions are the sensors there is a
noisy is it fully calibrated is it easy
to use
can you implement the semi unsupervised
unsupervised or even supervised learning
on that probably there is still a race
here and that's the reason it's
interesting market today because we
focus in the AI portion instead focus on
delivering physics and dynamics for the
sensors itself or the hardware
controller strategy we can deliver
also I mentioned in the earlier that
equal
works you have BCU you have you have PCM
Platzer modules and all the stuff those
are part of the pecan expert what our
PPO network so one other thing that is
interesting in the Vito networks that's
going to happen in next four five years
is absolutely vehicle security v2v v2x
v2i those are what happens in the
hardware and then or the controls
perspective it's more into making the
car becoming way way much aerodynamic or
what we call as a visual dynamics today
this is also combinations between the
controls PIDs proportional integrative
and derivations and also you have
hardware in loop those are the stuff
that what happens in in the core
technologies of selling car now it's
easy knowing nothing it's really hard
this is extremely hard and that's the
reason are in the stirrup
we focus in delivering discover portion
today with combinations of delivering
the hardware now what kind of work I do
how it works I mean how can I we deliver
this fast right what oh yes things I do
want make a new cars when we talk to
John that for example or talk with the
motorist companies in China you think
that oh you want to build a new cars you
want to build a future transportation
right you want to build a fancier car
that can transform into something wrong
you've all right platformers for example
like robotics right let's stop it that's
not how our main business model that's
not what we want liver and I talked to
my mom and then my mom's yeah you want
to make a hybrid electric cars no it's
not and my friends things my car is like
alien especially like Steve mentioned
that my car like from another space and
the government things I'm building
killing machines thinking about that how
about if it drives crazy and then how
about your insurance and all the stuff
well we sort of kind of roll it up
because we have a license to to do that
to test on the road and was it what I
think I do I do is have fun in cars
but actually what I do is using GTA 5 to
run our segment interview frames per
second that's what makes interesting now
what it means in terms of autonomous
driving so we talked a lot in the
beginning about how the process of how
autonomous driving works right we have
where ECU's we have Harvard layer lower
layer than the car itself
what's the real-time operating system in
the car and then again the big three
layers here is the sensing and thus
assessing and also the controls of the
cars these are the core problem today
that OMS trying to get into the consumer
market using AI or people learning in
today's one day now here's what our
achievements is I came back in China so
we took some couple pictures from China
Road and have our applications in the
road and luckily we always get correct
here we got the AV the our application
for object detection actually works
really well one it means that 100%
recognizing this is a Carm here and then
here to the bunch of cars here so it
recognized that I'm pretty happy with
this I'm pretty satisfied and I'm pretty
optimistic of course I mean you will
have nor did you do it how about with my
area or what it means by my house right
so well tell you what the overall
results of the a mic m84 this is
actually ninety eight point one percent
there is a chance that it still fails in
the small and heavy terrain like in the
sandstorm we didn't attempt this in the
text and song so we move the next how we
solve this one of a way is we using
learning driving behavior how Chinese
people try and sort of using simulator
either using gazebo or we have our own
closed loop simulator now Calipari's
lamp right what's the typical techniques
of doing slams that combined with the
deep learning this is interesting area
and then we
during this out pretty quickly because
we see that with it within the things in
one cloud and generating the data using
the slang techniques it's going to be
useful for the map so we have a platform
in our infrastructure that actually keep
generating the new data out based on the
point plus that we have now we test on
simulator we have our own framework that
leveraging from the real time not only
just only development like intentional
MX NAT pedal battle or cafe not only
that but we also test in the nursery
converse and that's the reason that RC
car is really useful in terms of
developing autonomous applications
before we go out on the road who is
doing this instead of one
LG's doing this and we thought that we
take the same approach like with all the
tests to make it ready better production
in the real car now out of the oldest
think again we go back and then we see
that how we test on the road how we map
exactly what how how way more did it how
way more did it using the Velo time so
they take the way of what they call as
reflecting so Rebecca Pat means that you
get you send the enterprise learning to
lay it out based on the point clouds so
on the left side is how it looks like in
the sunny day the result and the right
side is heavy rain night
so we correct the path called ghosting
ghosting is when when the lidar have
huge mean standard deviation and
standard deviations on the path of doing
the object tracking and using the salam
optimizations that we have we correct
that now this is actually how it it
performs in the highway driving so Oh
hopefully it front does it run
and it is Oh Oh
I'll see on the video later and this is
how we did the alchemist lat oh these
works we kept this cars to run more than
80 miles per hour fluctuates using a CC
adaptive cruise control so it runs
autonomously running this hundred Hill
racetrack similar to what happens in the
visual intelligence I'm using a calling
the how to be how our drivers react so I
think that's it thank you so much
Beijing oh one more thing I forgot to
say that there's more interesting thing
actually in in doing deep learning this
is just only one thing okay this is just
only one thing before I close I really
want to say this one of my friend that
worked in open AI
his name is dr. Ian good fellow which is
is really cool
professors and I really like him because
really smart guy I always want to hang
out with him in Stanford but um the way
you tell me that AI is a big pictures
right the huge pictures oil hole circles
of AI now inside of the eye the way you
learn this is you know more about
machine learning right I thought a lot
about unsupervised learning semi I'm
sorry supervised learning how you get
the feedback from label data and also
you take a regression you take the
statistic approach those are what
machine learning is so what machine
learning is is actually combinations of
the like that statistic approach and
make it output into something meaningful
for example you think about this way
have you ever have you ever thought
about how how you can tell to have a
more productive eighty testing
for the grocery market store when you go
to grocery market and you see that why
the aisle is so good how come they know
that I need sugar
I need tea I need a copy this week how
come they know that one of the things
that you can think about that is
actually soft using machine learning
don't ever think about depending on this
because deep learning is just gonna
makes you a little bit vague about this
really really vague
so machine learning can solve this by
using unsupervised learning
before you do a tea testing this is a
really good example for production
operation management that actually they
can put the iOS correctly based on the
customer behavior one other way to solve
this is using the k-means and clustering
that's a very popular technique and
that's still in terms of apply
statistics now you get that knowledge
which you already understand that good
enough at least you can you can code it
in a tubular notebook or using psych it
and heirs or all the stuff now you learn
about what it's called representation
learning this is interesting because I
earlier I mentioned about reinforcement
learning you have a for example in
self-driving car area this one
application is when you are in the
intersection you are in the intersection
and you're here and you have a traffic
light if United States if you're in and
traffic light you need to turn right if
you are here in the traffic light and
subtly green you need to move forward
and in turn light for example even
though the traffic light is red so these
types of the things that fall every
presentation learning and then this is
really good because it's called
combining with the fury of project shoot
I'm suddenly lost in my time oh yes the
game theory right you have a discount
factor you have a Q factor Q learning
these are what happens in really a
representation building orbit what
nowadays we call and reinforcement
learning now after you get the
a sense then you move to keep learning
those are the things that we learn today
we have label training data we want to
output model and output ways those are
the key how the AI assembles together I
hopefully this makes everybody clear and
I think this from Professor Ian foot
hello thank you so much
