Move 37; it's finally here.
Hello world! It's Siraj, and I've been teaching a course called "Move 37" for the past ten weeks
I wanted to culminate it all, every RL technique we've talked about, with this one video.
I'm gonna explain what Move 37 is, and three reasons why it's so significant for our future,
in terms of human jobs, health, and lifestyle.
The Go board game is very popular in large parts of Asia; it's very old and some players have dedicated their entire lives to mastering it.
When polled, most AI researchers believed it would take several decades for computers to reach the standard of a human expert player.
It's a very challenging game, with more potential board positions than there are atoms in the universe.
A player has to learn how to recognize abstract patterns in hundreds of pieces placed across the board,
and even experts have a hard time explaining why a particular move is either benificial, or problematic.
So when DeepMind's AI program, called AlphaGo Zero, beat the world champion at Go much earlier than expected, history was made.
Starting with the training set of recorded games that contained over 30 million moves made by expert Go players,
the AlphaGo Zero algorithm used a model proven to be able to learn from image data directly called a Convolutional Neural Network.
It was fed a series of game states as input meaning the positions on the board.
The network learned how to predict the next move, and learned to predict the outcome from different arrangements on the board.
Since Go has so many potential moves, a brute force search would take way too long;
even longer than it'll take Apple to release an augmented reality device.
So instead, the convolutional network was deployed in the context of a search algorithm called a Monte Carlo Tree Search.
This tree search was used to initially explore many possible moves on the board,
then focus that exploration over time as certain moves were found to be more likely to lead to wins than others.
AlphaGo used the neural network to both predict moves to help guide which branches of the game tree to search,
and it used it to evaluate the positions it encountered during its search.
This allowed it to intelligently search upcoming moves, and ultimately beat the world champion at the game.
But even more important than AlphaGo's win was one moment during the game; the moment that caused Lee Sedol to leave the room.
[AlphaGo played this move, which I wanna hear more about in a second, but, uh... Lee has left the room. He left the room after that move.]
It was a move that seemed terrible to everyone who saw it. But it turned out that Move 37 was an incredible move to play,
and it was instrumental in helping AlphaGo win the game.
Somehow, a computer program knew something about the game that we didn't.
Somehow, it's intuition was both different, and better, than human intuition.
One of the players, Fan Hui, said:
AI thinks you're beautiful too, Fan. *winks*
Move 37 offered us a glimpse at what an intelligence looks like that thinks differently than we do,
but is still very much capable of accomplishing tasks.
Interestingly, Lee played his own move in a later game with AlphaGo, since they were playing through a series of them,
which helped him win one of the games.
Move 78, called the Divine Move, or Brillaint Tesuji by some. (captioner's note: tesuji means "a good Go move" in Japanese")
The designers of AlphaGo called it a "one in ten-thousands move", because
AlphaGo calculated that there was a probability of 1 out of 10,000 that a human would play that move.
Lee's Move 78 caused AlphaGo to make sub-optimal decisions in the next round.
Lee learned that AlphaGo was superior at playing the right moves to optimize for small gains,
but it could fail in extreme situations where the gain of points in one move would be more important.
This was an example of intelligence augmentation:
Better designed algorithms lead to better performances, and better performances help humans make better decisions.
Humans who make better decisions can create environments where algorithms fail; i.e. "divine moves",
and learning about these failures helps us design better algorithms.
The AlphaGo algorithm influenced Lee's gameplay, and vice versa.
We can work together with this machine intelligence in many different ways,
to further extend our capabilities across every industry.
This kind of technology can help us achieve goals that we could not do alone, and I'll outline the three biggest ones.
The first is healthcare; cancer, Alzheimer's, and a host of other major diseases plague humanity,
many die every year because our best scientists haven't been able to find a cure.
In the USA, the path to develop a new medicine takes at least ten years on average,
and costs an average of $2.6 billion US dollars.
And less than 12% of the candidate medicines that make it into phase 1 clinical trials are approved by the FDA.
There are billions of possible lead molecules that could be used as potential cures,
so a scientist has to use their intuition to pick one that would most likely have the desired properties for curing the disease in question.
The process of synthesizing and testing a single new molecule in a lab could cost thousands or tens of thousands of dollars,
so the early stage guessing process is really important.
We can use AI to help generate a potential molecular structure, say using a model like Generative Adversarial Networks,
which have shown a lot of promise generating different types of useful data.
In these dueling networks, where one progressively becomes a better generator
by trying to fool the other into thinking what it's generated is real through a feedback loop,
feeding in vast quantities of molecular data that no human could process all at once could result in some incredible advances.
And human scientists could test what they've generated to see if it's a valid cure.
Along the way, they'd come up with more clues in the right direction, tweaking their algorithm  as they go along.
This is an example of the intelligence augmentation process realized.
AI could also help offer a second diagnosis for a doctor to help aid in their diagnosis,
giving them yet another perspective to consider to best help their patient.
The human body is itself an extremely complicated system, absolutely rich with data.
AI can help us ask questions that we never even thought to ask, and help individual humans by learning about their specific health needs,
delivering personalized medicine or treatments when they need it.
The second is software design.
We humans are not the best at writing software.
And when a software project becomes big enough, like say the Google Chrome browser,
it's hard for a single human, besides Jeff Dean, to understand the entire code base.
Imagine; a form of programming, where we give high level input to our machine,
and it decides on the implementation details by itself.
Take consensus algorithms, for example; the hype around cryptocurrencies has been around for almost a decade now,
and we still haven't figured out a way to create a scale-able blockchain consensus algorithm,
that doesn't demand so much electricity to run.
Proof-of-work is a human designed algorithm and so are the different blockchain consensus mechanisms.
There's so much theory that goes into designing these, from economics to cryptography to distributed systems,
to sometimes even physics.
It's a hard problem that will require a radically different way of thinking to come up with the solution;
perfect for an AI.
Google's AutoML is one example of AI automated software design.
It learns how to tune hyper-parameters for specific models,
and companies can use this all-in-one solution to fit their business needs.
Just give it a data set, and an objective, and in the ideal scenario, it learns which model to use,
what its hyper-parameters should be, and how it  should clean and regularize the data.
And this principle of AI assisted design doesn't just have to apply to software; it can be any creative endeavor.
Brandmark uses AI to generate a whole host of potential logos and themes for a new startup.
Just give it some initial parameters of what you want, and watch as it gives you many options.
And Google's Magenta project has resulted in some very cool web demos that let humans play music alongside AI.
The AI can learn your style of playing, and adapt, and you can do the same.
In this way, the duet becomes greater than the sum of its parts.
In fact, for any idea that we have, we can use AI to visualize it, assess the different possibilities,
and pick the best one, be it a scientific or creative endeavor.
And the third is a personal assistant.
The amount of data in the world doubles every two years. We are constantly getting flooded with a whole host of options,
of content, of choices, from the Internet, and different algorithms across the applications that we use
are constantly manipulating our behavior, tweaking our opinions and attention through subtle design tricks,
engineering the reward signals that are native to human psychology,
to get us to do something; whether that's clicking on an ad or buying a specific product,
we have to parse through all this noise for ourselves, to see what is the most meaningful to us.
This is a very, very hard task.
A personal AI assistant, stored locally, trained on our data, with a transparent backend,
could help us navigate the complexity of life in ways we did not think possible.
It could know us better than any human could, concatenating data points about us,
from our web browsing history, to our heart rate; it could tell us how to best achieve our goals,
whether that's in fitness or in school, in a career or in a relationship.
We can use this feedback to help guide our own decisions.
Move 37 also catalyzed the current AI renaissance that China is currently undergoing.
In the West, Go isn't as common a game, but in China, it's a highly respected game,
many even believing that it's a sacred game, connecting humans to the universe in its complexity.
AlphaGo defeating the world champion at a game thought to be so sacred in China was huge; bigger than in the West,
because the general population is more familiar with just how complex the game is.
The Chinese government promptly created an AI development plan to become the world leader in AI by 2030 and it's definitely executing on it.
Ultimately, automation technology will help free humans from labor based jobs,
and create new classes of jobs that we actually enjoy doing.
I hope you liked the video! Please subscribe for more, and for now, I've gotta try to play Go... somehow.
So... thanks for watching!
