SANDY BLYTH:  Hello, and
welcome to our Frontiers in Machine
Learning online event.  I'm
Sandy Blyth, and it's a pleasure for
me to introduce you to this
Microsoft Research event.  This is
something new for MSR at a
unique time for all of us, when so many
of us cannot travel to have these
discussions in person. 00:00:23
Frontiers in Machine Learning
continues a long-standing investment
in engagement and collaboration
with academia and the broader
research community and replaces
our North America Faculty Summit
for 2020.  That event has for over
20 years been an opportunity to
gather together and renew old
acquaintances, meet new people, and
discuss research of common interest.
Now, while this event cannot
fully replace the experience of an
in-person summit, we do hope
to achieve many of the same
objectives.  We've had input from
many members of the community on
the talks and the topics of our
agenda, and I hope you find the
time worthwhile.
00:01:01
You can see that from the
agenda we're aiming for a balance of
presentations and panel
discussions, with opportunities to ask
live questions of the participants,
as well as explore additional
materials, and do some virtual
networking.  As Frontiers continues
our investment in research
connections, so too we continue with
and expand our support for
students and faculty in pursuit of
state-of-the-art research across
a wide variety of disciplines.
And I'd like to take just a moment
to overview some of those award
programs from around the
world that I hope you will find of
interest.
Our Research Fellows Program
in our Bangalore, India, lab is a
predoctoral program that provides
one- or two-year roles in our
MSR India lab and offers access
to state-of-the-art technology and
the chance to work side by side
with world-class researchers from
across a variety of disciplines.
Our Ada Lovelace Fellowship
offers a full three-year tuition and
stipend for Ph.D. students
in their second year at a North
American university who are
from underrepresented groups in
computer science.  It aims to
address some of the structural
obstacles to diverse students
by providing an additional year of
support beyond that of our
North America Ph.D. Fellowship.
The North American Ph.D.
Fellowship continues and remains a
two-year program for third-year
students in North American
universities and, like the Ada
Lovelace Fellowship, offers an
opportunity to interview for an
internship at MSR and attend our
Ph.D. Summit, which is a two-day
workshop we hold annually in one
of our North America labs.
The MSR Asia Fellowship
offers a unique program, including
mentorship, research,
networking, and other academic
opportunities, and includes a
cash award, an optional internship
in our Beijing lab, and attendance
at select MSR Asia events.
The MSR Dissertation Grant also
recognizes the value of diversity
in computing and supports Ph.D.
candidates in their fourth year or
later from groups, again, that
are underrepresented in computing.
And we're pleased to have recently
announced the newest group of
10 students who have been
awarded the Dissertation Grants for
2020.  This is our fourth year of
offering the Dissertation Grant,
and this was by far the most
competitive year yet.  Our
congratulations to this
year's outstanding recipients.
Our MSR Faculty Fellowship is a
two-year, $100,000-per-year award
which recognizes promising
new faculty at North American
universities whose talent marks
them as emerging leaders in their
fields.  Here as well, we've
recently announced our 2020 Faculty
Fellows from nearly 200
nominations and who are distinguishing
themselves across a diverse
set of research interests.
Congratulations to our
2020 Fellowship winners.
Now, we've recently added two new
awards in EMEA and Latin America
for students in their third
year or beyond at a university in
those regions.  And here are
the first set of award winners from
both EMEA and Latin America,
and they each receive a cash reward
to help them complete their
research and also to have travel and
accommodations provided to
attend our Ph.D. Summit.  And we offer
our congratulations
on the award.
Finally, we continue our Ph.D.
Scholarship program in EMEA which
provides an annual payment of
up to three years for supervisors
and students to do collaborative
work on research themes aligned
to our MSR UK lab in Cambridge,
England.  Recipients may also
receive an internship to one of
our worldwide labs during the term
of that scholarship.
These are just some of the
faculty and student scholarship
programs that are available,
and if you'd like more information, I
encourage you to check out
the academic programs tab on our
Microsoft Research website.
Now, Frontiers is a virtual
event as a result of the COVID-19
pandemic.  MSR is currently
supporting a robust research agenda on
the novel coronavirus as well
as extending that research to
prepare for future pandemics.
And we're pleased to have
announced just last week awards of
funding to nine different projects
involving collaboration between
Microsoft and 19 different
institutions to advance knowledge on
infection prevention and control,
treatment and diagnosis, the
ethical allocation of resources,
mental health, and return-to-work
topics.
Thank you for the collaboration,
and congratulations to these
institutions where researchers
continue to work with MSR on
pandemic preparedness.
Also of note, Microsoft Research
and our Microsoft AI for Health
team have partnered with
CIFAR, the Canadian Institute for
Advanced Research, on their AI
and COVID-19 Catalysts Grants
Initiative.  We're proud to
support the program in accelerating
COVID-19 research by leveraging
our Azure high-performance
computing research.
In addition to these programs,
the pandemic has brought into
sharper relief the need to
upgrade the resource available for
research.  MSR supports world-class
research and echoes the call
for a national AI research resource,
as recommended by a recent
National Security Commission
on AI.  Eric Horvitz, the Microsoft
chief scientist, was a
commissioner on this report.
In addition to providing AI
supercomputing cycles, openly
available data sets are necessary
to advance the state of the art.
Microsoft Research's Open Data
Repository has made available
curated data sets that Microsoft
researchers have used in
conjunction with their own
research.  The site is enabled to
simplify access to data sets and
enable the reproducibility of our
research.
I'm pleased that we have today
made the source for ODR available
on GitHub.  By example, the
Emergent Alliance, which is a
not-for-profit community which
aims to inform future economic
decision-making and to aid in
societal recovery post COVID-19, has
recently used those sources from
GitHub and instantiated their own
Open Data Repository
of relevant resources.
Machine learning is a rapidly
changing landscape, and within the
larger scope, it's important
for us to recognize the dramatic
events and profound societal
change underway as we all look to act
and address systemic racial
injustice around the world.  I won't
recount the things that I'm
proud of Microsoft that it's doing
because there's so much yet to
be done.  We must do better, and we
will do the work.
We'll sustain the three company-wide
multi-year efforts.  We'll be
increasing representation and
our culture of inclusion by adding
over $150 million to our
current diversity and inclusion
investments and committing
to and reporting against specific
representation of blacks and
African Americans in leadership
positions in the U.S.
We'll be engaging our ecosystem
to use our balance sheet and
engagement with our suppliers
and partners to create new
opportunities.  And we'll be
strengthening our communities with
the power of data, technology,
research, and partnership to
improve the lives of blacks and
African American citizens across
our country and the safety and
well-being of our own employees in
the communities in which they
live.  The opportunity and the
obligation for change is here.
You expect it of us, and we expect
it from ourselves.  We're
going to act with intention.
So clearly there's a lot going
on around us and with MSR, and I
encourage you to stay in connection
with us through any of the Web
and social channels and
to subscribe to our featured
communication.  We are at
our best when we're in close
collaboration with this community,
and I hope you'll find that
some of the very best of that
is on display this week during
Frontiers in Machine
Learning.  Thanks.
And now I'm pleased to
introduce a fireside chat between
Christopher Bishop and Peter
Lee.  Christopher Bishop is a
Technical Fellow and the lab
leader of our MS UK lab in Cambridge,
and Peter Lee is corporate
vice president of research and
incubation at Microsoft.
CHRISTOPHER BISHOP:  Thank
you very much, Sandy, for that nice
introduction.  Hello, and a very
warm welcome to the Frontiers in
Machine Learning event.  My
name is Chris Bishop.  I am a
Technical Fellow at Microsoft,
and I'm also the lab director of
the Microsoft Research Lab in
Cambridge, UK.  Today I'm delighted
to be talking with Peter Lee,
who is a corporate vice president
and who is in charge of research
and incubation at Microsoft
Worldwide.  Peter, it's a real
pleasure to have you with us today.
PETER LEE:  Thanks, Chris.
It's really, really exciting to be
here.
CHRISTOPHER BISHOP:
Great.  And congratulations on your
relatively new role now as head
of research and incubation.  I
guess you started, what, two or
three months ago.  Maybe we could
just sort of dive right in, and
perhaps you could tell us a bit
about, well, first of all, why
did Microsoft choose to bring
research and incubation together
under the same roof, as it were?
PETER LEE:  Yeah, you know,
I think in a way, that question is
really central, and it's left as
an exercise, actually, to all of
us in Microsoft Research and
in the incubations.  But I think
there are a couple of factors
that went into the thought process
behind this.
One is that over the last few
years, certainly in the Satya
Nadella era from Microsoft,
research and research-powered ideas
and, maybe most importantly,
researchers themselves have been
getting increasingly involved in
creating new technologies, new
engineering capabilities, and
actually new lines of business and
new products for Microsoft.
And I think that's been a direct
response to the way the industry
has been going.  So when you
look at things like where silicon is
going in the cloud or the whole
idea of confidential computing or
all of the kind of intensity
of activity in large-scale NLP
pre-trained models, all of
those things and more are all
fundamentally research powered
and, furthermore, require the kind
of mindset and world
view from researchers.
And so the idea of linking
together what we do in Microsoft
Research and sort of making it
more possible to capture emerging
ideas and turn them into new
possibilities from Microsoft that's a
so-called incubation function,
there's a desire to somehow make
that work even better than it
has over the last few years.
And I think from the perspective
especially from Microsoft
Research, it's pretty exciting
because, you know, if you think
about people like Doug Burger or
Galen Hunt or Lily Chang or many,
many others who have sort of
gotten involved now in creating very
significant new possibilities, or
people like Johannes Gehrke, who
started off as researchers
and then had stints leading very
significant groups and now
coming back to Research, that sort of
interplay between research and
what the company is doing at the
large scale is just getting
incredibly important.  And so it has
sort of just made logical
sense to try to create a single
organization that just tries to
maximize the benefit of that sort
of thing.
There's one other element,
too, just -- and I know I'm being
slightly long-winded here,
but I think there's also been the
desire in this setup to help
organize and reunify all of Microsoft
Research under one roof. As
you know, it's been a little bit
fragmented over the last few
years, and the hope is that we'll be
able to build on the reputation
and the impact and thought
leadership of Microsoft Research
as a whole by bringing everything
together.
CHRISTOPHER BISHOP:  Thanks,
Peter.  And I agree, I think it is a
very exciting development, not
only the reunification but, as you
say, bringing us close to
incubation.  Many researchers love the
fact that at Microsoft we have
the opportunity to reach hundreds
of millions of people, and just
anything that reduces the friction
of that has got to
be a good thing.
Actually, when you took on the
role, of course, you had a rather
unusual start.  I think you're,
what, a few days into the role,
and then our chief executive,
Satya Nadella, asked you to sort of
put things to one side and really
focus on the company's response
to COVID-19 and think about
how there might be opportunities for
technology to really help the
world respond to this pandemic.
So can you tell us a little bit
about that experience and some of
the projects that were spun up
and any sort of early results that
we have?
PETER LEE:  Yeah, sure.
And, first of all, let me just start by
saying it has been kind of a
crazy time.  And I know crazy is both
a positive and a negative,
just given the seriousness of the
pandemic crisis.  And also,
Chris, let me thank you for your
patience because we made this
change, and then you and I were put
back together, working very
closely.  And then five days after
that change happened, I had
to kind of put things on hold for a
little while.  And your patience
with me I think is -- I really do
appreciate a great deal.
Because, you know, what happened
was we made this change and we
created this new research and
incubation organization.  I think
that happened on a Monday.
And then on that same week, on
Thursday, Satya, Kevin Scott,
and Kurt DelBene asked me to put
things on hold and focus on
coordinating Microsoft's response,
science and technology
response to the COVID-19 crisis.
And there was frustration because,
you know, it was hard to know
how to get your ideas heard
and seen and how to recruit and
mobilize resources.  So one of
the things that just to try to get
a handle on this was we worked
with the garage to stand up the
Hack Box platform that's used for
our annual hack-a-thons, and we
created a Hack Box site so
that people who just wanted to
volunteer their time could browse
projects that were being posted
and join them.
If you had an idea for a project,
you could write them down, the
descriptions, and get them
surfaced and recruit people.  And then
we had a V team set up involving
mostly people from Microsoft
Research to triage through
all of those projects and try to
surface ones that might benefit
from more focused attention.  And
that whole process I
think was really good.
One thing that Microsoft has
really trained itself to do is to do
hack-a-thons.  And so by the time
we closed that down, there were
1,100 Microsoft employees who
had joined projects, and there were
186 projects that were set up.
And each project had, on average,
about five people or so.
Out of that, there were a couple
dozen that kind of got plucked
out and really got a lot of very
focused attention.  And some of
them have had tremendous
impact.  One, of course, was in direct
crisis response for hospitals
and clinics that involved the Health
Bot technology that's built on the
Bot framework.  And the problem
there was, you know, people were
flooding emergency departments in
hotspot areas, were calling
call centers that were manned by
nurses.  And it was just
people getting overwhelmed.
The Health Bot was set up in
collaboration with the U.S. CDC to
have a self-assessment protocol
and then a smart handoff to a
telehealth session to a call
center or possibly to an emergency
department or a
drive-up testing center.
And by now, over 2,100 hospitals
and clinics around the world have
installed this.  And the official
Self-Assessment Bot for the CDC
is using this.  So that's been
tremendous.  And the hospitals that
we talk to are reporting a 30
to 40 percent reduction in call
center or telehealth
volume as a result of that.
Another project is -- has to do
with diagnostics.  Working with
Adaptive Biotechnologies, we've
been involved in a deep analysis
using machine learning of T cell
response to COVID-19.  And that's
resulting in a data set called
Immune Code.  That's an open data
set for researchers.  And it is
already looking like, in the first
tranche of data that's been
published, that there is a new type of
diagnostic that's based on T cell
response as opposed to antibody
measurements or direct virus PCR
analysis that would be much more
precise, much higher sensitivity
and specificity, as well as catch
and diagnose disease early on.
And that's been sort of in a realm
of a large number of other
activities in support of new drug
therapies and vaccines.
And then there's been just
tremendous amount that just has to do
with public health, just analyzing
capacity and utilization for
things like where are the next
hotspot areas, where are the
vulnerable populations in various
countries and how well do those
things match up with supply of
intensive care units, ventilators,
PPE equipment and so on.
And so just a tremendous number
of projects like this that I think
we should all feel really proud
of.  I think Microsoft's response
just really has made a
difference and continues to make a
difference.  And out of that
whole hack-a-thon effort, over a
third of the participants and
over a third of the projects came
out of Microsoft Research,
which I think is just really amazing.
And it's really brought Microsoft
Research front and center in the
company's response to COVID-19.
CHRISTOPHER BISHOP:  Clearly,
healthcare more generally is a major
opportunity, really, for machine
learning to have impacts, I know
something that's very close to
your heart.  And I wonder if we
could just step back a little bit
because, of course, before you
took on your current role, you
spent several years building up the
healthcare activities in Microsoft.
And can you share with us
some of your thinking about
Microsoft's strategy in the healthcare
space or even why Microsoft,
why should Microsoft be involved in
healthcare at all?
PETER LEE:  Yeah, and, Chris,
you shouldn't give me too much
credit for this because you know
you yourself have been very much
involved in this, and the Cambridge
lab in particular.  And it has
been something that has involved
not just research but also the
commercial business, teams in
Azure and in experiences in devices.
You know, for me, I think the
way I thought about this was in
three stages:  relevance, value,
and transformation.  And they
kind of came in stages.
When Satya asked us to take on
healthcare, the first order
of business was this issue of
relevance.
And what I mean by relevance is
how would the stakeholders in the
world of healthcare understand
that Microsoft had something to
offer.  So how could we be
relevant to the healthcare industry, to
healthcare providers, hospitals,
clinics, health systems, to
insurance, providers and other
payers, to the bio pharm industry,
to medical technology companies,
startups and so on.  So there's a
relevance there that we had to
somehow figure out how to earn.
Because, by doing that, then we
could get into collaborations and
partnerships and
start to learn more.
But then there's also relevance
within Microsoft because
healthcare is one of these
areas where everyone has direct
experience with healthcare.
Everyone has an opinion.  That
experience tends to be colored
by people's personal contact with
doctors and nurses and hospitals
but is largely ignorant of the
much, much bigger world of
healthcare that happens behind the
scenes.
And so we had to work to
earn credibility and relevance
internally.  And so to do that,
you know, that really meant trying
to find the right partners across
Microsoft.  And early on I made
the decision to partner closely
with the commercial business led
by Judson Althoff and by
Jean-Philippe Courtois to really identify
key early partners that we
could work with, like the NHS, like
Humana, like Walgreens and so
on.  And by doing that, we earned
both internal and
external relevance.
Then the second stage was value,
and that was largely about data
and AI.  There's a tremendous
amount happening with healthcare
data today around this problem
of what's called interoperability,
really trying get health data
flowing in a standardized format to
where it needs to be and making
it more susceptible to machine
learning and data analysis.
So we've done a huge amount
of work to sort of evolve Azure,
Dynamics, and Microsoft 365
so that they speak the language of
health data.  So you hear things
like FHIR and SMART.  And so
these are sort of new emerging
standards for health data.  And
then the AI is tremendously
fundamental and important.  Huge
amount of that health data is
unstructured text.  So NLP and
machine reading become incredibly
important; computer vision, to
really understand medical
imaging; understanding molecules;
understanding the human
genome.  All of these things.
Understanding the immune system
and the immunome.  All of these
things end up being fundamentally
machine learning and AI
problems.  And so that's
another area that we've really been
focused on.  And trying to build
up that technology stack for each
of these things and then get
things out as products has been the
big challenge.
And your question, Chris, why
are we doing this, my favorite
example of this is to try to
get people within Microsoft to
understand the global market
for healthcare is estimated to be
about $7.5 trillion.  Now, you
know, what does that mean?  Let's
just take one company that we
work pretty closely with in the U.S.
It's a company called Optum.
And what Optum does is they handle
the data for medical claims
to route them from healthcare
providers to payers and then the
remittances from payers back to
providers.
So that data stream that goes
back and forth is a very important
function in the healthcare
system in the United States.  And
there's a lot of data analytics
that helps facilitate that in both
directions.  And Optum is the
second largest provider of that
service in the U.S.
healthcare system.
So that niche market in Optum
sustains a company, Optum, that has
the same head count and the
same annual revenues as Microsoft all
up.  And so if you think about
the possibilities in this massive
shift that's happening right now
of healthcare to the cloud, there
is no reason why healthcare in
our cloud shouldn't be bigger than
all of Microsoft's current
business combined.
And, of course, one of the
most interesting points of our
collaborations together has been
about with Novartis.  And so it's
just something that we've all
been so excited about.  So it'd be
great to hear from you a
little bit more about that.
CHRISTOPHER BISHOP:  Yes,
this is a really exciting opportunity.
It's also very different sort of
mode of operation, I suppose, for
Microsoft Research.  I've been
privileged to be part of MSR for
over 23 years now, and
historically we would do a lot of basic
research, from time to time we
would transfer technology into
products, the products would
ship, customers would get to use
them.  And so it would have
real-world impact, but it was this
very -- this long chain of process
by which we connected with the
real world, as it were.
And now in the partnership with
Novartis, working directly with a
customer.  And it's very
exciting, I think, very relevant in the
world of machine learning in
this new data-driven world because
we're no longer thinking about
one-size-fits-all technology that's
sort of put on a disk and shrink
wrapped and sent around the
world, things are much more
bespoke now, bespoke to a particular
domain and even to a particular
collaborator and a particular
application.
And so we're working very
closely with Novartis under this new
five-year partnership.  It was
signed last year.  It kicked off in
January of this year.  There
are different components to it, but
the piece that I'm looking after
is the research component, the
longer-term component
of this collaboration.
And it's really a peer-to-peer
partnership between scientists in
Microsoft Research and scientists
in Novartis, bringing together
their amazing expertise in pharma
and amazing data that they've
been building up, along with
our expertise in machine learning
and, of course, leveraging
Microsoft's cloud with the storage and
the very powerful compute that we
have, and seeing how together we
can go after some really tough
challenges that just wouldn't be
possible for either
organization to do on their own.
So we've started a number of
projects.  We're probably not yet
talking publicly in detail about
those projects, but I can share a
sense of the kinds of things that
we're working on together.  One
of the things we've done is to
think about leveraging the sort of
expertise in technology that we
already have in Microsoft Research
and seeing how that's applicable
to some of the challenges in
Novartis.
For example, we have a project
called NRI that was started a few
years ago, and this really
looks at medical imaging and in
particular the segmentation of
three-dimensional medical images
such as CAT and MRI.  And there
are various applications for this,
but one important one is
so-called radiation therapy planning.
So if somebody has a solid
tumor that's going to be treated with
radiation, then there's some
software that has to optimize the
three-dimensional shape of
the beams to sort of maximize the
damage to the tumor and minimize
damage to surrounding tissue, and
especially to vital organs.
And so in order for that
software to work, it needs a
three-dimensional map of that
tumor.  That's where NRI comes in.
So at the moment, radiation
oncologists will take the 3D scan and
they'll literally go through
this slice by slice with the stylus
on the computer screen marking
out the boundary.  And that can
take 20 minutes for a simple
case.  Or if it's metastasized, there
are multiple tumors, it can take
several hours.  It's painstaking.
It's tedious.  You kind
of have to be accurate.
And this is where NRI can
really help the workflow for the
radiation oncologists by
automatically, and in the space of a few
seconds, producing a sort of
candidate segmentation.  And the
human expert can then go over
and fix up any little details that
they want to change.  But it
really speeds up that process.
That technology is actually being
used in research environments,
quite extensively in a local
hospital in Cambridge called
Addenbrooke's, a very -- one
of Europe's largest teaching and
research hospitals, and we're
delighted to see that technology
being explored in that way,
in clinical practice, effectively.
PETER LEE:  You know,
Chris, the thing that I think that
particular example highlights
is, again, how important Microsoft
Research is.  Because it's not
that -- it wasn't possible for the
NRI application to just take
an existing off-the-shelf machine
learning or computer vision
system or even an off-the-shelf
machine learning algorithm
or computer vision algorithm.
Something new had to be
developed specifically for that
application to work well.
And, you know, it really takes a
world-class research organization
to be able to do anything like
that.  And so it really highlights
just how important the
partnership with an organization like
Microsoft Research is.
CHRISTOPHER BISHOP:
Yeah, I think that's right.  I think it's
that intersection of both deep
research and real-world application
that gets a lot of researchers
excited.  I mean, for me, it's the
thing that gets me out of bed
in the morning, the fact that we
have an opportunity to have
impact on the real world, in the case
of healthcare, improve lives,
save lives.  And yet that's enabled
by, first of all, tackling some
very hard research problems.  So
it's that combination of deep
research and real-world impact that
for me, at least, is
tremendously exciting.
PETER LEE:
Yeah, for sure.  Yes.
CHRISTOPHER BISHOP:  Another
nice example of something we're doing
with Novartis really goes to the
heart of their business, which of
course is creating new drugs,
new therapies, which effectively
means discovering new molecules.
What's interesting here is that
the nature of the data is
rather different from many other
applications.  If you think about
let's say imaging again, images
tend to come at a fixed size or
you can resample to a fixed size.
So the neural network always
gets data in the same format, the
same dimension, as it were.
But molecules are interesting
because obviously they vary in size
and shape and configuration,
and so you can't take a simple
representation of a molecule
and treat that as the input to a
neural network because it
has this variable configuration.
There's some techniques called
graph neural networks that were
already pioneered in Microsoft
Research which address that problem
of how to take machine learning
and apply it to data which has
variable size and structures,
things like molecules.  And so,
again, that's a really nice
example of taking some of the deep
research from Microsoft Research
and combining that with the
expertise that Novartis has in
understanding the relationships
between the structures and
molecules and their biological
activity.  And that's a project
which it would be difficult to
imagine either group on their
own doing such a good job, but
together we can do things
that I think are really very, very
interesting.
PETER LEE:  Yeah.  And,
again, you know, I think one of the
interesting scientific challenges
is the -- you can't hope to
solve this problem purely based
on the data nor purely on the
basis of our understanding of
the chemical processes.  You really
need a combination of the two.
CHRISTOPHER BISHOP:  Sure.
And I -- and I -- I think one of the
really interesting things about
healthcare, of course, is the
opportunity for real-world impact
and benefiting society.  I think
also healthcare really throws
a spotlight on many, many deep
challenges, research challenges
in machine learning.  And, in
fact, I know we're going to be
looking at many of those during
this Frontiers in
Machine Learning event.
And this may be a good moment,
then, just to play a little taster
video.  So this is a little clip by
Besmira Nushi, who's a senior
researcher at Microsoft, and
she's going to be leading a session
in this event on machine
learning reliability and robustness.
Let's just have a
little look at that clip.
BESMIRA NUSHI:  Hi, everyone.
My name is Besmira Nushi, and I'm a
researcher in the adaptive
systems and interaction group at
Microsoft Research.  This
year at the Frontiers in Machine
Learning event, we are organizing
a session on machine learning
reliability and robustness to
discuss recent work on reliability
guarantees of machine learning
algorithms but also on how such
guarantees translate
to the real world.
This session will review properties
of machine learning algorithms
that make them more preferable
than others from a reliability
lens, such as consistency,
interpretability, or generalization
instances.  And second we will
also review tooling support that is
needed for machine learning
developers to verify and build with
these properties in mind.
We will have three well-known
researchers in the field to share
their work and thoughts.  Tom
Dietterich is a Distinguished
Professor at Oregon State
University, and he will talk about novel
category detection in machine
learning and vision.  Ece Kamar is a
senior principal researcher in
MSR, and she will share her work on
blind spot detection in the open
world.  And finally, Suchi Saria,
who is a professor at Johns
Hopkins University, will talk about
the work that is done in her lab
for safe deployment of machine
learning under various
data shifts with applications in
healthcare.  In the last 30
minutes, we will also have an open
discussion among the speakers
and the audience.  Thanks so much
for attending, and very much
looking forward to meeting you
virtually at the session.
CHRISTOPHER BISHOP:  Okay,
so that's a pretty impressive lineup of
participants that Besmira has
assembled for that, and I'm really
looking forward to that session.
So, Peter, we've talked a lot
about COVID-19, and, of
course, another big impact from this
global pandemic has been this
amazing shift towards remote working
and working from home and an
associated sort of explosion in the
use of remote collaboration
technologies, things such as Microsoft
Teams.
Now, Johannes Gehrke is a
Technical Fellow at Microsoft, and he
recently transitioned into our org
to become the managing director
of our complete portfolio of
research activities in Redmond.
Johannes, prior to that,
was responsible for things like
large-scale engineering
activities in Microsoft Office and
specifically in the area of the
scalability of AI and Microsoft
Teams.
Before joining Microsoft, he was
a professor at Cornell University
where his scholarly work earned
many accolades, including election
as an ACM Fellow and an IEEE
Fellow, and he also won the 2011 IEEE
Computer Society Technical
Achievement Award.
So I think Johannes is probably
the ideal person to share with us
his thoughts on the changes
that we're seeing in productivity and
the technology that supports
that, more specifically how machine
learning can further empower us
in that world.  So a few days ago
I spoke with Johannes about his
new role, and I asked him about
the impact of remote working
and also the power of machine
learning to improve this
technology.  So let's have a little look
at that interview.
We're delighted that you've
joined Microsoft Research.  I know
you've only been in role a very
short time, but can you tell us a
little bit about the new
role and what it will involve?
JOHANNES GEHRKE:  Yes, I'm
in the role now for about four weeks,
and it's really a privilege to be
here.  I mean, there's so many
amazing people, there's really
a huge breadth of research going
on, and you'll see this throughout
this conference as well.  So I
think what I can bring to
the role is that I have seen the
different parts of research,
both at a university and now I'm
experiencing here at Microsoft
Research, but I also bring a good
understanding of what it means
to be both in a startup as well as
in a product team.
So especially here in the
last seven years while I was at
Microsoft, I've gained a deep
understanding and a lot of empathy
of what it means to ship
products and how to scale software
development to hundreds of
engineers, how to think when you're in
a product group, when you have
to develop new features, when you
have to talk to customers.
And one of the challenges
in working with all the existing
systems, because every new
system that you build here at Microsoft
often has some legacy components
to it, and you have to bring
these legacy components into the
new world.  So I hope to leverage
this breadth of knowledge both
from the research side as well as
from the Microsoft internal
side here in my new role.
CHRISTOPHER BISHOP:  So our
focus here, of course, on machine
learning.  So what are your
thoughts on the role that machine
learning can play in terms of
productivity, tools, and technology?
JOHANNES GEHRKE:  Yeah,
so there are -- there are quite a few
interesting ideas.  I think the
first one, of course, is to look
at the audio and video stack
and look at where there is often a
lot of old control theory,
whether we can replace that with
machine learning.
We are recently -- we're about to
ship noise suppression where we
basically take an old stack option
noise suppressor and replace it
with machine learning.  And the
advances there are really amazing.
And this is also a good example
where machine learning research is
playing a big role, but also the
gap between the papers that were
published and actually what can
be shipped was really big.  And so
we had to do a lot more work to
make the models more performant
and also work for the large
variety of noises that we actually see
in practice.  So basically the
whole control plane, maybe even the
data plane of the audio/video
stack can be replaced with machine
learning, in my opinion.
Second of all, there might
be very interesting user-facing
features.  If you think right now,
we have a feature where I can
raise my hand, but then people
forget to take their hand down or
when they're done speaking.
So I think there is a lot of
user-facing features where
we can just ease the level of
interaction through the subtle
signals that we usually see when we
talk with each other one on
one or in a physical that we don't
really have in a
virtual setting yet here.
CHRISTOPHER BISHOP:  I
think it's really interesting, isn't it, to
see just how ubiquitous machine
learning is becoming and, as you
say, having at these sort of
more traditional problems are now
being replaced with machine
learning solutions that many times
work better because they're
tuned to the particular data or the
particular environment in which
they're actually used instead of
being sort of general purpose.
I think that's -- I think that is
one of the big frontiers of
machine learning these days.
Thank you very much, Johannes.
And if you'd like to hear more
from Johannes, then the full
15-minute interview will be available
in the highlight section of the
website.  So, Peter, I think it's
petty exciting that we've managed
to persuade Johannes to come and
join us in Microsoft
Research, don't you think?
PETER LEE:  Yeah, well, let's
face it:  Johannes is genetically a
researcher, so he belongs in
Oregon.  So I think it's great.  And
I know we're all very excited to
have him.  I've actually tracked
Johannes's career every since
he started his professorship.  He
finished his Ph.D. at the
University of Wisconsin.  Actually, his
Ph.D. advisor is Raghu
Ramakrishnan, who's now in Azure, of
course, also a Technical Follow.
And Johannes did some wonderful
work as a professor at Cornell
and then joined Microsoft. 00:42:44
Now, I think the intent when he
joined Microsoft was always to be
a part of Microsoft Research,
but he quickly got sucked into some
significant engineering
leadership opportunities in various
product groups.  And so it's
just thrilling to have him be here.
And, you know, as always,
almost everything is infused with
machine learning.  And so
maybe, Chris, this brings us back to
you, because you've been, of
course, one of the pioneers and one
of the worldwide leaders in
machine learning for the past 30 years
or more.  It'd be interesting to
get your thoughts on, you know,
how has the field changed and
evolved over the 30 years that
you've been in it.
CHRISTOPHER BISHOP:  I think
for me, though, the biggest shift
over those 30 years has been
really in terms of the emphasis of
the field.  Because for most of
those 30 years, certainly for the
first 20 of those 30 years, if I'm
honest, machine learning didn't
really work that well.  There was
a lot of excitement.  Everybody
understood the potential.
Intellectually it was fascinating.  But
the reality was that the performance
of many machine learning
systems was really not adequate
for real-world use.  There were
maybe a few niche applications,
but mostly it didn't really live
up to the promise
or the excitement.
And, of course, that's really
changed in the last decade, and
especially through the development
of deep neural networks and
deep learning and scaling up to
large data sets and large amounts
of compute.  And so today
we're in a world where there are
literally thousands of applications
of machine learning.  I mean,
most people have used several
already today, probably without even
knowing it.  It's
becoming ubiquitous.
And that means that, although
we continue to have a strong focus
on the performance in the
sense of the accuracy of machine
learning, we always want to make
it more accurate.  Because we're
now using machine learning in
real-world applications, it opens up
a whole raft of new challenges.
I think of it as a sort of
penumbra of research questions
that surround the core question of
just getting the machine
learning to work at all.
I think of things like biases
that creep into the predictions
because of biases in the data
set, for example, thinking about
fairness, thinking about
explainability, thinking about causality,
if we actually want to make
interventions on the basis of
predictions.  Adversarial issues.
You know, nobody was going to
attack my NeurIPS paper 20 years
ago, but once you put something
online and you've got a hundred
million people using it, there are
then adversarial agents out
there, people with ill intent who will
attack it in all sorts of ways
for a variety of different reasons.
So we have to worry
about those issues as well.
And actually this is probably a
good point to show another of the
taster videos, because I
think this is very relevant to this
discussion.  This is one by Rich
Caruana.  He's a senior principal
researcher at Microsoft Research,
and in this video he's talking
about an upcoming session
that he's organizing at this event
called interpretable machine
learning.  So let's hear from Rich.
RICH CARUANA:  Hi, I'm Rich
Caruana, a machine learning researcher
at Microsoft Research in Redmond.
This session is about saving
lives by using interpretable
machine learning in healthcare.
Because healthcare data is
very complex, it's critical to use
interpretable machine learning
methods to make sure that the
models we train are safe to deploy
and used on real patients.  One
challenge is that most patients
are already receiving treatment,
and that can cause confounding
in the data.  For example, a model
might learn that high blood
pressure looks like it's good for you
because the treatment given to
you when your blood pressure is
high lowers your risk compared
to healthier patients who have
lower blood
pressure to begin with.
There are many ways this kind of
confounding can cause models to
learn to predict very risky
things.  In the first presentation,
I'll talk about problems like this
that we see in healthcare data
thanks to the interpretable machine
learning methods we're using.
In the second presentation,
Ankur Teredesai, from the University
of Washington, is going to talk
about fairness in machine learning
when it's used for healthcare.
And in the last presentation,
Marzyeh Ghassemi, from the
University of Toronto, will talk about
how interpretable, explainable,
and transparent AI can actually be
dangerous when used in healthcare.
Looks like an exciting lineup,
so please join us.
PETER LEE:  Wow, Chris, that's
just awesome.  And, you know, what
an exciting session and some
amazing speakers.  You know, the
whole event is about and in
fact called Frontiers in Machine
Learning.  And so it'd be
interesting I think for people to hear
what are your -- what are Chris
Bishop's views about the important
frontiers and where is the field
heading over the next few years?
CHRISTOPHER BISHOP:  Okay.
Thanks, Peter.  That's a great
question.  Well, of course, in
one sense my answer is that this
whole event, as you say, is
about those frontiers, and really it's
not for me to provide the answers.
I really encourage people to
dive deep into this event,
engage with all the different
activities.  We've got an
incredible lineup of amazing people,
external people and great
people from within Microsoft.  And
between them, I don't think
we'll arrive at all the answers, but
we'll certainly touch on many of
the key issues and hear some very
interesting viewpoints
on these many frontiers.
I certainly don't have all the
answers.  But I'll just offer a
couple of thoughts, maybe, on
things which I think are trends that
we're seeing right now that I
think are very exciting.  One of
them is I suppose, in a
sense, fairly obvious, and it's the
scaling.  One of the reasons why
machine learning works so much
better today is because we've
learned to scale, to scale the size
of the data sets, to scale the
size of the learning algorithms,
the models in terms of the number
of parameters.  And, of course,
we've had to scale up the
compute in order to be able to train
large models on large data sets.
And that trend looks set to
continue.  When we think about the
developments in natural language
models, for example, there's no
sense that we've reached some
sort of asymptote.  There's every
indication that as we bring larger
data sets, bigger models, more
compute to bear, we'll see
more and more improvement in
performance, more and more
of these emerging properties.  It's
really been quite remarkable.
So a real challenge for the
field is how we stay on that curve,
how we continue to see these
massive increases in the performance
of machine learning hardware.
And, of course, that's something
that's of great interest to
Microsoft, and we're doing a lot of
work in that space at the
moment.  So I think that's one very
important trend, and I
think that's set to continue.
The other one really relates to
the fact that machine learning is
really about data.  Data sits at
the heart of machine learning.
And as we seek to bring the power
of machine learning to more and
more domains, we talked a lot
about healthcare as a great example,
and many other domains, where
the data that's being collected, the
data that's available, the data
we could potentially gather in the
future clearly has a lot of
potential to bring great benefit to
society.  But much of that
data is also very sensitive, very
personal, in the case of healthcare
data, is a great example.  But
data, generally, we need to be
very careful about data both from a
privacy point of view and
from a security point of view.
And this is I think a very exciting
and very important frontier.
It's one where Microsoft in many
ways has taken a lead in terms of
the ability to provide confidentiality
for machine learning within
the cloud.  We're the first cloud
provider to deploy technology
that allows data to remain
encrypted, not only when it's being
transmitted over the Internet
and stored, but right up to the
point where it's actually
inside the processor.
So the decryption only happens
in the processor, and it means even
somebody in the data center,
with physical access to the chip,
would only see encrypted data
going on and off the chip.  They
still wouldn't have access to
the data.  So very high levels of
security and privacy.
And that allows some really
interesting scenarios.  So we know
that machine learning not only
benefits from more data, but it
benefits from diverse data.
Sometimes you can bring several data
sets together, and you can get
more than the sum of the parts.
And the question is how can,
let's say, different organizations,
different people, how can they
bring their data together and pool
that data for machine learning
without simply having to give other
people or other organizations
direct access to the data.
Well, there's confidential
machine learning that opens up that
possibility, the idea that data
can be brought together, it's only
decrypted on the chip, it's
used within the chip to train a
machine learning model, and
that machine learning model is then
made available or its predictions
are made available to the
providers of the data.  It was
trained on the pooled data, so it's
more effective, more capable,
and yet at no stage did any of those
entities have access to the data
from the other entities.  And in
fact, at no stage did Microsoft
have access to any of the data.
So I think that intersection of
privacy with the machine learning
is going to be a very important
area in the years to come.  But
those are just examples of
frontiers in machine learning, and
we'll see many more important
frontiers over the next few days.
Which actually I think is a good
moment to play our final taster
video.  This one comes from
Stavros Volos.  He's a senior
researcher in Microsoft Research,
and he's going to be leading a
session on accelerating
machine learning with confidential
computing.  So here's Stavros.
STAVROS VOLOS:  So hi, everyone.
I'm Stavros Volos, a researcher
in the Microsoft Research
Cambridge, and I'm chairing the session
on accelerated machine learning
with facial computing.  So in this
session, we have an exciting
agenda with topics across the whole
confidential computing stack.
Okay, so let's talk more about
confidential machine learning.  So
today's clouds are spending an
increasing amount of compute cycles
on machine learning tasks.  One
key concern of these systems is
the privacy of the data being
analyzed as well as the results of
such analysis.  So these
concerns have raised the need for
confidential ML platforms.
Now, the goal of this system
is to provide strong security and
privacy concerns to cloud tenants.
A key block in these systems
is confidential computing
hardware which is trusted by cloud
tenants.  In turn, the hardware
provides the assurance to remote
entities that their data code and
models can remain protected from
privileged attackers and cloud
administrators throughout the
computation of the job.
Now, in this session, our speakers
will present applications and
advances in confidential AI
problems.  First we'll find out how
entities can securely collaborate
and then train accurate models
using sensitive data while
relying on confidential computing
hardware.  Then we'll learn how
cloud accelerator systems can be
designed to provide strong
security guarantees, overcoming their
performance limitations of
CPU-based confidential computing.  So I
hope you enjoy this session and
would like to hear your questions
at the end of
this talk.  Thanks.
PETER LEE:  Well, you know,
again, it just seems so interesting.
And, you know, as you were
saying earlier, Chris, before playing
that video, there's so much
happening just in terms of scale.
And, in fact, I think even
specialists have a hard time
appreciating just the scale
that we're operating at right now.
It's just stunning.  And, you
know, by the way, this also brings
us back to the beginning of
our conversation about why bring
research and
incubations together.
CHRISTOPHER BISHOP:  Yeah, I
think it does seem extremely natural
in this new world to bring
research and incubation so close
together, as you say.  I think
it's a very natural thing to do and
very exciting.  And because of
this ubiquity of machine learning,
it means the machine learning
is not only showing up in lots of
different places, but it's really
impacting society in very new
ways that we haven't seen
before.  In fact, Microsoft researcher
Mary Gray, she's a senior
principal researcher in Microsoft
Research, and she's going to
be leading a panel discussion
tomorrow, in fact, which will be
talking about how we can push on
the machine learning frontiers
in ways that deliver better social
equity, which is a topic that,
of course, is very much on our
minds these days.  So
I'm very excited about that.
So, Peter, it looks like we've
pretty much used up our time.  I
think for the last 10 minutes or
so we'd like to just open this up
to questions.  Now, my colleague
Rachel Howard has been monitoring
the feed.  So, Rachel, do
we have any questions?
RACHEL HOWARD:  Thanks,
Chris.  Thanks, Peter.  So, Chris, I think
I'll come to you first as we've
had a few questions related to
data privacy.  Perhaps I'll read a
couple, and you can cover them
both at the same time.  So we
have:  Since healthcare data is
sensitive and private, there is
a trade-off between maintaining
privacy while explaining any
high-level insight on how to approach
this.  And the other is:  Is
there any research on secure
multi-party computation
to maintain data privacy?
CHRISTOPHER BISHOP:  Thanks,
that's a great question.  I think
it's actually true in general
there is this tension between the
desire to create value-added data
and the need to protect data and
preserve privacy.  And there
isn't a sort of one-size-fits-all
answer to this, but some research
that we're doing in Microsoft
Research really aims to get to
the heart of this and address that
trade-off.  And you heard me
talk a little bit about it already
there, the idea that, of course,
it's very easy to protect data
when it's at rest or when it's
being transferred from one place to
another because it's encrypted,
but to get value add to the data,
you need to decrypt it.
So the idea of this secure
computation is to decrypt the data only
on chip, and the goal really is
to be at the stage where even if
somebody were in the data center
and even if they had access to
all the passwords, and even if
they had clips and could measure
the signals going in and out
of the pins on the chip, they still
wouldn't be able to see the
data, they would just see what
appeared to be random
noise, just encrypted data.
So that's the goal.  And that's
very powerful in general.  It's
particularly powerful in machine
learning when you want to do this
aggregation of data from
different sources, different people,
different providers and train up
models on aggregated data because
those models are often better
than models just trained on single
sources of data.
But there are still open research
questions.  So we've made a lot
of advance.  The technology
that we developed in Microsoft
Research is now deployed in
Azure.  Microsoft was the world's
first company to have this
technology deployed live in its cloud.
But there are still open
questions.  There are interesting
questions about leakage of
information via trained models.  So
there's a lot of research still
to be done in this space.  Peter,
I don't if you want
to add anything.
PETER LEE:  Yeah, I think,
you know, all of the leadership and
leading research now that is
deployed and is a standard part of
Microsoft Azure I think has been
tremendous.  But as you were
saying, there's still a lot more
that has to be done.  And, you
know, there's also a range of
things.  If you have healthcare data
that's in the FHIR, standard FHIR
format, we have anonymization
APIs that meet the legal
standards for anonymizing.
But it's not really the same as
really kind of locking down and
protecting people's identities.
And so the need for more
research, both in the
silicon architectures, in core,
cryptographic algorithms and
protocols, all the way to AI I think
is still a major focus, especially
for you in your lab, Chris.
CHRISTOPHER BISHOP:  Yeah,
absolutely.  I saw there's also a
question there about homomorphic
encryption, which is also very
interesting.  It's sort of a -- I
view it as a complementary
technique.  It's one that
produces very, very high levels of
security and privacy.  But it
perhaps lacks the generality and the
scaling that confidential computing
offers.  So I think right now
confidential computing looks
like a very practical technology that
we're already using real scenarios,
but there's a lot more work to
be done in this space.
PETER LEE:  There's another
aspect that I think about with
research, because even if we
don't necessarily feel we can have a
general homomorphic encryption
deployed, let's say, in product
form today, it dramatically
influences our thinking.  It makes us
think a little bit differently about
the whole problem and how we
might approach it.  And so it
gives us sort of more room to be
creative, then, with
how we go about this.
CHRISTOPHER BISHOP:
Yeah, I also think actually it's very
beautiful.  It's surprising that
you can do it, at least to me,
the fact that you can do more
homomorphic encryption and compute
data without decrypting it.
It's sort of magical.  So it's kind
of inspiring as well.
PETER LEE:  Yep.
CHRISTOPHER BISHOP:  Thanks,
Rachel, do we have anything else?
RACHEL HOWARD:  We do.  So,
Peter, can you perhaps share a little
bit more about Microsoft's
approach to fairness in AI?
PETER LEE:  Well, there again,
there's a range of aspects.  And in
the chat, I posted a paper,
that it's on my reading list, about
biases, analysis of biases in
NLP-trained models.  But stepping
back for a moment, of course,
the technologist in all of us is
looking for tools.  And in tools,
things -- frameworks like SHAP
and LIME, where we have very
intensive research and development
going on, give us an ability to
create models that then can be
analyzed for different
kinds of biases.
So if you wanted to ask a
question, is this model, let's say,
biased with respect to age, you
know, ageism or race or gender,
these SHAP and LIME and similar
kinds of frameworks give you an
ability to ask those questions to
models, do an analysis and get
some insights into
whether that's true or not.
And that has actually already
started to have an impact, for
example, as Microsoft works,
for example, with the financial
services industry, you know,
where ageism, for example, in the
denial of a credit application
is actually illegal.  And that has
then created a great deal of
interest across Microsoft Research in
whether applications of
frameworks like that and tools like this
could be useful, say, in healthcare
settings.  And so that's one
aspect.
But then popping up a level,
there's just also generally the
policy, how should we behave
and think and conduct our research
and deploy technology in a
responsible way, in a way that really
gives us a chance that these
technologies, as they develop, are
used in the most ethical way
with the most positive societal
implications.
And so we try to work in that
span of just actual concrete tools
that researchers and developers
can use all the way to thinking
about the influence of these
technologies on our policy thinking.
Chris, I know you've also been
really pushing a lot of this as
well in your own
direct research.
CHRISTOPHER BISHOP:  Yeah,
absolutely.  And it's interesting, I
mean, healthcare is one of
the fastest growing areas in the
Cambridge lab, for example, but
in the Redmond lab and others as
well.  And it's a great domain
that highlights, I think really
brings all of these issues into
very sharp focus, and not least
because the potential upside
when we can address them is so
enormous, the opportunity to
improve lives, improve healthcare
outcomes and so on.  So,
although a lot of these issues are very
broad and very generic, I
personally feel particularly passionate
about the healthcare space is a
great domain to stimulate research
simply because it's so motivating
at least.  That's something I
find personally.
So I noticed time is getting
on a little bit.  I know we haven't
addressed all of the questions,
but perhaps in the interest of
time and with the next session,
we should perhaps draw this to a
close.  First let me say a big
thank you to Peter.  It's been
enormous fun chatting with you.
We could obviously chat all day.
It's been quite a lot of fun.  A
big thank you to the team for
pulling this together, for Rachel
for triaging, asking questions.
So we're very shortly to the
next session.  This is going to be
led by Susan Dumais.  She's a
world-leading researcher and also a
Technical Fellow at Microsoft,
and she is going to be leading a
panel called Machine Learning
Conversations.  So I do hope you can
join us for that.  And meanwhile
just a big thank you again to
Peter, and I hope you all have a
very stimulating and informative
event over the next few days.
PETER LEE:  Thanks, Chris, and
thanks, everyone, for being here.
