Hannah: Welcome back to the
6th episode of DeepMind, the podcast.
My name is Hannah Fry,
I am a mathematician
who’s worked with data and algorithms
for the last decade or so.
And I spent the last year
at DeepMind -
an organisation that is trying
to solve intelligence
and then use it to solve
some of society’s problems.
There are an awful lot of people
working in the field
of artificial intelligence,
moving forward our
understanding of the whole area
and for many of them it is
a terrifically exciting place to be.
We’re breaking new frontiers of problem
solving and seeing great leaps ahead.
But before any of that
hits the outside world,
the first inklings of new breakthroughs
here at DeepMind
come in the form
of regular poster sessions.
Female voice: Our goal here was to um
have a model
that can carry out
the following simple tasks
where I’m going to give you
a number and a secret symbol
and what the sum
between those two are,
and you have to infer from that
what the value of the symbol is,
but it requires our agents
to have some properties
that we think are desirable
like learning to learn,
like having a memory and processing
those memories ourselves.
Male voice: We’re studying
analogical reasoning -
analogical reasoning is very important
because it’s a key to scientific
discovery and also human reasoning -
the main question we ask
is how can we design neural networks
that are able to do
analogical reasoning.
Female voice: My poster is about
verification of neural networks.
In this day and age,
when we deploy neural networks
into the real world applications,
we want to make sure that
these neural networks are safe,
for example if you have
an image classifier,
we don’t ever want to predict a cat
to be like a car or something like that.
Female voice: I always say DeepMind
is a bit like academia on steroids,
like it is still academia,
but we have a lot of compute,
a lot of great people
clustered together,
a lot of help to manage ourselves,
so yeah.
Hannah: while there is obvious
excitement about AI research,
this new era of artificial intelligence
also comes with concerns.
There is an ease about the way it might
be implemented, used and abused.
For the rest of this episode,
we are looking at the more
human side of technology,
and the fight to find a future of AI
that works for everyone.
In 2017, DeepMind set up dedicated teams
working on how AI impacts
ethics and society.
With the aim of making sure that
the algorithms designed in this building
are a positive force for good.
But hang on -
I know what you are thinking -
surely algorithms aren’t ever good
or bad in and of themselves,
it’s how they’re used that matters.
After all, GPS was invented
to launch nuclear missiles,
and now helps to deliver pizzas.
And speakers playing pop music on repeat
have been deployed as a torture device.
Isn’t the technology itself
just neutral?
Verity: Good question!
And um I think something
a lot of people say and believe
and I can see why they say that.
Hannah: This is Verity Harding,
co-lead of DeepMind Ethics and Society.
Verity: I think there’s a famous saying
about as long as there’s been fire,
there’s been arson.
You can use something
that’s for good you can use it for bad.
But It think actually is
we’re developing increasingly
sophisticated technologies
that have real impact on people’s lives.
It’s not really
an acceptable thing to say.
You can’t be building something
that’s going to have
this kind of monumental impact -
or potentially transformative impact
and not care about
how it’s going to be used.
Hannah: Is that part
of the concern then?
That technology that might have been
built for one purpose
ends up being used
in a different way?
Verity: I think definitely
that’s some of it.
I think definitely that’s some of it.
Because you could force a situation
where you’re building
facial recognition tool
because you want to allow somebody
to quickly find pictures
of their husband or wife or mom or dad
and that’s a great thing
but that facial recognition technology,
once developed,
could of course be used
to target political dissidents
and pick them out of a crowd
and you know, so I think,
that’s definitely
one of the concerns
that you might create something for one
purpose and it be used for another.
Hannah: On the topic of facial
recognition, Brad Smith,
the President of Microsoft,
recently refused a request
by a US police department
to install their algorithm
in cop cars and body cameras,
and he's publicly called for more
careful thought and societal dialogue
about potentially regulating
the use of the technology.
And here at DeepMind more generally
there is a strong sense
that the people behind the science
have a duty to investigate the wider
and perhaps less predictable
impacts of their work.
Verity: I don’t think it’s okay
to build something -
whether that be a product or a service
and put it out there
and and just hope
that you make the world a better place.
It think it’s important that you
are deliberate and intentional
about why you’re building this,
who are you building it for,
what are you hoping to do?
What is your intention
with this technology?
And if you start from that premise
then I think you are more likely
to get to a better outcome
where you do the good that you hoped
you were going to do.
Hannah: The problem is that
without these steps,
it’s very easy for unintentional
consequences to creep up on you.
You only need to look at the new stories
about social media
from the past few years
to see just how much algorithms
have changed our society
in unexpected ways.
Lila Ibrahim is DeepMind COO
and has over 20 years’ experience
working in the tech sector.
She has seen first hand
how hard a booming industry
has found it to keep up
with being responsible.
Lila: In 2006 I went into the middle
of the Amazon
and we built a computer lab
and health care,
so we put in internet
and computers, etc.
and we knew we had a responsibility
not to just leave it there
but to train folks to take care of it,
to think about the sustainability.
But I think that’s kind of
where things tend to end.
Ethics means something
very different now,
and responsibility means
something very different now
because technology is
in everybody’s hands.
It’s no longer limited to a few people
for a specific application.
Um it’s a lot easier
to get into the tech sector
and to make technology
that can have value to people,
and at the same time
that comes a lot of responsibility
that I don't think in general the
tech sector has taken into account.
Hannah: But the last few years
have shown
how hugely transformative
and disruptive AI can be
and brought sharply into focus
the very possible negative
outcomes of ill-thought
through technology.
But as Verity told me,
the tide is slowly beginning to turn.
Much of the drive
for a conversation about Ethics
is coming from within
the technology community itself.
Verity: 3 years ago in 2016
some of the scientists from different
ah labs at different companies
met at a conference
and were talking about how excited
they were about the potential for AI
to do a lot of good,
ah but acknowledging that -
a technology that’s so powerful
that it has the potential
to be transformative in a very good way
must also have the potential
to be very transformative in in
in not so good a way,
and so they came together to say well
what can we do about it?
Hannah: And so the partnership on AI
was born.
It includes members
from Amnesty International,
Electronic Frontier Foundation,
the BBC and Princeton University
amongst many, many others.
And together they are hoping
to come up with best practices in AI
making sure that society stays firmly
at the forefront of engineers’ minds.
Verity: So the partnership of AI
interestingly was founded
by the biggest tech companies
so it was founded by DeepMind
but also Google,
Facebook, Amazon, IBM and Apple.
One thing that’s really interesting
about the partnership and AI
is that the board membership
is made up of independent board members
and representatives
of the company
and so it’s creating a space
where those different groups
aren’t siloed from each other,
having debates in different rooms
and not listening but somewhere
where honest people
with the best of intentions
can come together
and challenge each other
and scrutinise each other
and hold each other accountable
but also have frank,
open honest debate
about issues where reasonable
people can disagree.
I really believe that the outcome
of that will be better decision
making both in companies
but elsewhere as well.
Hannah: Does it sometimes get
quite heated in those conversations?
Verity: You know my experience of it
is that it doesn’t get heated
but it’s passionate,
so people aren’t angry with each other
and and there’s not aggressive argument,
but people are very honest,
and very open and very challenging.
But that’s been received
really well in all cases.
Hannah: How do you protect
against rogue companies
just who are not part of these groups,
just doing whatever they want.
Verity: If enough companies
and enough groups sign up to something
and it becomes the norm,
it’s then really obvious
when people aren’t doing it.
And I do think people are kind
of being called out for that -
it will no longer be tenable
to not operate
in the way
that everybody else is operating.
Hannah: It’s not just
theoretical concerns about runaway
applications of AI
that’s prompting these conversations,
but real examples of algorithms
that have already been let loose
on the world with real question marks
about whether their benefits
outweigh their harm.
A notorious example is the use of AI
in the criminal justice system -
now you may have heard
of these algorithms already.
When a defendant appears in court,
the AI can assess a defendant’s chances
of going on to commit
another crime in future,
and that risk score is then used
by a judge to help decide
whether the defendant
should be awarded bail,
and in some cases, how long
someone’s sentence should be.
There is good justification
for something like this
because there is
an enormous amount of luck
involved in the human
judicial system.
Studies have shown that if you take
the same case to a different judge,
you will often get
a different response.
If you take the same case
to the same judge on a different day,
you’ll often get
a different response.
Judges don't like giving the same
response too many times in a row,
and so if a series of successful cases
of bail hearings have gone before you,
your chances
of being successful fall
and there is even evidence to suggest
that judges tend to be a lot stricter
in towns where the local
sports team has lost recently.
Using AI to help make these decisions
can help to eliminate
a lot of that inconsistency,
but you have to tread pretty carefully.
Verity: if you without thought
and care
and due attention to the history
of racial prejudice
in the criminal justice system,
build something that claims to be able
to predict somebody’s likelihood
of reform and rehabilitation,
and reoffending, then it is likely
at least in my view
that that’s going to fail.
If you build something with the
intention of addressing those biases,
and you work to include
the community in some way,
there could then potentially be
a beneficial outcome maybe,
but I haven’t seen it yet.
Hannah: And by fail you’re really
talking about
treating black defendants
differently to white defendants.
Verity: Absolutely! And once you tend
to look at the algorithms
and the data that they’ve been,
they’ve been built on,
um oftentimes you can see
where they were built on data
that was already biased of course
this was the outcome.
Hannah: The issue came
to public attention in 2016
after a group of US investigative
journalists from Pro Public published
a damning report of one particular
company’s criminal risk scores.
Their study showed
that the algorithm was twice
as likely to wrongly categorise
black defendants
as being likely to re-offend
than white defendants.
Now I should just point out that
DeepMind does not build these systems,
but the whole industry alongside
the partnership of AI
has been part of the conversation
about how to address them.
One of those people is William Issac,
a social scientist at DeepMind.
He says that the 2016 ProPublica
investigation
made people realise
that switching over to algorithms
doesn’t make decisions
any more objective.
William: Even with AI and ML tools
you are getting into
the social environment
where you actually have
the same norms,
the same kind of like systematic
biases, they’re still all present.
So it’s really hard to say
that somehow this will replace
all of the kind of subjective,
preconceived notions
about certain groups,
or historical biases
against them,
and that you can start all over again
and so I think that was the wakeup call
was that it’s not as objective
as it seems and that as a result,
we still have to grapple
with those questions.
Hannah: The problem is that the data
which gives the algorithm
predictive abilities
are questions like how many times
were you arrested as a juvenile,
but if you are say a young,
black man in America,
it doesn’t matter
how law-abiding you are,
the chances are that you will have had
many more negative interactions
with the police
than someone exactly like you,
who happens to be white.
And if you’re using that data to dictate
who deserves to be given bail or not,
then you are in serious
risk of perpetuating
societal imbalances going forwards.
This is Silvia Ciappia,
a staff research scientist at DeepMind.
Silvia: Researchers don’t
fully understand what this fairness
is about they also look like messier
in the sense that it involves,
it is not purely
technical problem
and it’s very difficult to understand
what how to define fairness,
and it’s difficult to separate
the technical part
from the ethical one.
Hannah: This is an important point,
because defining exactly what you mean
by “fair” is surprisingly tricky.
Of course you’d want
an algorithm
that makes equally accurate predictions
for black and white defendants,
the algorithm should also
be equally good
at picking out the defendants
who are likely to reoffend
whatever racial group they belong to.
And as Pro Publica
pointed out,
the algorithm should make
the same kind of mistakes
at the same rate
for everyone regardless of race.
Ethically you’d want
all of those things to be true,
but technically that’s not
always going to be possible.
If you’re data set has bias in it,
there are some kinds of fairness
that are mathematically
incompatible with others.
And even if you could guarantee
all of these things,
there are still a number
of ethical issues to contend with.
How do you measure fairness,
who’s excluded from your definition,
how do you make
those decisions transparent,
and ultimately how
do people contest
the decisions made
by those algorithms?
See, I told you
it was tricky.
Coming at this from two
very different perspectives,
William and Silvia started
looking into the bigger issue
of fairness in algorithms.
William: Even though we had
kind of different frameworks,
me as a social scientist and Silvia
as a machine learning researcher,
the actual overlap between
how we would approach this
and basically the assumptions
that are embedded within it
were remarkably similar
and actually part of what we’re saying
is like oh look at these papers
in social science
that are kind of making
this same point,
they just hadn’t actually had a way
to actually communicate that formally.
Hannah: You’re listening to a podcast
from the people at DeepMind.
In April 2019, William and Silvia
co-published a paper on fairness in AI
entitled a causal Bayesian
network’s viewpoint on fairness.
In it they show that no matter
how fair algorithms might be,
if the data they’re
learning from is biased,
we still can’t
trust their results.
Silvia: I don’t think it’s possible
to find technical solutions
that are completely
satisfactory.
At some point we need
to take decisions
whether the kind of fairness
is acceptable or not,
but we can advance a lot
and that’s why we need
more researchers involved
- and not just machine
learning researchers
but researchers
from different communities
to be raising awareness
about this this problem, find solutions,
but as we would never be able to find
completely satisfactory solutions
from a technical viewpoint,
and at that point
we need to take decisions
- is it important to talk
about these such that, that we are -
something that is missing
at the moment.
William: I do think this is
fundamentally
like a societal,
ethical question and challenge.
And it will require
lots of stakeholders to address.
If you have let’s say a data set
of facial recognition tool
that’s designed to find
missing children -
what threshold do you say
as a society well
you say okay this is acceptable,
if we maybe are less successful
at identifying children
with darker faces,
what threshold do we say
that’s acceptable,
because that’s not
a technical question,
that’s a social and political question,
a normative question.
Even if you do have a classifier
or a facial recognition software
that’s fair, the application
of it may be in unfair ways.
And so that might present
a second question
that is separate
from the actual kind of like
if you decide on a threshold
that if you’re just using it
in a neighbourhood
in a predominantly one group
or one ethnicity,
that presents a whole
other set of challenges
for whether or not that’s an ethical use
of a particular technology.
Hannah: You can’t assess whether these
algorithms are good or bad in isolation,
they don’t exist on their own.
You have to place them in the context
of the worlds that they are being used,
like the criminal justice system
or in healthcare.
Here’s Verity
Harding again.
Verity: This is what I mean by it being
a kind of much bigger discussion
that that potentially the use
of algorithms is highlighting.
My fear is that people kind of get
a checkmark that says:
we tested
and this algorithm isn’t biased
and therefore you should feel free
to use it.
And that to me isn’t going far enough.
I think there needs to be
a further discussion then about um
but is this making those decisions that
were already bad worse, or more quickly,
and therefore more of them
and you know that that kind of thing.
Hannah: But things are changing.
Here’s William on what has happened
since that Pro Publica story broke.
William: They’re going back and
reconsidering
what measures they collect
rather than going back
and trying to create
more robust data sets,
thinking about who is collecting
the actual data itself.
Will it be ever perfect?
Will we have bias-free,
purely pure data,
no I don’t think that’s--I
don’t think that’s ever going to happen.
But I do think that people
will be skeptical
when people ask about
what data sets you use
and they don’t get
a satisfactory answer,
but I do think people will ask -
is this data set representative?
Does it have balance
across different groups?
So people will start asking questions
and interrogating data
sets and models more aggressively
and I think that will lead
to better outcomes.
Hannah: And crucially,
more people are now being included
as part of the conversation.
William: In the aftermath
of some of my work
and on many others
on predictive policing,
many cities in California actually
started implementing citizen boards.
So when police departments wanted
to acquire a new police technology,
that included uses of machine learning
or artificial intelligence,
that they had to go
in front of a citizen board
and actually have
the local community evaluate
the tool for different metrics
including fairness and bias.
Hannah: Getting different voices
involved in the conversation
is essential to making sure
that we build a future
that belongs to all of us
because what seems obvious
to just one person
just wouldn’t occur
to another.
Your perspective is hard coded
into the work that you create.
There are clear examples of this
everywhere outside of AI -
able-bodied people
designing buildings
that disabled people can’t use
or new types of plasters
that only work if your skin
is one particular colour -
presumably the same
as the designer’s.
And the algorithms that we’ve created,
they’re really highlighting this issue.
Like the ones used
to automatically screen
CVs and predict which candidates
will fit best in a company.
Here’s Verity Harding again.
Verity: If it’s based on historically
discriminatory hiring decisions
by either intentionally
or unintentionally by us humans,
um then it’s going to kind of
recreate those patterns.
Hannah: Like if you’ve got a company
where white men have succeeded
Verity: Yes
Hannah: and you’re looking
for candidates who will succeed,
it’s going to pick out
white male CVs.
Verity: Yes, exactly.
And if the people building the
technology are white males as well,
then the likelihood of paying attention
to that potential bias
and being aware of it,
I mean we all have our blind spots,
then then the likelihood increases.
Hannah: We’ve seen driverless cars
that don’t spot pedestrians
with darker skin tones.
Tumour screening algorithms
that aren’t as effective for patients
with ethnicities
other than white European,
and lots and lots and lots
of issues around gender.
All of this is
kind of inevitable
unless you have a range of different
viewpoints in your design process.
Verity: The most important thing
in my point of view for ensuring
that these things are um
if not biased,
but that you are being intentional
about what you’re building,
and aware of the potential bias
is that your team is a diverse team
is that you have a broad
set of voices involved,
and it’s actually much simpler to do
that than it’s suggested.
Hannah: And the issue
of gender diversity
has been a particular
focus of late.
Verity: I think there’s
plenty of young women and girls
who are really excited
by science and stem subjects
and it’s an easy get out to say
that there aren’t enough women in STEM
and that’s why work
forces aren’t diverse,
but actually it’s much
more about making sure
that it’s a safe space
for women and girls to work,
that they’re not discriminated
against once they’re there.
That you’re able to not
just attack and hire them
but that you’re able
to keep them and
and make sure that it’s a place where
they feel comfortable working.
And so I think it’s much
more important that we look
at how women are treated in science
than just dismiss it
as something that girls aren’t
interested in at a young age.
Hannah: Lila Ibrahim, DeepMind CCO
is very conscious that diversity
is still a problem
in the tech sector as a whole.
Lila: Talk about things
that keep me up at night.
Right, so, here I am,
a professional of ah 25+ years
with an engineering background.
A mom also raising 9 year
old twin daughters.
I would have hoped by now
we would have solved the problem
and yet we haven’t
- we’re like at the same,
the numbers are flat.
Hannah: But there are steps
being taken to address it.
Lila: There’s the short-term stuff
you can do
which are things like you diversify
your candidate pools,
you, if you’re doing
university recruiting,
you look at a broader
range of universities
and ones that have um that have
a broader student representation
and have support structures
often in place
to help the students through
their academic and communities.
Um you look at job descriptions
and ensure that you don’t have
unconscious bias
reflected in your
job descriptions.
You - so once you’re
in the recruiting pipeline,
then you need to make sure candidates
have the right experience.
We are being very deliberate
about how we invest back in education.
AI is something that will
change future generations -
so how do we make this a field
that is more accessible -
so for example, um whether it’s funding
diversity scholars at universities,
or funding AI chairs in universities
to try and increase the pipeline,
and I think that helps fuel
some of the academic aspects
as well as support
are like long-term recruiting.
Hannah: This isn’t just tokenism
that we’re talking about here,
this is about making
better technology.
Lila: Diversity and diverse perspectives
will create a drive faster and safer
and with just a better a better result
because one of the things I worry about
is how do we avoid
our own internal bias?
A lot of the work
around deep reinforcement
learning started from specific pockets
and many people grew up in those labs
or those universities and you know
they bought their former colleagues
and so we have our pretty
strong network of people
who have known each other
for a long time which is fantastic
and they can really advance
certain aspects of our of our work
and yet there are other areas that are
emerging um how do you teach curiosity,
how do you um how do we ensure
that we minimise bias
and the code that we’re writing?
Who’s to say ah what intelligence is
and isn’t unless you have
a better representation from society?
And that’s just
on the research side.
On the operations side too you think
about things like okay,
think about public policy,
like if you’re asking governments
to think about how they’re going
to treat artificial intelligence
then you want people
that are representative
of the of the constituents
of the population.
If we want to ah
be focused on education,
making sure that we’re not just
focused on the specific schools
but our broader range so we’re bringing
more people into the space.
I just think it’s going to be imperative
for us to truly solve intelligence
that we’re just going to need
to have more diversity.
Hannah: So is part of the well
solutions,
probably a bit grand a word to suggest
but it’s part of the way forward
making ethics a kind of keystone
at every stage of the process rather
than having it as an afterthought.
Lila: Oh, absolutely. It we have to
be thinking about our responsibility
for the technology we develop
and to candidly to society as a whole -
every step along the way.
And I think that there’s something
quite special
about being headquarters
out of London versus being
based out of the Silicon Valley -
I love Silicon Valley,
it’s where my career
has really developed
and yet you’re surrounded
by technologists,
you know from the billboard signs
to the marketing and promotion.
Here in London it’s so multicultural
and I feel like
it’s part of your daily life.
You need to be thinking about
the work that you’re doing
and how it’s going to impact
all the people around you.
Hannah: But of course we can’t just
leave the solutions
solely in the hands of the people
who are designing these things.
It’s our future, too.
The public and government
should also have a hand in this.
Verity: My impression is that people
want to understand what they’re using
and want to understand
what makes it work and how it works.
But they more importantly
want their representatives
and the people tasked
with keeping them safe
and secure to understand it too
and I think that’s where we’ve seen
a bit of a breakdown in recent times.
Hannah: If you want to know more
about ethics, diversity and fairness,
then head over to the show notes
where you can also explore
the world of AI research
beyond DeepMind.
And we’d welcome your feedback
and questions
on any aspects
of artificial intelligence
that we’re covering in this series.
So if you want to join in the discussion
or point us to stories or resources
that you think other listeners would
find helpful, then please let us know.
You can message us on Twitter,
or you can email us -
podcast@deepmind.com
