DALE MARKOWITZ: Hi, I'm Dale.
This is Making With ML.
And today, I want to
talk about something
you've probably
experienced if you've ever
been on the internet.
I'm talking about user-generated
content that violates
a platform's terms of service.
[MUSIC PLAYING]
Almost all content
platforms, whether they're
YouTube, or Twitter, or OKCupid,
or "The New York Times",
have some strategy for
dealing with bad user input.
And this is called their
moderation strategy.
Now, you can imagine
what's OK to post on Reddit
or a dating app might not
fly on "The New York Times",
so these platforms have to
consider what is allowed
under their terms of service.
These platform
maintainers also have
to consider the implementation
of their moderation strategy.
Let me give you some options.
Option number one-- you could
hire a team of human moderators
to go through every piece
of user-submitted content
before it's posted to OK it.
This is called pre-moderation.
And as you can imagine, it's
really good at making sure
that nothing bad slips
through the cracks,
because human
beings are actually
looking at what's being posted.
But it also doesn't
scale very well,
and it takes a large
human moderation
team that scales with the
size of your user input.
So you'd only use it if it's
extremely, extremely important
that no bad bits get
through the cracks.
Option number two-- you
could ask your users
to flag bad content for you and
then send this flagged content
to a human moderation
team for later review,
maybe in a sort
of priority queue
where they figure out what's
important to review first.
This is called post-moderation,
because the content
is moderated after it's
already been posted.
For example, when you're served
a YouTube video or a Google ad
and you have the option
to report that content,
you're seeing a
post-moderation strategy.
The advantage to
doing this is that it
allows your human
moderation team to scale.
But the disadvantage,
of course, is
that then your users have to
actually see this potentially
offensive content.
And in some instances,
this would be unacceptable.
Both of these approaches
have major drawbacks.
You're either having your human
moderation team go through
every single piece of
content before it's posted,
or you're potentially subjecting
your users to content that
should have been
flagged but that wasn't.
And neither of these approaches
work for real-time moderation,
like for example, if we're
doing a video stream or chat
messages, which brings us
to option number three--
AI-powered moderation, which
is what we're building today.
The idea is to use
machine learning
to analyze content like, text,
speech, images, video, and flag
the content the instant
that it's created.
You could then send that content
to a team of human moderators
so that your users
never have to see it,
but your moderation
team can scale better.
Now you might
wonder, why not use
AI to just moderate your
entire moderation solution?
So let me add a disclaimer here.
Machine learning models
are always probabilistic,
which means that they
invariably make mistakes.
And sometimes they're
biased, meaning
that they make mistakes in
particularly bad, consistent,
and potentially
embarrassing ways.
So I always recommend
using AI in conjunction
with human beings, having
this human in the loop.
For example, you might have
your algorithms put comments
into a moderation queue
for your moderators
to review, or allow
users to appeal decisions
that were made by the AI.
So knowing that,
today I'll show you
how to make a simple AI-powered
moderation bot for the chat
platform, Discord.
Let's get started.
Today, we're going
to build a bot that
sits in a Discord server
and analyzes user messages.
When the bot detects
a message is toxic,
it will flag it with
a dynamite emoji.
It'll also keep
track of insulting,
flirtatious, spammy, and
nonsensical messages and flag
them with an appropriate emoji.
It'll keep a scoreboard
of emojis or karma
for each user in the chat room.
And if a user types too
many toxic messages,
they'll get kicked
from the room.
Now, don't worry if you've
never done any machine learning
before because,
for this project,
we're going to use
the Perspective
API to do the complicated
machine learning bits for us.
It takes in strings of
text and produces scores
for different attributes,
like toxicity, insultingness,
explicitness, and
even just incoherence.
To create these models,
the Perspective team
use millions of comments
from data sources,
like the New York
Times, Wikipedia,
and many other open data sets.
If you want to
see how they built
the model and the
architecture behind it
and even learn how
to train your own,
check out the
Perspective API website.
To use Perspective,
you need to fill out
a quick Google form explaining
how you plan to use the API.
Once your project
is approved, you
should be able to
use it in your app.
Next, go ahead and download
the Making with ML GitHub repo
and navigate to the
Discord moderator folder.
Here's all the code you'll
need to run your bot.
The file,
Perspective.js shows you
how to use the Perspective
API in JavaScript.
At the top, you can see
all the different types
of attributes the
tool recognizes,
like insult, toxicity,
spam, incoherence,
flirtation, and more.
Now, let's connect
the Perspective API
to a Discord bot.
To do that, you'll need to
sign up for a Discord developer
account.
Once you have an account,
create a new application.
Then create a new
bot by clicking
on Bot in the left-hand panel.
Give your bot an avatar and
a cute name, and you're set.
The brain of our Discord
bot lives entirely
in the file Discord.js.
Here at the top, you
can see our emoji map,
which tells our bot how to react
to different user comments.
Flirtations get a kissy
face, and spam gets a fish.
Down in the Evaluate
Message function,
we call the
Perspective analyzeText
function to detect
attributes in the message.
If an attribute is found,
we react with an emoji.
And if the user has sent
too many toxic messages,
we temporarily kick
them from the channel.
The next step is to run your
Discord bot from the command
line.
Awesome.
Now, it's ready to be
added to your server.
In the Discord developer
panel under OAuth,
select the permission
your bot needs--
sending messages, adding
reactions, and kicking users.
Copy the URL above
into your browser.
And from here, you'll be able
to add your bot to a server.
Now if we go to our
Discord server, boom.
The moderator bot
is in the channel,
keeping track of everything I
type and reacting with emojis.
If you want to build this
Discord moderator by yourself,
the code is in GitHub.
And you can follow along
in my very detailed blog
post in the description.
And there you go.
Now you have your very
own AI-powered chat bot
moderator for Discord.
Now, the moderation
strategy that our bot used
is very simple.
It just kicks users
from the channel
when they say too many things
the AI thinks is toxic.
But like we discussed
earlier, that's
actually not a
very good strategy
for a real in-production
moderation,
because the model
can make mistakes.
In fact, the Perspective website
specifically says, this API
is not designed for fully
automated moderation solutions.
And they even list a
bunch of known biases
that the model has.
Just think of it this way.
AI is just one
piece of the puzzle,
and it works better when human
partners are in the loop.
That's all for now.
If you want to learn more
about moderation and AI,
make sure you check
out the blog below.
And let me know what
problems you want to see
solved with AI in
the comments below.
But remember, those
comments are moderated, too.
That's all for now.
See you next time.
