CS50 Live first looks today at GitLab.
GitLab is a popular source code
hosting site, much like GitHub.com,
that developers can use in order
to store their code centrally,
in order to version control
it, share multiple copies,
as well as share it with other users.
Unfortunately, GitLab ran into
a bit of an issue very recently.
The whole incident started
when they saw this.
They support a feature
known as Snippets,
whereby users can
create snippets of code,
much like GitHub Gist, whereby users
can upload small snippets of code
to share them with other people.
Unfortunately, having some
1.5 million snippets of code
created over the course
of just a few days?
Not normal.
In fact, this seemed to be the
result of some spamming behavior
by some adversarial folks online.
Moreover, GitLab also notice that one or
more spammers seemed to be using GitLab
inappropriately, as a content
delivery network, or CDN,
whereby they were serving up
files in ways that they shouldn't.
Now unfortunately,
these kinds of attacks
resulted in a bit of a ripple
effect on their back-end databases.
Particularly, GitLab
posted the following.
"We are experiencing issues
with our production database
and are working to recover."
Now unfortunately, just
minutes later did they
post, "We accidentally
deleted production data
and might have to restore from backup."
Now what exactly happened?
Well, it's quite common for databases
to be replicated from one to another
so that you have a
primary and a secondary,
the latter of which is a backup
of the former in real time.
As part of the diagnosis
challenge for figuring out
why the databases were slowing
down in terms of this replication,
one of GitLab's system
administrators very deliberately
executed a command quite like this.
Now what is this command?
Well at the front of
this command is "sudo,"
which says execute the following
command with administrative, or root,
privileges.
What is the command to be executed?
Well, rm -rf apparently.
And rm, you might know, is to remove
files or folders from a system.
-r though means recursively.
Delete the following thing recursively,
so that any directories inside of that
also get deleted.
And unfortunately, the
"f" -rf means forcibly,
which means don't even prompt
the human to confirm or deny
that he or she wants to do this.
Now the system administrator meant
to execute this command deliberately
on their secondary database,
db2.cluster.gitlab.com,
so that they could resume then
the replication from their primary
to their secondary database.
Unfortunately, it appears
to have been late at night,
and this was a stressful
situation, and darn it
if this command weren't executed
on db1.cluster.gitlab.com,
the actual primary database.
Now no big deal, surely we have
backups all over the place.
So we can just restore from
backup, and our customers
will be perfectly
happy and on their way.
Unfortunately, out of five backup
or replication techniques deployed,
GitLab reported that, "None are working
reliably or set up in the first place."
Indeed, if you'd like to read their
whole post-mortem in which they
discussed exactly what went wrong, and
how, you can check out this URL here.
But the moral of the
story, for our purposes,
is please, please beware
the rm -rf, especially
if you're not just deleting
some directory of your own,
potentially your customers as well.
