September 2018.
I was tired, I was exhausted,
I’d just got back home from something,
and I didn't have the energy to cook.
So I went to my phone,
opened up a certain food-delivery app that’s
popular in the UK, and I ordered pizza.
Now, I know: that food-delivery company’s
employment practices are questionable,
there are more ethical ways to get dinner
delivered.
But I was tired, and I was hungry.
As are a lot of their drivers.
But that was the Night of the Multiple Orders,
when a bug in that app meant that some people
around Britain ended up with
identical food orders
being delivered two or three times,
and others got nothing at all.
And I nearly got caught up in the chaos.
To explain what happened,
I need to tell you a story about two generals.
The Two Generals' Problem is a
classic of computer science,
and it goes like this: picture a valley.
In the middle of the valley is a
heavily fortified castle.
On the edges of the valley are two armies.
The generals of these armies know that the
only way they can win a battle and overwhelm
the castle is if they both attack
at the same time.
A single army isn't going to make it.
They need the combined strength from
both sides of the valley to win.
And the only way they can communicate
is by sending messengers on a
risky path through the valley.
And General A won’t know what the right
time is until everyone’s already in position.
How can those two generals coordinate
to make sure they
attack at the same time?
This is a magical computer-science-land problem,
by the way,
so reasonable suggestions like “semaphore”
or “telescopes” don’t apply.
On the surface the problem seems trivial.
General A could just send a message to
General B with a proposed time.
Say, 8 o'clock.
But the messenger has to pass
through the valley,
and if they’re spotted, they’re, um,
not going to make it to the other side to
deliver the message.
So how does General A know that General B
received the message?
The messenger might not have made it.
And if that happens,
A will attack, B won’t, and they’ll lose.
So maybe they arrange it so General B has
to send an acknowledgment back,
and General A will only attack
if that acknowledgement arrives.
But that now runs into the same problem:
how does B know that A has received
the acknowledgement?
If it doesn’t get through,
A won’t attack, B will, and they’ll lose.
So, General A could send another acknowledgement
for the acknowledgement.
But how do they know that message
has gotten through? Well,
General B could send an acknowledgment for
the acknowledgement for the acknowledgement
and so on, and so on, and so on.
This problem is unsolvable.
I know, it feels like there should be some
hacky workaround
like sending 200 messengers,
and sure,
that would probably work
in the real world.
But this is magical information-theory
computer-science land.
Under these strict rules,
there is never a guarantee,
there is no certainty,
there is no arrangement that can be made,
there is no way,
that the two generals,
the two computers sending data,
can agree that the message has definitely
been received and acknowledged.
Now, with computers you’re not usually dealing
with such high stakes.
If you are in computer science and working
on a problem that involves potential loss of life,
I really hope you aren't watching a series
called "The Basics".
Anyway. I was ordering food.
And I put my order together, I tapped ‘pay’,
I put my fingerprint on my phone’s reader.
I got the little Apple Pay progress bar,
and the little tick.
And then I got a message from the app
saying that there had been a problem,
and my order had failed to go through.
Would I like to try again?
And I was about to.
I was about to hit ‘pay’ again.
And then something, just in the back
of my head, said: hang on.
There was that little tick saying
payment had worked.
And I’m enough of a computer nerd to go
"I’m not sure I believe that failed".
So I checked the ‘order history’ page.
It took a few tries to load,
but when it finally did, there was my order.
Processing. It had gone through, but
the acknowledgement hadn’t come back.
Or, rather, something had gone wrong
on the app’s servers,
and the logic they’d written
thought it had failed when it hadn’t.
So I sat tight,
I hoped that my food would arrive,
and I figured that the engineers were probably
having a very bad day.
They really were.
Because I wasn't the only one.
People all over the UK ordering via the app
were going to the payment screen,
hitting the button and getting
"try again".
And a lot of them did.
Again, and again, and again.
They were General A,
and the app’s server was General B,
and they were part of a real-life,
complicated version of the Two Generals' problem.
Imagine all the customers as General A,
sending message after message to General B.
B received the messages,
dutifully took the money from the credit card
every time --
they attacked --
but something had happened that stopped
the confirmation message getting through.
According to the flood of angry reports on
Twitter,
sometimes the restaurant would realise the
problem and just send one order.
Sometimes the restaurant wouldn’t realise,
and three different drivers would arrive
with three identical orders.
Sometimes no food would arrive at all.
The app’s customer service line was swamped.
To be clear: this was not the sort of thing
that is one engineer’s fault.
When something goes this drastically wrong,
there have been many poor decisions made over
a long period of time.
A single human error is never a root cause.
So what else could the app team have done?
How can you solve the Two Generals' problem
in the real world?
Well, first, maybe no-one should have
been able to place two identical orders
on the same credit card,
for the same restaurant,
within a few minutes of each other.
That seems like the sort of thing there should
have been a check for?
But the real solution is an
“idempotency token”, or an “idempotency key”.
This is a unique value generated on
the app, or on the web site:
and it’s a shopping cart ID, basically,
and it’s sent along with the order.
it's not just for shopping carts, though:
the idempotency token could be attached to
instructions to delete the oldest log file,
or send a text message,
or anything that you only want to happen once.
The server stores the idempotency key to keep
track of the request.
And if another request arrives with the same
key attached,
then the server knows it’s already
dealt with that request.
So it doesn’t fulfill it again; instead it
knows that the reply didn’t get through,
so it just sends back a copy of that first
acknowledgement again.
Now, that still won’t help if none
of the messengers get through,
if the connection completely fails,
but for real-world problems,
humans will notice that.
Idempotence means that you can request the
same thing multiple times
and it’ll only ever happen once.
That’s the way to fix the
Two Generals' Problem.
I was lucky.
I placed one order,
I was charged for one order,
and one order of food arrived
half an hour later.
Next time, I’ll just cook for myself.
This series of The Basics is sponsored by
Dashlane, the password manager.
I mentioned in the previous sponsored section
that they sync all your passwords
and payment details between
all your devices
without ever
knowing those passwords.
Which sounds a bit like magic.
When you sign up to Dashlane, you choose a
Master Password.
And incidentally,
you can do that by going to
dashlane.com/tomscott for a free 30-day trial.
Anyway, when you sign up,
you pick a single Master Password,
and that is never transmitted over the internet.
Not even to Dashlane,
not to their servers, nowhere.
If you don’t know that password,
all that private data just looks
like random noise.
So: when you sign up to a new website and
Dashlane generates
a long, complicated password
for you,
it is bundled up and encrypted using your
master password, that only you know.
That encryption takes just long enough,
a few fractions of a second,
that there’s no way to
brute-force it back open.
That encrypted bundle gets sent to Dashlane:
they just see random noise with a label saying
‘please synchronise this’.
So they pass the bundle on to your other devices,
and those devices, and only those devices,
can decrypt it because, at some point,
you’re going to open up the app and type
in your Master Password.
In truth, it’s actually a little
more secure than that,
because behind-the-scenes they also generate
a different key for each device you log in to,
but that is a whole other level of security
that I have actually found it impossible to
explain in a script.
But, in the massively unlikely event that someone
did compromise Dashlane’s servers,
or bribe some employee, it wouldn’t work.
All they could do is watch those packets of
random noise get shuffled around.
All your data stays on your own devices.
Which means,
if you lose your Master Password,
Dashlane can’t help you.
But that’s fine,
because now you’ve just got a single password
to remember.
That is massively more secure than reusing
the same password
or variations on a password
everywhere online.
Like I said last time:
you should use a password manager.
So: dashlane.com/tomscott for a
free 30-day trial of Dashlane Premium,
and if you like it you can use the code “tomscott”
for 10% off at purchase.
