- Hello, everyone I'm Adam
Bradford with D & D Beyond
and we are here for the
first development update
and community Q and A
session of the decade.
We'll take a look at what
we're going to discuss today.
Latest updates, we're back in the saddle.
So, we have taken a big long break
and everyone is feeling completely
refreshed and recharged.
It was really, really
nice to have that time.
It always is every year.
We appreciate everyone out
there being patient with us.
I know that you probably missed
some articles over the time
and all of that
but it is an important time
for our teams to be able
to recharge the batteries
and really just hit the ground
running when we come back
for the new year.
So, this Monday was really
the first day that folks
were starting to come
back into the office.
We're really, really excited.
Everybody is already pushing ahead.
Doing, you know, all the
work that we're gonna start
here in 2020.
And then we did have some
downtime over the weekend.
And as we shared,
as that downtime was occurring
I'm going to touch base
at a high level.
I don't want to bore
anybody with, you know,
overly technical details.
But I certainly would like,
in the spirit of transparency,
as hopefully all of our fans
know at this point in time
if this isn't your first
development update,
you probably are aware
that we value transparency
above many, many other things
and we want to be open
and honest with our community.
So, we'll share at a
high level what went on
and most importantly what
we're doing to address that
and to do our best to ensure
that it doesn't happen
or if it every happens again,
it's as minimal as humanly possible.
And so, we'll go through that as well.
And then talk about upcoming.
And then there should be plenty of time
for questions at the end
because again we just
started back this week.
Not too many new things have
been released the former week
but we'll start getting into
that swing as we get into
the subsequent weeks here of 2020.
All right, so the Sapphire Dragon.
That's the same one that is in that
I think it's 300 dollar dice set
that you could buy
that had the sapphire in it.
So, that Sapphire Dragon is
available now on D & D Beyond.
Go check it out.
All the proceeds from D & D Beyond
will go to Extra Life.
It's a great cause.
And we've had a whole lot
of people already pick up
the Sapphire Dragon.
So, just jump on the bandwagon.
I love gem dragons
because I love psionics.
So, you need to check it out.
We are, as we reported
at the end of last year,
working on a player
mobile app that will have
the character sheet in it.
I can't wait to reveal a little bit more
about that pretty soon.
Encounter tracking is in progress.
We showed off some demo
videos the last time
we were together.
But we are in progress on that.
Digital dice rolling.
Very excited about the direction here.
And I think we will have
some alpha discussions
before too much longer on that.
And then a couple of other things
that came in this last week.
We have removed the mobile device limit
that previously existed.
Where if you subscribed,
you could have unlimited devices.
If you did not subscribe,
you could only use it on
I believe the limit
was something like six.
This was because of just
some problems ultimately that
that limit was causing.
Especially when we start
to think about things
like having separate apps for
separate types of functions.
And so we went ahead
and removed that limit
at this point in time.
It probably is not going
to impact you much at all
but we did wanna make you aware that
that had happened.
And then finally,
we had a billing name change.
So, when we were acquired by Fandom
we changed over the billing statements
that people would
receive for subscriptions
and even product purchases.
And I believe it said Curse initially
and then it ended up saying Fandom
when we were acquired there.
And that was producing some confusion.
And so at this point and time
those billing statements
and notices that you get
should say D & D Beyond.
So, hopefully that's going
to eliminate the confusion
about what it was that you bought.
So, those are the latest updates.
And now I'm going to
talk a little bit about
the downtime that we had, you know,
I believe it's Saturday and
Sunday coming into this week.
So, a few things here.
And I wanna be abundantly
clear that these few things
that I'm about to say are not remotely
offered up as excuses.
Nothing that I'm going
to talk about here today
is an excuse or anything like that.
You know, this is something that we take
very, very seriously.
I think that in the, you know,
two and half years of existence
that we've had for D & D Beyond
this was the first, you know,
what we would classify as
major outage of the service.
And really if you're
looking at it that way,
we have done pretty well
with that up to this point.
Again, not offering that as
any kind of justification
or excuse but saying that
that means a few things for us, right?
That, you know, so first of all
since this hasn't been
something that's happened,
we've had downtime,
we've had scheduled downtime,
we have been down, you know,
an hour or two here and there
and we got right back up quickly,
this one was longer than an hour or two.
And so, you know,
one of the factors that played into this
is since this hasn't happened yet,
our teams haven't had a lot
of practice at being able to
respond to something like this happening.
So, you know, one of the
positives that came out of this
was it did actually
allow our teams to engage
with something more serious like this.
And we really have in our
retrospective tightened up
a lot of the processes.
And again, we have top notch teams.
Some of the best developers, you know,
that I have ever worked
with throughout my,
I'm getting too old, long career.
And so again, I think that we're in
a good place going forward.
But this was a little
bit of a new situation
that we were coming up against.
So, thinking about it
also from that perspective
the timing was about the worst
that it could've possibly been.
As I said, we were on a holiday break
and it was literally the
last weekend of that break.
That did not effect,
obviously we had people on call
and everything else
but it was very, very
unfortunate timing that
that happened during our break.
Again, not offered as excuses.
Just kind of putting those
up as descriptive factors
into what, you know, was
going on here as well.
So, then what happened.
Our web nodes.
So, we're in AWS.
And our web nodes maxed out their CPU.
And naturally when,
so if you don't know what CPU is,
to be honest I don't think
I did until a few years ago,
but it's just how much the
processing is happening
on the computer itself.
So, as we're on these nodes
they maxed out their CPU.
The natural response to this
is to scale up the web nodes
because it just,
that's something that you can
do relatively easily in AWS.
So, we did the normal action that would
normally happen there.
But where the real problem came in
was that some of the
metrics that we had set up
and configured to tell us, you know,
feedback on that change of
scaling up the web nodes,
they were just inaccurate.
So, again the configuration was not proper
and they told us that
we were using, you know,
less CPU but the slowness
of the site did not improve
at that point in time.
So, as a result of the
inaccuracy of those metrics
from that, you know, analytics
software that we were using
we chased a lot of red herring.
So, people were just, you know,
slamming their head against
the wall trying to figure out
what on earth was going on
because it just seemed
to not make any sense.
And really the truth kind of
came out that nothing else
was going to make sense
(laughs)
because we were just acting
on some imprecise information.
So, we scaled up the database node,
which is another, you know,
step that you would take here.
And when we did that it
caused some other problems
in other services that we had.
So, we fixed those services,
which is nice.
So, if we ever have to
scale up the database node,
we are able to understand, you know,
what impact that had on
some of our other services.
Fixed those services but that
still was not doing anything
for the site slowness
and some of the outages
that were happening.
So, it was at that
point in time, you know,
hours into this
that we discovered the initial
scale up of the web nodes,
metrics were not correct,
and we saw that it was
not actually utilizing
all of the nodes properly.
We got all of that fixed.
Web nodes scaled up for real
and then the site stabilized,
and everything was okay,
and of course everything has
been okay since that time.
So, ultimately again none of
this is offered in any way
as an excuse.
We really are just doing this
in the spirit of transparency.
That this is something that
has not happened before.
And then the final thing
that I'll share here is
as this happened we had another,
we'll call it a champagne problem,
but we had another factor
that played heavily into this.
And that was that as of this Saturday,
we don't if it was because
of post holiday gifting,
post holiday gaming,
whatever was happening,
we had twice the number of
concurrent users on the website
than we have ever had.
So, take the record number of
concurrent users that we had
had on the website at one time
and Saturday as all this kicked off,
we had double that record number, okay?
So, again, you know,
the service is growing
just leaps and bounds
but this, that rapid of a growth
and that rapid of a hit
was kind of what started
and kicked off a lot of this.
So, again, spirit of transparency.
We wanted to share what was going on.
So, our response to
this is we have updated
all of our configurations to
make sure that we're unlocking
all available CPU on all
these massive web nodes.
The metrics and everything
are tying in now.
We're confident that we
have the right information
if anything like this happened again.
And then the other thing
is a longer term solution
that we have already been
in the process of for about
a year and a half now.
And that is over time
migrating all of our services
from a kind of monolithic
framework that everything
is currently based on
to what is called micro services.
So, if you're in the industry out there,
if you're a developer engineer,
you probably know what that is.
And we do have some micro
services that are functioning
properly but this is
something that we will be
continuing to do over time.
And every single time that we
spin up a new micro service
these kinds of issues
are going to be mitigated
more, and more, and more.
So, this is also why it's important
that as we are making
changes to, you know,
and adding new enhancements
and features to the website,
we will be taking a little
bit of time here or there
to make sure that we are
appropriately migrating
those services into the micro services
where they need to be
to make sure that we can continue to scale
with the just, hungry rapid
demand that we're seeing
for the service.
And you know, that
continues to surprise us.
So, I welcome any questions
about any of this.
I might not be the person
that can fully answer it
if it's too technical.
But we're gonna move on from here.
If you have questions, we
can absolutely get to that
during the Q and A time.
The one thing that I can say for sure is
we genuinely do appreciate our fans.
We appreciate your patience.
We know that it's very inconvenient.
I remember the feeling of, you know,
thinking about a raid the entire day.
And thinking about, you know,
how my warrior tank
was going to, you know,
appropriately place
the boss that everybody
was going to be fighting
and all that stuff.
And then getting home
and sitting down at the computer
and battle.net's down.
Like I remember how frustrating that is.
I know that it is inconvenient.
And so, really do empathize.
And we are going to continue
to do everything that we can
to keep downtime and outages
to a minimum and do our best
to eliminate them entirely.
So, we are really committed to doing that.
So, we do appreciate you
and one of the things that we want to do
to show that appreciation is
we do have some discount codes
that you know, actually
we haven't used these
discount codes before.
But if you have not picked
up the Legendary Bundle,
you can use "legendary
thanks" to take 150 dollars
off the Legendary Bundle.
"Source book thanks" will do 75 dollars
off the Sourcebook Bundle.
And then "book thanks," five dollars off
a single digital book.
So, by all means take
advantage of these codes
if you haven't collected
all of that content yet.
And again, we do appreciate you.
We appreciate the responses
during the downtime.
One thing that, you know,
we were doing our best
during the, you know,
hours that we were
experiencing some problems,
we were trying to keep you updated.
It looked like you appreciated that
so obviously we'll always
continue to be as open
and transparent as we can.
And we appreciate everything
that you are all doing for us
out there because of course we
couldn't do this without you.
And we do appreciate your patience
as we went through this incident.
All right, let's talk about
upcoming for a second.
So, we are working on Unearthed Arcana
class features variants.
As I previously explained,
this was a little bit of a
hand grenade that was dropped
into our portal, you know,
just before the holiday break
but we are getting to this
and we know that a lot of
players out there are wanting
to use everything that is included
in that play test content.
So, we're working on it.
Apple authentication,
it's going to be required
for Apple's app store
so we're pushing into that.
Other upcoming things this year,
we're gonna start looking
at AL support, we hope.
And then a potential target for 2020
is also inventory enhancements.
That's another thing
that we're talking about
after we get past the things
that we're currently working on,
which are dice rolling,
encounter tracking, and so on.
So again, this isn't a complete list
but these are some of the
things that we are working on
in, you know, the,
kind of the upcoming window.
And another upcoming and latest
might blur together a little bit.
So, for instance,
the Unearthed Arcana
class feature variants,
that probably should actually
be up since it's in progress
with the latest.
So, that just didn't get moved here.
But these are the things
that we're starting
to target for going
forward into this new year.
