Fried: Good afternoon.
My name is Ben Fried.
I'm Google's CIO,
and I'm here with
some of my colleagues
from Google Corporate
Engineering
to talk to you about how
we write enterprise apps
for Google on App Engine.
Now, at some point in the
evolution of this slide deck
of this presentation,
we realized that a better name
for this might have been
"Enterprise Design Patterns
for App Engine".
Unfortunately, that stroke
of genius came to us
after the print deadline for the
I/O schedule had already passed,
so we have the slide deck title
that you see in front of you.
Now, before I go any further,
I thought it might be good
to get to know a little bit
about all of you.
How many of you are here
because you're writing apps
for App Engine, for enterprise
purposes of some sort?
Okay, cool.
How many of you are here
because you'd like to write
enterprise apps on App Engine
and you think maybe we have
something to share with you?
All right, awesome.
How many of you would just like
to write apps for App Engine?
How many of you are here
by mistake?
Okay, all right,
well, you know,
they say that honesty
is the foundation
of any strong relationship,
and I'm glad that we have
that to begin with.
So to give you
a little road map
for what we're going
to be talking about,
I'm going to do some kind of
high level CIO-ish stuff
about what I've learned
and what we've learned
as a result of using App Engine
to solve Google's own problems.
Then Eric and the Justins
are going to show you
actual design patterns
that they've developed
and that other engineers
who we've worked with
have developed to solve
Google's problems
and show you
a bunch of actual code,
and I'll close it up
and we'll take Q&A at the end.
So, you know, we run Google
on App Engine,
like the slide says.
Over two years ago,
two and a half years ago,
we made the decision
that App Engine
would be our first choice
for the hosting platform
for the apps that we write
to run the business of Google,
and over the last
two and a half years,
dozens of engineers
have written hundreds
and hundreds of applications,
for every conceivable purpose
inside the company.
And the variety,
the scope of this stuff,
is pretty incredible.
It ranges from things like
managing our fleet of Macs,
doing software update,
operating system update,
and patching and pushing
to these Macs--I'm told
one of the largest enterprise
fleets of Macs in the world--
to running the system
that does 360-degree employee
performance review
and promotion management,
to doing online
scheduling of classes,
to the software
that our recruiters use
to schedule interviews
for job candidates,
software used
by our lawyers.
You know, in every corner
of Google,
there's an app running
on App Engine,
serving some important
business purpose.
As a result of all
that experience,
we've learned an awful lot
about App Engine,
and hopefully we have some
useful things to share with you
about how to actually make App
Engine work for your business
in interesting ways,
for solving enterprise problems
in interesting ways.
Now, you know, so, based on
this audience survey,
I assume that many of you,
hopefully most, are kind of
fellow travelers in the cause of
App Engine for the enterprise,
and my goal over
the next three slides
is to give you a few really
good arguments
for bringing App Engine in
for solving enterprise problems
in your organization.
And you know,
as we thought about it,
as we thought about it
and over two plus years
of having actually lived this,
it's interesting
what we've observed,
which is that the reasons
why App Engine
is such a great choice
and such an obvious choice
for building Internet-based
services, you know,
consumer Internet-based
services,
also actually have
tremendous applicability.
Those reasons have
tremendous applicability
inside the enterprise as well,
although it was often
not obvious to us at the time
when we went down this path,
that the arguments for doing,
why App Engine,
and why the Cloud hosting
is so great
for consumer services,
would be just as powerful
for the enterprise.
So the first one of these
reasons that came to us
was this idea
that App Engine apps,
you can access and use
from absolutely anywhere.
Right?
And why this is
such a killer feature
for the enterprise
can become clearer
when you think about the fact
that we're using App Engine
to manage software
and operating system updates
for our fleet of Macs, right?
Well, as Sumdar mentioned
at the keynote today,
most computers are laptops
these days, not desktops, right?
And that means that they are
as likely to not be connected
to our corporate network
as they are to be connected
to our corporate network.
And what's great about
App Engine is it means
that we can reach
and patch these Macs
no matter where they are.
For Googlers who are at I/O,
or at an airport,
or when they're at home,
they still get the operating
system updates,
and in fact
what we've noticed
is that software updates happen
off our corporate network
a tremendous
percentage of the time.
Another--you know,
one of the truisms
I think that--of modern
enterprise life
is that this idea that--
of a tightly controlled
corporate network
that you give blanket access
to a large number of people to
is kind of becoming antiquated
in the world of security threats
and interesting
business partnerships
that modern companies
all have, right?
So, the great thing
about App Engine apps
is you don't need a separate
remote access solution
to make them available
to people.
You also don't need to give
people a separate VPN
or remote access technology
to use it,
so you're not exposing
your corporate network.
You're not exposing apps
other than the ones
you might want to use
to App Engine.
It's really, really nice
if you want to, for example,
make a set of apps
only available
to a contingent workforce.
You can do that
without needing to expose
your corporate network
in any way,
and we've certainly seen
this to be a big,
big benefit for us
as Google expands the nature
of its business relationships.
So accessing from anywhere,
a great advantage
of using cloud-based platform
as a service,
also, really, really cool and
important for the enterprise.
I don't know why I chose
pink pigs here.
But the...
obviously scalability is
what App Engine and the Cloud
is all about,
and this makes sense
if you want to build a service
to be used by
hundreds of millions of users.
But as it turns out,
in the enterprise,
if you think about it,
we all have a whole bunch
of scaling problems as well.
Now the form
that this often takes
that many people can relate to
is that there's
a whole bunch of apps
in the enterprise portfolio
that have enormous
periodicity in their usage.
An example
that I mentioned earlier
is the performance management,
the employee performance
review system, right?
You know, it turns out that
like the five or ten minutes
before the system closes
are the time when it has a load
like a hundred times
the rest of the year, right?
It's funny how some people are
deadline-motivated, I guess.
Most people are
deadline-motivated, I guess.
And who wants to have
to kind of build out
or manage a hosting environment
at the capacity
of that last five minutes
or ten minutes, right?
You know, the sad reality of
the life of the IT professional
is that it's when your apps
are under the most load
and the busiest
and that we need them the most
that we can least afford
to have them.
You know, we suffer denial
of service from, you know,
usage--you know, powerful usage
patterns within your company,
and hosting on App Engine
gives you all that.
But there's another
enterprise use case
that's a little less obvious,
but one that we discovered,
related to the first one
but different
and very, very powerful.
You know, inside the enterprise,
we often have this problem
that you rarely want to build
a testing or QA infrastructure
that is the same capacity as
your production infrastructure,
right? It's just, that can be
a lot of money
to spend on something
that you'll use for testing,
as opposed to
actually in production.
So all too often,
we only actually learn
how our apps behave
in production...in production,
which means we get
unpleasant surprises, right?
And again,
it stems from this fact
that you don't want to spend
the money on having a plant
of equal or greater capacity
to production
to understand
where all the needs
are and the performance curves
of the code that you've written.
With App Engine, you know,
it's brilliant, right?
You just deploy a test instance,
throw as much synthetic load
at that test instance
as you want--
you know, 10 times,
100 times, 1000 times,
whatever your expected
peak load is--
observe how it behaves,
and understand exactly
what you're going to be in for
on your application's
busiest days.
There are certain regulated
industries I'm familiar with,
where I know
that the ability to do this
is not just
an important feature,
but something that regulators
actually demand as well.
And I guess it goes
without saying
that one of the nice things
about App Engine
as opposed to a lot of other
ways you could choose to do
scaling and capacity planning
and so on
is that you don't
have to do anything
to get the extra capacity.
App Engine just scales up more,
spins up more instances
to solve your problem,
to handle the load
that's being thrown at it.
And then,
when that load goes away,
those instances idle
and go away
and you only actually paid
for the cycles that you needed.
It's a really nice model,
far less capital-intensive,
than the, kind of traditional
way of doing things.
Yeah, I don't know
what's up
with the orange floor
tile puller either,
but this was the image I got
when I searched for
systems administrator.
[laughter]
But, and again,
this makes perfect sense
when you think
about this point,
that you do a lot less
administration with App Engine.
It's a complete
rich platform.
App Engine is doing
all this work for you.
You know, the typical
alternative that we face
is something
that's the equivalent
of a virtualized version
of bare metal, you know,
onto which you have to manage
an entire web hosting stack
yourself, right?
I mean, you need to make
a pretty significant investment
in administration
and operations teams
in order to run that
web hosting stack
with load balancing
and web servers
and application servers,
and, you know, on and on and on
and on and on.
Whereas with App Engine,
you mostly just think
about solving the problem.
Now what we've seen after
two and a half plus years
of doing this,
many, many applications
that we're running at Google,
is that App Engine apps
typically take 1/10 or less
the system administration effort
of apps running on a traditional
web hosting environment
that you might find inside
your corporate network.
And you know,
what's great about that,
the thing that I think
is a great case to be made
in the argument for App Engine
in the enterprise,
is that it means that the IT
people, the enterprise people,
get to spend way more time
doing new things,
innovating, building new apps,
and way less time
operating the things
they've already done.
And I certainly
got into computing
because I liked
building new things,
innovating,
solving new problems.
That was far more attractive
to me than operating solutions
to yesterday's problems,
and I really, really liked
being able to go
to my management
and show that, as we made
this move to App Engine,
we spent a lot less time
thinking about
systems administration
and operations stuff
that's really important,
but only tangential
to the business of
the problem solving that we're
supposed to be doing.
So, okay, that's kind of my
30,000-foot CIO level fly-by
about why we think App Engine
is really, really good
for the enterprise and some of
the benefits that we've seen.
They're going to translate
that into something
that you can actually use.
I'm very happy that Eric
and the Justins are here
to talk about actual
development patterns
that they've discovered
in writing the applications
that we use at Google.
They'll show you a bunch of code
that implements that.
Justin McWilliams
is going to start off
with three design patterns
that he's discovered.
McWilliams: Thanks, Ben.
Hello, everyone.
I'm Justin McWilliams.
I'm an engineer on the corporate
platforms engineering team
in New York City.
This team is responsible for
building tools and applications
to help manage our fleet
of workstations within Google.
You'll see this
in my patterns.
So the first pattern I'd like
to talk to you about
is considering App Engine
as a web service host,
not simply a website
hosting platform.
So yes, App Engine
is an HTTP server,
but that doesn't mean
it needs to serve HTML.
It's actually good for
running a web service
that other services and clients
can integrate with,
and as of the App Engine 1.5
release yesterday,
it's actually good for consuming
other external web services
as well.
Now, when building your
web service on App Engine,
you have various
authentication options.
Given to you by the platform
are Google Account-Auth,
and OAuth, but you're not
limited to just those two.
You could build whatever
custom authentication protocol
you'd like.
For example, something
certificate based,
or a shared secret.
Now a good example of where
we use App Engine in this way
is the Simian application
that Ben mentioned.
This is our enterprise class OSX
software deployment solution.
So the idea here is we have
a bunch of OSX machines
in our environments,
and they have programmatic
clients on them
that are interfacing with
their App Engine application.
Now, we use HTTPs for encryption
of our traffic here,
but we can't use it
for authentication
for a couple of reasons.
First off, the App Engine
front ends
all share the same SSL
wild card certificates.
And therefore to our clients,
all App Engine front ends
look the same.
My app, your app, his app,
they're all the same.
Now second, there is
no client HTTPs access
on your app as well,
so it's not clear
which client is connecting.
So what we do is we store
some certificates
on the OSX machines,
and in the Datastore,
and we use these to sign
messages that get sent
back and forth,
verifying the authenticity
of both the clients
and the server.
Now once this goes through,
we send a cookie
down to the client
that is used for
subsequent requests.
Those include fetching
a manifest out of Datastore,
which is basically a list
of packages to install.
And then we're going to fetch
the packages out of Datastore--
or sorry, out of Blobstore,
and install them.
After that happens, it reports
the installation results
up to the server, which is
stored back in Datastore.
And then, we're going
to log out,
so that cookie
is expired immediately.
But we have cron jobs
that run regularly
that clean these cookies up,
just in case
that logout didn't work.
And there's
other cron jobs.
I'll talk about one of them
a little bit later--
basically report
generation.
So, I mentioned that App Engine
is good for other services
to integrate with.
Here's an example.
So we have this system at Google
where employees can request
commercially licensed software,
maybe something like Adobe
Photoshop, for example.
So what happens currently is
the employee makes that request,
their manager approves it,
and then a technician has to
manually install that piece
of software on their machine.
We want to get to a point where
these systems are integrated,
so when that approval
goes through,
the systems
talk to one another,
and that package
is automatically pushed down
to their machine.
So next App Engine pattern
I'd like to talk to you about
is using agents to sync with
your corporate services.
Now, last year, I told you
how you could use
secure data connector
to accomplish this.
This will work
in a similar way.
So you're going to use
HTTP PUT or POST,
or BulkLoader,
to push your data up
into your App Engine
application.
But now you have data
in two different places
and of course you're going
to want to make modifications
on the App Engine side.
So you're going to expose
the cloud state
and those changes
via some HTTP end points.
And then you're going to have
a cron or an agent
within your corporate
environment
that's pulling that end point,
looking for that state.
When it sees the changes,
it can make those changes
on your corporate services,
and then report back
up to App Engine that
those changes were made,
so that all systems
are in sync.
Now this particular pattern
we use in various applications
within Google,
but I'll give you an example
of something you might be
able to relate to
within your corporate
environments.
So, let's say you have
an antivirus solution
of some sort
and you're not happy
with the interface
it provides for you.
Maybe you're not able
to delegate access
to different levels
of technicians,
or maybe you're not happy with
the reporting that it gives you,
so you want to build your own
interface for that.
Here's how you can do it
on App Engine.
So you're going to have
a bunch of workstations,
obviously, running
antivirus clients,
and they're going to report
some events
up to a server
of some sort.
In this case,
the example is SQL,
but it could be
any API or service
within your corporate
environment.
So, let's say that it reports
like a virus notification
up to that server.
You're going to have an agent
that's looking at that server
for changes,
and when it sees them,
HTTP PUT or POST pushes it up
into App Engine,
and now your admin can see
all of those virus notifications
on the App Engine side.
Now you can imagine that admin
is going to want to run
some, like, full disk scans
on some of those machines.
So we can click a button.
That action
is going to get exposed
through some HTTP endpoint,
which you have
another agent monitoring,
and then it's going
to make some changes
within that SQL server
for that antivirus solution,
and eventually that client's
going to run
that full disk scan.
Now, you can imagine that after
that full disk scan is complete,
it's going to report
some more information
up to that SQL server,
which will then propagate
its way into App Engine,
so the admin gets a nice,
consolidated view
of that machine.
So the last pattern
I'd like to talk to you
about is using query cursors to
accomplish iterative fetching.
So, the idea with any app
in general,
especially in App Engine,
is you want to respond
to your users and clients
as quickly as possible.
And any expensive tasks
that you have to do,
you defer to the background.
In App Engine,
you can accomplish this
using TaskQueue Tasks.
So, query cursors allow you
to do--
They allow you
to step through queries.
So with this you can accomplish
UI pagination,
or simply check-pointing
during large query operations.
Query cursors can be
expressed as strings,
and therefore
they're very portable.
You can save them in
Datastore and Memcache,
or wherever else you want
to store your data.
They're also persistent,
so you can use them
across requests,
and continue where you left off
on a previous request.
Now, a good example
of where we use this
is back in our Simian
application.
I mentioned these programmatic
clients are posting
install results up to our app,
and we want to respond
to those, uh, those clients
as quickly as possible,
so they can continue on doing
whatever else they need to do,
installing further packages
or whatever.
So what we do is we simply
save those results
to the Datastore very quickly,
and then we have a cron job
that runs and analyzes
the results
to generate some reports
for our admins.
So the first thing
this cron's going to do
is pull the stats
out of the dictionary.
Or, sorry, dictionary
of stats out of Datastore.
And we use a little helper
method here,
just in case this is like,
the first time
the function has run,
and it's also--it's also
Memcache-backed.
So we're going to
set up our query.
We're looking at all installs,
oldest to newest,
as they come in.
And then we're going to pull
our query cursor
out of the dictionary.
Again, this was saved
on the last execution
of this function.
If that cursor exists,
we're going to attach it
to our query
so that we pick up where
we left off the last time.
Then we're going to fetch
1,000 installs out of Datastore.
Now, 1,000 here is
completely arbitrary.
Could be 10, 100, it's going
to work the same way here.
If there's no more installs
to process,
we'll just return out,
but I mentioned this is a cron,
so in a few more minutes
when this cron kicks off again,
hopefully there's more installs
to process.
Once we do have some installs
to process,
we'll loop over them,
analyze whatever we need to,
and update that install
stats dictionary.
And then, of course,
we're going to save it
back into the Datastore.
And this helper method
also updates Memcache,
so our reports
go live immediately.
Of course, we're also going to
have to save our current
query cursors to Datastore,
so that the next time
this function runs,
we continue
where we just left off.
And then finally, we're going to
defer the execution
of this function to happen
one more time in the background
using TaskQueue Tasks,
so that just in case
that magic number we chose,
the 1,000 or 10 or 100
or whatever, wasn't enough,
and there's actually
further installs to process.
So it's important to note that
query cursors and cron jobs
are not directly tied
in any way.
We could easily kick off
this same exact code
based on any other criteria,
maybe every 10 installs
that are reported
to our server.
We would just fire this off,
and things would work
just as you see here.
So next up, I'd like to welcome
Eric to the stage.
He's going to talk to you
about some creative ways
of using Blobstore.
Schoeffler: Thanks, Justin.
Hi, everyone.
I'm Eric Schoeffler,
and I'm here from the staffing
applications team
out in New York City.
Now basically, the team
works on the software
we use to get people
hired at Google.
I'm here today to talk to you
about an alternative
to using Datastore
on App Engine.
So I saw a lot of you
raise your hands
saying you worked
on App Engine before.
Now, if you've stored
any data on App Engine,
you've probably used
Datastore.
And Datastore works really well
when you're dealing with
a few records at a time.
So let's say you had
an employee database,
and you wanted to look up
how many vacation days
an employee has,
or you want to update
the phone number
of an employee.
Datastore is great at that,
because you're only dealing
with a few records
at a time.
But what if you
need to do a report
on everyone
in the database?
It turns out Datastore
is actually fairly slow
when you do something
like that.
It actually becomes more
efficient to store your data
as a blob in Blobstore,
deserialize that information
into memory,
and then you can run
whatever sort of operations
you'd like to on that data.
Now, obviously this has
some limitations.
When you're updating your data,
you have to update
that entire blob,
so that means
you can't individually update
one record at a time.
You have to update
the whole thing.
The other thing is, you're
pulling out all the data
whenever you want
to read anything,
so that means
that you need to pull out
that large blob of data
into memory,
which leads to the second
limitation.
There's a limitation of RAM on
whatever machine you're using.
You're pulling out that entire
database into your RAM,
so that needs to be able
to fit there.
But there's going to be
certain cases
where this is going
to work out fine,
and these limitations aren't
going to be a problem.
And in those cases, you're
going to get some benefits.
There are a couple benefits.
The first one is flexibility.
So when you're dealing
with Datastore,
you have to deal with the query
language that it provides.
So if you wanted to do
some sort of query like,
finding all the people
in a company
who started two years ago and
have done over 50 interviews,
that's impossible
with Datastore,
and that's because
you're not allowed to do
multiple filters--
multiple inequality filters
on multiple properties
in the same query.
But with Blobstore, since
you're pulling all your data
into memory, you can do whatever
sort of data manipulation
and filtering that you'd like to
on that data.
Now, you can even do something
as arbitrary as finding everyone
in the company who started
on an odd week of the year.
The other benefit you're going
to get with Blobstore is speed.
It's speed in upload
and speed in access.
So, whenever you upload
all your data,
all you have to do is upload
that one blob.
Now with Datastore,
to upload 50,000 records could
take over 30 minutes;
but with Blobstore,
the equivalent 5 MB blob
will take only
about 5 seconds to upload,
so that's pretty quick.
The speed benefit is also there
with accessing the data.
So I could just talk
to you about it,
but I'd rather show you.
So what I've done here is
I've created 50,000 records,
each one of these records
has a hair color,
an arbitrary test score,
and 100 bytes of arbitrary data.
So essentially, we have 50,000
records of about 100 bytes each,
and I've uploaded that
to Datastore as 50,000 records,
and I've uploaded it
to Blobstore
as a serialized CSV file.
It's about 5 MB.
And what you see here is
a front end that allows me
to filter based on
hair color
and find the average test score
of those people.
So the first thing
I'd like to show you
is a small result set.
So I'm going to look for people
with blue hair.
There's not many people
with blue hair in this database.
And you'll see that Datastore
returns extremely quickly.
It's actually less
than 20 milliseconds.
With Blobstore, it takes
a little bit longer
because you're pulling in
all of the data.
But what happens if we look
at a larger result set,
say people with black hair?
It turns out there's 25,000
people with black hair
in this data set.
And what you'll see
is that Blobstore returns
in the same amount of time
that it did
when we were looking
for one record,
but Datastore takes
a little longer.
Now that was only half
of the records in that database,
so what if I don't
filter at all?
What if I'm looking for everyone
and trying to average out
the scores of everyone?
You'll find that in that case,
Blobstore returns in about
the same amount of time,
around two seconds,
and with Datastore,
it takes even slower.
It's nearly twice as long
as it took
when we were looking for
people with black hair.
And the reason for that
is because Datastore
has some overhead
for pulling in a row.
So the more rows you pull in,
the longer it's going to take.
But with Blobstore,
you're always pulling in
all of the data, so whether
you're looking at one record,
or 50,000 records,
it's going to take
the same amount of time.
So now let's look at that demo
100 more times.
Now, I'm not going to make you
watch me do it 100 more times.
I've already done
the work for you.
So what you see here is
that last demo,
run 100 times on both
Blobstore and Datastore--
Blobstore in blue,
and Datastore in red--
and as it hits the gram
as latency for both of them.
And what you'll see
is some variations
for both Blobstore
and Datastore.
But in general, you'll see
two different spikes,
so there's one spike
at two seconds for Blobstore,
and one spike at 11 seconds
for Datastore.
So what this is showing you
is when you're dealing
with 50,000 records,
Blobstore is going to be
a lot faster than Datastore.
So now that I've showed you
that Blobstore can be faster
than Datastore,
you're probably wondering,
"How do I implement this
on my own app?"
Well, it's a three-step process.
The first step is you have to
allow your app to accept blobs.
So to do that, you need
to create two servlets.
The first servlet
is an upload servlet,
and what that does is,
it creates a one-time use
upload URL
that you can upload
your blob to.
Now when you create that,
you have to specify
a callback servlet,
and in this case, the callback
servlet is post upload.
And that's the second servlet
that you need to create.
This callback is called once
Blobstore has stored that blob.
And what it does is it gives
you metadata about the blob
in the form of a blob
key object.
Now if you want to access
that blob in the future,
you need to store
that blob key somewhere.
So I would suggest storing it
in Datastore.
That way you can access it
whenever you need it.
Now, remember, we talked about
Datastore being fast
when you're dealing with
a few records?
Well since you're only pulling
in one blob key at a time,
it's going to be very quick,
and the time is negligible
to pull out that data.
So, now once you've added
the ability to upload blobs,
you need to create a blob
and actually upload it.
So to do that, you need to
take all of your records,
and serialize them up
into a file somehow.
In the example I gave,
we serialized it as a CSV file.
Now, obviously that's not
the fastest
serialization method,
and in fact
in production for Java,
we use protocol buffers.
But you're going to
have to find the one
that works best
for the environment
that you're running on.
So, for example,
in Python,
you wouldn't want to use
protocol buffers,
because they're a little bit
slower in Python.
So once you've done that,
you need to upload that blob.
So you use the servlets
you created in Step 1
to upload the blob,
and then once the blob
has been uploaded,
it's ready to be accessed
whenever you need that data.
So to do that, you need
to pull out the blob key
that you stored in Datastore,
use that blob key to pull out
the stream that represents
the blob,
and deserialize that blob.
Once you've deserialized it,
you have all the data in memory
and you can do whatever
you'd like to with that data,
and that brings us back
to the flexibility thing
that we were talking about
before.
Since the data's in memory,
you can run any sort of
arbitrary filters,
or any sort of manipulation
on that data
that you'd like.
Now, if you're anything like me,
hair color, test scores--
they're not very interesting.
It's really fake.
So how can this be used
in the real world?
Well, it turns out I do that
in a project called Lineup.
And Lineup is
an interview-scheduling tool
that we use here
at Google.
You see, at Google,
we have this problem
where we have a lot of
candidates and a lot of people
who can interview
those candidates.
And what we want to do is find
the best interviewers
for each candidate.
So what we do is
we store information
about all of our employees,
so every employee
has some information,
like what program
and languages they know,
how many interviews
they've done,
what their job level is,
and we pack that all up
into a blob
and we push that blob
up to Lineup.
Then, whenever a recruiter
needs to set up an interview,
they just describe
the candidate,
and we go through that blob,
and we score every employee
based on how well they match
a candidate,
and then we rank them.
We go and find their
availability in Google Calendar,
and we schedule them
based on how well
they match that candidate.
And this way,
everyone's happy
because the interviewers
are interviewing candidates
that they actually understand,
and the recruiter doesn't have
to spend all that time
figuring out the schedule,
and can spend more time
talking to the candidate
and getting them
excited about Google.
So, the bottom line here
is that there are two different
data storage solutions
in App Engine.
Now, you might have heard
of Blobstore before,
but you probably thought of it
as just a storage system
for blobs like images, PDFs,
and that sort of thing.
But it's actually useful for
your main data source as well.
And both Datastore and Blobstore
can be useful
in different situations.
So when you're dealing with
a few records at a time,
Datastore is great,
or when you have
a very, very large database,
Datastore is great
because you don't have to pull
all that data into memory
at any given time.
But with Blobstore,
you can do large updates
and you can grab
a lot of information
all at once very quickly.
It also allows you
the flexibility
to do any sort of queries
that you'd like on that data.
So, with that I'd like to
pass it off to our next Justin,
who's going to talk about
using the channel API
for collaborative apps.
Fagnani: Thanks, Eric.
So my name's Justin Fagnani,
and I'm an engineer
in corporate engineering
down in Mountain View,
and today I'm going to talk
about how to implement
a real-time collaborative
editing system
using App Engine
and Google Web Toolkit.
So, real-time
collaborative editing
has been popularized by many
apps in the past few years,
especially Google Docs.
And this feature
is really awesome to have
when you have multiple people
editing the same document
at the same time.
So, what we're going to do here
to implement this
is we're going to use
the command pattern.
We're going to use
the App Engine channel API,
and we're going to use some
internals of GWT serialization.
Now, notice
that the command pattern
in the related
operational transformer
are patterns
that are commonly used
to implement
real-time collaboration.
You can find out about those
in more detail online,
but today I'm mostly going
to talk about the channel API
and how to wire things
together.
All you really need to know
about the command pattern
is that instead of sending
entire objects back and forth
like a traditional
web application,
we're just going to send
each edit as it happens.
So, collaboration relies on
sending messages
from the server
to the client,
and traditionally this has
been very tricky to do well.
It's actually been somewhat
impossible also on App Engine
due to the 30-second
request time out.
Luckily, though,
the App Engine team
has released
the channel API
and this takes care of
all the technical details
of setting up
this server-to-client
communication channel for us.
So that's great, we can
have collaboration, right?
Unfortunately
it's not that simple.
We still have to do
a lot of things ourselves,
specifically the channel API
only allows you to send
a simple string message
to one client at a time.
So if you want to send
a message to all of your clients
or to a group of clients,
you have to build and maintain
a registry on your own.
Also, we would like to be able
to send something more
than a simple string message,
say a GWT serialized object.
So that brings us to our three
key responsibilities here.
First, we need to
build and maintain
a list of the active clients and
what objects they're editing.
Then we need to be able to
serialize and de-serialize
our command objects,
so that we can send them
as a string
over the channel API.
And finally, we need to be able
to execute the commands,
and then dispatch them
back to all the clients
that are listening
to that object.
So here are our main
components.
We have a command object,
which I've talked about.
This object represents
a user action,
and it's the instructions on how
to modify a particular object.
The command object also performs
this modification,
and it performs it the same
on the client and the server,
thanks to the wonders of GWT's
Java to JavaScript translation.
Then, on the client,
we have the CommandController,
and this coordinates
all the command-related activity
in the browser.
The client is responsible
for opening a channel
and registering
that it's listening
to a particular object,
and it also periodically
pings the server
to let the server know
that it's still listening.
And then, most importantly,
it's responsible for sending
and receiving command objects
to and from the server.
On the server, we have something
called CommandService,
and this is the brains
of the whole operation.
The CommandService builds this
mapping of objects to clients
that are editing them.
Uh, it, you know,
listens and makes sure
that the clients
are still listening,
and then it also sends
and receives the commands
to and from the clients.
And finally we have something
called the CommandExcecutor.
The CommandExcecutor
is a helper that loads objects
that the commands
need to edit,
and then invokes
commands on them.
The CommandExcecutor
is a little bit different,
possibly, on the client
or the server.
We can have different behavior--
load objects from Datastore
or, you know,
from some cache
that you have in the browser
differently.
So here's a diagram that shows
how these pieces fit together.
If we start over here
on the left,
you see that we have
the CommandController,
and it sends a command to the
CommandService over at GWT RPC,
where the service receives it
and then executes it
via server side
CommandExcecutor.
The executor
will produce a result,
which is also a command,
and this command is serialized
and then sent over
the channel API back
to the listening clients,
where it deserializes it
and executes it via the client
side CommandExcecutor.
The important thing to note here
is that we are sending commands
to the server over GWT RPC,
and back to the clients
via the channel API.
So I talked about
how the channel API
does not help us keep track
of our clients.
We have to do it ourselves.
So here's a little bit
on how to do that.
This is the interaction
between the command
controller on the client,
and the CommandService
on the server.
So the first thing that happens
is that the CommandController
is going to open a channel
for a specific object.
When the CommandService
receives that call,
it's going to store this mapping
from an object ID
to a list of the clients
that are editing it,
and the timestamp, so the last
time it's seen that client.
And it stores this mapping
in Memcache.
Then, periodically,
the CommandController
is going to call KeepAlive
in the CommandService
to let it know that it's
still listening.
When the service receives
this KeepAlive call,
it's going to make sure that it
still knows about that client.
It's going to check
that it has a valid mapping
for that client
and that object ID,
and if it does,
it sends an okay.
If it doesn't,
it sends an error.
The client,
if it receives an error,
has to reopen the channel and
then resync the entire object.
So let's look
at some code now.
So this is a method
on the CommandService.
This is open channel
that lets the client register
for a particular object.
So the first thing we do here
is we load a mapping
for the object that we're
interested in out of Memcache.
And then we have to put
a new entry for the map.
This is our client ID,
and then the last time
we've seen or the time that this
method was called here.
We put that into
our mapping.
And then next we need to store
the mapping back into Memcache.
Now, notice that we're using
GetIdentifiable
and PutIfUntouched
on Memcache here.
This lets us perform
this whole action
as an atomic operation
so that another server
or App Engine instance
cannot modify the mapping
for this object
as we're doing this.
So next, let's talk about how
we actually execute a command.
So the first thing
that happens here
is that a user performs
some action in the browser.
And we need to process
this action into a command
and give it to the command
controller.
The CommandController
will then call execute command
on the CommandService,
and that sends it to the server
over at GWT RPC.
And then, the CommandService
will execute this command
via the server side
CommandExcecutors,
and it gets the result command
and broadcasts it back
to the appropriate
listening clients.
Finally, all the clients
that are listening
for that particular object
receive the command
over the channel API,
deserialize it,
and execute it locally.
So, here we can see
how we execute the command
on the server.
The first thing we do is we pass
the command to an executor.
And, I gloss over how we locate
the particular executor
we need a little bit,
but typically what you'll have
is a dictionary of executors
where you look up one
based on the command type.
Next, we get the result command
from the executor,
and we serialize it
via a helper class,
and I'll go over this helper
class in a second.
So that gives us a string.
Then we can load all
the channels that are listening
for this particular object
out of Memcache.
And this function here
is a helper
that just gets the map out
of Memcache like you saw before.
And finally, once we have
a list of all of our clients,
we can iterate through them,
and for each one
we can individually send
the serialized command message
over the channel API.
So this is the channel API
call right here, "send message".
So now let's talk about
how we get GWT serialization
to work over the channel API.
So normally GWT serialization
only works for GWT RPC calls,
and the compiler
takes care of making this
transparently
happen for you.
But we can get this to work
with arbitrary strings,
and then therefore
the channel API.
So the first thing we have to do
is we have to create
a dummy method
on the CommandService.
This dummy method forces GWT
to include the command class
in the serialization policy.
So you can see
we have a snippet
of the CommandService
interface here.
And we've defined a method
literally named dummy method
that just returns
a command.
So the implementation of this
doesn't do anything,
because it's never called,
but we need it to be able
to serialize the object.
After that, we need to locate
the RPC manifest file,
which will help us load
the serialization policy file.
And once we have
those two things,
the dummy method and
the serialization policy file,
we can call encode response
for success with our command,
and it will turn our command
into a string for us,
and then that's what we send
over the channel API.
On the client, things are
a little bit simpler.
We can simply cast
our CommandService to
a SerializationStreamFactory.
From the StreamFactory,
we can give it
our serialized message
and it will give us
a stream reader,
and then we can call read object
and get our command object back.
So here's the code on
how to do serialization.
On the server, we have
a command serializer class.
The first couple lines here,
we're just loading the dummy
method via Reflection,
and then using
some file system operations
to load
the serialization policy.
In your project,
you'll have to include
the serialization policy
in your build, you know,
ant or something like
that to be able to get in,
so I left that out here.
But once we have those,
then you see we can call
encode response for success
with the command object,
and we get a string.
On the client side, we have
the command deserializer.
Here we're casting
our service
to a SerializationStreamFactory.
Here we're using
the StreamFactory
to create a stream reader
out of the message.
And then finally
we can call read object,
and we get a command object
back.
So, I hope that was
an incredibly brief
but hopefully helpful overview
on how to implement
real-time collaboration
on App Engine using GWT.
So, quick summary here.
So, the first thing we did
is we had to build
this map of clients and objects
they're listening to
and we stored
that in Memcache.
Then we need to be
able to serialize
and deserialize objects,
and we forced GWT
to let us do this by creating
this dummy method,
and then finally we need
to be able to execute commands
and dispatch them
to the correct clients.
We use that mapping that
we established in Step 1.
So I think this is a great
enterprise feature actually,
and I hope that you consider
using the channel API.
It opens this up to a lot
more applications
and developers than may have
previously considered it.
Thanks, and I'll hand it
back off to Ben.
Fried: Thanks.
[applause]
So I asked the App Engine team
last week
approximately how many apps
does Google itself have
running on App Engine?
I guess in Hollywood
they like to say,
"Everyone wants to direct."
I think at Google,
everyone wants to write code.
Maybe everyone wants
to write App Engine code.
The reason I say
that is that the answer
I got from the App Engine
team is that,
if you look across
all the different environments
and all the different users,
there's something north of
10,000 unique application IDs
that are assigned
to Google purposes.
I wouldn't claim
that that means
there's 10,000 production
applications running
on App Engine for Google;
far from it.
A lot of that
is individual experimentation,
small tools, testing,
things like that,
but hopefully, that number
gives you a sense of the scale
and the way in which Google
is using App Engine
for its own purposes.
And, you know, it's based on
that experience that we've got
that Justin, Eric,
and Justin gave you
some tips based
on their experience,
a lot of cumulative wisdom
of how to get useful things
done
for the enterprise
with App Engine.
Hopefully you found that useful
and then you can tie that back
to those arguments
I was making about
why App Engine is this great
enterprise tool
at the beginning.
So, one thing to note
before we take Q&A,
some good news
for all of you
is that we've posted sample
code for all the patterns
that we went over,
for--for these five patterns.
There's sample code
for all of them available
on code.google.com.
You can look
for them there.
I think the--our sample code
continues to get better.
We're going to post more online
over the next couple days,
but it's a lot there.
Take note of the URL,
and Simian and a handful
of other projects
that we run on App Engine
are fully open source
and available for you
to download,
and you know,
copy, duplicate,
use in your own environment
as you see fit.
With that, I think we're ready
to take questions.
If anyone has questions,
please come to one of the mics
so that you can be recorded
on video for,
you know, all of history.
Oh, and people here will tend
to shoot T-shirts at you
while you do that, so...
man: So, I work in Detroit
in the automotive industry,
and we have a lot of--
we can generate a lot of data.
32 gigs, 64 gigs can be
a common use case
for some of our customers.
And we need to do
some processing
on that in the Cloud,
so App Engine sounds like
it'd be great for me.
But I'm wondering
why should I use App Engine
and not something like
Amazon EC2 to do my processing
when I have a lot more control
with EC2?
I'm kind of wondering
which direction to go with,
if you could offer me some...
Fried: Right.
man: You know,
convince me, I guess.
Fried: We have
some of our friends
from the App Engine team
here to talk about it.
This is Greg Dalesandre,
but can I take
the first shot at it,
and you can tell me
if I screw it up?
Dalesandre: Okay.
Fried: Okay, so, I mean,
as CIO guy,
let me give you my perspective
on why App Engine is better.
It goes back to that third
point that I made, right?
I mean, you can...
you know, if you want
the abstraction
of some virtualized version
of bare iron,
you can have that,
but with great control
comes great responsibility.
I might rather not worry
about that responsibility.
I might rather worry about
solving the actual problems
that we've got
and allow the Google SREs
to worry about keeping all
that stuff running yourself.
I mean, if you're getting
virtualized bare metal,
you've got to build
that complete hosting
environment yourself,
monitor it yourself.
You know, you have
to worry about kind of
provisioning mechanisms
and so on.
You know, I mean, it's--
I view that as kind of
the assembly language
of cloud hosting,
and, yes,
in assembly language,
you can make a machine do
exactly what you want it to do,
but, you know,
we all want to move on
and operate at a higher level.
That's my take on it.
Greg, did you...
Dalesandre: Yeah.
What he said, um...
But, in all seriousness, I--
a lot of our customers
that we talk to initially start
using App Engine
because the scaling
seemed appealing to them.
And in the end, what they found
that they really actually loved
about App Engine was the fact
that they didn't have to have
a team of people maintaining,
and, you know,
every new update
that came around,
they have to apply
the new updates.
And I've worked
at a bunch of start-ups,
and I know what happens
at start-ups all the time
is--is, you know,
you'll build something,
and then as soon as
it starts scaling,
you're scrambling
to try to figure out
you know, how to--
how to, um,
keep things alive.
And so it's that, like,
not needing to worry
about patching things or
figuring out who's managing it
or who's carrying the pager
and all those things
that I think is why
you'd use App Engine
instead of just
the bare metal.
Fried: I mean, since we started
using App Engine,
I think we've net added
no system administrators
to the teams that support this
fleet of corporate applications.
Right? And that's a pretty
powerful statement.
I might be off
by one or two,
but that's kind of
directionally right.
Schoeffler: You got
one more over here.
Oh.
man: Oh, oh.
Fagnani: Quick,
shut off the camera.
man: I have a question
on the channel example
that was given.
I started using Memcache
to store the list of clients
that we're connecting.
What happens when, you know,
it's no longer in Memcache?
Fagnani: That's a really
good question.
So, there's a couple of things
to understand
about the channel API.
The first thing
is that a channel
is only valid
for two hours.
You get a token,
send it to the client,
and then it opens
the channel, right?
So your channels are basically
going to be going away anyway.
And Memcache ends up
being fairly stable,
so the choice I made here
is that, you know, assume
that the channels
can come and go,
and that you need to build
a resyncing mechanism
for your object,
so that you can open the channel
and download
the whole object again.
Once you make that assumption,
you know,
whether or not Memcache
gets cleared
doesn't really matter.
And the fact that it's faster
and simpler
was an advantage over
storing these in Datastore.
You could certainly store
these mappings in Datastore,
cache them in Memcache,
and use a two-layer mechanism
to get to them, but I don't know
if it's actually worth it.
man: So the channel API then
wouldn't really be used
to allow--
I'm just thinking like--
for developing, like,
a Google Docs type thing?
For collaboration, you know,
and people that could be,
you know, in there for, you
know, their entire workday.
I mean, if there's
a two-hour limit, or can--
every time it's used,
is it extended for,
you know,
another period?
Fagnani: No, it doesn't
extend it for another period,
but basically, you know,
you're on a network,
you have connectivity issues,
right?
The laptop might close,
it might go off the network,
things like that.
So you have to build
in this mechanism
to be able to say, okay,
we got disconnected,
and we might have missed
a whole bunch of commands.
We need to resync
the entire object.
And if you have, you know,
an easy way to do that,
then these other concerns
kind of go away,
because you can say, Oh,
we might have lost a connection.
Just reset the object.
So that's the kind of philosophy
I've taken there.
So I think
that two-hour token window
doesn't really matter either.
man: Okay, thanks.
Fagnani: Welcome.
man: We developed enterprise
mobile applications
that often require
a middleware of some sort,
and our customers say we need
to be able to support it,
much like the first question
that was asked.
What kind of guidance
can you give us
for folks who don't want to
kind of relinquish that control,
but want the flexibility
of what App Engine offers?
And being the developers
that we are,
how do we kind of
message this
and try and get through
the download app engine?
What kind of guidance there?
Fried: I didn't quite
understand the point,
the middleware-related
to control point.
Is the idea that people want
to run the middleware
in their own kind
of data center?
man: Exactly.
To be able to structure it,
and be able to control
and support it from there.
Fried: I mean, I'm--
I don't think I--
I may not fully
understand the question,
but, I mean, I think
there are a few options
that are available to you.
One is you can have the same app
running on App Engine
in Google's Cloud and running,
you know, more or less
in some other internal
environment.
There are App Engine
emulation systems
that you can get
to run inside,
and that way people can kind
of get their feet wet
with both environments, right,
and understand
the benefits of one
and the costs of the other
and so on.
It's really, really easy
to make that migration,
especially if you make
the decision
from the beginning to code
for App Engine, right?
I mean, the thing
you really need to learn
is you need to--
to code for App Engine
is you need to think
through things like
your data
integration issues,
and, you know, what storage
techniques you're going to use,
but if you make the decision
that it's going to be
an App Engine app,
maybe you know,
whether or not you host it,
and from day--
where--no matter how
you host it from day one,
that's going to make it very,
very easy to move in
and kind of get people
to do that.
I think once you start trying
to run an app engine, you know,
to answer the first person's
point about control--
Control is also
highly illusory, right?
I mean, I've had lots of
conversations with people
from a previous job of mine,
in fact.
You know, people think
because stuff
runs in their data centers,
they have a lot of control
and they don't necessarily
realize how many people
have keycard access
to their data centers,
or what happens to tapes
when they move off site,
or how do I know someone
didn't pull that disk out,
or how do I actually know
everyone who was rude
on my systems, or you know--
control is also great
until you have some massive
data center failure,
and then it's all your fault,
too, right?
So, um...
control has pluses
and minuses.
I think that the economies
of scale that you get
from, you know, the expertise
of cloud hosting
and kind of the dynamic
scalability you get from it
make the App Engine choice
really, really compelling.
And people discover that
after they start to use it.
Greg, did you have...
Dalesandre: No, again.
I think Ben handled
the question well.
Ooh!
Oh, yeah, look at that.
Um...yeah, again,
I don't know
if that answered the question,
but there are a number of people
who essentially ease into it,
because one of the things
that, you know, it's really easy
after you've been using
App Engine for a while
to understand how there are
things that you want to tackle,
and then there are things
you want somebody else
to tackle, right?
You know, you want
to worry about--
about essentially
the useful part
of the development,
not necessarily the part
where it's not only useful
for you to sort of manage
your own machines,
it's, um, it's also--
I think, an area
that you have to
build up expertise in
that's not an important
sort of core competency
of your organization.
And so, it's only when you
sort of start getting into that
for a while that you start
to understand,
when you're running on something
like App Engine as a platform,
that you don't need to deal
with those things anymore.
It becomes just very--
you realize the advantage of it.
I don't know if that
makes sense, though. Is that--
man: Yeah, that's helpful.
Dalesandre: Okay.
man: Thanks.
all: Oh!
Dalesandre: Are you
all right?
Fried: Too much caffeine.
man: We developed lots of apps
on App Engine
for our clients
in South America,
fairly large clients
with 10,000 users.
And most of the time,
when we are previewing the work,
most of our clients
ask about security.
You know, how much secure
would the app be in their hands.
How should we address
our clients
about security
of Google App Engine?
Fried: So, I mean, I am,
as Google's CIO,
I get asked that question
a lot by people
who are interested in Google's
cloud applications
and cloud technologies,
and here's my perspective
on it, right?
Is that, um...
Google is a great place
for security people to work,
because, you know,
everyone wants
to attack us, right?
And so as a result, the best
security people in the world
come to us, right?
And we get--
there's this ec--
it's not just
an economy of scale,
it's a benefit of scale
that we get, right?
That it's, uh...
it is, you know--
and previous
to working at Google,
I worked in the financial
services industry,
where we had certain ways
we could incent really talented
people to work for us.
But even so, you know,
my experience is,
it's very, very hard,
even for great companies
that don't have this kind of
profile that Google does,
to attract the kind of security
people they need
to really be able to make
hard guarantees
about what security means
or not.
The thing about Google is that
security is so important to us,
it really is part
of our business.
We hire the best people
in the world to work on it.
They're attracted
as great people come up
that want to work here
for us at Google, right?
And they keep on
getting better.
What we keep on doing
keeps on getting better.
Security is part
of our business.
For, you know,
for other companies,
you know,
for your clients,
it's a tax to secure
their own corporate networks
and enterprises, right?
And taxes are things
people prefer not to pay.
For us,
it's a business benefit.
It's something
we need to do.
It's something
we need to be excellent at
in order to be the business
that we're in.
So my argument is that,
you know, over time, actually,
you know,
the big cloud providers
will have this incredible
advantage in security
that even great enterprises
won't be able to have,
just, you know,
given the nature
of the way
the world is working.
man: Are you guys thinking about
some sort of certification,
like SAS 70, or whatever?
Dalesandre: Um, so...
we're currently working
on SAS 70 certification.
Um, we, uh...
I believe the stage
we're in right now
is that we've, um,
we've gotten to the point
where we've satisfied
the criteria,
but then you have to do
a review later
to actually get
the certification.
So we're--the review comes,
I believe, a year later.
And so, we're
in that waiting period
after we've hit the cert--
the requirements
when you're waiting
for the review.
So, certification,
absolutely.
The other thing I would say,
just to Ben's point,
is that App Engine was built
with security in mind
from the beginning.
A lot of the limitations
that people didn't like
about, you know, time limits,
those sorts of things,
the reason we did all of that
is because
from the very beginning we knew
that we had to build something
that was sand-boxed
in such a way
that there was no risk ever
of any sort of, essentially,
security problems happening.
And we've had an incredible
track record because of that,
because we built it
from the ground up,
assuming we needed
to have security in mind.
Fried: Yeah.
man: Thank you.
woman: Hi, this
is great stuff.
I work at Chalet Tech,
and we actually are building
lots and lots of apps
on App Engine,
and we've run out of our
ten-per-user quota.
Is there--I mean,
what we really want to do
is sort of have either
an account whitelisted,
so that it could create like an
advent account or subs account.
Is that the model going forward,
or is there going to be
more flexibility on users
being able to create more apps?
Dalesandre: So we just
talked yesterday--
I don't know if you guys
want to know--
but we just
announced yesterday
that we'll be leaving preview
later on this year,
and when we do, we're
introducing a new account type,
which will allow you
to essentially get
as many apps as you want
and operational support
for $500 per month.
And I imagine that's probably
what you guys would want to do.
woman: Yes. Yeah.
Thank you.
Dalesandre: We can
talk afterwards
to get you apps
ahead of time,
you know, before
that goes online though.
woman: Okay, thank you.
man: Just a very simple
question.
Do you have any insights to get
App Engine to work in China?
McWilliams: It works
in China for Simian.
We update all of our OSX clients
that are in China with Simian,
so what restrictions
are you seeing?
man: We built a few applications
on App Engine,
and they--they work good
outside China,
but they--the access to them
are blocked in China.
Dalesandre: So, it's, um...
there might be
restrictions in place
for some of the ISPs that people
are trying to access it through.
So I don't know if we can give
a specific answer
without sort of knowing more
of the specifics around
sort of what it is
that's not working
and what you're
trying to do.
man: Well, it's--
we build...
[man speaking indistinctly]
Actually very simple.
But they are soon to be used
by general consumers in China,
and--
Fried: Maybe the thing
for us to do is kind of
have someone
from the App Engine team
go offline with you
and have them--
Dalesandre:
Yeah, I think that--
We can talk afterwards
about this.
Fried: And follow up.
Nomination.
man: Hi, I have
a short question,
or two questions.
Follow up on that 25--or 20--
$500 a month.
Is that 24/7 phone support?
Dalesandre: So it is
going to be 24/7 support.
It's ticket-based support
with a guaranteed response time
that I'm not going to quote
right now.
I believe it's eight hours in
response for operational issues,
and then if you have
developer issues,
like if you want to ask, "How
should I build X, Y, and Z,"
you'll be able to pay
per ticket
for additional
sort of developer support.
man: So that actually
involves phone?
You could actually
talk to somebody?
Dalesandre: So it's tickets,
and once with
the ticket process,
they might engage in phone calls
if that's the best way
to solve the problem,
but it's a ticketing system
is the way to engage
in the process.
man: And a different question
is, the Blobstore.
If you load the data into Python
or whatever,
how do you search--
like, do you have a SQL language
to search it, or how?
That's just
a simple question.
Schoeffler: So, essentially,
once you've
pulled it into memory,
it's just like
a list in memory
and you can do whatever
you want to with it there.
It's based on how
you deserialilze it,
so if you deserialize it into
a list or a map or whatever,
it's there in memory
and you can do whatever
you want to with it there.
man: So you could just use
Python to do it or--
Schoeffler: You'd have a Python
dictionary in memory
at that point, yeah.
man: Thank you.
man: Hi.
I was more curious
about test patterns you have.
So you would program to
anticipate possible failures
in the Cloud somehow,
or if you guys have
any specific type of suites
that you're--
uh, code suites
that you run that you
make in-house to kind of
mock up these failures
to anticipate it in your code.
Fagnani: So I do notice
that some people
use an App Engine app to monitor
other App Engine apps.
You know, obviously there are
some systematic problems there
that you could miss
that you monitor
if an app goes down.
Um, we--we've...
created some frameworks
that help us, too,
like load testing
on App Engine.
That takes care of, like,
disabling security
and authorization and stuff.
In terms of monitoring
App Engine itself,
I mean, I think that the App
Engine team has--
Dalesandre: Right.
We're actually going
into trusted tester program,
well, today
with a monitoring API
that is in fact one--
it's an App Engine app
that runs to monitor
your other App Engine apps,
if that's sort of what
you're concerned
about in terms of making sure
that things
are running appropriately
and sort of within
the parameters you want.
man: Not so much
in actual run time
as the development
process.
Say you want to--say you
want to mock-up a failure
of the Datastore and then you
want to see how your app would--
Dalesandre: So--
so we have--
Right. So we have, like,
unit-testing frameworks
for both Java and Python
that are online
if you want to
check those out.
McWilliams: Yeah.
When there's Datastore outages,
it's going to throw
a certain type of exception,
so you can certainly
write test code that will
throw that exception
and handle it in
an appropriate manner.
man: But there's no
standard methods
how you guys go about it?
Fagnani: Well, if you use--
I mean, we use dependency
injection heavily,
so, you know,
the way I typically
write an application is that
I have objects inject the
service that they want to use,
like the Memcache service
or the Datastore service.
So in your test,
there's an interface,
you can just mock it
through a failure.
man: Okay.
Fagnani: Yeah.
Fried: I'm sorry,
we're getting the high sign
that the next team
is to come in and talk,
but thank you all very much.
[applause]
