(uptempo electronic music)
(train horn)
- [Voiceover] Hi everyone.
My name is Konstantin Tennhard,
and I have to warn you,
this is not going to be a funny talk.
I'm German, we don't do that.
(audience laughter)
But I actually don't
live in Germany anymore,
last year I moved to Ottawa, Canada,
and there's so many Ruby
developers in Ottawa.
I actually work for Shopify,
and, in fact, we have
so many Ruby developers
that we just don't know where to put them
and we sent them all down to Kansas
to speak at RailsConf.
And, so, Ed already gave a talk,
and there's two of us speaking
later in the afternoon
about how we test and about sprockets.
To continue with the
shameless self-promotion,
you can find me on Twitter and GitHub.
My handle is @t6d.
And, in my spare time, I'm
maintaining a couple of libraries
that you might find interesting.
One of them is Action Widgets,
which is a UI component
micro framework, I'd say,
for Ruby on Rails.
In fact, the slides you see here on screen
are powered by action widgets.
I'm maintaining Smart Properties,
which is supercharged
Ruby attribute accessors,
as well as processing
pipeline for Ruby on Rails
to model complex business processes.
And finally, the one I
wanna talk today about,
Request Interceptor, which
is my most recent one.
And it allows you to simulate
Foreign APIs with Sinatra.
So, at its core, this
talk is all about testing,
and specifically, one type of test,
tests that involve HTTP connections.
So most of you might know the
libraries VCR and Webmock,
which are usually used to
stamp out individual requests,
or in case of VCR, replay requests
that have previously
been sent to remote API.
I want to present a
different approach today
and talk about how we can
use Sinatra to simulate
a foreign API within our test suite.
And that is essentially the core idea
behind request interceptor.
So, I guess I best show you
how to use the library first
and then, throughout the talk,
we dive deeper and deeper
into how it actually works internally,
to the point where I'll show
what kind of meta-programming
techniques I use
to hook into the net HTTP library
to make all that magic happen.
So, yeah, I just mentioned it,
Request interceptor does modify net HTTP,
just like Webmock and VCR.
There's no clean way to
sort of interject yourself
into what net HTTP does,
so some trickery is
required to make that work.
But I get back to that later.
The idea is that you can
use any Rack compatible app,
and use it as a request interceptor
that sort of intercepts an HTTP request
sent out by your
application and reroutes it
to your Rack app, which will then
handle the request in line.
And, in fact, all you
need to know, essentially,
is that request interceptor
implements run method,
which takes a Rack application
as well as a host name pattern.
So the host name pattern is important
to know when request interceptor sort of
starts intercepting requests.
It will actually look at the HTTP requests
and only redirect the
requests to your own Rack app
if it matches the host name.
Otherwise the HTTP request
will be made just as a
regular remote request.
And in the code example
here, I define the probably
most minimal Rack app you
could potentially implement.
It's simply a Lambda
statement that returns
an array with a status code,
no headers and a message, Hello!
And then I use Request
interceptor to intercept
all requests that go to
anything that ends with
any hosted ends with example outcome.
I do my HTTP request and
then assert on the equality
of the response being, Hello!
The problem with bare metal Rack apps
is that they are very inconvenient.
To sort of implement something
more feature complete,
you wouldn't necessarily want
to go with Rack directly,
instead you want to pick
something that has a little more,
that provides you with a
little more convenience.
And for me, this convenience
is sort of given by Sinatra,
which sort of combines
simplicity as well as
provides you with a lot of flexibility
on how to simulate these API endpoints.
And for those of you
who don't know Sinatra,
it is a Ruby micro web framework,
and it's based around a very simple idea.
You have a Sinatra application
that provides you with
more or less, well,
the most important methods are
'get,' 'post,''put, and 'delete,'
which correspond to the HTTP methods
and they allow you to
define request handlers
in your Sinatra application.
So they take a path as the first argument
and then a block and the block defines
how requests are being handled.
So a simple Sinatra app
looks something like that.
You don't even need to wrap
it in a class or anything.
It provides you with some
magic to make this work,
and you require the library,
you define that your
application is handling
anything that comes into slash hello,
and in this case it
returns 'Hello Sinatra!'
So given this conciseness
and this simplicity,
Sinatra was an excellent
choice to sort of model APIs,
and therefore makes a great combination
with request interceptor.
In fact, I went further because
of this great combination,
it's the combination
that I would suggest for you to use
instead of using Requetinterceptor.run .
With just any Rack app, I
would recommend using Sinatra
and request interceptor
gives you a define method,
which allows you to define
a new Sinatra application
with some extra goodness.
So request interceptor
allows you to define
the host name pattern.
Again, just as we've seen before,
where we submit the host name
pattern and the application
to the run method,
we now define it right on the application
and then we just define it
as a regular Sinatra app
with all of our endpoints that we need.
And the result of this
define call is a class again,
which is a Sinatra application
with the added benefits,
and one of those benefits is that
this application provides
you with an intercept method.
And the intercept method is just
a convenient wrapper for you around run.
So instead of having to pass in everywhere
where you want to use an interceptor,
remember which host name you want to match
and which application to pass,
you can just call intercept
on your interceptor
provided with a block,
and then, again, fire off an HTTP request
and assert that the correct
message is returned.
And then, more importantly,
in order to test this,
you probably want to know
how many requests you made,
which requests you actually made,
and what the request
and response data was.
And, to make this possible,
the intercept method
returns a transaction lock.
So it's simply an array
of request interceptor transactions,
and these transactions are simply structs
which give you access to
the request and the response
that was made within the block.
And these are instances of
Rack: Mock Request and
Rack: Mock Response.
Just as other libraries
usually use for testing,
Rack applications, I essentially use these
to carry all the data
for further inspection.
And then the example down below shows you
how you can, for instance,
assert on the path of the
first transaction log entry.
And in this case I'm just asserting
that my program called
the path 'hello' of example.com.
You can also nest them
in case you want to have,
you communicate with multiple APIs,
and at Shopify, I was on the team
that implemented Uber Rush integration.
We did that as a separate app,
so for us, Shopify was, we
also treated Shopify as an API,
just as you would if you
develop an app for Shopify.
And then we treated Uber
as our other service,
so our application was,
actually, had to communicate
with both of these services.
And it's often necessary
that you know exactly
which requests were sent where,
and that is why Request
interceptors do support nesting.
So both of these interceptors
write a separate transactional log.
And, yes, of course the
innermost interceptor
takes precedence so if you,
you can actually have two interceptors
responding to the same domain,
or to the same host,
in which case, the innermost would win
and intercept the request.
Another important feature is
that you can customize an interceptor
for an individual test,
because the idea is that you
generally outline your service
that you are modeling in a single file.
And then, customize it to certain behavior
to fit sort of the needs of your test.
Let's say you want to
model an error response
for one particular endpoint.
You would take your interceptor
call the .customize method on it,
and then override the
previously defined endpoint.
And, Sinatra is smart enough
that if you redefine an endpoint,
it, the new endpoint will take
precedence over the old one.
And in this case, we're just
switching the hello endpoint
to send another message.
Previously, it was 'Hi, RailsConf!'
and now it's 'Bonjour, RailsConf!'
So now that you have a basic understanding
on how they work, I want to talk a little
about the advantages in
comparison to VCR and Webmock
that I think exist when
using request interceptors.
For me, one of the
biggest advantages is that
the code isn't cluttered
throughout your test suite.
Instead, what we do, is we have one file
that defines a particular service,
in our case, Uber or
Shopify, that implements
all the endpoints we are
usually communicating with.
And then, we customize this interceptor
to specific needs in our test suite,
but if you sort of want to
see in one go what your app
is actually communicating
with, you would just
open the file and look at
the interceptor definition.
Another advantage for me is
that interceptors provide
greater power and flexibility because
we're talking about a Sinatra application.
You can literally go as
far as you want with that.
You could have, theoretically,
in a memory database
that sort of keeps state,
if you wanna simulate entire workflows
or you can keep it super simple
and return static responses
from your endpoints, so
it's really up to you.
Then, of course, since it's
essentially just one file,
you can also go further and package it
into a Ruby gem, let's say.
You build a surface that
other developers use
and you have a public API and now you want
to make it easier for people
to sort of integrate your
service, you could provide them
with a predefined interceptor
they can use in the test suite,
so they don't even think
about hitting your API
with, like, requests
from their test suite.
And then finally,
and that is, personally,
for me, super important,
is that the code is just very readable,
which is in the nature of
the Sinatra application.
And I personally think it's more readable
than having these Webmock stubs
sort of scattered around your test suite.
Instead you have this
one single application
that defines how your interceptor works.
And then there's more, there's features
that I am not sure
if you could simulate
them with Webmock or VCR,
and so, I wanna talk a little bit about
more advanced concepts on how
to use these interceptors.
A big one for me is
simulating network requests.
Request interceptors
are set up in a way that
they propagate errors or exceptions
that are being raised
in one of the endpoints.
So, I specifically disabled
seen address functionality
to handle exceptions
and propagate them
through the entire stack,
which allows, for instance, to simulate
that a host is unreachable
simply by raising the
appropriate exception,
which makes it very easy
to test your application
or the library you are building,
whether it's robust enough
to handle these error cases.
And then of course, Sinatra gives you
a lot of tools that you can leverage
to make interceptor definition even easier
and make the code more readable.
And one of the most
important things is probably
that, being a standard Ruby class,
you can just define private helper methods
that you can use
throughout your interceptor
and throughout the customizations
you use in your test suite.
In fact, you can just apply
standard object orient
design principles, too,
and all that Ruby gives you to sort of
make your interceptors as readable
and as easy to use as possible.
Then there is the possibility
of using Sinatra's before
and after callbacks
that run before or after a
particular endpoint is hit.
And you could, for instance,
utilize an after callback
to automatically encode data into JSON,
let's say you are modeling a JSON API.
It's tedious if in any endpoint,
you always have to remember
that you, as a last step,
have to call to JSON on whatever
you're sending over the wire.
So just define it once in a block
and, in this case, I look at the response,
and if it's an array or
hash, I encode it into JSON.
And then of course you have the ability
to use Rack Middleware, and in this case,
we model both Shopify
and Uber interceptors
as API, as JSON APIs.
So we always wanted to
decode the incoming JSON
so we can easily work with
that in our interceptors,
and Sinatra provides you
with a method called use
that allows you to inject Rack Middleware
that runs before your
actual endpoint is hit.
Now that you have sort of an understanding
on how you use interceptors and
why they might provide a nice
alternative to VCR or Webmock,
I actually wanna dive deeper
into some of the internals,
because I just think
it's interesting to see
some of the powerful
features Ruby provides
and just as a, sort of, learning exercise.
So, in the beginning of the talk,
I showed you that Requestinterceptor.run
is sort of the core
of the whole idea.
And, in fact, this is the
concrete method implementation
as it exists in the library,
and there's essentially six steps
and I will go over all of these six steps
to sort of showcase how you can mess with
an existing Ruby library
that doesn't provide you
with the ability to sort
of do this in a clean way.
So the first step, is because
you can reuse an interceptor,
is to clear the transaction log.
That's very easy, I just cleared the array
that keeps all the transaction entries
from the previous run,
and then I cache the
original net HTTP methods
because we have to make sure that
once the block finished its execution,
we restore net HTTP to
its default behavior.
And then I override the net HTTP methods
with a custom implementation
just as Webmock does as well,
and then I execute my test,
and now my test will essentially use
these overridden net HTTP methods.
And then finally, I
collect my transactions
and then eventually restore
net HTTP to its former glory.
And, the last part
happens in an ensure part,
so it's always guaranteed to run,
and, so that it doesn't
happen that your test suite
actually gets into a state
where net HTTP is not
in its original state.
So, as I said, it's easy to
clear the transaction log,
so I just wanna skip that
and talk about caching
the original methods.
There's three methods you need to override
if you wanna do something
like incepting HTTP requests.
There's start, finish and request.
Start and finish sort of take care of
opening the TCP connection,
and then request performs
the actual heavy lifting.
And the way caching works
in request interceptor,
you have now a concrete
request interceptor instance
at your hand that is currently
handling your test case,
and I just assign these
methods to instance variables.
And what instance method
gives me is an unbound method,
so I essentially save the
original method implementation
and just put them, for now,
in an instance variable.
And then I replace these three methods
with my own implementation.
Start and finish are pretty boring,
I just make sure that
net HTTP thinks it has
an open TCP connection
it is communicating with,
but in fact, I don't need one because
of how they redirect to the
Sinatra application is working
and I'll show that in a second.
And then I define a new request method,
which is a little more interesting.
The interceptor instance itself
is currently handling your test case,
has a request method of its own.
And all I really do is I take the data
that would usually go
to net HTTP requests,
and redirect it to my interceptor,
and then I also pass in
the interceptor itself.
I won't show the code for
request interceptor request
because it's a little more complex,
but I at least wanna
explain what is going on,
and you can always take
a look at the source code
if you're interested.
So the first thing I do is
I try to find an appropriate interceptor,
meaning I look at the HTTP request
and then look at the host of this request,
and now go through my
list of host name patterns
and stored applications
and see if one matches.
If I find one, I now build a mock request
and mock request, the
initializer of mock request,
takes an Rack application
as its first argument.
Once I have that mock request initialized,
I can call the methods
get, post, put, delete
on them to simulate an
actual HTTP transaction.
And, once that happened, I
get back a mock response,
which I now have to transform
into a net HTTP response
to make net HTTP believe
that it actually just talked
to a remote service.
And then I log the transaction,
meaning now I'm taking
the mock request and the mock response
and just writing them
in my transaction log
so they can be further
analyzed in a test suite.
The interesting thing is what
happens if no interceptor
actually matches your host name,
because I wanted to implement
it in an unobtrusive way,
I didn't want it to block
just any HTTP communication,
especially to be still
compatible with Webmock and VCR.
So, what happens is,
my current net HTTP instance,
which is now in this weird state
that it talks to the Sinatra application
has to be restored to actually be able
and perform network requests.
And the way I did this is
shown on the next slide,
but once I restored it, I
essentially performed the request
as if there would never have been
any interceptors in the way.
And, method.
Restoring works by utilizing
Ruby's defined method,
we actually can not just take a block
but it can also take an unbound method.
So the ones we previously
stored in instance variables,
we can now rebind to net HTTP
and we can even rebind them
to concrete instances of net HTTP,
and it is sort of happening
when the request interceptor
doesn't find the matching application,
it rebinds the original methods to the
concrete net HTTP request
that's currently going on,
and then just calls request again,
and performs the request as
if nothing ever happened.
So that was essentially the internals
of how the request cycle
works in request interceptor,
and if you can compare that to Webmock
there's certainly similarities
with the difference
that you define a step within your test
and in this case, I redirect
to the Sinatra application.
I previously mentioned that
there's error propagation
that you can utilize to sort
of simulate network errors.
I just wanted to quickly
show how this works,
it is very simple because
Sinatra supports it
by just using particular
configuration statements,
so all you need to do to sort
of have a Sinatra application
actually raise an exception
and not handle it,
and have the calling code
take care of that exception,
is you disable show exceptions
and you enable raise errors,
and by that, you sort of switch Sinatra
into an aggressive mode,
which does not make sense
if Sinatra runs as your
production application,
but it makes a lot of
sense to sort of simulate
these network request errors.
Well, I do have further plans
for request interceptor.
So one thing I wanna implement
is these supportive traits,
sort of similarly like the
factory grow mechanism,
where you can define what
your factory is building
and then give it a certain trait
of how it is actually building
and I want that for interceptors as well
because I was running into the
issue that I was simulating
the same endpoints several times.
And, what I did so far was
just having a lot of these
customized request interceptors,
but what I actually want is
just in a particular test case,
I wanna have a name where I can refer to
an endpoint definition and say
I want my interceptor to run
with a faulty implementation of my hello,
and the faulty implementation
could either be
raising a 500 or raising a network error.
And, I wanna support different adapters,
so I don't want to just stop at net HTTP,
the next thing I wanna
implement would be Faraday,
because Faraday would give me exposure
to several other libraries
because I don't really wanna do,
mess around with each of
these libraries individual.
That is sort of the two goals
I have in mind right now
to bring this library forward.
And that basically brings
me to the end of my talk,
and I just wanna quickly summarize
what I've been talking about.
So request interceptors sort of provide
a third alternative to VCR and Webmock.
The thing I like most of them
is that I have a concise
service definition in one place,
instead of scattering this definition
across the entire test suite,
and they provide me with an easy mechanism
to customize them if
there is the requirement
in a certain test.
And then finally, Sinatra provides me
with a lot of simplicity and flexibility,
which ultimately leads
to very readable code,
which is just something I greatly enjoy.
If you are interested to take
a look at the slides again,
because I know it was a lot
of content I was going over,
they are available online.
Thanks a lot for your attention.
(audience applause)
(uptempo electronic music)
(train horn blowing)
