Hi, this is Steve Purcell, on behalf of Tweag.
We're a distributed software lab and consultancy
doing all kinds of interesting things at the
intersection of research and industry. You
should check us out at tweag.io.
I'm going to show you some nice tooling we've
been working on which can help you with your Python projects. It uses Nix. If you don't
know Nix, that's okay -- perhaps this will
pique your interest.
So as a starting point I have a simple Python
application, this is a Flask web application
which, when it starts up, uses the Requests
library to fetch an image from the internet,
then it uses the pillow library to resize
that image to a smaller version, and then
when you visit the /image URI, you should
get the smaller version.
Let's try running that here on my Mac.
I have a Python installed, but it's the system-wide one.
And it doesn't have the Flask module installed.
And I don't want to install these dependencies
system-wide.
What a Python programmer would normally do
at this point would be use a virtualenv, but
instead I'm going to use the Poetry tool,
which manages that for us along with resolving
dependencies and having an installable description
of the project.
I also don't have poetry installed.
I'd like to install that just for use within
this project. I'm going to create a shell.nix
file.
It has some packages which, by default, come
from the system-wide installation if they're
not passed in.
And I would like to have python and poetry
available in here.
When I run nix-shell, inside it I can see
there's a "poetry" installed, but this is
bash, and I'd really rather work from my normal
shell environment, so I'm going to use direnv
to give me that integration.
I'm going to tell direnv to use nix, and now
it uses the information from the nix environment
to populate environment variables - including
$PATH - in my host shell, such that poetry
and python3 are the ones that have been installed
via Nix.
If I pop out of this directory I'm back to
my system Python, pop back into it and I'm
there as before. So that's quite handy.
Let's create an empty poetry project.
Non-interactively, called "imgapp" to match
the python module name.
And let's add some dependencies.
Poetry creates a virtualenv for us, it creates
a pyproject.toml manifest file for the project,
and it also saves the exact versions of all
of the mutually-compatible dependencies it
resolved into a lockfile so that they can
be reinstalled later.
I'm going to tell Poetry that we have a script
called "imgapp" which is the "main" function
in the "imgapp" module.
Now I can say "poetry run imgapp" to run my
script inside the virtualenv with the dependencies
and Flask starts happily.
Let's make a request.
Hello Rick!
So that's working.
Now to be able to install this on a server
and have it co-exist happily with perhaps
other Python projects we would need to think
about writing wrapper scripts or perhaps even
making Docker containers.
Let's see how Nix can help us with that.
I'm going to make a default.nix file.
I'm going to use the poetry2nix facility,
which was created by my colleague Adam
And what this does is translate the poetry
lockfile into a set of Nix-installed packages.
Fantastic.
Now I've been given a "result" symlink to
the actual location inside the Nix store directory
where everything has been built,
and if I look at
"result/bin/imgapp",
this is a handy wrapper script which runs the program
for me.
And Nix knows about all of the system dependencies
for that as well as the Python version and
the Python libraries.
Let's run that.
And I get the same result as before.
Now, this doesn't use a virtualenv. This is
just a packaged-up script with a specific
Python interpreter that only has those necessary
packages installed into it.
The nice thing about this is that we can use
the various Nix facilities for more conveniently
working in a team or across multiple machines,
so we have access to things like binary caches
for shared builds, or we could deploy the
entire dependency tree you just saw to remote
machines using NixOps and then switch between
one of those trees and another atomically
as you upgrade your application.
But we can also make a Docker image. I'm going
to make a different version now of the default.nix
and I have one that I saved from earlier.
This is slightly more complicated.
Here I use an overlay to add our program - using
mkPoetryApplication as we saw before - to
the overall set of packages that are known
to Nix.
I have one particular set of packages here,
and then I have another set of packages for
which I've overridden the target system to
be Linux.
Now remember I'm on a Mac here and if I ask
Nix to build me a Docker image I would really
like it to contain Linux binaries and not
Darwin / Mac binaries.
So, to do that, I can use the dockerTools
package and, using the host packages, I'm
going to build a script which when I run it
will produce the Docker image, and inside
the Docker image, I would like there to be
the linuxPkgs build of my new custom
Flask application package.
So now if I say "nix build" the attribute
"docker" from this set of two attributes over
here, let's see what happens.
I didn't save it.
Now some magic happens and our build is shipped
off to a machine that knows how to build for
Linux.
This uses the brand new "nixbuild.net" service
which you should definitely check out if you're interested in Nix.
Now the latest build result is a script which, when you run it, copies the various remotely-built
Linux build artifacts on-the-fly into a Docker
image which it then outputs on standard output.
I can load that into my local Docker daemon
using "docker load".
We even have multiple layers to get some degree
of re-use.
Now let's run that Docker image.
I'm going to expose its port.
And there is our Linux binary running inside
Docker on my Mac.
Let's try making that request one more time.
And there we go.
So I hope that's been an interesting demo
of some of the handy things that you can do
using Nix. I definitely encourage you to check
out more information at Nixos.org, and of
course don't forget to visit the Tweag website
and read some of the articles about what we've been doing
Thanks very much for checking in!
