Poetry: "Dependency management and packaging
made easy"
Samuel Roeca
>> I'll wait one more minute to make sure
people who are filtering in get a chance to
make it.
Welcome everyone.
My talk today is called poetry, dependence
and management package made easy.
If you've gone to the poetry code base you'll
notice I completely plagiarized the tag line
of the code base itself.
So go there, it's pretty self explanatory.
Today I'm going to tell you a brief story
how I found my way to the world of poetry
and I'll go over some jargon, and a bit of
history of dependency management and packaging.
Finally I'll show you features of poetry and
through slides, show some examples.
If there's some extra time I'm going to do
my best and try to do some live examples of
Poetry, with all the dragons and dangers entail
at a live demo and hopefully a bit of time
for questions either afterwards or at the
end.
So me, my name is Samuel Roeca.
That's my git picture.
My name is papa Sam.
I'm one of those people whose X box user name
has followed them longer than it should have.
I'm a happy user of Linux.
I love to tinker I spend most of my time in
vim and some of you should too.
I run the New York City meet-up.
I'm also the head of engineering at a digital
marketing agency called Keppler group.
We're building some cool internal code bases
to help differentiate us in the marketplace
and that's also a picture of me back home
with a colleague.
I'm originally from Hawaii.
So if I seem a bit off, maybe that's why.
They're ear all normal, so maybe it's just
me.
Some so examples of Python that I've written
and deployed to GitHub and PyPi, I wrote a
library called latex build.
And then fast forward into 2019 there was
a dark period where I didn't do much.
I wrote a library called TOML-sort.
It's on PyPI.
I 
wrote my own language server called Jedi language
server.
There's a more famous one called py LS.
I tried to write my own.
And finally this presentation is written in
a framework called hover craft and deployed
or run with poetry.
So the three projects at the end use poetry.
The first one uses setup tools and I'll talk
about how I made that transition.
So here's story time.
We had a problem at my company.
In late October 2018, some users reached out
to us and said we can't get in to the application.
We're like what happened?
We didn't change any code.
But people were logging in and immediately
being redirected to the authentication microservice.
And this was a problem.
It wasn't a security problem because nobody
could get in, but it was still a problem.
So we wracked our brains and we looked at
our requirements.TXT and it looks kind of
like this.
This is an example.
Maybe we've got to dig deeper so we went to
flask and read its source code and saw it
had these version constraints.
And we're looking around here and okay, that
might be something to look at.
It's called it's dangerous.
Maybe that's where the problem lies.
And it turns out, it's dangerous had just
reached version 1.0.
It's hilarious.
The default algorithm for signing cookies
had changed.
So people -- all we had done is redeploy our
authorization micro service.
We hadn't changed any code.
But because a dependency of a dependency had
changed the latest version was installed,
and the signing algorithm differed between
our micro service and the rest of our services,
which were using a different algorithm.
And nobody could get in.
And that was a really fun journey.
Wait as a great way for an organization and
an individual to discover the importance of
dependency management.
And we considered a lot of alternatives at
this time and ended upset willing on poetry.
Which had just come out recently.
The author of poetry's name is Sebastien.
I don't know how to pronounce his last name,
he hasn't told me.
His GitHub name is sdispater.
I did not write poetry.
I've participated in the discussion but I
have not written this library.
I'm just a huge fan.
I'm going to use a lot of words.
Python we have a module, and when I say module
I mean a file that contains Python code.
A package in the context of Python and in
the context of this talk is a directory of
Python modules.
You can see this area we have my package with
two modules inside of it.
Finally a distribution is an archived module
or package can be in tar format, can be in
zip format.
Take al your assets and smush them up into
one thing.
That would be an example of a distribution.
So when I use those words, that's what I mean.
Dependency management.
This isn't quite as well defined in the Python
community so I'll use Wikipedia's definition.
It's the process of installing, upgrading,
configuring and removing insert my own, Python
distributions, used by your project in a consistent
manner.
So my project depends on mush mash tar files
living in high PI.
Those would be my dependencies.
Managing those dependencies for my project
would be dependency management.
Cool?
Packaging.
It's a new concept.
It's the process of creating and uploading
a Python distribution to be used by yourself
and others.
So taking my code, mushing it into a distribution
intended for distribution.
That would be packaging.
Sort of like this person giving a package
to the other one.
A little bit of history.
These words, packaging and dependency management
haven't been very separated in the Python
community historically.
And we'll maybe see why.
I refer to these in periods of three.
We have antiquity and middle ages and modernity.
In the period from 1990 to about 2008 we had
a lot of stuff happening.
1990 we probably packaged and distributed
things through floppy disks or other techniques.
And this tool came out in 2000 it's still
around in the standard library.
PyPI was created, I got a lot of these facts
from inside the cheese shop so if you're curious
about a deeper dive into some of these details,
check him out.
Then setup tools.
The tool that I use to build lay tech build
came out.
Virtual end helps us sandbox our dependencies
from our operating system dependencies.
Then we have PIP, this tool we all know and
love.
A developer who was tired of people complaining
built PIP and that's how you came to life.
You can read his blog post.
Then we get to the middle ages.
There was a rift between 2008 and 2011.
The maintainer of PIP kind of step add way
and the Python packaging authority was born
in 2011 and I think that was kind of taking
this period of anarchy and trying to protect
ourselves from chaos and then we got to Conda
in 2012.
PIP tools, py builder, two tools I'm not familiar
with but I know they exist, came out around
that time so they're fairly old.
The wheel format, this is just a package code
came out in 2013 then we had semantic versioning,
we had PyPi standards, a lot of stuff happening
in the middle ages.
I think we reached the period of modernity
around 2016 when we're starting to formalize
some of these standards in a way that is not
too tool specific.
Gives us a lot of flexible.
This py project.TOML along with independent
builds were proposed and this just blew the
roof off.
Anything was possible at this point.
So 2017, kind of unrelated to those first
two sections, pipenv was created.
It's a tool that's on the official tutorial
-- this is a politically charged topic these
days so I'm not going to touch it.
I'm going to talk about Poetry.
I just love it.
Give it a chance.
Poetry was created in 2018 on the heels of
pipenv.
Some messages were sent.
PIP there were hella changes made in 2018
-- there were 10 releases through August 2,
5201.
In 2019, six in 2018 and zero in 2016.
When I say hella changes, I mean hella changes.
It's PyGotham today.
So here we are.
So here's Poetry in comparison to the rest
of ecosystem.
This is not to say that poetry is the only
tool that can do both dependency management
and packaging.
I am simply arguing that it's the only tool
that does them both really, really well.
On the left we have setup tools, twine, set-up
configuration manifest.
On this side we have tools you're familiar
with, poetry sitting smack in the middle.
That's that.
So as I've already pointed out, poetry is
feature rich, it does a lot.
Don't read all of this, just know there's
a lot of things it does.
It adds things, removes them, publishes them.
Poetry also exposes a familiar interface.
And this is the reason I was attracted to
it at first.
I have been experimenting a lot with other
languages.
I really like RUST, it's got a cool package
manager, dependency manager called Cargo and
I guess the creator of Poetry liked RUST as
well because they resemble each other a bit.
And he says that.
He's like it's a good interface.
We should stick with it.
And similarly Yarn, a couple things missing
by default but we have the ability to add
packages, the ability to publish them, to
upgrade, a familiar interface.
And there was a talk, the keynote yesterday,
attracting people from other communities,
trying to build an easier community to contribute
to.
I think similarities to tools across the programming
ecosystem is important.
So dependency management.
I label this a walk in the park, because this
is a problem that's being solved by a lot
of tools.
I'm trivializing the problem.
It's harder than I'm making it sound.
We've got pipenv here is an example.
In order to manage dependencies with pipenv,
we run this command, we install our dependency,
install a development dependency, and, look.
We have our dependencies managed.
Let's not go into too much detail about how
that works.
Side by side with poetry.
You initialize poetry, you add flask, you
get something similar.
The point is multiple tools solve this problem.
Dependency management is a problem that is
currently being solved in many places.
Dependency management is not the only problem.
Let's think about packaging.
Or taking our distribution, sending it somewhere,
making it available to others.
This is a little more daunting at least to
me when I was exploring tools in the Python
ecosystem.
Let's do a warm-up.
We're going to use a distributed package.
We're not going to build one, we're just going
to use it.
That's all have you to do.
PIP install and we're done.
Hurray!
All right.
Now for the real challenge.
Let's make a distributable package.
Okay.
That's fun.
I'm sure there's plenty of things I need to
do and oh, my God I have to think about all
these things.
I have to do product scaffolding, manage the
version, control who can publish versions.
There's a lot here don't read it all.
I go to the Python packaging authority official
tutorial for how to package Python code.
Step one, manually create your project structure.
Do that.
Follow the tutorial.
Step 2.
Manually write a setup.py.
Kind of looks like this.
This is one of the simplest versions you'll
see.
Step 3.
Build distribution with setup tools.
So we get these tools, they might already
be installed, I'm just adding this here to
make it seem more complex to serve my argument.
So Python, you execute setup.py with two arguments
and in the distribution folder we have these
guys.
Now we'll use that tool twine to upload it.
So we've got to get a new tool it uploads
to pypi.
And of course there's a billion tools time
to study.
You get into a study group, spend a couple
weekends on it, and then you take the blue
pill.
You're like I published latex build, that's
enough.
I'll take three years off.
That's what most developers find themselves
in.
Now let's cover the packaging tutorial with
Poetry.
We wrote a package!
Okay.
I'll break it down for you.
Step one project scaffolding.
We don't have a structure.
Now Poetry knew my package, we do have a structure.
Poetry comes with project scaffolding as part
of the project, much like Rust's cargo.
This is great.
Step 2, dependency management.
That whole project that's a walk in the park.
We'll add a dependency and it will install
the dependency along with all of its subdependencies
and I'll get into lock files later.
Edit software.
So at the end of the day we're still going
to have to edit our software.
Poetry doesn't do that for us yet, but I think
with some artificial intelligence machine
learning who knows, we'll edit the stuff and
use requests.
Finally we'll build our distribution.
It builds.
And we'll publish our distribution.
We'll share it with the world.
Publish this to PyPi but if you have a package
index you want to share it with as well, that's
how it works.
So that's it.
That's the steps broken down.
What I want to do through now is a bit deeper
dive into dependency management.
Because what I showed before was setup.py
I kind of made a joke about it, but this is
the form of dependency management with poetry.
When you think about your project, you have
some metadata associated with it.
Again, this is in the py project TOML which
is a Python PEP product.
And we have tool.poetry along with your project
name, its version, obviously it's a great
project.
And then the version of Python that your project
is supposed to run on.
That's kind of cool.
It's baked into the expression itself.
We also have dependencies that are required
for your project to run.
So in this case flask.
It's not the same example that I showed before.
That was requests.
Down here we have development dependencies
or things that I need while I actually develop
the project.
I don't want to install those when I deploy
what I'm using here to a server, I want to
have these dependencies when I develop.
And down at the bottom we have tool.poetry.scripts.
Which gives me and we can get into this later
and this is really cool.
None of this is terribly different from a
super well-written setup.py but it's structured.
It's easier to read.
Semantic version.
If you've worked with package.JSON you've
become familiar with patch releases you get
a squiggly up line.
Minor release ranges, you get a caret.
For those of you who aren't familiar with
semantic version, it breaks down a project
version into three numbers separated by dots.
First number, would be 8.
That represents the major version.
If you're going to make a breaking change
to your library, you will bump the major version
because your users will probably be sad if
they changed versions.
Version 9 would be a minor version.
So if you're making updates that aren't breaking,
you'll bump that features.
And patch releases are for bug fixes.
So something is wrong, I'll bump the patch.
All of these are backwards compatible.
These came out in PEP440 some time in the
history of Python.
We don't really support these by default.
Poetry gives you some nice syntactic sugar.
I mentioned dependencies of dependencies.
Poetry has this file which specifies the exact
version of a dependency that's been installed
for an application.
That's important we no longer get hit by the
it's dangerous situation even if we're not
explicitly referencing the version in our
pyproject.toml.
It does all sorts of cool things.
So at the end of the day why use poetry.
And I thought I would leaf it up to Joseph
brod ski.
He said poetry amounts to arranging words
with the most effective and externally inevitable
sequence.
He was talking about poetry itself, which
this kind of translates well to the project,
because I think poetry has a slack, familiar
interface, that makes developing Python packages
and applications easy.
And that's great.
Poetry is not perfect.
It's still being perfected.
I think of this as kind of -- this is the
environment before something like poetry.
We had the penny far thing.
Now we have this bicycle.
It's not a perfect bicycle by any means.
It could always be a better bicycle.
We may have a slightly different design, it
might be called something else.
But I think poetry resembles that thing that
we need to get to.
A tool that bridges the gap between applications
and libraries, makes dependency management
and packaging easy and makes everyone want
to hop on board.
So please join us in the land of poetry.
And what I'll try to do, I think I have a
little bit of time, is show a quick example
in my terminal of how to actually 
quickly make a Python package.
Now we'll go ahead and add requests.
It created a virtual environment.
That's nice.
A lot of these features aren't terribly different
from pipenv.
Then we're going to build a package.
We have a wheel.
You know what?
Screw it.
You know what PyPi needs right now?
It needs my package.
Here you go PyPi.
I won't do that.
We can't be bad members of the community.
That would be polluting PyPi and that's something
we have to think about.
Does poetry make packaging too easy.
But that's a great problem to have.
So hopefully we all want to use poetry now
because I think it's awesome.
And that's all for my content.
Thank you very much for listening.
[ Applause ] we have time for questions.
[Off microphone]
As far as continuous integration goes, we
documentize all of our packages.
All of this runs in docker.
The lock file is generated outside of docker.
We'll have our lock file added to our version
control system.
Then we build our docker image.
The docker build will be referencing the lock
file itself.
So our dependencies are frozen in the context
of an application.
When we want to make up dates to the lock
file, that will be done outside of our integration
system.
That's manual.
[Off microphone]
No.
Maybe I should but I don't know what those
are.
Thank you very much for that question.
And any more questions I would be happy to
take outside.
[ Applause ]
