In building your Python application and its dependencies for production, you want to make sure that your builds are predictable and deterministic. Therefore, always pin your dependencies.
Update: A newer blog post about the future of pip-tools is available too: Better Package Management.
Pin Explicitly ¶
Don’t ever use these styles in
Instead, pin them:
If you don’t, you can never know what you’ll get when you run
Even if you rebuild the env every time, you still can’t predict it. The outcome
relies on a) what’s currently installed, and b) what’s the current version on
Eventually, all of your environments, and those of your team members, will run
out of sync. Worse even, this cannot be fixed by rerunning
pip install. It’s
just waiting for bad things to happen in production.
The only way of making your builds deterministic, is if you pin every single package dependency (even the dependency’s dependencies).
WARNING: Don’t pin by default when you’re building libraries! Only use pinning for end products.
The biggest complaint from folks regarding explicit pinning is that you won’t benefit from updates that way. Well, yes, you won’t. But think of it. It’s impossible to distinguish between a new release that fixes bugs, or one that introduced them. You are leaving it up to coincidence. There is only one way to retake control: pin every dependency.
Check for Updates Automatically ¶
So: we want to pin packages, but don’t want to let them become outdated. The solution: use a tool that can check for updates. This is exactly what I built pip-tools for.
pip-tools is the collective name for two tools:
It will check for available updates of all packages currently installed in your environment, and report about them when available:
$ pip-review requests==0.14.0 available (you have 0.13.2) redis==2.6.2 available (you have 2.4.9) rq==0.3.2 available (you have 0.3.0)
You can also install them automatically:
$ pip-review --auto ...
or interactively decide whether you want to install each package:
$ pip-review --interactive requests==0.14.0 available (you have 0.13.2) Upgrade now? [Y]es, [N]o, [A]ll, [Q]uit y ... redis==2.6.2 available (you have 2.4.9) Upgrade now? [Y]es, [N]o, [A]ll, [Q]uit n rq==0.3.2 available (you have 0.3.0) Upgrade now? [Y]es, [N]o, [A]ll, [Q]uit y ...
It’s advisable to pick a fixed schedule to run
pip-review. For example, every
monday during a weekly standup meeting with your engineering team. Make it
a point on the agenda. You discuss
pip-review’s output, inspect changelogs,
or just blindly upgrade them. The important part is that you do it explicitly.
You have the chance to run with the upgraded versions for a while in
a development environment, before pushing those versions to production.
pip-review solves the problem of how to check for updates of pinned
pip-dump focuses on the problem of how to dump those definitions
into requirements files, managed under version control.
Typically, in Python apps, you include a
requirements.txt file in the root of
your project directory, and you run
pip freeze > requirements.txt
periodically. While this works for simple projects, this doesn’t scale. Some
packages are installed for development, or personal, purposes only and you
don’t want to include those in
requirements.txt, going to production, or
visible to your other team members.
pip-dump provides a smarter way to dump requirements. It understands the
convention of separating requirements into multiple files, following the naming
requirements.txtis the main (and default) requirements file;
test-requirements.txt, or actually,
*requirements.txt, are secundary dependencies.
When you have a
dev-requirements.txt file in your
project, with the following content:
# requirements.txt Flask==0.9 # dev-requirements.txt ipython
Then simply running
pip-dump will result in the following output:
# requirements.txt Flask==0.9 Jinja2==2.6 Werkzeug==0.8.3 # dev-requirements.txt ipython==0.13
It keeps the files sorted for tidiness, and to reduce the chance of merge conflicts in version control.
You can even put packages in an optional file called
.pipignore. This is
useful if you want to keep some packages installed in your local environment,
but don’t want to have them reflected in your requirements files.
pip-tools 0.x is relied on by many already on a daily/weekly basis. It’s
worth noting that we’re working on Better Package Management too,
which will be the future of
pip-tools. If you want to contribute, please