Back around 2010 I was manually managing production-critical CPython installations, compiled in special ways. I learned a lot about how to properly isolate Python installations from each other. I also learned how important it is to never confuse a “system Python” (as many Linux distributions have one) with a Python build that you can be in control of. Since then I have never touched a system Python anymore (it is the system’s Python after all, and none of my business).

For developing, testing, and running my software, I needed to manage custom Python installations. That involved building CPython from source, many times. At some point around 2015 I adopted pyenv for doing exactly that. Back then, I also adopted pyenv-virtualenv for managing so-called “virtual environments” derived from the individual Python builds managed by pyenv.

I come to say: this method has been a joy to use over the last years. It has never let me down; it has always provided me with simplicity, reliability, a predictable outcome based on which serious engineering work could be done. I have adopted it for continuous integration pipelines; I use it for building container images, towards more reproducible test environments. It always works.

I want to share the boring stuff I did today with pyenv/pyenv-virtualenv, as I have done it for years, always with a smile, thinking and appreciating “that just worked as expected”.

Today, for a program I am developing right now, I wanted to set up a CPython 3.6.9 build, with a virtual environment derived from it to install specific dependencies into it. I started with

cd ~/.pyenv && git pull

for updating my local pyenv installation to learn about current Python releases (I have been using the same pyenv installation for years, migrated it between development machines). I issued the following command for pyenv to download a CPython 3.6.9 source archive, and to build it:

pyenv install 3.6.9

I found that the resulting build didn’t provide the lzma module because I didn’t have the corresponding header files available on my Fedora installation. I installed the relevant header files with a

dnf install xz-devel

and simply repeated

pyenv install 3.6.9

This resulted in a CPython build including the lzma module. I then went ahead and derived a virtual environment from that build, for the specific purpose of developing my current pet project:

pyenv virtualenv 3.6.9 venv369-goeffel

I then activated said virtual environment with

pyenv activate venv369-goeffel

and started to use it for specific development work.

Not spectacular, right? That’s it, precisely. The power here comes from the simplicity and reliability. A big thank you goes out to the pyenv maintainers.

I think the following shows that I should start deleting old things, but I think it also shows the use case where pyenv/pyenv-virtualenv really shines:

$ pyenv versions
* system (set by /home/jp/.pyenv/version)
  2.7.15
  2.7.15/envs/gipcdev-2715
  3.5.2
  3.5.2/envs/venv352-analysis-2
  3.5.2/envs/venv352-beets
  3.5.2/envs/venv352-bouncer
  3.5.2/envs/venv352-bouncer19
  3.5.2/envs/venv352-bouncer19-2
  3.5.2/envs/venv352-bouncer-open
  3.5.2/envs/venv352-dcos-commons-simumation
  3.5.2/envs/venv352-jupyter
  3.5.2/envs/venv352-modern-crypto
  3.6.6
  3.6.6/envs/dcos-docker
  3.6.6/envs/dcos-e2e-zk35
  3.6.6/envs/gipc-py-366-gevent14
  3.6.6/envs/messer-hdf5
  3.6.6/envs/pandas-dev-env
  3.6.6/envs/test-gipc-100-release
  3.6.6/envs/test-gipc-101-release
  3.6.6/envs/venv366-bouncer-enterprise
  3.6.6/envs/venv366-bouncer-open
  3.6.6/envs/venv366-dataanalysis
  3.6.9
  bouncer-deptest
  cryptotest
  dataanalysis
  dcos-docker
  dcos-e2e-zk35
  gipcdev-2715
  gipc-py-366-gevent14
  messer-hdf5
  pandas-dev-env
  pypy2.7-5.9.0
  pypy2.7-5.9.0/envs/pypy27-gevent-gipc-dev
  pypy27-gevent-gipc-dev
  pypy3.5-5.9.0
  pypy3.5-5.9.0/envs/pypy35-gevent-gipc-dev
  pypy3.5-6.0.0
  pypy3.5-6.0.0/envs/venvpypy35600-gipc
  pypy35-gevent-gipc-dev
  pypy-5.6.0
  pypy-5.6.0/envs/venvpypy560-gevent120-gipc
  test-gipc-100-release
  test-gipc-101-release
  venv2710-kazoo
  venv2710-pysamltest
  venv2710-samldev
  venv342
  venv342-bouncer
  venv342-bouncer2
  venv342-bouncerdeps
  venv342-bouncerdeps2
  venv342-bouncerold
  venv342-bouncertmep
  venv342-pyoidc
  venv342-samldev
  venv342-test
  venv342-tmp
  venv343-bouncer
  venv343-saml
  venv344-bouncer
  venv344-consensus
  venv344-dev
  venv344-gunicorndev
  venv345-bouncer
  venv345-cryptography-patch
  venv345-kazam
  venv352-analysis-2
  venv352-beets
  venv352-bouncer
  venv352-bouncer19
  venv352-bouncer19-2
  venv352-bouncer-open
  venv352-dcos-commons-simumation
  venv352-jupyter
  venv352-modern-crypto
  venv366-bouncer-enterprise
  venv366-bouncer-open
  venv366-dataanalysis
  venv-dcos-342
  venv-graphtool
  venv-kazam
  venvpy3boto3
  venvpypy35600-gipc
  venvpypy560-gevent120-gipc
  venvtmp