Back around 2010 I was manually managing production-critical CPython installations, compiled in special ways. I learned a lot about how to properly isolate Python installations from each other. I also learned how important it is to never confuse a “system Python” (as many Linux distributions have one) with a Python build that you can be in control of. Since then I have never touched a system Python anymore (it is the system’s Python after all, and none of my business).
For developing, testing, and running my software, I needed to manage custom Python installations. That involved building CPython from source, many times. At some point around 2015 I adopted pyenv for doing exactly that. Back then, I also adopted pyenv-virtualenv for managing so-called “virtual environments” derived from the individual Python builds managed by pyenv.
I come to say: this method has been a joy to use over the last years. It has never let me down; it has always provided me with simplicity, reliability, a predictable outcome based on which serious engineering work could be done. I have adopted it for continuous integration pipelines; I use it for building container images, towards more reproducible test environments. It always works.
I want to share the boring stuff I did today with pyenv/pyenv-virtualenv, as I have done it for years, always with a smile, thinking and appreciating “that just worked as expected”.
Today, for a program I am developing right now, I wanted to set up a CPython 3.6.9 build, with a virtual environment derived from it to install specific dependencies into it. I started with
cd ~/.pyenv && git pull
for updating my local pyenv installation to learn about current Python releases (I have been using the same pyenv installation for years, migrated it between development machines). I issued the following command for pyenv to download a CPython 3.6.9 source archive, and to build it:
pyenv install 3.6.9
I found that the resulting build didn’t provide the lzma module because I didn’t have the corresponding header files available on my Fedora installation. I installed the relevant header files with a
dnf install xz-devel
and simply repeated
pyenv install 3.6.9
This resulted in a CPython build including the lzma module. I then went ahead and derived a virtual environment from that build, for the specific purpose of developing my current pet project:
pyenv virtualenv 3.6.9 venv369-goeffel
I then activated said virtual environment with
pyenv activate venv369-goeffel
and started to use it for specific development work.
Not spectacular, right? That’s it, precisely. The power here comes from the simplicity and reliability. A big thank you goes out to the pyenv maintainers.
I think the following shows that I should start deleting old things, but I think it also shows the use case where pyenv/pyenv-virtualenv really shines:
$ pyenv versions * system (set by /home/jp/.pyenv/version) 2.7.15 2.7.15/envs/gipcdev-2715 3.5.2 3.5.2/envs/venv352-analysis-2 3.5.2/envs/venv352-beets 3.5.2/envs/venv352-bouncer 3.5.2/envs/venv352-bouncer19 3.5.2/envs/venv352-bouncer19-2 3.5.2/envs/venv352-bouncer-open 3.5.2/envs/venv352-dcos-commons-simumation 3.5.2/envs/venv352-jupyter 3.5.2/envs/venv352-modern-crypto 3.6.6 3.6.6/envs/dcos-docker 3.6.6/envs/dcos-e2e-zk35 3.6.6/envs/gipc-py-366-gevent14 3.6.6/envs/messer-hdf5 3.6.6/envs/pandas-dev-env 3.6.6/envs/test-gipc-100-release 3.6.6/envs/test-gipc-101-release 3.6.6/envs/venv366-bouncer-enterprise 3.6.6/envs/venv366-bouncer-open 3.6.6/envs/venv366-dataanalysis 3.6.9 bouncer-deptest cryptotest dataanalysis dcos-docker dcos-e2e-zk35 gipcdev-2715 gipc-py-366-gevent14 messer-hdf5 pandas-dev-env pypy2.7-5.9.0 pypy2.7-5.9.0/envs/pypy27-gevent-gipc-dev pypy27-gevent-gipc-dev pypy3.5-5.9.0 pypy3.5-5.9.0/envs/pypy35-gevent-gipc-dev pypy3.5-6.0.0 pypy3.5-6.0.0/envs/venvpypy35600-gipc pypy35-gevent-gipc-dev pypy-5.6.0 pypy-5.6.0/envs/venvpypy560-gevent120-gipc test-gipc-100-release test-gipc-101-release venv2710-kazoo venv2710-pysamltest venv2710-samldev venv342 venv342-bouncer venv342-bouncer2 venv342-bouncerdeps venv342-bouncerdeps2 venv342-bouncerold venv342-bouncertmep venv342-pyoidc venv342-samldev venv342-test venv342-tmp venv343-bouncer venv343-saml venv344-bouncer venv344-consensus venv344-dev venv344-gunicorndev venv345-bouncer venv345-cryptography-patch venv345-kazam venv352-analysis-2 venv352-beets venv352-bouncer venv352-bouncer19 venv352-bouncer19-2 venv352-bouncer-open venv352-dcos-commons-simumation venv352-jupyter venv352-modern-crypto venv366-bouncer-enterprise venv366-bouncer-open venv366-dataanalysis venv-dcos-342 venv-graphtool venv-kazam venvpy3boto3 venvpypy35600-gipc venvpypy560-gevent120-gipc venvtmp