Introduction
An introduction to pyenv and poetry for python!
There’s no denying it. Python is one of the most widely used programming languages today. Especially within Data Science. However, there are few languages that can cause such chaos and frustration as python. Handling different python-installations and dependencies can be a nightmare!
Let’s be thankful for pyenv and poetry, candles that light up the python darkness! In this post I will show you how to install these two tools, and some basic usage examples.
pyenv
pyenv is a python installation manager. It allows you to install and run multiple python installations, on the same machine. pyenv manages the different versions for you, so that you will avoid the chaos illustrated in the above picture. Don’t ever again install a python version any other way!
Setup & get started with pyenv
You can follow the steps below for installing on macOS. I’m using Homebrew, but see the pyenv documentation for alternative installation methods.
# Install pyenv.
brew install pyenv
# Add pyenv initializer to shell startup script.
echo -e 'export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init --path)"
eval "$(pyenv init -)"' >> ~/.bash_profile
# Reload your profile.
source ~/.bash_profileWith pyenv installed, you can install any python version that you would like. Below I am installing some python versions. Depending on which directory you’re in, pyenv will point to your specified python version.
# See all available python installations.
pyenv install --list
# Install some python versions.
pyenv install 3.7.5
pyenv install 2.7.17
pyenv install 3.8.0
# See all python installations that you have installed.
pyenv versions
# Set the default/global from one of the python versions.
pyenv global 3.8.0
# In the current directory, set the python version. This creates the file .python-version.
pyenv local 2.7.17
# To see which python is currently being used.
pyenv versionpoetry
poetry is a packaging and dependency manager. It resolves your library dependencies, and can build and publish your project to be distributed on your private pypi repository. In the beginning of December 2019, version 1.0.0 was finally released!
The main file of your poetry project is the pyproject.toml file. Define the requirements, the dev-requirements and project metadata in this file. poetry uses the .toml file to resolve the dependencies of your defined requirements, and creates the poetry.lock file. Then poetry creates a virtual environment and installs everything from the .lock file.
(Some alternatives to poetry for virtual environments are virtualenv, conda, venv or pipenv. I’ve used them all, especially pipenv. However, I believe that poetry is superior!)
Setup poetry
As of writing, poetry is not yet available on Homebrew. Follow the steps below for installing it.
# Install poetry via curl
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python
# Add poetry to your shell
export PATH="$HOME/.poetry/bin:$PATH"
# For tab completion in your shell, see the documentation
poetry help completions
# Configure poetry to create virtual environments inside the project's root directory
poetry config virtualenvs.in-project trueWith poetry installed, I can now create a python project. I will define a project catered for data science. poetry will install a virtual environment based on python 3.7.5, that I have already installed with pyenv. I will include data analytics libraries like pandas and scipy, along with jupyter notebook. In addition I will include the ML libraries scikit-learn and a specific pre-release version of tensorflow. Finally, I will specify black and flake8 for formatting and style guiding.
# Specify the python version for the local directory using pyenv
pyenv local 3.7.5
# Create a new project, and directory
poetry new hugos-ds-poetry-demo
# Specify some libraries
cd hugos-ds-poetry-demo
poetry add pandas numpy scipy tensorflow=2.1.0rc2 tensorflow-text matplotlib scikit-learn jupyter ipykernel
# Specify some dev libraries
poetry add --dev black flake8With the above commands, I have created a pyproject.toml file as well as a poetry.lock file. poetry has installed the virtual environment for this project in hugos-ds-poetry-demo/.venv. The pyproject.toml file looks as follows.
[tool.poetry]
name = "hugos-ds-poetry-demo"
version = "0.1.0"
description = ""
authors = ["Hugo Hjertén <hugo.hjerten@jayway.com>"]
[tool.poetry.dependencies]
python = "^3.7"
pandas = "^0.25.3"
numpy = "^1.18.0"
scipy = "^1.4.1"
tensorflow = "2.1.0rc2"
tensorflow-text = "^2.0.1"
matplotlib = "^3.1.2"
scikit-learn = "^0.22"
jupyter = "^1.0.0"
ipykernel = "^5.1.3"
[tool.poetry.dev-dependencies]
pytest = "^5.2"
black = "^19.10b0"
flake8 = "^3.7.9"
[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"Get started with poetry
# The contents of my directory.
.python-version
hugos_ds_poetry_demo/
.venv/
README.rst
hugos_ds_poetry_demo/
example.py
poetry.lock
pyproject.toml
tests/I have added the hugos-ds-poetry-demo/example.py file. It’s contents are the the same as below.
'''An example file that imports some of the installed modules'''
import pandas as pd
import numpy as np
import tensorflow as tf
import tensorflow_text as text
import matplotlib.pyplot as plt
if __name__ == "__main__":
# If the modules can't be imported, the following print won't happen
print("Successfully imported the modules!")I can now run this python script from within that virtual environment.
# Try running the script outside your virtual environment. This won't work.
python hugos_ds_poetry_demo/example.py
# Run the script within your virtual environment, using the 'run'-command.
poetry run python hugos_ds_poetry_demo/example.py
# Spawn a shell within your virtual environment.
poetry shell
# Try running the script again, after having spawned the shell within your virtual environment.
python hugos_ds_poetry_demo/example.pyConclusion
Above I have just provided you with the bare minimum to get you started. For more details, please see the documentation for pyenv and poetry! Your python life will be that much easier once you start using them both!
Extra!
Use clean python versions
If you intend to use poetry and pyenv, use clean python versions. In your pyproject.toml file you define the python version dependency, and individual library dependencies. pyenv gives the possibility of installing python versions like ironpython, cpython, anaconda and miniconda versions. For example, the anaconda python installation comes with many libraries preinstalled. These preinstalled libraries won’t show up in your pyproject.toml or poetry.lock file. Any dependencies baked into the non-clean python version will not be specified. This makes it very difficult to share and recreate your environment/project.
Using jupyter with poetry
jupyter works with kernels, and will not work out of the box with your virtual environment that poetry created for you. If you wish to work in a jupyter notebook based on your virtual environment, you need to create a kernel for that virtual environment. The code below explains how. The prerequisite is that you have added both jupyter and ipykernel as dependencies in your poetry project.
# Make sure to run the command within your virtual environment. The 'poetry run' command ensures this
# Best practice is to use the same name for your kernel as the project.
poetry run ipython kernel install --user --name=hugos-ds-poetry-demoDeath of virtual environments – PEP 582
Developing with python is constantly becoming easier and better, thanks to libs and enhancements by the growing python community. One such Python Enhancement Proposals (PEP) is PEP 582. This will avoid the steps to create, activate or deactivate “virtual environments”. So let’s cross our fingers for this to be implemented asap! :)
Problems installing python versions with pyenv
No tool is perfect, and unfortunately there are some drawbacks and issues when installing python versions with pyenv. For instance, there is a delay from when a new python version has been released, to when it is available in pyenv. New versions of python are available with upgrades of pyenv.
When installing a python version with pyenv you might encounter the following error, zipimport.ZipImportError: can't decompress data; zlib not available. This is explained in more detail here, but the following fix might work for you.
# If you are experiencing zipimport.ZipImportError, the following fix might help!
brew install zlib
brew install sqlite
export LDFLAGS="${LDFLAGS} -L/usr/local/opt/zlib/lib"
export CPPFLAGS="${CPPFLAGS} -I/usr/local/opt/zlib/include"
export LDFLAGS="${LDFLAGS} -L/usr/local/opt/sqlite/lib"
export CPPFLAGS="${CPPFLAGS} -I/usr/local/opt/sqlite/include"
export PKG_CONFIG_PATH="${PKG_CONFIG_PATH} /usr/local/opt/zlib/lib/pkgconfig"
export PKG_CONFIG_PATH="${PKG_CONFIG_PATH} /usr/local/opt/sqlite/lib/pkgconfig"Another error that you might encounter is ar: internal ranlib command failed. For me, moving/removing the ar and ranlib directories did the trick.
# If you are experiencing 'ar: internal ranlib command failed', the following fix might help!
sudo mv /opt/local/bin/ranlib /opt/local/bin/ranlib-backup
sudo mv /opt/local/bin/ar /opt/local/bin/ar-backup
Received the following question on linkedin:
Nice read! What features is it that are superior in poetry compared to for example Conda, and how is the transition when you move from development to production?
Response:
Happy you liked it! :) An advantage with conda is that it’s language-agnostic and has support for binary packages. With conda you can install your specific python version directly into the virtual environment, which you cannot with poetry. The main contributor of poetry stresses the fact that poetry is not a python version manager. That’s why you need pyenv also :)
However, what Poetry does really well is resolve dependencies for the libraries that you require in your project. Another major benefit of poetry over conda is how easy it is to recreate and share the environment, superior over the standard setup.py/requirements.txt files.
More or less, pipenv has these benefits over conda also. But where poetry wins over pipenv is that it is less buggy. It feels more mature and tested, even though its younger :) Even though pipenv is the recommended library dependency manager… With the recent release of 1.0.0 most of the previous smaller bugs are fixed!
Also, packaging your project and publishing your custom python library to i.e. pypi, is super-easy with poetry!
And we found going into production with poetry is really forgiving and easy!
Long response… I guess I believe in poetry ;)
Great article, Hugo!
If I may ask, what is the advantage of creating the .venv/ folder in the project’s root directory rather than in the default location?
Hi Ian! Happy you appreciated the article :)
Creating the
.venvdirectory in the project’s root folder is definitely a question of personal taste. I like having everything gathered in the same directory, rather than having the source code in one place and the executables in another. But the biggest reason for me is to make it easier to point out my executables to my IDE. No matter the python project, VS Code always finds the python executable with all of its libs in.venv. I find this very practicle!Just wondering, do we not end up with multiple copies of the same libraries each time we create a project, which ends up taking more disk space. Is that what most people are willing to do as trade-off? Thanks,
Hi James! Yes, you’re completely right, each virtual environment is its own copy. I think it’s worth mentioning that you should make sure to only include the specific libraries that you actually need in your project. I.e, don’t include libraries that “might” come in handy later, but rather add the library first when you actually have a need for it.
It sounds to me like you often use the same libraries in your different python projects? Either way, unless you have thousands of projects with tons of identical libraries in each project, I would claim that the little extra used disk space is worth it. Good luck!
Hi Hugo, thanks for the good job. Had nice time reading your article. I have a concern about one of the code samples:
# Try running the script outside your virtual environment. This won’t work.
python src/example.py
# Run the script within your virtual environment, using the ‘run’-command.
poetry run python src/example.py
should be not ‘src’ but ‘hugos_ds_poetry_demo’ where you created your file to test the dependencies.
Thanks for taking the time to read the post, and pointing out the typo! I have updated the post :)
Hey Hugo,
You used Pyenv to install a Python version and Poetry to create a virtualenv using that version. I tried using a virtualenv created with Pyenv in Poetry but this makes a lot more sense so thanks for clearing it up :)
Happy it could be of help to you! :)
Hey Hugo, FYI this article is now somewhat outdated since the Poetry installation process has changed slightly making some commands you’re suggesting obsolete, like export PATH=”$PYENV_ROOT/bin:$PATH”.
Hi Hugo,
interesting article. I experienced a bug (most likely) in Poetry.
The bug becomes a problem if you are working in a folder with a space in the folder name, e.g. “/Volumes/Macintosh SR0/Users/mklengel/Development” and using “poetry config virtualenvs.in-project true”.
I’ve reported the possible bug here: https://github.com/python-poetry/poetry/issues/3643
My current workaround is “poetry config virtualenvs.in-project false” – I don’t like this, because VS Code doesn’t find the .venv folder in this case.
Regards
Michael
+1 for ‘poetry config virtualenvs.in-project true’ I didn’t know about that.
What is the benefit to use pyenv with poetry here?! , poetry install a version of python you specified in toml file in default location : ../cache/.poatry/your_virtual _env/python
and every package you installed it’s in python folder in the same path, and every project can have isolated python version installed in virtual env folder (in poetry path) , so I think one must be used, (poetry or pyenv ) , don’t both
Just leaving this here as food for thought.
1. pipenv integrates seamlessly with dotenv. poetry refuses to and requires upgrade to 1.2 prerelease + plugin
2. pipenv falls neatly under pyenv, thus respecting local versions. pipenv will even use pyenv to install a missing version for you. poetry is not integrated and does not respect pyenv local version. feels more like 2 separate tools to manage and synchronize.
3. pipenv has a simple npm-like scripts syntax in Pipfile. poetry has a much more obscure and finnicky scripts syntax that relates more to python package management than the environment shell.
Thank you for this helpful site!
You have a typo:
echo -e ‘export PYENV_ROOT=”$HOME/.pyenv”
should read:
echo -e ‘export PYENV_ROOT=”$HOME/.pyenv”‘
missing the closing single quote