When it comes to Python development on macOS, I rely on a combination of two tools that have served me exceptionally well over the past few years: Pyenv and Poetry. Pyenv provides an elegant solution for managing different Python versions on my system, while Poetry simplifies dependency management and the creation of virtual environments for my projects. In this article, I will guide you through the process of setting up Pyenv to install a specific Python version and then using Poetry to create a virtual environment for your project.

Setting up Pyenv

  1. Install Homebrew:

If you are not using Homebrew already, proceed to install it using the following command:

1/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
  1. Install Pyenv:

Use the following shell command,

1brew install pyenv

and then add the following lines to the shell profile file ~/.zshrc or its equivalent:

1   # Python configuration -----------------
2path+="$HOME/.pyenv/bin"
3if command -v pyenv 1>/dev/null 2>&1; then
4  eval "$(pyenv init -)"
5fi
6alias brew='env path+="${PATH//$(pyenv root)\/shims:/}" brew'

The last alias is to ensure that brew works seamlessly with the global Python environment set by pyenv (see here).

  1. Install a Specific Python Version:
1pyenv install -l # to list all the available python versions
2env PYTHON_CONFIGURE_OPTS="--enable-framework" pyenv install 3.11.5

The inclusion of --enable-framework in the install command addresses potential compilation errors during package building (see here).

  1. Set Global or Local Python Version:

Now that pyenv is installed, set the Python version globally or locally. Setting globally provides access to this Python version from anywhere:

1pyenv global 3.x.x

Setting locally provides access to this Python version from a specific directory or project:

1vim .python-version  # add this file to the root of the project
2pyenv local 3.x.x   # add this line to the above file

Make sure to restart the shell before proceeding.

Using Poetry for Dependency Management and Virtual Environments

  1. Install Poetry:

Install Poetry using the following command:

1curl -sSL https://install.python-poetry.org | python3 -
  1. Initialize Poetry in the Project:
1poetry init

Follow the prompts to provide project information, and Poetry will generate a pyproject.toml file.

  1. Install Dependencies and Create Virtual Environment:
1poetry install
  1. Activate the Virtual Environment:

Activate the virtual environment with:

1poetry install

To exit the virtual environment, type exit.

To have more control over the structure of the Python project, an alternative approach is to create a project by following these steps:

 1# create a folder for the project
 2mkdir ~/Data/Work/Projects/projFolder
 3
 4# create a virtual env
 5poetry new --name pkgName --src .
 6
 7# install any dependencies
 8poetry add biopython
 9poetry add numpy
10poetry add pandas
11...

The resulting folder structure will look like this:

1.
2├── README.md
3├── poetry.lock
4├── pyproject.toml
5├── src
6│ └── pkgName
7│     └── __init__.py
8└── tests
9	└── __init__.py

Additionally, I prefer to segregate the dependencies of my Python project into main and dev groups:

1# add dependencies to the development section
2poetry add --group dev ipykernel black mypy ruff

Looking at the pyproject.toml, we can observe two sections: the main section, which comprises the core dependencies of the project, and the development dependencies installed under the dev group.

 1[tool.poetry.dependencies]
 2python = "^3.11"
 3typer = "^0.9.0"
 4bio = "^1.5.9"
 5loguru = "^0.7.0"
 6plotnine = "^0.12.1"
 7pyarrow = "^12.0.0"
 8
 9[tool.poetry.group.dev.dependencies]
10black = {extras = ["all"], version = "^23.3.0", allow-prereleases = true}
11jupyter = "^1.0.0"
12mypy = "^1.3.0"
13ruff = "^0.0.270"
14pandas-stubs = "^2.0.1.230501"

Updating dependencies

To update the project dependencies, we can manually edit the pyproject.toml and run the poetry install command again. Alternatively, we can automate this process using a Poetry plugin called poetry-plugin-up (repo). This plugin updates the version numbers of the dependencies listed in pyproject.toml to their latest compatible versions while respecting version constraints.

1poetry self add poetry-plugin-up
2poetry up

To update all versions to the latest available compatible versions, use the --latest flag.

Moreover, it is possible to update a specific dependency, only the dependencies listed in the main group, or all of them except the dependencies listed in the dev group.

1poetry up foo bar
2poetry up --only main
3poetry up --without dev

Installing dependencies from repositories

In some cases, you may find it necessary to install Python packages directly from repositories, especially when dealing with cutting-edge features, bug fixes, or custom modifications not yet included in official releases. This approach allows you to access the latest developments or specific branches, tag, or revsion of a project.

1poetry add git+https://git@github.com/has2k1/plotnine.git

Or, you can edit the pyproject.toml to include the specific branch, tag, or revision:

1[tool.poetry.dependencies]
2plotnine = {git = "https://git@github.com/has2k1/plotnine.git"}
3plotnine = {git = "https://git@github.com/has2k1/plotnine.git", branch = "dev"}
4plotnine = {git = "https://git@github.com/has2k1/plotnine.git", tag = "v0.12.4"}
5plotnine = {git = "https://git@github.com/has2k1/plotnine.git", rev = "f78a0fd"}

Install packages as editable

Installing packages from local folders and in editable mode is another powerful technique that facilitates a dynamic development environment. This mode, often referred to as “editable” or “develop” mode, allows you to work on the source code of a package directly within your project. I use it when actively developing or debugging a Python package and needing to test changes in a live environment. Installing the package in editable mode within my project enables me to experiment without the need for repeated installations.

1poetry add ../my-package/             # install a local package
2poetry add --editable ../my-package/  # install a local package in editable mode
3poetry add ../my-package/dist/my-package-0.1.0.tar.gz
4poetry add ../my-package/dist/my_package-0.1.0.whl

Or, you can edit the pyproject.toml manually:

1[tool.poetry.dependencies]
2my-package = {path = "../my-package/", develop = true}
3my-package = {path = "../my-package/dist/my-package-0.1.0.tar.gz"}
4my-package = {path = "../my-package/dist/my_package-0.1.0.whl"}

Conclusion

Pyenv ensures that I have the correct Python version for my projects, while Poetry simplifies dependency management and virtual environment creation. With these tools in my arsenal, Python development becomes efficient, reliable, and reproducible on my system.