Python venv management
2 min read

Python venv management

When you create a project in python, you should create requirements.txt to specify dependencies, so other people can have the same environment when using your project.

However, if you don’t specify module versions in requirements.txt, you could end up with people using the wrong module version, where some APIs can be deprecated or have different behaviors than older versions.

Another issue is that maybe you’re working on a few python projects, each uses different python versions (eg. projectA uses python3.6, projectB uses python3.9, etc).

Enters pyenv and pipenv (I will discuss about poetry later), where you can easily switch python versions, and have different environment (with python version locking) for projects you’re working on.

Installing pyenv

Follow instructions here. For windows, use this.

Useful commands

# list available python versions
pyenv install --list

# install specific version
pyenv install 3.8.0

# list installed versions
pyenv versions

# activate new env
pyenv shell 3.8.0 # support multiple version

# config venv
pyenv virtualenv 3.8.0 my-data-project

# set env per folder/project
pyenv local my-data-project

Installing pipenv

Notes: make sue pyenv is installed, and remove anaconda / miconda & python3 installed via official installer from your system. Then run:

$ pip install pipenv

# run this command every time pip installs a .exe
$ pyenv rehash

pipenv workflow

pipenv --python 3.7

# install a specific module
pipenv install jupyterlab==2.2.9

# install from existing requirements.txt or from Pipfile definition
pipenv install

# remove venv
pipenv --rm

# running inside venv
pipenv run jupyter lab
pipenv run python main.py # is equivalent to `pipenv shell && python3 main.py`

Windows only

$ pyenv install 3.7.7 # see Pipfile for required python version
$ pyenv local 3.7.7 # IMPORTANT. global / shell doesn't work with pipenv
$ pyenv rehash
$ pip install pipenv # done once per pyenv python version
$ pyenv rehash
$ pipenv --python 3.7
$ pipenv install
$ pipenv run python tokenization_sandbox.py

Notes

  • On linux/mac, do not use system python. OS updates would mean python version upgrade, in turn making all your installed modules gone. Use python installed via pyenv instead.
  • On windows, start fresh with pyenv.
  • Do not use anaconda distribution. It does too much background magic that can make things harder to manage environment property. In addition, venv definition from anaconda is often doesn’t work cross-platform (eg. venv def from windows wouldn’t work on mac due to different wheel binary versions).
  • Always create venv via pipenv per each project. Although you can have a playground venv via pyenv, so you can shell into it and do a quick analysis / scripting on an adhoc basis.
  • I heard good things about poetry but it doesn’t integrate with pyenv natively. It would work if you use it to publish python modules, since it simplifies a lot of processes.
    • poetry also picks up the wrong python version from pyenv. And if you sync python version via pyenv, it has to be the same python version across all OSes, including minor version. pipenv doesn’t have this restriction, and it also picks up the correct python version from pyenv by default (via pipenv --python 3.8).