Managing your computational environment

Author
Affiliation

Kyle Niemeyer

Oregon State University

Published

January 6, 2025

Introduction to virtual environments

Power of Python for research comes from rich ecosystem of third-party packages

These include general numerical and scientific tools like NumPy, SciPy, and Pandas, and domain-specific software like Cantera. You should rely on mature packages that provide useful features when possible—don’t reinvent the wheel!

But, each software package has its own set of dependencies, and might require a specific version of some other package (like when a newer version breaks or changes some functionality).

Your results may depend on the specific versions and combination of packages you have installed.

This is particularly important for research analysis.

🚨 As a rule, don’t install packages into your global environment!

… even if the documentation says pip install x

Best practice: use virtual environments

  • Virtual environments are lightweight installations of packages in a particular location (like in your working directory or project project) that build on top of your system Python installation.

  • Virtual environments isolate the set of dependencies for a project or package from each other, and from your system configuration.

  • These are intended to be easy to set up, disposable, and replaceable.

  • On a Mac or Linux machine, I recommend using the built-in venv module to create a virtual environment for your work, then install inside that using the pip installer.

  • On Windows, your best bet is probably the Anaconda Distribution. Anaconda comes with hundreds of scientific packages already, and also lets you set up system-wide virtual environments.

Using venv to manage virtual environments

Creating and activating a virtual environment

python3 -m venv .venv

This creates a new directory .venv with:

  • .venv/bin with link to Python, pip, and activation scripts; and
  • .venv/lib/site-packages where packages will be installed.

To activate the environment, source the activation script:

source .venv/bin/activate
Tip

Try this yourself. .venv/bin will be added to your PATH, and your shell will usually indicate you are “in” a virtual environment.

To leave the virtual environment, run the deactivate function:

deactivate
Note

Closing/terminating your shell will automatically deactivate the virtual environment, and you’ll need to manually activate whenever you start a new session.

Installing packages

pip install <package>

Alternatives for managing virtual environments

pip family

Tools that all work similarly to venv:

  • venv: comes with Python, simple
  • virtualenv: installable package with same interface as venv, but faster and with more options
  • uv: written in Rust and extremely fast

In addition, there are project management tools that both handle environments, packaging, production, etc.:

These work a bit differently, but can be quite powerful particularly for locking specific versions of dependencies. Consider returning to these when we are thinking about packaging and distribution.

Conda family

Conda is an open-source tool for managing virtual environments across your system, originally designed for Python package management but now supports a wide range of software (e.g., compiled C++ dependencies).

Virtual environments created with Conda are not location-dependent like venv, but the same approach applies: do not install in the base/default environment.

Conda is recommended if you expect to need non-Python dependencies.

conda config --set auto_activate_base false  # turn off the default environment
conda env create -n some_name  # or use paths with `-p`
conda activate some_name
conda deactivate

Conda-based package managers

Some alternatives to conda exist and offer some advantages, including speed:

  • Miniconda: minimal Conda installation without all the packages that come with the full Anaconda distribution
  • Pixi: fast, Rust-based package manager and workflow tool Some other
  • Micromamba: tiny version of Mamba, statically linked C++ binary

References and further reading

Schreiner, H., Niemeyer, K., Holland, J. G., Dang, G., & Restrepo, M. I. (2024). “INTERSECT: Packaging” [Data set]. https://github.com/INTERSECT-training/packaging

Scientific Python Library Development Guide