Distributing Packages / Deploying
Software Development for Engineering Research
Kyle Niemeyer. 26 January 2022
ME 599, Corvallis, OR
from time import sleep
print("This is my file to demonstrate best practices.")
def process_data(data):
print("Beginning data processing...")
modified_data = data + " that has been modified"
sleep(3)
print("Data processing finished.")
return modified_data
def read_data_from_web():
print("Reading data from the Web")
data = "Data from the web"
return data
def write_data_to_database(data):
print("Writing data to a database")
print(data)
def main():
data = read_data_from_web()
modified_data = process_data(data)
write_data_to_database(modified_data)
if __name__ == "__main__":
main()
We've talked about version-controlling, licensing, structuring, testing, and object-orienting your software.
What about actually deploying it so that others can install and use it?
Note: this can be hard, especially when considering all the various systems you may want/need to support.
First step: package the software.
This means creating a file distributable to a wider audience, that can install your package on their system.
Depends somewhat on the package manager they use (e.g., pip/pipenv, conda)
Package management options
- Source-based: link to code. Good for dynamic languages, less for compiled.
- Binary: popular for both dynamic and compiled. Create packages for various architectures/systems.
- Virtualization: package both software and environment.
Comment on Python package management
Unfortanately, managing packages in Python is an ever-evolving environment.
pip and conda are currently the most-used, but now there is pipenv
Do the best you can! Fortunately the basics should be compatible with future systems.
Python: pip and PyPI
-
pipcan be used to install Python packages from source or from PyPI -
pipenvis a newer replacement for pip that allows environments and handles dependenies better -
We will tell
piphow to install your package by creating asetup.py(orpyproject.toml) file for thesetuptoolspackage to use. -
setup.py/pyproject.tomlgoes at the same level as your package's source code directory.
setup.py
/compphys
|- __init__.py
|- _version.py
|- constants.py
|- physics.py
/more
|- __init__.py
|- morephysics.py
|- evenmorephysics.py
/assets
|- data.txt
|- orphan.py
/tests
|- test_physics.py
|- test_morephysics.py
...
setup.py imports and calls setup() function, which can both install
a package locally and make/upload to PyPI
-
pipcan be used to install Python packages from source or from PyPI -
We will tell
piphow to install your package by creating asetup.pyfile for thesetuptoolspackage to use. -
setup.pygoes at the same level as your package's source code directory.
try:
from setuptools import setup, find_packages
except ImportError:
from distutils.core import setup
setup(
name='compphys',
version='0.1.0',
description='Effective Computation in Physics',
author='Anthony Scopatz and Kathryn D. Huff',
author_email='koolkatz@gmail.com',
url='http://physics.codes',
classifiers=[
'License :: OSI Approved :: BSD License',
'Intended Audience :: Developers',
'Intended Audience :: Science/Research',
'Natural Language :: English',
'Programming Language :: Python :: 3',
],
license='BSD-3-Clause',
python_requires='>=3',
zip_safe=False,
packages=['compphys', 'compphys.more', 'compphys.tests'],
# or find automatically:
package=find_packages(),
package_dir={
'compphys': 'compphys',
'compphys.more': 'compphys/more',
'compphys.tests': 'compphys/tests',
},
include_package_data=True,
# or you can specify explicitly:
package_data={
'compphys': ['assets/*.txt']
},
)
Better: have setup.py copy in README, version information from
_version.py file in your package, and include dependency information.
Add a setup.cfg file with some additional configuration options
Add a CHANGELOG file to describe how your code changes with each version.
Add a MANIFEST.in file to tell PyPI to bring other files.
from codecs import open
from os import path
import sys
here = path.abspath(path.dirname(__file__))
with open(path.join(here, 'compphys', '_version.py')) as version_file:
exec(version_file.read())
with open(path.join(here, 'README.md')) as readme_file:
readme = readme_file.read()
with open(path.join(here, 'CHANGELOG.md')) as changelog_file:
changelog = changelog_file.read()
desc = readme + '\n\n' + changelog
try:
import pypandoc
long_description = pypandoc.convert_text(desc, 'rst', format='md')
with open(path.join(here, 'README.rst'), 'w') as rst_readme:
rst_readme.write(long_description)
except (ImportError, OSError, IOError):
long_description = desc
install_requires = [
'numpy',
]
tests_require = [
'pytest',
'pytest-cov',
]
needs_pytest = {'pytest', 'test', 'ptr'}.intersection(sys.argv)
setup_requires = ['pytest-runner'] if needs_pytest else []
setup(
name='compphys',
version=__version__,
description='Effective Computation in Physics',
long_description=long_description,
author='Anthony Scopatz and Kathryn D. Huff',
author_email='koolkatz@gmail.com',
url='http://physics.codes',
classifiers=[
'License :: OSI Approved :: BSD License',
'Intended Audience :: Developers',
'Intended Audience :: Science/Research',
'Natural Language :: English',
'Programming Language :: Python :: 3',
],
license='BSD-3-Clause',
install_requires=install_requires,
tests_require=tests_require,
python_requires='>=3',
setup_requires=setup_requires,
zip_safe=False,
packages=['compphys', 'compphys.more', 'compphys.tests'],
package_dir={
'compphys': 'compphys',
'compphys.more': 'compphys/more',
'compphys.tests': 'compphys/tests',
},
include_package_data=True,
)
Contents of setup.cfg:
[metadata]
license_file = LICENSE
[aliases]
test=pytest
[tool:pytest]
addopts = -vv --cov=./
filterwarnings =
ignore::ResourceWarning
Contents of MANIFEST.in::
include LICENSE
include AUTHORS.md
include CHANGELOG.md
include CONTRIBUTING.md
include compphys/tests/*.txt
-
Install the package from source:
python setup.py install -
Create a source-only ZIP file:
python setup.py sdist - Create an account on PyPI
-
Upload:
twine upload dist/* - Subsequent versions: increase the version number (tip: use Semantic Versioning)
New: pyproject.toml
-
There is a newer standard for specifying project information:
pyproject.toml -
This is intended to replace
setup.pyand related files forpip - This is still evolving, and you might choose to adopt this approach, but it is currently not required (I think)
Contents of basic pyproject.toml:
[build-system]
requires = [
"setuptools>=42", "wheel"
]
build-backend = "setuptools.build_meta"
Semantic Versioning: Given a version number MAJOR.MINOR.PATCH, increment the:
- MAJOR version when you make incompatible API changes,
- MINOR version when you add functionality in a backwards-compatible manner, and
- PATCH version when you make backwards-compatible bug fixes.
To start: set initial development release at 0.1.0 and increment minor version for subsequent releases.
Contents of _version.py:
__version_info__ = (0, 4, 2, 'a1')
__version__ = '.'.join(map(str, __version_info__[:3]))
if len(__version_info__) == 4:
__version__ += __version_info__[-1]
Contents of CHANGELOG.md:
# Changelog
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
## [Unreleased]
## [0.1.0] - 2015-10-06
### Added
- Answer "Should you ever rewrite a change log?".
### Changed
- Improve argument against commit logs.
- Start following [SemVer](http://semver.org) properly.
## [0.0.8] - 2015-02-17
### Changed
- Update year to match in every README example.
- Reluctantly stop making fun of Brits only, since most of the world
writes dates in a strange way.
### Fixed
- Fix typos in recent README changes.
- Update outdated unreleased diff link.
## [0.0.7] - 2015-02-16
### Added
- Link, and make it obvious that date format is ISO 8601.
### Changed
- Clarified the section on "Is there a standard change log format?".
### Fixed
- Fix Markdown links to tag comparison URL with footnote-style links.
## [0.0.6] - 2014-12-12
### Added
- README section on "yanked" releases.
## [0.0.5] - 2014-08-09
### Added
- Markdown links to version tags on release headings.
- Unreleased section to gather unreleased changes and encourage note
keeping prior to releases.
## [0.0.4] - 2014-08-09
### Added
- Better explanation of the difference between the file ("CHANGELOG")
and its function "the change log".
### Changed
- Refer to a "change log" instead of a "CHANGELOG" throughout the site
to differentiate between the file and the purpose of the file — the
logging of changes.
### Removed
- Remove empty sections from CHANGELOG, they occupy too much space and
create too much noise in the file. People will have to assume that the
missing sections were intentionally left out because they contained no
notable changes.
## [0.0.3] - 2014-08-09
### Added
- "Why should I care?" section mentioning The Changelog podcast.
## [0.0.2] - 2014-07-10
### Added
- Explanation of the recommended reverse chronological release ordering.
## 0.0.1 - 2014-05-31
### Added
- This CHANGELOG file to hopefully serve as an evolving example of a
standardized open source project CHANGELOG.
- CNAME file to enable GitHub Pages custom domain
- README now contains answers to common questions about CHANGELOGs
- Good examples and basic guidelines, including proper date formatting.
- Counter-examples: "What makes unicorns cry?"
[Unreleased]: https://github.com/olivierlacan/keep-a-changelog/compare/v1.0.0...HEAD
[0.1.0]: https://github.com/olivierlacan/keep-a-changelog/compare/v0.0.8...v0.1.0
[0.0.8]: https://github.com/olivierlacan/keep-a-changelog/compare/v0.0.7...v0.0.8
[0.0.7]: https://github.com/olivierlacan/keep-a-changelog/compare/v0.0.6...v0.0.7
[0.0.6]: https://github.com/olivierlacan/keep-a-changelog/compare/v0.0.5...v0.0.6
[0.0.5]: https://github.com/olivierlacan/keep-a-changelog/compare/v0.0.4...v0.0.5
[0.0.4]: https://github.com/olivierlacan/keep-a-changelog/compare/v0.0.3...v0.0.4
[0.0.3]: https://github.com/olivierlacan/keep-a-changelog/compare/v0.0.2...v0.0.3
[0.0.2]: https://github.com/olivierlacan/keep-a-changelog/compare/v0.0.1...v0.0.2
conda
-
pipis great for pure Python source codes, but many research codes use multiple languages including compiled code. - Conda: cross-platform binary manager designed for deploying research software.
- Handles multilanguage and non-Python code, runs on all operating systems, and runs in user's home space—it does not try to install on the system.
conda
- Fastest way to install: Miniconda distribution
- By default, searches Anaconda Cloud, but you can add additional channels to search for packages (even your own!)
-
Recommendation: add conda-forge channel via
conda config --add channels conda-forge
Build a conda package (on your system)
-
Create an Anaconda cloud
account, install the client, and login:
conda install anaconda-client anaconda login - Install
conda-build:conda install conda-build conda config --set anaconda_upload no -
Create a new directory
conda.recipeand in it create the filemeta.yaml:
{% set data = load_setup_py_data() %}
package:
name: compphys
version: {{ data['version'] }}
source:
path: ..
build:
number: 0
script: python setup.py install --single-version-externally-managed --record=record.txt
requirements:
build:
- python >= 3
- setuptools
run:
- python
- numpy
test:
imports:
- compphys
requires:
- pytest
- pytest-cov
commands:
- pytest -vv --pyargs compphys
about:
home: data['url']
license: BSD 3-Clause
license_file: LICENSE
license_family: BSD
summary: data['description']
description: data['long_description']
dev_url: data['url']
-
Build the package:
conda build conda.recipe -
Upload your package to Anaconda cloud:
anaconda upload $HOME/miniconda/conda-bld/*/compphys*.tar.bz2 -
Go see your package at
https://anaconda.org/[username]/compphys! -
Others can install your package using
conda install -c [username] compphys
Now you have three ways of making your software installable by people:
-
Create
setup.pyto allow others to download your repo and install manually (at minimum, do this). -
Upload your package to PyPI, making it easily installable using
pip. -
Create and upload a
condapackage, easily installable via your personal channel.
Advanced deploying
Although at this point it may be sufficient to manually build and deploy packages for pip and conda, you can also use GitHub Actions or Travis CI to do this for you.
Main idea: when you tag a release using Git (e.g., git tag -a v0.1.1 -m 'v0.1.1'),
GitHub Actions will run your tests, and if successful build and deploy to PyPI and/or Anaconda.
name: Python package
on:
push:
# Build on tags that look like releases
tags:
- v*
# Build when main is pushed to
branches:
- main
pull_request:
# Build when a pull request targets main
branches:
- main
jobs:
build:
runs-on: ${{ matrix.os }}
continue-on-error: ${{ matrix.experimental }}
strategy:
matrix:
python-version: [3.7, 3.8]
os: [ubuntu-latest, macos-latest, windows-latest]
experimental: [false]
fail-fast: false
steps:
- name: Check out the repository
uses: actions/checkout@v2
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip setuptools wheel
python -m pip install tox tox-gh-actions tox-venv
- name: Test with tox
run: tox -v
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v1
with:
token: ${{ secrets.CODECOV_TOKEN }}
file: ./coverage.xml
conda-build-and-upload:
runs-on: ubuntu-latest
defaults:
run:
shell: bash -l {0}
steps:
- uses: actions/checkout@v2
- uses: conda-incubator/setup-miniconda@77b16ed746da28724c61e1f1ad23395a4b695ef5
with:
auto-update-conda: true
conda-build-version: 3.21
auto-activate-base: true
activate-environment: ""
show-channel-urls: true
miniforge-version: latest
- name: Install conda-build dependencies
run: conda install -q anaconda-client conda-verify ripgrep
- name: Run conda build
run: conda build conda.recipe
- name: Upload package to anaconda.org
run: |
anaconda -t ${{ secrets.ANACONDA_TOKEN }} upload $CONDA/conda-bld/*/package*.tar.bz2
if: github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags')
pypi-build-and-upload:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/setup-python@v2
with:
python-version: 3.8
- name: Install build dependencies
run: python -m pip install --upgrade pip setuptools wheel pep517
- name: Build the source and binary distributions
run: python -m pep517.build --source --binary --out-dir dist/ .
- name: Publish a Python distribution to PyPI
if: github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags')
uses: pypa/gh-action-pypi-publish@v1.3.1
with:
user: __token__
password: ${{ secrets.PYPI_PASSWORD }}
Required steps: generate those secure strings for passwords/upload tokens
-
Anaconda: go to your Anaconda.org account settings,
and in Access generate an API token. Then, in your Travis CI
account, go to the repo in question, and create an environment variable with the name
ANACONDA_TOKEN. Add the value of the API token you just created. Copy the secure string intometa.yaml. -
PyPI:
install Travis CI command line utility, then
use command
travis encrypt --add deploy.passwordto generate thesecurestring formeta.yaml.
Next: add setup.py or pyproject.toml
at minimum to your projects, and consider making available via
PyPI and/or Anaconda!