Test Coverage, Continuous Integration, & Documentation
Software Development for Engineering Research
Kyle Niemeyer. 17 April 2019
ME 599, Corvallis, OR
First: test coverage
Meaning: percentage of code for which a test exists, determined by number of line executed during tests
pytest-cov
Instructions:
-
Install
pytest-covusingpip/conda -
pytest -vv --cov=./ - Look at coverage; are you at or near 100%?
-
Get more detailed information by having it create a report:
pytest -vv --cov=./ --cov-report html
Example
# content of test_sample.py
def inc(x):
if x < 0:
return x - 1
return x + 1
def test_answer():
assert inc(3) == 4
$ pytest -vv test_sample.py --cov=./
main.py::test_answer PASSED
---------- coverage: platform darwin, python 3.5.4-final-0 -----------
Name Stmts Miss Cover
-----------------------------
main.py 6 1 83%
Coverage report
# content of test_sample.py
def inc(x):
if x < 0:
return x - 1
return x + 1
def test_answer():
assert inc(3) == 4
def test_answer_negative():
assert inc(-2) == -3
$ pytest -vv test_sample.py --cov=./
main.py::test_answer PASSED
main.py::test_answer_negative PASSED
---------- coverage: platform darwin, python 3.5.4-final-0 -----------
Name Stmts Miss Cover
-----------------------------
main.py 8 0 100%
Coverage overview: work towards 100%
Use coverage to help you identify missing edge/corner cases
Next: continuous integration
Meaning: ensure all changes to your project pass tests through automated test & build process
Getting started with Travis CI:
- Sign in at travis-ci.org with your GitHub account
- Go to your profile and enable the repository for the software you want to build & test.
-
Add a
.travis.ymlfile to the repository -
Add info to configuration file telling Travis CI what to do:
what programming language (
language: python), what it needs to build your package (install), and how to run the tests (language: script)
Using pip
language: python
python:
- "3.5"
- "3.6"
install:
- pip install -r requirements.txt
script:
- pytest -vv --cov=./;
Contents of requirements.txt
#
####### requirements.txt #######
#
###### Requirements without Version Specifiers ######
pytest
pytest-cov
#
###### Requirements with Version Specifiers ######
# See https://www.python.org/dev/peps/pep-0440/#version-specifiers
numpy >= 1.12.0
pint >= 0.7.2
#
Using conda
language: python
python:
- "3.5"
- "3.6"
install:
- if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh;
elif [[ "$TRAVIS_OS_NAME" == "osx" ]]; then
wget https://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-x86_64.sh -O miniconda.sh;
fi
- bash miniconda.sh -b -p $HOME/miniconda
- rm miniconda.sh
- source $HOME/miniconda/etc/profile.d/conda.sh && conda activate
- conda config --set always_yes yes --set changeps1 no
- conda update -q conda
- conda update -q --all
- conda config --append channels conda-forge
- conda env create -qf test-environment.yaml;
- source activate test-environment;
script:
- pytest -vv --cov=./;
Contents of test-environment.yaml
name: test-environment
channels:
- defaults
- conda-forge
dependencies:
- python=${PYTHON}
- numpy>=1.12.0
- pytest>=3.2.0
- pytest-cov
- pint>=0.7.2
Getting started with Travis CI (contd.)
-
git add .travis.yml test-environment.yaml, commit, and push to GitHub - Check the build status page to see if the build passes or fails
-
(Optional) Add a status badge to your README:
(Get this from your Repository page on Travis CI)
GitHub will use Travis CI to tell you if Pull Requests are safe to merge!
Last: documentation
Professor Carole Goble in “Better Software, Better Research”:
One of my favorite #overlyhonestmethods tweets (a hashtag for lab scientists) is Ian Holmes’s “You can download our code from the URL supplied. Good luck downloading the only postdoc who can get it to run, though.”
Value of documentation
- The value and extent of your work is clearer if it can be understood by colleagues.
- Documentation provides provenance for your scientific process, for your colleagues and yourself.
- Documentation demonstrates your skill and professionalism.
Documentation is easier than you think.
- Documentation pays for itself with the time it saves in the long run.
- Documentation requires little effort beyond writing the software itself.
Types of documentation
- Theory manuals
- User and developer guides
- Code comments
- Self-documenting code
- Generated API documentation
User and developer guides
README: sits in top-level directory and
contains all the necessary information for installing, getting started with, and understanding the accompanying code.
May be accompanied by other specific files:
LICENSE, INSTALL, CITATION, ABOUT, CHANGELOG
README example
SQUIRREL, version 1.2 released on 2026-09-20
# About
The Spectral Q and U Imaging Radiation Replicating Experimental Library
(SQUIRREL) is a library for replicating radiation sources with spectral details
and Q and U polarizations of superman bubblegum.
# Installation
The SQUIRREL library relies on other libraries:
- The ACORN library www.acorn.nutz
- The TREEBRANCH database format API
Install those before installing the SQUIRREL library. To install the SQUIRREL
library:
./configure
make --prefix=/install/path
make install
...
Comments
Comments provide a way to insert metainformation about code intended for people, right next to the code:
def the_function(var):
"""This is a docstring, where a function definition might live"""
a = 1 + var # this is a simple comment
return a
Bad Comments
Also possible to pollute code with unnecessary cruft:
def decay(index, database):
# first, retrieve the decay constants from the database
mylist = database.decay_constants()
# next, try to access an element of the list
try:
d = mylist[index] # gets decay constant at index in the list
# if the index doesn't exist
except IndexError:
# throw an informative error message
raise Exception("value not found in the list")
return d
Useful Comments
Code written cleanly will have its own voice. Use intelligent naming to make most lines of code clear without comments, then use comments sparingly to help explain reasons or complicated sections:
def decay(index, database):
lambdas = database.decay_constants()
try:
lambda_i = lambdas[index] # gets decay constant at index in the list
except IndexError:
raise Exception("value not found in the list")
return lambda
Self-Documenting Code
Naming: a class, variable, or function name should tell you why it exists, what it does, and how it is used.
Simple functions: functions should be small to be understandable and testable; they should only do one thing.
Consistent style: use a consistent, standardized style; e.g., select variable and function names according to the PEP8 style guide for Python.
Guidelines for naming:
# packages and modules are short and lowercase
packages
modules
# other objects can be long
ClassesUseCamelCase
ExceptionsAreClassesToo
functions_use_snake_case
CONSTANTS_USE_ALL_CAPS
# variable scope is *suggested* by style convention
_single_leading_underscore_ # internal to module
single_trailing_underscore_ # avoids conflicts with Python keywords
__double_leading_and_trailing__ # these are magic, like __init__
Docstrings
docstring: comment placed immediately after a function or class definition, typically enclosed by three pairs of double quotes:
def <name>(<args>):
"""<docstring>"""
<body>
docstrings are available within Python via help() and
iPython's magic command ?, and Sphinx picks them up.
Docstrings (more)
Make docstrings descriptive and concise; you can explain the arguments of a function, its behavior, and how you intend it to be used.
def power(base, x):
"""Computes base^x. Both base and x should be integers,
floats, or another numeric type.
"""
return base**x
Sphinx: automate generating documentation
Sphinx can be used to automate the generation of HTML documentation; we can even use it with Travis CI to automatically build and deploy the docs on GitHub Pages.
For now, let's just make sure your docstrings are suitable for Sphinx.
Numpy-Style Docstrings
def function_with_types_in_docstring(param1, param2):
"""Example function with types documented in the docstring.
`PEP 484`_ type annotations are supported. If attribute, parameter, and
return types are annotated according to `PEP 484`_, they do not need to be
included in the docstring:
Parameters
----------
param1 : int
The first parameter.
param2 : str
The second parameter.
Returns
-------
bool
True if successful, False otherwise.
.. _PEP 484:
https://www.python.org/dev/peps/pep-0484/
"""
More examples at the Sphinx documentation
Google-Style Docstrings
def function_with_types_in_docstring(param1, param2):
"""Example function with types documented in the docstring.
`PEP 484`_ type annotations are supported. If attribute, parameter, and
return types are annotated according to `PEP 484`_, they do not need to be
included in the docstring:
Args:
param1 (int): The first parameter.
param2 (str): The second parameter.
Returns:
bool: The return value. True for success, False otherwise.
.. _PEP 484:
https://www.python.org/dev/peps/pep-0484/
"""
More examples at the Sphinx documentation
Get started with sphinx
-
mkdir docs -
cd docs -
sphinx-quickstart(accept defaults if unsure; answer "yes" for question aboutautodoc) -
sourcedirectory holds.rstfiles for user guides, theory manuals, etc. separate from the autogenerated API documentation