Python Packaging Best Practices

Python is a versatile and popular programming language. According to the TIOBE index, it is currently the world's most popular programming language. It has found extensive use and engineering and scientific applications, including robotics. Python is popular because it has clear/flexible syntax, a "batteries included" standard library, easy interfacing to native code, and a massive ecosystem of packages providing additional functionality. This document provides an overview of best practices for packaging Python robotics software for distribution.

There are many tutorials, blog posts, and whitepapers on Python packaging. This guide is intended to provide a quick overview of the most important aspects of packaging, with references to more detailed guides.

Python Package Formats

Python packages typically bundle the Python *.py files, resources files, and any additional native modules/libraries used by the package. Native modules and libraries are files containing machine code that can only operate on specific systems. These modules are typically developed using C/C++, Rust, Fortran, or other native compiling software language. The native code can be interfaced with Python using a number of methods, include ctypes, cffi, manual programming, or generative tools like SWIG.

Python packages can either be "source" or "binary".

• Source packages: Contain Python script files, resource files, and the native source code for any native modules. Source packages typically compile on installation on a users computer.
• Binary packages: Contain Python script files, resource files, and native modules/libraries. These packages are tied to specific operating systems and computer architectures.

For "pure python" packages, the distinction between source and binary is less strict, and typically a single package can run on all systems with a high enough Python version that supports all the required dependencies.

Python Module Design

Python packages should be designed carefully so that they are easily reused. Avoid unpredictable interactions between packages by being very deliberate in the design of the API and how dependencies are used.

Pure-Python Packaging Best Practices

This document focuses on the packaging pure-Python packages. These packages contain only Python script files (*.py) and resource files. Because there are no native modules, the package should run on any operating system and system architecture with a recent enough Python version that supports all the required dependencies.

See Also

• https://python-packaging.readthedocs.io/en/latest/minimal.html
• https://packaging.python.org/en/latest/
• https://setuptools.pypa.io/en/latest/userguide/index.html
• https://docs.readthedocs.io/en/stable/tutorial/index.html
• https://godatadriven.com/blog/a-practical-guide-to-setuptools-and-pyproject-toml/

Example Packages

The following packages will be used as examples:

• abb_motion_program_exec
• general_robotics_toolbox
• drekar-launch

PyPi and pip

PyPi (https://pypi.org) is the central repository for Python packages. The related tool pip is used to install packages, either from local files or the PyPi repository. It is included with Windows, but the python3-pip package may need to be installed on Linux.

Package File Structure

A python package will typically have the following package structure. In this example, the package name is my-example-package and it contain the module my_example_module

• src/
  • my_example_module/
    • __init__.py
    • my_submodule.py
    • my_other_submodule.py
• docs/
  • conf.py
  • index.rst
  • requirements.txt
  • example_module/
    • sphinx documentation files
• test
  • test_my_example_module.py
• examples/
  • examples using the module
• scripts/
  • scripts used during development
• .gitignore
• .readthedocs.yaml
• LICENSE
• README.md
• setup.py (and/or) pyproject.toml

This file structure represents one possible directory structure, but it is far from the only one possible. See the many other packaging guides for details on other possible options.

This package can be managed using pip and uploaded to PyPi if desired. Once installed, the package modules can be imported:

from my_example_module import my_submodule, my_other_submodule

These submodules are now imported. The contents of the __init__.py file will be placed in the my_example_module root package.

src/

The src/ directory contains the Python module and scripts. In this case, the my_example_module folder will be packaged. Some developers omit the src directory and have the module directory in the root of the repository. However, experience has shown it is better practice to use a src/ directory.

README.md

The readme file may be the most important file in the repository. It should give the user an overview of what the package does a quick guide of how to use it.

Common sections of a readme file:

• Introduction
• Documentation
• Installation
• Getting Started / Quick Start / Usage
• Examples
• Building (if applicable)
• License

Even if there is no additional documentation, always include a good readme!

examples/

The examples directory should contain simple examples demonstrating how to use the module.

scripts/

When developing a module for a scientific project, there will often be scripts that are executed to accomplish some task that use the module. Place these general purpose scripts in the scripts/ directory, or make the module executable.

An better option to a scripts/ directory is to make modules executable. Running:

python -m my_example_module

will invoke the __main__.py module in the directory next to __init__.py. If developing a package that will be called from the command language, the -m option in a package is the preferred way to distribute. See also "entry points" in pyproject.toml or setup.py that can create callable command line programs from modules.

tests/

pytest is the most common testing framework used for Python. It will scan the test directory and find methods and classes that start the test and execute each one as a unit test. See Effective Python Testing With Pytest.

Python tests can be automatically executed by GitHub when push using GitHub Actions. See Building and testing Python for more information on Python testing on GitHub. Also see the example modules GitHub workflow files.

.gitignore

All Python packages should use Git, and regularly push to a Git repository for backup and sharing. The .gitignore file specifies files that should not be committed to git. A template gitignore can be found on GitHub: https://github.com/github/gitignore/blob/main/Python.gitignore

LICENSE

A standard license should be used. For open-source robotics packages, the BSD 3-clause or Apache 2.0 are the most common:

• https://opensource.org/license/bsd-3-clause/
• https://www.apache.org/licenses/LICENSE-2.0

setup.py and pyproject.toml

setup.py and pyproject.toml contain the metadata and building instructions for the package. For pure-python packages running on Python 3.6 or greater, only a pyproject.toml file is required. See the example and links above for more information on creating a pyproject.toml.

setup.py can be used to compile or package native modules. This is an advanced use case and the specific solutions used by different developers can vary widely.

If non-Python resource files are used, it is necessary to specify they should be included.

See A Practical Guide to Setuptools and Pyproject.toml.

.readthedocs.yaml and docs/

Read-the-docs is a service that automatically compiles and serves documentation. The documentation is compiled using sphinx, a popular tool for documenting Python and other software languages. Sphinx uses Python Docstrings to document classes and methods. Docstrings are text literals in the source code that describes the class/method, and any parameters/return/exceptions. Sphinx can use a tool called autodoc to read these docstrings and generate the appropriate files.

The contents of the docs/ folder contains the files required to build the documentation. conf.py contains the instructions to build the documentation. The documentation can contain both API documentation and general instructions on how to use the package. See the Read the Docs tutorial and the example modules for instructions on using Read the Docs.

The example modules have additional code in conf.py to include the README.md file. This is recommended

Developing the Module

Modern Python development uses "virtual environments". Virtual environments create an isolated Python installation where packages can be installed and removed without affecting the system installation. All installed packages are local to the virtual environment, and different modules can have different virtual environments with different packages installed.

Create a virtual environment in the root of the repository:

python -m venv venv

This directory is ignored by the default .gitignore file so it will not be committed. Linux users may need to install python3-venv.

The venv can now be activated:

Windows:

venv/Scripts/activate

Linux:

source venv/bin/activate

The path of the local prompt is modified so that the local virtual environment is called when python is called.

Now, the module under development can be "editable" installed.

python -m pip install -e .

This means that the Python virtual environment will point to the files in scr/, rather than copying into venv/site-packages. This makes any changes to the files in src/ will apply immediately.

Any scripts executed while the virtual environment is active will use the source files in src/ for the module.

See also Python Virtual Environments: A Primer

Distributing the Package

There are several popular Python distribution options:

• Local install
• Direct install from Git
• PyPi (https://pypi.org/)
• Anaconda.org aka conda (https://anaconda.org/)
• Debian/Ubuntu Apt
• ROS

This section focuses on distribution and installation using pip.

The simplest installation option is to use the unpacked repository. If the repository is local, it can be simply installed:

python -m pip install .

This will copy the files to site-packages of the Python installation.

Another option is to install directly from the GitHub repository. For instance, to directly install the general_robotics_toolbox run:

python -m pip install git+https://github.com/rpiRobotics/rpi_general_robotics_toolbox_py.git

Keep in mind in these use cases, pip cannot automatically resolve dependencies that are not on pypi.org. They will need to be installed in order if only available on Git.

Uploading to pypi.org requires a "wheel" archive of the package. This can be accomplished using the "build" package:

Install the "build" and "twine" packages (one time only per virtual env)

python -m pip install build twine

Build the package:

python -m build

Upload the wheel to PyPi.org:

python -m twine upload dist/*

See Packaging Python Projects for more information.

Once uploaded, the package can be installed using pip from any machine:

python -m pip install my-example-package

Distributing Python packages for ROS is typically accomplished by adding a link to the Python PyPi project. See https://github.com/ros/rosdistro/blob/master/rosdep/python.yaml

Python Packaging Best Practices

• Always include a README.md
• Carefully design public API interface (classes and methods)
• Use the formatting specified in PEP 8 - Style Guide for Python Code
• Use a src/ directory
• Use a unique and descriptive name for packages
• If packages are not widely used, host on GitHub or a private package repository
• Always use docstrings and type annotations where possible
• Use a pyproject.toml or setup.py to define dependencies. If not relevant, use a requirements.txt file
• Do not copy and paste code! Create reusable functions!
• Support Python 3.7 or greater
• For newer projects, use a pyproject.toml instead of setup.py if possible
• Use a documentation system like Sphinx and/or Read the Docs to keep documentation up to date
• Include additional instructions beyond the API documentation
• Include the readme in the full documentation
• Always use GitHub Actions to test your code an the supported platforms
• Use file objects to read and write files rather than passing filenames
• Use "relative" imports
• Test in all configurations the software will run
• Do not modify sys.path! Use the packaging system to place the packages in the correct locations