Skip to content

Instantly share code, notes, and snippets.

@yene
Last active October 16, 2024 18:33
Show Gist options
  • Select an option

  • Save yene/9b8dc8235fd308b51de6c173628a06d5 to your computer and use it in GitHub Desktop.

Select an option

Save yene/9b8dc8235fd308b51de6c173628a06d5 to your computer and use it in GitHub Desktop.
pinning python dependencies and checking hash for airflow

Warning

This may faile with some packages. We recommend to just use uv and create the requirements.txt with uv export

Airflow provides a constraints file with the exact versions used. Here is how we can make the installation (almost) reproducible by relying on the hashes.

requirements.in holds the dependencies with constraints.

# airflow and constraints
apache-airflow[celery]==2.9.0
-c https://raw.githubusercontent.com/apache/airflow/constraints-2.9.0/constraints-3.8.txt

with pip (and pip-tools)

# create requirements.txt with hashes and exact versions by provided constraint.
# The requirements.txt is commited to Git.
pip-compile --generate-hashes --allow-unsafe
# Install and verify hashes, will exit if hashes have a missmatch. 
# This step is usually run during docker build.
pip install --require-hashes -r requirements.txt

with uv

# create requirements.txt with hashes and exact versions by provided constraint.
# The requirements.txt is commited to Git.
uv pip compile requirements.in -o requirements.txt --generate-hashes
# Install and verify hashes, will exit if hashes have a missmatch. 
# This step is usually run during docker build.
uv pip install --require-hashes -r requirements.txt
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment