Remote Logging to Azure Blob Storage

Firt of all, Remote Logging to Azure Blob Storage works with Airflow Version >= 3.0.5, therefore make sure you're using that. We tested it with Airflow 3.0.6:

### Blob Logging works only on Airflow >= 3.0.5
images:
  airflow:
    repository: apache/airflow
    tag: "3.0.6"
  pod_template:
    repository: apache/airflow
    tag: "3.0.6"

Before you begin, make sure you have Azure Storage Account with Blob Containers ready!

Afterwards, create Azure KeyVault secret containing WASB Connection Settings, following this format:

# Variant One:
{"conn_type": "wasb", "login": "__STORAGE_ACCOUNT_NAME__", "password": "__STORAGE_ACCOUNT_SHARED_KEY__"}
# Variant Two (Recommended):
{"conn_type": "wasb", "host": "__STORAGE_ACCOUNT_HOST__", "login": "__STORAGE_ACCOUNT_NAME__", "password": "__STORAGE_ACCOUNT_SHARED_KEY__"}

command template:

az keyvault secret set --vault-name __KEYVAULT_NAME_GOES_HERE__ -n apache-airflow-blob-log-conn --value '{"conn_type": "wasb", "host": "__STORAGE_ACCOUNT_HOST__", "login": "__STORAGE_ACCOUNT_NAME__", "password": "__STORAGE_ACCOUNT_SHARED_KEY__"}'

Then, create file airflow-blob-logging-conn-secret.yaml containing ExternalSecret with the connection details:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: airflow-blob-logging-conn-secret
spec:
  secretStoreRef:
    name: secret-store-keyvault
    kind: SecretStore
  target:
    name: airflow-blob-logging-conn-secret
    template:
      data:
        AIRFLOW_CONN_WASB_LOGGING: "{{ .airflowAzureBlobLogConn | toString }}"
  data:
    - secretKey: airflowAzureBlobLogConn
      remoteRef:
        key: apache-airflow-blob-log-conn

in your overlay/ENV and then enable it for all Pods in values-override.yaml:

# Auth for Blob Storage
# The naming convention is AIRFLOW_CONN_{CONN_ID}
extraEnvFrom: |
  - secretRef:
      name: airflow-blob-logging-conn-secret

After doing so, you can set up Remote Logging via Environment Variables as follows:

env:
  # Enable Remote Logging
  - name: "AIRFLOW__LOGGING__REMOTE_LOGGING"
    value: "True"
  # Delete Local Logs when Remote Logging is Enabled
  - name: "AIRFLOW__LOGGING__DELETE_LOCAL_LOGS"
    value: "True"
  # Directory inside Blob Container
  # needs to start with 'wasb' or 'wasb/' or 'wasb://'
  - name: "AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER"
    value: "wasb/logs"
  # Blob Container Name
  - name: "AIRFLOW__AZURE_REMOTE_LOGGING__REMOTE_WASB_LOG_CONTAINER"
    value: "airflow-dev-logs"
  # Auth for Blob Storage
  # The naming convention is AIRFLOW_CONN_{CONN_ID} (lowercase value here)
  - name: "AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID"
    value: "wasb_logging"

Please note that all these variables are mandatory! After following these steps, it should work.

IMPORTANT There are some caveats:

1. Value of AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER Docummentation here suggest that the path should be wasb://path/to/logs however for some reason on version 3.0.6 it's NOT true! Following that advice creates wasb:/noname/path/to/logs directory! Common sense would also suggest value like wasb://[email protected]/airflow-logs but it doesn't work as well. It would create wasb:/noname/[email protected]/airflow-logs directory!
2. According to Soruce Code of Airflow, value of AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER HAS to start with wasb or wasb/! See here for version 3.0.6.
3. Value of __AIRFLOW__LOGGING_REMOTE_LOG_CONN_ID when creating Airflow Conenction via Environment Variable must be lowercase! For example, when the variable with connection params is AIRFLOW_CONN_REMOTE_LOGGING then variable value will be remote_logging.