Logging can become as complicated as your imagination permits in Python. There are modules in the standard library that cater for all scenarios I have encountered and probably many more I have not.
The documentation has extensive sections on logging. The key materials (in rough order of reading) being:
- Logging HOWTO
- Logging Cookbook
loggingmodulelogging.configmodulelogging.handlersmodule
In an attempt to distill these options to that should just work, I am documenting my own logging recipes here.
In all cases, the best way I have found for providing configurable logging is through a YAML configuration file. Perhaps once I have tested these options for many years this flexibility might be unnecessary but for debugging a complex Python application I find flexibility to be essential.
For parsing YAML the best library I have used by far is ruamel.yaml.
t has recently migrated development to SourceForge which I find
particularly painful to navigate. I hope it migrates to something less
trashy like Gitlab/Github soon.
pip install ruamel.yamlimport ruamel.yaml- Documentation
- Source
We don't care what else is in this file, so long as a logging key
exists at the root which contains a dictionary value matching Python's logging.config.dictConfig(config) schema.
This style applies to an application run by a human, typically in user space. It may require root privileges but a human runs the program and wants to see what is happening as it happens, and gets a log file to avoid messing around with pipes and file names.
This configuration starts by printing the location of the log file (pre- and post-date-expansion) to ensure the user knows where to find the log file. This location can be tweaked and any date format strings will be expanded. The directory will be created if required so grouping log files by year, month, or day becomes easy.
The default is that time-only-stamped information goes to stdout,
whilst the log file gets extra debug information (including the Python
module, process and thread IDs). This can all be tweaked in the
configuration without needing any changes to the program code.