We recently merged a change to allow configuring the loggers more selectively in an experiment. It’s a small step in the direction of improving our logging in general, but is already proving useful.
Python supports configuring loggers using configuration files, or directly using dictionaries (anyone familiar with Django may have experience with this latter approach). Since we already use Hydra to configure our experiments using YAML, it’s easy to add this to our existing logging configuration, without requiring a full refactor of the rest of how that works just yet.
For example, some of the work I’m doing on the MuJoCo simulator has involved tracing what is happening, and we have some decent debug logging for the areas I’m working on. Turning on DEBUG for the entire system would be far too noisy, so I want to select just the modules (and their children) that I want to see logs for.
With these changes, I can do this by adding a few lines to my experiment configuration.
experiment:
config:
...
logging:
...
loggers:
# Turn logging on for all of the MuJoCo modules
tbp.monty.simulators.mujoco:
level: DEBUG
# Enable some motor policy logging
tbp.monty.frameworks.models.motor_policies:
level: DEBUG
Because of the way logging.config.dictConfig works, it ignores the custom keys we usually have in the logging section and just focuses on the ones it recognizes.
More detailed documentation is available in the Python docs. Worth noting though, it currently uses incremental configuration which limits what we can do. We’d need to migrate the rest of our logging setup to use this to be able to unlock those extra capabilities.
Let us know if you run into problems using this feature.