I’ve been reading through the documentation on unsupervised continual learning and exploring the codebase to understand the current logging infrastructure.
From what I can tell, several useful metrics already exist in the experiment outputs and logging utilities, including metrics related to:
graph/object associations
matching behavior
number of possible matches
graph memory statistics
unsupervised learning performance
Before I start implementing anything, I wanted to ask a couple of questions:
Is this roadmap item expected to go through the RFC process, or would a direct pull request be appropriate?
Are there specific metrics, visualizations, or WandB dashboards that maintainers would particularly like to see added for unsupervised continual learning experiments?
Is there an existing design or direction for this task that I should follow?
My current thought is to focus on exposing existing unsupervised-learning metrics through WandB rather than introducing new experiment logic, but I’d appreciate guidance before getting started.
I don’t think this warrants an RFC. You can probably just do this via a direct pull request.
Regarding what metrics to add, @vclay pointed out that a good place to start would be adding the metrics that we report in our unsupervised learning benchmarks: Benchmark Experiments
From the coding perspective, I think this will require reading additional properties from data["BASIC"] and or data["DETAILED"] inside the WandB handlers in wandb_handlers.py, but I haven’t looked at it in detail, so perhaps more complexity will surface as you work on it.
One consideration I would keep in mind is to see if these can be added without adding any, or as little as possible, new methods elsewhere in Monty. The reason being is that I hope we can move away from this logging path next quarter (26Q3) toward the approach described in 333 Telemetry collection is configurable down to the Python module level.
Just to add to Tristan’s response and to save you some time searching: The metrics reported in the unsupervised learning benchmarks tables are currently obtained by running the print_unsupervised_stats command on the saved .csv table (Running Benchmarks). It would be amazing to have these logged to wandb! Thanks for your interest in contributing
The PR focuses on logging the unsupervised benchmark-style metrics currently produced by `print_unsupervised_stats`, while keeping the implementation contained in the existing WandB logging path as much as possible.
I also avoided changing learning behavior, matching logic, graph construction, or experiment logic.
I would appreciate any feedback on whether the metric names and implementation approach match what you had in mind.
