While trying to reproduce Monty’s compositional-object recognition (the lvl2 comp models from pretrained_compositional_objects_v3), I looked inside the high-level LM’s graph memory — specifically the composition of object_id nodes stored in its learning_module input channels, which I’d like to call the parent’s “BOM” (bill of materials), if that’s OK. What I found surprised me (figure below):
The logo child is almost never written in any of the 10 logo-bearing parents (e.g., 002_cube_tbp_horz stores 368 cube nodes and just 1 logo node; several store none at all);
The disk family is heavily cross-contaminated with cube (44–51% in the composites — 007_disk_tbp_horz actually stores more cube nodes than disk).
So the child channels don’t seem to carry the information that distinguishes these composites from their bare shapes; whatever discrimination happens appears to rely on the high-level LM’s own coarse patch.
Reading the code (EvidenceGraphLM.get_output), child-channel content comes from the lower LM’s output, which only becomes usable once the lower LM reaches a match — a unique candidate with a settled pose (I also noticed the official “NOTE: May want to relax this check” comment right there). A small, flat logo that is coplanar with its host face seems to have a very hard time crossing that gate, which would explain why it never makes it into the parent’s graph.
The Hierarchy-or-Heterarchy paper describes the higher-level LM as consuming the lower LM’s “object ID at pose” as features (arXiv:2507.05888). So my questions: what role does pose precision play in these failures? What does the “settled pose” requirement in the match/upload criterion mean for small, flat child objects? Has the team run ablations on pose tolerance here?
I may well be misreading something — corrections very welcome. Happy to share the exact dissection steps if useful.
These are all great points, thank you for bringing them up. The results you found are consistent with what we’d expect from the configs used in your experiments. Note that:
In the commit edc597c402e4, we have removed the lvl2 configs from Monty and replaced it with base_infer_objects_with_stickers_comp_models.
It is to be expected that the lower-level LM does not switch between child objects since these configs are missing burst sampling.
These configs are also missing recent work from @hlee on the 2D sensor module which allows a lower-level LM to learn and infer the logo on the surface of the object by processing the perceived edges as morphological features. With the 2D SM + 3D SM working together, the heterarchy will look more like the image below.
We are currently working on rolling out these updates to the configs. We’re also working on other policy improvements and changes to how we represent symmetry in a learning module. These changes aim to improve the compositional experiments performance, so stay tuned!
Thank you @rmounir, this is really helpful — and thanks for the pointer on the newer configs. I was dissecting the deprecated lvl2 set, so I’ll re-run against base_infer_objects_with_stickers_comp_models before drawing further conclusions.
I still want to know about this: I kept running into rotationally-symmetric child parts (e.g. a cylinder) rarely reaching a settled pose, which felt related to the get_output “May want to relax this check” note. Is the symmetry change aimed at letting such parts contribute their identity without a unique pose, or is it addressing something separate?
No rush at all — really appreciate you sharing the roadmap, I’ll keep following along.
Good question. The symmetry change we are working on will definitely help with this issue. The main problem is that with symmetric objects, there isn’t one correct pose for an object, and the lower-level LM sends up one of the possible symmetric poses arbitrarily. That means the higher-level LM will have a hard time relying on the pose that it receives. I wrote up a slightly more detailed description of the problem and current thoughts on how we could address this here, in case you are curious:
This solution would have the nice effect that we wouldn’t be testing symmetric poses in the lower-level LM anymore, and therefore also wouldn’t take a long time to converge/rediscover that the object is symmetric.
You are working at the cutting edge of what Monty is currently capable of, and as you can see, we are currently making changes on at least 4 aspects of the compositional modeling implementation (configs, policy, symmetry, feature and movement input channels). There will be a lot of changes related to those benchmarks in the next weeks, so stay tuned for updates (and maybe don’t expect too much of them at the moment, we are addressing remaining shortcomings)
