Over the past month or so I have been working on recognizing compositional objects. I stepped into quite a few pitfalls, made some findings, and tried some new things — I’m writing them up here to ask for your thoughts. One scoping note up front: what I’m attempting is not the logo-sticker line of compositional objects, but the kind assembled from 3D geometric parts; the logo line is in my future plans.
1. The full library of compositional objects we built
The ready-made composites in the official compositional dataset are mostly logo stickers (flat prints attached to a carrier object); composites assembled from 3D geometric parts are sparsely covered. But the questions I wanted to answer systematically are of the “parts × spatial relations” kind — which ways of assembling parts are easy to recognize, which are hard, and where exactly the pipeline breaks. Three or five composites are not enough for that: early on I repeatedly experienced “tuned it on one composite, broke on the next”, and realized I needed a fixed exam paper.
So, using the basic parts from the official dataset as blueprints (cube, cylinder, sphere, disk, washer, mug, cone, triangular prism — 16 kinds in total), I procedurally assembled a full library of 48 composites, each made of 2–3 parts. The exam dimensions include:
- Stacking (three parts stacked, e.g. cylinder + cube + sphere);
- Side by side (a mug next to a cone);
- Nesting (a sphere sitting inside an open box);
- Piercing (a rod passing through a cube);
- Size contrast (a large cube + a 4.5 cm small sphere);
- The hole family (disk, washer, torus — parts distinguished mainly by their holes);
- Similar families (a deliberately designed exam point: multiple composites share parts, and some are even in subset relations — “cube + cylinder” is a proper subset of “cube + cylinder + sphere”).
The exam protocol is fixed: each composite is recognized once at each of three rotations (0°, 90°, 180°) — all numbers below use this protocol — 144 episodes in total.
Figure 1: Five representative composites — stacking, nesting, side-by-side, size contrast, piercing.
2. The current setup: the official compositional architecture + a few modifications
The base architecture follows the official one exactly: child learning modules recognize parts and report part identities upward; the parent LM uses the official evidence mechanism to accumulate “which part appears at which location” in the composite’s reference frame, and finally decides the composite’s identity.
One premise I must state honestly: the current version skips the learning phase. The official tutorials describe supervised pretraining as an experimental control — “ensuring the model learns the correct models so we can focus on the inference phase”. I pushed that idea to its extreme: I synthesize the parent and child object models directly from the parts’ mesh geometry (I call this “perfect assembly”), fully decoupling the recognition problem from the learning problem so I could first get recognition right. The next step is to return to real learning (see §5 for why — that decision is itself one of the biggest takeaways of this project).
On this base, I made several modifications to the recognition side. Each one corresponds to a concrete, measured problem:
- Handling feature mismatches in child-LM evidence scoring. I measured that “large-envelope parts” (cube, open box and the like) always pick up a little evidence even on other parts’ surfaces, while the official feature evidence has a lower bound of 0 (a mismatch simply adds nothing) — so an incumbent wrong hypothesis is never penalized. I introduced a bounded negative contribution for consistent feature mismatches, specifically to treat this “wrong incumbent feeding on its size” capture phenomenon.
- Curvature-conditioned evidence memory speed. Evidence accumulates with very different dynamics on curved versus flat segments; I switch the evidence memory weights based on the currently sensed surface curvature — decaying memory on curved steps, official behavior unchanged on flat steps.
- Motion pattern: continuous sweeping with turn-backs at part edges, replacing large-jump sampling. This came from a lesson that took a long time to pin down: the official displacement machinery assumes continuous small-step sampling, and large jumps displace the pose hypotheses off the model’s surface entirely.
- Repositioning to look again: when the parent race shows “the leader keeps flipping” (a behavioral signal of an undecided race), trigger a reposition around the object, steered by elevation toward surface bands not yet seen; paired with a “reacquire the object” guard (after consecutive off-object steps, look back at the center of recent sensed locations, or jump back to the pre-reposition pose).
- Two platform-level fixes. First, “getting stuck inside an object”: a jump action occasionally lands the camera inside a part’s geometry, and the official JumpToGoal post-jump check only asks whether the depth at the image center is ≥ 1 meter — inside an object the center depth is close to 0, so the jump is judged as a fine view and is not undone (the official code carries a TODO comment saying this should be replaced with a percept.on_object check; I implemented the fix in that direction, and I verified that this check is still unchanged on the latest main — happy to write this up as feedback if useful). Second, state self-check and cleanup at episode boundaries.
- The current setup is single sensor stream (the parent listens to one child channel). I did experiment with feeding multiple sensor streams into the parent; the problems I ran into deserve a separate write-up, so I won’t expand here.
3. Results
The headline number (object-level recognition accuracy, full library): 48 composites × 3 rotations = 144 episodes, 71 recognized correctly — 49.3% accuracy; 10 of the 48 composites were correct at all three rotations. Every number in this post comes from the same code and the same exam protocol.
A supporting number (part-level readings, diagnostic subset): beyond the object-level outcome, I also wanted to know how often the child LM reads the right part name for whatever part the sensor is currently standing on — this is a metric at a different level, not a second accuracy. Since per-step recording is expensive, it is done only on a diagnostic subset of 8 composites: part-identity readings inside composites are 64.4% correct (2758/4285 steps, sensed locations attributed to ground-truth parts); as a reference, the same parts presented in isolation read at 83.3% (6323/7587 steps). The same part, once placed inside a composite, reads markedly worse — and that gap is the direct source of the failure classes in §4.
(The repositioning ablation was also run on this subset: with repositioning turned off, in-composite part readings drop from 64.4% to 60.1%, while object-level results are 15/24 vs 16/24 — a ±1 difference within the jitter we see across repeated experiments. Repositioning’s net contribution is mainly in part readings and time-on-object; for composites like the three-part stack, sensor wandering time drops substantially.)
Two correctly recognized examples (our standard diagnostic figure; the four panels are: ① the composite’s true topology; ② the object model; ③ sensed locations during the test, colored by the child LM’s readings; ④ the parent’s evidence trace):
Figure 2: Sphere + cylinder, recognized correctly at all three rotations. In panel ④ the correct model’s evidence leads for most of the episode.
Figure 3: Triangular prism + sphere, recognized correctly at all three rotations.
Reproduction materials: the 48 composite meshes, the perfect-assembly records, stock-official configs that run directly on current tbp.monty main, and the scoring script are published at monty-realworld/compositional (data and records in the accompanying Release). Running them gives an official-baseline score — the exam is public, and you are welcome to bring your own ideas to it; the mechanism code is available on request.
4. Why recognition fails (and questions for the team)
The failures concentrate into four causes. Two typical examples:
Figure 4: Large cube + 4.5 cm small sphere, wrong at all three rotations. Panel ③ shows the sensed locations landing almost entirely on the large cube — the small sphere collected only 7 sensed locations across all three rotations, with zero correct part readings; panel ④ shows the correct model never rising to first place in the parent race (0/259 steps). The parent knows “this is a large cube plus something small”, but inside the “cube + small part” family it can only keep guessing wrong.
Figure 5: Cube + cylinder, judged as “cube + cylinder + sphere” (its proper superset). While moving on the subset object, every observation is compatible with the superset’s model — the extra sphere is merely “not seen”, and in the official evidence mechanism “not seen” does not count as evidence against.
The four failure causes:
- The observation-scale problem for small parts (Figure 4): at the standard viewing distance the patch is essentially filled by the large part; the small part gets almost no sensed locations, so the child LM never has a chance to report it.
- Similar families / the subset blind spot (Figure 5): families of composites that share parts, especially subset relations — every observation on the subset object supports the subset and the superset models alike, and the superset wins by having more scoring opportunities. I did try, in near-tie situations, actively repositioning to the predicted location of the superset’s extra part to verify its absence; the mechanism works (absence detection and penalties both function), but the verification jump itself perturbs those near-tie races that were about to be won — the gains and the perturbation cost roughly cancel.
- Context carried across parts: after the sensor walks from part A to part B, the lead accumulated for A in the child’s evidence race suppresses B’s correct hypothesis — even when B is being observed on its most recognizable surface band. I tried a family of “race state treatments” here (boundary resets, lead capping, etc.); within a single-race framework I could never keep both “long-range accumulation” and “cross-part isolation” at once.
- Trajectory sensitivity of near-ties: races with close evidence are highly sensitive to small changes in the observation sequence; any mechanism that alters the trajectory (even one meant to gather more information) reshuffles the outcomes of these races.
Questions for the team:
- Q1: For subset/superset families (like Figure 5), is there a recommended way to discriminate within the evidence framework? For example some form of “absence evidence” (a negative contribution when the expected part is not observed at its predicted location)? As far as I can tell the evidence-related future-work items don’t directly cover this scenario.
- Q2: Is there an official vision for handling child-LM evidence across parts (across objects)? The future-work item “rapid evidence decay as unsupervised memory resetting” discusses objects changing in the environment — can walking from one part to another inside a composite be seen as a miniature version of the same problem?
- Q3: Small parts and observation scale: when a part is much smaller than the patch (Figure 4), is there a recommended multi-scale / zoom observation scheme?
- Q4: We implemented the JumpToGoal post-jump TODO (switching to a percept.on_object check), and in our tests it eliminates the “stuck inside an object after a jump” failure. It is now submitted as a PR (tbp.monty#1160) — reviews welcome.
5. Next steps
First priority: returning to real learning. This decision comes from what I consider the most important realization of this project: the official matching mechanism tests “whether what you report now matches what you yourself reported back then”, not “whether your report is correct”. Perfect assembly writes god’s-eye ground truth into the models, but a child observer has its own temperament — it makes systematic mistakes at particular places (for example, the boundary band of a cylinder next to a cube is often read as cube). In a report-against-report architecture those systematic deviations are actually signal (if learning made the same mistake, the recurrence at recognition time matches), and perfect assembly throws that signal away entirely. In other words: perfect assembly is not an upper bound of real learning — it is a substitute that is weaker in a crucial dimension. So the next step is unsupervised learning-while-recognizing (what the official tutorials call the more natural but more challenging default learning setup), building the models from scratch, with part identities individuated by the machine’s own “new object” decisions.
Second: starting the logo-composite line, after aligning with the new official dataset (compositional_objects 1.2 merged the symmetrically ambiguous sticker variants).
Appendix 1: Directions that turned out wrong (briefly)
- The “ignore pose for now” route. At the very beginning I assumed pose could be sidestepped: let multiple child LMs each report part identities and judge the composite by their union (pose-independent part-set aggregation). It does work on composites whose parts differ a lot, but it hits a ceiling quickly on geometrically balanced composites and similar families — “which parts” is not enough; the arrangement information of “where each part is and how it is oriented” is the key to family discrimination, and that cannot be had without pose. Only after returning to the official pose-based route did the progress in this post happen.
- Treating perfect assembly as a lossless scaffold. As described in §5, I realized very late that it discards the “reporting behavior” dimension of signal — which is also the direct motivation for returning to real learning.
- Naively wiring multiple sensor streams into the parent cost me dearly; the lessons deserve their own write-up, so I’ll leave them out here.
Appendix 2: A note on LLM-assisted development
This project was carried out with LLM assistance. A few more words on the division of labor and the process, for those who are using or considering this way of doing research.
Division of labor: I myself propose the hypotheses, the experiment directions, and the code logic — including why each mechanism is designed the way it is, how the experiment matrix is laid out, and which results count versus which need retesting; the LLM handles code implementation, running tests and experiment queues, plotting, and the English translation of this post. Directional judgments (which route to continue, which to archive and cut losses on) are always mine.
A few hard process rules (all paid for with pitfalls): First, any new mechanism gets a design document that I review before code is written. Second, after a mechanism lands, probes must verify it actually executed before looking at downstream metrics — instrumentation written by the LLM can itself silently fail, so we require every instrumentation hook to “count its consumption”; zero consumption is treated as zero mechanism, and this rule has caught several cases of “thought it was changed, but it never took effect”. Third, every conclusion must carry its denominator, and “confirmed facts” must be separated from “interpretation”; whenever a control cell is missing from the comparison matrix, it gets run. Fourth, every stage’s findings — including failures and wrong directions — go into a numbered findings ledger; all the material in this post comes from that ledger.
One honest assessment: the LLM makes “a dozen probe rounds a day, each with full telemetry and figures” something one person can afford; but it also makes the mistake of overstating conclusions — this very post contains one causal attribution that was later retracted and re-established with a control experiment. The human does the doubting, the machine does the legwork — so far this combination works for me.