At the end of my previous post on compositional-object recognition (Recognizing 3D compositional objects: a month of pitfalls, findings, and new attempts) I wrote that the next step would be to go back to real learning. That step is done now, and this post tells it in the order it happened: why the work led to continual learning, how the experiment is set up, how the loop protocol runs, how each object entered the graph memory, how the system decides on its own “who is this, and is it new”, and the defects found along the way. Every number comes with its denominator, and there are a few questions for the official team at the end.
1. Why “learning the objects well” turned into continual learning
The compositional-recognition work in the previous post skipped learning altogether: the object models were synthesized directly from the meshes. Only later did I realize that Monty’s matching asks “does your report now agree with your own report from back then”, and that an observer’s systematic biases at particular locations are part of that signal. A synthesized object model throws that signal away. Synthesis is not an upper bound on real learning; it is a weaker stand-in. That was the reason to go back to real learning.
The first attempt at real learning followed something close to the official supervised pretraining: a fixed number of rotations, a fixed tour, learn each object in one go. It stalled on a plateau. Seven versions of the material strategy on the writing side (number of rotations, density, spatial distribution, extra coverage of discriminative regions) all left the object-level readout between 65% and 69%, while official supervised material under the same recognition configuration reached 81.7%. Going through the candidates one by one showed that the geometric quality of our object models was already at the sub-millimeter level of the supervised ones, so quality was not the gap; and no static density rule could serve every object at once: the mug needs dense material, the cube collapses as soon as it gets dense. A learning recipe fixed in advance cannot be right for all objects at the same time.
So sufficiency should not be decided before learning; it should be judged dynamically during later recognition, and topped up when it falls short. That is where the continual-learning loop starts.
2. Experimental setup
Objects and environment: the 14 basic geometric objects of the official compositional-objects dataset 1.1 (cone, ellipsoid, mug, H-frame, capsule, H-tube, triangular prism, disk, cylinder, washer, open box, torus, cube, sphere), 60 to 100 mm in size, in the official simulator.
System configuration: a hierarchical configuration of 7 learning modules and 8 sensor modules (official architecture; the layering is our own), with recognition read from LM_0. Recognition uses the official default unbounded evidence scoring (past weight 1, present weight 1, no forgetting) and the official default hypotheses updater. Every mechanism in this post lives in an overlay outside the core graph memory; the core code is untouched, and a PR will follow.
Learning views: an object is learned the first time with a single exploration (called a tour below; “minimal first learning”). The tour is orthogonal alignment plus dwell modulation: the sensor first aligns with the surface, then decides how long to stay from the local curvature, and takes a few extra looks where the top three competing object models differ. No number of views is preset; the tour ends when it reaches the step cap or when coverage stops growing.
Poses: objects are learned at rotation 0 deg; the test rotations are 0, 90 and 180 deg about the vertical axis, one episode per rotation; the encounter round of section 5 uses an additional slanted rotation of 35 deg.
Scoring convention: at the end of an episode, whether the most likely object reported by LM_0 equals the object’s own model; if so the episode counts as correct, otherwise as wrong. Every “n/42” in this post uses this convention: 14 objects x 3 test rotations = 42 episodes.
Zero labels: the system names its own object models (new_object0, new_object1, …). The protocol keeps a ledger on the side, used in exactly two places: to decide whether a check passed, and to tell the targeted write-in which object model to write into. Learning itself never uses the ledger. So the loop results in this post belong to the setting “at repair time the protocol knows which object it placed”, which corresponds to re-inspection with a work order in production; the setting with no identity at all is in section 5.
3. New components (names and one-line descriptions; details in the upcoming PR)
- Tour motion: orthogonal alignment plus dwell modulation, shared by recognition and learning; look more where needed, stop where not.
- Difference-directed looks: during learning, the next destination is chosen from where the top three competing object models differ.
- Plateau termination: an episode ends early once the readout is stable and evidence has plateaued.
- Scoring regime by scenario: the continual-learning loop uses the official default unbounded scoring; a frozen graph memory used only for recognition uses fully bounded scoring (accumulated x0.9 plus current x0.1). The comparison is in section 7.
- Write when confident: during recognition, once the leader is far enough ahead and the readout has plateaued, the current observation is merged into the low-density region of the leading object model.
- Targeted write-in: during repair, self-check and sentinel checks, the write target is the object model of the object the protocol placed; the location on the model is taken from that object model’s most likely hypothesis (MLH); only observations 6 to 25 mm from the object model’s nearest node are written.
- Trusted-band write gate: a patch on the previous item: the write position must also lie within 25 mm of the object’s first-learning nodes. Why it is needed is the last item of section 7.
- Sentinel checks: after each new object is learned, the two old objects most entangled with it (counted from how often they appear in the top two during the new object’s self-check) are re-tested.
- Judge: the module that decides, during learning and on encounters, “which object model is this, or is it new”; it consists of a vote across learning modules plus an alignment-residual gate.
4. The continual-learning loop
4.1 Protocol
Object by object: minimal first learning → self-check on the 3 test rotations (writing on) → if not all correct, repair, at most 2 repair rounds per alarm (a repair round = re-recognising the object’s 3 rotations with writing on) → after the object enters the graph memory, sentinel checks on the 2 most entangled old objects → repair any old object that is now recognized wrongly → next object. After all 14 objects, writing is switched off and a 42-episode final check is run. Apart from the single first-learning tour, all learning happens inside recognition: self-checks, sentinel checks and repairs are recognition episodes with writing on and a designated target. The final check is the only episode set that recognises without writing.
4.2 Five iterations
Figure 1: left, final-check scores of the five iterations; the dashed line is 37/42, the score of a graph memory learned from official supervised material (true labels, 14 views) under the same recognition regime. Right, the immediate-recovery rate of repair rounds, i.e. the share of rounds after which all 3 rotations are recognized correctly.
The protocol did not come out in one piece. It went through five iterations, each changing one variable, and Figure 1 reads left to right as those five. The starting point is batch learning, learn once and never change, at best 36/42. The first loop used bounded scoring plus write-when-confident: also 36/42, and only 3 of 56 repair rounds recovered immediately; repairs were made but did not take. The second loop switched to targeted write-in: 39/42, immediate recovery 8/59. The third switched recognition scoring to unbounded: 41/42, immediate recovery 13/39. The fourth added the trusted-band write gate: 42/42, immediate recovery 16/41, 104 minutes in total. That last version is the current regime. Re-running it from scratch reproduced the sequence of self-check, sentinel and repair events exactly, and the final graph memory was identical node for node across all 14 object models.
From the second loop on, the zero-label graph memory beat the supervised reference under the same recognition regime. This is not an information advantage: the supervised graph memory had true labels throughout, while ours had no labels at first learning and only one piece of protocol information, “which object is present”, at repair time.
4.3 Object by object
Figure 2: one panel per object, numbered in order of entry. The vertical axis is the number of the object’s 3 test rotations recognized correctly; the horizontal axis is minutes since the loop started. Grey shading marks the time before the object entered the graph memory. Blue dots are self-checks after first learning, orange triangles are sentinel checks triggered by a later part, green squares are repair-round results, and the black star is the final check at minute 104.
Walking through from the first object.
Object 1, cone (minute 0). Memory is empty. The protocol places the cone in front of the sensor, the system tours it once, the judge compares against the object models in the graph memory (there are none) and rules it new, creating object model new_object0. Self-check on the three rotations: 3/3.
Object 2, ellipsoid (minute 1.3). After first learning the judge compares the observations with the graph memory’s only object model, rules it new, and creates new_object1. Self-check 3/3. Sentinel check: the cone appeared most often in the top two during the ellipsoid’s self-check, so the cone is placed again and recognized, with the result [correct, wrong, correct]: the cone’s 90 deg rotation has been taken by the ellipsoid. Learning a new object has damaged an old one. The protocol immediately opens a repair round on the cone; after one round it is [correct, correct, correct]. In Figure 2, panel 1, the orange triangle at 2 around minute 3 and the green square right after it are these two steps.
Object 3, mug (minute 4.3). Ruled new at first learning. Self-check [correct, wrong, wrong]; repair round 1 gives [correct, correct, wrong], round 2 gives [correct, correct, correct]. Sentinel checks: the cone again loses its 90 deg rotation, this time to the mug, and recovers after one repair round; the ellipsoid is 3/3.
Object 4, H-frame (minute 10.1). Self-check [correct, wrong, correct], recovered after one repair round. Sentinel checks: cone 3/3; mug [correct, correct, wrong], recovered after one round.
The remaining 10 objects follow the same cycle. 10 of the 14 self-checks after first learning were not all correct: minimal first learning really is shallow, and the loop fills it in. Of 25 sentinel checks, 16 caught a recognition error, and every one of them was repaired back to 3/3 in the rounds that followed; 16 of the 41 repair rounds recovered immediately. The cone was damaged by later objects most often (5 times), the H-frame and H-tube 3 times each. Final check 42/42.
Sentinel checks only cover the two objects that compete hardest with the new one; the cylinder, washer, torus, cube and sphere were never checked in the whole loop. They were all correct at the final check this time, but the protocol does not guarantee it: in an earlier loop the washer was never checked either, and carried one recognition error all the way to the final check (section 7).
4.4 What the repairs wrote
Figure 3: observations written into each object’s model during the loop (bars) and the number of repair rounds the object received (right column). 1872 writes in total, all into the object model of the object that was present, 0 into a wrong object model.
The amount written follows need: H-tube 326, open box 248 and cone 243 at the top; sphere 14 and washer 40 at the bottom. Parts that were always recognized correctly were hardly written to; objects that kept getting entangled with each other were written to a lot.
Figure 4: how the cone object model grew, batch by batch. The light surface is the object’s mesh placed as the simulator placed it, aligned to the object model’s frame; dark grey points are nodes present before the panel, colored points are the nodes added in that panel. The text under each panel gives the cause, the number of nodes added and the recognition result at the time.
The cone object model has 338 nodes in 9 batches. First learning wrote 95 nodes in rows along the generatrix, covering one side of the cone only; the self-check, although 3/3, still wrote 60 nodes that filled the gaps between the rows; the sentinel checks triggered by the ellipsoid and the mug each lost the 90 deg rotation, and the repairs wrote 67 and 55 nodes, mostly around the base rim; the three checks triggered by the capsule, the prism and the washer each lost the 180 deg rotation, and the repairs wrote 23, 15 and 13 nodes; the two false-alarm checks wrote only 4 and 6. There is still a sheet of points below the cone, off the surface; where they come from is the last item of section 7.
Figure 5: the mug object model, 540 nodes in 7 batches. The first 6 batches come from the loop; the last batch of 217 nodes comes from the encounter round of section 5, merged after the judge recognized “this is the mug” without being told anything.
First learning wrote 231 nodes covering the front of the body and the handle; the self-check lost both the 90 and 180 deg rotations to the cone, and two repair rounds wrote 54 nodes; after the H-frame entered, the 180 deg rotation was lost again, to the H-frame, and a repair wrote 19; the three checks after that were all correct and wrote only a few nodes. The median distance from node to surface is about 1 mm in all seven batches.
5. Deciding on its own whether an object is new
Inside the loop, “which part is placed this time” is told to the system by the protocol. In the real world that information does not exist: the system has to decide for itself whether what it meets is one of the object models in the graph memory or something it has never seen. This section tests exactly that, on the graph memory learned by the loop, with no identity information given.
Setup: 6 known hard objects (mug, H-frame, triangular prism, disk, washer, open box) and 3 newly made 3D objects (an L-block, clearly a new shape; a hexagonal prism and a frustum, close to the cylinder and the cone respectively) are mixed into one encounter stream. Each appears for one episode at 35 deg about the vertical axis, a rotation that was neither learned nor tested before. The judge is a vote across learning modules plus an alignment-residual gate, with unbounded scoring for recognition; if it rules “known”, the episode’s observations are merged into that object model through the official `update_memory` path, i.e. the existing object model is extended with the new points; if it rules “new”, nothing is written and nothing is created. Judgment quality cannot be read from the new objects alone: the share of known objects wrongly judged new has to be reported at the same time, and the difference between the two is called discriminability, otherwise a judge that calls everything new would score full marks.
Figure 6: for each object in the unlabeled 35 deg mixed encounter stream, the judge’s decision, the number of nodes merged, and the median distance from the merged nodes to the object’s true surface.
Result: all 6 known objects were recognized and merged into the correct object model (6/6), with 0 wrong merges and 0 wrongly judged new; all 3 new objects were judged new (3/3), with 0 wrong merges; discriminability 100 percentage points. The merged material lies on the surface: the median distance from merged nodes to each object’s true surface is 0.4 to 2.5 mm. A 42-episode recognition check after merging gives 42/42; merging did not damage any old object. The last batch of the mug in Figure 5 is the 217 nodes merged here.
Boundary and next step: the sample is small, 3 new objects, 6 known objects, a single 35 deg rotation. After an object is judged new, nothing is written or created yet; the hand-over to learning an object model for it has not been built. The judge ran without its pre-merge guard (re-checking against the most easily confused rival object model before merging). Replacing the protocol-given identity in the loop with the judge’s own decision, so that the whole loop runs autonomously, is one of the planned next steps; both the judge and the loop components exist already, and my initial judgment is that it is quite feasible.
6. Summary and boundaries
Starting from zero labels, the 14 geometric objects were learned into a graph memory under the protocol “minimal first learning, top up dynamically during recognition”, with a final check of 42/42, reproduced exactly on re-run. Writing is allocated by need; objects that were always recognized correctly were hardly written to. In an encounter stream with no identity information, the system recognized the known objects by itself and merged the new observations into them, and judged three newly made objects to be new. On slanted rotations of 20, 35 and 70 deg that were never learned or tested, the same graph memory recognises 39/42.
Two observations. First, “holding up” in a continual-learning loop does not mean never getting worse; it means getting worse, being caught and being repaired: 16 of the 25 sentinel checks caught a recognition error, and all 16 were repaired. Second, the writing side and the scoring side form a positive feedback pair: unbounded scoring keeps the correct object model’s pose hypothesis coherent through the episode, the hit rate of targeted write-in rises with it, and the material written pays off in the very next episode. This loop does not exist under bounded scoring.
Boundaries: simulation only; 14 geometric objects; test rotations only about the vertical axis; “which part is placed” is protocol-given at repair and check time; sentinel selection by competition co-occurrence misses old objects that do not compete with the new one; 17% (59/338) of the cone object model’s nodes are still more than 15 mm off the true surface, see section 7; section 5 has a small sample and no hand-over from “judged new” to learning; the whole run takes 104 minutes, a little over 7 minutes per object, and the speed levers that exist reduce writing and are not yet tuned to the current regime.
7. Defects and lessons found along the way
- Conservation of the total under static material strategies. Seven writing-side strategies only moved recognition errors from one object to another; no density rule could serve every part, which is what led to dynamic top-up.
- The two faces of the scoring regime. On a frozen graph memory bounded scoring wins by a wide margin (36/42 vs 22/42 on the same batch-learned graph memory); inside the loop unbounded wins (41/42 vs 39/42). Changing the scoring alone was negative everywhere; it only paid off together with targeted write-in. The official docs leave “bounded evidence may be more reasonable” open; our data say the answer depends on the scenario.
- Duplicate object models created when judging new hijack later checks. Creating an object model on every “new” verdict in an encounter setting made the second appearance of the same part split votes between the old and the new object model, and one check fell to 11/42. The fix was to separate the verdict from object model creation: judged new, nothing written, nothing created.
- Two engineering issues in official components. The grid object model anchors on its first observation and rejects some merges; the quaternion field order differs between the environment and the sampler. There is also a crash candidate: unbounded scoring combined with the burst-sampling hypotheses updater desynchronises the evidence and location arrays; we worked around it with the default updater and will write it up as an issue for the official team to confirm.
- Sentinel selection by co-occurrence can miss an object. In an earlier loop the washer was never selected after entering, and carried a wrong 180 deg rotation to the final check; once found there, 3 repair rounds fixed it with the whole graph memory still at full marks. In practice this is “notice the ambiguity the next time the washer shows up and relearn on the spot”; in the protocol, errors found at the final check are not yet fed back into repair.
- Targeted write-in put nodes in the wrong place. Found last, and the one with the largest effect.
Figure 7: left and middle, the final cone object model before and after the gate, red points more than 15 mm off the true surface; right, the share of off-surface nodes in each object’s final object model, before and after.
Targeted write-in transforms an observation into the object model’s frame using the object model’s most likely hypothesis (MLH) at the time; in episodes that were recognized wrongly that hypothesis is itself wrong, the node is written in the wrong place, 17% (389/2305) of writes before the gate were more than 15 mm off the true surface, and that material still scores in the evidence accumulation. The trusted-band gate (the write position must lie within 25 mm of the first-learning nodes) brought off-surface writes down to 5% (99/1872), the final check up from 41 to 42, and pruning the remaining off-surface nodes still leaves 42/42. The official unsupervised merge also transforms new observations with the detected pose, so the exposure is the same; that is where the second question in section 8 comes from.
8. Questions for the official team
1. If Monty learns tens of thousands of objects in the real world by continual learning, does recognition degrade with the number of object models, and do compute and memory become unmanageable? The paper notes inference cost grows linearly with the number of known models; how far along are the mitigation paths you have in mind (model merging, hierarchical decomposition)? 2. The official unsupervised merge transforms new observations with the detected pose. In episodes where the pose hypothesis is wrong, does it write observations off the surface in the same way? Have you considered a geometric gate before merging that does not depend on the pose hypothesis, or post-hoc cleanup of nodes that later observations never support (the permanence weights mentioned under Deal with Incomplete Models in future work)? 3. The bounded-evidence item in future work is paused. Our data say bounded is better for recognition on a frozen graph memory and unbounded is better while learning during recognition. Would you consider switching by scenario, or do you prefer to solve it within one regime?
Appendix: on LLM-assisted development
As in the previous post: the experimental direction, the hypotheses, the mechanism design and every decision are mine; the LLM implements, runs the experiment queues, draws the figures and translates into English. One more hard rule was added this time: for any new mechanism or new figure, verify from step-by-step telemetry that “it really happened, and the figure is the data” before looking at metrics. The last defect in section 7 was tracked down that way; I first assumed the figure was wrong.
