From single parts to compositional objects: the whole first, then the details

[W_Foxalike]

Having finished the continual-learning work on single parts（From zero labels to 42/42: an unsupervised continual-learning loop on 14 geometric parts - #4 by W_Foxalike）, I went back to compositional objects and found a completely different situation. A compositional object has two levels: you have to judge the parts and the whole, and often one part is familiar while the other has never been seen.

Carrying the single-part procedure over as it is means missing the forest for the trees. The way Monty works now is closer to the blind men and the elephant: it starts from one small patch of surface and accumulates evidence step by step. The problems I run into fall into a few groups. A whole episode can be spent going around one part while the other part is never seen at all. The evidence from a small part gets drowned out by a large one. When two compositional objects share the same base, their evidence curves run almost on top of each other. The hardest one is deciding that something is new: there is no way to tell “never seen before” from “that side has not been looked at yet”.

Eyes do not work like that. There is an impression of the whole first, and then attention goes to the details to tell things apart. I later read what nleadholm wrote in “Attention and Model-Free Segmentation” in June 2026, that attention spreads from broad to narrow, which is the same thing I had in mind.

I plan to follow this line, in roughly five steps:

measure offline how far the overall shape on its own narrows the candidates;
put that into recognition, producing a list of candidates first and using local observations to confirm or rule out each of them;
make the next observation land where the top two candidates differ most, rather than wherever the walk happens to go;
when every candidate has been ruled out, call it new, and separate out the part that has not been seen before so it can be learned on its own;
close the loop, learning while recognizing, so that new parts and new compositions both end up in memory.

I will post the progress of each step as it comes, whether it works or not. If anyone has thoughts on any of the steps, they would be very welcome.

[Bryan_Leavelle]

This sounds like the right way to go about it, I have found success following the same rule! Cant want to see what you come up with. I’ve also found adding as many sensors as possible only helps.

[W_Foxalike]

Step 1: how far can the whole shape alone narrow the candidates?

The first step asks one question. Before touching any local detail, just look around the object quickly and get a rough point cloud. Compare its overall shape with every compositional object in memory. How high does the true one rank?

I used the simplest comparison I could: shape only, no color or curvature, rotate and keep the best angle.

It worked better than I expected. With 23 compositional objects in memory, the true object was in the top 5 in 87% of cases from the start, and 95% on the training set after tuning.

But the top 3 got stuck at 79%. The reason is one family of objects: the same cube base with different small attachments. There are 10 of them in memory (Figure 1). The whole shape can tell “this is the cube-base family”, but which member it is comes down to luck: within the family the true object ranks first only 44% of the time, yet it is in the top 5 every time (Figure 2, left).

This is actually a natural division of labor. The whole shape picks the family; the difference inside the family sits in that small attachment, and that is a job for local observation. So the candidate list is not a fixed top 3. I widened it to the top 7, which keeps the whole family for the next step.

On the test set the numbers dropped noticeably at first. There were two causes. The test set contains parts never seen during training, so failing to recognize them is no surprise. And a few records were incomplete. For the ring on a thin rod, memory had learned only the ring; the rod was missing entirely, so no amount of accurate shape matching could rank it.

After the records were completed, the true object made the top 7 on the test set in 97% of cases instead of 90%, and the base family made the top 3 in 100% instead of 80% (Figure 2, right).

My conclusion: the whole shape is good enough to narrow the range, not to decide. It turns a 1-in-23 question into a few-in-one-family question. Also, keeping the test set only for acceptance and never for tuning paid off: it caught an overfit tuning choice on the first day.

[W_Foxalike]

An interlude: incomplete records

While working on step 1, I also switched on the official hypothesis-driven observation: take a look where the top two candidates differ most. I expected it to help. Instead accuracy dropped (from 56 to 52 correct out of 69 episodes).

The problem turned out not to be the observation policy but the records themselves. The place where two candidates “differ most” was often not a real shape difference, but a hole that one of the records had never learned. Looking into a hole doesn’t help.

So I compared each of the 23 compositional records with the real shape, and 7 were clearly incomplete. The clearest case is the ring on a thin rod: the record contained only the ring, the rod had never been learned at all (Figure 1). Worse, what was missing was often exactly the small attachment that tells members of the same family apart.

The cause is how learning works. The first learning stops after a while; after that, the system only learns more when it confuses the object with another one, and only at the place of confusion. A part that nothing competes with is never filled in. The rod on the ring object was more than 90% missing, yet its self-check still passed three out of three, because nothing in memory forced it to look at the rod. More steps don’t help either: places the policy never goes to stay unvisited however many steps you give it.

My fix was to change what counts as “learned enough”, from “not confused” to “nothing left unseen”: first scan around the whole object once, use that shape as the reference, go back to any region the record doesn’t yet explain, small unexplained regions first, and stop only when everything is covered.

Result: only 1 of 23 objects remained incomplete (the top of the sledgehammer, which a horizontal tour can’t see), and small attachments such as thin rods and small balls were mostly filled in. The cost is a longer first learning, 62 seconds median.

The biggest surprise came next. With these more complete records swapped in, the previous recognition method got worse: from 56 to 42 correct out of 69 episodes. Whole-shape matching got much better: on the test set, the true object made the top 3 in 97% of cases instead of 77% (Figure 2).

My reading: the previous method told family members apart largely because their records were incomplete in different ways; the shape of what was missing acted as a fingerprint. Once the records are complete, the shared parts of a family match every record equally well, and the only difference left is the small attachment, which the previous tour doesn’t reliably visit. Global-first works the other way round: align the whole first, then look where things differ, so the more complete the records, the better it works.

This also raised another question for me. If global-first requires a very complete first learning, learning gets slower, and the system stores a lot that isn’t needed “at the time”. People don’t learn objects that way. Unless we are deliberately studying something, we use a rough impression, until one day we mix it up with something else and it costs us, or someone points it out, and only then do we look more carefully. Learning with no feedback at all is like a child exploring the whole world with no teacher and no parents. I increasingly think continual learning needs a small amount of external feedback, without being fully supervised: learn roughly on your own, and only when a mistake happens and feedback arrives, take a targeted closer look at the pair of objects that were confused. Once the five steps of the original plan are done, I’ll try this idea.

[W_Foxalike]

Step 2: into recognition, candidate list first, then verify

Step 2 wires the whole shape into the recognition process itself: a quick look around at the start gives a candidate list, and the local observations that follow only confirm or rule out candidates on that list.

The first version was very direct: following the candidates on the list, it jumped from region to region, ranked a short list for each region on its own, and vetoed whatever didn’t fit. It did worse than before, for two reasons. Jumping between regions lost coverage: the places where the sensor dwelled were cut in half, and parts were missed. And the per-region veto overruled results the learning modules had gotten right, downgrading them to “not sure”.

Once coverage was restored, the part lists were indeed cleaner and more complete than before, but composite-object recognition was only 27% correct, lower than the original 41%. At that point I did an error analysis and found the problem was structural, not a matter of some parameter.

The original rule compared the list of parts read in this episode with the part list of each composite object in memory. That requires both sides to be clean. Each side makes its own mistakes and the probabilities multiply: a 55% clean rate squared is about 30%, which matches what I measured. Getting higher would need both sides at 88%, beyond what the part readings can deliver.

Meanwhile, in the same observations, another signal had been kept in a secondary role: using the whole shape alone, the true object ranked first 61% of the time and was in the top 3 88% of the time. I had put the strong signal behind the weak one.

So I reversed the order. The whole shape ranks the candidate list first. Part readings no longer decide; they only check: the part families that were read must be on the candidate’s part list, otherwise that candidate is dropped. When several candidates in the same family can’t be told apart, the position of the small attachment on the base breaks the tie.

Stacking the three steps, the training set went from 27% to 61%, 79% and 88%; the test set to 73%, 77% and 80%.

Recognition errors went down as well, to 3% to 8% on the training set. The remaining errors are concentrated in two objects. Sphere half-sunk in a cube: the sphere cap and the top of the cube fall in the same observation window, and in none of the three placements was the sphere family read. Sledgehammer: the head was read as a cylinder or not read at all. Both are failures of part-level perception, not of the composite-level decision.

Looking back, what made the difference in this step wasn’t the new “jump region by region” mechanism, but turning the whole shape from a reference signal into the primary decision. The value of local observation lies in checking and breaking ties.

[W_Foxalike]

Steps 3 and 4: look where they differ, and judge new when nothing fits

Once the whole shape has narrowed the candidates down to one family, the top two in that family often score very close, almost a tie (I’ll call it a near-tie below). Step 3: find where the two records differ most, go there for a close look, and see which record better explains what is observed there. Step 4: if after the close look even the chosen record can’t explain the observations, judge the object new; if there is a gap between two pieces, judge them as two pieces placed apart.

Step 3 tripped me up twice at first. The first time was the mallet and the sledgehammer, two objects with the same handle and different heads. The largest difference fell at the bottom of the handle, but the mallet’s record happened never to have learned the bottom of the handle, so all three episodes were judged to be the sledgehammer. So I changed it to compare only where both records have content, and the second time it backfired on another pair: a small ball on the top face of a cube, one at the centre of the face, one near a corner. The real difference was exactly where one record has a surface and the other doesn’t, and the rule excluded it; all that was left to compare was the cube faces both have, and the comparison turned into which record was denser. Across the batch, the first-ranked candidate was replaced 13 times and 7 of those were wrong, all of them switched to the centre one. In the end I went back to comparing all the differences, on the condition that the records are completed first: when a record is complete, “there is nothing there” is itself evidence.

The hard part of step 4 is composite objects that contain a part never seen before. I prepared three composites, each with one part that isn’t in the part library. By whole shape alone they can’t be told apart from the known composites; what separates them is the surface normals in a close look. So the system takes one close look where the chosen record explains the observations worst. Take the pyramid on a cube: on position alone, the record explains nearly 90% of the points; requiring both position and normal to match, only 5% to 22% remain, while for known composites, even at their own worst spot, the lowest is still 30% (Figure 1). Pieces placed apart are handled with the same idea: go to where the two pieces meet, take a close look at a narrow band, and measure the largest empty stretch. Where a thin rod comes out of a cube face there is a groove, and the empty stretch measured there is about 9 mm, which at first was taken for a gap; real gaps are at least 22 mm, so the threshold went from 8 mm to 15 mm.

With both steps in place, 36 of 39 training episodes were recognized correctly, and in near-ties the first-ranked candidate was replaced 19 times, every time correctly; on the test set, all 30 episodes of known composites were correct. Of the 9 episodes of composites with an unseen part, 6 were judged new; the same close-look check made a decision in 38 episodes on known composites and never wrongly judged one new. All 6 episodes of pieces placed apart were judged correctly, and none of the 9 episodes of touching pieces was judged as placed apart. The one that was never judged new is the pentagonal-prism rod: the pentagonal prism is too similar to the hexagonal-prism head of the bolt in the library.

Later I switched the simulator from Habitat to MuJoCo and learned the records again, and two more problems came up. The first was in step 3, the mallet again: all three episodes went to the sledgehammer once more. This time the points on the thin handle in the record were nearly twice as dense as on the head, so alignment pulled the whole record toward the handle end, and the bottom of the handle was off by nearly a centimetre (Figure 2). After switching both alignment and scoring to a copy of the record thinned to even density, the mallet got two episodes back, and the training set went from 34 to 38 correct episodes, with the test set unchanged.

The second was in step 4, judging new: all three episodes of the pyramid on a cube were recognized as the cone on a cube. This time the cone record wasn’t dropped by the part check and went into a near-tie; the discrimination only compares the top two, and once it’s done the winner isn’t checked again. Adding a check on the winner didn’t stop it either: at the spot the cone record explains worst, it still explains 70% to 90% of the points. On Habitat it was judged new because the record checked there was far off in shape.

Looking back, both steps compare observations with records, and each went wrong in one direction. Step 3 shows that a difference between two records isn’t necessarily a difference between the objects: gaps in a record and uneven density both create false differences. Step 4 shows that observations matching a record don’t necessarily mean it is the same object: an unseen part close in shape still matches at the precision of the current close look; the pentagonal prism matched the bolt, and the pyramid matched the cone.

[W_Foxalike]

Step 5: learning while recognizing, asking only when unsure

The step 5 establishes a loop between recognition and learning, marking the first instance where the system learns about composite objects while simultaneously identifying them. I consider this—along with the initial step of perceiving the object as a whole—to be the most critical components of this framework. Once this loop is in place, the system can autonomously fill in gaps in its knowledge and learn about previously unseen composite objects. The recognition accuracy approaches that of a system that already knows the true identities of the objects; throughout the entire process, the system only queried an external source for the correct identity once (in the experiment, this was handled by a program providing the ground truth, effectively simulating the presence of a teacher).

	After first learning (start)	With the loop	Filled with true identities (reference)
Training set, recognized correctly (39 episodes)	30 (77%)	35 (90%)	36 (92%)
Test set, recognized correctly (30 episodes, not in the loop)	—	28 (93%)	29 (97%)
Two never-seen composites, recognized after being learned (6 episodes)	0	4	not learned
Identities asked from outside	—	1 in the whole batch	one for every write

The key to successful integration lies in the system’s ability to determine an object’s identity autonomously. Previously, when an object wasn’t correctly identified—often because its data record was incomplete—its identity had to be manually provided from an external source for each instance. Furthermore, any composite encountered for the first time was simply labeled as “new,” with no mechanism to recognize it upon subsequent encounters. Now, the system evaluates each object three poses and determines its identity based on a majority vote. If the object is identified as one already present in the library, the current observations are used to fill in any missing data in its existing record; if it is deemed new, a fresh record is created. It is important to note that the part library itself remains unchanged during this process: the system learns the record for the composite , but individual components that have not been seen before are not yet added to the part library as standalone entries. In the initial library, 13 out of 23 records had gaps; 7 of these were in the training batch. After a full cycle of iterations, all gaps were successfully filled—with zero errors and no redundant records created.

The trade-off is that learning new tasks can impair performance on previously learned ones—an issue that still requires a solution. For instance, after the “Pyramid on Cube” task was mastered, the new record was added to the candidate list; however, this resulted in a reduction of one episode for training and two episodes for testing.

Allowing the system to perform its own “gap-filling” carries a major risk: it might misidentify an object and then record data based on that error. To prevent this, a two-layered safeguard is in place: multiple verification checks are conducted before any data is written, and if an error is detected, the entire batch of entries can be retracted. In an early version, a pyramid-like structure atop a cube was confidently misidentified as a cone atop a cube; this passed a close-up visual inspection, and the data was subsequently recorded under the “cone” category.The two-layered check works as follows: if the observed surface orientation fails to match the recorded data (e.g., multiple flat planes mapped to a single curved surface), that specific entry is invalidated; furthermore, if the existing record is already complete, no additional data is added. The retraction mechanism functions by grouping each set of entries into a “batch.” When the object is re-examined, the system checks whether the points in that batch align with the current observations. In the case of the misidentified cone, for instance, half of the recorded points were visible but never matched the actual surface, while only about a quarter were correctly mapped. By deleting the entire batch, the system removes the erroneous data, leaving behind only the original, accurate points. Since the incorrectly recorded points are often just a few millimeters away from the actual surface—making it impossible to distinguish them on an individual, point-by-point basis—the only viable solution is to retract the entire batch.

There is no need to verify the identity after every single entry; I only ask for clarification when the system itself is uncertain. This approach yields the same results while cutting the total time by more than half. Previously, I would perform a spot check against the ground truth for potentially confusing entries every single time—a process that accounted for over half the total duration, yet never once revealed a regression across 50 iterations. I initially considered providing feedback only when an error occurred, but that would effectively mean having someone scrutinize every single response. Now, I only intervene when the system cannot reach a confident consensus from three episodes—and even then, only after a closer manual review fails to resolve the ambiguity. If the answer exists in the database, it is retrieved and added as usual; if it is missing, it is labeled as “new,” and a fresh record is created. Since replacing the spot-check routine, the results have remained consistent across every batch, and the total processing time has dropped from 8 hours and 13 minutes to 3 hours and 42 minutes—further optimized to 2 hours and 24 minutes. In the entire batch, the only instance requiring manual intervention involved an entry—“pyramid”—that was flagged during the initial screening.

Loop can bridge some gaps, but it cannot compensate for “thin” initial learning—that is, when the recorded data is too sparse. In the first step, relying solely on overall shape for ranking can cause the true object to be discarded. By halving the angular range of the initial observation—reducing the time required from 75 seconds to 36 seconds—I ended up with less than half the recorded data points, mostly confined to a single side. Consequently, the right number of training batch dropped from 35 to 18, with two errors occurring during the process. Since the observation spans a full circle, most of it falls outside the scope of these “thin” records; instead, the data appears more similar to other records that happen to share the same component and are well-represented on that specific side. The number of instances where the true object ranked first fell from 29 out of 39 to 18. This is how a bolt ends up being misidentified as a block atop a cylinder—a result that is also reflected in the observation data (Figure 1).

Looking back at the two ideas at the end of post 3, a small amount of feedback has worked out, and learning roughly at first hasn’t yet. It is enough to give feedback only when the system itself is unsure, but it won’t ask about a mistake it is confident in: the pentagonal-prism rod was recognized as the bolt in all three episodes, and it was stopped only because the bolt’s record happened to be complete already. Learning roughly has to wait until matching by whole shape alone can tolerate incomplete records; people can recognize things from a rough impression, and the system can’t do that yet.

Looking back, the two ideas mentioned at the end of the third post have been partially validated—though the concept of “learning just the gist” hasn’t quite panned out yet. The system currently triggers a request for feedback only when it’s uncertain; it won’t ask for clarification regarding errors it feels “confident” about. For instance, it misidentified a pentagonal-prism rod as a standard bolt across three episodes—a mistake that was only caught because the entry for “bolt” happened to be complete. To truly “learn the gist,” the system would need to be able to tolerate incomplete records by relying on overall shape matching—a capability it currently lacks, unlike humans, who can easily recognize objects based on a rough impression.