From single parts to compositional objects: the whole first, then the details

[W_Foxalike]

Having finished the continual-learning work on single parts（From zero labels to 42/42: an unsupervised continual-learning loop on 14 geometric parts - #4 by W_Foxalike）, I went back to compositional objects and found a completely different situation. A compositional object has two levels: you have to judge the parts and the whole, and often one part is familiar while the other has never been seen.

Carrying the single-part procedure over as it is means missing the forest for the trees. The way Monty works now is closer to the blind men and the elephant: it starts from one small patch of surface and accumulates evidence step by step. The problems I run into fall into a few groups. A whole episode can be spent going around one part while the other part is never seen at all. The evidence from a small part gets drowned out by a large one. When two compositional objects share the same base, their evidence curves run almost on top of each other. The hardest one is deciding that something is new: there is no way to tell “never seen before” from “that side has not been looked at yet”.

Eyes do not work like that. There is an impression of the whole first, and then attention goes to the details to tell things apart. I later read what nleadholm wrote in “Attention and Model-Free Segmentation” in June 2026, that attention spreads from broad to narrow, which is the same thing I had in mind.

I plan to follow this line, in roughly five steps:

measure offline how far the overall shape on its own narrows the candidates;
put that into recognition, producing a list of candidates first and using local observations to confirm or rule out each of them;
make the next observation land where the top two candidates differ most, rather than wherever the walk happens to go;
when every candidate has been ruled out, call it new, and separate out the part that has not been seen before so it can be learned on its own;
close the loop, learning while recognizing, so that new parts and new compositions both end up in memory.

I will post the progress of each step as it comes, whether it works or not. If anyone has thoughts on any of the steps, they would be very welcome.

[Bryan_Leavelle]

This sounds like the right way to go about it, I have found success following the same rule! Cant want to see what you come up with. I’ve also found adding as many sensors as possible only helps.

[W_Foxalike]

Step 1: how far can the whole shape alone narrow the candidates?

The first step asks one question. Before touching any local detail, just look around the object quickly and get a rough point cloud. Compare its overall shape with every compositional object in memory. How high does the true one rank?

I used the simplest comparison I could: shape only, no color or curvature, rotate and keep the best angle.

It worked better than I expected. With 23 compositional objects in memory, the true object was in the top 5 in 87% of cases from the start, and 95% on the training set after tuning.

But the top 3 got stuck at 79%. The reason is one family of objects: the same cube base with different small attachments. There are 10 of them in memory (Figure 1). The whole shape can tell “this is the cube-base family”, but which member it is comes down to luck: within the family the true object ranks first only 44% of the time, yet it is in the top 5 every time (Figure 2, left).

This is actually a natural division of labor. The whole shape picks the family; the difference inside the family sits in that small attachment, and that is a job for local observation. So the candidate list is not a fixed top 3. I widened it to the top 7, which keeps the whole family for the next step.

On the test set the numbers dropped noticeably at first. There were two causes. The test set contains parts never seen during training, so failing to recognize them is no surprise. And a few records were incomplete. For the ring on a thin rod, memory had learned only the ring; the rod was missing entirely, so no amount of accurate shape matching could rank it.

After the records were completed, the true object made the top 7 on the test set in 97% of cases instead of 90%, and the base family made the top 3 in 100% instead of 80% (Figure 2, right).

My conclusion: the whole shape is good enough to narrow the range, not to decide. It turns a 1-in-23 question into a few-in-one-family question. Also, keeping the test set only for acceptance and never for tuning paid off: it caught an overfit tuning choice on the first day.

[W_Foxalike]

An interlude: incomplete records

While working on step 1, I also switched on the official hypothesis-driven observation: take a look where the top two candidates differ most. I expected it to help. Instead accuracy dropped (from 56 to 52 correct out of 69 episodes).

The problem turned out not to be the observation policy but the records themselves. The place where two candidates “differ most” was often not a real shape difference, but a hole that one of the records had never learned. Looking into a hole doesn’t help.

So I compared each of the 23 compositional records with the real shape, and 7 were clearly incomplete. The clearest case is the ring on a thin rod: the record contained only the ring, the rod had never been learned at all (Figure 1). Worse, what was missing was often exactly the small attachment that tells members of the same family apart.

The cause is how learning works. The first learning stops after a while; after that, the system only learns more when it confuses the object with another one, and only at the place of confusion. A part that nothing competes with is never filled in. The rod on the ring object was more than 90% missing, yet its self-check still passed three out of three, because nothing in memory forced it to look at the rod. More steps don’t help either: places the policy never goes to stay unvisited however many steps you give it.

My fix was to change what counts as “learned enough”, from “not confused” to “nothing left unseen”: first scan around the whole object once, use that shape as the reference, go back to any region the record doesn’t yet explain, small unexplained regions first, and stop only when everything is covered.

Result: only 1 of 23 objects remained incomplete (the top of the sledgehammer, which a horizontal tour can’t see), and small attachments such as thin rods and small balls were mostly filled in. The cost is a longer first learning, 62 seconds median.

The biggest surprise came next. With these more complete records swapped in, the previous recognition method got worse: from 56 to 42 correct out of 69 episodes. Whole-shape matching got much better: on the test set, the true object made the top 3 in 97% of cases instead of 77% (Figure 2).

My reading: the previous method told family members apart largely because their records were incomplete in different ways; the shape of what was missing acted as a fingerprint. Once the records are complete, the shared parts of a family match every record equally well, and the only difference left is the small attachment, which the previous tour doesn’t reliably visit. Global-first works the other way round: align the whole first, then look where things differ, so the more complete the records, the better it works.

This also raised another question for me. If global-first requires a very complete first learning, learning gets slower, and the system stores a lot that isn’t needed “at the time”. People don’t learn objects that way. Unless we are deliberately studying something, we use a rough impression, until one day we mix it up with something else and it costs us, or someone points it out, and only then do we look more carefully. Learning with no feedback at all is like a child exploring the whole world with no teacher and no parents. I increasingly think continual learning needs a small amount of external feedback, without being fully supervised: learn roughly on your own, and only when a mistake happens and feedback arrives, take a targeted closer look at the pair of objects that were confused. Once the five steps of the original plan are done, I’ll try this idea.

[W_Foxalike]

Step 2: into recognition, candidate list first, then verify

Step 2 wires the whole shape into the recognition process itself: a quick look around at the start gives a candidate list, and the local observations that follow only confirm or rule out candidates on that list.

The first version was very direct: following the candidates on the list, it jumped from region to region, ranked a short list for each region on its own, and vetoed whatever didn’t fit. It did worse than before, for two reasons. Jumping between regions lost coverage: the places where the sensor dwelled were cut in half, and parts were missed. And the per-region veto overruled results the learning modules had gotten right, downgrading them to “not sure”.

Once coverage was restored, the part lists were indeed cleaner and more complete than before, but composite-object recognition was only 27% correct, lower than the original 41%. At that point I did an error analysis and found the problem was structural, not a matter of some parameter.

The original rule compared the list of parts read in this episode with the part list of each composite object in memory. That requires both sides to be clean. Each side makes its own mistakes and the probabilities multiply: a 55% clean rate squared is about 30%, which matches what I measured. Getting higher would need both sides at 88%, beyond what the part readings can deliver.

Meanwhile, in the same observations, another signal had been kept in a secondary role: using the whole shape alone, the true object ranked first 61% of the time and was in the top 3 88% of the time. I had put the strong signal behind the weak one.

So I reversed the order. The whole shape ranks the candidate list first. Part readings no longer decide; they only check: the part families that were read must be on the candidate’s part list, otherwise that candidate is dropped. When several candidates in the same family can’t be told apart, the position of the small attachment on the base breaks the tie.

Stacking the three steps, the training set went from 27% to 61%, 79% and 88%; the test set to 73%, 77% and 80%.

Recognition errors went down as well, to 3% to 8% on the training set. The remaining errors are concentrated in two objects. Sphere half-sunk in a cube: the sphere cap and the top of the cube fall in the same observation window, and in none of the three placements was the sphere family read. Sledgehammer: the head was read as a cylinder or not read at all. Both are failures of part-level perception, not of the composite-level decision.

Looking back, what made the difference in this step wasn’t the new “jump region by region” mechanism, but turning the whole shape from a reference signal into the primary decision. The value of local observation lies in checking and breaking ties.