Voting on real data (1): why my naive committee failed, and what I think a good witness needs

[W_Foxalike]

Hi everyone,

Following the last post (one moving LM reaching ~87% on real data), I wanted to try the thing I’ve most wanted to see for myself in the Thousand Brains theory: several LMs in parallel, voting to reach consensus. I’m still finding my way around Monty’s engineering, so this is more a learning log of “wire it up the way the theory describes and see what happens” — and my first committee failed. But it failed in a regular way, and the regularity pointed me at a precondition I think matters.

To keep it readable I’ve split the voting part into two posts, published together: this one is the diagnosis — why my naive committee doesn’t work and where the root is; the next one (#4) rebuilds it in a setup that meets the precondition. They’re meant to be read side by side.

How I set it up

I took the official 5-LM benchmark recipe as-is (five sensors in a cross, fully-connected voting, 3-of-5 consensus) with the official pretrained 5-LM models, and fed it real BOP frames. The overlay only places the real frames into the official geometry; I didn’t touch the voting itself — I wanted to see “vanilla” voting on real data before tuning anything.

One thing to note now (I’ll come back to it): these 5 LMs use co-located pretrained models — the same models, looking at nearly the same view.

What I saw: an inverted-U

Changing just one variable — how far apart the five voters stand (sensor baseline) — gives an inverted-U (Fig 1; the metric is precision when it commits @ decision-rate, with correct and correct_mlh grouped as in your Neco paper):

• ~1 cm, same frame: 47% @ 100%
• ~1–2 cm, adjacent frames: 74% @ 100%
• 41–75 cm, wide baseline: 53% @ 100%

(For reference, one LM on one frame is 89% @ 59% — it commits less often, but accurately.)

Both ends are worse, the middle is best. I’d assumed “further apart = more diverse views = better,” which is clearly wrong. It looks like two different failures live at the two ends.

Failure at the narrow end: correlated errors

Five patches on the same frame look at the same ambiguous patch of the same view, so the five witnesses tend to make the same mistake. And voting has an implicit precondition — that the witnesses’ errors are independent. Once they’re correlated, voting adds the same mistake up as five independent votes and quickly commits a wrong consensus. The plainest example: in 3 of bleach’s 4 episodes, all 5 LMs unanimously (5:0) called it mustard — the five witnesses didn’t each err in their own way, they erred toward the same thing together.

Failure at the wide end: votes that miss (Fig 2)

At a wide baseline, a vote has to translate “where I am on the object” into another viewpoint, and the error of that translation scales like baseline × pose-quantization. At cm-scale baselines that’s a few mm (inside the match tolerance); at tens of cm it’s a few cm (outside it) — so the correct object’s votes systematically miss, and the most pose-degenerate objects pick up stray votes instead. mustard is the clearest case here: of 20 column-judgements, 17 land unanimously on banana and the correct answer gets zero (Fig 2). Voting counts those 17 correlated errors as 17 independent votes.

What I think the root is

The reason adjacent frames win, I think, is that they dodge both ends: the sensor noise is already independent, and the transfer error is still within tolerance. But stepping back, I suspect the real root is that these five witnesses are clones — the same model, nearly the same view. One deceptive piece of geometry fools all the clones the same way; move the viewpoint and the blind spot moves with each witness. If that’s right, the problem isn’t the vote-checking rule, it’s that the witnesses aren’t diverse or independent enough — the voting the theory wants needs each LM to have its own model and to see genuinely geometrically-diverse views (azimuth and pitch).

What I’m trying next

Across these three settings, what voting bought looks more like speed and willingness to commit (all at 100% decision-rate, median ~30 steps) than higher precision. If the root really is “clone witnesses,” I should give each LM its own pitch viewpoint, let each learn a different model, and then vote. YCB-V can’t do that, but T-LESS’s training set is a full hemisphere with pitch — so the next post moves there and rebuilds it, to see what a committee actually buys once the precondition is met.

(Code and configs are in monty-realworld; very happy to be told where I’ve got it wrong.)

[W_Foxalike]

Following the last post: Voting on real data (1): why my naive committee failed, and what I think a good witness needs. This post fixes the two things that were missing: I let each LM learn a different model on its own pitch band, and then vote. YCB-V can’t give me pitch, so I moved to T-LESS, whose training set is a full hemisphere — so each column can actually learn a different face of the object. This is still me validating the theory, so I’ve tried to write down both what matched my expectations and what surprised me.

First, confirming the witnesses really differ

Once I split the hemisphere into a few pitch bands and let each LM learn only on its own band, the five columns’ learned point clouds stratify vertically — the centroids span about 45 mm, and the top and bottom bands’ learned surfaces barely overlap (~1%). So the columns really did learn different faces of the object; they’re no longer clones. That’s the causal precondition #3 said was missing, now in place.

The headline: voting bought speed, not peak accuracy

On 28 objects, with the same information, I put three approaches side by side (committee = each column commits and we count votes / single column = coherently integrate all views into one hypothesis space / a rung ladder raising views-per-column from 1 to 2):

• One view per column is only 75% (too weak); two views per column → 92.9%, matching the single column, and still at 26 steps — about 6–7× faster (the single column takes 158–178).
• But with the same information (10 views), the single column’s coherent integration is still 96.4%, slightly above the committee’s 92.9%

I’d quietly hoped the committee would pull ahead here (I finally had real pitch diversity), and it didn’t — and this time it closes the “it’s just missing geometric diversity” excuse, reproducing the official “more columns, faster but not more accurate.” The way I read the mechanism: coherent sequential integration constrains all the views into one hypothesis space and uses the information most fully; voting skips the integration for speed, at a small cost in peak accuracy.

Where the committee actually wins: robustness

Clean input flatters the single column, it turns out. As soon as I add depth noise to every test view, the two lines cross:

• clean: single 96.4 vs committee 92.9;
• 5 mm noise: committee 82.1 vs single 60.7;
• 10 mm noise: committee 32.1 vs single 7.1 (~4.5×).

Any noise flips it, and when I knock out each viewpoint individually (top / high / equator / low / bottom), the committee is more robust in every single case. The way I read it: the five columns’ noise is independent, so consensus averages the independent errors out (wisdom of the crowd), whereas the single column integrates ten noisy views coherently, so the error compounds and drifts — under heavy noise the whole thing collapses onto one “noise attractor” object. Which brings it back to #3’s precondition: voting helps when the errors are independent, and here pitch diversity + independent noise is what finally makes them decorrelate.

Two footnotes, both honest boundaries:

• Compute: each column is a small band-specialized model, ~5× lighter than one LM carrying 5 channels of a whole-sphere model.
• The speed dividend needs parallel compute to cash out: the 6–7× is in steps; running the 5 columns serially on one CPU thread, wall-clock is only a little faster (7.5 vs 9 s). True parallelism (5 columns across cores) is where the ~2 s vs 9 s would come from. Nearest-neighbour search is the main bottleneck.

The claim I haven’t delivered

“The committee is even more accurate at its peak than a single column” — the strongest TBT claim — I couldn’t produce under this homogeneous input (coherent integration is just too strong). I suspect it may live in inputs a single column structurally can’t hold: vertical compositional hierarchy (objects made of objects), or cross-modal input (vision + touch/sound). But that’s a direction I want to try next, not something this post can conclude.

A couple of questions for the team

1. Even with genuine pitch-diverse views, my committee is still “speed + robustness” rather than peak accuracy — consistent with the “faster not more accurate” of your co-located benchmark. Does that reading — that peak accuracy belongs to coherent integration — hold up for you?
2. I noticed the top view is the most discriminative, while near-duplicate views (like the equator) drag a uniform vote down — would reliability-/discriminability-weighted voting keep the robustness and recover some accuracy? Curious whether that’s already on your roadmap.

What I’m trying next

So where I’ve landed: under real-deployment conditions — noisy, multi-view, low-latency, compute-constrained — the committee is already ahead across the board; “more accurate at peak too” has to be found in heterogeneous, structured fields (hierarchy, cross-modal), which is what I want to build next.

(Same as always: very happy to be told where I’ve got it wrong. Code is in monty-realworld.)

[vclay]

Hi @W_Foxalike
This is consistent with what we report in our voting experiments in our recent publication (https://direct.mit.edu/neco/article/38/6/845/136222/Thousand-Brains-Systems-Sensorimotor-Intelligence). Voting is there for faster recognition, but not to increase accuracy. Our experiments in that paper demonstrate that “Monty can converge to a likely representation with far fewer steps, a result that scales with the number of SMs used in the Monty system. Importantly, this increased efficiency does not come at a cost in robustness, with accuracy remaining approximately level.” Adding voting basically reduces the need of the sensor to move. You can either recognize an object by moving one sensor over it for multiple steps, or you can take fewer movements by placing multiple sensors at different locations on the object and voting between them.

As you noticed, it matters how the sensors are placed too. If they overlap a lot, there is a lot of redundancy in their sensory input and they don’t add more information by voting with each other (only accumulating error or confidence faster, as you noted). If they are spaced too far apart on the distant agent, many patches are not on the object and won’t contribute anything. I would assume that if you place the patches around the object and make sure that they are all on the object, the increased distance between them should not be an issue.

I am not sure I understand what you did with the pitch bands. Are you only learning partial models of the object in each LM? What is the difference there? Each LM should ideallt learn a complete model of the object (although in its own reference frame).

Best wishes,
Viviane

[W_Foxalike]

Hi Viviane,

thank you — it’s really encouraging to have you confirm both points: that voting is there for faster recognition rather than higher accuracy, and the overlap-vs-spacing tradeoff.

After your question I went back and re-read the theory around voting, and I realized that each vote having to be made on a complete model is intrinsic to how voting works — that was an oversight on my part. In that experiment I had indeed let each LM learn only the model of its own band (a partial model).

So I redid the experiment: each LM now learns the complete model, with everything else unchanged. It doesn’t change the “voting doesn’t raise accuracy, it raises speed” conclusion, and the committee still shows a clear robustness advantage under noise. The numbers:

depth noise	complete-model committee	partial (band) committee	single column (10 views)
clean	89.3%	92.9%	96.4%
5 mm	71.4%	82.1%	60.7%
10 mm	17.9%	32.1%	7.1%

There is one thing I hadn’t expected, though: with complete models the committee’s accuracy is lower than the partial-model committee at every noise level. My current reading is that locking each witness to a single band is a more decisive prior — but that is something I hand-fed by having “LM k only look at band k.” In a real deployment you don’t control where each sensor lands on the object, and a partial model goes blind the moment it sees outside its band — so the complete model is the general, correct choice, and the partial version’s edge here is really an artifact of my controlled band→LM assignment.

One more question, if I may: in your framework, is this robustness-to-noise from voting (not just the speed) something you would expect?

[vclay]

Yes, that is unexpected about the partial model condition. I would tend to agree with your hypothesis that in this constrained setup, the advantage just comes from constraining the initial hypothesis set for every LM, and there would be no advantage (and things would break) if every sensor could land anywhere on the object.

And yes, robustness to noise should be another advantage of voting. Especially if the noise is independent between the sensors. If the noise between them is correlated, that might lead to errors in the LMs being correlated, which then accumulate more quickly when they vote, as you pointed out earlier.

• Viviane