Recognition through movement on real RGB-D data: what one moving sensor buys, and the failure it introduces

[W_Foxalike]

Hi everyone,

This follows my first post, where the takeaway was that the single-view wall is an information wall: the local geometry one patch sees isn’t enough to separate similar objects. This post lets one LM move across several real viewpoints within a single episode and accumulate evidence, to see how much “recognition through movement” delivers on real data. It delivers — but movement also introduces a new failure mode, and fixing it turned out to be more interesting than I expected. Still no changes to Monty’s core; everything is in subclasses.

Making the sensor move

Accumulating evidence across frames has a prerequisite: patch locations, displacements, and pose vectors from different frames must live in one shared coordinate frame, or the accumulation is meaningless. Two things (in a subclass): unproject with the full intrinsics fx/fy/cx/cy (the previous post noted the environment hard-codes the iPad’s hfov); and use the ground-truth camera poses BOP provides (cam_R_w2c / cam_t_w2c) to express every frame’s point cloud in a shared world frame.

An honest boundary: on a real robot this step corresponds to extrinsics from arm kinematics or SLAM; here I use the dataset’s ground truth to isolate the one thing I want to test — whether movement itself disambiguates. One engineering byproduct: with true intrinsics, answer-correct (including timeouts where the answer was right) went from 22/32 to 27/32 — the unprojection geometry got honest.

It works: 75%, and one clean proof

Across the 8 objects with multiple viewpoints: 6/8 confident-correct (75%), versus 12% for the single-view baseline.

The cleanest proof is master_chef_can (Fig. 1). Each of its 4 frames on its own times out — the leader and runner-up stay locked together, exactly the pattern from the last post’s Fig. 2. Chain the 4 frames into one episode and it commits, correctly, at ~137 steps — a few steps after the second viewpoint arrives. Same object, same LM; the only difference is whether the sensor moves.

One caveat: part of this gain comes from frame quality (I picked frames with high visibility). “Seeing more clearly” and “seeing from more angles” stack here — it isn’t all movement.

But movement introduced a new failure

In some episodes, within one or two steps after a viewpoint switch, the leading object’s evidence suddenly drops by more than half, and the episode then commits wrong, or misses a commit it should have made (Fig. 2, top). This isn’t noise or natural decay — the leading hypothesis is being deleted.

Diagnosis: the deletion step misfires on the switch transient

This traces to the official BurstSamplingHypothesesUpdater, which deletes “unpromising” hypotheses by the slope of their evidence growth (deletion_trigger_slope). The mechanism: on the step where the viewpoint switches, the new viewpoint’s first observations haven’t aligned with the existing hypotheses yet, so every hypothesis’ evidence wobbles and its slope briefly goes negative. The updater deletes this batch of “apparently stalled” hypotheses — including the correct one. Within a viewpoint it works well; the problem is confined to the short transient right after a switch.

Fixing it, official knobs first

Following “don’t reinvent the wheel,” I first swept the official knobs: deletion_trigger_slope levels, and fixed cooldowns (min_eval_steps, and pausing termination right after a switch). None of the five configurations simultaneously gave zero errors, no slowdown, and kept within-view pruning — a fixed cooldown is a blunt instrument: too short does nothing, too long kills the fast, correct commits that happen within a viewpoint.

The official future-work actually has a matching idea: the out-of-model-movements (OOMM) grace period — “after a large movement, give hypotheses a few steps of amnesty before judging them.” I instantiated that for viewpoint switches as a small fix (in a subclass): reset the updater’s slope window on a switch, so the known large action (“change viewpoint”) gets a few steps of deletion amnesty while within-view pruning continues as before. Result: 7 correct / 1 wrong, median ~135 steps (Fig. 2, bottom).

A few observations for the team

1. The slope-based deletion criterion misfires on known large actions like a viewpoint switch. Rather than tuning the deletion threshold, it might be cleaner for the environment to give the LM an explicit “I just made a large movement” signal that the deletion logic can defer on — curious how the team sees this direction.
2. A fixed cooldown can’t satisfy both sides (fast within a view, stable right after a switch); a context-dependent grace period fits better than a global threshold.
3. True-intrinsics unprojection is necessary for cross-frame consistency, not just an accuracy nicety — worth exposing the full intrinsics at the environment level.

What I’m trying next

One LM can move now and reaches 75%. But the signature of the Thousand Brains theory is many LMs in parallel — each sensing part of an object, voting to reach consensus. Next I want to get voting working on real data: several LMs looking at the same object from different positions, then voting, to see whether it buys back speed, or precision too.

(Code and configs are in monty-realworld; happy to hear where my fix is wrong.)

[vclay]

Cool idea combining movement across several frames! I am a bit surprised why the most likely hypothesis receives such negative evidence after a new frame is shown. Is this because the camera location in the dataset is noisy and the images don’t “stitch together” well? Can you visualize how point clouds from different frames look like in the shared world frame after your projections? The solution you came up with sounds very interesting, I just want to make sure I understand the problem that it solves first.

Maybe @rmounir has more thoughts on this thread here too (as he is the architect of Monty’s burst sampling algorithm)

Best wishes,
Viviane

[rmounir]

Welcome to the forum, @W_Foxalike ! This is a great problem setup and really nice investigations/debugging. Interesting dataset too, I’m thinking this could be a good one to extend our Monty Meets World benchmarks because it introduces sensor movements which comes with more realistic noise, e.g., lighting variability, between frames.

on the step where the viewpoint switches, the new viewpoint’s first observations haven’t aligned with the existing hypotheses yet, so every hypothesis’ evidence wobbles and its slope briefly goes negative.

I think this part requires some more investigation. I don’t see why moving the sensor would cause any misalignment. In the habitat simulator, the distant agent can jump to new viewpoints and continue accumulating evidence without any issues. My suggestions are below.

• Try to use the data to learn a model of any object first. I suspect that there may still be unresolved issues of intrinsic camera parameters or noise in camera pose. tbp.teleop can be very handy here because you can visualize the learned 3D graph nodes in the LM buffer point by point while controlling the agent. You can start with one frame/viewpoint, and see if moving the agent on the surface of the object results in a reasonable model shape in the 3D graph. You can also inspect the quality of the features you’re getting from the SM processing the dataset’s depth maps (i.e., surface normal).
• The next step would be to learn and stitch the graphs from two frames given the camera poses. This may require a few transformations, but also straightforward after you get a couple of graphs from two or more viewpoints with tbp.teleop. If you can build a reasonable model from two viewpoints (even if very sparse), then we can rule out dataset noise and any config bugs.
• It is an interesting idea to fix the issue by resetting the slope window when we expect noisy observations due to large movements. Another way is to maybe try setting the hypotheses age to 0. This is another knob to tune, because we currently guard against deleting a hypothesis if its age is below some threshold. Every evidence update increments this age value by 1.

I think visualizing a single episode live with tbp.teleop would be very useful in debugging these experiments, so if you face any problems with running it on your project, let me know

[W_Foxalike]

Hi Viviane, good question — “not stitching together” was the first thing I ruled out. After projecting both frames’ point clouds into the shared world frame using the camera poses, I ran a nearest-neighbour registration check: the overlap region has p10 ≈ 1–4.6 mm (well under the 10 mm max_match_distance), so it isn’t gross misalignment. The overlaid clouds are attached — you can see the two frames interleave in the overlap and cover two adjacent faces of the can.

Given that, the collapse looks more like the deletion mechanism over-punishing the transient from a new viewpoint’s first observations. That said, I agree the data layer should be nailed down first, so I’ll also run the fuller verification rmounir suggested.

[W_Foxalike]

Hi rmounir, thanks — it’s an honour to have the author of the resampling/burst updater look at this directly. Your skepticism is fair; let me put the two pieces of counter-evidence I already have on the table, then the verification I’ll add.

1. Registration: the two frames’ point clouds in the shared world frame have p10 ≈ 1–4.6 mm (< max_match_distance 10 mm), so gross misalignment is ruled out (figure in my reply to Viviane just above).
2. 2×2 isolation: setting deletion_trigger_slope to -inf (deletion off) makes the collapse disappear — it becomes 7 correct / 0 wrong across the objects. That’s what pointed me at deletion as the amplifier.

But you’re right that the root of the transient could still be residual intrinsics/pose noise, and I shouldn’t stop at “turning off deletion fixes it.” So I’ll follow your three steps: (a) learn a model of one object from this BOP data to expose any intrinsics/pose issues; (b) learn and stitch two frames’ graphs from the camera poses to check data quality; (c) visualize a single episode live with tbp.teleop. I’ll also try your suggestion of resetting the hypothesis age to 0 (via the existing min_age guard) against my slope-window reset — if the age reset is cleaner, I’ll switch to it.

I’ll post what I find, and may well take you up on the offer of help. Thanks!

[W_Foxalike]

Hi rmounir — I ran the checks you suggested. Two things to report: one on the data, one on your age-reset idea.

On the data (you suspected intrinsics / pose noise). Three checks all point to it being clean:

• Registration — the two frames’ point clouds, projected into the shared world frame, land within p10 ≈ 2 mm of each other (well under the 10 mm max_match_distance). That’s the overlaid-clouds figure in my reply to Viviane.
• A model built from scratch — per your suggestion to learn one first, I trained a model off these frames: 878 graph nodes, extent 133×138×171 mm, a coherent object. Noisy intrinsics or poses would smear it into a blob; they don’t.
• SM feature quality — the surface normals you pointed me at. I pulled them off the learned graph nodes and measured them: unit-length (mean |n| = 1.00), smooth across the 6 nearest neighbours (mean |cos| = 0.93, where 1 is perfectly smooth and ~0.3 is random), and outward-facing. Clean features, not noise. This is the very thing you wanted to inspect (and what tbp.teleop shows) — I just measured it statically over the whole model instead of live.

(Fig 1: SM surface normals from the learned model — unit-length, smooth, outward-facing.)

On your age-reset idea. I tried it as-is — reset _hyp_age to 0, leaning on the min_age guard. Honest result: on its own it isn’t enough; it’s the same as no fix (bleach still confused, 6/8). The reason is that the min_age=5 immunity is shorter than the 10-step slope window — the moment immunity lifts, the window still holds the post-switch dip, so the correct hypothesis gets deleted anyway. What actually works is also clearing the slope window (buffer → NaN), so that when immunity lifts the slope only reflects post-switch data. The two together give 7/8.

(Fig 2: five-way frame-switch-grace ablation — age-reset alone = no fix (6/8); combined fixes bleach (7/8).)

The thing I’d stress: every one of these interventions touches only the deletion tracker’s internal buffers — not the accumulated evidence, and not any point clouds or poses — and a purely-mechanical reset still cures the collapse. That’s why I read it as a deletion-timing problem rather than misalignment: the evidence dip right after a switch is just the natural cost of changing viewpoint, and it’s the deletion step that turns that transient into permanently dropping the correct hypothesis.

One concrete takeaway, fwiw: for a known large movement, the grace immunity seems to need to span the slope-window length (~10) and clear the slope history, not just reset the age counter — otherwise the deletion logic still misfires during the transient.

On tbp.teleop: the StepHook it imports wasn’t in the tbp.monty version I’d pinned, so I bumped my core to a matching version and it attaches to my experiments now. The SM surface-normal quality you suggested looking at, I went at statically for this (Fig 1, same feature); teleop’s live view is the same thing, just point-by-point, in the sensor frame, and drivable. Thanks for taking the time on this — it’s been really helpful.

[rmounir]

Thanks for the detailed reply, @W_Foxalike . Nice debugging.

878 graph nodes, extent 133×138×171 mm, a coherent object. Noisy intrinsics or poses would smear it into a blob; they don’t.

Do you happen to have a visualization of the resulting LM graph in 3D (from tbp.teleop or matplotlib visualization of the stored graph nodes) and how it compares to the object you learned it from? My thinking is that the graph can become distorted with incorrect calibration, e.g., FOVH, without smearing into a blob. This will affect the stitching between viewpoints as well. This is different from the plots you shared with Viviane (directly based on the data). I’m more worried about the learned graphs inside the LMs which, if incorrect, would surface any SM calibrations issues and explain why the evidence drops between viewpoints.

teleop’s live view is the same thing, just point-by-point, in the sensor frame, and drivable.

Apart from the smoothness of the surface normal w.r.t. the nearest neighbors, did the surface normal directions make sense (when compared to the viewfinder) as you move the sensor on the object surface. Did the SN directions look reasonable?

Thanks.

[W_Foxalike]

Thanks, this is exactly the right push — and you’re right that “looks coherent” is not “isn’t distorted”: a wrong FOV can warp the graph without ever smearing it into a blob. Two things — a correction first, then a cleaner test.

A correction first. The “878-node” model I quoted was misdescribed. Going back to it, that graph was an unsupervised merge of three objects — my zero-label learner had accreted mustard + master_chef + cracker into one new_object0. So “a coherent object” was the wrong words: three co-located items superimposed, not one object. Thanks for making me look again.

On FOVH specifically. In the multi-view path there is no single FOV to get wrong: each frame is unprojected with the full per-frame intrinsics from BOP scene_camera.json — x=(u−cx)/fx·d, y=(v−cy)/fy·d. So fx, fy, cx, cy are read per frame, not a scalar FOV I set. (The hard-coded iPad FOV only survives as a fallback in the single-image saccade path — the one the FOVH PR parameterizes — and isn’t on this path.)

The distortion test you asked for (first figure). I took one clean single object — master_chef_can; a can is a good probe because a wrong FOV would turn it elliptical or conical — and put its learned LM graph next to the ground-truth CAD mesh, then overlaid the two with rigid ICP only, no scale. Point-to-surface residual is median 2.6 mm / p90 5.5 mm on a 102 mm-diameter can. Because I disallowed scale, a wrong metric scale (which a wrong FOV produces) would show as a systematic offset — it doesn’t — and there’s no elliptical/conical bow. So for this object the learned graph is geometrically faithful, not warped.

On the normal directions (your second question) — two ways. Statically, on that graph 99% of the nodes’ surface normals point outward and adjacent-normal agreement (|cos| to the 5 nearest neighbours) is 0.94 — sensible directions, not just local smoothness. And I took your suggestion literally and watched it live through your teleop viewer (second figure) as the policy drove the sensor over the object: as the patch climbs the bottle’s face, the sensed surface normal stays a stable outward normal, matching the local surface in the view finder — which is what a roughly constant-azimuth cylindrical sweep should give. Nothing looked off. (Two small local tweaks to run teleop on this data: the view-finder outline assumed 4-channel RGBA and broke on our 3-channel frames, and I ran the monitor plotter headless to dump the frames.)

What I take from this. The graph isn’t warped and the normals are sensible both statically and live, so a gross calibration artifact seems unlikely as the cause of the between-viewpoint evidence drop — which lines up with the earlier result that the drop is deletion timing on the transient (the slope tracker firing during the switch) rather than data misalignment. That’s where I’ve landed; very happy to be pushed further if you see a hole.

Thanks again — this one made me catch a real sloppiness in how I’d described that earlier model.

[rmounir]

Thanks @W_Foxalike, this looks reasonable.

If you are looking for more debugging pointers, I would still take two LM graphs learned from two different frames/viewpoints and plot them in the same common reference frame. The LMs already represent features in this common RF so it would be enough to just plot the graph nodes and normal directions as stored in the LM. This would surface issues with the camera pose transformations between viewpoints.

The normal directions on the first figure look right, but the sensed surface normal on the second figure look off (they are not pointing at a direction normal to the surface of the object), but perhaps they are just missing a transform in this visualization?

Thanks.

[W_Foxalike]

Hi rmounir, thank you — that was a sharp catch in particular.

On the normals in the second figure: you’re right, it was a missing transform in the visualization. The second figure (the teleop Input Feature inset) plots the raw sensed normal from the LM’s input buffer, still in the sensor/input frame — it hasn’t been rotated into the object’s reference frame, so overlaid on the object view it doesn’t look surface-normal. The normals actually stored in the LM graph (the first figure, and the one below) are already in the object frame, and those point the right way.

I made the two-viewpoint plot you suggested (below): master_chef_can, with a single-viewpoint LM graph learned from frame 0 and from frame 2 separately, reading the nodes + stored normals straight out of the LM. To make the “camera-pose transform between viewpoints” check quantitative, I rigidly registered the two viewpoints together (one shared transform, no scale) onto the GT CAD —

The two viewpoints cover adjacent sectors and both land on the same real can surface: frame 0 is a median 6.1 mm point-to-CAD, frame 2 a median 4.2 mm (a single frame is noisier than the earlier three-frame fused 2.6 mm, as expected).

• If the camera-pose transform between viewpoints were off, one sector would sit tens of mm off the surface; both sitting on the same surface at a few mm says the inter-frame transforms are consistent.
• The stored normals point radially outward and agree between the two viewpoints (right).

One small suggestion: the teleop Input Feature inset would read more intuitively if it rotated the sensed normal into the currently displayed object frame before drawing it — right now it plots the raw normal from the input buffer, which is why it looks off.

Thanks again — these rounds of questions made me learn a lot.

[rmounir]

Great, thank you for plotting these graphs! The nodes and surface normal directions look reasonable.