Hi everyone,
I’ve been following the Thousand Brains theory and this project for a while, and I recently decided to move from reading along to actually building — doing some engineering validation and a bit of exploratory research. This is the first thing I picked up: running Monty’s pretrained models zero-shot on real RGB-D data, to see what happens when object models learned in simulation meet a real camera. The data is YCB-V from the BOP benchmarks (real captures, with ground-truth camera poses provided, so it’s reproducible). This post is a record of what worked out of the box, what broke, and how much the cheap fixes buy back. I didn’t touch Monty’s core code — all adaptation is done through subclasses.
Setup
- Models: the official supervised-pretrained 77-object graphs (the
surf_agent_1lm_77objfamily), no re-training. - Data: YCB-V test scenes converted into the format
SaccadeOnImageEnvironmentexpects (a depth image + RGB). That environment was written for iPad captures, and it turns out to be a ready-made bridge between simulation and real images. - Two small adaptations (in a subclass): widen the depth clip from the iPad’s 0.4 m to 2 m; use the dataset’s intrinsics for field of view (the environment hard-codes the iPad’s
hfov). - Test set: 8 YCB objects × 4 frames each = 32 recognition episodes.
First results: two failure modes
| Outcome (official CSV category) | Episodes | Share |
|---|---|---|
correct (converged, right object) |
4 | 12% |
correct_mlh (timed out, most likely hypothesis was right) |
9 | 28% |
confused (converged, wrong object) |
12 | 38% |
confused_mlh (timed out, most likely hypothesis also wrong) |
4 | 12% |
patch_off_object (saccade never landed on the object) |
3 | 9% |
A model that scores near 100% on the simulated benchmark is confidently-correct only 12% here; add the timed-out-but-right episodes and 41% get the answer right. Setting aside the 3 pipeline misses (patch_off_object, where the saccade never found the object), the recognition failures fall cleanly into two kinds (Fig. 1):
- Failure A — the answer is actually right (the most likely hypothesis points at the correct object), but the episode times out before the terminal condition is met. The model won’t commit. (
correct_mlh, 9 episodes.) - Failure B — it converges quickly onto the wrong object, which is worse than not committing. (
confused, 12 episodes.)
(Fig. 1: the 32 episodes broken down by outcome category.)
A tiny logging tool
The official step-by-step log (DetailedJSONHandler) is ~1.3 GB per episode here, too heavy to leave on. I wrote a lightweight logger that records, per step, only the top-5 candidate objects’ evidence and the sensor’s pixel location — about 150 KB per episode. Every diagnosis below comes from it (Fig. 2).
(Fig. 2: evidence of the top candidates vs. step count for one episode — the winner never pulls far enough ahead of the runner-up to trigger termination.)
Diagnosis without new experiments
Re-reading the existing logs is enough to locate the mechanism behind each failure:
- Of the timed-out episodes, every one is stuck at the same place: the possible matches (the set of still-viable objects) never narrows to one. The terminal condition needs the runner-up’s evidence to fall below 80% of the leader’s (
x_percent_threshold=20), and from a single real viewpoint there are always a few geometrically-similar objects staying close. - Of the wrong-object episodes, most converge at a very low absolute evidence —
object_evidence_thresholddefaults to 1, which is essentially no floor.
Both failures share one root cause: from a single viewpoint, local geometry doesn’t separate similar objects. A small patch of a can’s wall and a small patch of a pitcher’s wall look nearly the same at patch scale.
Cheap fixes and where they stop
| Fix (config-level only) | Effect |
|---|---|
object_evidence_threshold 1 → 50 |
stops most low-evidence wrong convergences; 4 episodes flip from wrong to correct |
| library 77 → 8 objects | marginal — the runner-up confuser simply takes over |
| background masking (GT masks) | pitcher_base 0/4 → 4/4; background depth was polluting the patches |
| frame-quality filter (visibility ≥ 90%, depth coverage ≥ 70%) | one object’s frames turned out to be 83–89% occluded — it had never been fairly tested |
| all combined | confident-correct 12% → 28%; precision 60% |
Config-level fixes have a clear ceiling. Past 28%, the remaining failures all point the same way — not enough information, rather than bad judgment.
A few observations for the team
- Timed-out episodes are all stuck on the possible-matches convergence criterion; the symmetry-confirmation logic never runs in these episodes (it comes after object uniqueness).
object_evidence_thresholddefaults to 1, which is effectively off on real data — an empirical sweet spot is around 50 (for this 8-object, single-view setup).SaccadeOnImageEnvironmenthard-codeshfovand the depth clip for the iPad; these would be worth parameterizing (I plan to send a small PR).- Real-sensor depth dropout (depth = 0) is currently unprojected into “a surface at the camera origin”; treating it as an invalid observation would be more correct (also a small PR).
What I’m trying next
The single-view wall is an information wall: the local geometry one patch sees isn’t enough to separate similar objects. Next I want to let the sensor move — accumulating evidence across several real viewpoints within one episode (using the dataset’s ground-truth camera poses for a shared world frame) — and see how much “recognition through movement” delivers on real data.
(Reproduction commands, the converter, and all configs are here: monty-realworld. Happy to hear where my adaptation choices are wrong.)
