# Running Monty zero-shot on real RGB-D data (BOP YCB-V): what breaks and what cheap fixes buy **URL:** https://forum.thousandbrains.org/t/running-monty-zero-shot-on-real-rgb-d-data-bop-ycb-v-what-breaks-and-what-cheap-fixes-buy/1157 **Category:** Monty Code **Created:** [July 11, 2026, 3:02pm UTC](https://forum.thousandbrains.org/t/running-monty-zero-shot-on-real-rgb-d-data-bop-ycb-v-what-breaks-and-what-cheap-fixes-buy/1157 "2026-07-11T15:02:52Z") **Posts on this page:** 1 **Showing post:** 3
### Author: ![vclay](https://yyz2.discourse-cdn.com/flex034/user_avatar/forum.thousandbrains.org/vclay/32/4_2.png) [@vclay](https://forum.thousandbrains.org/u/vclay) #### Post date: [July 13, 2026, 8:39am UTC](https://forum.thousandbrains.org/t/running-monty-zero-shot-on-real-rgb-d-data-bop-ycb-v-what-breaks-and-what-cheap-fixes-buy/1157/3 "2026-07-13T08:39:05Z")
Hi @W_Foxalike first of all, amazing work! Thanks a lot for sharing all this here. This is super interesting for us. Great idea about testing on the YCB-V dataset for sim-to-real transfer. This is a nice reproducible test bed. To get a better idea of this task, could you share some of the images that you use to test Monty on? As far as I understand, YCB-V is a video dataset, how did you select the frames, and do they contain several objects or only one? Those are some really good catches and fixes (hfov is definitely a gotcha, so if you can make a PR for that, this would be great!). In my mind the terminal condition is mostly there to have some criterion for terminating an episode early to avoid computational cost. If we only show one object and Monty has low evidence for all others, we don’t have to keep matching. But one could also set it less strictly to shorten episodes or more strictly to always take `max_steps` before ending an episode. I think of this as mostly an experiment parameter as opposed to a Monty feature that you can adjust depending on what you want to measure. Monty itself has a most likely hypothesis at every step and can use this to act. It doesn’t need to wait until any terminal threshold is reached to be able to use its hypotheses for intelligent action. This is why we, for example, group correct and correct\_mlh together in our recent publication evaluating Monty ([https://direct.mit.edu/neco/article/38/6/845/136222/Thousand-Brains-Systems-Sensorimotor-Intelligence](https://direct.mit.edu/neco/article/38/6/845/136222/Thousand-Brains-Systems-Sensorimotor-Intelligence)) (while still separating them in some plots like figure 5 since, of course, being more confident earlier is better). We will probably need to work more on this parameter as we introduce more hierarchy (since it also plays a role in when an LM sends a message to another LM) since, as you pointed out, being able to uniquely identify an object isn’t always possible if you just have one viewpoint and know about several objects that look similar/identical from this view. The `object_evidence_threshold` and the finding that confused episodes often have a low confidence are also interesting results. Thanks for sharing this. I wonder whether adjusting the `object_evidence_threshold` would also help with our performance on the [Monty Meets World Benchmarks](https://docs.thousandbrains.org/docs/benchmark-experiments#monty-meets-world). Finally, one small tip: It looks like you wrote a nice custom logger and plotting for what you need already, but you might be interested in the `tbp.teleop` tool @rmounir published recently [Controlling the Monty agent live with `tbp.teleop`](https://forum.thousandbrains.org/t/controlling-the-monty-agent-live-with-tbp-teleop/1151). It’s been a useful debugging tool for our research team lately and can make some nice live visualizations of what Monty sees during an experiment and what its hypotheses look like without having to save large detailed stats files. As far as I know, we haven’t tried it on the `SaccadeOnImageEnvironment` yet, so I am not 100% sure if it will work out of the box for what you are doing, but maybe worth a try. Again thanks a lot for sharing this here and I’m looking forward to reading your further updates! - Viviane --- _[View the full topic](https://forum.thousandbrains.org/t/running-monty-zero-shot-on-real-rgb-d-data-bop-ycb-v-what-breaks-and-what-cheap-fixes-buy/1157)._