Hi @W_Foxalike
first of all, amazing work! Thanks a lot for sharing all this here. This is super interesting for us.
Great idea about testing on the YCB-V dataset for sim-to-real transfer. This is a nice reproducible test bed. To get a better idea of this task, could you share some of the images that you use to test Monty on? As far as I understand, YCB-V is a video dataset, how did you select the frames, and do they contain several objects or only one?
Those are some really good catches and fixes (hfov is definitely a gotcha, so if you can make a PR for that, this would be great!).
In my mind the terminal condition is mostly there to have some criterion for terminating an episode early to avoid computational cost. If we only show one object and Monty has low evidence for all others, we don’t have to keep matching. But one could also set it less strictly to shorten episodes or more strictly to always take max_steps before ending an episode. I think of this as mostly an experiment parameter as opposed to a Monty feature that you can adjust depending on what you want to measure. Monty itself has a most likely hypothesis at every step and can use this to act. It doesn’t need to wait until any terminal threshold is reached to be able to use its hypotheses for intelligent action. This is why we, for example, group correct and correct_mlh together in our recent publication evaluating Monty (https://direct.mit.edu/neco/article/38/6/845/136222/Thousand-Brains-Systems-Sensorimotor-Intelligence) (while still separating them in some plots like figure 5 since, of course, being more confident earlier is better).
We will probably need to work more on this parameter as we introduce more hierarchy (since it also plays a role in when an LM sends a message to another LM) since, as you pointed out, being able to uniquely identify an object isn’t always possible if you just have one viewpoint and know about several objects that look similar/identical from this view.
The object_evidence_threshold and the finding that confused episodes often have a low confidence are also interesting results. Thanks for sharing this. I wonder whether adjusting the object_evidence_threshold would also help with our performance on the Monty Meets World Benchmarks.
Finally, one small tip: It looks like you wrote a nice custom logger and plotting for what you need already, but you might be interested in the tbp.teleop tool @rmounir published recently Controlling the Monty agent live with `tbp.teleop`. It’s been a useful debugging tool for our research team lately and can make some nice live visualizations of what Monty sees during an experiment and what its hypotheses look like without having to save large detailed stats files. As far as I know, we haven’t tried it on the SaccadeOnImageEnvironment yet, so I am not 100% sure if it will work out of the box for what you are doing, but maybe worth a try.
Again thanks a lot for sharing this here and I’m looking forward to reading your further updates!
- Viviane