Thanks for sharing this @W_Foxalike , those are some interesting findings and good questions. Also thank you for opening that related PR.
One high-level point to highlight - since you’re already exploring compositional objects, you may be interested in checking out the feature fork we have. It has quite a few elements that currently aren’t in tbp.monty, like attention and a model-free segmentation system, but which are helpful for learning compositional models. It isn’t supported (although we’re working on pulling the key elements into tbp.monty), but I thought I would highlight it to you, given your interests. We should also have a video out soon highlighting the results we’re getting around compositional models with these features.
Q1. Yes this is actually a known issue, and something @jmwright also recently asked about - please see the link here, where I highlight the relevant resources to check out. In the fork I shared above, there is a partial solution - if a hypothesis testing jump fails to reach its target, then this is used to decrement evidence for that hypothesis.
Q2. If I understand your question correctly, there are two aspects to this. One is updating hypotheses sufficiently quickly as incoming evidence no longer matches predictions; this is what burst sampling is for, and it may be that you need to tweak the burst sampling parameters in your experiments. The other, longer term point is the role of top-down connections, which we are currently in the process of implementing. We would expect that, with a learned compositional model, the parent LM would be able to tell the child LM what to expect, and thereby ensure quicker transitions in hypotheses. Does that answer your question?
Q3. In the long term, we would expect deeper hierarchies, where lower-level LMs would be particularly suited to modelling small objects. However, I’m not sure if it’s entirely necessary in your case, or if the issue is more that the policy does not spend sufficient time on the smaller object. I would suggest directly visualizing the intersection of the SM patch size and policy by using tbp.teleop. Some screenshots showing how large the SM patch is relative to the object will be quite illuminating on this question. Let me know if you get stuck using teleop, it’s a very useful tool!
Q4. Thanks very much. If you’re able to add a unit test displaying the case of when Monty is inside an object, that would be brilliant and much appreciated.