I made progress on this, and in simulation at least, the glass now terminates correctly every time. Previously it led on evidence throughout but could never eliminate the mug, so every episode ran to its ceiling. There are some caveats though.
I have a branch of Monty here that implements a negative evidence mechanism. All of the configs and utilities to reproduce my results were made self-contained and placed here, and that directory holds a readme which explains the whys and hows of this experiment.
Caveats:
- This departs from the RFC in a significant way. It introduces a sort of “sudden death” mechanism because a graded penalty can’t eliminate - an object scores the max over its hypotheses, so knocking the leader down just promotes the next-best pose, and runs would still hit their ceiling. The penalty suggested in the RFC degrades gracefully, my method will not. It can give a confident wrong answer instead of an abstention. That didn’t happen at the threshold I settled on, but it did at another one I tried.
- It needs a much larger step budget. The glass now takes 643 to 1092 steps, and the hardest case 2396, against a naive scan policy that naturally ends at 307. Cases that already terminated are unchanged or slightly faster. The budget is not what does it, though. With the mechanism off, 3000 steps scores exactly the same as the baseline, 9 of 15 (five objects at three rotations each). The ray carving method can eliminate multiple hypotheses at once, which helps.
- There are only 5 test CAD objects (only 3 of which have been printed and used on the stereo rig), and it is not known how well it will work with large libraries of objects.
- The simulated agent circles the object, so this doesn’t show a single viewpoint is enough. That’s still an open question for my stereo rig to answer.
- Related to the item above - I have not tried this on my stereo sensor data yet. I only used the idealized CAD models for now, just to see if I could make the method work. If used, the data from my stereo rig will have to be “fixed” so that voids in the middle of an object, due to a lack of differentiation, don’t read as negative evidence and a strike against the model.
Also, due to my lack of experience with Monty, I used LLM assistance quite a bit. I wrote the code or have reviewed it, but my depth of Monty knowledge isn’t enough yet for me to recognize nuanced problems in the generated code. View this as an experiment to try to solve a specific problem with my rig (mug vs glass), and not a claim of a polished, finalized solution that’s free of bad assumptions.