I did an experiment to see if I could train Monty on CAD models, then 3D print those models, and then get Monty to recognize the prints on a low-cost stereo rig that I’ve built.
It was a simple experiment with only three objects and a static rig (no turntable):
- A mug
- A glass that was just the mug with the handle removed
- A block that is essentially a large calibration cube
The glass and the mug were speckled using stick-on labels. Bare glossy filament gives a block matcher nothing to correlate, so curvature came back much flatter than it is. I don’t think that’s specific to a low-cost rig, it’s why commercial depth cameras often project their own pattern.
The setup worked as a simple test and gave some encouraging results. The mug and the block were both identified correctly against the CAD-trained library, and a deliberately unspeckled control was misidentified as the block because the glossy plastic cost the curvature.
There was one interesting failure mode. I ran into trouble that I think is linked to the glass being a subshape of the mug. Here’s the theory on the failure mode: when one object’s graph contains another’s, a capture of the contained object can’t discriminate, because the absence of a handle is never an observation. Demonstrated by the mug capture resolving to rig_mug in 2468 matching steps, while the glass capture ran to a 25,000-step ceiling with rig_mug and rig_glass both still in play. rig_block was correctly eliminated in that run, so it isn’t just noise. A turntable should help recognize a mug, but I’m less sure it helps with the glass. A hypothesis scores -1 when an observation lands where its graph has no node within max_match_distance, but that only fires when the observation falls outside the model. A mug capture disconfirms the glass that way, because the handle lies beyond the glass’s extent. A glass capture never falls outside the mug, so no viewpoint produces the evidence that would rule the mug out. Am I understanding that correctly, and is there any mechanism that penalizes a hypothesis for a stored feature the observation has never visited?
My repository was created to explore the idea of a low-cost, dual camera wildlife trap that could identify and count species using stereo depth perception. It seems like Monty may be a good fit for that purpose eventually. I do wonder if the failure mode outlined above may matter for that application specifically though. A juvenile is often close to a subshape of an adult, and species that differ by one feature - antlers, tail length - might sit in the same relationship as my mug and glass. That said, the conventional alternative has its own problems: large training sets often do not exist for endangered/rare species, and so it can be hard to properly train a traditional ML setup to accurately identify what species (and how many) are in the frame. Some scientists have resorted to having AI generate images to train their setups.
Disclaimer: AI assistance was used throughout the experiment.