Replicable real-world test bed: 3D printing + stereo rig

Research and Theory
1

This thread is for discussion related to the third example bullet given in the post Major Updates to our Future Work Research Items , which is Set Up Replicable Real World Test Bed . I have opened this so that I can ask some questions about that example task.

This first post will be about my hardware, but others are welcome to enter the thread with their own solutions and thoughts.

Potential Implementation

I have created a stereo camera rig, and have used the RGBD data from that rig with Monty to identify (just a few) real world objects which were 3D printed from their source CAD. The rig is 3D printed and uses low-cost camera modules. With some effort, I think this rig could be tailored to be replicated by the community for real-world Monty research. The intention has been to eventually integrate a simple turntable that Monty could command to specific angles for the poses. At this time, there is no plan to vary camera elevation as well, so the stereo cameras would effectively orbit around the object in a single plane. Habitat simulations suggest that this approach should work.

Questions

  1. What should the benchmark look like: how many objects, how many and which rotations, and which metrics (accuracy, rotation error, steps) should be recorded? Should it mirror the sim benchmarks directly?
  2. Is the intended protocol sim-to-real (learn the CAD mesh in sim, recognize the print on the rig), learning on the rig, or both? I currently learn the CAD in sim and then recognize on the rig.
  3. Would a captured dataset (stereo pairs plus turntable pose) be a useful first deliverable, like Monty Meets World and the ultrasound set, before live control? Is there a preferred format?
  4. The turntable moves the object, not the sensor. Is reporting that as the equivalent sensor orbit acceptable? What pose (turntable angle) accuracy counts as “accurately tracked”?
  5. Should the dataset ship calibrated depth maps, raw stereo pairs, or both? The item mentions exploring other ways of measuring depth.
  6. Stereo depth processing leaves voids on low-texture surfaces, and Monty can’t currently tell missing depth from empty space. Is representing “unknown depth” on the roadmap or of interest?
  7. Any cost target or constraints on 3D printers and materials? The rig (without the turntable) is probably below $50 right now. I would guess it will stay under $100 or $150 with the turntable.
  8. Should the object/CAD set be only cleanly separable shapes, or also include a small “hard pairs” set like mug/glass and bar/i for off-object work? I have these hard pairs done in CAD already.
  9. Should it be possible to also use the rig with non-printed objects (spam can, TBP mug, etc)? The stereo rig does struggle with glossy objects without much texture, so using random real-world objects would probably be hit-and-miss.
  10. What is the standard process for pinning Monty for a hardware rig or simulation? Is it acceptable to pin to a sha, or should it always be to a release version?

Tagging @nleadholm

1 Like
2

This is great, thanks for looking into this further Jeremy.

  1. If you’re able to, I would say 10 objects would be a good target, as fewer may be too simple. It would also be good to have objects that share some amount of morphology, like you’ve already done with the mug and cup - that way the task isn’t too trivial. Other examples might be having a cube vs. an elongated rectangle, a cone, and a pyramid. Essentially local features shouldn’t be sufficient to immediately discriminate the objects. Ideally it would be possible to have the objects in three rotations of i) upright, ii) upside down, and iii) on their side (where the side vary from time to time). It might be worth thinking about what this means for stability when designing the objects. Note that it will generally be difficult to assign the ground-truth rotation (even if upright, what is the rotation about the vertical access?), so the main question will be whether Monty can correctly infer the object, and how many steps this takes.
  2. Ideally both - but starting with sim2real would be reasonable. Again, because it is difficult to define the ground-truth rotation of the object, then the options when learning will be either i) learn as much as possible given a single orientation of the object on the turntable (e.g. upright), or ii) use unsupervised learning, and (eventually, probably not the existing version of Monty) the different models learned at different orientations will be merged together.
  3. Yes if you could have something like Monty Meets World, but where you also have turntable information, that would be a really nice, parallel deliverable. This would mean that, even for those who don’t want to build the entire rig, they could still experiment with real world data offline, including finding ways of improving performance.
  4. Yeah that is fine. It would however make sense to formulate the “efferent motor copy” of rotation such that the sensor patch perceives itself moving. See the example of get_state in the EverythingIsAwesome repository.
  5. Both would be amazing, as then others could explore alternative depth measures, as you mention.
  6. I think the cleanest thing for now would be to treat this similar to an off-object observation. If this is the case for all points on the object, then it wouldn’t be possible to learn it.
  7. If you can keep it under $100, that would be great. But I think $150 would also be reasonable. If the depth measurements are the main bottle-neck, then maybe having the option to swap in an alternative, more expensive option for this could be interesting.
  8. Per (1) we want to make the task relatively hard. However, please don’t distinguish them by a tiny, barely perceptible feature (e.g. a cube with a small dent on one surface).
  9. I think it would be nice if people can in principle test these, even if it doesn’t work well for now.
  10. I don’t have any strong view on this point. However I would have thought that pinning to a release version would be simplest. If you need to use something else for some reason, then it’s something we can discuss with the engineering team.
1 Like
3

This is really cool stuff dude. Im walking a similar path right now, 3d printing parts and jamming sensors into a robot. Im very interested in seeing your results. From my experience:

  1. the “motor” in “sensorimotor” is legitmately one of the best parts about Monty and you should try to use it. A few servos and a night at the printer and you can have a moving apparatus that takes Monty’s output easily. Its extremely cool, and its already in there, for free.
  2. add some ears! I have an implementation of it here
  3. Any thoughts on whisker-senses or using piezos/ copper tape for touch? Cameras do not do well up close, and I have found that the eye and touch share a failure mode that leads to stacking errors.
1 Like
4

the “motor” in “sensorimotor” is legitmately one of the best parts about Monty and you should try to use it.

I plan to have a turntable, but that will be the only DoF for this setup, other than being able to manually adjust the height of the cameras. I don’t see a reason that a stereo camera rig like this couldn’t be mounted on the end of a robot arm, but it does sound like there are complications with that setup that I’m not quite ready to tackle.

add some ears!

Multi-modal sensing would be cool, but this rig will probably stay with just vision. A reference build of your echolocation setup would be really cool though.

Any thoughts on whisker-senses or using piezos/ copper tape for touch?

I spent just a little time looking into a low-cost touch sensor, but haven’t gotten very far with it. Piezos is one option that I looked into. That route has its own issues, but it should be possible to make it work. This stereo rig will absolutely fail at close range, and I’ve proven that many times when trying to do calibrations.

Other options I’ve considered, that would raise the cost too much for this specific rig, but that are still interesting.

Neither is a touch sensor, but would probably allow the robot to get much closer before it loses it’s ability to resolve (see edit below for radar). Radar can also give you direction and speed, which could be really interesting as Monty gets more capable (i.e. wildlife population tracking including direction and speed of travel). I don’t think that touch or radar are viable for Monty with just a turntable, but I could be wrong about radar. I’ve never used it in a project.

Edit: Actually, I may be wrong about the radar sensor being able to get closer than the stereo cameras. I found the range listing for one, and it starts at 1.2 meters, whereas this stereo rig works from about 350 to 700 millimeters. The RGB-D camera I listed can go down to 100mm though.

2 Likes