Recognition through movement on real RGB-D data: what one moving sensor buys, and the failure it introduces

Monty Code
3

Welcome to the forum, @W_Foxalike ! This is a great problem setup and really nice investigations/debugging. Interesting dataset too, I’m thinking this could be a good one to extend our Monty Meets World benchmarks because it introduces sensor movements which comes with more realistic noise, e.g., lighting variability, between frames.

on the step where the viewpoint switches, the new viewpoint’s first observations haven’t aligned with the existing hypotheses yet, so every hypothesis’ evidence wobbles and its slope briefly goes negative.

I think this part requires some more investigation. I don’t see why moving the sensor would cause any misalignment. In the habitat simulator, the distant agent can jump to new viewpoints and continue accumulating evidence without any issues. My suggestions are below.

  • Try to use the data to learn a model of any object first. I suspect that there may still be unresolved issues of intrinsic camera parameters or noise in camera pose. tbp.teleop can be very handy here because you can visualize the learned 3D graph nodes in the LM buffer point by point while controlling the agent. You can start with one frame/viewpoint, and see if moving the agent on the surface of the object results in a reasonable model shape in the 3D graph. You can also inspect the quality of the features you’re getting from the SM processing the dataset’s depth maps (i.e., surface normal).
  • The next step would be to learn and stitch the graphs from two frames given the camera poses. This may require a few transformations, but also straightforward after you get a couple of graphs from two or more viewpoints with tbp.teleop. If you can build a reasonable model from two viewpoints (even if very sparse), then we can rule out dataset noise and any config bugs.
  • It is an interesting idea to fix the issue by resetting the slope window when we expect noisy observations due to large movements. Another way is to maybe try setting the hypotheses age to 0. This is another knob to tune, because we currently guard against deleting a hypothesis if its age is below some threshold. Every evidence update increments this age value by 1.

I think visualizing a single episode live with tbp.teleop would be very useful in debugging these experiments, so if you face any problems with running it on your project, let me know :slight_smile:

1 Like