Design of Final-Year Undergraduate Project with the Inclusion of the TBP!

Monty Code Projects
30

Hello all, sorry for the delay. It’s about time for a project update.

I’ll first share how I ran my own version of Monty Meets World without XCode and using an iPads back facing LiDAR sensor instead of the front facing TrueDepth Camera.

For starters, to snap the correct depth picture I used this app, allowing me to take a 32-bit float TIFF image, each pixel representing the distance (in metres) to a point in the scene. This app should be compatible with apple devices that have a LiDAR (such as iPhone pro and pro max - 12 onwards, or some iPad pros). It should be noted that the app is a little clunky, but works for the intended purpose.

Unfortunately, I haven’t built a method to stream the images captured to Monty (as is done in the original Monty Meets World). A manual method is required instead.

  1. Use the Depth Camera RAW app on a LiDAR device to capture an image of one of the Monty Meets World pre-scanned 3D objects (I used the Numenta/TBP mug), ensuring the object of interest is located in the centre of the image.
  2. Locate the TIFF and corresponding JPEG image files on the LiDAR device. I found them within the app files.
  3. Use the attached scripts (“jpg_to_reshaped_png.py” and “tiff_to_http_payload.py”) to convert the TIFF file into the accepted http payload and the JPEG into the reshaped PNG (you will need PIL and numpy python packages installed). Example terminal commands are as follows: $ python jpg_to_reshaped_png.py <your_tiff_file_location_here>.tiff <your_jpg_file_location_here>.jpg and $ python tiff_to_http_payload.py <your_tiff_file_location_here>.tiff
  4. From running the example commands you should have two files called depth_x.data and rgb_x.png in the same folder you have saved the scripts in.
  5. Make a folder in following location of your fork of the Monty repo ~/tbp/data/worldimages/<your_scenes>/<your_object> (for instance mine was ~/tbp/data/worldimages/zachs_scenes/tbp_mug), and move your depth_x.data and rgb_x.png files here.
  6. Change the “x” in both files to an index (starting from 0). The experiment will run in this order, based on the scenes and versions you have provided for eval_env_interface_args in the experiment YAML (see next step). If you just have the one pair of images, replace “x” with “0”.
  7. Create an experiment YAML in ~/tbp/src/tbp/monty/conf/experiment/<your_experiment_name> (I called mine zachs_monty_meets_world.yaml)
  8. Write the following experiment config in your YAML file:
defaults:
  - /experiment/config/eval@config
  - /experiment/config/logging/parallel_evidence_lm@config.logging
  - /experiment/config/monty/patch_and_view@config.monty_config
  - /experiment/config/monty/learning_modules/clear_learning_module_configs@config.monty_config
  - /experiment/config/monty/learning_modules/default_evidence_1lm@config.monty_config.learning_module_configs
  - /experiment/config/monty/args/clear_monty_args@config.monty_config
  - /experiment/config/monty/args/defaults@config.monty_config.monty_args
  - /experiment/config/monty/motor_system/clear_motor_system_config@config.monty_config
  # the following config moves 20 pixels at a time (this doesn't work very well when reduced to say 8)
  - /experiment/config/monty/motor_system/informed_no_trans_step_s20@config.monty_config.motor_system_config
  - /experiment/config/environment/world_image@config.env_interface_config
  - /experiment/config/environment/init_args/clear_env_init_args@config.env_interface_config
  - /experiment/config/environment/init_args/monty_world_standard_scenes@config.env_interface_config.env_init_args
  - /experiment/config/environment_interface/clear_eval_env_interface_args@config
  - /experiment/config/environment_interface/world_image@config.eval_env_interface_args

_target_: tbp.monty.frameworks.experiments.object_recognition_experiments.MontyObjectRecognitionExperiment
config:
  model_name_or_path: ${path.expanduser:${benchmarks.pretrained_dir}/surf_agent_1lm_numenta_lab_obj/pretrained/}
  n_eval_epochs: 1
  show_sensor_output: true
  env_interface_config:
    env_init_args:
      data_path: ${path.expanduser:${oc.env:MONTY_DATA}/worldimages/<your_scenes_folder_name>/} # add your path to the scene folder here
      patch_size: 30 # change the sensor patch size
  logging:
    run_name: my_monty_meets_world
    wandb_group: my_monty_meets_world
# To see stats and plots in W&B, uncomment the following lines...
    # monty_handlers:
    #   - ${monty.class:tbp.monty.frameworks.loggers.monty_handlers.BasicCSVStatsHandler}
    #   - ${monty.class:tbp.monty.frameworks.loggers.monty_handlers.ReproduceEpisodeHandler}
    # wandb_handlers:
    #   - ${monty.class:tbp.monty.frameworks.loggers.wandb_handlers.BasicWandbTableStatsHandler}
    #   - ${monty.class:tbp.monty.frameworks.loggers.wandb_handlers.BasicWandbChartStatsHandler}
  monty_config:
    monty_args:
      min_eval_steps: ${benchmarks.min_eval_steps}
  eval_env_interface_class: ${monty.class:tbp.monty.frameworks.environments.embodied_data.SaccadeOnImageEnvironmentInterface}
  eval_env_interface_args:
    scenes: [0] # which scene folder do you take the version from, if all are located in the same folder you can repeat for multiple images: i.e. [0, 0, 0]
    versions: [0] # Change this for specific versions in the specified scene folder: i.e. [0, 1, 2]
  1. Run the experiment with $ python run.py experiment=<your_experiment_name>

There are few modifications to the original Monty Meets World YAML here, one of them being the patch_size. The Depth Camera RAW app are notably smaller than the examples taken for the original Monty Meets World and so the sensor patch size has to be decreased accordingly.

One of the things I didn’t manage to solve was how many pixels the sensor patch moves. Because the Depth Camera RAW images are smaller, the default 20 pixel movement of the sensor patch on the image at each step seemed quite hectic. I created alternative motor system configs that reduced this (to 8 or 12 pixels etc) but this seemed to greatly decrease correct detection. I’m not yet sure why! For now I’ve kept it as is.

2 Likes