Special guest @Zachary_Danzig shares his undergraduate thesis with the team. Zachary’s hard work brought Monty onto real robotic hardware. His project combined a stereo camera (to act as a distant agent), a time-of-flight depth sensor and a UFactory Lite 6 robotic arm (to act as a surface agent) to explore how multiple sensing modalities can work in the real world for learning and object recognition in Monty. Zachary’s proof-of-concept thesis lays the foundation for future robotics research with Monty.
Main Video
If you want to follow Zachary’s journey putting this project together, you can retrace his footsteps on the forum: https://forum.thousandbrains.org/t/de…
0:00 Introduction
0:08 Introducting Zachary Danzig
1:15 Robotic Object Recognition for Thousand-Brains Systems
1:49 Background: Discovering the Thousand Brains Theory
3:57 Project Aim and Objectives
4:38 Project Timeline
5:11 Monty Meets World Proof of Concept: iPad LiDAR
6:04 Background Research
7:25 Sensor Research
8:09 A Dual-Agent Project Approach
9:09 Sensorimotor System Flowchart
10:09 Distant Agent Setup
11:05 Surface Agent Setup
12:16 Translating Abstract Goals to Robot Motion
14:39 Running through the Experiment
16:39 Filtering Time of Flight Data
17:17 Reference Frames
18:15 The Dataset
18:55 Distant Agent Experiments
21:16 Demoing a Distant Agent Experiment
22:08 Distant Agent Results (Supervised Training Evaluation)
22:24 Distant Agent Results (Rotation Invariance Experiment)
24:44 Distant Agent Results (Continual Learning Experiment)
27:52 Surface Agent Demo
28:18 Surface Agent Results (The Hardware Reality)
29:08 Challenges Faced: Time of Flight Sensor Scattering
30:07 Challenges Faced: Interfacing with the Lower-Level Workings of Monty
30:39 Challenges Faced: Working with a Changing Codebase
31:43 Tips for Future Community Projects
33:29 Extensions to the Project
38:09 What I Would Do With More Time
39:38 Q&A: Tactile Sensors Discussion
44:49 Q&A: Safety Features and Motor Constraints
53:47 Thank you Zach!
