Special guest @Zachary_Danzig shares his undergraduate thesis with the team. Zachary’s hard work brought Monty onto real robotic hardware. His project combined a stereo camera (to act as a distant agent), a time-of-flight depth sensor and a UFactory Lite 6 robotic arm (to act as a surface agent) to explore how multiple sensing modalities can work in the real world for learning and object recognition in Monty. Zachary’s proof-of-concept thesis lays the foundation for future robotics research with Monty.
Main Video
If you want to follow Zachary’s journey putting this project together, you can retrace his footsteps on the forum: https://forum.thousandbrains.org/t/de…
0:00 Introduction 0:08 Introducting Zachary Danzig 1:15 Robotic Object Recognition for Thousand-Brains Systems 1:49 Background: Discovering the Thousand Brains Theory 3:57 Project Aim and Objectives 4:38 Project Timeline 5:11 Monty Meets World Proof of Concept: iPad LiDAR 6:04 Background Research 7:25 Sensor Research 8:09 A Dual-Agent Project Approach 9:09 Sensorimotor System Flowchart 10:09 Distant Agent Setup 11:05 Surface Agent Setup 12:16 Translating Abstract Goals to Robot Motion 14:39 Running through the Experiment 16:39 Filtering Time of Flight Data 17:17 Reference Frames 18:15 The Dataset 18:55 Distant Agent Experiments 21:16 Demoing a Distant Agent Experiment 22:08 Distant Agent Results (Supervised Training Evaluation) 22:24 Distant Agent Results (Rotation Invariance Experiment) 24:44 Distant Agent Results (Continual Learning Experiment) 27:52 Surface Agent Demo 28:18 Surface Agent Results (The Hardware Reality) 29:08 Challenges Faced: Time of Flight Sensor Scattering 30:07 Challenges Faced: Interfacing with the Lower-Level Workings of Monty 30:39 Challenges Faced: Working with a Changing Codebase 31:43 Tips for Future Community Projects 33:29 Extensions to the Project 38:09 What I Would Do With More Time 39:38 Q&A: Tactile Sensors Discussion 44:49 Q&A: Safety Features and Motor Constraints 53:47 Thank you Zach!
Thanks for the interesting presentation. Just on tactile sensors (a favourite subject of mine), there doesn’t appear to be any viable technology out there. The GelSight technique may work for small patches but it won’t scale to large areas. Touch is the third of the big three senses and it will be difficult for any robot to function in the real world without it. A robot will require a touch sensitive skin covering its entire surface. If it could be temperature and moisture sensitive as well that would be good. And self-repairing.
Today’s humanoid robots most likely just have tactile sensors on their finger tips and nothing more. An artificial skin will need to be a tough, flexible pressure sensitive membrane with the pressure data muxed onto a communications bus running around the robot body. Arrays of tiny magnets and hall effect sensors may be feasible, or some kind of pressure sensitive resistor material.
Thanks Alex! I was very interested in the homemade tactile sensor you shared with me at the end of last year. I very much agree with the sense that there is much more progress to be made in tactile sensing, and the technology is nowhere near the biological equivalent today! Looking forward to the next breakthrough in this field as it will bring many new opportunities for robotics!
Hi @Zachary_Danzig , following on from the hall effect sensors discussed in another thread I have discovered that you can buy 2mm diameter magnets very cheaply and SOT-23 packaged Hall Effect sensors. So it would be possible to make a pressure sensor array with 3 or 4mm resolution reasonably cheaply. If built on flexi-pcb material it could even wrap around things, so making sensitive hands and feet is doable. If I had the time a would like to trying building one.Then comes the problem of how to handle the sensor data
Definitely sounds like something that could be used by Monty. 3/4mm precision is pretty reasonable for learning the surface of an object. Let me know if you ever attempt this project because I’d be interested to see the outcome!
It’s really inspiring to see Monty running on real robotic hardware — this is a direction I’m very interested in too.
One idea I keep coming back to: instead of moving the sensor over the object, could the arm hold the object and turn it in front of a fixed depth camera? The sensor-to-object pose would then come from the arm’s kinematics. It’s also closer to how we get to know a new object ourselves — we pick it up and turn it around, but we only have one pair of eyes.
Hi @Zachary_Danzig, got this design in my head now and can’t stop thinking about how to build it. I’m thinking of an 8 x 8 array of sensors which would be about 3 cm x 3cm square. The magnets would be encapsulated in a 3D printed rubbery sheet placed over the SOT-23 packaged hall effect sensors. On the back of the PCB would be the analogue to digital conversion and interface electronics.
If you were to connect it to Monty what would the interface be? a USB serial port perhaps, generating a stream of 64 8-bit pressure values 50 times per second for example.
Other array layouts could be possible, concentric circles maybe, but I’m not sure if anything is gained by that. The concept of ‘sharp’ would be a few sensors having strong signals while the others have weak signals when pressing. ‘Hard’ and ‘soft’ would be the ratio between pressure signals and end effector motion against the surface. In other words, on a soft surface the surface gives so the end effector can move further and pressure signals are small, while ‘hard’ is small end effector motion and high pressure signals.
There is a large temporal element to touch sensor data, which I believe Monty has limited support for at present?
