Sounds good, and yes for the stereo camera, as long as this is on a fixed base/does not physically move, then SaccadeOnImageEnvironment is indeed the best option.
1 Like
Hi @nleadholm, hope you’re well! I’m back from my 2-week Taiwan Scholarship Trip and am continuing progress on this project. I should be able to extend my deadlines by 2 weeks to gain a bit of time to make up for the time I’ve been away.
Any news on if the refactoring of the motor system is complete, or should I remain wary?
1 Like
Great to hear, when will you officially stop working on the implementation? And is that the same time as when your write-up is due?
Re. the refactor, @sknudstrup and @tslominski have made a lot of progress, but there are still some pieces that that they are bringing together. There is likely to be some pretty significant further changes at least until the end of March. But yeah it should hopefully make your work easier, and I wouldn’t worry about that stopping you - it’s more just something to be aware of in terms of pulling from upstream regularly.
If you are worried about it for any reason, you can also consider pinning your own repository to a particular version of tbp.monty, and working from there. That might be something to consider in your final weeks so that there isn’t any risk of upstream changes breaking your implementation.
The original report and poster submission is on the 20th of April, but I should get a 2 week extension on that so deadline should be 4th May.
To give me time to finalise the report and the analysis I will aim to stop with my implementation around April 20th, with the goal to have experimental data before then.
Classes have come to an end for me for now and it is Easter break for the next 3 weeks. I’ll be staying at my uni so I should be able to spend significantly more time working on the project (and hopefully make some good progress).
Thanks for letting me know, I’ll continue to frequently pull from upstream for now and consider pinning towards the end of the project 
Ok great, sounds good. Let us know if any blockers come up or you reach a stage where you want to revisit the interaction of multiple independent agents at inference time.
1 Like
Some examples of the Surface Agent in motion. This implementation isn’t perfect and doesn’t often find the correct normals or retain the object in view. It does still attempt to follow the surface of the object.
Surface Agent on inverted Bowl:
Surface Agent on TBP Mug:
4 Likes
Very cool, thanks for sharing Zach!
One thing I’m curious about - what is the desired_object_distance you have set for the surface agent policy? I believe the minimum distance for your ToF sensor is around 30cm? It seems that at some points the system gets quite close. I wonder if desired_object_distance is set too low, then the agent might be moving very close and then getting confused as the quality of the depth readings degrade?
Also, have you tried with any objects that have a more matte surface? I can imagine that very glossy surfaces (like these two objects) might interfere with the ToF sensor’s readings.
1 Like
Also do you have some visualizations you can share on what the entire ToF image typically looks like, vs. the sensor patch that is actually passed to Monty? As you’ve also noted, the sensor patch should be quite small relative to the object, so if the code which extracts the surface normal and curvature is getting a view of an entire object (rather than just a smaller patch), this could cause issues.
1 Like
Hey Niels,
desired_object_distance is set to 0.12 (so should be 12cm). The datasheet for the ToF sensor is a bit wishy-washy and states both a minimum of 15cm and 20cm. I am below both these values though so definitely something I’ll try today in the lab. I chose 12cm originally as I didn’t notice a difference and the sensor appears to degrade about ~5cm and below. I was trying to compromise between sensor patch size and distance (obviously the 100x100 array ToF sensor isn’t loads to work when cropping to make a sensor patch small enough to be on the object at a distance).
I’ve got some random matte objects I’ve found that I’m bringing with me today to test.
I haven’t got any visualisations to share of that currently, I’ll get some if I get the chance. Currently I’ve been eyeballing it based on the physical display of the ToF sensor, altering the size of the sensor patch is another thing I will be trying today.
I expect today will probably be my last day in the lab before my deadline on Monday. I’m headed there now and intend to spend a bit of time attempting to fix the Surface Agent output but mainly I will be running formal experiments to collect data. I may have to cut my losses with the surface agent output if I don’t find a fix today but will see what I can do.
1 Like
Ok great, thanks for that info. It sounds like you’ve tried some reasonable trade-offs given the constraints that are here. E.g. I can see now why you have tried having it closer given the resolution of the sensor.
One lesson learned both from your experience and another collaborator we have is that we want to provide some better guidance on what kind of depth cameras are suitable to use with Monty, including details around what properties to look for and how to set them up. I’ll aim to ask you about this again in June for your feedback if you’d be happy to provide some comments.
Your plan for today sounds sensible, you’ve already accomplished a lot. Best of luck with the final experimentation!
2 Likes
Hello all, I have officially submitted this dissertation project and thought I’d share the poster and report with the community! These are both still subject to change if any errors are found. Anywho, I hope you enjoy!
Robotic Object Recognition For Thousand Brain Systems.pdf (2.4 MB)
The project wasn’t perfect and I encountered a lot of difficulty with the ToF sensor and the Surface Agent, which I think ended up being down to the limitations of the sensor. But all in all it was a very interesting project!
The appendices is rather lacking as I just didn’t get a chance to include many items in it before the submission, and was lower priority. However, anything that would have made it into the appendices is accessible through what I have posted here or found in my Github fork.
There is a public Github repo here with my fork of tbp.monty. Beware that it is a tad messy but is formatted reasonably - I’ve used a mixture of LLM agents to help with writing the code. I hope to clean up bits of my parts of the repo at some point soon, in particular the RGBD data collected by the Distant Agent might be better located (currently found in multimodal_monty_meets_world/zed_api/incoming_scenes/). However, it should be noted that there is no intention to merge this with the main tbp.monty repo, this project was a prototype of sorts and may serve as inspiration to others who want to do something similar!
I’m looking into some potential expansions for the project involving scanning the dataset with some 3D scanners to be used in simulation. This dataset of scans will be made available to the community once I have collected them. The following is a sneak peak of this:
Happy building!
6 Likes
Really amazing project! Thank you for sharing the code, thesis, and all the updates during the project 
If you are interested, you could make a PR to our project showcase page in our documentation to add your project and links to the repository and further resources there so the project remains visible to others looking for inspiration on what to do with Monty.
Congratulations to submitting your thesis and such a cool project!
Best wishes,
Viviane
2 Likes
Awesome work!!! Congrats on the amazing project!!
4 Likes
Here are the results for Accuracy Vs. Rotation for each object individually (Experiment 2 - described in the report), as requested recently by the TBP team.
Experiment 2 - Rotation Invariance by Object (Distant Agent Setup)
Dataset
This example of the mc_fox shows the different positioning orientations, rotating clockwise from ORI0 to ORI3. ORI4 is a random orientation.
ORI0 and ORI4 for each of the remaining objects.
Uncropped view. The ZED 2i stereo camera had to be at least 30cm away for accurate depth readings. This impacts the data that the sensor patch is able to collect
Results
Accuracy is measured per episode. An episode counts as correct when the result is “correct” or “correct_mlh”.
Per-object accuracy by orientation
Experiment results can be found here.
5 Likes