Hi everyone, I’m Ashray. My background is in neuroscience and biomedical research, followed by work in data engineering, Python/data-science teaching, mentoring and apprenticeship curriculum development.
I’m starting to explore Monty and would love to contribute while learning. Developing educational materials seems a useful place to begin. The tutorials I’ve explored so far mainly use experiment configurations. Would a complementary Jupyter notebook walkthrough be useful—for example, stepping through a small experiment, inspecting the outputs and visualising how object hypotheses change as observations arrive? I may have missed existing resources, so pointers would be welcome.
Before choosing a topic, I’d appreciate suggestions about gaps or small projects that would genuinely benefit the community. I’d also be happy to collaborate with anyone interested in developing something together. I can initially commit around 3–4 hours a week.
First, thank you for offering to help out! We’ve been looking to expand into more interactive educational resources to help the community learn about Monty and the Thousand Brains Theory, and I believe your idea of a complementary Jupyter notebook walkthrough would be a useful step in that direction.
One project I’d suggest is a guided walkthrough of how Monty recognizes an object as observations arrive, building on the inference tutorial: how does Monty determine whether it’s observing a mug or a banana?
The idea would be to run Monty live within the notebook, with the learner controlling when it advances. They could inspect each observation, see how the evidence changes, and make predictions before continuing.
For an MVP, I’d suggest:
Set up a short inference episode based on the inference tutorial. Start with a setup cell that loads the pretrained model, creates the simulator, and prepares an episode based on the inference tutorial.
Let the learner advance one step at a time. We’d want Monty and the simulator to stay in memory between cell executions, allowing the learner to advance the experiment by rerunning a stepping cell. This would require some additional code to handle the experiment’s state. Monty has a run_step() method to build on top of here.
Display what Monty is seeing and inferring. Show the current sensory patch, the object itself, and the updated evidence for each hypothesis (the mug and the banana).
Add the teaching material. Include short explanations and questions at selected points throughout the notebook. The questions need to work with a constantly changing state, as no two runs are guaranteed to be identical. Some examples: “Did the new observation increase or decrease Monty’s ambiguity about the most likely hypothesis?” or “Based on the current observation, which object does Monty currently believe is the most likely hypothesis?” or “Before advancing, do you think the next observation will strengthen or challenge that hypothesis?”
With your 3–4 hours a week, one short lesson would be a good initial scope. Your perspective as someone learning Monty will be particularly helpful in identifying which steps need a clearer explanation. I would be here to help you, to discuss the teaching material, and to answer any questions you have.
Does this idea appeal to you? We also have a future work section in our docs that outlines different projects that would be valuable to us, in case any of them appeal to you. The above suggestion would fall into the ‘Build a Tutorial’ item under the ‘oss-communication-improvements’ section.
Some other useful visualizing educational tools we already have that are worth exploring and potentially building an interactive tutorial or lesson around (although I think building with them would be better for a follow-up project, rather than your initial project):
tbp.plot (useful visualizations for existing experiment data)
tbp.teleop (live visualizations during a Monty experiment, you can either configure it to run manually or visualize it as it runs automatically)
Thank you for the thoughtful suggestions and the offer of support. The step-by-step inference walkthrough appeals to me, particularly the opportunity for learners to make predictions and inspect how Monty’s hypotheses change.
My interest comes partly from teaching Python and data science to apprentices, many of whom are curious about how intelligent systems work. Monty’s explicit separation of sensing, learning and action appealed to me as a way to explore intelligence through interacting building blocks. Being able to set a movement policy, follow the observation–action loop, and explore how changing one component affects the system’s behaviour seems a particularly useful aid to learning.
At this stage, my work is still exploratory. I’ve reproduced a pretrained inference experiment and explored sampling patches from video frames. I encountered a simulator setup issue when running inference inside Jupyter, although the terminal run worked. I’m now exploring a trace hook to inspect how observations, evidence and actions change during an episode.
I’ll spend some time exploring the architecture and the resources you suggested, with your proposed walkthrough in mind. I’ll come back with some initial thoughts by early next week. Thank you again—it’s helpful to have a concrete direction that would benefit the community.
Excellent, that plan sounds great to me, Ashray! Have a lot fun exploring and playing around with Monty, and I look forward to hearing back from you next week!
Hi Benji, I’ve reviewed the inference tutorial more closely and wanted to confirm my understanding of the MVP: initialise a pretrained mug/banana episode once, retain Monty and the simulator between cell executions, and let the learner advance one step at a time while inspecting the sensory patch and evidence.
Would it make sense to include a short introductory cell plotting the stored mug and banana graphs, using the existing pretraining analysis example? I’d keep how those graphs are constructed as a separate follow-up lesson.
A few implementation questions:
Which simulator/version should I target? The current tutorial uses MuJoCo, while my working setup uses Habitat.
You mentioned building on run_step(). Is there an existing example that also handles the episode counters and stopping conditions?
For the evidence display, which method or data structure would you recommend using to compare mug and banana?
My terminal inference and trace capture work, but Habitat stalled during setup inside Jupyter. If I can’t resolve that, would it be acceptable for notebook cells to control a separate Python process that keeps Monty and the simulator in memory between steps?
Your understanding of the MVP is correct: initialize a pretrained mug/banana episode once, keep the experiment alive between cell executions, and let the learner advance it while inspecting the observation and evidence. Each episode would show one object, with both mug and banana available in Monty’s memory of learned models as possibilities.
Responses to your other questions below:
I think it’s a wonderful and useful idea to include an introductory graph plot of the mug and banana graphs. That would help give learners a concrete picture of what Monty already knows (and therefore an idea of what Monty is trying to match new evidence data against). Perhaps this cell can have the title: “the models Monty will compare its observations against” or something similar (I threw that idea in because it came to me, but I really want to leave as many of the implementation details to you as possible, so you feel the most freedom and excitement in building this project, so feel free to use that title if you like, and feel free to completely discard it and try something else if you think it’ll work better). I agree with you that we should save how those graphs are constructed for a separate follow-up lesson.
I suggest targeting MuJoCo, using Monty’s current release (v0.57.0) for your work. That release removes Habitat support, as we’ve moved fully to MuJoCo going forward. See this post for context. And perhaps, if we’re lucky, that issue you were having with Habitat inside Jupyter won’t be a problem with MuJoCo.
Yes, check out run_episode_steps in the same file for inspiration. It’s responsible for calling run_step and preserves the next actions, increments counters, and checks whether the episode should finish. Note that `run_episode_steps` itself is set up to run the whole episode straight through, so a Jupyter notebook wrapper would need extra code written to allow advancing one step at a time, then waiting for user feedback before the next step.
For displaying the evidence,I recommend playing around with EvidenceGraphLM.evidence_for_each_graph(). It returns the object IDs and the maximum evidence score among the pose hypotheses for each object with a non-empty hypothesis space. This can be used to give us a straightforward mug-versus-banana comparison. I recommend labeling these as something like “evidence scores” to make it clear they’re not probabilities (a score of 10 does not mean a 10% chance of being correct). Perhaps teaching something very basic about how evidence in Monty works might make sense here. See this doc for more context on evidence in learning modules in Monty.
Since evidence is just a number in Monty, it could be a simple table that says something like ‘Hypothesis’ | ‘Evidence’ and underneath that the next row can be ‘mug’ | 3.2 and ‘banana’ | 9.8 (if 3.2 and 9.2 were their evidence scores, respectively).
Before evidence accumulates, we can show something like “Awaiting observations.” But evidence starts at a count of 0, so just showing 0 in a table could also be enough.
One important aside: some movements won’t change the evidence scores, because the sensor module can filter observations whose features haven’t changed enough. I’m not sure if you’ll run into that in development, but if you do, then it makes sense to have the notebook make that understandable rather than implying every click must change the scores.
Yes, a separate Python process would be an acceptable fallback, if you run into a similar issue with MuJoCo stalling during setup inside Jupyter. The notebook could send an “advance one step” command to a process that keeps Monty and the simulator alive, then receive the observation and evidence to display.
Does everything above make sense? Please ask any clarifying questions you have.
cool ideas, and superb undertaking. I realized, for education, something hassle-free is indeed needed. One can go the python/config way - yes, but perhaps, for today’s concentration spans of users, a UI would be great. That’s why I’m pursuing A Rewrite of Monty in Elixir - as “promised”
the one step at a time is not yet what one’d want but there’s replay of an experiment after it’s finished. While this software is not yet en par with Monty, there’s a python live UI version: What if (?) Monty had a live view
One can perhaps optimize either one for learners - simplifying UX, cross-linking to Monty docs, etc.
There is an MCP to control the experiments in the Elixir version, and it should contain references of where to find docs. Perhaps, some learners might prefer this UX as well.
Perhaps, @Maths, you’ll find such a system of value as well