Hi @AgentRev ,
sorry for taking so long to respond to your post! Those are some good questions and thoughts. I think it may be useful to clarify a few potential misunderstandings first:
The analogies you mention here
-
- Visual Sensor Modules = Retina → lateral geniculate nucleus (LGN) → V1 → V2 → V4;
- Object Learning Modules = Inferior temporal gyrus (IT) / lateral occipital cortex (LOC) → perirhinal cortex (PRC);
- Action Spaces, Policies, Goal States = rudimentary version of V5/MT → posterior parietal cortex (PPC) → posteromedial cortex (PMC), in addition to basal ganglia.
are not exactly how we would think about the mapping of Monty onto the brain. Instead, everything that is in the neocortex would be learning modules (including columns in V1, V2, V4, as well as MT, IT, PPC,…). Sensor modules are whatever sensory processing happens before information goes to the learning modules. This processing does not require structured models and is much more basic. The thalamus (which includes LGN) is a place where all the information is initially routed through before it reaches the cortical columns. This is where we propose that the reference frame transformations happen (more details in this paper: [2507.05888] Hierarchy or Heterarchy? A Theory of Long-Range Connections for the Sensorimotor Brain particularly section 3.2). In Monty, this piece is included in each learning module, similarly to how the Thalamus is sometimes conceptually described as the seventh layer of the neocortex.
The whole premise that the TBT is built on is that no matter where in the neocortex a cortical column is located, it works the same way. The only thing that determines what kind of models it learns is where it gets its input from (I can talk in a lot more depth about this part if this is confusing). So it’s not like we plan to write custom learning modules or other components for the different areas of the neocortex. They will all work the same and just connect to different sensor modules and actuators.
Take the example of learning 2D surface models (one of our ongoing research projects): the LM is untouched, and we simply write a custom sensor module that extracts different features and 2D movement along the surface of objects.
Or learning behavior models (how objects move and change over time): One of the really big key insights we had last year was that we can use exactly the same mechanism we use for modeling static objects (+adding a timing input). The LM that learns behaviors simply receives changes instead of static features as input. This is described in a bit more detail here: Object Behaviors
Here is one of our research meetings where we talk about how the exact same column structure could lead to learning various types of models simply by providing different input: https://youtu.be/nWRWb5c3zJk?si=gLBu8sjpCyqYdIB0
The hippocampus is outside the neocortex and does very fast associative learning (for example, quickly building up a map of the room you are in). One of our proposals is that the basic mechanism which cortical columns are built on, first evolved in the entorhinal cortex and was then generalized into the cortical columns. We will have to see if some special modifications are necessary to model the role of the entorhinal cortex, but for now, our thinking is that we can just use a learning module that learns very quickly and forgets quickly again. Here is a research meeting video where we explore the role of the hippocampus a bit more https://www.youtube.com/watch?v=tmljWGLgM70 (there are some more videos around this about to be released in the next weeks).
So in Monty, we don’t plan to add fundamentally new components to the system. Instead, we will need to add a few more tweaks to the general-purpose learning module algorithm (such as adding a temporal component to its models and using the models more for action outputs and planning). As @evanuno mentioned, the large range of applications would then be enabled by people using this general-purpose learning unit (LMs) and providing it with different data in different settings. The idea is that when someone has a custom type of sensor, all they have to do is write a new sensor module that translates the raw sensor data into the CMP (the language that any learning module can understand) and potentially a custom motor system that translates LM output goal states into the raw action format the specific actuators understand.
In terms of fundamental capabilities that we will need to add to Monty, this video might be useful: https://youtu.be/Iap_sq1_BzE?si=sRAVtWKmm_qjfYp6 The TL;DR is that there are 3 more big items on our roadmap:
- Modeling compositional objects (including scenes): We have a solid theory and built prototypes for this last year and are integrating them into Monty at the moment. Monty has a basic ability to learn compositional models already, and the first half of this year is dedicated to making it more robust and performant.
- Modeling object behaviors: We developed a good theoretical understanding of how the brain could do this last year, as well as * a concrete plan of how we would implement this in Monty. This year, we plan to prototype those ideas.
- Learning causal models & producing actions to achieve goals: This is the biggest open question in our theory at the moment and our main focus for brainstorming in the next year.
There are a couple of other capabilities that will need to be added (like scale invariance, learning associative connections between LMs, attention, and object deformations), but we have some good ideas around those as well, and hope that they will fully clarify once the main 3 items above are also solved. We wouldn’t expect those to be new Monty components either, but instead tweaks to the learning modules or the information routing between them. Those topics frequently come up in our research meetings when we talk about the big open questions, and we’ve also made some progress on them in the past year.
I can go into more detail on how any of the bullet points you listed are planned to be tackled in Monty and how each LM (or specialized SMs/motor systems) should implement them if you like.
I hope this makes sense and maybe simplifies the big picture view for you a bit more.
Best wishes,
Viviane