Can Monty's Sensorimotor Framework Be Adapted to ARC-AGI-3?

General Introductions
1

Can Monty’s Sensorimotor Framework Be Adapted to ARC-AGI-3?

Hi everyone, I’m an independent researcher and developer interested in brain-inspired AI.

I’ve been studying both Monty and ARC reasoning, and I’m curious whether Monty’s sensorimotor framework could serve as a foundation for solving ARC-AGI-3.

My interest in this began while looking for approaches to ARC-AGI-3. After seeing the limitations of many current deep learning approaches, I started exploring more brain-inspired directions. I looked into predictive coding, world models, and various ANN-based approaches, but none fully convinced me as a path toward general intelligence.

That search eventually led me to The Thousand Brains Theory.

What attracted me to the theory is its proposal that intelligence emerges from many sensorimotor models of the world rather than a single centralized representation. The more I read about it, the more promising it seemed as a possible route toward AGI.

Naturally, that led me to Monty.

After spending time studying the Monty codebase and documentation, my current understanding is that Monty is primarily focused on sensorimotor object recognition and object understanding through active exploration. This is already a major step beyond passive perception and static pattern recognition.

However, ARC-AGI-3 appears to require capabilities beyond object recognition, including:

  • Strategy formation

  • Learning through action and feedback

  • Adaptive world-model construction

  • Reasoning over sequences of observations and actions

In other words, ARC-AGI-3 is a game-learning environment where an agent must discover rules through interaction rather than simply recognize objects through exploration.

This raises a question that I find particularly interesting:

Can the principles behind Monty be extended from sensorimotor object understanding to general sensorimotor reasoning?

One aspect of Thousand Brains Theory that especially interests me is Jeff Hawkins’ suggestion that the same cortical mechanisms used for modeling objects may also underlie higher-level thought. He also mentioned that abstract reasoning can be viewed as operating in higher-dimensional reference frames, just as object recognition operates in spatial reference frames. This caught my attention because when I study different domains, I often find myself thinking what if i increase dimensions for this.

I don’t know whether this idea is correct, but it seems like a natural extension of the reference-frame concept and potentially relevant to ARC-AGI-3.

Some questions I would love feedback on:

  1. Does Monty need an additional reasoning, planning, or memory layer on top or can it be all handled by cortical columns alone?

  2. Could higher-dimensional reference frames provide a mechanism for abstract reasoning?

  3. Has anyone experimented with combining Monty and ARC-AGI before?

I also see ARC-AGI-3 as an interesting benchmark for Thousand Brains Theory. If the same cortical mechanisms and reference-frame principles extend beyond object recognition to general reasoning, ARC-AGI-3 seems like a natural place to test that hypothesis.

2 Likes
2

Hi @Klgv_Klgv

Those are excellent questions!

We think Monty will eventually be an excellent fit for the ARC AGI challenge. Since Monty aims to have all the abilities of the human neocortex, the ARC AGI challenge is definitely within the scope of our vision. We’ve also been thinking more about the challenge over the past months, especially the capabilities that are still missing from Monty to solve it.

Unfortunately, as you already noticed, Monty isn’t quite at the mature, capable stage where it could be applied to the ARC AGI challenge. There are several major capabilities that we still need to incorporate into the algorithm. Luckily, those align well with our existing research roadmap, and so hopefully, as we make progress on that, solving the challenge will come closer to reach. The three major items on our roadmap are:

- Modeling compositional objects (actively being implemented & tested): We live in a compositional world, objects are composed of other objects, and our learned models should make use of that fact, both for efficiency and generalization.

- Modeling object behaviors (theory outlined & implementation scoped but not started Object Behaviors ): The world is constantly changing, objects move and behave in various ways, and models need to learn how an object can change (based on time passing or actions that are being applied)

- Causal interaction/achieving goals (working on the theory): This will be the key for any application (including ARC AGI) where the task goes beyond perception and actually involves leveraging learned models to change the state of the world. Part of this will also be how to set goals and break down goals into sub-goals. It includes learning causal models, simulating the outcome of actions, and using models to plan how to achieve new goals.

Since we still have a lot of open questions about the last one (our process is to try to figure out how the brain could solve a set of problems first, and then test those ideas in Monty), I can’t say how soon Monty will be useful for ARC-AGI. However, if you are interested in contributing to any of the items on our roadmap to help us speed up the process, or you have ideas for potential solutions, we would be excited to hear about that.

Regarding your specific questions:

  1. Does Monty need an additional reasoning, planning, or memory layer on top or can it be all handled by cortical columns alone?

The idea is that it can all be handled by cortical columns/learning modules. Our current learning modules may have to be extended to have better planning/reasoning abilities, but we wouldn’t need a separate component in the system. You might find this part of our recent Q&A useful: https://www.youtube.com/watch?v=50YPPzQ0BfI&t=505s

  1. Could higher-dimensional reference frames provide a mechanism for abstract reasoning?

Potentially. The idea of the TBT is that we would also use reference frames for higher-level, more abstract concepts and reasoning within them. It is unclear whether those actually need to be higher-dimensional or if most/all more abstract concepts could actually just be embedded in 2D or 3D space. The thing that would make them more abstract is that the input to the columns does not come directly from sensors but from other columns that have already processed the input much more and express more “abstract” information (like the identity of a person as input and the reference frame representing the relationship between different people).

  1. Has anyone experimented with combining Monty and ARC-AGI before?

Not as far as I know. Since Monty can’t model object behaviors yet and doesn’t have the ability to change the state of the world (it can set goals, but those are all about where to move its sensors next), it would likely involve several larger changes/additions to Monty to get it to work on ARC-AGI.

I hope this helps!

Best wishes,

Viviane

3 Likes
3

Hi @vclay,

Thank you for the detailed reply. The roadmap explanation, especially around object behaviors and causal interaction, helped me better understand both Monty’s current capabilities and its future direction.

I also found your comments on abstract reference frames very interesting. I’ll spend some time exploring the idea of higher-dimensional reference frames further, as I find that direction particularly intriguing.

Thank you again for taking the time to answer my questions.

Best wishes,
Klgv_Klgv

4

@vclay

I have one more question. From what I understand, attention is an important area that is still being worked on within the Thousand Brains framework, and it seems like a key component for active exploration, planning, and higher-level reasoning.

Are there any papers, blog posts, discussions, or research notes on attention that you would recommend? I’d be very interested in learning more about the current thinking and progress in that area.

Best wishes,
Klgv_Klgv

5

Good question! Attention has been coming up a lot in our research meetings lately, however often not as the main topic so it is a bit hard to sift through our many meeting recordings and find the relevant parts. We have a playlist for brainstorming videos that touch that topic a bit more here: https://www.youtube.com/playlist?list=PLXpTU6oIscrkyfYCeM2oRWCejPNzO0ZzL but it is still a lot and probably not as organized as you would hope for (try using the chapter titles to find the specific parts around attention. We just had another meeting where we talked a lot about attention last week and I can post here once that is published.
Hopefully, we will develop more clarity around the topic in the next few weeks (as we are actually planning to implement attention in Monty as part of our compositionality work) and can share a more succinct write-up.

Best wishes,
Viviane

1 Like
6

I myself have been experimenting on my own HTM/TBT derived system and testing it on ARC-AGI 3’s public tests, plus an assortment of offline tests I made (Sokoban and Tetris being among them).

I didn’t start from Monty’s codebase, I made my own and took a different direction (very heavy on HTM, learned reference frames rather than assumed Euclidean). Still working on building the base architecture to the full spec, and on very small toy tests.

A major theme I’ve had come up while trying to test these ideas is that to do ARC-AGI 3, it needs a system for exploring its environment, forming a hypothesis, and committing to testing that hypothesis. For example, in the Sokoban test, when it has no prior knowledge of the game it would have to learn there’s a goal to reach, learn you can push the block, hypothesis that maybe it should push the block to the target tile, commit to doing the whole thing, and then testing the hypothesis by going to the goal cell at the end.

Some of the things I’ve done towards that end are trying to make a reward system and basal ganglia. Took a bit of an active inference direction, but it’s hard to make that work for the depth the Sokoban example requires.

Hypothesis formation and testing seems to be very under-researched in the AI space. I’d like to know if you know any good sources, examples, or ideas that would help.

1 Like
7

Hi @anon_anon ,

I actually went down a very similar path at first. I also thought we needed explicit mechanisms for exploration, hypothesis formation, commitment, and testing. But the more I tried to model those as separate cognitive stages, the more the architecture grew in complexity. In my experience, increasing architectural complexity tends to make the system much harder to reason about and stabilize.

One thing that pushed me toward a different perspective was studying a course on Complex Systems. It reinforced the idea that nature’s most complex systems are often governed by surprisingly simple underlying rules. The complexity we observe is usually an emergent property of many simple components following the same local principles, not because nature explicitly implements every high-level cognitive process as a separate module. That’s one of the reasons I find the Thousand Brains perspective compelling—the idea that many cortical columns following similar local algorithms can collectively produce intelligent behavior.

Because of that, I’m becoming skeptical of trying to model intelligence the way we consciously describe our own thinking. When we say, “first I form a hypothesis, then I commit to testing it,” that’s an introspective description of our experience, not necessarily the computational mechanism producing the behavior.

My intuition is that behaviors that look like hypothesis formation and experimentation should emerge naturally, without explicitly engineering a “hypothesis module” or a “commitment module.” Even if you manage to make it work through explicit engineering, I think it would mostly solve that specific problem. The solution becomes tied to the assumptions you built into it, making it difficult to generalize. In contrast, if the desired behavior emerges from simple underlying principles, it has a much better chance of transferring to completely new environments without requiring new hand engineered mechanisms.

So rather than trying to engineer things directly, I’d focus on finding the simplest underlying principles. If those principles are correct, behaviors like exploration, planning, and hypothesis testing should emerge on their own. This is the direction I’ve been leaning toward lately. I think there’s probably more to learn from emergence and self-organization than from trying to replicate our own conscious reasoning step by step.

2 Likes
8

Yes indeed. Like a trickle of water finding its way across a slightly sloping floor to the lowest point. Is it using lots of strategy and planning to figure out the route :slightly_smiling_face:

I suspect in many ways intelligent behaviour is often the same, the behaviour happens naturally because that’s what flows out of the neural net. Later we attribute it to strategy and planning.

1 Like