2026/06 - Attention and Model-Free Segmentation

[TBP-Announcements]

@nleadholm and @sknudstrup discuss attention as an area-based constraint, something distinct from target poses, that can influence where Monty focuses its resources. We explored concrete approaches to combining model-free and model-based methods, and how algorithms like flood-fill can interact with salience maps to identify regions of interest. Scott presented some early results on using model-free segmentation algorithms on our compositional dataset. There was a discussion around the difficulty of how model-based signals might influence attention, particularly when unexpected sensory input occurs.

Summary Video

Summary - Attention and Model-Free Segmentation

Main Video

2026/06 - Attention and Model-Free Segmentation

0:00 Introduction
1:08 Attention Is a Constraint-Area, a Different Structure than Target Poses
2:30 Salience Is Different Than Attention
2:48 Model-Free Attention Scenarios
6:33 Live Demo: Comparing Different Algorithms
11:06 How to Think About the Challenge of Identifying Compositional Objects (Like a Logo on a Mug)
19:04 Gestalt Psychology Principles and Other Ideas on How to Group Things Together as One
22:35 Initial Seed Location Can Be Defined Without Moving Sensors in Model-Based Policies
25:57 Multiple Learning Modules Can Each Define Their Own Locations to Refine the Attentional Area
35:28 Attention Upscales and Downscales Target Poses Based on If the Target Pose Falls within the Region
44:01 What Makes Something Interesting
49:49 Prediction Error as a Curiosity Signal
51:59 A Radical Idea: Are Target Poses Still Needed?
1:04:23 Question and Discussion - What are the Corticospinal Monosynaptic Neurons Doing if There Is No Target Pose?
1:07:27 Final Brainstorming on Attentional Regions
1:13:48 Wrap-Up

[Falco]

Some musings… probably wrong. Sorry if I waste your time…

Situation: Your eyes rest on a coffee mug, but you’re not really watching. You’re listening to a somewhat intriguing piece of music. And then suddenly your focus shifts.

What happens to all the columns connected to your eyes and ears, and also the many further upstream that govern your extensive knowledge about music and kitchenware? They still get flooded by sensorial and secondary inputs. Do those columns process these inputs?

Instead of attention using a specific area (bucket tool reference) that ‘commands’ what the cortex should be processing on, what if it is inhibiting everything else? Instead of an algorithm using Gestalt principles recruiting relevant columns up front for the task of identifying an object, the inhibitor starts from a tiny ‘straw diameter’ signal to gradually but very quickly extend the uninhibited area until the object is recognized. Along the way, features and child objects may become identified. Once something useful is identified, the expensive column processes are instantly put back to rest. Inhibited.

Moreover, I think free will is an illusion. We are not consciously looking for interesting objects. Instead our brain is randomly peaked by interesting signals and milliseconds later we are convinced that it was our intension all along. That conviction is conscious, and so does become the content of our investigation: we become conscience of the object we’re looking at. What happened just before is deterministically chaotic. (I know this is a controversial position, but bear with me).

So, when my eyes point towards something, some columns become aware of dark pigments. Triggered by chance, some process orders to extend the un-inhibited area… Black pixels form a cluster on a white background… maybe in a shape some columns recognize… Wake up those columns. It’s a capital T… What’s a T doing there? More un-inhibiting… it seems the T is at the start of a cluster of letters… More un-inhibiting. A related column ‘knows’ that an uppercase is usually at the start of a word and for that, my eyes need to saccade to the right. Expensive process, but we’ll allow it for now… lowercase letter h… more letters… five or six more… too expensive, ignore all those and saccade to the end of the cluster… letters nd… More un-inhibiting. Another column gets triggered by an 8 letter word starting with Th and ends in nd. ‘Thousand’. Activate the SDR object ID ‘Thousand’ and put all those expensive processes to sleep.

Why did the process ignore the white space around the T? Deliberately? No, just an accident. It could have picked up on it, and I would have focused on the word in front or underneath. And if I had, I would have been convinced that it was my intention all along.

Why did I focus on that word ‘Thousand’? I shouldn’t have. The music was much more interesting and I’ve seen that logo many times before. It didn’t enrich my experience or expand my knowledge. It was a mistake. But one that I fully own, because I am convinced it was a deliberate decision.

It wasn’t.

[nleadholm]

Interesting thoughts, thanks for sharing @Falco .

I agree with the general premise that there will be many factors influencing what grabs our attention, yet subjectively we may always feel like that is what we “wanted” to attend to. As an aside, I’m with you on the free will as an illusion, but I’ll keep the discussion here focused on attention.

How exactly attention should spread or change moment-to-moment to efficiently recognize objects is an interesting one. In general we’ve imagined it going in the opposite direction. Attention would initially be very wide, and try to recognize large objects (or even whole scenes). If it is successful, then we’ve rapidly identified lots of useful information (i.e. we’ve pulled up a model that can explain a significant chunk of our environment). Only if something is unexplained do we narrow our attention down, and potentially start learning a new compositional object.

For example, if you’re looking at a novel Chinese character, you need to focus your attention in on each stroke to understand it. In contrast, people can typically recognize a familiar word without focusing on the individual letters.

As another example, imagine walking into a room on an alien spaceship: you would start by looking around the objects to understand what they are. If they are too alien to be recognizable, you would narrow your attention to understand what they are made of (are there screens? things for an appendage to grab? buttons?). Perhaps the material itself is some alien technology, and you would go up to study its texture and specular qualities in detail.

This may not always be the most efficient approach - perhaps it depends on certain priors we have about what we are going to see whether we start with narrow or broad attention, but that’s at least how we’ve often envisioned it.