Monty can hear now! Lyon's cochlea → SAI → learning module. Melodies as graphs, transposition invariance falls out

Research and Theory
1

Yo! Sysadmin / audio-DSP / app-dev guy here, been circling this repo for years without a way in, because I’m not a vision or touch person. Many years ago I read On Intelligence, then Thousand Brains, and have been trying to keep up with Numenta ever since. Three weeks ago I found Richard Lyon’s Human and Machine Hearing and realized Monty was deaf, and that I could fix it.

This is what’s in the repo (I’m still cleaning it up, ignore the references to GODOT ports):

  • A real cochlea. No spectrogram here. Lyon’s full CARFAC cascade plus the Stabilized Auditory Image, running in real time, verified against the google/carfac Python reference. Every constant is a published physiological parameter. A hearing scientist can lesion it. An engineer using REAPER can analyze tracks in real time with it.
  • An AudioSM. SAI frames become Monty percepts: (channel, log-lag) as location, a small harmonic-timbre vector as the feature. Same skeleton as CameraSM. No audio-specific code anywhere in the learning module.
  • Objects learned from one exposure, as graphs. A siren is a path, a drone is a knot, a guitar riff is a tangle you can read the chord changes off of. The JSFX makes an excellent guitar tuner, and I’ll be using that aspect to develop new ideas for my open-source JSFX suite.
  • Innate transposition invariance. Because the frame is log-lag, a key change is a translation, and Monty matches anywhere in the tests so far. A tune learned in A minor is recognized in C minor, first hearing, same step count as the in-key take. Tested on a synthetic sweep and on my own guitar, +3.0 semitones on the nose.
  • Timbre as texture. Adding the harmonic vector to each node took a drone from “timed out, hedging” to certain in 26 steps. Still no audio-specific code in the LM, which is the point of the theory, and Im happy to report that it’s holding.

Two sensors are better than one, especially if they’re speaking the same language.

I went crazy and kept asking questions, and the system started measuring things I didn’t ask it to. It told two guitars apart by how much the hollow-body’s resonances spawn rival percepts compared to a solid-body Ibanez 7-string. It found a ruler bug in my tolerance geometry (Hz where it should have been cents) by producing a rad result that died under its own control. Every wrong guess turned into an experiment, and every experiment ran in about five minutes. Around 100 runs in one enjoyable, particularly-for-coding, day. I even got to play guitar. Hypotheses died by the pound, my friends, and I think I gained a gram of understanding from it.

At first I was happy just to connect the ideas of two researchers I’ve grown to respect. Then I started running tons of experiments. I have a day job that has nothing to do with neuroscience or AI research, so I’ll keep running these at around two afternoons a week and posting results. But this is just fun for me, and could be useful in other more serious hands than mine. And that brought me here.

Plots, configs, the frozen baseline, and the percept logs are all in the repo: [link]. Everything above has a registered prediction and a control behind it. For what it’s worth, the results ring true to my… ears… and much of the tuning was done through 25 years of audio engineering and electronics experience. It’s nice to be able to put a graph to the mystical world of “tone,” and I couldn’t have done it without y’all’s work on Monty and Lyon’s on the cochlea.

What I’d love from this crowd: eyes on the AudioSM contract (am I abusing the pose plumbing?), thoughts on where mode should live (it’s visible in the residual and invisible in the verdict, which smells like a hierarchy question) and anyone who’s thought about a learning-module-level notion of time, because right now a tune is the same graph at any tempo. Time and phase are just as good indicators of difference as the shape of the signal itself, and something I have taken advantage of in many-a-plugin. I think the model would benefit greatly from a sense of time.

edit: ran an echolocation experiment. Please enjoy the graph below, showing a bat coorrectly guesing the shape of a room on sound alone.

bat_waterfall
bat_waterfall1395×1136 166 KB

edit 2: late night experiments. 10 bats search with no knowledge to get around the obstacle. Bat 11, fed with the map built with Monty, immediately takes the shortcut. Bat goal includes cost of achieving its goal. Dope.

blind_cartography
blind_cartography1830×1070 146 KB

9 Likes
2

ah shit the plot has thickened. new info soon

1 Like
3

OK new fun information:

Audiomonty/eyes - two apertures, for finding stuff. A head that swivels to point ears at a target. I simulated a bat hunting a moth. Before the head swivel, the bat would get get the moths on a horizontyal plane almost everytime, but if the moth escaped vertically, the bat would mis it. Now, the head turns to the sound, the eyes track the moth - total moth annihilation!

Forgive my excitement but by jove yall have made something amazing here.

head_tilt
head_tilt2400×1140 177 KB

night_hunt
night_hunt2550×960 154 KB

3 Likes
4

This is very cool. I’ve been experimenting with making non-spatial sensor (time-series) data work with Monty, and it seems like I could draw a lot of inspiration from your work.

Time and phase are just as good indicators of difference as the shape of the signal itself, and something I have taken advantage of in many-a-plugin.

Phase is what I’ve been focusing on recently. For instance, phases of the the dry-down cycle of soil moisture with a drift axis, coupled with a plant health metric, to automatically learn/adapt watering for plant health. I’m new enough to Monty that I’m still not sure how practical this is, but I’m playing around with the idea. I believe that HTM had the concept of series/time built into it, but my understanding is that did not get carried over into Monty (different research focus).

2 Likes
5

Awesome. That’s a fantastic idea, and here’s the thing I’m seeing in mine: Lyon’s SAI takes the cochlea’s output and turns time into geometry. It’s an axis-picking game — what does this sense actually do, and how do you translate that into Monty LM speak? The best part is that Thousand Brains seems to be right and the brain is already built. The job is to look at the physics that rule the process (lots of low-hanging fruit out there for biosensors), make the data transformation, and send it to Monty. The AudioSM in my repo is the template for exactly that: any sensor plus a made-up spatial axis, learning module untouched. Steal it.

I’d be REALLY interested to see anything you come up with for the soil and time experiments — a dry-down cycle sounds like a closed path through (phase, moisture, drift), which would make it an object Monty can recognize and a pose it can track.

And on phase: as an audio engineer, this is the thing I’m most interested in. Comb filtering is a shape and provides locations. Phase cancellation is a great way to infer physics from perceived silence.

2 Likes
6

I went HAM.

Monty now lives in REAPER. Running real time. I’ve had an insane week, but this is crazy.

A learning pass or two on a lick and its minor or backward played variant gives me the information to tell the two apart.

This consumes and logs beats in reaper (not time) as place. Learning in a place assigns a shape to that place. A variant, learned necessarily in another place (time or space, same difference), can still be found when regions are swapped. After, the region can live in both, and are recognizable and seperable. In other words, a bird is a bird, no matter the forest in this model. But some birds are more likely to show up in more places than others.

There is a LUA script that exports the brain state as well. This is tweakable, a nice way to see a lot of information about how works with always on Monty versus single-shot Monty, what I found didn’t, and understanding it better as an accumulator over time than a winner take all bar fight that ends abruptly and never repeats.

Ive tried to leave Monty as unchanged as I can for a port like this. Some stuff I left out because it made no sense over many accumulative runs in time. But Monty now lives in a real time environment coded in the best real time platform ever, and its on my plugin suite now. Ill get it up soon and send a link. Ive got about a thousand uses for this alone I can think of. And all I did was plug stuff into monty.

Screenshot_20260913_223939
Screenshot_20260913_223939552×696 53.9 KB

3 Likes
7

Well, this is pretty cool! I wasn’t aware of the CARFAC stuff. Very neat. Would you mind briefly describing what Stabilized Auditory Image means?

Q: You say Monty percepts contain “log-lag”. When I hear log-lag, I think of the YIN method or something autocorrelation-based. From what Im seeing, CARFAC works like a fancy filter bank. If so, how does lag come out of it?

Q: I used to have a copy of Reaper, but I’m on Logic pro now. I think I could probably try this out if it’s a VST plugin. Is this something you’ve been thinking about distributing? It’s very cool.

Also, am I right in thinking that there’s no time-dependence being captured? As in, playing C-E-G would produce the same graph as playing E-C-G ? It’s not a critique – that’s how Monty would work out-of-the box right now since we don’t currently handle time in a time-like (i.e., strictly ordered) way. I was just curious if that’s something you’ve played with or tried to “solve”.

Also, for sharing purposes I might recommend forking tbp.monty and adding your changes there. I’m a little unclear as to how your repo is interacting with tbp.monty, though I see tbp.monty is gitignored.

Very cool work, thanks for sharing!

2 Likes
8

thanks dude!

the SAI (stabilized auditory image) sits on top of CARFAC. CARFAC is the filterbank giving per-channel neural activity, then the SAI strobe-triggers each channel like an auto-triggering scope so anything periodic locks into a stable 2D image of channel by lag. so you’re right that the lag isn’t coming from CARFAC, it’s the SAI. sum that image across channels and the marginal acts like a summary autocorrelation, first peak is the pitch. strobe-triggered, not YIN. pitch in SAI is the accumulation across frequency channels at a shared lag, and you plot that lag on a log scale

on running it, it’s JSFX, so honestly the easy path is just grab reaper since it runs native there. logic is my former nemesis that led to me building my own better plugin suite and I flat out refuse to play that game anymore. not trying to be rude, i just have very strong opinions about tech oligarchs and how they made computers suck over my lifetime. Get REAPER, and go Linux while you’re at it :wink:

you nailed the time thing. unordered graphs can’t tell C-E-G from E-C-G, same as stock Monty. i added order memory (directed edges), a tempo-invariant inter-onset-interval axis, and a “rail” that pins each note to a beat position, and with those on it separates them. when i say “rail” i mean the amp builder, arduino goof an electrical engineer sense - a thing that connects the other things. ground and positive. but here it’s a hum, basically a temporal reference frame, the time analog of the spatial reference frames Monty already uses. there’s a shared pulse the events get phase-referenced to (think neural entrainment / oscillatory phase coding, Large’s resonance work), so a note’s position is its phase against that pulse. tempo-invariant because phase is relative.

it’s a knob though since strict order gets brittle. idk how stoked i am about making a knob for any of this: but the less i fuck with monty and focus on jamming wires into it that is connected to as many sensory devices i can find, the better my results.

last thing, tbp.monty is gitignored just because i never touched it, it’s reference only. my stuff is a from-scratch JSFX reimplementation, nothing lives inside it, but i can put mine in its own repo if that’s easier to browse.

thanks again, this is a blast.

Edit sorry if cursing is not allowed here

3 Likes
9

Awesome, thanks for the SAI info! I hadn’t seen this before – looks like I’ve got some reading/playing to do.

I tried Reaper a few years ago on a Windows machine, but I never managed to get the input latency down to usable length (I don’t think I tried on a Linux boot). After getting a mac a couple years ago, I was pleasantly surprised to find that my old Logic Pro license was still good. I’m just a hobbyist now, and I usually don’t do much programming in my free time so ¯\_(ツ)_/¯. Good to know I could easily use Reaper though if I get the itch to write my own plugins.

Cheers man, neat stuff.

1 Like
10

gotcha! yea that is strange - maybe i’ll give the vst a whack for the heck of it :wink:

1 Like
11

@Bryan_Leavelle thanks for implementing this, I was away for some time and only just now got round to taking a proper look at your post. Overall this looks really interesting. If you are able to build off of tbp.monty like Scott suggested, and implement it such that the native EvidenceGraphLM is used for the actual model learning and inference, that would be really cool. In other words, if you could implement a fork with your new sensor-module and dataset, and demonstrate this working, then this would be evidence of a clear new modality in Monty.

You also asked about time - while this is not currently implemented in Monty, it is on our roadmap, and you can read more about how we picture implementing it here: Global Interval Timer

1 Like
12

@nleadholm o, i can do that. the problem is that i’ve become obsessed with making multi input realtime monty on a rasp pi. im very close, but i can for sure focus up later tonight and make a CARFAC/SAI with the correct inputs for Monty today. I’ll update here when i make it. thanks for looking!

1 Like
13

Ok great, no rush but yeah that would be very cool to see!

1 Like
14

ok i hope i did this right but here’s the fork. the only other fork i have ever made is for a guitar pedal project hehe. this took some peeling apart, but this is the CARFAC → monty pipeline and audio LM. let me know if you have any questions.

after looking at your post wanting a timing mechanism, i think yall are kinda just describing a DAW. REAPER comes with almost everything you need stock: compute forward, time accumulated between events, ability to trigger actions based on them. i’m working on a headless reaper rack that could be pretty cool.

15

That’s great, thanks! Would it be possible to setup a downloadable dataset, environment, and associated configs so that someone else can run it end to end? That would mean i) others could use the fork as is, and we could potentially feature it on our projects showcase page, and ii) there could be an actual performance benchmark looking at Monty’s ability to recognize the auditory objects, as well as plotting what those objects look like within the LM.

16

yep! i’m working on a way to distribute a data set - since it’s 96k audio the package gets quite large. I’ll add to it sometime soon

1 Like
17

packaging it up now but here is a little figure explaining how audio objects get graphed

IMG_4695
IMG_46952080×992 330 KB

18

Done. The fork runs end to end now: download the dataset, then pretrain, eval and plot. It’s all in AUDIO.md on the audio-sensor-module branch. Held-out recognition is 5/7, and both misses get the tune right and the key wrong. Transposition is 1/2. There’s a plotter for the LM graphs too. One known limit is written up at the bottom: a held note never lets go, because Monty’s evidence can’t say “no”, and this is something i’ve built workarounds for in other various projects, but not here.

2 Likes
19

That’s great, thanks Bryan!

One question from having a quick look at the learned models - are the edges meant to represent anything specific? We’ve generally moved away from edge/displacement matching, although may have a minor role for it in the future. Currently they can still be created as a side-effect of the default DisplacementGraphLM implementation, but given your configuration, I don’t believe they would be used during matching. My question is whether, given they show up in your visualizations of the models, they signify anything to you.

Another question is, given your own background in music, would you describe some of the below failures as more or less wrong?

  1. held-out chords_am → chords_cmaj (relative major, same notes)
  2. held-out swing_am → swing_cm (same riff, minor third up)
  3. transposed swing_cm → swing_maj (same riff, major)

E.g. I would have thought of “held-out chords_am → chords_cmaj (relative major, same notes)” as a more significant failure, since that suggests a bag-of-features-type detection, while “held-out swing_am → swing_cm (same riff, minor third up)” is more desirable, since they are the same riff using a minor chord. Is that consistent with your own interpretation?

20

Thanks! Good questions.

Edges: nope, they don’t mean anything musically. They’re just the nearest-neighbour links the DisplacementGraphLM builds during pretraining. My axes are log-lag (pitch) × cochlear place,
with no time axis, so an edge just joins notes that sit close in pitch. Eval runs on EvidenceGraphLM, so they never get used. I left them in because they make the shape easier to see, but I should’ve said so.

Failures: yep, that’s exactly how I hear it as a musician. chords_am → chords_cmaj is the bad one, and it’s bag-of-features by construction. No time axis means a riff gets stored as a
pile of notes, and A minor and C major are the same pile (all the notes for each exist within each others scale: Cmaj6 and Am7 are literally the same notes inverted ). swing_am → swing_cm is the good kind of wrong: same riff up a minor third, which is just a slide along the pitch axis, so Monty
treats it like the same object moved. That’s what a transposition should be. swing_cm → swing_maj sits in the middle: same rhythm and contour, different intervals. A player would call it
“the same lick, but major.”

Where I keep landing is order. Position alone can’t tell same-notes-different-order apart; time has to be part of the object. The other thing I ran into: in a real-time version I’ve been
building, stock Monty couldn’t vote a hypothesis down. Features score 0 to 1, so a contradiction just withholds support, and the old answer never dies. I added a “no” (a contradicting
feature costs evidence) and suddenly the columns could argue: a three-sense rack (eye, touch, ear) switched objects in about 1.3 s instead of never. Happy to share the writeup if you
want it.