Monty can hear now! Lyon's cochlea → SAI → learning module. Melodies as graphs, transposition invariance falls out

[Bryan_Leavelle]

Yo! Sysadmin / audio-DSP / app-dev guy here, been circling this repo for years without a way in, because I’m not a vision or touch person. Many years ago I read On Intelligence, then Thousand Brains, and have been trying to keep up with Numenta ever since. Three weeks ago I found Richard Lyon’s Human and Machine Hearing and realized Monty was deaf, and that I could fix it.

This is what’s in the repo (I’m still cleaning it up, ignore the references to GODOT ports):

• A real cochlea. No spectrogram here. Lyon’s full CARFAC cascade plus the Stabilized Auditory Image, running in real time, verified against the google/carfac Python reference. Every constant is a published physiological parameter. A hearing scientist can lesion it. An engineer using REAPER can analyze tracks in real time with it.
• An AudioSM. SAI frames become Monty percepts: (channel, log-lag) as location, a small harmonic-timbre vector as the feature. Same skeleton as CameraSM. No audio-specific code anywhere in the learning module.
• Objects learned from one exposure, as graphs. A siren is a path, a drone is a knot, a guitar riff is a tangle you can read the chord changes off of. The JSFX makes an excellent guitar tuner, and I’ll be using that aspect to develop new ideas for my open-source JSFX suite.
• Innate transposition invariance. Because the frame is log-lag, a key change is a translation, and Monty matches anywhere in the tests so far. A tune learned in A minor is recognized in C minor, first hearing, same step count as the in-key take. Tested on a synthetic sweep and on my own guitar, +3.0 semitones on the nose.
• Timbre as texture. Adding the harmonic vector to each node took a drone from “timed out, hedging” to certain in 26 steps. Still no audio-specific code in the LM, which is the point of the theory, and Im happy to report that it’s holding.

Two sensors are better than one, especially if they’re speaking the same language.

I went crazy and kept asking questions, and the system started measuring things I didn’t ask it to. It told two guitars apart by how much the hollow-body’s resonances spawn rival percepts compared to a solid-body Ibanez 7-string. It found a ruler bug in my tolerance geometry (Hz where it should have been cents) by producing a rad result that died under its own control. Every wrong guess turned into an experiment, and every experiment ran in about five minutes. Around 100 runs in one enjoyable, particularly-for-coding, day. I even got to play guitar. Hypotheses died by the pound, my friends, and I think I gained a gram of understanding from it.

At first I was happy just to connect the ideas of two researchers I’ve grown to respect. Then I started running tons of experiments. I have a day job that has nothing to do with neuroscience or AI research, so I’ll keep running these at around two afternoons a week and posting results. But this is just fun for me, and could be useful in other more serious hands than mine. And that brought me here.

Plots, configs, the frozen baseline, and the percept logs are all in the repo: [link]. Everything above has a registered prediction and a control behind it. For what it’s worth, the results ring true to my… ears… and much of the tuning was done through 25 years of audio engineering and electronics experience. It’s nice to be able to put a graph to the mystical world of “tone,” and I couldn’t have done it without y’all’s work on Monty and Lyon’s on the cochlea.

What I’d love from this crowd: eyes on the AudioSM contract (am I abusing the pose plumbing?), thoughts on where mode should live (it’s visible in the residual and invisible in the verdict, which smells like a hierarchy question) and anyone who’s thought about a learning-module-level notion of time, because right now a tune is the same graph at any tempo. Time and phase are just as good indicators of difference as the shape of the signal itself, and something I have taken advantage of in many-a-plugin. I think the model would benefit greatly from a sense of time.

edit: ran an echolocation experiment. Please enjoy the graph below, showing a bat coorrectly guesing the shape of a room on sound alone.

edit 2: late night experiments. 10 bats search with no knowledge to get around the obstacle. Bat 11, fed with the map built with Monty, immediately takes the shortcut. Bat goal includes cost of achieving its goal. Dope.

[Bryan_Leavelle]

ah shit the plot has thickened. new info soon

[Bryan_Leavelle]

OK new fun information:

Audiomonty/eyes - two apertures, for finding stuff. A head that swivels to point ears at a target. I simulated a bat hunting a moth. Before the head swivel, the bat would get get the moths on a horizontyal plane almost everytime, but if the moth escaped vertically, the bat would mis it. Now, the head turns to the sound, the eyes track the moth - total moth annihilation!

Forgive my excitement but by jove yall have made something amazing here.

[jmwright]

This is very cool. I’ve been experimenting with making non-spatial sensor (time-series) data work with Monty, and it seems like I could draw a lot of inspiration from your work.

Time and phase are just as good indicators of difference as the shape of the signal itself, and something I have taken advantage of in many-a-plugin.

Phase is what I’ve been focusing on recently. For instance, phases of the the dry-down cycle of soil moisture with a drift axis, coupled with a plant health metric, to automatically learn/adapt watering for plant health. I’m new enough to Monty that I’m still not sure how practical this is, but I’m playing around with the idea. I believe that HTM had the concept of series/time built into it, but my understanding is that did not get carried over into Monty (different research focus).

[Bryan_Leavelle]

Awesome. That’s a fantastic idea, and here’s the thing I’m seeing in mine: Lyon’s SAI takes the cochlea’s output and turns time into geometry. It’s an axis-picking game — what does this sense actually do, and how do you translate that into Monty LM speak? The best part is that Thousand Brains seems to be right and the brain is already built. The job is to look at the physics that rule the process (lots of low-hanging fruit out there for biosensors), make the data transformation, and send it to Monty. The AudioSM in my repo is the template for exactly that: any sensor plus a made-up spatial axis, learning module untouched. Steal it.

I’d be REALLY interested to see anything you come up with for the soil and time experiments — a dry-down cycle sounds like a closed path through (phase, moisture, drift), which would make it an object Monty can recognize and a pose it can track.

And on phase: as an audio engineer, this is the thing I’m most interested in. Comb filtering is a shape and provides locations. Phase cancellation is a great way to infer physics from perceived silence.

[Bryan_Leavelle]

I went HAM.

Monty now lives in REAPER. Running real time. I’ve had an insane week, but this is crazy.

A learning pass or two on a lick and its minor or backward played variant gives me the information to tell the two apart.

This consumes and logs beats in reaper (not time) as place. Learning in a place assigns a shape to that place. A variant, learned necessarily in another place (time or space, same difference), can still be found when regions are swapped. After, the region can live in both, and are recognizable and seperable. In other words, a bird is a bird, no matter the forest in this model. But some birds are more likely to show up in more places than others.

There is a LUA script that exports the brain state as well. This is tweakable, a nice way to see a lot of information about how works with always on Monty versus single-shot Monty, what I found didn’t, and understanding it better as an accumulator over time than a winner take all bar fight that ends abruptly and never repeats.

Ive tried to leave Monty as unchanged as I can for a port like this. Some stuff I left out because it made no sense over many accumulative runs in time. But Monty now lives in a real time environment coded in the best real time platform ever, and its on my plugin suite now. Ill get it up soon and send a link. Ive got about a thousand uses for this alone I can think of. And all I did was plug stuff into monty.

[sknudstrup]

Well, this is pretty cool! I wasn’t aware of the CARFAC stuff. Very neat. Would you mind briefly describing what Stabilized Auditory Image means?

Q: You say Monty percepts contain “log-lag”. When I hear log-lag, I think of the YIN method or something autocorrelation-based. From what Im seeing, CARFAC works like a fancy filter bank. If so, how does lag come out of it?

Q: I used to have a copy of Reaper, but I’m on Logic pro now. I think I could probably try this out if it’s a VST plugin. Is this something you’ve been thinking about distributing? It’s very cool.

Also, am I right in thinking that there’s no time-dependence being captured? As in, playing C-E-G would produce the same graph as playing E-C-G ? It’s not a critique – that’s how Monty would work out-of-the box right now since we don’t currently handle time in a time-like (i.e., strictly ordered) way. I was just curious if that’s something you’ve played with or tried to “solve”.

Also, for sharing purposes I might recommend forking tbp.monty and adding your changes there. I’m a little unclear as to how your repo is interacting with tbp.monty, though I see tbp.monty is gitignored.

Very cool work, thanks for sharing!

[Bryan_Leavelle]

@sknudstrup

Well, this is pretty cool! I wasn’t aware of the

Also, for sharing purposes I might recommend forking tbp.monty and adding your changes there. I’m a little unclear as to how your repo is interacting with tbp.monty, though I see tbp.monty is gitignored.

Very cool work, thanks for sharing!

thanks dude!

the SAI (stabilized auditory image) sits on top of CARFAC. CARFAC is the filterbank giving per-channel neural activity, then the SAI strobe-triggers each channel like an auto-triggering scope so anything periodic locks into a stable 2D image of channel by lag. so you’re right that the lag isn’t coming from CARFAC, it’s the SAI. sum that image across channels and the marginal acts like a summary autocorrelation, first peak is the pitch. strobe-triggered, not YIN. pitch in SAI is the accumulation across frequency channels at a shared lag, and you plot that lag on a log scale

on running it, it’s JSFX, so honestly the easy path is just grab reaper since it runs native there. logic is my former nemesis that led to me building my own better plugin suite and I flat out refuse to play that game anymore. not trying to be rude, i just have very strong opinions about tech oligarchs and how they made computers suck over my lifetime. Get REAPER, and go Linux while you’re at it

you nailed the time thing. unordered graphs can’t tell C-E-G from E-C-G, same as stock Monty. i added order memory (directed edges), a tempo-invariant inter-onset-interval axis, and a “rail” that pins each note to a beat position, and with those on it separates them. when i say “rail” i mean the amp builder, arduino goof an electrical engineer sense - a thing that connects the other things. ground and positive. but here it’s a hum, basically a temporal reference frame, the time analog of the spatial reference frames Monty already uses. there’s a shared pulse the events get phase-referenced to (think neural entrainment / oscillatory phase coding, Large’s resonance work), so a note’s position is its phase against that pulse. tempo-invariant because phase is relative.

it’s a knob though since strict order gets brittle. idk how stoked i am about making a knob for any of this: but the less i fuck with monty and focus on jamming wires into it that is connected to as many sensory devices i can find, the better my results.

last thing, tbp.monty is gitignored just because i never touched it, it’s reference only. my stuff is a from-scratch JSFX reimplementation, nothing lives inside it, but i can put mine in its own repo if that’s easier to browse.

thanks again, this is a blast.

Edit sorry if cursing is not allowed here

[sknudstrup]

Awesome, thanks for the SAI info! I hadn’t seen this before – looks like I’ve got some reading/playing to do.

I tried Reaper a few years ago on a Windows machine, but I never managed to get the input latency down to usable length (I don’t think I tried on a Linux boot). After getting a mac a couple years ago, I was pleasantly surprised to find that my old Logic Pro license was still good. I’m just a hobbyist now, and I usually don’t do much programming in my free time so ¯\_(ツ)_/¯. Good to know I could easily use Reaper though if I get the itch to write my own plugins.

Cheers man, neat stuff.