Quick one. I put a cortex in a DAW this weekend and it runs at seven percent of a core. Here is what that means and what I had to throw away to get there.
The tools, for people who don’t make records
REAPER is a digital audio workstation made by Justin Frankel, the guy who wrote Winamp and then decided the whole industry was doing plugins wrong. Inside REAPER there is a scripting language called JSFX. It runs on a JIT compiler called EEL2 that Frankel wrote about twenty years ago so people could write audio effects in a text file and have them run at sample rate, no compiler, no toolchain, edit and hear it. Most people use it for little EQs and utility plugins. I have been abusing it for years to build amp sims and, this month, a cochlea.
The important property of a JSFX for what follows: it never stops. It runs on every sample as long as the transport is on. There is no main function, no return, no exit code. It is a loop with state and the world pouring through it.
My understanding of Monty in one paragraph
Monty is the Thousand Brains Project’s reference implementation of Jeff Hawkins’s theory of the neocortex. A learning module stores objects as graphs of features at locations in a reference frame, and recognizes them by matching displacements as a sensor moves. Sensors are modules that translate raw input into that feature at location format. Multiple modules vote. It is open source, Python, and very good. Last week I built it an ear, a log polar eye, and a fingertip, and wrote it up. This weekend I ported the ear’s brain into a plugin.
What stayed
The percept stayed. Each frame of the stabilized auditory image with a salient ridge becomes one feature at location: lag in octaves for pitch, membrane place in octaves for where the ridge sits, and timbre read through the cochlea as ratios at the first few harmonics. Frames with no ridge produce nothing. Silence never testifies.
The matching rule stayed, flattened to the auditory frame. Per object, per node: new evidence is path memory times the best old evidence one displacement back, plus how well this node’s features fit right now. That recurrence is displacement matching. A melody walks its graph and compounds. A wrong object gets crumbs.
The graph stayed, with the same few millimeter dedup the Python earned the hard way. A vibrato is a cluster, not a point.
What got juiced out
The terminal condition. Monty is episodic. It accumulates evidence until a condition fires, announces a match, resets, and starts over. That is correct for a benchmark and wrong for an instrument. I deleted it and put a leak in its place. Evidence carries forward every frame and decays through silence. The readout is a standing opinion that strengthens while the sound keeps behaving like the object and drains when it stops. No verdict is ever final because no episode ever ends.
The quorum. Stock Monty declares a match when enough modules are confident, without checking they are confident about the same thing. That is how two witnesses naming different suspects convict a bowl. In the plugin, two objects near a tie is reported as TORN, out loud, not resolved by picking the taller bar.
The reset. Episodic Monty wipes the sensor and the evidence between trials. An always on ear has no trials. It has a room, and the room is the first object it should learn, so I gave it a slot for the background and let it subtract its own floor. The body touching itself.
What did not make sense for an always on computer
Halting, generally. A Turing machine’s whole identity is that it returns. A nervous system’s weird signature is that it doesn’t. Every failure I found in the episodic harness traced back to stopping: the fast sense adjourns the meeting before the slow sense votes, dissent never gets heard, absence never gets counted. Take out the stop and those go from structural to transient.
Wall time. A DAW has beats, and beats are what a musician’s sense of place is made of. So place in this brain is the bar, not the second. A lick learned at bar 9 is an object with an address. A variant learned at bar 40 is a different object with a different address. Swap the regions and the ear still finds them, because the shape carries the identity and the address is only a vote.
Winner take all. The first version picked the tallest bar and moved on. The one I kept has a tally: every time the sensory evidence contradicts whoever holds a bar, the holder’s claim on that bar drops a notch. Territory is earned by being right there repeatedly, not by having arrived first. Getting your ass kicked at your home bar enough times and the room reconsiders who lives there.
How the graphs stabilize
They stabilize because almost everything in the system is a leaky integrator with a different time constant. Evidence leaks per frame. Nodes that never get revisited are the first evicted when a slot is full. Home authority is a running tally, not a flag. Order memory only records a transition once per proven note, so one noisy frame can’t write a false edge. And the note clock: instead of thousands of frame percepts per bar, the brain gets one honest observation per note, so a tune ends up around sixty nodes for a hundred and fifty percepts. That economy is the thing that made the graphs readable, and readable is the thing that made the bugs findable.
The result, for the record: one learning pass on Minor Swing in major and one in minor, a few thirds a semitone apart, and the plugin tells them apart live, mid swap, on each other’s territory. A couple of takes. Not a million.
The brain saves as an FX preset. Mine is called FIRSTGOODBRAIN. You can open it in a text editor and delete a node.
Fun weekend. I am tired and will probably put this toy down for a few days. Yall are cool.
