The embodied AI sensor module I designed enables cortical columns to autonomously develop functional areas based on input signals, driving exploration through surprise and pain sensation (nociception).
The hardware bill of materials (BOM) is as follows:
5-DoF Robotic Arm: Equipped with 5 motors to capture both position and current feedback.
e-flesh Tactile Sensor: One electronic-flesh (e-flesh) tactile sensor for haptic feedback.
Dual RGB Cameras: Two 720p RGB cameras for visual input.
Dual 6-Axis IMUs: Two 6-axis gyroscopes/inertial measurement units (IMUs).
Dual Microphones: Two microphones for auditory perception.
Can you explain more about the software setup behind this? Are you using Monty? You mentioned that you designed a sensor module, what does this SM do? Is it designed to work with all of your sensors, or do you have one for each modality? Did you modify LMs to incorporate surprise and pain/reward for learning?
I like the e-flesh tactile sensor design, not something I had thought of.
The sense of touch is often undervalued when it comes to sensorimotor intelligence. This is partly due to the lack of technology capable of emulating the sense in its full resolution and sensitivity.
Also, knowing what to do with the data is not easy. Ideally it should be processed in the same way as video might be processed, a stream of data telling the brain about an object’s shape, weight and texture.
All architectural decisions must be based on empirical neuroscience data. Components without biological counterparts or those violating neuroscience facts shall not be introduced.
Handcrafted feature variants → Should be replaced by competitive self-differentiation; Fixed single-column mapping → Should be a macrocolumn group; Secondary columns directly connected to sensors → Should be connected to the primary column’s output.
7
The use of cosine similarity as a core matching mechanism is strictly prohibited. It must be replaced by biologically plausible attractor dynamics iteration or synaptic weight competition.
hippocampus.query() original single-step cosine → Fixed to Hopfield 2-step WTA; MotionLabelGuide.listen_and_act() original cosine → Fixed to a two-step dual-confirmation.
8
Zero tolerance for non-biological technologies. Any technical component must map to a specific neuroscience entity and mechanism; engineering convenience cannot be used as a justification to introduce non-biological components.
Cosine matching (no neuroscience counterpart) → Hopfield attractors; Static point features (no variance) → Perceptual distributions (mean + variance); Fixed precision → Should be dynamic attentional modulation.
This is my latest DIY upgraded tactile sensor, the eflesh-mini, designed to be mounted on a finger.This is my latest DIY upgraded tactile sensor, the eflesh-mini, designed to be mounted on a finger.
Tactile sensing can operate on a variety of problem domains. For example, it can be used to sense an item’s texture (e.g., cloth, sandpaper, vegetable, wood) or recognize encoded patterns (e.g., Braille). I’m curious what domains your sensor can work in.
If you’re looking for test cases, you might want to check in with the Blind community. There are lots of games, teaching tools, and such that might be applicable.
I appreciate the quest for biologically inspired purity, though my own interests lean more toward explicitly hybrid approaches. There is a real danger in taking computer-friendly shortcuts; they may well miss some capabilities that a pure system could achieve.
That said, I see no reason why the training approach (etc) should not take advantage of impure support systems. For example, what is the harm in letting an LLM receive the available inputs and provide its own observations for purposes of comparison, evaluation, supervised learning, etc?
On another note, you say:
Any technical component must map to a specific neuroscience entity and mechanism; engineering convenience cannot be used as a justification to introduce non-biological components.
This constraint may be unrealistic, if carried to extremes. For example, because modeling every impulse, neuron, and synapse explicitly is impractical given available hardware, Monty’s design (and TBT in general) assume that modeling at the level of cortical columns is enough.
However, this leaves the modeling of fine-grained activity and structure (e.g., impulses, neurons, synapses) up to the developers. Similarly, the learning modules use a lot of Python for geometric calculations. Is this (unacceptably) impure?
You seem to be developing a taxonomy for modeling approaches and a set of rules for their use. This seems quite worthwhile; it will be interesting to see how things develop.
