Hi @sergioval — thanks, both of these are things I keep circling back to.
On navigating a semantic space. I find this really appealing too — the symmetry with moving through physical space is the TBT picture exactly. Where I get stuck is that reusing Monty’s reference-frame machinery needs a metric space, but the object IDs that LMs pass around are discrete labels with no metric — so there’s no “semantic pose” to move through. Trying to put a metric on a semantic space instead runs into two things: a semantic similarity is only a scalar, not the multi-axis coordinate structure a reference frame needs; and embeddings are unstable — a small change in the input can move the embedding a lot, so distances come out uneven and path integration breaks down. That’s the part I can’t see past yet. (A related discussion runs into similar difficulties here: applying TBP to text understanding ( Discussion on Applying TBP Theory to Text Understanding: Challenges and Potential Pathways - #4 by vclay ).) Do you see a way to give that space enough structure to actually navigate?
On the parent operating at a larger spatial scale. I think “larger scale” here means a larger reference frame, not more input. Along a compositional (part-of) edge, the parent object’s frame spatially contains its children — the whole mug’s frame holds the handle and the logo at their poses — so its spatial extent is bigger by construction. But the parent LM doesn’t receive more raw sensory bandwidth for that: its input is just the child LMs’ ID + pose, which is coarse and low-frequency. So a bigger scale ≠ more or finer input. And it needs no global “height” ordering — only the local parent-contains-child relation along each compositional edge, which sits fine on top of a heterarchy that has no global depth.