Hi @SafinVladislav, these are good questions.
Object Individuation: What does Monty use as an individuation criterion? In other words, what actually counts as an “object”?
Conceptually, Monty treats an object as a collection of features at locations in a common reference frame. Operationally, during pretraining/learning we currently present objects in isolation, so the learning setup supplies the initial object boundary, i.e. observations from an episode are used to learn one object model. During inference, this assumption is not required as burst sampling lets Monty update its hypotheses when incoming observations shift from one object to another.
Reference Frames: When is a new reference frame created? Is it when the old one no longer fits the incoming data?
I think you have the right idea. As Monty moves over an object, it accumulates evidence for hypotheses over the object models already in memory. When no stored model remains a possible explanation of the observations, Monty reports a no_match and creates a new object model (which has its own object-centric reference frame). If an existing model remains a plausible match, Monty associates the observations with that model’s reference frame rather than creating a new one. During learning, those observations can then update and extend the existing model.
Dimensions: What is a “dimension” in Monty, and can new ones be added?
Hm, I’m not sure what you mean exactly by “dimension”, but two different things come to my mind:
-
A spatial dimension in which movement occurs (current Monty expects movement in 3D Euclidean space)
-
A space of features, in which a Monty’s sensor module can extract.
Would you mind clarifying what you mean in this question?
Sensorimotor & Geometry: How does Monty recognize objects regardless of their position and orientation? Does it apply geometric transformations to them, and can these transformations be learned?
As I mentioned in the first answer, an object in Monty is a collection of features at locations. During recognition, Monty maintains a list of hypotheses about both the object identity and pose. As Monty moves, it measures a displacement in the common body/world frame. For each of its hypotheses, it rotates that displacement and the sensed feature pose into the learned model’s reference frame, then checks whether the stored features and incoming observations match. Depending on how well (or not well) these match, Monty increments (or decrements) evidence for each hypothesis. The docs on Evidence Based Learning Module explains more about hypothesis and evidence updates.
This process continues until only one hypothesis is left (where Monty has recognized the object and its pose) or some other terminal conditions are met. The rules for applying rotations and translations are currently explicit geometric operations, not learned functions. The transformations are described mathematically in our DMC paper and the Reference Frame Transformation page goes into details of how poses and movements are mapped between frames.
Manipulation vs. Observation: Are the models Monty builds designed to facilitate object manipulation, or are they purely observational?
Yes! The learned models of Monty are designed to facilitate object manipulation and not just observation. Since they encode an object’s features in a structured reference frame, they can be used to predict observations and guide actions toward particular locations or features. For example, we have a hypothesis-driven action policy, which uses learned models to choose informative movements for faster object recognition. We are also actively exploring how to use these same learned models to generate goals and interact with the objects (this playlist has recordings of discussions related to your question).