Hi @Tongjian_AN ,
sorry about the late reply to this and thanks @AgentRev for the great response and analogy!
We haven’t thought much about text-only learning because this seems fundamentally different from how our brains learn. But I can try and give some of my personal thoughts on this, especially in the context of your specific problem.
Monty (or more specifically, the learning modules inside Monty) implements a general-purpose algorithm for learning structured models of things it is sensing. A learning module (LM) learns how features are arranged relative to each other. So, in the context of language,e this could be models of letters composed of strokes, words composed of letters, or sentences composed of words.
When stacking LMs hierarchically, they can learn compositional models. So models that are composed of other models. This is an extremely powerful concept and capability. A lower LM may learn models of letters composed of strokes. The recognized letter would then be the input to a higher-level LM, which could learn models of words composed of letters. The higher-level LM doesn’t have to relearn how each letter looks exactly (i.e. the strokes of it), it just stores an arrangement of letter IDs, and if a letter appears multiple times, it just references that model of the letter learned at the lower level twice. This is super useful to efficiently learn the large variety of words composed of a limited number of letters.
When it comes to the next level up, you may learn models of sentences. But you wouldn’t learn a model of every possible sentence out there (there are simply too many!). Some higher-level models can be more general (e.g. learning general arrangements of types of words like nouns and verbs). Those models may reflect the grammar of the language you are learning and can be used to bias which words at the lower level to expect next.
Getting a bit more towards your concrete application, to actually learn the meaning of words is a bit trickier with a language-only setup. The data just isn’t there. All you can learn are statistical regularities. You can learn that in a similar context (i.e. higher level model), two different words may occur (top-down prediction to the word modeling column), which could be a way Monty may learn synonyms. Similarly, classes of words (like the mechanical equipment) could be learned based on lower-level word models appearing in the same context in the higher-level sentence model. However, I am having a hard time thinking about a good way to learn logical relationships (like > than or 30%) from text data only. I guess with sufficient examples, it could learn the combinations of 1>2, 2>3,… and with enough examples that would work but it isn’t really a solution where the system understands numbers or what it means for a number to be larger than another one.
One very speculative idea for tackling this type of task could be to have “logical/conceptual” columns that receive some kind of formal or conceptual representation of the text. Again, this is a very unformed idea, and I would have to think through it much more. But, for example, when presenting the number “3” to an LM (the language one), another LM (the logic one) could get a more visual representation of 3 dots. When the language LM receives “>”, the logic LM could receive two number representations next to each other where one to the left is larger than the right one. Or it could learn a behavior of subtracting the left from the right number and seeing if it remains a positive number. The two LMs could then learn associative connections (what we call lateral voting) between their models.
These ideas are not very fleshed out at all but what it hints at is that this would then not be a text-only system anymore, but it wouldn’t require some other perceptual input for learning the meaning of words. As I mentioned before, I don’t think you can extract any real meaning of words, beyond statistical correlations with other words, without having non-language models to associate them with. If language is not grounded in models that were learned through experiencing what the language is describing, I don’t see a way that it could represent this information.
I hope this makes sense and isn’t too disappointing.
For what it’s worth, I do think it would be exciting to explore language modeling in Monty. I don’t think it would require any modification to the LM for Monty to ingest text data. For instance, you could have a sensor module that sends letter IDs as features to the LM as well as their location in the string of text. Or you could take a lower-level approach and have a vision LM extract strokes and learn letters first. Or you could take LLM-like token representations and send those as features to learning modules. Movement would basically be in 1D space (backwards and forwards in the text). But Monty could learn complex, compositional, and nested models of the language. The main issue I mentioned is about associating the learned language models with meaning and understanding what the words refer to, to solve tasks intelligently.
Best wishes,
Viviane