Thank you Niels for the thorough answers and references.
I want to try to refine my question and get your feedback on a specific idea that I presented, that is, using only a middle layer of an LLM as a gateway for Monty to learn a linguistic model.
For example, here is one paper analyzing the roles of different layers in LLaMA. What I was hoping is to shortcut only the fundamental part of language, basic syntax, basic meanings of words and sentences (maybe also through audio). Then, let Monty develop real structural concepts and nuances that are a combination of the LLM’s basic processing of speech and Monty’s higher-level spatial reasoning. Does that make sense?
1 Like