Thank you for the update @AgentRev ! Great to see progress… I hear deer can become pests real quick!
I’m looking forward to running this to see what I get. I didn’t have time to fully review everything today. However, I can offer some high-level overview comments…
I see telemetry nested under tbp.monty.frameworks.loggers.telemetry. I’m only highlighting this because perhaps we are not seeing the relationship between telemetry and loggers the same way. I view telemetry as a peer of normal logging, and a replacement for “loggers”. In code, the way that I would surface that would be by putting telemetry in tbp.monty.telemetry. The “frameworks” organizational structure is deprecated and we’ll move out of it eventually. Or in tbp.monty.frameworks.telemetry if you don’t want to skip steps 
. A peer relationship.
In a similar vein, I see
# TODO telemetry: separate config for log level
self.telemetry = TelemetryEmitter("experiment", level=logging.INFO)
self.post_episode_telemetry = PostEpisodeTelemetryConsumer(
level=logging.INFO, handlers=monty_handlers, output_dir=self.output_dir
)
self.post_episode_telemetry.subscribe()
as part of MontyExperiment.init_monty_data_loggers(), again, sort of nesting it under “loggers.” I think it would be easier to compartmentalize telemetry from the beginning, even introducing MontyExperiment.init_telemetry() for now.
DISCLAIMER: I’m not sure I fully understand the approach. I will attempt to provide some feedback, but perhaps I misunderstood how things are wired. Is there an overall diagram that you maybe have that could give a high-level view of how everything is connected?
Here’s where I’m confused..
I see a self.telemetry.snapshot(...) inside of MontyExperiment.post_episode(), but the snapshot that is emitted uses self.logger_args.
I think what I’ve been attempting to describe differs. I think perhaps I can best demonstrate the difference by focusing on MontyExperiment.post_episode():
def post_episode(self, steps):
# ...
self.telemetry.snapshot(
level=logging.INFO, # TODO telemetry: adjust log level
event=PostEpisodeTelemetry.from_logger_args(
logger_args=self.logger_args,
model=self.model,
emitter=self.__class__.__name__,
),
)
self.post_episode_telemetry.pump() # consumes above snapshot
# ...
I see here telemetry being collected at the episode boundary by introspecting Monty itself. What I expected to see instead was some sort of EpisodeTelemetryHandler that was subscribed to all telemetry events, and as each telemetry event was emitted by the code throughout Monty, it kept a running track of episode specific telemetry. Then, when post_episode() was called, something like self.episode_telemetry_handler.post_episode() gets called and this emits the equivalent of the above.
The main difference I am highlighting is that what I’ve been trying to describe is that PostEpisodeTelemetry event is being assembled step by step, event by event, by some sort of EpisodeTelemetryHandler, and when prompted by the experiment that the episode is over, this EpisodeTelemetryHandler flushes its internal state in form of the PostEpisodeTelemetry. The EpisodeTelemetryHandler.post_episode() is not the moment telemetry is gathered, that’s already been happening step by step, event by event. post_episode() only generates the PostEpisodeTelemetry from already gathered data.
Another way of saying this, the only input into creating PostEpisodeTelemetry would be the events emitted on the telemetry stream. So, the only way of creating a PostEpisodeTelemetry would be to listen in on the telemetry stream and read the data that we need to keep for the episode.
For example, one of the episode stats is episode/correct.
class EpisodeTelemetryHandler(logging.Handler):
_episode_correct: int
def __init__(self, ...) -> None:
# ...
self._episode_correct = 0
# magical future Python version
def _emit(self, record: logging.LogRecord) -> None:
match record.msg:
case LearningModuleTerminalState(terminal_state=terminal_state):
if terminal_state == "match":
self._episode_correct = self._episode_correct + 1
# ...
# today's Python version, something like...
def emit(self, record: logging.LogRecord) -> None:
event = record.msg
if getattr(event, "kind", None) == "LearningModuleTerminalState":
terminal_state = event.terminal_state
if terminal_state == "match":
self._episode_correct = self._episode_correct + 1
elif ...:
# ...
if ...:
# ...
def post_episode(self) -> None:
# TODO: emit the `PostEpisodeTelemetry` event
# reset for next episode
# ...
self._episode_correct = 0
Now, instead of going through loggers and eventually calling get_graph_lm_episode_stats(LM) which sets "primary_performance", which eventually makes its way to the episode_correct count, the learning module itself is responsible for emitting the relevant telemetry. In this particular case, I think this (on first glance, I might be wrong, but the idea is to show the shape of my proposal) would be in GraphLM.update_terminal_condition(...):
def update_terminal_condition(self):
"""Check if we have reached a terminal condition for this episode.
Returns:
Terminal state of the LM.
"""
possible_matches = self.get_possible_matches()
# no possible matches
if len(possible_matches) == 0:
self.set_individual_ts("no_match")
if (
self.buffer.get_num_observations_on_object() > 0
): # lm has gotten input during episode
self.buffer.stats["detected_location_rel_body"] = (
self.buffer.get_current_location(input_channel="first")
)
# 1 possible match
elif (
(
self.buffer.get_num_observations_on_object() > 0
) # had observations on object
and len(possible_matches) == 1 # We have it narrowed down to 1 object
):
object_id = possible_matches[0]
pose = self.get_unique_pose_if_available(object_id)
if pose is None: # No pose determined yet
logger.info(f"Pose for {self.learning_module_id} not narrowed down yet")
else:
self.set_individual_ts("match")
logger.info(f"{self.learning_module_id} recognized object {object_id}")
# > 1 possible match
else:
logger.info(f"{self.learning_module_id} did not recognize an object yet.")
return self.terminal_state
I would say that everywhere you see self.set_individual_ts(...) (ts stands for “terminal state”) is an event emission point for the learning module. For example, instead of, or perhaps in addition to:
self.set_individual_ts("match")
It would be something like:
telemetry.emit(
level=telemetry.INFO
event=LearningModuleTerminalState(
kind="LearningModuleTerminalState",
id=self.learning_module_id,
terminal_state=TerminalState.MATCH
)
)
With this inline emission… now the implementation of def event(...) in EpisodeTelemetryHandler would find this event, and add it to it’s running tally.. and once experiment called EpisodeTelemetryHandler.post_episode(), then EpisodeTelemetryHandler would emit the PostEpisodeTelemetry, which would show up on the telemetry stream after all the previous events. It is important to note that EpisodeTelemetryHandler does not prevent the propagation of telemetry events. However, it is an additional consumer of telemetry stream and keeps a running tally so that it can emit PostEpisodeTelemetry when told to do so.
I gotta run today, sorry. But I wanted to give you some initial comments in case you’ll get around to reading them before I have a chance to read through all of the code.
Thank you again!
One could write an 
