That’s great, thanks for the update @YiannosD ! We have some paper submission deadlines this week, and then another in-person meetup next week, so I won’t get around to a proper response until the week of April 20th. In particular, I want to have a proper look through your code and configs to give some more detailed advice.
Just a few early comments in the meantime:
- Thanks for sharing the meshes; I loaded those and have attached screenshots of the models in case anyone else is following and is interested in what these look like. Related to this:
- If you are able to visualize the learned models with the colors for each point, that would be helpful. E.g., do we see points corresponding to the trunk in the models?
- It looks like the generic model might be missing a texture element?
- Re. hyperparameters
- As Scott mentioned - it’s worth checking the initial position of the agent at the start of the episode; since these objects are large, we may often be starting inside the tree.
- You mention the trees are ~1.0 meter, and have set
max_sizeappropriately - I just wanted to sanity check this, as I’m surprised the tree models aren’t bigger if they are meant to reflect real-world dimensions? - If you are using the surface agent during learning, you could set the number of steps even higher. That will help with building larger, more complete models.
- Re. learning the trunks, it might be helpful to have an episode where you either rotate the tree to be viewed by the agent from below, or where you translate the agent down, in order to increase the chance that we start somewhere on the trunks. From the models you’ve shared, it looks like the agent always spends most of its time on the foliage at the top.
- Once you’ve played around with the above, I’d be curious if you see any interesting / consistent changes in the clustering as you move from, say 1 –> 10 –> 20 –> 40 epochs of learning.
- In terms of model clustering and the dataset
- Given what the tree objects look like, I would expect things like pine and spruce to be a single model - this has happened, which is encouraging, although interestingly the model doesn’t look like what I would expect. In particular, it seems to just be a blob of points - do you have a sense for the scale of this model vs. the actual objects?
- Similarly, a generic “all tree model” might not be a bad thing, but it would be easier to assess if we had a more diverse dataset. If you get a chance, it could be interesting to add something like 10 cars/trucks to the dataset. Then it will be easier to see whether we distinguish trees from vehicles with generic models, while also having more specific models within each class.
