[R] [N] Toolformer: Language Models Can Teach Themselves to Use Tools - paper by Meta AI Researchswegmesterflex t1_j8d3t4r wrote
Had this idea and was planning to play around with it when I had more free time. Good to see some evidence it’s a promising direction. I speculate you can actually get a LOT out of this if you’re clever with it. A tool for long term memory could be done by having a lookup table with text embeddings as keys. A tool for vision could be made with an image captioning model + maybe some segmentation to get a richer text description of the image. Many more things you could come up with, that I think could work well if you find some clever way of turning them into text.
MysteryInc152 t1_j8ppoiq wrote
I'd rather the basic senses at least (vision as well as audio) be pretrained as well. We know from Multimodal chain of thought as well as scaling laws for generative mixed modal language models that multimodal models far outperform single modal models on the same data and scale. You won't get that kind of performance gain leveraging those basic senses to outside tools.
Viewing a single comment thread. View all comments