Disastrous_Elk_6375 t1_j8cd4x4 wrote
Reply to comment by radi-cho in [R] [P] OpenAssistant is a fully open-source chat-based assistant that understands tasks, can interact with third-party systems, and retrieve information dynamically to do so. by radi-cho
I think it will depend on how small the LLMs that it uses are. If they can be run on consumer GPUs, then it will probably take off. If you need to rent 8xGPU servers just for inference, probably not.
Stablediffusion took off because in the first two weeks you could run it on 4GB VRAM GPUs. Then when "finetuning" aka dreambooth came along, it went from 24 to 16 to 8 GB in a matter of weeks. Same effect there.
Zondartul t1_j8hd29v wrote
The plan is to make it kinda good and train in (on industrial hardware) and then distill it down to a smaller model that ideally can fit in a consumer GPU. It's going to be big at first but they do want to make it small eventually.
Disastrous_Elk_6375 t1_j8hdb2r wrote
Do you know if distilling will be possible after instruct finetuning and the RLHF steps? I know it works on "vanilla" models, but I haven't searched anything regarding distillation of instruct trained models.
Zondartul t1_j8i1bdg wrote
Sorry, I just casually watch Yannic Kilcher's YT videos, so I don't know much else.
Viewing a single comment thread. View all comments