Viewing a single comment thread. View all comments

curiousshortguy t1_jad9s4t wrote

Reply to comment by Beli_Mawrr in [R] Microsoft introduce Kosmos-1, a Multimodal Large Language Model (MLLM) that can perceive general modalities, learn in context (i.e., few-shot), and follow instructions (i.e., zero-shot) by MysteryInc152

it is, you can probably do 2 to 8 billion on your average gaming pc, and 16 on a high end one

28

AnOnlineHandle t1_jaeshwf wrote

Is there a way to convert parameter count into vram requirements? Presuming that's the main bottleneck?

7

metal079 t1_jaeuymi wrote

Rule of thumb is vram needed = 2x per billion parameters, though I recall pygamillion which is 6B says it needs 16GB of ram so it depends.

12

curiousshortguy t1_jaf3aab wrote

Yeah, about 2-3. You can easily shove layers of the networks on disk, and then load even larger models that don't fit in vram BUT disk i/o will make inference painfully slow.

10