Viewing a single comment thread. View all comments

Gatensio t1_jd6rixk wrote

Reply to comment by KerfuffleV2 in [D] Running an LLM on "low" compute power machines? by Qwillbehr

Doesn't 7B parameters require like 12-26GB of RAM depending on precision? How do you run the 30B?

1

KerfuffleV2 t1_jd7rjvf wrote

There are quantized versions at 8bit and 4bit. The 4bit quantized 30B version is 18GB so it will run on a machine with 32GB RAM.

The bigger the model, the more tolerant it seems to quantization so even 1bit quantized models are in the realm of possibility (would probably have to be something like a 120B+ model to really work).

3

ambient_temp_xeno t1_jd7fm8a wrote

I have the 7b 4bit alpaca.cpp running on my cpu (on virtualized Linux) and also this browser open with 12.3/16GB free. So realistically to use it without taking over your computer I guess 16GB of ram is needed. 8GB wouldn't cut it. I mean, it might fit in 8gb of system ram apparently, especially if it's running natively on Linux. But I haven't tried it. I tried to load the 13b and I couldn't.

2

ambient_temp_xeno t1_jdcpvhv wrote

*turns out WSL2 uses half your ram size by default. **13b seems to be weirdly not much better/possibly worse by some accounts anyway.

1