Viewing a single comment thread. View all comments

ThatInternetGuy t1_j2d5nkm wrote

Reply to comment by 3deal in An Open-Source Version of ChatGPT is Coming [News] by lambolifeofficial

170GB VRAM minimum.

So that's 8x RTX 4090.

18

3deal t1_j2d8bj5 wrote

I mean, for a startup it is not very expensive for all the benefit it gives.

13

Disastrous_Elk_6375 t1_j2de4o2 wrote

Can the 4090 pool their VRAM? I always thought that LLMs need GPUs from the A/V series so that they can pool memory. Am I wrong in thinking that?

4

zaptrem t1_j2e2lvb wrote

You can do pipeline parallelism via FairScale and HF Accelerate on any identical (and sometimes non identical) GPUs.

3