Viewing a single comment thread. View all comments

hx-zero OP t1_j03zy85 wrote

Reply to comment by Acceptable-Cress-374 in [Project] Run and fine-tune BLOOM-176B at home using a peer-to-peer network by hx-zero

Yes, it's technically possible to integrate GPT-NeoX in our code instead of BLOOM (requires some work, but it's not too hard).

Also, it may be possible to fit GPT-NeoX into 20 GB of VRAM (i.e., one 3090) using recent LLM.int8() work: https://huggingface.co/blog/hf-bitsandbytes-integration We use this approach to make BLOOM consume as few memory as possible in Petals.

6