[D] Is it possible to train LLaMa?

Submitted by New_Yak1645 t3_11nhl03 in MachineLearning

Most AI is impossible to train(like chat GPT)

Dose LLaMa can be trained?

Although the dataset is very hard to get, It would be nice if LLaMa can be trained.

When searching for reddit, this topic cannot be searched, so I hope it becomes a discuss about HW or availability.
Thank you.

0

Comments

You must log in or register to comment.

CKtalon t1_jbnccl7 wrote

If you have a few thousand A100s, sure? The dataset is fairly easily obtainable.

The next difficulty is the technical knowhow to train such LLMs.

16

UnusualClimberBear t1_jbngux4 wrote

Training from scratch required 2048 A100 for 21 days. And it seems only to be the final run.

I guess you can start to fine-tune it with much lower resources, 16 A100 seems reasonable as going lower will require quantization or partial loadings for the model.

7

potatoandleeks t1_jbnl6se wrote

Wow, they cost $15k a piece. So that's $30 million just for the GPUs! But since you only need them for 21 days, can probably sell them later on craigslist

6

SomewhereAtWork t1_jcf9g5p wrote

Would it be possible to train a quantzised model?

1

ch9ki7 t1_jbneot3 wrote

I would start searching on huggingface.co

2

Raise_Fickle t1_jc1tb4r wrote

Any idea for finetuning llama on multi-gpu setup?

1