Viewing a single comment thread. View all comments

singularpanda OP t1_j3e6asd wrote

Reply to comment by f_max in [D] Will NLP Researchers Lose Our Jobs after ChatGPT? by singularpanda

Yes. That's the benefit of in the big companies. However, for a lot of NLP researchers like me, we do not have that many gpu resources(I believe most of the companies also cannot afford this).

5

f_max t1_j3eagrm wrote

Right. So if you’d rather not shoot to join a big company, there’s still work that can be done in academia with say a single A100. Might be a bit constrained at pushing the bleeding edge of capability. But there’s much to do to characterize LLMs. They’re black boxes we don’t understand in a bigger way than maybe any previous machine learning model.

Edit: there are also open source weights for gpt3 type models w similar performance. Ie huggingface BLOOM or Meta OPT.

3

singularpanda OP t1_j3elwu4 wrote

Seems recently, not too much paper are doing on them. Don't look at details. Maybe models like OPT is still too large?

1

f_max t1_j3frqfb wrote

They have a sequence of models ranging from 6B params up to 175B largest, so you can work on smaller variants if you don’t have gpus. There’s def some papers working on inference efficiency and benchmarking their failure modes if you look around.

1

Think_Olive_1000 t1_j3tnkld wrote

Dude that's why you ought to put everything into NLP find a way of producing better results for cheaper on less expensive hardware and you'll be the talk of the town. I think everyone would love to have an unrestricted local version of chatgpt on their phones. Do the research!

1

currentscurrents t1_j3eo4uc wrote

There's plenty of work to be done in researching language models that train more efficiently or run on smaller machines.

ChatGPT is great, but it needed 600GB of training data and megawatts of power. It must be possible to do better; the average human brain runs on 12W and has seen maybe a million words tops.

2

singularpanda OP t1_j3eohh7 wrote

Yes, it is quite costy. However, it seems not easy to modify it in our research as it is not open.

1

KBM_KBM t1_j3g7swj wrote

https://github.com/lucidrains/PaLM-rlhf-pytorch

Similar to chat get architecture you can play with this

2

singularpanda OP t1_j3gdv9p wrote

Thanks! Yes, there are many similar things. But the ChatGPT seems to have the most amazing performance.

1

KBM_KBM t1_j3gere2 wrote

True but practically training a gpt model is not computationally cheap. I think instead of making such generalized language models we need to focus more one subject specific language models.

1

f_max t1_j3frhxs wrote

Megawatt sounds right for training. But kilowatts for inference. Take a look at tim dettmer’s work (he’s at UW) on int8 to see some of this kind of efficiency work. There’s definitely significant work happening in the open.

1