Submitted by ortegaalfredo t3_11kr20f in MachineLearning
wywywywy t1_jb97nl6 wrote
Nice one.
With dual 3090s, I think 30b should be possible in 8bit?
polawiaczperel t1_jb98qce wrote
Even with one rtx 3090 https://github.com/oobabooga/text-generation-webui/issues/147#issuecomment-1456626387
ortegaalfredo OP t1_jbaswga wrote
Interesting, will research more into that code, its exactly what I need to run 33B.
Currently using a single card it's still too slow to use it as a chatbot.
Viewing a single comment thread. View all comments