Is there a blog post or a paper comparing open source / open weights models? I know flant t5 is really good at instruction following, but I am specifically refering to performance after finetuning. Preferably it compares models from somewhere around 1b to 11b parameters.

Comments

You must log in or register to comment.

borisfin t1_j8upatv wrote on February 17, 2023 at 1:57 AM

There is some interesting comparisons found in the flan t5 paper. Checkout the paper "Scaling Instruction-Finetuned Language Models". Hope this helps.

AbsoluteCondui t1_j8viwrj wrote on February 17, 2023 at 6:23 AM

thanks

adt t1_j8v1vlp wrote on February 17, 2023 at 3:37 AM

For models, see my up-to-date list of models:

https://docs.google.com/spreadsheets/d/1O5KVQW1Hx5ZAkcg8AIRjbQLQzx2wVaLl0SqUu-ir9Fs/edit#gid=1158069878

For performance, Papers with code keep good benchmarks:

https://paperswithcode.com/area/natural-language-processing

https://paperswithcode.com/task/question-answering

farmingvillein t1_j90m0ab wrote on February 18, 2023 at 8:52 AM

> For models, see my up-to-date list of models:

Which tab is germane to OP's request?

> but I am specifically refering to performance after finetuning.

So far as I can tell, there is nothing here that is responsive to OP's query. But there is a lot here--perhaps I read too quickly.

Franck_Dernoncourt t1_j8v206a wrote on February 17, 2023 at 3:38 AM

For summarization: Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, Tatsunori B. Hashimoto. Benchmarking Large Language Models for News Summarization. arXiv:2301.13848.

rshah4 t1_j9cndve wrote on February 20, 2023 at 11:10 PM

Check out this great post that includes fine tuning Flan-T5, Language Models vs. The SAT Reading Test:

https://jeffq.com/blog/language-models-vs-the-sat-reading-test/