Jean-Porte t1_j6hif9e wrote on January 30, 2023 at 10:50 AM

T5 is fine-tuned on supervised classification. Trained to output labels. That's why it outperforms GPT3.

Generative models are not as good as discriminative models for discriminative tasks. A carefully tuned Deberta is probably better than chatGPT. But ChatGPT has a user-friendly text interface. And the glue-type evaluation is not charitable to chatGPT capabilities. The model might internally store the answer but it could be misaligned to the benchmark.

I always wonder why we don't try to scale-up discriminative models. Deberta-xxlarge is "only" 1.3B parameters, and it outperforms T5 13B.

mettle OP t1_j6ilm6q wrote on January 30, 2023 at 4:27 PM

Is there some alternative benchmark that measures factual accuracy of output?

Or is that impossible to use and create because any model would overfit that data?

Jean-Porte t1_j6imfho wrote on January 30, 2023 at 4:32 PM

LAMA, truthfulQA, MMLU, and many others

mettle OP t1_j6imy6h wrote on January 30, 2023 at 4:35 PM

perfect, thank you!