Viewing a single comment thread. View all comments

Qumeric t1_itkg1un wrote

Reply to comment by rehrev in [R] Scaling Instruction-Finetuned Language Models - Flan-PaLM- Google 2022 - 75.2% on five-shot MMLU / Forecasters expected this SOTA would need until 2024! - Public checkpoints! by Singularian2501

There are things called prediction markets and prediction aggregators. See https://www.metaculus.com/questions/11676/mmlu-sota-in-2023-2025/ for up-to-date predictions for MMLU SotA.

Also, see https://www.lesswrong.com/posts/arveXgFbJwascKtQC/forecasting-ml-benchmarks-in-2023 for a detailed prediction from a single forecaster.

2