Building on YEARS of ideas. They were cool, but without transformers they wouldn't exist. Without infrastructure code, they wouldn't exist. Without years of hardware improvements, they wouldn't exist. Without the ideas of normalization and skip connections, they wouldn't exist. Etc. (and this isn't even including all the alleys that were chased down, to find out they didn't work. Which isn't as clear, but definitely contributes to research).

GATO didn't even have that much to show for it, the long hoped-for skill transfer was not really there. DALLE 2 builds on CLIP and diffusion, ChatGPT builds on GPT3 and years of RL research.

You're saying something along the lines of "x is better than what came before, so the step to x is bigger than the sum of all the steps before that" and that is the worst take i've ever heard. It's definitely not how research works.

And goddamn it I'm getting deja vu cuz this bad take has been said before on this subreddit.

This rebuttal better? I'd be happy to go and list essential moments in AI in the past decade if it isn't.