I agree with most of what you say, but please note that one key difference between diffusion models and language models is size, so compute costs. Diffusion models are really tiny compared to language models.