A problem with your hypothesis is that, in diffusion, as the matter becomes less dense, the rate of expansion slows. In the universe, the rate of expansion is increasing. Also, as someone else mentioned, from our perspective everything is expanding in all directions at an increasing rate. In a diffusion system, you can easily observe diffusion rate and concentration gradients.