"the brick brown big wall" would have a different meaning.

It makes it sound like "brick brown" is a type of colour.

And then "big wall" is sounds like something different than just a "wall", it's its own noun.

The strict order you would normally put these words actually allows us to have more flexibility in the words you use. For example, having a colour called "brick brown" is made possible. If we allowed any random order to describe things, that couldn't happen.