I would read “The brown big brick wall” as being a wall made of big bricks that were brown.

I think what happens is that if the adjectives go out of order your brain re-parses the list and assumes that they’re out of order for a reason and that there’s some kind of nesting is going on. some things are modifying other adjectives (sometimes nouning them) rather than the base noun.

Like “the old big cat” makes me assume that we are assessing “big cats” and talking about the old one.