Adjectives are descriptive words that can combine in numerous and ambiguous ways. "brick brown" could be interpreted as a color description and not actually implying brick as a material at all. A lot of the ordering we have internalized is probably about disambiguation.