That’s one of those envisioning things at play. Most people can’t detect slight variations in color even when they can see them, so while you do indeed see a gradient your memory tends to trim it down to less variance. It’s estimated that we can discern around a million distinct colors, but remembering that is difficult for all but a very small number of people. Most of just remember the idea of a rainbow and then reconstruct the imagery later.