Best explanation I found as to how / why it truly matters is that if you're at 90% Volume on a song because your phone can't power your headphones all the way.

Say you're listening to a song with a very high dynamic range like something classical or some Jazz.

The soft portions of the song that are very quiet all the way to the loud booming portions could have a difference of 25db.

When you're at 90%volume, your headphones can't put out more than the Maximum, and so you lose out on all that extra detail and range the song has.

And when your headphones aren't powered enough to bring that low small whisper portion of a song into an audible level, and then be able to bring the louder booming portions into it as well, then you're losing out on either the sound from the quiet tones or the more loud higher details. And when it's not properly powered, that's the issue you struggle with and face.

There may be some flaws with this, I'm not an engineer. This is my understanding after reading a fair amount on it and asking some similar questions.