Sound is an entity that travels through solid/liquid/gas and hits you, that is your entire body. Yes your ears hear that sound but there is much more to it than that , which gets discarded by headphones. Especially so for low-end / subsonic frequencies. Your bone marrow, your spinal fluid and all the way to your synapses between your neurons vibrate with low frequencies.

This is the body-feeling difference in the comparison. And then there is what we call the sound stage, that open / deep perception of the stereo composition. Let alone coloration through added harmonics of cabinets, room, objects etc. (Which actually effects the sound produced but makes it more alive, like an entity that shares the same space with us)

So no guilt to be felt here. If you want precise monitoring of audible frequency range, headphones indeed can offer an awesome alternative. But if you want to float in music and let it massage you, a stereo speaker setup is where it's at.