Go to your local audio shop and ask to try out some amps. This way you can decide for yourself. You might hear a big difference or you may find it hard to notice. No two sets of ears are identical.

I’m happy with the headphone amp built into my separates integrated amp, it sounds way clearer than listening directly from my CD player or IPad and the bass feels fuller and punchier. Personally I don’t hear enough of an improvement from anything I’ve tried compared with my current setup to justify dropping cash on a dedicated amp but to each their own. Maybe you’ll find one that makes sense for you.