In my entirely personal opinion, as someone who enjoyed the book but was weary on the idea of watching many seasons of a TV show adaptation of this particular story: short answer, yes.

Longer answer: I think seasons 1-3 are definitely much slower and sadder than they need to be, a long trauma parade, which does have good character arcs, but it's kind of a slog. But Seasons 4-5 really pick up and take a new direction that to me is much more exciting and engaging and very worth it. I rarely tell people to skip episodes when watching shows but this is one where I'd say you could skip some of seasons 2-3 if you're feeling burnt on the trauma of Gilead but want to see the story move forward.