It may be a little of both as the saying goes. Studies of vaccine antibody persistence in a few diseases suggest that if you’re not regularly exposed to the organism in question, the antibodies fade faster, sometimes much faster than in communities where the organism is endemic. Even though the people are not getting reinfected at least not obviously. So (and this is just my idle speculation) since influenza famously does as you said , drift , maybe we don’t get the benefit of reawakening the vaccine with repeated exposure. But back to actual science: it remains a mystery why influenza vaccines fade so incredibly fast , sometimes within a month it’s starting to fade.