In most use cases it is very helpful for there to be exactly 24 * 60 * 60 "seconds" in a day, but it's not all that important that a second be an SI second.

If you insist that a second is a physical second from an atomic time source you will eventually accumulate some noticable difference between the official time and the astronomical time.

The general public can probably accept a much greater deviation in that than the one second that has been the practice for the last few decades, but if the astronomical community thinks it is worth keeping world time in sync with astronomical time then they should bear the brunt of the adjustment.

A single second amortized over a full day is not detectable by commodity time sources. So if you want to make the adjustment do it that way. Lie about how long the seconds are, just make your lie so small that it falls below the natural variation in quartz time sources.