I am exactly in the same position as you. Go towards the games you want. Cloud gaming is becoming big you can use your xbox 1 s for lots of games if you have gamepass ultimate. The way I see it if I get a series x than my xbox 1 s doesn't get used but if I get a ps5 I can still use the xbox 1 s for old games and for cloud gaming.

Ps5 has amazing exclusives and you can use that for the next gen games.

Many games are also cross platform.

Just some advice in the end compare all your factors before buying. In the end both consoles are great.