If you’ve got a bathroom fan, get a look at the damper in the line. If it is not sealing when not in use, it may be the cause of your problem.

Just remodeled. Got a brand spanking new bath fan & the damper wasn’t tight. Bathroom smelled like a musty attic & was 20 degrees warmer in the summer until I installed an aftermarket damper. I don’t know if it is a design flaw in the fan or my contractor just got a bad one.