Depends on how concerned you might be with him getting pissed off (because he’s family). If your not concerned just leave. If he reached out ask him for a copy of the lease and remind him that he told you to get the fuck out.
Take a minute to document the time and place (and any other details) when he made that statement in case he denies saying it. But…I’m not a lawyer!!!