> Child is both a British and American citizen, I am a British Citizen with a green card

You have every reason and right to stay in the UK then. Coming back to America, as he wants, is a bad idea, and he knows it (likely why he wants you to do it).