Whoever wrote the scene where Ultron wakes up conscious, bodiless, frightened and angry, in utter darkness, gets it. Elon Musk, who has warned of “summoning a demon”, gets it.

That said, Nick Bostrum posits that a simulated human brain that simulated everything (right down to individual neurons) would effectively be a brain and that consciousness, being an emergent property of a brain’s system, would occur all on its own. If general AI becomes conscious/sentient, it should be possible to simulate bodily input/output experiences like a human does, helping it grow like a child - so it hopefully empathizes with the human condition as it matures.

Aiming for a bodiless black box consciousnesses with access only to programmers and Google is a terrifying, and highly immoral, gambit.