I'll say up top, there is no manner to answer anything you have put forth in regards to consciousness until there is a definition for consciousness. So, apologies if you find these answers wanting or unsatisfying, but until there is a testable and consistent definition of consciousness, there is no way to improve them.

> isn't it possible the AIs we end up creating may have a much different, "unnatural" type of consciousness?

Sure, but we aren't discussing the future or AGI, we are discussing LLMs. My comment has nothing to do with AGI but yes, that is a possibility in the future.

> How do we know there isn't a "burst" of consciousness whenever ChatGPT (or its more advanced future offspring) answers a question?

Because that isn't how feed-forward, deep neural networks function regardless of the base operation (transformer, convolution, recurrent cell, etc.). We are optimizing parameters following statistical methods that produce outputs - outputs that are designed to closely match the ground truth. ChatGPT is, broadly, trained to align well with a human; the fact that it sounds like a human shouldn't be surprising nor convince anyone of consciousness.

Addressing a "burst of consciousness", why has this conversation never extended to other large neural networks in other domains? There are plenty of advanced types of deep neural networks for many problems - take ViT's for image segmentation. ViT models can be over a billion parameters, and yet, not a single person has once ever proposed ViT's are conscious. So, why is this? Likely, because it is harder to anthropomorphize the end problem of a ViT (a segmented image) than it is to anthropomorphize the output of a chatbot (a string of characters). If someone is convinced that ChatGPT is conscious, that is their prerogative but they should also consider all neural network of a certain capacity as conscious to be self-consistent with that thought.

> Even if we make AIs that closely imitate the human brain in silicon and can imagine, perceive, plan, dream, etc, theoretically we could just pause their state similarly to how ChatGPT pauses when not responding to a query. It's analogous to putting someone under anesthesia.

Even under anesthesia, all animals produce meaningful neural signals. ChatGPT is not analogous to putting a human under anesthesia.

I think we will eventually discover that consciousness is closely tied to the brain's ability to interact on a quantum level with the real world and that maintaining the unique superposition of quantum states is what is unique. Any discrete silicon-based computer will only be an approximation of that at best.