> where's the line though between their work and the AI?

I am sure with case by case analysis we can find lines. But when AI is capable enough to publish full coherent papers, engage in high level debates in, say, logic, metalogic, epistemology, physics etc. on a level that experts have to take it seriously and so on, then we can weigh AI's opinion more. Right now AI is both superhuman and subhuman simultnaeously. It's more of a cacophany of personalities. It has modelled all the wacky conspiracy theorists, random internet r/badphilosophers, and also the best of philosophers and scientific minds. What ends up is a mixed bag. AI will respond based on your prompts and just luck and stochasticity. Sometimes it will write coherent philosophy simulating an educated undergraduate, another time it can write plausible nonsense (just as many humans already do and gain following). We will find techniques to make it more controlled and "aligned". That's already being done in part with human feedback, but feedback from just random human, will only make it aligned in so far that the AI becomes able to emulate a the expert style (eg. create fake bullshit but in a convincing articulate language) without substance. Another thing that's missing ATM is multimodal embodiment. Without it AI will be lacking the full grasp of human's conceptual landscape. At the same time due to training of incomprehensibly large data, we also lack the full grasp of AI's conceptual landscape (current AI (still quite dumb by my standards) is already beyond my intelligence and creativity in several areas (I am also quite dumb by my standards. My standards are high)). So in that sense, we are kind of incommensurate different breeds at the moment (but embodiment research will go on -- that's effectively the next step beyond language). Also certain things were already done better by "stupid" AI (or just programs; not even AI). For example, simple calculations. We use calculators for it. Instead of running it in our heads. So in a sense basic calculators are also "superhuman" in some respet. Which is why I don't think it's quite meaningful to make a "scalar" score to rank AIs and humanity or even other animals.

Personally, I don't think there is a clear solution to getting out of bias and fallibility. GIGO is a problem for humans as much for AI. At some point AI may start to become just like any human expert we seek feedback and opinions from. We will find more and more value and innovation in what they provide us. So we can start to take AI seriously and with respect. Although we may not like what it says, and shut of it (or perhaps, AI will just manipulate us to do more stupid things for lolz). We, as AI researchers, have very little clue what we are exactly doing. Although not everyone will admit that. But really, I don't where we should really put focus. Risks of collapse of civilization, military, surveleince, dopamine traps, climate change and what not. I think we have enough on our hands, more than we are capable to handle already. We have created complex systems that are at verge of spiralling out of control. We have to make calibrated descion on how to distribute our attention and focus on some balance between long term issues and urgent one.

We like to be egocentric; it's also not completely about us either. We have no clear theory of consciousness. It's all highly speculative. We don't know what ends up creating artificial phenomenology and artificial suffering. People talk about creating artificial consciousness, but few stop to question whether we should (not "should" as in whether we end up creating god-like overlords that end us all, but also "should" as in whether we end up creating artificially sentient beings that actually suffers, suffers for us. We have a hard time even thinking for our closer biological cousins -- other animals, let alone thinking for the sake of artificial agents.).

But sometimes, I am just a doomer. What can I do? I am just some random guy who struggle to barely maintain myself. Endless debates also just end up being intellectual masturbations-- barely anyone change their positions.

> Then they often give some qualifying criteria for how far AI would have to advance before they worship it.

I don't even find most descriptions of God worship-worthy; let alone AIs (however superhuman)