What freaks me out the most are the social ramifications of AIs that pass as humans to the majority of people. We're still figuring out how to healthily interact with social media and soon we're going to be interacting with entirely artificial content that we're gonna anthropomorphize onto other humans. In the US we're dealing with a crisis of trust and authenticity, I can't imagine generative text models are going to help with that.

I worry about a lot of bad AI/ML made by interns making decisions that have huge impact like in the justice system, real estate ect.

>I remember as recently as 2015 at ICLR/ICML/NIPS you’d get side-eye for even bringing up AGI.

You still do, imo rightfully so

None of those concerns have to do with the intrinsic nature of machine learning, though. Right now it's another tool that can automate tasks previously thought impossible to automate, and sometimes it does that task much better than humans could. It's another wave like the Industrial Revolution and the assembly line.

Some people will inevitably use this technology to destroy things on a greater scale than ever before, like using the assembly line to mass produce missiles and tanks. But trying to put a leash on the technology won't accomplish anything because technology isn't inherently good or evil.

Now, if the state of ML were such that sentient AI was actually on the horizon, not only would this way of thinking be wrong, we'd need to rethink the concepts of humanity and morality altogether. But it's not. Not until these models manage to improve at tasks it was not trained to do. Not until these models become capable of accurate self-evaluation of its own performance without human input.

What I’m hoping is that EY’s long term vision for AI existential risk is thwarted by the inevitable near term issues that will come to light and inevitably be raised to major governments and powerful actors who will then enter a “collective action” type of response similar to what happened with nukes, etc. the difference is that any old 15 year old can’t just buy a bunch of AWS credits and start training a misaligned nuke.

What you mention about a ChatGPT like system getting plugged into the internet is exactly what Adept AI is working on. It makes me want to bang my head against the wall. We can say goodbye soon to a usable internet because power seeking people with startup founder envy are going to just keep ramping these things up.

In general though, I think my “timelines” are longer than EY / EA by a bit for a doomsday scenario. LLMs are just not going to be the paradigm that brings “AGI,” but they’ll still do a lot of damage in the meantime. Yann had a good paper about what other factors we might need to get to a dangerous, agentic AI.

One would have to consider the ultimate consequences (including paradoxical ones) of those things too.. Like would it really be catastrophic if social media became unusable for the average user? The 1990s are usually considered the last halycon era... Maybe thats a feature not a bug!

As far as drone swarms, those are definitely terrifying, but then there will be drone swarm countermeasures. Also, is it really much more terrifying than Russia throwing wave after wave of humans at machine gun nests?

I view a lot of the ethics concerns as a bunch of people projecting their fears into a complicated world, and then drastically overextrapolating. This happened with the industrial age, electricity, the nuclear age and so on and so forth.

Much like that there are issues like what recommender systems are recommending, how credit models are scoring, why your resumes are being discarded without being seen by a human being and lots of other minor mundane daily things that we take for granted and are actually dystopic for the people at the short end of the stick. These are systems that need to be fine tuned and we treat their judgements as holy lines in the Commandment stones. The AI dystopia is already real.

> EY tends to go straight to superintelligent AI robots making you their slave.

First, I don't think he ever said that they will make us slaves, except possibly as a joke at the expense of people who think the AI will care about us or need us enough to make us slaves.

Second, I am frustrated by the fact that you seem to think that only the short-term threats matter. What's a more short-term threat: nuclear contamination because of the destruction of the ZPP in Ukraine, or all-out nuclear war? Contamination is more likely, but that doesn't mean that we wouldn't be stupid to ignore the potentially farther away yet incredibly catastrophic outcome of nuclear war. Why can you not be worried about short-term AI issues but also acknowledge the possibility of the slightly longer term risk of superintelligent AI?

This is depressingly typical as an attitude and not at all surprising as the top comment here, unfortunately.

The shorter-term problems you mention are important, and I think it would be great for technical and policy-minded people to try to alleviate such threats. But it's also important for people to work on the potential longer term problems associated with AGI.

OpenAI, and organizations like them, are racing towards AGI - it's literally in their mission statement. The current slope of ML progress is incredibly steep. Seemingly every week it looks like some major ML lab comes up with an incredible new capability with only minor tweaks to the underlying transformer paradigm. The longer this continues to happen, the more impressive these capabilities look, and the longer we see scaling curves continue with no clear ceiling, the more likely it looks that AGI will come soon, say, over the next few decades. And if we do succeed at making AI as capable or more capable than us, then all bets are off.

None of this is a certainty. One of Yudkowsky's biggest flaws imo is the certainty with which he makes claims backed with little rigorous argument. But given recent discoveries, the probability of a dangerous long term outcome is high enough that I'm glad we have people working on a solution to this problem, and I hope more people will join in.

Re: Things that will happen rather soon, I think it's important that we also look at AI's impact on nonhuman animals. I argued it in this paper. AI-controlled drones are already killing animals, some AIs are helping factory farming, language models are showing speciesist patterns that might reinforce people's bad attitudes toward some animals (ask ChatGPT about recipes of dog meat vs chicken meat, or just google "chicken" to see whether you see mainly the animal or their flesh)

Actually, even for things that could happen in the far future, I think it's extremely likely that what AI will do will immensely impact nonhuman animals too.

>What about when ChatGPT

I mean, we're facing even more important dilemas right now, with ChatGPT's saftey rails. What is it allowed to talk about, or not? What truths are verbotten?

If the plurality of Internet content is written by these sorts of algorithms, that have hardcoded "safety" layers, then dream of truly open access to information that was the Internet will be that much closer to death.