I hope they are very wrong

What are BCIS and FDVR?

There’s a video I watched that stated one of the pluses of moving away from fossil fuels and coals is that if there’s ever some catastrophic event there are a good fallback to basically restart industrial society

It could also do something like Make solar 1000 more efficient or create synergetic algae to capture carbon, a lot of possibilities but I wouldn’t pin our plans on it, it’s not a given

What do you mean? Also if it closely resembled what we consider to be conscious does it even matter I guess?

Terrifying to consider

Anytime in the history of earth when a smarter being emerged it either killed off or completely dominated the rest. Why would this trend stop?

Yeah that’s likely not very hard to do

Some bad game theory. So we all have to try because another might, but if they do so it could literally end humanity?

We don’t even think climate change is real here, we don’t act rationally. The issue will be politicized via populism, expect severe backlash

Why would it not make sense to refrain from creating AGI? I would love to see an actual risk/benefit analysis done.

I disagree completely. You’re assuming rational actors. Have you seen American politics? It’s a disaster. Also, maybe some people don’t want their entire life revolved around AI, and with the social upheaval automation from narrow AI is about to rapidly cause from job replacement, who do you think they will come for? I say it’s very likely

You may be right, I guess we'll see right? I think it can write something very very similar probably just based on ML inputs even if it's not entirely original, but we'll see very soon

I think you’re underestimating how fast AI has been moving and how fast it will improve. It’s been breakneck pave the past two years for content and scaling will obviously keep going

That’s simply not true, it will.

You should read the article. Some very smart people but the chance at 10%, also look at AI scaling, it follows the concept of exponential growth, it will explode in capabilities so fast once it gets going.

I have commented about how nuclear non-proliferation needs to keep going and even accelerate on this sub and /News and it’s met with many downvotes lol

Summary

We expect that there will be substantial progress in AI in the next few decades, potentially even to the point where machines come to outperform humans in many, if not all, tasks. This could have enormous benefits, helping to solve currently intractable global problems, but could also pose severe risks. These risks could arise accidentally (for example, if we don’t find technical solutions to concerns about the safety of AI systems), or deliberately (for example, if AI systems worsen geopolitical conflict). We think more work needs to be done to reduce these risks.

Some of these risks from advanced AI could be existential — meaning they could cause human extinction, or an equally permanent and severe disempowerment of humanity.2 There have not yet been any satisfying answers to concerns — discussed below — about how this rapidly approaching, transformative technology can be safely developed and integrated into our society. Finding answers to these concerns is very neglected, and may well be tractable. We estimate that there are around 300 people worldwide working directly on this.3 As a result, the possibility of AI-related catastrophe may be the world’s most pressing problem — and the best thing to work on for those who are well-placed to contribute....

I’d like it to stop right now personally. The risks far outweigh the benefits.

Yes it is but don’t mention that on this sub they will downvote you into oblivion

Exactly. It’s startling how little awareness or concern there is about this in the general public let alone a FUTURE subreddit

Now imagine these get proliferate and AI starts controlling these. Not so easy to "just pull the plug" now is it?

And what do you do when AI controls everything connected to the internet? Not so easy to shut off is it?

Why do we want to do this, still feels like the risk is not worth it....