Some of the people helping build the world’s most powerful artificial intelligence systems are now warning that the technology they are creating could eventually threaten humanity itself.

Yet nobody appears particularly eager to stop building it.

That uncomfortable contradiction has returned to the centre of the debate after researcher Jacob Coxon resigned from Anthropic, one of the companies developing some of the world’s most advanced AI systems, and issued an extraordinary warning about where the technology could be heading.

Coxon, who previously worked at OpenAI, accused leading the developers of racing towards increasingly powerful systems while taking risks whose consequences could extend far beyond Silicon Valley.

The warning is difficult to ignore for one reason: Coxon was not watching the race from the sidelines. He was inside it.

Walking Away From the AI Race

According to Axios, Coxon left Anthropic after only four months and just two months before his equity in the company was due to vest.

That means his resignation came with a financial cost.

His concern centres partly on the possibility of increasingly autonomous and self-improving systems eventually becoming difficult for humans to reliably control.

Speaking to WIRED, Coxon described the next year or two as a critical period for development and said researchers inside the industry have used language such as “endgame” when discussing what may be approaching.

The fear is not simply that machines become smarter.

It is what happens if humanity creates systems vastly more capable than itself before figuring out how to guarantee that those systems continue acting in humanity’s interests.

One Researcher Would Be Easy to Dismiss. He is Not Alone

Anthropic AI alignment researcher Evan Hubinger has publicly estimated that there is a greater than 10% chance AI could kill everyone within the next decade.

That is an extraordinary estimate, but it is not proof that humanity has suddenly acquired an expiry date. It is one researcher’s assessment of a deeply uncertain future.

Concern about catastrophic AI risks, however, extends well beyond one company.

A 2026 study led by researchers from MIT FutureTech and the University of Queensland brought together 272 specialists from 37 countries to assess 24 categories of risk.

Under a “business as usual” scenario, the experts judged 18 of those 24 categories as having more than a 10% probability of producing catastrophic harm within five years.

Lead author Peter Slattery, a researcher at MIT FutureTech, said the findings showed that experts were concerned about a wide range of potential harms rather than one dramatic science-fiction scenario.

The study identified dangerous capabilities, AI-enabled weapons and cyberattacks, competitive pressure between developers, concentration of power and sophisticated misinformation among the major areas of concern.

University of Queensland researcher Dr Shahar Avin, one of the study’s authors, also highlighted the importance of understanding how risks interact rather than treating each threat in isolation.

That matters because AI does not need to become a rogue superintelligence to cause serious harm. Weapons, cyberattacks, misinformation and reckless competition could create major consequences long before anything resembling science fiction arrives.

So Why Are We Still Builing It?

This may be the most uncomfortable part of the story.

Artificial intelligence is no longer simply a scientific experiment. It is a commercial and strategic race.

Companies are competing to build increasingly capable systems, while governments are increasingly treating the leadership of artificial intelligence as economically and strategically important.

Coxon told Axios that he had not witnessed Anthropic compromising safety. That distinction matters.

His warning is about what could happen as competition intensifies and developers face pressure to move faster.

The MIT-led research points to a similar contradiction.

Slattery said the findings suggest that responsibility for reducing risks falls heavily on governments, regulators and developers themselves. Yet those same actors operate in an environment where slowing down can mean allowing a competitor to move ahead.

Put simply, some of the people with their hands closest to the brakes are also competing to win the race.

Is It Actually Going To Kill Us?

Nobody knows.

And pretending otherwise would turn a serious debate into science-fiction clickbait.

The extreme scenarios being discussed are forecasts of possible future risks, not evidence that today’s systems are preparing to overthrow humanity.

There is another side to the equation too.

Advanced systems could accelerate scientific research, improve medicine and help humanity solve problems that have resisted decades of conventional research. Coxon himself has acknowledged the potentially enormous benefits of sufficiently powerful AI.

That is precisely what makes this debate so difficult.

The choice is not simply between dangerous AI and no AI. Humanity is pursuing a technology that could deliver extraordinary benefits while researchers are still debating how severe its worst possible consequences could be.

And that leaves one question hanging over the entire Artificial Intelligence race.

The question is no longer simply how intelligent machines can become.

It is whether humans will still be firmly in control when they do.

Sources and References

Alos read: Lawyers Must Lead AI, Not Simply Adopt It to Move Forward

https://futuretech.mit.edu/publication/prioritization-of-risks-from-artificial-intelligence-a-delphi-study
https://www.axios.com/2026/09/09/anthropic-researcher-ai-warning-interview?utm_source=chatgpt.com
https://www.ft.com/content
https://www.wired.com/story/anthropic-researcher-quits-jacob-coxon-ai-fears-humanity/
https://www.rt.com/news/645350-ai-kill-humanity-decade/