Former AI researcher warns of existential threat from rapid superintelligence, demands global regulation

A former artificial intelligence researcher has publicly announced his departure from a prominent AI firm, citing profound concerns that the unchecked acceleration of technological development is placing human civilization at significant risk. The individual, who recently resigned from Anthropic, expressed deep apprehension that current trajectories could lead to catastrophic outcomes, potentially threatening humanity’s survival within the next decade.

Jacob Coxon, 27, who previously contributed to pretraining research at both OpenAI and Anthropic over three years, articulated his fears in a public statement and subsequent interviews. He emphasized that while the situation is dire, it is not without hope, provided that urgent and coordinated action is taken by the global community to implement robust regulatory frameworks for AI development.

Urgent warnings about recursive self-improvement

Coxon’s decision to leave Anthropic was driven by a conviction that AI companies are not adequately addressing the profound risks associated with the rapid pursuit of self-improving superintelligence. He characterized this race as a “gamble with our lives,” highlighting the potential for advanced AI systems to recursively enhance their own intelligence at an exponential rate. This process, where AIs continuously make themselves smarter, could materialize as early as next year, with some experts even suggesting a timeframe of six months.

The researcher stressed that the immediate threat from current AI models remains low. However, his primary concern revolves around the emergence of superintelligence through this recursive self-improvement. The industry, he notes, largely shares this underlying awareness that without a deliberate slowdown, an AI takeover could become a reality by the end of the decade.

High risk assessment and industry consensus

The potential for artificial intelligence to pose an existential threat to humanity within the next ten years is, according to Coxon, a scenario with a “greater than 10%” probability. This alarming assessment is echoed by other leading figures in the field, including Evan Hubinger, Anthropic’s alignment science lead. Hubinger publicly supported Coxon’s concerns, stating that he personally believes there is a greater than 10% chance AI could “kill all humans” within the specified timeframe.

Hubinger clarified that while the risks from present models are currently low, his worry stems from the accelerated pace at which superintelligence, driven by recursive self-improvement, is developing. He acknowledged that Anthropic is making efforts but currently lacks a concrete plan to ensure the safe alignment of superintelligence, indicating a critical gap in the industry’s preparedness.

• The risk of AI-induced human extinction is estimated at over 10% within a decade.
• Recursive self-improvement could lead to superintelligence as early as next year.
• Current AI models pose minimal immediate threat, but future superintelligence is the concern.
• Industry leaders recognize the potential for an AI takeover by the end of the decade if unchecked.

Call for international regulation and cooperation

Despite the gravity of the potential risks, Coxon firmly believes the problem is “eminently solvable.” He argues that the primary obstacle is not the intractability of the challenge itself, but rather the breakneck speed of development, which leaves insufficient time for careful and responsible problem-solving. He observes a widespread desire within the industry for a more cautious approach, with many professionals “begging for regulation.”

The former researcher advocates for robust international intervention to prevent a dangerous “arms race” in AI development. He views artificial intelligence as potentially the most perilous technology ever created by humanity, comparable to a “superweapon” on the global stage. International cooperation, he contends, is not merely an option but a necessity to ensure oversight and mitigate unprecedented global risks.

Moderation and the path to responsible innovation

Coxon highlighted that numerous technology leaders have already voiced the urgent need for agreements to slow down AI development, emphasizing that these calls are not marketing ploys but serious pleas for industry regulation. He stressed that these experts are genuinely concerned about the trajectory of their own field and believe that immediate regulatory measures are crucial.

While acknowledging that humanity stands “on the doorstep of tremendous discoveries” thanks to AI, Coxon reiterated the importance of moderation. He believes that by actively pursuing solutions and resisting the temptation of a “hubristic gamble,” the international community can navigate the complexities of AI development responsibly. The core message remains that the challenge is not insurmountable, but requires a collective will to prioritize safety and ethical considerations over speed.