Anthropic co-founder warns AI is a car with a gas pedal but no brakes

The artificial intelligence industry is moving rapidly towards systems capable of enhancing themselves without any human involvement — and has put in place no mechanism to decelerate or halt that process if something goes wrong.

That is the stark warning now being issued publicly by Anthropic, and it is one coming directly from within the company's own leadership.

The blog post that sparked the debate

In a post published by The Anthropic Institute, co-founder Jack Clark and institute leader Marina Favaro cautioned that "full recursive self-improvement" — AI systems capable of autonomously developing their own successors — is considerably closer than the industry had previously estimated.

"If systems are capable of fully building their own successors, the ways we secure them, monitor them, and shape their behaviour all grow much more important," they wrote.

The authors stopped short of predicting outright catastrophe, but declined to rule it out either. They called on companies across the sector to consider slowing or pausing frontier AI development to allow safety research adequate time to catch up.

Clark takes the warning to a mainstream audience

During a Thursday evening appearance on CNN, Clark used accessible language to convey the scale of his concerns to a general audience. "When I look down at the car we're driving, all I have is a gas pedal. I don't have a brake pedal," he told Anderson Cooper.

When Cooper pressed him on whether he was troubled by the kind of "AI kills humanity" scenarios depicted in science fiction, Clark did not sidestep the question. "Yeah, we read the science fiction and watch science fiction here as well, so it's not lost on us," he said.

Addressing the competitive contradiction

The most obvious challenge to Anthropic's call for collective restraint is the fierce competitive dynamic that exists between AI companies — a dynamic that Anthropic itself is very much part of, having filed for an IPO that could value the company at tens of billions of dollars.

Clark, however, pushed back on the notion that competition makes cooperation impossible, drawing on historical precedent. He argued that "during the height of the Cold War, during highly volatile interactions between countries that are competitors, there were still ways in which they could stabilise certain elements of the arms race."

In his view, where there is sufficient political and corporate will, similar guardrails could be established for AI development today.