Jacob Coxon Leaves Anthropic: What He and Former Employees Have Warned About the AI Race

Jacob Coxon resigned from Anthropic on Sept. 8 and said the company and OpenAI are racing toward self-improving superintelligence.

“They are racing straight to self-improving superintelligence and gambling with our lives,” Coxon wrote in a series of posts on X.

Coxon, 27, said he had spent the previous three years doing pretraining research at OpenAI and Anthropic before leaving the AI industry altogether. He argued that the stakes are understood differently at the two companies: At OpenAI, he said, many employees have not fully internalized the risks. At Anthropic, he said, the risks are understood, but the company is “locked in a race to get there first.”

“The people building AI earnestly believe that it could kill us all by the end of the decade,” Coxon wrote.

Evan Hubinger, Anthropic’s Alignment Science Lead, responded that Coxon was right that researchers genuinely believe AI could kill all humans. Hubinger said he personally puts the probability above 10% within the next decade.

“I believe Anthropic is trying its best,” Hubinger wrote, “but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.” He said current models pose low risk and that his concern is superintelligence emerging through recursive self-improvement.

Anthropic’s Warnings Did Not Start With Coxon

Coxon is the latest Anthropic researcher to publicly raise serious concerns, but not the first. Mrinank Sharma, who led Anthropic’s Safeguards Research Team, resigned in February. 

“The world is in peril,” Sharma wrote, adding that the danger was not limited to AI or bioweapons. He also described pressure on organizations to set aside their values. His work included safeguards against AI-assisted biological threats.

Coxon worked on pretraining, the process used to build the underlying models, rather than on a dedicated safety or safeguards team.

OpenAI’s Safety Split Came Earlier

The same conflict became public at OpenAI in 2024. Jan Leike, who co-led OpenAI’s Superalignment team, resigned in May after saying he had disagreed with leadership over the company’s priorities. “Safety culture and processes have taken a backseat to shiny products,” Leike wrote. He later joined Anthropic.

OpenAI co-founder and chief scientist Ilya Sutskever also left that month. He later founded Safe Superintelligence Inc., whose stated focus is building safe superintelligence.

Daniel Kokotajlo, an OpenAI governance researcher, resigned in 2024 after saying he had lost confidence that the company would handle AGI responsibly. He refused to sign a non-disparagement agreement, forgoing roughly $2 million in vested equity rather than sign, and later became an advocate for protections for employees who warn about AI risks.

Those departures did not establish that OpenAI was ignoring safety. They showed that senior employees had reached serious disagreements over how safety should be weighed against rapid AI development.

The Race Is Now an Industry-Wide Concern

The concern is not limited to people who have left frontier labs. In July, more than 1,100 employees at major AI companies signed a statement asking the U.S. government to help develop technical and governance tools that could deliberately pace the frontier of automated AI development. The statement warned that automated AI research could accelerate beyond humanity’s ability to understand or control the resulting systems.

The signatories included employees from OpenAI, Anthropic, Google, and Meta. The request was not for an immediate halt to AI development. It was for governments to create mechanisms that could buy time if automated AI research began moving faster than safety and oversight could keep up.

Coxon said he ultimately could not accept entering that “endgame” from inside a private company and urged researchers to question whether they should launch increasingly powerful systems without a rigorous understanding of how those systems work.

His departure does not establish that Anthropic or OpenAI will create uncontrollable AI, nor does Hubinger’s estimate amount to an official company forecast. But the record shows a recurring concern among people who have built, studied, or tried to constrain frontier systems: the technology may be advancing toward capabilities that its developers do not yet know how to reliably control.