OpenAI's and Anthropic's senior scientists join Hinton and Bengio in asking governments to prepare for an intelligence explosion

The Cambridge Programme on AI Science & Policy published a working paper titled “What if automating AI R&D triggers an intelligence explosion?” Coverage dates it to Monday 28 September, after The Wall Street Journal reported it first. The CASP page itself shows no date.

The author list is the story. Its 22 names include Jakub Pachocki, OpenAI's chief scientist; Jack Clark, a co-founder of Anthropic; Eric Horvitz, Microsoft's chief scientific officer; and Geoffrey Hinton, Yoshua Bengio and Andrew Barto. People who run research inside the frontier labs are putting their names to a warning about what that research could do.

The argument, as the abstract sets it out, is that AI systems already write much of the code inside the companies that build them and are on track to automate most AI research within a few years, possibly all of it, and that humans could lose control of what comes out. The paper's ask of governments is in one sentence: “Policymakers should urgently obtain more visibility into the automation of AI R&D, develop ways to steer and constrain an intelligence explosion, and prepare society to adapt to an intelligence explosion's impacts.”

News reports say the full paper also proposes reporting by labs on their progress, auditors placed inside companies and preparations for pausing some research. Those details could not be read in the PDF during this run, so they are not stated here as the paper's own. It is a working paper, not a peer-reviewed result.

• Confirmed The paper, hosted by the Cambridge Programme on AI Science & Policy, has 22 authors including Pachocki, Clark, Horvitz, Hinton, Bengio and Barto. CASP
• Confirmed “Policymakers should urgently obtain more visibility into the automation of AI R&D, develop ways to steer and constrain an intelligence explosion, and prepare society to adapt to an intelligence explosion's impacts.” CASP
• Reported Published Monday 28 September, after The Wall Street Journal reported it first. The Next Web