I wrote an AI textbook — how long until AI can do it better?
LLMs are stagnant at long-form non-fiction writing, a prerequisite for autonomous science
“Models being stagnant in long-form, non-fiction writing should be alarming to those reliant on models autonomously solving grand, open science problems in the near future.”— Sam Altman
Nathan Lambert, who just finished writing an RLHF textbook with LLM assistance, argues that current models surprisingly fail to organize and compellingly present well-established scientific knowledge in long-form non-fiction. He frames this as a critical capability gap: if models can't compress and present known science coherently, claims about near-term autonomous scientific breakthroughs are premature. The piece is a counterweight to hype around results like Claude's Riemann Hypothesis progress, arguing AI science wins will remain narrow and cherry-picked until this generalization problem is solved.