The Problem With Testing AI Architectures at Small Scale | Jerry Tworek, Core Automation
AI architectural research fails because it tests at insufficient compute scale
“to get to any interesting results you need certain level of compute to even see the capabilities in the model”
Jerry Tworek of Core Automation argues that a systemic flaw in AI research is validating architectures at small scale before scaling up — but many architectures, especially those using reinforcement learning, require a minimum compute threshold before any meaningful capability emerges. This challenges the standard 'prove it small, then scale' research paradigm. The implication is that promising architectures may be getting discarded prematurely because researchers lack the compute budget to reach the threshold where results become visible.