A Princeton-led team posted a paper today describing the first AI evaluation designed around a deceptively simple idea: give an AI agent a genuinely open research question \u2014 one whose answer is unknown \u2014 and then have the scientists who asked it grade what comes back. If AI benchmark scores have been telling us something systematically misleading about whether AI can actually do science, this paper begins the process of finding out what is actually true.<\/p>\n
The paper, “Can AI agents conduct open-ended AI research? Early evidence from two case studies,” posted this morning to arXiv (CRUX open-world research paper<\/a>), climbed to the top of arXiv’s cs.AI daily listing within hours of publication. Its lead authors are Sayash Kapoor, a Princeton computer science PhD, and Arvind Narayanan, a Princeton CS professor and director of the Center for Information Technology Policy \u2014 the same team behind AI Snake Oil, one of Nature’s best books<\/a> of 2024. Their co-authors span Georgetown’s Center for Security and Emerging Technology, Johns Hopkins, the UK AI Security Institute, Stanford, and a dozen other institutions.<\/p>\n The paper describes two case studies \u2014 both drawn from real, unpublished AI research projects. An agent is handed the central question of an active investigation, given access to the tools and data that would normally be available, and then evaluated by the scientists who originally posed the question. That sounds obvious. It is also, according to this team, essentially unprecedented in rigorous AI evaluation.<\/p>\n Why Every Other AI Research Evaluation Gets This Wrong<\/p>\n The standard approach to evaluating whether AI can conduct research has been to build a benchmark: curate a set of tasks with known answers, grade them automatically, and publish a leaderboard. That approach has generated a library of evaluations \u2014 PaperBench (can AI replicate published AI results?), EXP-Bench (can AI conduct AI research experiments?), ResearchGym (can AI rediscover methods withheld from published papers?), MLR-Bench \u2014 each more sophisticated than the last and each sharing a structural flaw: every task has a known answer.<\/p>\n This matters because of a principle the paper does not name but that runs through its argument like a thread. Economist Charles Goodhart noted in 1975 that any statistical regularity tends to collapse once it becomes a control target; anthropologist Marilyn Strathern later distilled it to the version everyone quotes: when a measure becomes a target, it ceases to be a good measure. AI benchmarks are classic Goodhart targets<\/a>. AI benchmarks are Goodhart targets. Modern reinforcement learning training directly optimizes for them. Any task specified precisely enough to benchmark is also specified precisely enough to optimize for, which means a model can learn to score well on a test without acquiring the underlying capability the test was meant to proxy.<\/p>\n The construct validity in research<\/a> problem \u2014 whether the measurement actually measures what it claims to measure \u2014 is structural, not incidental. It cannot be patched by making benchmarks harder, adding more tasks, or using better automatic graders. It only disappears when the task is too open-ended to specify as an optimization target. Which is exactly what open-ended research questions are.<\/p>\n EXP-Bench research experiment benchmark<\/a>, from a University of Michigan team, illustrates the ceiling of the current approach. It curated 461 research tasks from 51 top-tier AI publications, gave agents a research question and incomplete starter code, and had them formulate hypotheses, design experiments, execute them, and analyze results. On individual experimental components like design or implementation correctness, scores occasionally reached 20\u201335%. On complete, executable experiments, the success rate was 0.5%.<\/p>\n That is the state of the art with known answers. The Princeton paper asks what happens with unknown ones \u2014 and proposes the only evaluation framework designed to find out honestly.<\/p>\n How Author-Graded Open Research Works<\/p>\n The methodology is called “open-world evaluation” \u2014 a term the Princeton group introduced formally in a companion open-world evaluations paper<\/a> posted in May. Its defining features set it apart from every benchmark-based approach:<\/p>\n The task is real and the question is open. The agent receives the central research question of an actual, unpublished investigation \u2014 not a question whose answer exists somewhere in a training set or a published paper. Genuine uncertainty is not simulated; it is structural.<\/p>\n The graders are the original scientists. Because the question is genuinely unresolved, there is no automated grading rubric. The only people positioned to assess the agent’s output are the researchers who have spent real time thinking about the question. Their evaluation is qualitative, expert, and impossible to game by optimizing for an automated score.<\/p>\n The sample is deliberately small. The companion paper explicitly addresses this: open-world evaluations typically involve one to a handful of tasks, examined in depth rather than at benchmark scale. This is not a limitation to be overcome \u2014 it is the feature that enables the qualitative log analysis and expert review that automated grading forecloses.<\/p>\n The paper describes this design as a “third way” between narrow benchmarking and peer review. It trades the scalability and reproducibility of benchmarks for ecological validity: the tasks are real, the stakes are real, and the judges are real experts, not anonymous reviewers under time pressure or automated test suites.<\/p>\n In exchange, the authors are candid about what they give up. Open-world evaluations cannot cleanly rank models against each other \u2014 run-to-run variability across this class of task may exceed the difference between two capable models. They are not reproducible in the way a benchmark run is. And they require domain expertise to design and interpret that not every research group can supply.<\/p>\n What they can provide is something standard benchmarks structurally cannot: evidence about what AI agents can actually do when no one has pre-specified what “correct” looks like.<\/p>\n Who Is Behind the Paper<\/p>\n The authorship list runs to more than twenty researchers and is worth pausing on.<\/p>\n Helen Toner at Georgetown CSET<\/a>, interim executive director at Georgetown’s Center for Security and Emerging Technology, is listed as a co-author. Toner was an OpenAI board member in November 2023 when the board voted to remove Sam Altman as CEO \u2014 a decision that collapsed within days under pressure from investors and staff. She has since led CSET’s work on AI governance and policy. Her involvement here signals that this evaluation framework is not positioned as a purely academic exercise; it is also a governance instrument, a way for institutions and policymakers to get honest answers about AI research capability independent of company-produced benchmarks.<\/p>\n Gillian Hadfield at Johns Hopkins<\/a>, the Bloomberg Distinguished Professor of AI Alignment and Governance at Johns Hopkins and a Canada CIFAR AI Chair at the Vector Institute for Artificial Intelligence, is also listed. Hadfield’s research focuses on building AI systems that integrate into human normative systems \u2014 the legal, regulatory, and cooperative frameworks that allow complex societies to function. Her participation reflects a judgment that AI research capability, correctly measured, is a governance question as much as a technical one.<\/p>\n Rishi Bommasani of Stanford, who leads the Foundation Model Transparency Index<\/a> project, contributed alongside several researchers from the UK AI Security Institute.<\/p>\n The breadth of the coalition \u2014 spanning computer science, law, economics, AI governance, and national security policy \u2014 is itself a statement about what the team believes this methodology is for. It is not designed to serve the needs of AI labs benchmarking their own models. It is designed to serve the needs of institutions and researchers who need empirically grounded, independently verifiable answers to questions that billions of dollars in AI investment currently ride on.<\/p>\n The Larger Stakes: What Benchmark Scores Cannot Tell Us<\/p>\n The paper’s significance goes beyond methodology. It lands in the middle of a debate whose financial and policy consequences are enormous.<\/p>\n Forecasts of explosive AI progress \u2014 the scenarios in which AI recursively improves itself, compressing decades of research into years \u2014 hinge on a specific empirical claim: that AI agents can automate AI research. If that claim is true, it follows that the pace of AI development is about to accelerate in ways that demand urgent regulatory and safety responses. If it is false \u2014 or if the evidence for it consists entirely of benchmark scores that are structurally subject to Goodhart optimization \u2014 then the urgency of those scenarios rests on evidence that this AI R&D automation forecasts paper<\/a> argues cannot support it.<\/p>\n Until the Princeton paper was posted this morning, both sides of that debate were arguing from the same corrupted evidentiary base: benchmark performance on tasks with known answers. Advocates and skeptics of rapid AI progress have had essentially no direct evidence about what AI agents can do when the answer is genuinely unknown.<\/p>\n The two case studies in this paper do not settle the question. Two data points are not a body of evidence. But they are, the team argues, the right kind of evidence for the first time \u2014 evidence produced by an evaluation paradigm that does not allow the capability under test to be optimized away before the test runs.<\/p>\n The CRUX project evaluation schedule<\/a>, which produced this paper as its second and third evaluations (following a first study on autonomous iOS app development and publication), plans to release new evaluations every one to two months. If that cadence holds, the next twelve months should produce a small but growing body of evidence that could, for the first time, give AI researchers, policymakers, and investors a measurement they can actually trust.<\/p>\n What Benchmark-Based Evaluation Gets Right \u2014 and Why It Cannot Be Replaced Entirely<\/p>\n The Princeton team is careful not to argue that benchmarks are worthless. The companion paper explicitly notes that several benchmarks remain unsaturated<\/a> and useful, including SciCode, MMLU-Pro, and Humanity’s Last Exam. Benchmarks remain the primary tool for tracking progress across model generations at scale, for measuring efficiency, and for identifying average-case performance across large populations of tasks.<\/p>\n