Hugging Face and alphaXiv published their ICML 2026 Open Reproductions report on August 13, after a challenge that ran from July 15 to August 2. The initiative attempted to reproduce findings from 2,226 accepted papers, representing 34% of the conference.
The project began with 6,341 ICML 2026 papers indexed alongside their abstracts and main scientific claims. Its 1,221 community members then published 6,816 Trackio logbooks documenting reproduction work. Together, those records show not only which claims could be checked, but also the conditions and methods used by each team.
Participants could use tools including Claude Code, Codex, Cursor and OpenResearch’s orx. Each attempt produced a static Hugging Face Space containing the report, executed code and resulting artifacts. A complete agent trace could also be published in a Dataset, giving other researchers material to inspect instead of leaving the result inside a private session.
Verified results were only part of the picture
An automated Logbook Judge, running on the open-weights GLM-5.2 model, assigned each claim one of four verdicts: verified, falsified, toy or inconclusive. The verdicts for 35,908 claims were frozen in a public dataset when the challenge closed.
Of the papers examined, 1,103, or 51%, had at least one independently verified claim. Two hundred and sixty-six papers were fully reproduced with every extracted claim verified, while 632 were partially reproduced without any claim being falsified. Across the project, real experiments confirmed 3,978 individual claims.
Those figures do not translate into a single reproduction success rate. Some claim was falsified or contested in 496 papers, representing 23% of those examined. Forty-nine papers had every claim falsified and none verified, while 242 papers received opposing verdicts from independent reproduction teams working on the same claims.
Another 502 papers produced only small-scale evidence, and 280 produced no result that could establish anything. Missing artifacts were the most common reason for the latter outcome. An unresolved verdict therefore does not mean that a paper was shown to be wrong; in some cases, the available material could not support a stronger conclusion.
Human review remained necessary
Thirty-five participants reported having falsified a result. The challenge team rechecked every reported falsification by rereading the paper and logbook, then rederiving the mathematics or reimplementing the experiment. Authors of several papers confirmed findings reported by the challenge, and two arXiv corrections were being prepared when the report was published.
The report gives concrete examples of why reproduction still requires judgment. A paging reproduction measured an additive term growing as 0.38 ln k; an extended reimplementation through k = 1,024 confirmed that growth at roughly nine standard deviations. In a paper on an attention forward-pass and Frank-Wolfe method, three teams found counterexamples after 224, about 3,800 and 6,416 steps.
Other checks found a mismatch between inverse KL in a self-distillation theory and direct KL in the published default code. In a paper on three Transformer projections, about 66% of evaluated positions were padded EOS tokens, changing the reported 3.1% quality cost to about 9.4% after correction.
According to the report, agents working alone could get trapped in local loops, misread scale-dependent behavior or mistake a unit mismatch for a falsification. The most reliable results came from workflows in which a human guided the agent, questioned an assumption or stopped a poorly founded experiment.
The challenge shows how open logs, code and artifacts can make AI-assisted reproduction more inspectable at conference scale. It also leaves a practical limit: automated verdicts do not replace human review. For anyone assessing AI research claims, the key distinction is whether a claim was verified, falsified or left inconclusive because the evidence or scale was insufficient.