Agents code fast, science validates slowly

An OpenAI report shows that agents speed up code writing but struggle with scientific validation, which remains dependent on humans.

Many analysis tools in use in laboratories were born as the code accompanying a paper, written by small academic teams with little engineering experience and without the time to handle packaging, testing, or maintenance. This observation is the starting point for an exploratory field report from OpenAI dedicated to eight scientific computing projects conducted with agents, primarily in the life sciences. Five utilized Codex alone, while three combined it with Claude Code. The case studies are written by the teams themselves and cover a broad spectrum, from routine maintenance to migrating from one language to another or refactoring to run natively on GPUs.

The clearest result lies in a shift. Engineering work ceases to be the limiting factor, and validating what the agent produces becomes it, which still relies entirely on human judgment. Contributors describe agents that are effective on well-defined requests but unable to tell if their own work is scientifically valid, and readily displaying confidence where the code contains obvious errors. Verification methods that hold up therefore rely on an external reference or a measurable target: strict equality of outputs, parity with an existing tool, expected statistical behavior, or answers known in advance thanks to simulated data.

None of these projects worked on the first try. The teams broke down tasks, iterated, and inserted intermediate benchmarks. The first implementations arrive quickly, then edge cases and subtle numerical discrepancies take much more time, with the last mile concentrating most of the effort.

The most interesting reservation concerns the aftermath. When rewriting costs less, nothing prevents multiplying neighboring rewrites, which fragments users and dilutes the expert attention necessary for the reliability of a single tool. Mature scientific software carries undocumented conventions, compatibility requirements, and an acquired trust that translating the source code does not reproduce. Two projects saw their modifications merged back into the original repository, while a third changed hands due to the lack of an active maintainer. Without an identified owner or a credible maintenance plan, the report warns, today's modern rewrite becomes tomorrow's abandoned code. The document presents itself as retrospective and exploratory.