OpenAI is evaluating the scientific judgment of agents with GeneBench-Pro

OpenAI launches GeneBench-Pro, an open-source benchmark of 129 computational biology problems to evaluate the scientific decision-making of AI agents.

Rather than memorizing facts or executing a protocol, GeneBench-Pro seeks to measure the discernment of AI agents when faced with ambiguous scientific data. This new open-source benchmark from OpenAI evaluates the quality of decisions made during a computational biology analysis. Each problem places the model before a realistic and messy dataset, a succinct experimental context, and a question related to a concrete decision. The model must explore the data, choose a relevant approach, iterate, then deliver an answer, with a wrong path leading to failure.

The evaluation brings together 129 problems across ten domains, from statistical genetics to cancer genomics and pharmacogenomics. Each case is synthetically constructed: the team controls the complete causal structure and directly simulates data generation, which allows for deterministic correction against known targets and avoids unintentional shortcuts. A portion of the problems was reviewed by external experts, who estimate that a single one would require a human specialist between twenty and forty hours of work.

Regarding results, OpenAI acknowledges significant room for improvement: even its most performant model, in its most advanced configuration, does not exceed one-third of the problems solved. The publisher is opening ten representative questions on HuggingFace and will entrust a subset of fifty questions to Artificial Analysis for an independent third-party evaluation.