Putting AI agents’ judgment to the test with raw lab data

Developed by OpenAI and Ligo, BioBasicsBench tests AI agents on 100 tasks drawn from real experiments, from interpreting images to checking controls.

A culture plate shows no colonies, so the control appears to have worked. The handwritten label on the rim changes the interpretation. The plate contains kanamycin, while the experiment uses ampicillin. In this BioBasicsBench task, GPT-5.6 Sol draws its conclusion from the absence of colonies. After reading the label, GPT-6 Astra realizes that the control doesn’t match the experiment and rules it out.

The benchmark developed by OpenAI and Ligo Biosciences tests this kind of judgment. To interpret an image or measurement, an agent also has to check how the data was obtained and what conclusions it supports. Scientists built the tasks using data from their own experiments. Lab results come with ambiguities

Some of the 100 tasks ask agents to read a gel, compare microscopy images, or extract a measurement from an instrument file. Agents may also receive protocols, handwritten notes, or videos of lab procedures. Nineteen tasks test whether agents recognize when there’s too little evidence to reach a conclusion. In one example, the detection ranges for two dyes overlap, so both dyes can contribute to the observed brightness. Astra spots the ambiguity and asks for additional controls before concluding that the cells are dead.

Each task was reviewed by a Ligo scientist who hadn’t written it, then by an outside researcher with a PhD who works in a lab, and finally underwent one more internal review. Responses earn points for each criterion they meet. Depending on the task, the rubric can assess the conclusion, the measurement, the reasoning behind it, or whether the agent identifies a gap in the evidence. Partial answers can earn credit

The evaluation covers seven OpenAI models across 37 combinations of model and reasoning effort. With five attempts per task in each configuration, the team scored 18,500 responses. At the shared “xhigh” setting, GPT-6 Astra earned an average of 66.7% of the available points, compared with 43.8% for GPT-5.6 Sol, a gap of 22.9 percentage points. At that setting, Astra averaged 4,704 output tokens per task, compared with 9,099 for GPT-5.6 Sol. Seeing the right detail and drawing the wrong conclusion

In another exercise, Astra correctly distinguishes two groups of bands on a gel. It then assumes the larger fragment is the expected product. Comparing the bands with the controls leads to the opposite choice in the reference answer. The model spots both patterns and assigns them the wrong identities.

In a photo of a culture plate, Astra also overcounts colonies by about 12%. It merges some neighboring colonies, misses others, and counts empty areas as colonies. With the plated volume and dilution held constant, that error inflates the estimated efficiency of DNA uptake by the same percentage.