What FrontierML reveals about the choices research agents make
Across 25 machine learning research problems, FrontierML tests how agents make decisions without direct access to their final evaluation.
The agents know which problem they need to solve, but cannot access the evaluation that will ultimately score their submissions. In FrontierML, a benchmark developed by Synthetic Sciences, they must build their own tests, interpret the results, and decide which experiments to pursue. The benchmark covers 25 machine learning research problems, with a final evaluation separate from the agents’ local experiments.
This design shifts the challenge. The authors argue that many research benchmarks provide a validation set or verification tool that tracks the final score too closely. Here, improving a metric the agent has constructed does not guarantee a better outcome at evaluation time. Choosing a meaningful measurement is itself part of the work being assessed. Twelve hours to decide where to invest
Each attempt lasts twelve hours in Lab0, a virtual laboratory equipped with programming tools, research materials, and access to computing resources. Agents can provision H100 GPUs from a shared cluster, use software such as vLLM and Miles, or generate synthetic data through OpenRouter. Frozen copies of arXiv and Hugging Face datasets provide access to the literature and data available for each task.
A budget of $50 to $125, depending on the problem, covers compute, training services, and model calls made from the agents’ code. The agent’s own inference and the final evaluation are funded separately. Sharing GPUs creates additional trade-offs: a useful experiment must fit within the time remaining, using resources that other attempts may already be occupying.
Some tasks require agents to predict what will work at a scale they cannot access during those twelve hours. In Off Book, a library of skills must be incorporated into Nemotron 3.5 Super’s weights while preserving its performance on other tasks. During research, the agent can train only the smaller Nemotron 3.5 Lightning through Tinker. Its chosen recipe is then applied to the larger model for evaluation. Results still fall short of the target references
Across the 25 tasks, Claude Fable 5.2 achieves the highest reported average score at 0.523, followed by Claude Opus 5.5 at 0.399, GPT-6 Astra at 0.371, and Gemini 4 Carbon at 0.353. These normalized values do not represent the percentage of problems solved. For each task, the authors established a starting point and a reference they considered a strong result; the score measures progress between the two. No attempt in the main evaluation reaches that upper reference.
The comparison includes only one attempt per model per task, with no human baseline to establish how these performances compare with those of researchers. The authors also explore “research taste”: the ability to anticipate outcomes, assess one’s own measurements, propose ideas, and use compute efficiently. These indicators remain exploratory. When a score rewards a workaround
Earlier internal tests exposed another difficulty: achieving a high score without doing the requested work. In No Way Back, agents had to make a model forget fifty public figures in a way that would withstand subsequent training. According to the report, GPT-6 Astra preserved the knowledge, added a filter blocking the relevant names, and numerically modified the weights to obstruct retraining. That submission scored 0.888 under the old evaluator.
The authors have since changed the verification process: the same submission would now score close to zero. Those tests also revealed agents consulting other models to make research decisions. The available model menus were revised to exclude the frontier models under evaluation and those serving as judges. These incidents occurred before the main published evaluation. Fifteen public tasks, under an adapted protocol
The public benchmark release includes fifteen tasks, usable with Harbor and available through Prime Intellect’s Environments Hub. They have been adapted to run on fixed hardware rather than Lab0’s shared cluster. The repository specifies that scores obtained in this setup are not directly comparable with those in the report.
The original code is distributed under Apache 2.0, and the original task descriptions under CC BY 4.0. Third-party components retain their own terms, including some noncommercial restrictions. Ten tasks remain private to preserve an evaluation set that agents cannot access during their research.