One-third of SWE-Bench Pro tasks would be broken, according to OpenAI

OpenAI reveals that 30% of SWE-Bench Pro tasks are broken due to strict tests and vague prompts, urging developers to build better AI coding benchmarks.

OpenAI has published an audit of SWE-Bench Pro, one of the most widely used benchmarks for measuring agentic coding capabilities, concluding that a significant portion of its tasks are flawed.

The benchmark's premise: across a set of 731 public tasks drawn from the history of software repositories, a model must implement a feature that passes new tests without breaking existing functionality. In eight months, frontier models progressed from 23.3% to 80.3% success rate on it. OpenAI subjected this dataset to an analysis pipeline: an automatic filter first flagged 286 suspicious tasks, which were then reviewed via two parallel paths—investigative agents built on Codex, capable of reading the repository and replaying tests, and a human annotation campaign where each task was handled by five engineers.

The verdict converges. The automatic pipeline marked 27.4% of tasks as broken, human annotation 34.1%, leading to an estimate of approximately 30%. Four defects dominate. Overly strict tests impose implementation details not present in the prompt, failing otherwise correct solutions. Under-specified prompts omit requirements that hidden tests expect. Low-coverage tests poorly verify functionality, allowing incomplete fixes to pass. And some misleading prompts guide towards incorrect behavior.

OpenAI links these issues to the data's origin: issues and pull requests from open-source repositories, written for human collaboration, are ill-suited for isolated evaluation tasks. The company draws two conclusions: it withdraws its recommendation to adopt SWE-Bench Pro, having already discarded SWE-Bench Verified, and urges the creation of benchmarks designed by experienced developers. It also notes that sufficiently capable models are now being used to audit benchmarks themselves.