AI-generated design faces a panel of professionals
Using 33 briefs, the Frontier Design Benchmark tests AI models against designers’ preferences and the limits of quality scores.
In the preference rankings, human reference designs outperform almost every model. On the criteria-based rubric, however, several AI models receive higher scores. This gap runs through the Frontier Design Benchmark, developed by Contra Labs with BenchFlow: avoiding execution flaws and winning over designers are different achievements.
The benchmark combines blind comparisons, a design critique rubric, and a tool that identifies recurring visual traits in AI-generated websites. Its findings cover websites and applications built from detailed briefs, rather than the full range of design disciplines. Bringing client briefs into the test
An analysis of 4,600 website projects posted on Contra informed part of the benchmark’s 33 tasks. Twenty-seven draw on real commissions, rewritten for fictional clients; six come from projects that startups assigned to designers on the platform. Brand identity, target audiences, and functional requirements accompany the requests. Examples include a festival website with ticket sales and a museum audio tour.
To collect model outputs, BenchFlow runs an agent in an isolated environment and records its work, cost, and final result. Models are accessed through OpenRouter; wherever possible, each uses its own lab’s agent framework. Human reference designs follow a different process: designers propose three directions in a moodboard, receive feedback, and then build the website using their preferred tools. The methodology therefore describes different production processes without establishing equivalent time or resource budgets. The rankings have already changed since launch
The launch thread placed designers first, winning 76% of their comparisons. The leaderboard reviewed now puts Gestalt, described as a model still in stealth, in first place with an Elo rating of 1,931. Human reference designs follow at 1,739, then Claude Fable 5.2 at 1,713, Claude Opus 5.5 at 1,691, and GPT-6.1 Sol at 1,668. The website now reports 4,311 blind votes.
These positions reflect preferences between two submissions responding to the same brief. They are neither a measure of compliance with a commission nor a general assessment of professional competence. For the undisclosed model, the publication states that Contra Labs is working with its team to continue evaluations and provide feedback.
The design critique rubric offers another perspective, examining typography, contrast, composition, accessibility, and brand consistency. The authors report a correlation of 0.94 between rubric scores and the models’ preference rankings. Human reference designs are the exception: flaws identified by the rubric do not prevent designers from frequently preferring them. Execution scores do not fully capture the panel’s aesthetic judgment. Visual habits do not establish origin
The Slop Detector examines characteristics that make a page resemble AI-generated work in its dataset. The authors identify different habits across model families: horizontal arrows on links in Claude outputs, diagonal arrows in GPT outputs, and content frequently arranged in cards in Gemini outputs. These observations describe the pages tested; they are insufficient to attribute a website to a particular model with certainty.
A low detector score does not guarantee a strong design either. In the published results, unfinished pages produced by DeepSeek V4 Pro look unlike typical AI-generated pages while ranking poorly with designers. The gallery is publicly viewable, but the full dataset is available only upon request.