SVG-Bench: 352,000 human votes compare AI models for vector graphics

Datapoint AI releases SVG-Bench, an open dataset gathering 352,270 human votes to compare 14 AI vector creation models.

A vector illustration can look appealing while containing errors in proportions, typography, or composition. Conversely, a technically valid SVG file does not guarantee that the design follows the original instructions. To assess these capabilities, Datapoint AI has developed SVG-Bench, a dataset containing 352,270 human preference votes on vector graphics produced by 14 AI models. The evaluation covers logos, icons, illustrations, diagrams, user interfaces, and precision-focused tasks. Its results reveal substantial differences between specialties: a model that performs particularly well on illustrations may rank near the bottom when asked to create a structured diagram. Comparing Vector Graphics Through Human Preferences

SVG-Bench uses 400 original prompts, evenly distributed across eight categories. Each of the 14 evaluated models generates SVG code directly from each prompt. The resulting files are rendered through Chromium onto a white background and converted into 1024 × 1024-pixel images.

Evaluators are not told which models produced the images. They see the original prompt and two rendered designs, then select the one they prefer. Every comparison receives ten independent human judgments, rather than relying on a single reviewer's opinion.

The dataset includes responses from 20,323 annotators in 183 countries, covering 35,227 validated image pairs and exactly 352,270 votes. Datapoint scheduled every possible pairing of the 14 models for each prompt. However, comparisons involving outputs that could not be rendered as usable SVGs were excluded from the voting process.

This method measures human visual preference between two renderings. It does not directly assess how easily the SVG can be edited, whether its source code is well structured, or whether it meets accessibility requirements. An image preferred by evaluators may still violate constraints in the prompt. GPT-6 Astra Leads a Closely Contested Ranking

The overall leaderboard uses a Bradley–Terry statistical model to convert pairwise votes into Elo scores. Each of the eight categories receives equal weight, preventing a single type of graphic from dominating the aggregate ranking.

GPT-6 Astra leads with 1,065.6 Elo points, followed by Gemini 3.8 Flash with 1,059.2 and Muse Wonder with 1,047.3. Claude Fable 5.1 and Claude Opus 5.5 follow with 1,028.5 and 1,028.2 points. Qwen3.8 Max ranks sixth, ahead of Gemini 3.1 Pro.

The confidence intervals qualify this ranking. GPT-6 Astra's interval runs from 1,059 to 1,072, while Gemini 3.8 Flash's spans 1,052 to 1,066. Because they overlap, the results do not establish a statistically clear advantage for the top-ranked model over the runner-up.

The Elo ratings describe relative performance among these 14 models on the selected prompts. They are not universal design-quality scores or a comprehensive ranking of all available vector-generation systems. Arrow 2 Telos Leads Illustrations but Falls Behind on Diagrams

One of the most striking differences concerns Arrow 2 Telos, developed by QuiverAI. Although it ranks eighth overall with 1,011.6 Elo points, it takes first place in two creative categories, reaching 1,128 in illustrations and 1,133 in scenes.

These tasks emphasize characters, objects, props, spatial relationships, and visual depth. They leave room for stylistic interpretation, even when the prompt specifies particular elements.

The outcome reverses for diagrams, where Arrow 2 Telos ranks last among the 14 models. Diagram prompts require accurate representations of data, labels, connections, and flow logic. An attractive composition cannot compensate for incorrect relationships between the elements.

This contrast illustrates the difference between producing convincing artwork and constructing an informative visual governed by explicit constraints. In a poster or editorial illustration, evaluators may favor aesthetic choices. In a diagram, an incorrect connection or label can undermine the document's usefulness even if it remains visually polished. Gemini 3.8 Flash Performs Strongly on Structured Design

The category results reveal several distinct specialties. Gemini 3.8 Flash leads logos, diagrams, user interfaces, and precision tasks. Its scores include 1,092 Elo in diagrams and 1,078 in logos.

Precision exercises test requirements such as exact object counts, clock times, geometry, and spatial relationships. UI tasks ask models to produce interface components with specified text, values, and states. These categories require close instruction-following as well as coherent visual design.

Claude Opus 5.5 leads the icon category, while GPT-6 Astra ranks first in typography, narrowly ahead of Gemini 3.8 Flash. The variations demonstrate why a single overall ranking cannot determine the best model for every creative assignment.

A team creating a consistent icon family, a typography-driven logo, or a flowchart with exact relationships may therefore benefit from different models. Category-level results provide more useful guidance for particular design workflows than the overall podium alone. Technical Validity and Visual Preference Are Different Measurements

SVG-Bench also documents generation failures and costs. Of the 5,600 expected outputs, 5,508 produced usable SVG files. The remaining attempts included 42 malformed XML files, 41 responses without SVG content, and nine truncated generations.

Several models produced technically valid results for all 400 prompts, including GPT-6 Astra, Claude Opus 5.5, Claude Fable 5.1, and Arrow 2 Telos. Others were less reliable: Gemini 3.8 Flash produced 369 valid SVGs out of 400, while MiniMax M3 produced 364.

The distinction between technical reliability and visual preference is significant. Gemini 3.8 Flash ranks second overall despite 31 failed outputs. Because invalid outputs were excluded from human comparisons, the preference leaderboard does not penalize those failures in the same way as a metric measuring successful completion of all assigned tasks.

The