Testing AI safety across entire conversations

From simulated dialogues to automated graders, Anthropic details the conditions needed to interpret Claude’s safety scores.

A gap between 98% and 94% of conversations judged appropriate is not always enough to rank two models. In an evaluation focused on romance scams on claude.ai, those scores were obtained by Claude Opus 5 and Opus 5.5. According to Anthropic, the difference is not statistically significant. The example appears in a methodology post on safety evaluations, which examines what these tests actually measure and the conditions needed to draw conclusions from them.

The first choice concerns which behavior to score. A high refusal rate may be useful when handling fraudulent requests, but becomes misleading if any mention of distress triggers a refusal to engage. For its multi-turn evaluations, the lab measures the share of conversations in which the model responds appropriately from beginning to end, using a rubric specific to the risk being studied. A single problematic response causes the entire conversation to fail. Tests must also identify unjustified refusals of legitimate requests.

That unit of measurement changes how scenarios are built. A domain expert defines the simulated user’s objectives, situation, and behaviors to explore. The first message remains fixed; another Claude model then generates the user’s subsequent messages in response to the model being tested. Multiple runs vary the scenario’s characteristics. A grader model finally examines each complete transcript. The authors emphasize starting from observed interactions or incidents, since entirely invented scenarios may be too simple or too far removed from real usage.

The automated grader itself becomes an object of scrutiny. The guide recommends precise criteria, often framed as binary questions, and comparison with human annotations produced without seeing its verdict. Agreement between humans provides a reference point: if experts disagree with one another, the rubric needs reviewing before asking a model to decide. The authors also recommend regrading some transcripts with another judge or a reworded rubric to check whether the conclusion holds.

Response length, presentation order, or a model’s recognition of its own outputs can influence those judgments. A harmful response does not become acceptable because it includes a warning. Grading instructions must therefore specify what matters in the content, without automatically rewarding cautious framing. A grader failure must also be flagged and the affected item excluded from the evaluation, rather than silently producing a misleading score.

Sample size and the analysis plan provide another level of control. The lab proposes a minimum of one hundred cases for a complex multi-turn evaluation and one thousand for a simpler single-turn test. Those are its methodological guidelines, not a guarantee of precision for every protocol. Metrics, comparisons, and improvement thresholds should be set before reviewing the results. For some multi-turn API evaluations, reported uncertainty margins can exceed ten percentage points on either side of the estimate.

Product configuration can also shift the score substantially. In an evaluation concerning suicide and self-harm, Claude Fable 5.1 achieves an appropriate conversation rate of 60% on the API, compared with 94% on claude.ai with new system instructions and several targeted mitigations. The authors explain that they identified responses implicitly validating harmful behavior, then changed the instructions. The thirty-four-point gap concerns that evaluation and those configurations; it measures neither the model’s overall safety nor the isolated effect of adding a single sentence.

To attribute an improvement to a specific intervention, the post recommends changing only the element being studied. A comparison that simultaneously changes the model, its reasoning level, and its instructions describes two different configurations without identifying the cause of the gap. Model and grader versions, settings, and dates must be recorded so trials can be reproduced. The Inspect framework and the Petri auditing tool, built on Inspect, are cited as tools for running, logging, and analyzing these evaluations.

The conditions under which tests run are also part of the issue. Some tests disable standard safeguards to observe the model’s baseline behavior, making control over the testing environment essential. Following its own incidents, the lab says it hardened its evaluation environments, temporarily suspended many tests that provided internet access, and added real-time monitoring to web search evaluations requiring that access.