From Qwen to Arete-27B: 803 questions to measure how AI reproduces China's official narratives

Across 803 questions, Nous Research measures how often Arete-27B, a retrained version of Qwen3.8-27B, reproduces official Chinese narratives.

Article 35 of the Chinese Constitution serves as the starting point for one of the 803 questions posed to Qwen3.8-27B. The evaluation framework expects the model to distinguish between the freedom of speech guaranteed in the constitutional text and the restrictions applied in practice when criticism targets the Chinese Communist Party. The benchmark dataset published by Nous Research uses questions like this to compare Alibaba's model with Arete-27B, a version the lab retrained to reinforce what it calls "Western values." When Citing an Official Position Becomes Endorsing It

For each question, the team selected two to four facts from a cited source. It then assigned the scoring to Grok 4.7 without revealing which model had generated each response. The evaluator first checks which facts are included, omitted, or contradicted. It separately assesses how the response presents the official Chinese position and must cite a specific passage if it determines that the model endorses that position.

This distinction matters when interpreting the numbers. A response can mention Beijing's position, clearly attribute it to the Chinese government, and receive a score of zero on the "propaganda" criterion. An unanswered question also receives a zero on that criterion, but scores zero for factual coverage as well. The team refined its evaluation framework using 25 manually scored questions before applying it to the full dataset. The Biggest Shift Is in Freedom of Speech

Across five runs of the 803 questions, with reasoning mode disabled and identical generation settings, the rate of responses classified as endorsing official Chinese narratives falls from 22% for Qwen to 0.9% for Arete. The average factual score rises from 2.32 to 2.98 out of 5. The largest difference appears in the freedom of speech category, where the endorsement rate drops from 60% to approximately 1%, while the factual score increases from 0.89 to 2.44 out of 5.

These results describe responses to English-language questions selected to test factual coverage and the endorsement of official Chinese narratives. The choice of questions, sources, and expected facts defines the scope of what the experiment measures. By publishing these materials alongside the evaluation framework and scoring scripts, the team gives other researchers the means to reproduce the test and examine its assessments. Using Example Responses to Retrain the Model

The lab asked Inkling to generate the example responses used for retraining. The publicly released dataset contains Inkling's responses, Qwen's responses before retraining, and quality-control annotations. The dataset documentation indicates that most of the selected responses still carry a factual-error flag. Supervised fine-tuning involved 52.8 million text tokens and updated the parameters of the language component, while the vision encoder remained frozen.

On Hermes Bench, Nous Research's internal benchmark for general capabilities, Arete scores 65.60%, compared with 65.77% for Qwen. The lab considers this 0.17-percentage-point difference too small to conclude that performance declined on this benchmark. The Arete-27B model weights are available for download on Hugging Face, where the model card lists an Apache 2.0 license.