An automated research loop for ERNIE 5.5 Flash

Applied to ERNIE 5.5 Flash, ERNIE Evolve cycles through code changes, experiments, and checks to improve inference.

A 33.7% increase in throughput while preserving bitwise-identical results: that is one of the gains Baidu reports for an ERNIE 5.5 Flash inference configuration. In its technical post on ERNIE Evolve, the lab describes an automated research system tasked with choosing directions, modifying code, and checking the effects of each change. This work was applied to optimizing the model while it was in development.

The starting point is a code repository, an objective, and defined evaluation criteria. Experiments run in isolated environments, and their results guide subsequent experiments. The lab places this approach within recursive self-improvement, or RSI: using AI to improve the systems that make it work. An evaluator beyond the agents’ reach

Research is organized into two nested loops. The outer loop explores several directions in parallel, drawing on previous experiments and a knowledge base. Within each direction, the inner loop selects a code version, formulates a hypothesis, and prepares a modification. Candidate versions form a tree that preserves the lineage of changes.

Evaluation is handled by a fixed program kept outside the repository being optimized and hidden from the agents. It checks correctness and performance on the specified workloads. Changes outside the permitted scope are excluded, and gains must exceed the chosen noise threshold to be accepted. A review component then examines the code and measurements to identify departures from the objective or attempts to exploit weaknesses in the test. Public comparisons across four metrics

On NanoChat AutoResearch, the lab reports a validation score of 0.8763 bits per byte, compared with 0.8878 for the selected public baseline, under a five-minute training budget on one NVIDIA B200 GPU. Lower scores are better. The optimization adds statistics on token sequences and retrieval from training documents, combining their predictions with those of the network.

On NanoGPT Speedrun, the reported results improve from 39.9 to 38.9 seconds to reach the target loss on one track, and from 2,690 to 2,590 steps on another. The comparison uses the best public results selected at the time of evaluation. On SOL-ExecBench, which focuses on GPU kernels, the average score of 0.819 remains below the 0.827 reference.

The ERNIE Evolve Results repository provides final implementations and logs for examining the measurements. For the first NanoGPT track, it reports 38.92 seconds across 18 runs using eight 80 GB H100 GPUs. These materials document results produced by the system; they do not release its complete orchestration. On P800, faster execution paths were already available

The application to ERNIE 5.5 Flash comprises three separate projects. On Kunlunxin P800 accelerators, four research rounds bring generation speed to 623.6% of the baseline, or 6.236 times the original speed. The model and deployment configuration remain unchanged, according to the lab. A substantial part of the gain comes from enabling optimized implementations that were already available, followed by fixes to scheduling and communication.

The deterministic inference project starts from a baseline already optimized by engineers. Generated tokens and their probabilities must remain bitwise identical, regardless of the other requests in the batch. After approximately three days of research, throughput increases by 33.7% under these conditions, according to the published measurements.

The third project concerns single-request decoding on GPUs, with one