A model recites two facts by heart and remains unable to connect them

The alphaXiv agent shows that a model with 100% recall on two facts remains unable to link them. The limits of fine-tuning for logic.

alphaXiv is launching autoresearch, a system that tasks an agent with reproducing a research paper, from code to figures. For the demonstration, the agent, built on GPT-5.6, replicated the results of a paper dedicated to a fine-tuning puzzle: why a model that has perfectly memorized a fact remains unable to use it.

The replicated finding is clear-cut. Across one thousand synthetic chains, the tested model, a 1.5-billion-parameter Qwen2.5, achieves one hundred percent recall by the fourth epoch—meaning the fourth complete pass over the training data: it reproduces every learned fact word for word. But as soon as it has to chain two of these facts together to answer a two-step question, accuracy remains at zero percent, and stays there until the thirty-first epoch. The model knows, yet does not know how to use it.

The mechanistic analysis does not solve the puzzle; it deepens it. By transplanting internal states from one pass of the network to another—a technique known as activation patching—the authors of the replication observe that the signal for the correct answer does indeed exist, located primarily in the early and late layers, and that it can be patched to be stronger. Yet, the correct answer never ends up ranking first. The knowledge leaves a trace in the model, but nothing brings it to the surface.

alphaXiv accompanies its demonstration with caveats that deserve to be shared: the exercise is not a benchmark, the observation remains anecdotal, and the approach only applies to a category of papers—interpretability, inference engineering, benchmarks—where reproduction does not require massive computing resources. The replication itself had to substitute synthetic data for an unpublished experimental dataset, which supports the paper's behavioral thesis, but not its detailed figures.