Skyfall AI argues that language models do not learn from experience

Skyfall AI releases the Morpheus benchmark, proving that large language models fail to learn from experience or adapt in persistent corporate environments.

Skyfall AI lab releases Morpheus, a benchmark that simulates persistent corporate environments, and a study that draws a harsh conclusion: large language models are not continual learners.

The setup stands out from typical benchmarks. Where classic reinforcement learning environments reset the world at the end of each episode, Morpheus resets nothing: past decisions weigh on what follows, objectives shift through configuration changes, and the consequences of an action sometimes appear only after several simulated days. The tasks involve logistics: allocating resources among competing priorities, scheduling shipments and receipts, all while the context deteriorates (data transmission failure, drop in warehouse capacity).

The counterintuitive result is the core of the study. On the simplest task, the tested models maintain stable performance across disruptions. Skyfall sees this not as proof of robustness, but as its opposite: stability means the model applies the same heuristic from start to finish, without adjusting its strategy. It does not adapt; it executes a recipe inherited from its pre-training that still happens to fit. In other words, performance measures the coverage of pre-training, not the ability to learn.

The illusion dissipates on the difficult task, where the chain of consequences grows longer. Performance collapses, with one of the models repeatedly dropping to zero without recovering any logic. Another failure is reported: unable to detect that conditions had changed, a model continued to apply an increasingly unsuitable policy, as the useful signal had slipped out of its context window.

The most unsettling argument lies in the opacity. When a reinforcement learning agent fails, the failure can be attributed to a specific mechanism (too slow to adapt, policy oscillation) and corrected. When a language model fails, since its weights have not changed, there is nothing to measure: context overflow, out-of-distribution training data, or reward misalignment produce identical curves. The breakdown is real, the cause untraceable.