Remembering without rereading everything: another memory for AI assistants

The Memorilla method condenses the useful information for each question into sixteen representations, without modifying the language model parameters.

Sixteen memory elements for answering from old exchanges or long documents: this is the format chosen in Memorilla’s main experiments. The project, led by a team bringing together Stanford and other institutions, proposes an external memory whose content passed to the model changes depending on the question. The documents remain stored separately; the sixteen elements form the synthesis that the model receives to prepare its answer.

The method presented by the researchers addresses a difficulty faced by personalized assistants: useful information may be found in an old conversation whose wording bears little resemblance to the current request. RAG, or retrieval-augmented generation, provides the model with passages drawn from a document base. A selection based on similarity to the question can leave out information phrased differently. Passing the entire history, in turn, increases the amount of text to process. The experiments compare these different strategies with the same language model, Qwen3-8B. A memory reconstructed for each question

Each document, for example a conversation or a chapter, is first transformed into a numerical representation by an encoder. This operation is performed only once. The representations are stored and reused; adding a document does not require re-encoding the previous ones.

At question time, a learned module consults this set and produces sixteen representations adapted to the request. They are injected directly into the input representations of the language model, alongside the question. This is therefore not a written summary that the user could read. The number of transmitted elements remains fixed in this configuration, while the stored collection continues to grow.

The encoder and language model parameters remain unchanged during training. Only the memory module learns, with around 194 million parameters in the main prototype. The modules published on Hugging Face are associated with Qwen3-8B and its Qwen3-Embedding-4B encoder. The setup requires access to the model’s internal representations and a module trained for it. Savings on the text sent to the model

The figure of 4,212 times fewer tokens corresponds to the largest gap measured, on NarrativeQA. Full-context reading there transmits an average of 67,385 tokens from the documents, versus sixteen memory representations. The calculation excludes the question and common instructions. It does not describe the total size of the stored documents, nor an equivalent reduction in all system costs.

The speed measurements concern a separate configuration. With 512 documents of 250 tokens each, the authors report about 0.74 seconds per answer, compared with 19.9 seconds in full context. The test uses Qwen3-8B on an 80 GB H100 GPU, one request at a time, and generation capped at 32 new tokens.

The software configuration also matters: full context uses YaRN to go beyond the context limit imposed on the model, with generation that does not benefit from a paged memory cache. The factor of about 27 depends on this setup. RAG is not timed in this comparison. The initial encoding of the documents remains, for its part, a cost that must be paid before the memory can be reused. An agent that makes better use of its own history

A second experiment focuses on TextWorld, a text-game environment in which an agent explores rooms and carries out a quest. Each completed turn enriches its memory. In this case, the module provides eight memory elements instead of sixteen, and its training uses the game rewards.

The acting model, Qwen3-4B, remains frozen. Its average success rate rises from 18.6% with history alone to 24.2% with the trained memory, a gain of 5.6 points. The two conditions have the same history and the same input budget. Here, memory is added to the text rather than replacing it: the experiment measures what it helps the agent extract from already available information. Linking information, at the risk of losing details

Across the eight main evaluations, the method comes out on top in five according to the metrics chosen by the team. In PersonaMem-v2, devoted to personalization, it reaches 44.6% correct answers, compared with 30.9% for the best competing method, xRAG. These results follow training of the module on the training sets of the evaluated tasks; the two personalization datasets also receive specific fine-tuning.

Compression can, however, lose names and other precise details. On NarrativeQA, passage retrieval remains better for certain questions asking to identify a person or an exact element of the story. The experiments concern text and models from the Qwen family. Moving to other model families or other domains still needs to be evaluated.

The two methods also succeed on different questions. On FactKG, which checks claims by linking several facts, Memorilla reaches 84.3% correct answers and RAG-50, 78.4%. At least one of the two gives the correct answer in 94.3% of cases. This last figure is an upper bound calculated with knowledge of the correct answers, not the result of a hybrid system able to automatically choose the best output.