Speed up AI responses without retraining models

Agents rewrite and test the code that powers models at Together AI. Throughput improves by up to 2.6x in published tests.

Making an already trained model run faster by letting agents rewrite and test the code that executes it: that is the job assigned to Together Compiler. Presented by the research team at Together AI, this system looks for optimizations tailored to the model, the available accelerators, and the requests received. It acts on the execution software used to produce responses.

Two figures summarize the announcement, but they measure different things. The factor of 252 refers to the number of optimization experiments the agents can try in a given amount of time. The announced throughput gain for the models reaches up to 2.6 times in the configurations described in the research report. The results are published by the system’s authors. Testing a change without restarting everything

Automatic optimization runs into an obstacle: each idea must be tried to find out whether it actually speeds up computation. Some systems restart the entire inference engine with every attempt, including loading the model and preparing its execution. These operations can take several minutes. During that time, the agents get little feedback on their proposals.

The system keeps one engine running and reloads new versions of the code to be tested into it. The model parameters remain loaded. Several agents work simultaneously and share their observations, in order to feed this infrastructure with candidates. The claimed factor of 252 corresponds to this experimentation capacity, compared in the study with a system that restarts the engine between measurements.

The tests are split between two levels. A first engine uses part of the model’s layers to quickly eliminate slow or faulty proposals. A second uses more of them to obtain a more representative measurement. In an experiment with GLM 5.3 and B200 accelerators, the authors report 1,354 candidates evaluated, compared with 168 using a single slower measurement level. The solution found is 7.2% faster in this comparison. Good optimization also depends on the requests

Code that performs well on synthetic data can lose its advantage in real-world use. Mixture-of-experts models only activate part of their components for each token, and this distribution depends on the inputs. The length of the prompts and the number of simultaneous users also change the work that must be done. Optimizing an isolated operation therefore is not always enough to speed up the full service.

The Together AI technical post describes several variations observed in its activity. Prompts can get longer as users move from chat to agent-driven tasks. Clients may also suddenly send more requests, or reuse less content already present in memory. A configuration suited to one moment can become less efficient a few days later.

An estimation tool first selects the most promising compute allocations across accelerators. It combines hardware characteristics and measurements to avoid trying every possible configuration. The agents then try to improve the code within this framework. When conditions change, the search can be restarted: this is what the company calls continuous compilation. Gains to be read together with their measurement conditions

The tests cover several models, including Kimi K3, GLM 5.3, MiniMax M3.1-Flash, and DeepSeek v4.1-Flash. The authors announce up to 2.6 times more tokens processed per second per accelerator, at comparable per-user speed, after twenty-four hours of optimization by tuning. The tests cover NVIDIA, AMD, and AWS Trainium2 accelerators. However, the Trainium2 case uses a different synthetic conversation set from the AgentX profile used for NVIDIA and AMD.

The highest figures highlighted for individual speed reach 1,111 tokens per second with MiniMax M3.1-Flash on B200 and 1,205 with DeepSeek v4.1-Flash on GB300, with a single simultaneous request. The metric used is the inverse of the 90th percentile of the delay between two tokens. It describes the speed of the stream received by a user, distinct from total throughput per accelerator and from the time needed to obtain a complete response.

Some of these comparisons also incorporate a change in workload distribution. For MiniMax, the system retains a model split across eight accelerators, compared with four in the published reference configuration. The observed gain therefore combines configuration choices and code optimization. It cannot be attributed entirely to the agents’ code changes alone.

The SemiAnalysis InferenceX project provides the comparison framework and public results. The new measurements presented here remain those of Together AI. Using this protocol does not amount to independent validation of a record by the organization that maintains it. An optimization already deployed, still reserved for some access

In its presentation of so-called polymorphic inference, the company says it uses the system on some of its on-demand model services. Customer access to the tool itself is announced in limited beta. The ability to directly optimize a dedicated self-service deployment is among the upcoming features.

The system is presented as a way to continuously adapt execution to usage patterns. The changes retained go through quality and functionality checks before deployment, according to the authors. The expected gain concerns speed or processing capacity; the published results do not establish a general improvement in the models’ knowledge or reasoning.

The researchers also tested their system on weather, biology, and time-series models, as well as on simulations of physical equations. The report gives speedups of 1.5 to 6.4 times compared with the initial implementations, with exact preservation of outputs in these tests. For weather forecasting, the authors study using the saved compute time to produce more forecasts and combine them.