Pipette measures models where they are actually meant to run

Pipette compares models, quantization, runtime, and device to measure the quality, speed, latency, and memory use of locally run AI.

A score obtained on a powerful server cannot tell you whether a model will respond quickly on a phone, fit within the available memory, or suffer a sharp performance drop as the context grows. Pipette sets out to measure these constraints by treating each deployment as a complete combination of model, quantization, runtime, and device.

Developed by Liquid AI in partnership with Artificial Analysis, the platform is intended for teams that want to run models locally on smartphones, laptops, PCs, or embedded hardware. Rather than ranking models solely from their original weights, it compares the versions actually deployed after they have been converted into formats suitable for a specific device.

This distinction applies in particular to quantization, which reduces the numerical precision of a model’s weights to decrease its size and memory requirements. Two versions carrying the same model name can perform very differently depending on the selected format. More aggressive compression makes local execution easier but can also weaken instruction following, reasoning, or response quality.

Pipette associates each result with four components: the model being tested, its quantization, the runtime version, and the hardware used. Context length, execution settings, operating system, and the exact request shape are also recorded. Results produced under different conditions remain separate instead of being combined into an average that would be difficult to interpret.

The initial release includes more than 10,000 measurements across more than 1,000 deployment configurations. It represents around 30 models and approximately 35 model classes, with several quantization formats and context lengths ranging from 256 to 8,192 tokens where device memory allows.

The first verified results cover a MacBook Pro equipped with an M5 Max chip, an iPhone 17 Pro, and a Samsung Galaxy S26 Ultra. A configuration based on an AMD Ryzen AI Max+ 395 with Radeon 8060S graphics is expected to be added later. The initial announcement therefore mentions four devices, although data for the AMD system was not available at launch.

The interactive dashboard lets users filter data by device, model, quantization, context length, runtime, and reasoning mode. Measurements include time to first token, total generation time, prompt-processing speed, response-generation speed, and peak RAM usage.

Dedicated views show how performance changes as context length increases. This factor can materially alter model selection. On a Galaxy S26 Ultra, for example, Granite 4.0 H 350M retains 78.4% of its generation speed between 256 and 4,096 input tokens, compared with 33.8% for Granite 4.0 350M using Q4KM quantization. Similar model sizes therefore do not guarantee comparable scaling as the amount of information increases.

Pipette also highlights tradeoffs between speed and quality. On an iPhone 17 Pro, MiniCPM5-1B processes a configuration with 2,048 input tokens and 256 output tokens in 3.47 seconds, compared with 4.12 seconds for LFM2.5-1.2B-Instruct. The latter, however, scores nine points higher on MATH-500 at the same quantization level. Neither model leads on both criteria at once.

The platform therefore avoids automatically naming a single overall winner. Its charts can display Pareto frontiers, representing configurations for which no competitor is simultaneously faster, less memory-intensive, and higher quality. The most suitable choice depends on the intended use: fast autocomplete, a conversational assistant, longer-document processing, or features that must work without an internet connection.

Performance measurements are collected on the relevant devices. Quality evaluations, however, follow a different process. The IFBench, GPQA Diamond, and MATH-500 scores currently published by Pipette use the same quantized model files but are produced on reference systems equipped with NVIDIA H100 80 GB hardware. They are then paired with the speed and memory measurements collected locally.

This approach assumes that the quality of a quantized file does not vary by device when the runtime correctly performs the same operations. It also avoids tying up a phone for the full duration of the evaluations. Still, the quality scores calculated on the reference system should be distinguished from the performance measurements actually obtained on the phone or computer.

Artificial Analysis provides a complementary view focused on models that can run on smartphones. It combines Pipette data with five evaluations covering instruction following, tool use, knowledge, resistance to fabricated answers, scientific reasoning, and mathematics.

The files tested in this mobile ranking use four-bit quantization or lower and must fit within 8 GB of memory, including the cache required for an 8,000-token context. At launch, 33 of the 41 evaluated builds successfully ran on the iPhone 17 Pro. According to Artificial Analysis, the time required to process the same request varied by a factor of 30, while peak memory ranged from approximately 0.4 GB to 6.9 GB.

These differences show why a model’s name or parameter count is not enough to predict its behavior. An architecture using partial activation can generate quickly while still retaining a large memory footprint. A slower model may perform better on a specific task, while a particularly verbose reasoning system can become impractical on a phone despite earning a high score when no time limit is applied.

To make its measurements reproducible, the protocol fixes the input and output shapes, uses deterministic decoding, and discards an initial warm-up run. Each measurement is then based on five repetitions. A preliminary check also monitors device temperature and load to avoid publishing results obtained under conditions considered unstable.

Each submission records the benchmark version, settings, model file, quantization, runtime, hardware, and operating system. The results pages provide access to the measured value and, where