Perplexity builds a custom engine for Qwen on M5 Macs

Perplexity has released Lily, a local Rust and Metal inference engine designed to accelerate Qwen3.6-35B-A3B on Macs with M5 chips, with its clearest gains appearing during text generation.

Fitting a large model into a Mac’s memory is not enough. The machine must also be able to read its weights, process a long prompt, and generate an answer at a usable speed. With Lily, Perplexity is moving away from general-purpose frameworks to build an engine dedicated to a single architecture and one generation of Macs.

The project accompanies Hybrid Compute, a feature that divides tasks between remote models and one running locally. Cloud models handle research and reasoning, while the local model can work with files and applications stored on the computer. To make this division feel seamless, Perplexity needed to accelerate the local side.

Lily is written in Rust and uses custom Metal operations developed for Apple hardware. Neither PyTorch nor MLX is involved during execution. Model loading, conversation management, generation, and GPU operations are brought together inside a single program.

This approach differs from MLX and MLX-LM, Apple’s commonly used tools for running models on its computers. Their main advantage is compatibility with many architectures. Lily takes the opposite direction: it gives up that versatility to organize every operation around one model.

That model is Qwen3.6-35B-A3B. It contains 35 billion parameters but activates only about 3 billion for each token it processes. A router selects eight experts from a pool of 256, alongside one shared expert used for every input.

This design reduces the required computation but makes execution more irregular. Two consecutive tokens may activate different parts of the model. The engine must identify the relevant experts, group the corresponding data, and load their weights without creating unnecessary exchanges with the CPU.

Qwen3.6-35B-A3B also combines 10 full-attention layers with 30 Gated DeltaNet layers. The former consult a memory that grows with the conversation. The latter compress earlier information into a fixed-size recurrent state.

These two mechanisms do not place the same demands on the computer. Attention becomes progressively more expensive as the context grows, while the recurrent state must be updated in token order. Lily therefore prepares different execution paths depending on the layer, text length, and generation phase.

Perplexity distinguishes between prefill and decode. Prefill is the processing of the request, conversation history, and attached documents before the first word appears. Hundreds or thousands of tokens can be handled together during this phase.

Decode begins afterward. The model usually produces its response one token at a time. This stage has little opportunity to reuse the same weights at any given moment and depends more heavily on how quickly the machine can move data from memory.

This difference explains Lily’s approach. During prefill, the engine prioritizes matrix operations that can reuse one block of weights across many rows. During decode, it follows a path better suited to processing a single row and mainly attempts to reduce the amount of data read for each new token.

Unified memory allows the CPU and GPU in a Mac to access the same physical memory pool. The model therefore does not need to be copied in full from one memory space to another. This design does not make data movement free, however: memory bandwidth remains a limiting factor when weights must be read again for every generated token.

The checkpoint used by Lily is quantized to 4 bits. Its 35 billion parameters occupy about 19.4GB, compared with roughly 70GB in bfloat16. The compressed values are accompanied by a scale and bias used to reconstruct them when needed.

One of the main optimizations performs this reconstruction directly during matrix multiplication. Lily does not create a complete temporary version of the expanded weights in unified memory. It expands only a small block, briefly holds it in faster GPU memory, uses it, and then moves on to the next.

According to an internal comparison using a 512-token prompt, integrating this conversion into the multiplication increased prefill speed by 77.4%. This figure measures the effect of the change within Lily itself, not its lead over MLX-LM.

Expert routing also remains on the GPU. The engine counts how many tokens have been assigned to each expert, determines their positions, and groups the corresponding data without asking the CPU to validate each step.

This arrangement launches more small operations but avoids interruptions caused by transferring decisions between components. In Perplexity’s 512-token test, keeping routing on the GPU nearly doubled Lily’s prefill speed.

Not every expert receives the same number of tokens. Lily therefore adapts the size of its compute blocks to the actual workload. A block that is too small increases setup overhead, while an oversized one leaves part of the GPU idle when only a few items have been assigned to an expert.

The engine also keeps Gated DeltaNet’s recurrent state in registers while scanning the prompt. This prevents it from being written to memory and read back at every stage. Perplexity reports a 5.6% prefill improvement at 2,048 tokens, smaller than the gains from expert processing because the experts account for most of the workload.

For long prompts, Lily works through successive chunks. The weights remain in memory, while the recurrent state and attention cache carry context from one chunk to the next. Previous information is not discarded, but temporary values for the entire text do not have to remain in memory simultaneously.

This approach limits the required working memory. It does not eliminate the rising computational cost of full attention: the 10 relevant layers must still process more information as the prompt grows.

During generation, Lily keeps the selected token on the GPU so it can be used directly in the next step. A copy is sent to the CPU for delivery to the application, but the engine does not wait for a full round trip before preparing what follows.

Perplexity says that generating a single token can launch