Perplexity reduces search time even before the answer is given

Perplexity details Ivy, Tulip, and ROSE, three components designed to accelerate embedding computation and result ranking on GPUs.

Before an answer appears in Perplexity, another operation has already begun. The service must convert the question into a numerical representation, retrieve the closest documents from its index, then rank the results by relevance. Saving a few milliseconds at each stage takes on a different significance when those steps are repeated at search-engine scale.

In a technical post dated September 4, 2026, Perplexity describes the infrastructure it uses to run its embedding and ranking models. The company organizes it around three components called Ivy, Tulip, and ROSE. The first prepares requests, the second groups and schedules them, and the third runs the models on accelerators.

An embedding represents text as a vector, meaning a sequence of values describing its position in a mathematical space. Two passages with similar meanings should produce nearby vectors, even if they do not use the same words. A question about the price of a train ticket can therefore be matched with a document discussing rail fares.

When an index is built, millions or billions of documents must be converted this way. Throughput matters more than the time required to process a single item. At search time, the priorities are reversed: a usually short query must be converted immediately so it does not delay the answer shown to the user.

Ranking happens after the initial selection. A scoring model examines a batch of documents and assigns each one a value to retain the most relevant results. This stage must process more text than a single query without adding too much delay before the answer is generated.

Perplexity wants to cover these different workloads with shared infrastructure. Its embedding models are generally small Transformers, sometimes derived from language models. The company therefore reuses part of the code already employed to serve its larger models.

Bulk document encoding resembles the prefill phase of a language model, when many tokens are processed simultaneously. A short query is closer to token-by-token generation: the GPU has less computation to perform, making the surrounding operations more visible in the total runtime.

Ivy serves as the HTTP entry point. Written in Rust, it receives calls from Perplexity services, parses the JSON, converts text into tokens, applies each model’s required formatting, and divides large batches into smaller chunks. Requests are then transmitted using an internal gRPC protocol.

This division prevents an entire batch from being sent to one instance, potentially overwhelming it while others remain idle. Ivy distributes chunks among several replicas and attempts to balance their workloads. The layer also incorporates the tokenization system developed in-house by Perplexity.

Tulip receives the gRPC requests prepared by Ivy. This server uses Rust, Tokio, and Tonic to track requests, assemble batches, and send them to the GPU. Its scheduler is deliberately simple: pending sequences are selected in the order they arrive.

That simplicity follows from the behavior Perplexity observed in small models. At the sequence lengths used in production, dense layers reportedly consume more time than attention. Cost therefore rises mainly with the total number of tokens rather than the number of sequences in a batch.

For a model with fewer than one billion parameters, Perplexity places GPU saturation at around 512 tokens. Once that threshold is reached, adding more requests to the same batch does not necessarily improve efficiency. Tulip can prioritize fast scheduling without constantly searching for the perfect combination.

ROSE, short for Runtime-Optimized Serving Engine, runs the model. This layer is developed primarily in Python and contains computation routines, model definitions, and implementations adapted to different graphics cards. It was originally designed for language models before being extended to embeddings.

Dense layers work identically in both cases because each token representation is processed independently. The main differences appear in attention. ROSE accepts inputs of varying lengths without artificially extending them to a common size. It also skips the KV cache needed for autoregressive generation because the encoder processes the complete sequence in a single pass.

This reuse reduces the work required to deploy a model derived from an existing language model. Perplexity says its `pplx-embed` model and Qwen3.5 decoding use some of the same computing components.

The optimization does not focus solely on what happens inside the GPU. When a batch is small, the time the CPU spends preparing and launching operations can exceed the actual computation time. The hardware then waits for instructions instead of working continuously.

CUDA graphs reduce that delay. Instead of launching each matrix multiplication, normalization, activation, and attention operation separately, the system records the entire sequence of actions required for one model pass. A single command can then replay the complete sequence.

The benefit is especially noticeable for short requests. Heavier batches already keep the GPU busy long enough to make launch costs less significant. With the small models studied, Perplexity places the tipping point at several thousand tokens and dozens of sequences.

This method creates another problem. A separate graph may be needed for every combination of token count and sequence count. The grid can quickly grow to several thousand configurations, requiring several minutes of preparation at startup.

Perplexity reduces the number of variations by grouping lengths into buckets of 64 or 256 tokens. The graphs are not all prepared when the server launches. A new configuration begins with a standard execution that loads the operations and memory areas. When that configuration appears a second time, the system records its CUDA graph. Subsequent calls use the recorded version directly.

This gradual capture shortens instance startup times and spreads the preparation work over several hours.