PhoneLLM releases a model built specifically for voice agents

Pipecat is releasing an open-weights model for automated calls, promising fast responses, accurate tool use, and an estimated cost of $0.0025 per minute.

Booking a table, checking a file, or transferring a call does not necessarily require several seconds of reasoning. With PhoneLLM Alpha 1, Daily and the Pipecat team are introducing a model specialized in common voice-agent tasks, where response speed matters as much as answer quality.

PhoneLLM is not an audio model. It does not transcribe speech or generate spoken responses on its own. It operates between a speech recognition service and a text-to-speech engine inside a pipeline such as Pipecat. Its role is to understand the transcribed request, decide which action to take, call the appropriate tools, and write a concise response before it is read aloud.

General-purpose models can perform these tasks, but their reasoning modes often introduce delays that are difficult to reconcile with the pace of a phone conversation. Disabling that reasoning reduces latency but may also weaken instruction following and tool use. PhoneLLM was trained specifically to work without this intermediate step.

The model is based on NVIDIA Nemotron 3 Nano 30B-A3B. It contains 30 billion total parameters but activates approximately 3.5 billion for each input. Pipecat fully fine-tuned its weights using NVIDIA’s NeMo framework and a collection of phone conversations and customer support scenarios.

Its training covers financial services, healthcare, retail, hospitality, and outbound calls. The goal is to produce concise, relevant responses that remain coherent throughout long conversations. Particular attention was given to the difference between saying that an action has been completed and actually calling the tool required to perform it.

In a typical failure, an agent might tell a caller that a reservation has been confirmed without sending any request to the booking system. PhoneLLM is intended to avoid this mismatch by collecting the necessary information, calling the appropriate service, and confirming the operation only after it has been performed.

To measure these behaviors, the team is also introducing PhoneBench v1, a benchmark focused on telephone agents. It evaluates speaking style, relevance, conversational coherence, tool-call accuracy, consistency between statements and actions, compliance with authentication or escalation procedures, and the final outcome for the caller.

Responses are graded by a panel of model judges calibrated against human evaluations. The scenarios, system instructions, and tool lists are kept separate from the data used to train PhoneLLM. This approach is intended to test whether the model can handle previously unseen situations, but PhoneBench remains a benchmark developed by the same team releasing the model. Its results have not yet been independently validated.

According to PhoneBench, PhoneLLM delivers performance comparable to GPT 5.6 Terra on typical voice-agent tasks while costing 94% less and reducing P95 time to first token by approximately 1.3 seconds. The comparison uses configurations with reasoning disabled or minimized, which reflects real-time constraints but does not represent every possible use of the models being evaluated.

On a lightly loaded NVIDIA B200, P95 time to first token reportedly falls below 100 milliseconds for a single request. This figure only measures PhoneLLM’s server-side processing. It does not include transcription, speech synthesis, application logic, or the full network path that contributes to the latency experienced by the caller.

To better represent production workloads, Pipecat and Modal also simulated complete conversations using voice-agent pipelines. Their optimized configuration can support 88 agent processes on one B200 while keeping P95 time to first answer token below 600 milliseconds, including the network overhead covered by the test.

The estimated cost of $0.0025 per minute assumes that density of 88 agents, an effective B200 cost of $13.392 per hour on Modal, and a 70% utilization rate. It covers the language model and does not necessarily include speech recognition, voice generation, telephony, or the other services required by a complete agent.

These results therefore depend on high-end infrastructure, strong utilization, and optimizations developed with Modal. A deployment with fewer simultaneous calls, different hardware, or a generic vLLM configuration will not automatically achieve the same cost per minute. Modal estimates that its settings approximately double the number of concurrent agents compared with a standard configuration.

The PhoneLLM Alpha 1 weights are available on Hugging Face. The model has a 262,144-token context window, currently supports English, and can be served with vLLM or SGLang. The team recommends setting the temperature to zero and disabling reasoning completely.

Pipecat’s modifications are released under the two-clause BSD license. Because PhoneLLM is derived from Nemotron 3 Nano, NVIDIA’s open model license continues to apply to the underlying model, including attribution and documentation requirements when redistributing it.

PhoneLLM can be deployed directly through Modal or installed on compatible infrastructure. Its 30 billion parameters in bfloat16 still require substantial hardware, even though only 3.5 billion are active during processing. Open weights do not necessarily mean lightweight or easy to run locally.

PhoneLLM reflects a broader shift toward using smaller models trained for specific jobs instead of deploying a general-purpose system for every interaction. For a voice agent, calling the right tool within a few hundred milliseconds may be more useful than producing an extended chain of reasoning. It remains to be seen whether the PhoneBench results hold across other languages, less predictable calls, and less favorable infrastructure.