An expression Layer between the LLM and an agent’s voice

LiveKit Expressive mode automatically adjusts emotion, pacing, and nonverbal sounds in voice agents based on the context of the conversation.

Expressive mode adds a layer to LiveKit Agents that controls how a voice agent delivers its responses. Instead of always sending neutral text to the TTS engine, the LLM can now determine from the conversation context whether a response should include more warmth, excitement, hesitation, a pause, or certain nonverbal sounds.

The system sits between the LLM and the TTS. LiveKit injects instructions corresponding to the active voice engine into the prompt so the model can add prosody markup directly to its response. The framework then normalizes those instructions, converts them into the format understood by the TTS provider, and removes them from the transcript shown to the user.

This layer makes it possible to switch providers without requiring developers to manually handle different markup syntaxes. Fish Audio S2.1 Pro, Inworld TTS 2, Cartesia Sonic models, and xAI TTS-1 are supported initially. Expressive mode works with any LLM, but on the voice side it requires a model available through LiveKit Inference that declares a compatible markup dialect.

LiveKit has also changed how text is sent to speech synthesis. According to its tests, sending each sentence separately caused the TTS to lose part of the emotional context and could lead to shifts in tone from one sentence to the next. The framework therefore batches multiple sentences before synthesis to preserve greater consistency across an entire response. LiveKit says it did not observe a meaningful latency penalty when using a fast LLM such as Gemma 4 31B.

The behavior remains configurable. Developers can adjust pacing, remove disfluencies, or disable sounds such as laughter, sighs, breathing, or vocalizations. Custom instructions can also supplement or replace the guidance automatically injected by LiveKit.

The expressive state can also be passed to the interface. LiveKit normalizes signals produced by certain providers into eleven mood categories that can be used to drive a visualizer or other visual indicator. Fish Audio, Inworld, and Cartesia expose this information through `lk.expression`. xAI can adjust vocal prosody, but does not currently provide this expression signal to the interface.

Expressive mode applies only to agents built with an STT, LLM, and TTS pipeline. Real-time speech-to-speech models do not use this layer because they handle their vocal characteristics directly.