Fish Audio teaches voice agents to recognize the right time to respond

Fish Audio launches a real-time transcription model capable of detecting interruptions, hesitations, and speaker turns for voice agents.

Fish Audio Teaches Voice Agents When to Respond

A user says "yes" while the assistant is speaking. Should it interrupt its response, or simply understand that the user is listening? This distinction, obvious in human conversation, remains difficult for voice agents to master. Fish Audio introduces Realtime STT, a new transcription model designed not only to recognize words, but also to interpret the flow of a conversation.

Available in preview under the identifier `transcribe-stream-1`, the system combines speech recognition, turn-taking detection, and conversational context awareness. The company claims a transcription error rate of 2.21%, which reportedly earned it first place among 20 models on the Cekura benchmark at its launch. Transcription aimed at understanding when the user has finished

Traditional speech recognition systems convert an audio sequence into text. For conversational assistants, that is not always enough.

A person might pause to think, nod along while the assistant is speaking, or be interrupted by background noise. If the agent treats every silence as the end of a sentence and every sound as a new utterance, the exchange quickly feels unnatural.

Fish Audio addresses this with a feature called Smart Turn Detection, designed to recognize when a user is genuinely taking the floor or concluding their turn.

The model analyzes what is being said and how the speech develops, rather than relying solely on a predefined duration of silence.

In documented behavior, a simple "mm-hm" spoken while the assistant is talking can be recognized as a backchannel—a signal of listening or agreement. The agent can then continue its response uninterrupted.

Conversely, if the user genuinely resumes speaking, the system emits a signal that allows the synthetic speech to pause and yield the floor.

This distinction represents a potential improvement in experience for voice agents, particularly in scenarios where both speakers frequently talk over each other. A claimed 2.21% error rate on Cekura

Fish Audio reports achieving a 2.21% Word Error Rate (WER) on Cekura's speech recognition benchmark.

This metric counts words replaced, omitted, or added by a transcription system. The lower the percentage, the closer the transcription is to the ground truth.

The result published by Fish Audio corresponds to first place among 20 evaluated models at the time of its announcement.

The Cekura benchmark compares systems on a dataset comprising 1,000 public audio clips and recordings from real telephone conversations. It also measures the speed of first-word delivery and the time required to yield a final transcription.

However, the scope of this ranking warrants clarification. Leaderboard results evolve over time with updates and newly added models. Consulting the benchmark today does not guarantee that the top spot held at launch remains a permanent ranking.

Furthermore, a 2.21% error rate does not mean the model will correctly transcribe 97.79% of all conversations, regardless of language, audio quality, or recording conditions.

Speech recognition benchmarks and turn-taking evaluations measure different capabilities. A strong transcription score alone does not guarantee that an assistant will always know the right moment to respond. Initial transcription within roughly half a second

Fish Audio's documentation provides several latency figures measured on its production service using synthetic speech.

Stage · Indicative Latency First transcribed words · 0.5 seconds after speech begins Probable end detection · 0.6 seconds after speech stops Final transcription · 1.1 seconds after speech stops

The model features a capability called Eager End of Turn, which emits a signal indicating an utterance is likely finished before its full transcription is finalized.

This allows the application to begin computing its response earlier. If the user resumes speaking, the system can cancel the provisional conclusion.

While this approach potentially reduces waiting time between turns, it introduces a trade-off: responding too quickly risks interrupting someone who was not actually done speaking.

The reported latencies do not necessarily include all end-to-end network delays, language model reasoning time, or audio response generation latency. Recognizing the user's voice amidst noise

Another common challenge for voice agents involves their acoustic environment.

A playing television, background chatter, or another person in the room can trigger speech recognition when the primary user hasn't made a request.

Fish Audio introduces detection for utterances that likely do not originate from the primary user. The system can flag these as secondary events, preventing the assistant from automatically responding to them.

The documentation specifically advises against interrupting the agent upon the very first detection of voice activity. The initial signal might stem from background conversation, a minor acknowledgement, or ambient noise.

Instead, the application should await confirmation indicating the user is explicitly taking their turn.

This separation of acoustic activity from conversational intent is especially crucial in automobiles, shared spaces, and noisy telephone environments. Conversational memory for improved name recognition

Realtime STT also leverages conversational context to refine its transcriptions.

Developers can supply a list of specific terms, such as names of individuals, products, or technical jargon. The documentation allows up to 100 custom terms, with a maximum length of 32 characters each.

The system can also ingest the text of sentences previously uttered by the assistant.

If the assistant asks the user to confirm a name or reference, the user's response can be interpreted with the preceding question in mind, rather than as an isolated audio fragment.

Recent turns and historical context provided at the start of a session can similarly serve as reference material. Transcriptions are able to maintain consistent speaker identifiers