Cloning voices and editing lines locally with ZONOS3-Mini

Zyphra’s text-to-speech model can clone a voice from an audio clip, adjust delivery word by word, and run locally in a browser.

To replace a word in a recorded line, Zyphra’s voice studio first finds where it’s spoken. Once the transcript is corrected, ZONOS3-Mini regenerates that section and splices it back into the original take. In the browser, the studio uses a transcript supplied by the user or already attached to the clip.

The same model can turn text into speech. It can pick up a voice from a reference clip and use it to read a new line. Settings for perceived age, accent, and vocal texture also let users shape a character’s voice from a starting voice. From the tone of a line to the emphasis on a word

In Zyphra’s demos, the same sentence is delivered three ways. The first take is neutral. The other two whisper either the beginning or the end. The model tracks its place in the text as it generates audio, so it can apply the direction to the selected words.

Other controls, including pace, emotion, and vocal projection, can apply to the whole line. Their effects can overlap. Speeding up delivery can also change pitch or loudness, Zyphra notes. The studio can generate several takes from the same script, then revise a specific passage in the chosen take. First audio in under a second on the test phone

Two changes reduce the amount of computation. The model has 800 million parameters and activates about 100 million at each generation step. Its codec represents one second of speech with 150 audio tokens, down from 774 before, according to Zyphra. That gives the model fewer units to generate before the decoder turns them into sound.

In its technical post, Zyphra reports time to first audio of under 570 milliseconds on a Google 9-series phone and under 150 milliseconds on an M3 MacBook Air. The company says both devices generate speech in real time. The browser studio runs the model on the device through WebGPU. The web runtime documentation puts the initial download at about 1 GB, with additional files for some features. A ranking based on 420 test combinations

On ZTTS2-Eval, Zyphra’s own benchmark, ZONOS3-Mini ranked first among the open-weight models tested, ahead of Fish S2 Pro and MOSS TTS v1.5. The score combines 420 combinations of metrics, languages, and test sets. The benchmark covers 35 languages for read speech and 20 for recordings collected in real-world settings. It measures factors including intelligibility, similarity to the reference voice, and speech rhythm.

The model weights are available on Hugging Face under the Apache 2.0 license. The runtime code on GitHub is licensed under MIT and includes the browser runtime and native versions for running the model locally.