An in-house codec to speed up speech in Liquid AI’s models

Built for speech in LFM3 models, LiquidCodec combines local processing, streaming, and audio tokens designed to make prediction easier.

Each 80-millisecond segment of speech is represented by 12 tokens in LiquidCodec, the audio codec developed by Liquid AI for LFM3-Audio-2B and LFM3-Omni-2B. This component converts a recording into a discrete representation that a model can process, then turns tokens back into sound. Its approximately 74 million parameters belong to the codec itself, separate from the multimodal models that use it. The work addresses both the fidelity of reconstructed audio and how easily a model can learn to predict its tokens.

The main change concerns this intermediate representation. In Mimi, the third-party codec used by the provider’s earlier audio models, successive codebooks correct errors left by the preceding ones. This dependency constrains prediction. The method adopted here, grouped finite scalar quantization, or GFSQ, divides audio features into independently quantized groups. A speech model can then predict multiple tokens within the same segment in parallel over several refinement steps.

A decoder turns this representation back into sound by directly predicting spectral coefficients. According to the technical post, this avoids certain progressive reconstruction operations used by other codecs. The encoder and decoder share a symmetric architecture based on LFM3 blocks. The encoder is causal: it processes the available signal without waiting for future audio, making it suitable for streaming input. The component works with mono speech sampled at 24 kHz.

On a single CPU thread of an Apple M4 Max, internal measurements report encoding at approximately 361 times real time and decoding at 87 times real time, using FP32 precision. For decoding, one second of audio therefore requires about 11.5 milliseconds of computation in the tested configuration. These figures measure the codec, rather than a complete voice assistant pipeline: they exclude the time a language model needs to prepare a response and predict the tokens. The 80-millisecond segments likewise describe how the signal is divided, rather than measuring response latency.

The weights available on Hugging Face can be loaded with Transformers. The repository contains the BF16 version; quantized decoders are not distributed separately but are included in the GGUF versions of LFM3-Audio-2B and LFM3-Omni-2B for llama.cpp. According to the provider, quantization-aware fine-tuning reduces the decoder’s footprint by a factor of 3.5 in Q40, with a small decline in reconstruction metrics. That reduction does not apply to the memory footprint of the complete assistant.

Distribution is governed by the LFM Open License v1.0. Despite its name, the license includes a commercial restriction tied to annual revenue, with a threshold set at $10 million. Businesses exceeding that threshold must obtain separate commercial permission. This is an open weights release, without unconditional commercial rights for every organization.

Training began with 264,000 hours across 107 languages, drawing on sources that included web recordings, audiobooks, and conversations. An initial stage taught the codec to reconstruct the signal by directly comparing its waveform and spectrum with the original. According to the team, metallic-sounding artifacts remained. A second stage, using 55,000 hours of cleaner speech, added networks trained to distinguish reconstructions from real recordings to refine perceptual quality.

The evaluation separates two questions: faithfully reconstructing a recording and producing speech from text. To compare the latter task with Mimi, X-Codec 2, and NanoCodec, the researchers trained speech synthesis systems using the same LFM3-1.7B backbone, compute budget, and 101,000 hours of speech in six languages. Results aggregate three generation runs per test. According to the team, repeating the experiment with Qwen3-1.7B produced consistent trends.

Across eight test sets, the provider reports the highest ViSQOL scores and lowest mel spectral distances for reconstruction, as well as the highest UTMOSv2 naturalness scores for speech synthesis. The intelligibility advantage, measured through word or character error rates, covers four of the eight test sets. The model card limits these findings to speech: music, singing, and environmental sounds have not been evaluated. The 107 pretraining languages therefore do not represent 107 languages validated by these comparisons.

The French results illustrate the differences between evaluation criteria. On FLEURS, the BF16 version records a 6.91% word error rate for speech synthesis, compared with 6.37% for X-Codec 2, despite a higher UTMOSv2 score. The version with a Q40 decoder brings that rate down to 5.26%.