IFStruct, Liquid AI's Benchmark for Structured Outputs

Liquid AI launches IFStruct, an open-source benchmark testing LLM structured outputs in JSON or YAML, where its LFM2.5-350M model beats Qwen3.5-4B.

Liquid AI addresses a recurring weakness in language models with IFStruct, an open-source benchmark that measures their ability to produce valid structured outputs, in JSON or YAML. The evaluation isolates a single criterion: adherence to the requested format, regardless of the quality of the generated content. Each test imposes a precise schema: fields, types, allowed values, numerical bounds, and expected number of elements. The verdict is binary; a response only passes if all constraints are met, with no added fields.

The interest lies in the variety of formulations. Requirements appear sometimes as a chat request in plain language, sometimes as a bulleted list, raw JSON schema, annotated example, or ASCII table. Half of the cases are rewritten in natural prose to mimic real-world usage, including requests that change their mind midway. Several cases intentionally force string escaping, a frequent breaking point in production.

Liquid AI accompanies the release with a highlighted result: its LFM2.5-350M model, trained by reinforcement on a dedicated training set, improves its success rate from 21.10% to 44.90%. According to the publisher, this score surpasses that of models more than ten times larger, such as Qwen3.5-4B and granite-4.0-h-tiny. The benchmark and test set are available on GitHub and HuggingFace.