From Image to Storyboard, Tencent Proposes a New Protocol to Evaluate AI Video

With OmniVBench and Omni-R2V, Tencent offers a benchmark and 340,000 training examples to evaluate the control of AI-generated videos.

AI video generation is no longer limited to transforming a text prompt into an animated sequence. Character images, camera movements, sketches, graphic styles, or storyboards now serve as references to guide models. But a video that appears convincing at first glance can deviate from the initial instructions: reusing the correct character without respecting its motion, maintaining a composition while altering the requested style, or mixing elements from multiple sources.

Researchers from Tencent, The Chinese University of Hong Kong, and The Hong Kong University of Science and Technology (Guangzhou) have developed OmniVBench, an evaluation benchmark dedicated to these scenarios. Their work is accompanied by Omni-R2V, a dataset of approximately 340,000 examples intended for training models capable of generating videos from multiple references. Evaluating what a video truly respects

Video generation rankings often rely on an overall assessment of the output. A sequence may receive a high score for its visual quality or general resemblance to a reference, without correctly reproducing the specific elements that were meant to be preserved.

The method described in the OmniVBench study distinguishes three dimensions: reference fidelity, instruction compliance, and intrinsic video quality.

The first verifies whether the expected features were preserved: character identity, object appearance, camera trajectory, style, or narrative progression. The second measures whether the model correctly applied the requested transformations. The third independently evaluates the technical quality, aesthetics, and physical plausibility of the output.

This separation allows for the identification of errors that are difficult to distinguish in a single score. A model can produce an aesthetically pleasing video while transferring a character's motion to the wrong subject or retaining details that should have been removed. Seven task families, from characters to storyboards

The benchmark gathers 813 evaluation cases, organized into seven families and 18 sub-tasks. The researchers cover content, motion, style, structure, and narrative references, multi-subject compositions, and combinations of references covering different aspects of the image.

In content tasks, the model must preserve the identity or specific characteristics of a character, object, or environment. Motion tests examine its ability to transfer an action or camera movement without unnecessarily copying other elements from the source video.

Structural references include line drawings, simplified geometric representations (greyboxes), and preparatory storyboards. Narrative tasks go further: producing a shot that is consistent with the preceding one, or transforming several storyboard panels into an animated sequence.

Finally, multi-reference evaluations require assembling elements from different sources. This can involve preserving the appearance of a character depicted in an image, applying motion sourced from a video, and adhering to a composition dictated by a sketch.

To evaluate these constraints, OmniVBench uses 12,172 instance-specific checkpoints. The criteria determine not only whether the references were used, but also whether their features were assigned to the correct elements in the final output. Seedance 2.5 and MiniMax H3 among the top-ranked models

The team evaluated 11 models, including Seedance 2.0 and 2.5, Kling 3.0 Omni, Gemini Omni, Happy Horse 1.0, MiniMax H3, and several open-source alternatives.

The overall leaderboard results place Seedance 2.5 in first place with 72.68 points, followed by MiniMax H3 at 72.41 points. Seedance 2.0 scores 70.91 points, ahead of Happy Horse 1.0 at 70.89 and Gemini Omni at 70.76.

This proximity in overall scores masks differences across specific tasks. Gemini Omni notably achieves 75.11 points on narrative references, while Seedance 2.5 reaches 73.35 points on structural references. In this evaluation, MiniMax H3 displays the best results among the models categorized as open source.

Above all, the analyses show that content references are generally better respected than motion or style references. Performance varies considerably depending on the type of reference used, even for the highest-ranked models. Therefore, a good result in preserving characters does not guarantee mastery over camera movements, narrative continuity, or complex compositions.

These results correspond to the models and configurations tested in the study. They do not constitute a general ranking of all video generation capabilities. A benchmark that breaks down errors

To construct the evaluations, the researchers first define the features that each reference must convey. Vision-language models are then used to prepare tailored criteria lists for each example, which are manually verified before use.

Reference fidelity and instruction compliance are evaluated separately. Video quality judgments rely on specialized tools focused on technical quality, aesthetics, and physical consistency.

The team also compared automatic evaluations with scores assigned by three human evaluators on a sample of 100 cases. Spearman correlations reach 0.77 for reference fidelity, 0.74 for instruction compliance, and 0.82 for video quality.

Detailed checklists show a higher correlation with human evaluations than an overall assessment, particularly for reference fidelity. Nearly 340,000 examples to train future models

In parallel, Omni-R2V aggregates exactly 339,570 training examples. The dataset covers seven reference families divided into 13 sub-tasks, which are distinct from the benchmark's 18 sub-tasks.

The examples associate one or more reference images or videos, instructions in English and Chinese, and a target video. They include live-action footage, 2D animation, and 3D animation, with durations reaching up to 20 seconds.

Its construction relies primarily on a professional video corpus owned by the team, supplemented by public data. Sequence-matching and