PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniVBench: A Benchmark and Large-Scale Dataset for Omni Reference-to-Video Generation

Wenxue Li, Peiyan Guan, Haoyang Jiang, Junxian Cai, Hualuo Liu, Chunjie Zhang, Chong Guan, Kai Huang, Songlian Li, Taiyi Wu, Yongjian Yu, Xiaotong Zhao, Alan Zhao, Eric Liu, Xi Chen, Yu Liu, Lei Zhu

arXiv:2609.22069v1cs.CV

TL;DR

Existing R2V benchmarks and datasets have limited coverage of heterogeneous reference settings, factor-level evaluation, and training data for diverse tasks. The paper introduces OmniVBench and the Omni-R2V Dataset as complementary evaluation and training resources, and reports clear model-performance gaps across benchmark task families and evaluation dimensions.

  • Problem

    Existing benchmarks cover limited reference types and compositions and mainly assess holistic consistency, while training datasets have fragmented task coverage and can require substantial preparation for reuse.

    Source: §1 · p. 2

  • Method

    OmniVBench evaluates diverse reference settings with factor-grounded checklists, while the Omni-R2V Dataset provides processed training samples built through task-specific reference–target pipelines.

    Source: §1 · p. 3

  • Results

    Evaluation of advanced open- and closed-source R2V models reveals clear performance gaps across task families and evaluation dimensions.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The benchmark's factor-grounded checklists enable fine-grained diagnosis beyond holistic reference consistency.

    Source: §1 · p. 3

Abstract

from arXiv · show

Reference-to-video (R2V) generation is evolving toward increasingly general and versatile reference control, giving rise to the emerging paradigm of omni R2V generation. However, existing benchmarks fall short of these emerging capabilities: their test cases cover limited reference types and compositions, and their evaluation protocols largely assess holistic reference consistency, overlooking whether reference factors are properly preserved, disentangled, and routed. Meanwhile, the high cost of constructing omni R2V training data makes suitable training resources scarce. To address these gaps, we introduce OmniVBench and the Omni-R2V Dataset for evaluating and training omni R2V models. OmniVBench expands R2V evaluation across broader reference types, fine-grained control tasks, and richer reference compositions, covering 7 task families and 18 fine-grained tasks spanning content, motion, style, structure, narrative, and multi-reference settings. We introduce factor-grounded evaluation with 12,172 case-specific checklist items, assessing whether intended reference factors are faithfully preserved, correctly disentangled and bound to their targets, and properly realized according to the instruction. We further introduce the Omni-R2V Dataset, bringing industrial-grade training resources for diverse R2V tasks to the broader research community. Drawing primarily on a large-scale corpus of professional video footage, it comprises 340K processed training samples spanning diverse reference types and multi-reference compositions. We develop task-specific pipelines for reference-target pair construction, offering a practical and scalable recipe for omni R2V data construction. Extensive evaluation of advanced open- and closed-source R2V models reveals clear performance gaps across task families and evaluation dimensions on OmniVBench, highlighting remaining limitations of current R2V models.

1 Introduction

As reference-to-video generation expands toward compositional and flexible control, the introduction identifies gaps in benchmark coverage, factor-level evaluation, and training data. It presents OmniVBench and the Omni-R2V Dataset as complementary evaluation and training resources, and reports performance gaps among current models.

1 Introduction

  • Existing benchmarks cover limited tasks and scenarios and largely assess holistic reference fidelity, leaving fine-grained controls and heterogeneous or compositional settings insufficiently evaluated.Omni R2V also requires assessing whether reference factors are selectively preserved, modified, suppressed, and composed correctly.Source: §1 · p. 2
  • Existing R2V datasets are often task- or reference-specific, while specialized pair-construction pipelines and unprocessed reference–video pairs make reuse and scaling challenging.The introduction identifies fragmented coverage and data preparation effort as barriers to broader training resources.Source: §1 · p. 2
  • OmniVBench spans 7 task families and 18 fine-grained tasks across heterogeneous reference types and compositional settings, with 12,172 factor-grounded checklist items.The checklists assess preservation, disentanglement, routing to targets, and realization according to instructions.Source: §1 · p. 3
  • The Omni-R2V Dataset provides 340K processed training samples across 7 task families, built primarily from professional footage through task-specific pipelines.The pipelines construct reference–target pairs and corresponding training instructions for diverse R2V tasks.Source: §1 · p. 3
  • Evaluation of advanced open- and closed-source R2V models reveals clear performance gaps across task families and evaluation dimensions.The authors say these results highlight remaining limitations of current R2V models.Source: §1 · p. 3

2 Related Work

Prior R2V benchmarks and training datasets cover only parts of the expanding reference-to-video task space. The related work identifies limited attention to heterogeneous reference factors and factor-level control.

Benchmarks for Reference-to-Video Generation

  • R2V benchmarks [33] [48] [56] focus mainly on content references and composition, leaving motion, camera, style, layout, story, and continuation underexplored.Their test cases and evaluation protocols also emphasize overall reference similarity and prompt alignment, with limited factor-level assessment.Source: §2 · p. 4

Training Data for Reference-to-Video Generation

  • Existing reference-conditioned datasets [56] [9] [3] remain centered on content references, with limited coverage of heterogeneous factors and their compositions.The related work contrasts these resources with the need for data spanning diverse instructions and single- and multi-reference settings.Source: §2 · p. 4

3 OmniVBench

OmniVBench probes R2V capability boundaries across diverse reference conditions, from single references to compositions of multiple references. It combines a broad task taxonomy with factor-grounded evaluation of reference fidelity, instruction realization, and video quality.

3.1 Reference-Centric Task Taxonomy

  • The taxonomy covers five single-reference categories—content, motion, style, structure, and narrative—and extends to multi-content and cross-aspect settings.Multi-content tasks compose entities or attributes; cross-aspect tasks combine controls such as content with motion, style, structure, or narrative.Source: §3.1 · p. 4, §3.1.2 · p. 6

3.1.1 Single-Reference Tasks

  • Content, motion, style, structure, and narrative tasks test entity preservation or transfer, selective motion and style transfer, recovery of incomplete structure, and story understanding.Motion references leave the intended motion undescribed; structure references require completing missing appearance details while preserving composition, poses, layout, and timing.Source: §3.1.1 · pp. 4–6

3.1.2 Multi-Reference Tasks

  • Multi-content tasks bind referenced entities or attributes to specified targets, while cross-aspect tasks jointly apply different factors without one reference overwriting another.Cases include direct entity composition, grounded factor composition, and combinations such as appearance images with a motion video.Source: §3.1.2 · p. 6

3.2 Benchmark Construction and Statistics

  • OmniVBench contains 813 evaluation cases across 18 sub-tasks, with references ranging from one image or video to multiple image–video combinations.Prompt semantics and length vary across categories, and reference videos span diverse durations.Source: §3.2 · pp. 6–7

3.3.1 Evaluation Capability Taxonomy

  • Evaluation separates Reference Fidelity, Instruction Realization, and Video Quality, measuring factor preservation, correct operation and target assignment, and output quality, respectively.Only relevant fidelity and instruction-realization sub-dimensions are evaluated for each case; video quality is assessed independently of reference fidelity and instruction compliance.Source: §3.3.1 · p. 7

3.3.2 Factor-Grounded Checklist Evaluation

  • Case-specific checklists instantiate relevant criteria from the instruction, references, roles, and target bindings, with separate questions for multiple factors and manual verification.RF and IR criteria are rated by a VLM; criterion scores are averaged and aggregated so active L2 sub-dimensions receive equal weight, while the three overall dimensions are averaged equally.Source: §3.3.2 · p. 7

4 Omni-R2V Dataset

Omni-R2V is a large-scale dataset built from a professionally curated video corpus to cover the seven OmniVBench task families. Its construction uses task-specific reference–target pairing and instruction generation.

4 Omni-R2V Dataset

  • The pipeline processes source videos, constructs task-specific reference–target pairs, filters candidates, and generates instructions.The central challenge is establishing relationships across heterogeneous reference types and expressing them in corresponding training instructions.Source: §4 · p. 8

4.1 Dataset Statistics

  • The dataset contains 339,570 training instances across seven task families, covering 2D animation, 3D animation, and live-action videos.Clip durations extend to 20 seconds; resolutions range from 480p to 2160p+, with 1080p and 2160p+ accounting for the majority.Source: §4.1 · p. 8

4.2 Data Construction Pipeline

  • Pair construction uses cross-pair matching or references extracted from or generated from the target, then filters candidates for task-relevant information and visual identifiability.For multi-reference tasks, references specify distinct entities or factors in the same target video; generated instructions identify each reference's intended role and target correspondence.Source: §4.2 · p. 8

5 Experiment

OmniVBench evaluates representative open- and closed-source R2V models across task families and capability dimensions. The results show comparable performance for the strongest open-source and leading closed-source models, alongside persistent cross-task and factor-level differences.

5.1 Experimental Setup

  • The evaluation compares closed-source and open-source R2V models, using MiniMax H3-Context-IR to preprocess references and instructions for MiniMax H3.The evaluated set includes six closed-source and five open-source models.Source: §5.1 · p. 9

5.2 Results

  • The strongest open-source models perform comparably to leading closed-source models, but no model is consistently strong across task families.Models generally score higher on content reference, with larger differences in motion, style, structure, narrative, and multi-reference settings.Source: Figure 6 · p. 9, §5.2 · p. 9
  • Several models score relatively high on target compliance but substantially lower on reference-factor disentanglement and routing, especially in multi-content and cross-aspect tasks.A model may follow the target instruction while copying irrelevant reference content or applying a factor to the wrong target.Source: §5.2 · pp. 9–10, Table 4 · p. 10
  • Models show distinct strengths and weaknesses across reference-fidelity dimensions; strong content fidelity does not necessarily translate to motion, structure, or narrative fidelity.The fine-grained comparison breaks Reference Fidelity into content, structure, motion, style, and narrative fidelity.Source: §5.2 · p. 9, Figure 7 · p. 10
  • Automatic evaluation agrees strongly with human judgments across RF, IR, and VQ, and factor-grounded checklists achieve higher human correlation than holistic evaluation.Human evaluation covered 100 cases across all seven task families and 18 sub-tasks, yielding 965 model outputs.Source: §5.2 · p. 10, Table 5 · p. 10, Table 6 · p. 10

6 Conclusion

The paper presents OmniVBench and the Omni-R2V Dataset as complementary resources for omni reference-to-video generation, using broad task coverage and factor-grounded evaluation to diagnose model behavior.

6 Conclusion

  • OmniVBench pairs with the Omni-R2V Dataset, which contains 340K processed training samples spanning diverse reference types and compositions.Source: §6 · p. 10
  • OmniVBench assesses reference fidelity, instruction realization, and video quality to support diagnostic analysis of fine-grained control and multi-reference composition.Source: §6 · p. 10
  • Evaluation of representative open- and closed-source models reveals strengths and limitations across diverse reference conditions.Source: §6 · p. 10

A.1.1 Detailed Omni-R2V Dataset Construction

The dataset uses task-specific construction pipelines to create reference–target pairs for varied reference factors and compositions. Its evaluation benchmark and training dataset are non-overlapping.

A.1.1 Detailed Omni-R2V Dataset Construction

  • The Omni-R2V Dataset constructs reference–target pairs through cross-pair matching or inverse construction, with operations tailored to each task’s required reference factors.The dataset and OmniVBench benchmark are non-overlapping.Source: §A.1.1 · p. 15
  • Content references match the same character across different clips, while style references pair semantically unrelated content from a shared source video to retain stylistic consistency.Source: §A.1.1 · p. 15
  • Motion references use character images animated with target action, while camera-motion references come from controlled synthesis, mined real videos, or generated trajectories.Source: §A.1.1 · p. 15
  • Structure references include line maps, greybox videos conditioned on scene geometry and target depth, and temporally ordered storyboard sketches checked for consistency and artifacts.Source: §A.1.1 · p. 15
  • Narrative references pair coherent adjacent shots, whereas multi-panel storyboards arrange target-video frames into a grid representing temporal progression and key visual states.Source: §A.1.1 · pp. 15–16
  • Multi-content references combine character, object, and scene materials; cross-aspect references pair content with style or structure references through complementary construction strategies.Source: §A.1.1 · p. 16

A.1.2 Prompt Statistics

The prompt-statistics figure summarizes prompt length and vocabulary across OmniVBench, including differences among its seven task families.

A.1.2 Prompt Statistics

  • The prompt-statistics figure presents overall word-count variation, task-wise prompt-length distributions, and vocabulary spanning subjects, scenes, attributes, actions, spatial concepts, and visual descriptions.Task-wise length differences reflect different reference configurations and instruction requirements.Source: §A.1.2 · p. 16

A.2 OmniVBench Construction Details

OmniVBench construction covers individual and compositional reference controls through task-specific sample designs. Human review and filtering check visual quality, reference clarity, and task fit.

A.2 OmniVBench Construction Details

  • All benchmark samples receive human review for visual quality, clarity of intended reference information, and consistency with task requirements, and filtering checks that reference information is identifiable but not stated in the instruction.Source: §A.2 · pp. 16–17
  • Content, object, character, and scene tasks vary references to test identity preservation, selective attribute transfer, and retention of environments while allowing instructed changes.Source: §A.2 · pp. 16–17
  • Motion tasks cover diverse subject actions and camera movements, testing transfer of action to new subjects and reproduction of viewpoint trajectories distinct from subject motion.Source: §A.2 · p. 17
  • Style references span 43 visual styles across 16 categories and test applying depicted styles to new content without copying reference subjects or scenes.Source: §A.2 · p. 17
  • Structure tasks use line art, rough storyboards, or greyboxes to test changing contours and layout, motion from sparse cues, and scene geometry alongside omitted appearance details.Source: §A.2 · pp. 17–18
  • Narrative references include story videos, multi-panel storyboards, and preceding shots, testing event order, continuous action, and shot continuity while accommodating instructed changes.Source: §A.2 · p. 18
  • Multi-content tasks combine 2–5 reference images, testing integration of characters, objects, and scenes or selection and role-binding of intended entities among distractors.Source: §A.2 · p. 18
  • Cross-aspect tasks pair content with motion, style, structure, or narrative references to test action transfer, identity and style separation, structural placement, and narrative roles.Source: §A.2 · pp. 18–19

A.3 Detailed Benchmark Composition

This section details OmniVBench’s 813 single- and multi-reference evaluation cases and its sample-relevant checklist protocol for assessing reference fidelity and instruction realization, alongside automatic video-quality components. It also presents factor-specific criteria, evidence, and human review procedures, illustrating how case-specific checklists distinguish performance.

A.3 Detailed Benchmark Composition

  • 813 evaluation cases span single-reference and multi-reference tasks, providing broad coverage of the R2V task space.Table A.1 summarizes the benchmark’s individual sub-task composition.Source: §A.3 · p. 19

A.4 Evaluation Capability and Criteria

  • A VLM-based activator selects criteria relevant to each sample, then instantiates checklist questions; Reference Fidelity uses a 1–5 scale and Instruction Realization a 1–3 scale.Video Quality bypasses checklists and uses automatic evaluation components.Source: §A.4 · pp. 19–20
  • Reference Fidelity criteria cover content, structure, motion, style, and narrative, while Instruction Realization criteria assess factor disentanglement and routing alongside target compliance.The criteria distinguish fidelity to reference factors from realizing textually specified targets.Source: §A.4 · pp. 21–22
  • Reference Fidelity scoring judges only the requested signal at relevant moments, requiring all core requirements and observable secondary aspects for a top score.The rubric cautions against requiring an action, event, transition, or pose in every frame.Source: §A.4 · p. 23
  • RF criteria judge instance identity and defining-detail consistency, while allowing permitted transformations but penalizing resulting loss of the requested signal or unverified required factors.Generic category resemblance is insufficient; defining structure and the severity of visible differences guide ratings.Source: §A.4 · pp. 23–24
  • RF assessment separately checks requested motion and camera behavior, named structural anchors, continuous completion between anchors, narrative events and relations, and specified visual style.Narrative scoring prioritizes semantic preservation, while style scoring ignores content or composition changes unless they are part of the requested style signal.Source: §A.4 · p. 24
  • IR scores each sub-question from 1–3 using visual evidence, evaluating whether instructions and factor-target assignments are realized without judging general video quality.A score of 3 means all core requirements and target assignments are satisfied; 2 indicates partial realization, and 1 indicates absence or fundamental error.Source: §A.4 · p. 24

A.5 Evidence Presentation

  • Evidence is tailored to reference roles: non-temporal references use 2-FPS generated frames, temporal references use ten matched-position frame pairs, storyboards use ordered anchors, and continuations show context first.Non-temporal sampling has a minimum of 8 and maximum of 20 frames; storyboard anchors are compared with twelve ordered generated frames.Source: §A.5 · pp. 24–25

A.6 Human Evaluation Protocol

  • Three annotators independently score each generated video for IR and RF on five-point scales using the stated criteria.The criteria are specified in Table A.5.Source: §A.6 · p. 25, Table A.5 · p. 25

A.7 More Results

  • Dataset quality review samples about 100 training examples per task family, with two annotators judging reference usability, reference–target consistency, and instruction correctness.The paper reports mean pass rates by family and criterion, including averages, and resolves disagreements through review; the supplied passage gives no pass-rate values.Source: §A.7 · p. 25, Table A.6 · p. 25

A.7.1 Qualitative Results

  • A representative multi-reference case shows RF/IR checklists capturing differences in reference fidelity, factor disentanglement and routing, and target compliance across models.Additional qualitative comparisons cover content, structure, narrative, style, and cross-aspect reference tasks.Source: §A.7.1 · p. 26, Figure A.2 · p. 26

Related papers

Loading 2609.22069v1…