PaperlayerView exact revision on arXiv

Source-linked AI summary

MultiRef-Compass: Towards Comprehensive Evaluation of Multi-Reference-to-Audio-Video Generation

Xiaohan Zhang, Yuqing Wen, Junlin Chen, Yuqi Tang, Yiting He, Lizhuo Shao, Weiming Zhu, Tengfei Liu, Yang Shi, Jialu Chen, Yuanxing Zhang, Huaxiong Li

arXiv:2607.14189v1cs.CVcs.SD

TL;DR

Unified evaluation of multi-reference-to-audio-video generation remains underexplored, despite the need to assess how models understand and compose multiple references while maintaining multimodal consistency. MultiRef-Compass introduces a benchmark and hybrid evaluation framework, and experiments reveal substantial limitations across multiple evaluation dimensions.

  • Problem

    Unified MR2AV evaluation remains underexplored, although it must assess how models understand, bind, and integrate multiple references while maintaining multimodal consistency.

    Source: Introduction · p. 1

  • Method

    MultiRef-Compass provides 350 curated samples spanning three composition settings and combines automatic metrics with rejudging-enhanced MLLM judging across four evaluation dimensions.

    Source: Introduction · pp. 1–2, Hybrid Judge Architecture · p. 4

  • Results

    Experiments show differentiated strengths across quality, reference consistency, and audio-visual consistency, while models still struggle with natural visual integration and cross-reference understanding.

    Source: Main Results · pp. 6–7

  • Takeaways & Limitations

    MultiRef-Compass provides a practical diagnostic testbed that distinguishes model capabilities and exposes failure modes not captured by existing benchmarks.

    Source: Conclusion · p. 9

    For comparability, the benchmark standardizes samples to three reference images, leaving substantially larger reference sets outside its current scope.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Multi-reference-to-audio-video (MR2AV) generation aims to generate coherent audio-video content conditioned on multiple references and textual instructions. Existing benchmarks mainly focus on text-driven generation, single-reference subject preservation, or isolated audio-video alignment, leaving the emerging MR2AV setting largely unexplored. Compared with these settings, MR2AV requires models to jointly reason over multiple references while generating synchronized visual and audio content. Models must not only preserve each reference faithfully but also correctly bind and compose multiple referenced entities into coherent audio-visual events. To address this gap, we introduce MultiRef-Compass, a unified benchmark for MR2AV generation. It comprises $350$ carefully curated samples constructed through a scalable and controllable asset-composition pipeline, covering multi-view subject preservation, multi-entity binding, and human-object-scene composition. To provide interpretable assessment, MultiRef-Compass defines an evaluation protocol with four dimensions: Basic Quality, Reference Consistency, Audio-Visual Consistency, and Instruction Following, using 14 sub-metrics. MultiRef-Compass integrates automatic metrics with a rejudging-enhanced MLLM-as-a-Judge framework, enabling scalable and auditable evaluation of both perceptual fidelity and reference-conditioned composition. Extensive experiments on eight representative MR2AV systems reveal substantial room for improvement across multiple evaluation dimensions, underscoring the need for a comprehensive benchmark and positioning MultiRef-Compass as a foundation for future MR2AV research.

Introduction

MultiRef-Compass addresses the underexplored evaluation of multi-reference-to-audio-video generation with a curated benchmark and hybrid assessment framework. It targets the challenges of binding multiple references and producing coherent audio-visual compositions, while experiments expose limitations in current systems.

Introduction

  • Existing benchmarks focus on X-to-audio-video generation or limited reference-to-video settings, leaving unified MR2AV evaluation underexplored.MR2AV evaluation must assess cross-reference understanding, compositional binding across visual and audio modalities, and natural visual integration—not reference fidelity alone.Source: Introduction · p. 1, Table 1 · p. 2
  • MultiRef-Compass contains 350 taxonomy-driven samples spanning multi-view subject preservation, multi-entity binding, and human-object-scene composition.A scalable, controllable asset-composition pipeline supports reproducible sample construction.Source: Introduction · pp. 1–2
  • Its hybrid evaluation framework combines automatic metrics with rejudging-enhanced MLLM-as-a-Judge assessment across four dimensions.The dimensions are Basic Quality, Reference Consistency, Audio-Visual Consistency, and Instruction Following; omni-reference metric routing supports image, video, and audio references alongside text instructions.Source: Introduction · p. 2
  • Experiments on eight representative MR2AV systems reveal substantial limitations across multiple evaluation dimensions.The paper presents MultiRef-Compass as a comprehensive benchmark for future MR2AV research.Source: Introduction · p. 2

Related Work

Image-conditioned video generation has expanded from single-image animation to richer multi-reference control, while existing benchmarks provide limited diagnosis of how multiple references are jointly understood and composed in audio-visual generation. MultiRef-Compass bridges these evaluation lines by jointly assessing multi-reference consistency and cross-modal alignment.

Image-Conditioned Video Generation

  • I2V advanced from single-image animation and customization, pose, identity, and localized-motion control [13] [10] [31] to spatiotemporal and trajectory guidance [28] [33] and multiple-image, heterogeneous conditioning [14] [23] [25] [16].These newer settings require models to infer relationships among references and compose them according to textual instructions.Source: Image-Conditioned Video Generation · p. 2

Reference-Conditioned Generation Benchmarks

  • Video benchmarks assess generation quality, prompt alignment, or image-conditioned subject consistency [11, 12, 18, 19, 29, 34, 36], while audio-video benchmarks evaluate quality, alignment, event-sound correspondence, and synchronization [3] [38] [32] [17].Source: Reference-Conditioned Generation… · p. 2
  • Existing benchmarks rarely distinguish same-identity multi-view references from distinct-entity compositions or diagnose how multiple references are jointly bound across multimodal conditions; MultiRef-Compass addresses this gap.Source: Reference-Conditioned Generation… · p. 2

MultiRef-Compass Dataset

MultiRef-Compass benchmarks generation of coherent audio-video content from multiple references and textual instructions, including multimodal references. Its construction controls reference types, entity combinations, audio conditions, and task difficulty, while its evaluation framework diagnoses performance across four dimensions.

MultiRef-Compass Dataset

  • The benchmark evaluates Basic Quality, Reference Consistency, Audio-Visual Consistency, and Instruction Following to diagnose failures beyond aggregate scores.Its three-stage construction comprises asset-pack curation, board-specific asset composition, and structured prompt generation.Source: Figure 1 · p. 3, MultiRef-Compass Dataset · p. 3

Data Construction

  • Asset packs combine manually verified multi-view subject references, objects, scenes, and optional video and voice references sourced primarily from royalty-free repositories.Subject packs use identities extracted from videos or 3–6 auxiliary references generated from a primary image, varying viewpoint, expression, and pose.Source: Data Construction · p. 3
  • The three main boards cover same-subject multi-view conditioning, heterogeneous subject-object-scene compositions, and multi-entity scenarios requiring interactions and role assignments.An extended fourth board tests joint conditioning on reference video and audio; its results are reported in Appendix 6.Source: Data Construction · p. 3
  • Each composition receives a structured prompt with video, audio, and speech fields, unified by GPT-5.5 and checked by human reviewers for grounding and consistency.Sample metadata records evaluation-relevant conditions so the router can select applicable metrics; the framework combines automatic metrics with rejudging-enhanced MLLM-as-a-Judge assessment.Source: Data Construction · pp. 3–4, Figure 2 · p. 4

Dataset Statistics

  • The benchmark contains 300 samples across three main boards and 50 challenge-board samples, with varied reference configurations and multimodal prompt requirements.70% use real-world references and 30% cartoon-style references; prompts average about 600 characters, with 60% specifying sound effects or ambient audio, 30% background music, 70% speech, 40% appearance constraints, and 30% multi-shot structure. These attributes are not mutually exclusive.Source: Dataset Statistics · p. 4

Evaluation Framework

MultiRef-Compass evaluates reference-conditioned audio-video generation through four dimensions, combining automatic metrics with MLLM judgments and a consistency-focused rejudging stage. Its metric suite provides diagnostics for technical quality, reference preservation and binding, audio-visual alignment, and prompt adherence.

Hybrid Judge Architecture

  • The hybrid framework combines automatic metrics and rejudging-enhanced MLLM judgments across Basic Quality, Reference Consistency, Audio-Visual Consistency, and Instruction Following.MLLM-based metrics use a 1–5 MOS scale, and the framework follows an omni-reference design for interpretable diagnostics across reference settings and modalities.Source: Hybrid Judge Architecture · p. 4

Metric Dimensions

  • Basic Quality assesses technical video and audio quality alongside anatomical plausibility, using DOVER++, Audiobox, and an MLLM checklist.Visual Technical Quality targets low-level visual defects; Audio Technical Quality covers clarity, naturalness, and audible artifacts; Anatomical Quality checks plausibility and 3D depth coherence across people, objects, and backgrounds.Source: Metric Dimensions · p. 5
  • Reference Consistency measures entity fidelity, detail preservation, and binding correctness, with an MLLM naturalness coefficient discounting evident copy-and-paste artifacts.Entity Fidelity compares detected human and object crops or full-frame scenes with references, combining face and body cues when available and matching multi-view subjects to their most similar reference view; MLLMs assess detail preservation and binding errors.Source: Metric Dimensions · p. 5
  • Audio-Visual Consistency checks speech-lip timing, event-sound matches, sound-source identity, and voice-reference timbre, while Instruction Following checks visual, audio, speech, and temporal requirements.LatentSync is applied to dialogue or monologue samples with stable frontal faces; MLLM checklists assess event sounds, source correctness, and prompt adherence, while SpeechBrain ECAPA-TDNN measures timbre similarity for Board 4 audio-reference inputs.Source: Metric Dimensions · pp. 5–6

Rejudging

  • Rejudging compares initial scores and rationales across model outputs, revisiting video evidence when needed and correcting only clear inconsistencies to improve item-level consistency and auditability.It does not add evaluation dimensions, change metric definitions, or rescore judgments that are already consistent.Source: Rejudging · p. 6

Experiments

Across eight MR2AV systems, MultiRef-Compass reveals uneven, capability-specific performance and persistent challenges in multi-reference coherence, instruction following, and synchronized audio-visual generation. Its benchmark scores align strongly with human preferences across all four dimensions.

Experimental Settings

  • The evaluation covers eight models, using 13 metrics across four dimensions on three main boards; Timbre Similarity is reserved for challenge board B4.Models receive three reference views; identical checklists and prompts are used for each sample, and Gemini 3.1 Pro serves as judge for MLLM-based metrics.Source: Main Results · p. 6, Experimental Settings · p. 6

Main Results

  • Gemini-Omni and HappyHouse 1.1 lead visual and audio technical quality, respectively; Kling 3.0 leads EF, while Seedance 2.0 leads BC and DP.Seedance also shows occasional face-region artifacts in real-person cases; SkyReels-V3 can earn inflated similarity scores despite copy-and-paste artifacts, which Ppaste discounts.Source: Main Results · p. 6, Figure 5 · p. 8, Figure 6 · p. 8
  • Gemini-Omni leads SLS, Seedance 2.0 leads ESM, and HappyHouse 1.1 leads SC; Seedance favors visual instruction following, Kling audio and speech, and Wan 2.7 temporal compliance.The differing leaders show that semantic event-audio matching and sound-source attribution, like instruction compliance across modalities, remain distinct challenges.Source: Main Results · p. 7

Further Analysis

  • Seedance 2.0 ranks first overall among closed-source models, while SkyReels-V3 outperforms Wan2.1-VACE but both open-source systems trail closed-source models substantially.Seedance's lead reflects a relatively balanced profile; open-source weaknesses are especially pronounced in Reference Consistency and Instruction Following.Source: Further Analysis · p. 7, Figure 4 · p. 8
  • Action and Wearing instructions separate closed-source systems more than Basic Setting and Detail, while open-source models perform worse across all four visual-task subdimensions.Incomplete Action and Wearing fulfillment shows that an aggregate Visual Task Following score can conceal failures on specific controllable attributes.Source: Further Analysis · pp. 7–8, Figure 4 · p. 8
  • Reference Consistency declines with more complex reference compositions, and no evaluated system is consistently strong across all four dimensions.The analysis also reports widespread anatomical-quality failures, speech without corresponding lip motion, weaker instruction following under complex constraints, and larger proprietary-model advantages in Reference Consistency and Instruction Following.Source: Further Analysis · p. 8

Human Preference Alignment

  • Benchmark win rates correlate strongly with human preferences: 0.8979 for BQ, 0.9380 for RC, 0.9217 for AC, and 0.9644 for IF.Four annotators compared outputs per dimension; preferences and benchmark scores were converted to pairwise outcomes and aggregated into model-level win rates.Source: Table 4 · p. 9, Human Preference Alignment · p. 9

Further Analysis

  • Horizontal rejudging applies a consistent semantic criterion across models and corrects overly strict or permissive judgments without changing evaluation dimensions or overall conclusions.For example, it treats picking up and placing a plant in a new location as satisfying a checklist instruction to adjust its position.Source: Further Analysis · p. 9

Limitations

MultiRef-Compass is a controlled diagnostic benchmark whose scope and evaluation tools have limitations, including reference-set constraints and sensitivity to dynamic faces. MLLM judging also adds cost and may introduce model-specific bias, while changing proprietary systems complicate reproducibility.

Limitations

  • The benchmark does not cover every creative domain, cultural style, or reference modality, and standardizes samples to three images, leaving larger reference sets outside scope.This standardization enables comparability across models with different conditioning interfaces.Source: Limitations · p. 9
  • Automatic evidence tools have limitations for cartoon subjects and faces with large pose or expression changes; SLS may favor stable frontal speakers and penalize natural facial or head motion.More robust synchronization metrics are needed for dynamic faces and complex multi-speaker scenes.Source: Limitations · p. 9
  • MLLM-based judging incurs additional cost and may inherit model-specific biases, while changes to proprietary systems and APIs complicate reproducibility.The authors report model versions and access dates to support reproducibility.Source: Limitations · p. 9

Conclusion · Appendix · A MR2AV Definition

MultiRef-Compass provides a benchmark and evaluation protocol for MR2AV generation, while the definition clarifies that the task jointly composes information from multiple reference sources rather than editing them.

Conclusion

  • MultiRef-Compass combines an asset-reuse sample-construction pipeline with automatic metrics and a rejudging-enhanced MLLM-as-a-Judge protocol.It is presented as a diagnostic testbed for future omni-reference audio-video generation systems.Source: Conclusion · p. 9
  • Its four evaluation dimensions are Basic Quality, Reference Consistency, Audio-Visual Consistency, and Instruction Following.Experiments on representative MR2AV systems show that the benchmark distinguishes model capabilities and exposes failure modes not captured by existing benchmarks.Source: Conclusion · p. 9

A MR2AV Definition

  • MR2AV synthesizes new audio-video content from multiple references and a textual instruction, with references potentially including images, videos, and audio clips.The inputs provide information to preserve or reflect in the generated content.Source: A MR2AV Definition · p. 12
  • “Multi-reference” means jointly interpreting and composing information from sources depicting different views of one entity or distinct subjects, objects, scenes, motions, and sounds.The textual instruction specifies the references’ intended roles, relationships, and interactions in the generated event.Source: A MR2AV Definition · p. 12
  • MR2AV is reference-conditioned generation, not an editing task: reference videos can provide identity, appearance, or motion cues, and audio clips can provide voice identity, timbre, or acoustic characteristics.Neither reference video nor reference audio is treated as source content to be directly edited.Source: A MR2AV Definition · p. 12

B Dataset Construction Details … B.4 Prompt Schema and Sample Records

MultiRef-Compass’s construction pipeline combines reusable assets and board-specific compositions to make multi-reference R2AV evaluation both controlled and diagnostic. Its structured prompts and recorded reference roles support reproducible assessment of fidelity, audio-visual consistency, and instruction following.

B Dataset Construction Details

  • The pipeline aims to reflect practical multi-reference R2AV workflows while keeping evaluation controlled and reproducible.It covers asset-pack collection, board construction, board-design rationale, and structured prompt generation.Source: B Dataset Construction Details · p. 12

B.1 Overview and Design Rationale

  • Reusable asset packs are recombined under board-specific rules, reducing redundant annotation and making sources of difficulty interpretable.The three core boards target multi-view subject preservation, heterogeneous reference composition, and multi-entity role binding; samples use three reference slots but contain three to seven atomic visual inputs when combined views are counted separately.Source: §B.1 · p. 12

B.2 Asset-Pack Collection

  • Subject packs combine identity-consistent images from permissively licensed or copyright-free videos with synthesized alternate views when needed.Diverse visible backgrounds discourage identity preservation through background shortcuts; audio packs span human and cartoon voice types to test speaker-type consistency.Source: §B.2 · p. 12
  • Object and scene packs provide recognizable, compatible non-subject references, while video packs supply dynamic subject references and metadata guides compatible pack matching.Object and scene cases are filtered to exclude ambiguous, occluded, or cluttered references; source videos are used as references rather than content to copy, and metadata helps avoid incompatible combinations.Source: §B.2 · pp. 12–13

B.3 Board Construction

  • Board 1 tests whether multiple views of one identity become a coherent dynamic subject without producing duplicate subjects.Its references vary in viewpoint, expression, pose, or speaking state, and evaluation considers identity and appearance over time.Source: §B.3 · p. 13
  • Board 2 tests whether subject, object, and scene references are preserved and composed coherently rather than ignored or allowed to dominate one another.Compositions include subject-object-scene mixtures, multi-view subjects with scenes, and multiple subjects in a shared scene; unlike Board 3, it does not focus on explicit role binding.Source: §B.3 · pp. 13–14
  • Board 3 tests role binding among entities, while extension Board 4 tests use of video or voice references alongside prompt-specified content.Board 3 can expose identity swaps, entity merging, attribute transfer, or misassigned actions; Board 4 accommodates limited current support for video or voice inputs and outputs without changing core board logic.Source: §B.3 · p. 14

B.4 Prompt Schema and Sample Records

  • Prompts use a difficulty- and board-dependent schema, then undergo LLM-assisted drafting and manual review.Fields may cover visual grounding, action and temporal structure, audio events, speech content, and the final instruction; the LLM drafts visual/video, audio, speech, and integrated prompts, which are refined to remove unreasonable, inconsistent, or unsafe requirements.Source: §B.4 · pp. 14–15

C Board4 Analysis · D Case Analyses

Board 4 adds audio-video references to more complex multimodal interactions, where Seedance 2.0 is more balanced across reference preservation, structural quality, and audio-reference use, while Kling 3.0 remains competitive on selected dimensions. Case analyses complement metric construction by illustrating protocol behavior across boards and how rejudging corrects inconsistent checklist judgments.

C Board4 Analysis

  • Kling 3.0 has slightly higher audio technical quality, while Seedance 2.0 leads in visual technical and anatomical quality.The higher anatomical quality score suggests better structural stability on this challenging subset.Source: C Board4 Analysis · p. 15
  • Seedance 2.0 leads on automated EF and MLLM-based binding and detail-preservation metrics, indicating more reliable preservation of referenced identities and fine-grained visual attributes.Source: C Board4 Analysis · p. 15
  • Seedance 2.0 performs better on event-sound matching and speech-lip synchronization, whereas Kling 3.0 scores higher on source correctness.Source: C Board4 Analysis · p. 15
  • Neither model reliably preserves input voice timbre on Board 4, although Seedance 2.0 achieves a relatively higher similarity score to the provided audio reference.The results indicate that current models struggle to use voice references faithfully even when generating plausible speech or audio events.Source: C Board4 Analysis · p. 15
  • Instruction-following results are mixed: Seedance 2.0 is stronger on visual and audio task following, while Kling 3.0 leads on speech content accuracy and temporal order following.Source: C Board4 Analysis · p. 16
  • Seedance 2.0 is more balanced across reference preservation, structural quality, and audio-reference use, while Kling 3.0 remains competitive on selected instruction and source-attribution dimensions.This overall pattern parallels the main benchmark, with Board 4 adding audio-video reference conditioning.Source: C Board4 Analysis · pp. 15–16

D Case Analyses

  • Case-level and board-level analyses show how the evaluation protocol behaves under different board settings and how rejudging corrects inconsistent checklist judgments.These analyses complement the metric construction details.Source: D Case Analyses · p. 16

D.1 Entity Fidelity Analysis

Entity Fidelity declines on the more complex boards, primarily because human preservation becomes harder, while stronger systems remain comparatively stable across boards.

D.1 Entity Fidelity Analysis

  • Average Entity Fidelity falls from 0.6930 on Board 1 to 0.5893 on Board 2 and 0.5748 on Board 3, driven mainly by declining Human EF.Human EF drops from 0.6930 to 0.5522 and 0.5409, while Object and Background scores stay around 0.65–0.68; complex interactions may also bias face/body similarity downward through profile faces, smaller faces, or partial occlusions.Source: §D.1 · p. 16
  • Kling and Seedance remain relatively stable across boards and rank among the top models on all three board totals.Other models show differing branch-level weaknesses, including human consistency or object and background preservation.Source: §D.1 · p. 16
  • As multi-reference composition and entity interaction grow more complex, human consistency becomes the dominant challenge for current R2AV systems.The harder boards stress human identity, face/body appearance, and temporal consistency under heterogeneous composition, interaction, occlusion, and pose variation.Source: §D.1 · pp. 16–17

D.2 Speech-Lip Synchronization Analysis · D.3 Rejudging Cases Analysis

Speech-lip synchronization scores depend on board complexity and sample filtering, while speech-content accuracy reveals language- and content-control differences that synchronization alone misses. Rejudging corrects both overly literal and overly permissive checklist judgments.

D.2 Speech-Lip Synchronization Analysis

  • Gemini-Omni retained 84 valid videos, compared with 47 for both Kling and Seedance 2.0, illustrating model-dependent sample retention after filtering.Residual motion can still affect SyncNet-based scores; more robust evaluation should handle dynamic motion, profile faces, occlusions, and other challenging speaking conditions.Source: §D.2 · p. 17
  • Strict speech-content accuracy was nearly identical for English and Chinese: 87.8% and 88.0%, respectively.Despite this aggregate similarity, model-level results showed clear language-specific behaviors.Source: §D.2 · p. 17
  • Kling scored 90.6% English and 97.6% Chinese accuracy, HappyHouse 1.1 scored 89.9% and 100.0%, and Seedance 2.0 scored 93.5% and 91.2%.Kling and HappyHouse 1.1 mainly had mild English-side errors; Seedance 2.0 showed occasional language-control and content-control errors, including wrong-language speech and additions beyond the required utterance.Source: §D.2 · p. 17
  • Gemini-Omni reached 98.1% English accuracy but 76.7% Chinese, while Wan scored 82.6% and 84.0% and Vidu 74.1% and 75.0% in English and Chinese, respectively.Gemini-Omni’s Chinese errors included substitutions, semantic drift, missing or unintelligible speech, and unrelated English; Wan and Vidu often omitted speech or generated irrelevant utterances, with occasional Chinese-to-English translation errors.Source: §D.2 · pp. 17–18
  • Board 3 is the most challenging setting, while Board 2’s sometimes-high VSLS scores warrant caution because fewer retained samples make scores sensitive to sample selection.VSLS should be interpreted alongside board type and retained sample count, not as a purely model-intrinsic score.Source: §D.2 · p. 17, Figure 11 · p. 18
  • Speech Content Accuracy measures preservation of requested language, lexical content, dialogue assignment, and utterance structure, so it should be evaluated separately from lip synchronization and audio quality.Fluent, synchronized speech can still fail to reproduce the required spoken content.Source: §D.2 · p. 18

D.3 Rejudging Cases Analysis

  • Rejudging corrected overly literal failures when picking up, carrying, or placing a plant changed its position, and lowered overly high backstage scores when evidence showed a main stage or ambiguous theater context.These cases illustrate how rejudging aligns scores with checklist intent and comparable visual evidence across models.Source: §D.3 · p. 18

D.4 Stability of MLLM-Dependent Evaluation

Repeated MLLM-dependent evaluation on 60 examples from the three main boards produced relative standard deviations below 1% across metrics and consistently small item-level MAEs.

D.4 Stability of MLLM-Dependent Evaluation

  • The complete evaluation pipeline was repeated five times on 60 randomly sampled examples using the same generated outputs and evaluation settings.EF and SLS were included because they use MLLM-assisted processing: EF for paste-naturalness calibration and SLS for pre-screening lip-sync measurements.Source: §D.4 · p. 18
  • Relative standard deviations remained below 1% across all metrics, and item-level MAEs were consistently small, supporting reliable model comparisons under repeated evaluation.The results suggest reported outcomes are not artifacts of judge stochasticity.Source: §D.4 · p. 18

E Detailed Metric Constructions and Prompts

This appendix explains how evaluation metrics are constructed, including metric-specific checklists, MLLM-as-a-Judge prompts, and automatic-metric implementation. Its instruction-following checklist uses prompt and metadata alone to assess audio tasks, exact speech, and explicit temporal structure.

E Detailed Metric Constructions and Prompts

  • The appendix covers checklist generation, MLLM-as-a-Judge prompts, and automatic-metric implementation, while omitting prompt output-format and serialization details for readability.The displayed prompt boxes show core components rather than those omitted specifications.Source: E Detailed Metric Constructions and… · p. 18

E.4 Instruction Following

  • The instruction-following checklist is fixed per sample and independent of generated videos, using only prompt and metadata without inventing unstated requirements.The same checklist is reused to evaluate videos from multiple models.Source: §E.4 · p. 27
  • Audio Task Following covers explicitly requested non-speech audio through 1–3 atomic questions, excluding speech-related criteria and marking speech-only prompts not applicable.Covered audio may include music, ambience, environmental sounds, foley, object sounds, footsteps, rustling, clicks, movement sounds, or explicitly requested silence.Source: §E.4 · p. 27
  • Speech Content Accuracy checks exact requested speech by extracting the expected speaker, quoted text, and language, and is not applicable when no exact text is specified.Vague requests such as “speaks” or “talks” do not create checklist items.Source: §E.4 · pp. 27–28
  • Temporal Order Following evaluates explicit structure only, with questions for scene segmentation, required scenes, and scene order when multiple shots or scenes are specified.The checklist also addresses explicit ordering constraints between visible events.Source: §E.4 · p. 28
  • For visible events, Temporal Order Following asks one atomic question per explicit ordering constraint, does not infer order from sentence position or include speech timing, and uses 1–3 questions when applicable.If neither multiple-shot/scene structure nor explicit action ordering applies, the category is not applicable.Source: §E.4 · p. 28

Related papers

Loading 2607.14189v1…