PaperlayerView exact revision on arXiv

Source-linked AI summary

T2AV-Compass: Towards Unified Evaluation for Text-to-Audio-Video Generation

Zhe Cao, Tao Wang, Jiaming Wang, Yanghai Wang, Yuanxing Zhang, Jiahao Wang, Jialu Chen, Miao Deng, Yubin Guo, Chenxi Liao, Yize Zhang, Zhaoxiang Zhang, Jiaheng Liu

arXiv:2512.21094v3cs.CV

TL;DR

T2AV evaluation remains fragmented across unimodal metrics and benchmarks that do not fully assess cross-modal consistency. T2AV-Compass combines broad prompt coverage with objective and subjective evaluation, and its results reveal continuing gaps in instruction following and audio realism.

  • Problem

    Unimodal metrics remain insufficient to capture audio–video consistency in timing, spatial cues, and semantics.

    Source: §2 · p. 2

  • Method

    T2AV-Compass combines objective evaluation of video, audio, and cross-modal alignment with subjective MLLM judgment of instruction following and realism.

    Source: §3.3 · p. 4, §5 · p. 9

  • Results

    Closed-source models dominate the compact-average leaderboard, while gaps are most pronounced in instruction following and audio realism; Dynamics and Sound Effects are especially challenging.

    Source: §4.1 · p. 7, §4.2 · p. 7

  • Takeaways & Limitations

    The benchmark provides fine-grained diagnostics of model capabilities and failure modes for text-to-audio-video generation.

    Source: §5 · p. 9

    The MLLM-as-a-Judge protocol has computational overhead, can reflect underlying MLLM biases, and depends on closed-source Gemini variants.

    Source: A Limitations · p. 13

Abstract

from arXiv · show

Text-to-Audio-Video (T2AV) generation aims to synthesize temporally coherent video and semantically synchronized audio from natural language, yet its evaluation remains fragmented, often relying on unimodal metrics or narrowly scoped benchmarks that fail to capture cross-modal alignment, instruction following, and perceptual realism under complex prompts. To address this limitation, we present T2AV-Compass, a unified benchmark for comprehensive evaluation of T2AV systems, consisting of 500 diverse and complex prompts constructed via a taxonomy-driven pipeline to ensure semantic richness and physical plausibility. Besides, T2AV-Compass introduces a dual-level evaluation framework that integrates objective signal-level metrics for video quality, audio quality, and cross-modal alignment with a subjective MLLM-as-a-Judge protocol for instruction following and realism assessment. Extensive evaluation of 11 representative T2AVsystems reveals that even the strongest models fall substantially short of human-level realism and cross-modal consistency, with persistent failures in audio realism, fine-grained synchronization, instruction following, etc. These results indicate significant improvement room for future models and highlight the value of T2AV-Compass as a challenging and diagnostic testbed for advancing text-to-audio-video generation.

2 Related Work

Prior evaluation work often isolates video or audio quality, while joint audio–video benchmarks still have gaps in fine-grained taxonomies and evaluation metrics. T2AV-Compass pairs broader benchmark comparison dimensions with a prompt-construction and checklist-generation pipeline.

2 Related Work

  • Unimodal benchmarks assess visual quality or audio fidelity in isolation, missing cross-modal semantic alignment and temporal synchronization central to realistic T2AV generation.VBench and EvalCrafter prioritize visual quality, while AudioCaps and AudioLDM-Eval focus on audio fidelity.Source: §2 · p. 2
  • The benchmark comparison spans prompt and event statistics, sound types, and evaluation dimensions including video quality, audio quality, cross-modal alignment, instruction following, and realism.T2AV-Compass constructs prompts from curated community sources and video inversion, then generates instruction-alignment and perceptual-realism checklists.Source: Table 1 · p. 3, Figure 2 · p. 3
  • Joint audio–video benchmarks probe open-domain generation or synchronization, but existing efforts remain limited in fine-grained taxonomies and evaluation metrics.JavisBench focuses on diverse open-domain generation; Verse-Bench and Harmony-Bench examine synchronized generation across acoustic scenarios.Source: §2 · pp. 2–3

3 T2AV-Compass

T2AV-Compass builds a diverse, complex prompt set from curated sources and real-world video inversion, then evaluates generated outputs through objective signal metrics and subjective checklist-based judgment. Its framework covers video and audio quality, cross-modal alignment, instruction following, and realism.

3.1 Data Construction

  • The three-stage construction pipeline combines taxonomy-based prompt design, multi-source collection, and real-world video inversion to build a diverse, complex evaluation set.Community prompts are deduplicated and sampled; LLM rewriting and human refinement add or filter constraints, while 100 video-inversion prompts are grounded in verified 4–10-second clips.Source: §3.1 · pp. 3–4

3.2 Dataset Statistics

  • The dataset reports category distributions across five annotation dimensions and summarizes audiovisual complexity through subject count, event structure, and audio composition.The paper also reports higher prompt token counts and greater semantic distinctiveness than concurrent datasets.Source: §3.2 · p. 4, Figure 3 · p. 4, Figure 4 · p. 4

3.3 Dual-Level Evaluation Framework

  • The dual-level framework evaluates video quality, audio quality, and cross-modal alignment objectively, alongside subjective instruction following and realism.Objective measures include text–video and audio–video semantic similarity, temporal synchronization error, and talking-face lip synchronization.Source: §3.3 · p. 4, §3.3.1 · p. 5

3.3.1 Objective Evaluation

  • Objective metrics cover low-level video integrity, visual aesthetics, audio quality, speech quality, and cross-modal consistency.Audio Aesthetic Score averages Perceptual Quality and Content Usefulness; NISQA estimates speech naturalness and intelligibility.Source: §3.3.1 · p. 5
  • Text–audio alignment uses CLAP similarity, while audio–video alignment checks semantic correspondence and temporal metrics measure event synchronization.DeSync scores the absolute onset offset, with LatentSync additionally used for talking-face lip synchronization.Source: §3.3.1 · p. 5

3.3.2 Subjective Evaluation

  • The MLLM-as-a-Judge framework uses reasoning-first, 5-point scoring across instruction-following and realism tracks to support interpretable error attribution.Instruction-following checklists operationalize prompt constraints across seven dimensions and 17 sub-dimensions; realism evaluates video plausibility and audio fidelity.Source: §3.3.2 · p. 6
  • Video realism scores motion smoothness, object integrity, and temporal coherence, while audio realism scores artifacts and material–timbre consistency.The audio metric checks whether timbre aligns with visible physical properties.Source: §3.3.2 · p. 6

4 Experiments

T2AV-Compass evaluations show that top leaderboard averages coexist with dimension-specific weaknesses: audio realism, complex dynamics, and long-horizon prompts remain difficult. Judge checks also assess agreement and ranking stability.

4.1 Main Results

  • Cascaded T2V→V2A systems can match end-to-end models on modality-specific quality, but often lag in global audiovisual alignment.Wan-2.2 + Hunyuan-Foley reaches 89.63 Video Realism; the passage attributes alignment lag to fragmented optimization.Source: §4.1 · p. 7, Table 2 · p. 7

4.2 Further Analysis

  • Dynamics is the hardest video instruction-following dimension, while Sound Effects are more error-prone than Speech/Music requirements.The analysis associates these patterns with complex motion and interactions, and with grounding physical sound events to prompts and visual events.Source: §4.2 · p. 7, Figure 6 · p. 8
  • Models show stronger visual integrity and temporal stability than audio realism, where artifacts, muffled timbres, and weak material–timbre grounding persist.The failure decomposition links the audio-realism bottleneck diagnostically to semantic matching, temporal control, and physical grounding, not as definitive causal attribution.Source: §4.2 · pp. 7–8, Figure 7 · p. 8

4.1 Main Results

  • 70.29, 69.83, 68.16, and 67.68 are the Average scores for Veo-3.1, Sora-2, Kling-2.6, and Wan-2.6, respectively, but no model dominates every dimension.LTX-2 leads open-source end-to-end models overall at 63.72 and Video Realism at 89.95; Wan-2.6 leads IF Video at 78.52 and IF Audio at 74.95.Source: §4.1 · p. 7, Table 3 · p. 8

4.2 Further Analysis

  • Failures rise from 8–10% for single subjects to 21–44% for crowds, while long-narrative prompts reach 63–80% failure across models.Wan-2.6 rises from 22% failure on single-event prompts to 63% on long-narrative prompts; LTX-2 reaches 80%.Source: §4.2 · p. 8, Figure 8 · p. 9
  • Gemini 2.5 Pro aligns most closely with human ratings on IF Video, IF Audio, and Video Realism; Audio Realism remains harder and benefits from human verification.Agreement was assessed on 50 prompts using L1 distance between human and MLLM four-dimension score vectors; lower distance is better.Source: §4.2 · p. 8, Table 4 · p. 9
  • Gemini 2.5 Pro matches the human-derived ordering of five T2AV systems in forced-choice pairwise comparisons aggregated with a Bradley–Terry model.The comparisons used Veo-3.1, Wan-2.6, PixVerse-V5.5, Ovi-1.1, and JavisDiT across the same 50 prompts.Source: §4.2 · p. 9
  • Gemini 2.5 Pro has subjective-score variability of CV ≤1.02%, while Gemini 2.5 Flash yields a five-system ranking correlation of Spearman ρ = 0.9248.The authors interpret these checks as indicating that comparative findings are not tied to one exact Gemini variant.Source: §4.2 · p. 9

5 Conclusion

T2AV-Compass combines taxonomy-driven prompts and dual-level evaluation to diagnose T2AV system capabilities and failure modes.

5 Conclusion

  • T2AV-Compass combines taxonomy-driven prompt construction with dual-level evaluation to assess video and audio quality, cross-modal alignment, instruction following, and realism.Experiments show that the benchmark differentiates model capabilities and exposes diverse failure modes in real-world settings.Source: §5 · p. 9

Impact Statement

The impact statement presents T2AV-Compass as a diagnostic benchmark for safer and more rigorous assessment of audiovisual generation.

Impact Statement

  • T2AV-Compass surfaces capability gaps such as Dynamics bottlenecks and weak material–timbre consistency, supporting more transparent assessment and robustness auditing.The statement says standardized evaluation can complement forgery-detection research amid misinformation risks from high-fidelity generation.Source: Impact Statement · p. 10

A Limitations

The authors identify evaluation cost, MLLM judge biases and dependencies, and prompt coverage as limitations, alongside planned extensions for longer videos and event-level sound attribution.

A Limitations

  • The MLLM-as-a-Judge protocol's computational overhead challenges large-scale, real-time evaluation.The authors also note intrinsic MLLM biases, reliance on closed-source Gemini variants, and limited coverage of rare physical interactions or niche artistic concepts.Source: A Limitations · p. 13

B Future Work and Insight

  • Future work plans to extend T2AV-Compass to videos longer than 10 seconds, add event-level sound-role attribution, and develop lightweight distilled evaluators.The stated aim for distilled evaluators is to reduce the cost of repeated model-development evaluation.Source: B Future Work and Insight · p. 13

C More Detailed Results

The detailed results show varied capability profiles across end-to-end models and composed pipelines, alongside diagnostic checks of audiovisual evaluation and metric independence. The appendix also reports human-annotation and MLLM-judge setup details.

C More Detailed Results

  • Tables 5 and 6 provide fine-grained comparisons across sub-dimensions, with dashes marking models that cannot generate speech.Source: Table 5 · p. 14, Table 6 · p. 14

D.1 Validation and Diagnostic Additions

  • Among 970 audio checklist questions, 307 (31.65%) test audiovisual synergy; human synchronization ratings correlate with the metric at SRCC = 0.9172.Source: §D.1 · p. 15
  • Resampling audio to 16 kHz changes four reported metrics by at most 0.0015, while sequential prompts have a higher audio failure rate than simultaneous prompts (48.8% vs. 40.7%).Across sound categories, biological sounds have the highest failure rate (50.3%), musical sounds the lowest (40.5%), and mechanical sounds are near average (42.9%).Source: §D.1 · p. 15

Proprietary End-to-End Models

  • Veo-3.1 scores highly across most dimensions, while Seedance-1.5 leads Audio Aesthetics (AA 7.403), A-V Alignment (0.2875), and Lip Sync (1.560).PixVerse-V5.5 leads Temporal Synchronization (DS 0.6627) and Speech Quality (SQ 1.824); other proprietary models show different mixes of strengths.Source: Proprietary End-to-End Models · p. 15

Open-Source End-to-End Models

  • LTX-2 has lower visual fidelity (VT 7.16) but competitive audio generation among open-source models; JavisDiT does not support speech generation.Ovi-1.1 performs consistently across evaluated dimensions, while JavisDiT also scores lower on visual-quality metrics.Source: Open-Source End-to-End Models · p. 15

Composed Generation Pipelines

  • Wan-2.2 + Audio Modules has the highest Visual Technical score (VT 13.43) but high synchronization error (DS ≈0.89), whereas AudioLDM2 + MTV leads Text-Audio alignment (0.2698) with lower video-quality scores.Source: Composed Generation Pipelines · p. 15

D.3 Metric Independence and Redundancy

  • After de-meaning per-sample metric z-scores to remove overall model capability, most metric pairs have absolute Pearson correlations below 0.3, indicating limited redundancy.Figure 9 presents the processed-metric correlation heatmap.Source: §D.3 · p. 16, Figure 9 · p. 16

D.4.1 Experimental Setup

  • The setup evaluates five T2AV models on 50 prompts, yielding 250 samples; five trained annotators rate instruction following and realism, while MLLMs use checklist or direct-assessment prompts.The human rubric uses a 5-point scale, with exemplar videos and training sessions to support consistent interpretation.Source: §D.4.1 · pp. 16–17

D.4.2 Comprehensive Results

The agreement analyses find that MLLM performance varies by dimension and model family. Human pairwise comparisons also produce a clear ranking among five systems.

D.4.2 Comprehensive Results

  • Gemini 2.5 Pro has the lowest overall agreement distance (L1 1.087) and is 1.1% worse than humans on Video Realism, while Qwen3-Omni is 107% worse than the human baseline on IF Audio.Source: §D.4.2 · p. 17, Table 6 · p. 17
  • Gemini variants agree with each other more closely (L1 0.702) than humans do (L1 0.949), while cross-family MLLM pairs disagree more (L1 1.322–1.476).Source: §D.4.2 · p. 17, Table 7 · p. 18

D.5.1 Experimental Design

  • Arena50 contains 50 prompts rendered by five T2AV systems, and trained annotators compare model pairs on instruction following and realism.Source: §D.5.1 · p. 18

D.5.2 Ranking Methodology

  • The Bradley–Terry model estimates latent strengths from pairwise win counts; these parameters are converted to Elo scores using OVI1.1 as the 1500-point baseline.A 400-point Elo gap corresponds to a predicted win probability of 90.9%.Source: §D.5.2 · p. 18

D.5.3 Complete Results

  • Human-derived rankings put Veo-3.1 first (Elo 2607.2; 89.45% win rate) and JavisDiT last (209.2); Veo3.1 wins at least 76.5% against any opponent, while Javisdit wins under 2% against all.PixVerse-V5.5 is mid-tier at Elo 1804.9; its matchup with OVI-1.1 is the most competitive, with 69.0% of wins for PixVerse-V5.5.Source: §D.5.3 · p. 18

D.6 Upper Bound Analysis of MLLM-as-a-Judge

Calibration against real-world videos shows large score gaps from current state-of-the-art systems on relations and dynamics, while revealing a possible auditory-judgment caveat.

D.6 Upper Bound Analysis of MLLM-as-a-Judge

  • On 100 real-world clips, the judge scores real videos above state-of-the-art outputs, including Relations (97.44 vs. 77.81) and Dynamics (90.41 vs. 65.14).The passage reports that these margins demonstrate checklist sensitivity to complex interaction logic and temporal motion consistency; Figure 10 compares SOTA and real-video scores.Source: §D.6 · pp. 18–19, Figure 10 · p. 19
  • Real-world clips exceed 90 in most instruction-following dimensions, supporting the authors’ view that checklist requirements are attainable.Source: §D.6 · p. 19
  • The real-video ceiling is lower for Sound Effects (83.50), which may reflect judge sensitivity to subtle environmental noise even in authentic recordings.Source: §D.6 · p. 19
  • Aesthetics scores are close for real videos and SOTA models (84.42 vs. 84.83), while larger gaps remain in other dimensions, including structural and logical alignment.Source: §D.6 · p. 19

E MLLM Evaluation Framework

The framework evaluates visual and audio instruction following with modality-specific metrics and assesses realism across five dimensions.

E.1.1 Score Normalization

E.1.2 IF-Video Calculation

  • IF-Video averages non-audio major-category scores, with each category score itself averaging its sub-dimension scores.Categories include visual dimensions such as Object, Attribute, Motion, and Scene.Source: §E.1.2 · p. 20

E.1.3 IF-Audio Calculation

  • IF-Audio combines normalized Speech, Sound Effects, and Music scores with a penalty for unwanted speech.The penalty is based on NonSpeech compliance and applies when speech appears where it should not.Source: §E.1.3 · p. 20

E.3 Realism

  • Realism assessment uses TCS, OIS, MTC, MSS, and AAS to cover temporal coherence, object integrity, material–timbre consistency, motion smoothness, and acoustic artifacts.Source: §E.3 · p. 21

F Experimental Setup

Experimental settings retain each system’s native generation configurations while specifying judge and synchronization-analysis settings.

F Experimental Setup

  • Video generation uses each system’s default frame rate and audio sampling rate; most systems output at 720p, Javis at 480p, and Kling-2.6 at 1080p.Source: F Experimental Setup · p. 21
  • Gemini-2.5-Pro paraphrases prompts for Ovi-1.1-based and composed-generation systems to align text with their pipeline requirements.Source: F Experimental Setup · p. 21
  • The technical quality evaluation framework is organized into five scoring dimensions and sub-dimensions.Source: Table 12 · p. 22
  • The MLLM judge uses deterministic decoding and samples visual frames at 2 FPS; synchronization analysis uses desync with offset_sec -2.0.Source: F Experimental Setup · p. 22

H.1 Crawl From Existing Data

This appendix section documents the construction and structuring of T2AV-Compass prompts and real-video references, including instructions and examples for rewriting audio-video into English prompts and captioning videos. It also specifies checklist-based binary QA and MLLM judging templates, criteria, scoring rules, and omission conditions for assessing instruction following and visual and audio realism.

H.1 Crawl From Existing Data

  • T2AV-Compass aggregates prompts from diverse sources and uses semantic clustering with inverse-square-root sampling to reduce common-topic dominance and surface niche scenarios.Candidate prompts are encoded with all-mpnet-base-v2 and deduplicated at a cosine similarity threshold of 0.8 before sampling by cluster size.Source: §H.1 · pp. 22–23

H.2 Real Video Collection

  • The YouTube reference set provides reproducible, high-fidelity clips spanning 5–10 seconds, 16:9, and at least 720p, from simple events to multi-step narratives.Videos were documented with source URLs and timestamps; post-processing unified resolution to 720p.Source: §H.2 · p. 23
  • Reference clips emphasize audiovisual complexity, including 1–4 sound effects, off-screen audio, human speech in 30%, and camera dynamics in 70%.The collection also reports a seven-domain thematic distribution: Modern Life & Drama (25%), Documentary & History (23%), Fantasy (17%), Sci-Fi (14%), Animation (14%), Horror & Humor (14%), and Commercial & Promotion (13%).Source: §H.2 · p. 23

I.1 Rewrite Prompt

  • The rewriting prompt prioritizes fidelity to a user's original creativity, theme, and emotion, while prohibiting resolution, aspect-ratio, or duration specifications.It selectively integrates subject, scene, motion, cinematography, style, and sound into one fluent paragraph.Source: §I.1 · pp. 24–25

I.2 Video Caption Prompt

  • The video-caption prompt asks for an accurate description rather than invention, selectively combining six modules and excluding API-controlled resolution, aspect ratio, and duration.Core modules are subject, scene, motion, and sound; cinematography and stylization are optional, and the result should be one coherent paragraph.Source: §I.2 · pp. 26–27
  • The video-caption prompt illustrates richly detailed English scene descriptions and directs the model to rewrite user-provided raw audio-video accordingly.Examples combine subject and setting details with camera choices, sound design, and dialogue.Source: §I.2 · p. 27

I.3 Checklist Extraction Prompt

  • Checklist extraction covers aesthetics, attribute, cinematography, and dynamics through prompt-specific visual criteria, including style, mood, appearance, quantity, lighting, framing, color grading, and motion.It calls for one objective binary question per applicable sub-dimension, with an expected “Yes”; unspecified features are set to null.Source: §I.3 · pp. 27–29
  • Dynamics QA targets temporal motion, interactions, transformations, and camera movement, distinguishing camera motion from static camera positions.Questions should concern processes during playback, and a question is omitted when the prompt does not specify that sub-dimension.Source: §I.3 · pp. 30–31
  • The Relations dimension is framed as evaluating how visual elements relate to one another.Source: §I.3 · p. 31
  • The checklist generates at most one expected-Yes binary question per mentioned spatial or logical relation, and omits questions for unmentioned relation types.Spatial relations cover object positions; logical relations cover composition, comparison, and inclusion.Source: §I.3 · p. 31
  • For sound effects, speech, and music, the checklist asks at most one objectively audible expected-Yes binary question per mentioned category, using null when a category is absent.Questions can target ambient or action sounds, speech content or voice characteristics, and music style or instruments.Source: §I.3 · p. 32
  • The world-knowledge checklist tests objectively verifiable inherent characteristics of named entities with one expected-Yes binary question, omitting questions when the prompt lacks relevant facts.It directs questions toward proper nouns and their factual attributes rather than difficult-to-verify scene locations.Source: §I.3 · p. 33

I.4 MLLM Judge-IF Prompt

  • The instruction-following judge rates video completion on a 1–5 Likert scale, from strongly incomplete to fully complete, and returns a score with an explanation.The required output is a JSON object containing a reason and integer score.Source: §I.4 · pp. 33–34

I.5 MLLM Judge-Realism Prompt

  • The smoothness judge distinguishes generation artifacts from plausible motion blur and adjusts blur tolerance to whether a scene is dynamic or static.It assesses flicker, tearing, frame-to-frame fluidity, and optical-flow consistency.Source: §I.5 · p. 34
  • The realism prompts score motion smoothness for artifacts, temporal fluidity, and scene-aware blur tolerance, and object integrity for anatomical plausibility, rigid-body stability, and consistency across frames.MSS uses a 1–5 scale; OIS examines structural integrity during motion.Source: §I.5 · pp. 34–35
  • Structural realism is scored from anatomical and rigid-body integrity, ranging from severe deformation to stable structure throughout motion.The scale also treats proportion problems and detail corruption as intermediate failures, while allowing tiny contour jitter in very fast motion or near occlusion boundaries at the good level.Source: §I.5 · p. 35
  • Temporal Coherence Score (TCS) evaluates whether objects persist and retain identity, distinguishing abnormal disappearance, appearance, flicker, or identity changes from plausible exits and occlusions.The 1–5 scale runs from frequent incoherence or major object loss to strong object permanence and behavior consistent with occlusion.Source: §I.5 · p. 36
  • Acoustic Artifact Score (AAS) judges technical audio fidelity independently of audio content and visuals, checking artifacts, temporal stability, and signal integrity.The rubric considers metallic tones, smearing, bandwidth loss, pops or dropouts, noise-floor changes, clipping, and hallucinated noise, with scores from 1 to 5.Source: §I.5 · pp. 36–37
  • Material–Timbre Consistency (MTC) compares sound-source materials and interactions with their audible textures, while checking whether reverberation fits the depicted environment.Its 1–5 scale ranges from severe material or spatial mismatch to convincing texture, weight, and scene-matched acoustics.Source: §I.5 · pp. 37–38

J Qualitative Case Studies

Two qualitative cases illustrate performance gaps on prompts requiring models to coordinate cinematic atmosphere or fine-grained visual details with audio cues.

J Qualitative Case Studies

  • The qualitative comparisons use representative models to make performance gaps intuitive.Source: J Qualitative Case Studies · p. 39
  • Veo-3.1 and Wan-2.5 handle the monk prompt’s lighting, auditory atmosphere, and physical reflections better than other baselines.The prompt specifies soft morning light and faint, distant cicada sounds.Source: J Qualitative Case Studies · p. 39, Figure 12 · p. 40
  • Veo-3.1 captures the honeybee’s shallow depth of field and texture, while other models struggle with the extreme close-up perspective.The comparison highlights its fuzzy texture and glistening liquid details alongside audio-visual alignment.Source: J Qualitative Case Studies · p. 39, Figure 13 · p. 41
Loading 2512.21094v3…