PaperlayerView exact revision on arXiv

Source-linked AI summary

AVBench: Human-Aligned and Automated Evaluation Benchmark for Audio-Video Generative Models

Jialiang Yang, Bin Xia, Ruihang Chu, Dingdong Wang, Wanke Xia, Zhun Mou, Tianyang Zhong, Yiting Zhao, Wenming Yang

arXiv:2605.24652v1cs.AIcs.CVcs.MMcs.SD

TL;DR

Audio-video generation has advanced, but evaluation remains limited in human-centric detail and specialized assessment of subtle cross-modal errors. AVBench combines fine-grained metrics with specialized evaluators that produce continuous confidence-based scores. Experiments report close alignment with human judgment, and the framework is positioned for systematic evaluation and optimization, though its current focus is short clips.

  • Problem

    Existing AV-generation benchmarks and generic evaluators provide limited fine-grained human-centric assessment and are not specifically trained to detect subtle cross-modal inconsistencies.

    Source: §1 · pp. 2–3

  • Method

    AVBench combines human-centric evaluations with specialized SFT evaluators that derive continuous alignment scores from normalized Yes/No prediction probabilities.

    Source: §1 · p. 3, §3.3 · p. 9, §3.4 · p. 10

  • Results

    AVBench’s evaluations align closely with human judgment across difficulty levels in core human-centric scenarios.

    Source: §1 · p. 4

  • Takeaways & Limitations

    AVBench is presented as a proxy for systematic evaluation and optimization, with continuous differentiable scores offering potential as a scalable reinforcement-learning reward signal.

    Source: §6 · p. 15

    AVBench currently focuses primarily on short-form video clips of 5–12 seconds.

    Source: §5 · p. 14

Abstract

from arXiv · show

Rapid advances in audio-video (AV) generation have enabled high-fidelity synthesis with synchronized sound, particularly for human-related scenarios involving speech and interactions. Yet evaluation for AV generation remains at an early stage, with only a few coarse-grained benchmarks for human-related scenarios and relying on limited preset evaluations with generic multimodal LLMs, leading to inaccurate assessments of model capabilities. To address these issues, we introduce AVBench, a fully automated benchmark tailored for human-centric AV generation. AVBench is built on two key designs for comprehensive and accurate evaluation: (i) Human-centric and fine-grained metrics. AVBench integrates ten evaluation dimensions designed for human-centered real-world scenarios, covering visual quality, audio quality, and multi-level consistency across modalities. These practical metrics capture human-related details that existing benchmarks often overlook. (ii) Specialized evaluators via preference learning. To address the lack of specialized training data, we construct large-scale supervision by transforming real-world videos into diverse training pairs with controlled perturbations. After fine-tuning on this high-quality dataset, the evaluators learn to reliably detect subtle cross-modal inconsistencies. Crucially, instead of producing discrete textual judgment, AVBench derives continuous evaluation scores from the model's prediction confidence on binary decisions. This probabilistic scoring mechanism enables a more reliable assessment than traditional VQA-style evaluation and aligns closely with human judgment. Taken together, AVBench offers automated evaluation for AV generation, demonstrates strong potential for data filtering, and serves as a differentiable reward signal for Reinforcement Learning from Human Feedback (RLHF).

1 Introduction

AVBench addresses gaps in evaluating human-centric audio-video generation with fine-grained metrics and specialized evaluators. Its continuous scores and difficulty-tiered test set support detailed assessment, and experiments report alignment with human judgment.

1 Introduction

  • Existing benchmarks and evaluators lack fine-grained human-centric assessment, specialized AV evaluation, and continuous scoring for subtle cross-modal inconsistencies.Single-modality video benchmarks neglect synchronization, while generic multimodal models are not specifically trained for AV evaluation.Source: §1 · pp. 2–3
  • AVBench adds evaluations of cross-modal consistency, speech-related factors, and video perceptual quality for human-centered scenarios.Its dimensions target perceptual factors in common human-centric settings.Source: §1 · p. 3
  • Specialized evaluators trained with hard negatives produce continuous confidence-based scores for more granular assessment than discrete outputs.The training includes sub-second temporal shifts, emotional mismatches, and LLM-driven semantic mutations.Source: §1 · p. 3
  • AVBench uses Normal and Hard test subsets to assess model performance in everyday and more challenging scenarios.The hierarchical structure is intended to reveal weaknesses across difficulty levels.Source: §1 · p. 3
  • AVBench achieves strong alignment with human judgment while assessing model capabilities across difficulty levels in core human-centric scenarios.The introduction reports this as the outcome of extensive experiments.Source: §1 · p. 4

2 Related Work

Native audio-video generation moves beyond cascaded video-then-audio systems toward unified synthesis, improving global coherence while leaving fine-grained control challenging. AVBench’s specialized evaluator training and curated test prompts respond to evaluation gaps left by existing paradigms.

2.1 Evolution of Native audio-video Generation

  • Cascaded T2AV systems generate video first and audio second, a separation associated with temporal misalignment and semantic dissonance.The audio generator lacks access to the video generation process’s latent dynamics.Source: §2.1 · p. 4
  • Native AV models jointly synthesize modalities and improve global coherence, but identity consistency and lip-sync remain difficult in complex multi-talker scenarios.The passage describes unified spatiotemporal-audio patches, hybrid attention, and synchronization of motion with audio cues across recent systems.Source: §2.1 · p. 4

2.2 Evaluation Paradigms for Audio-Video generation

  • Existing evaluation ranges from zero-shot single-modality metrics to VQA pipelines, which can be costly, discrete, and insufficiently human-centric.AVBench instead uses specialized SFT evaluators and continuous probabilistic scoring; its construction also separates evaluator training from test-set curation.Source: §2.2 · p. 5, Figure 2 · p. 5

3 AVBench

AVBench combines a human-centric test set with specialized evaluators trained on hard negatives. Its curation separates evaluator-training clips from manually verified prompts and stratifies evaluation by difficulty.

3 AVBench

  • AVBench has two pillars: a diverse human-centric test set and specialized evaluators SFT-trained on a 300K-sample dataset with multidimensional hard negatives.The dataset’s normal and hard subsets span language, speaker count, interaction complexity, emotional expression, and camera shot type.Source: §3 · pp. 5–6, Figure 3 · p. 6

3.1 Human-Centric Data Curation

  • The training corpus uses 30K human-centric real-world clips with dense annotations, while the independent test set has 470 manually verified prompts and leakage safeguards.Test prompts are quota-sampled, deduplicated against training clips, and rewritten to remove training-specific artifacts.Source: §3.1 · p. 6
  • The Normal subset covers stable everyday settings with 1–2 speakers, while the Hard subset includes noisy, occluded, overlapping, or emotionally intense interactions with 3–4 speakers.The test set is stratified by environmental and interaction complexity.Source: §3.1 · p. 7

3.2 Multi-Dimensional Hard Negative Mining

  • Hard-negative mining targets broad semantic shifts and subtle misalignments; textual negatives use balanced perturbation dimensions, minimal LLM edits, and sequence-matching validation.Edits change 1–3 words while retaining 90%–95% of the original structure; accepted samples have character-level similarity from 0.70 to 0.995.Source: §3.2 · p. 7

Automated Hard Negative Generation Pipeline.

AVBench constructs targeted hard negatives across audio-video, video-text, and audio-text alignment, then fine-tunes specialized evaluators to detect subtle inconsistencies and produce continuous scores.

Audio-Video (AV) Misalignment:

  • AV negatives test temporal synchronization, acoustic distortions, and semantic or environmental mismatches, including shifts of 0.2–3.0 seconds and conflicting speaker or ambient-sound cues.Perturbations include playback-speed and pitch changes, frequency filtering, overlapping speech, and mismatches such as indoor visuals paired with outdoor rain.Source: Audio-Video (AV) Misalignment · pp. 7–8

Video-Text (VT) Misalignment:

  • VT negatives target human-centric appearance, age, gender, emotion, and interaction, as well as spatial relations, state transitions, and motion trajectories.The latter perturbations address structural hallucinations in diffusion models.Source: Video-Text (VT) Misalignment · p. 8

Audio-Text (AT) Misalignment:

  • AT negatives test speech attributes and content, action–sound alignment, sound effects, acoustic environment, temporal structure, and world knowledge.Examples include identity inconsistency, logical reversals, and acoustic textures that conflict with described physical events.Source: Audio-Text (AT) Misalignment · pp. 8–9

3.3 Evaluator Model Optimization via Supervised Fine-Tuning

  • The evaluators learn alignment decisions from multimodal inputs paired with Yes/No labels, while the normalized probability of Yes supplies a continuous evaluation score.The training setup uses specialized multimodal backbones, with the LLM fine-tuned for AV and VT and both the LLM and connector layers fine-tuned for AT.Source: §3.3 · p. 9

3.4 The Comprehensive Evaluation Suite

AVBench combines cross-modal alignment and synchronization measures with independent audio and video quality measures across ten evaluation dimensions.

  • The suite has two modules: cross-modal alignment and synchronization, and unimodal generation quality, spanning ten dimensions.The cross-modal module assesses semantic alignment and temporal synchronization; the unimodal module assesses audio and video streams independently.Source: §3.4 · pp. 10–11
  • Specialized evaluators score audio-text, video-text, and audio-video consistency, while SyncNet alignment confidence and temporal-offset analysis assess lip-sync fidelity.The lip-sync measure is implemented through the LatentSync framework.Source: §3.4 · p. 10
  • Unimodal measures cover speech content accuracy, speech realism, audio quality and aesthetics, and video technical quality and aesthetics.Speech content uses transcription with keyword-completeness, lexical-accuracy, and hallucination components; audio quality uses NISQAv2 MOS, and video technical quality uses DOVER++.Source: §3.4 · pp. 10–11

4 Experiments and Analysis

AVBench comparisons across five T2AV models show performance differences that are especially apparent on the Hard Split. Video-text consistency is the main cross-modal bottleneck across models.

4.1 Main Results: Evaluating State-of-the-Art T2AV Models

  • Performance differences among the five T2AV models are particularly clear under the Hard Split conditions.Table 2 reports cross-modal, lip-sync, speech, audio, and video-quality measures across Normal and Hard splits.Source: §4.1 · p. 11, Table 2 · p. 12

Cross-Modal Alignment & Synchronization.

Unimodal Generation Quality

Across quality dimensions, T2AV models show distinct strengths, while specialized SFT evaluators outperform generic and zero-shot baselines on hard negatives. AVBench’s automated scores also correlate strongly with human preferences on several dimensions.

Unimodal Generation Quality

  • Kling 2.6 leads lip-sync scores, Wan 2.6 leads speech-content scores but has the lowest vocal realism, and Sora 2 produces the most natural voices.The results show that lip-sync, content accuracy, and vocal realism do not move together.Source: Unimodal Generation Quality · p. 12
  • Seedance 1.5 Pro leads in NISQA, Audiobox, and DOVER++, while Kling 2.6 has the highest video-aesthetic scores across both splits.These are distinct technical-quality and visual-appeal strengths.Source: Unimodal Generation Quality · p. 12
  • Across models, VT instruction-following remains a challenge, especially on the Hard Split, and technical quality or aesthetics do not compensate for limited realism and semantic depth.Source: Unimodal Generation Quality · p. 12

4.2 Effectiveness of the SFT Evaluator

  • Specialized SFT evaluators reach 0.8437 on Audio-Text, 0.9144 on Video-Text, and 0.9817 on Audio-Video hard-negative accuracy.Zero-shot CLAP reaches 0.4888 on Audio-Text and ImageBind 0.4991 on Audio-Video; prompted Qwen2-Audio reaches 0.2500 on Audio-Text.Source: §4.2 · p. 13, Figure 5 · p. 13

4.3 Human Alignment Validation

  • Automated scores correlate with human win ratios at ρ = 0.9488 for AT consistency, ρ = 0.9653 for VT consistency, and ρ = 0.9779 for Speech Content.DF Arena Bonafide, NISQA MOS, and SyncNet Score correlations are 0.9668, 0.8012, and 0.8194, respectively.Source: §4.3 · pp. 13–14, Figure 6 · p. 14

5 Limitation

AVBench currently focuses primarily on short-form video clips of 5–12 seconds, matching the default output lengths of some state-of-the-art models.

  • AVBench primarily targets 5–12-second clips; extending the framework to longer videos is future work.The paper proposes using the same methodology to build longer-video training data and evaluation benchmarks.Source: §5 · p. 14

6 Conclusion

The paper presents AVBench as an automated framework for human-centric audio-video generation, using specialized SFT and fine-grained evaluation. It concludes that the framework captures subtle cross-modal misalignments and yields continuous scores that correlate with human judgment.

  • AVBench assesses model capabilities across ten dimensions, combining semantic consistency, signal-level quality, and aesthetics.Its specialized SFT pipeline is trained on 300K samples with multi-dimensional hard negatives.Source: §6 · pp. 14–15
  • The framework derives continuous, differentiable scores from model confidence and demonstrates potential as a scalable reward signal for reinforcement-learning-based alignment.Source: §6 · p. 15
  • AVBench captures subtle cross-modal misalignments that traditional metrics often overlook and correlates strongly with human judgment.The paper presents it as a proxy for systematic evaluation and optimization of audio-video generation systems.Source: §6 · p. 15

7 Extended Details on Negative Sample Construction

AVBench constructs fine-grained negative samples by perturbing audio-video tracks and text descriptions, then organizes mismatches into modality-specific taxonomies and illustrates their construction.

7.1 Negative Sample Construction Pipeline

  • Audio-video negatives span eight levels, including semantic conflicts, sub-0.1-second and 0.5–2-second shifts, speed and pitch changes, added noise, and spectral filtering.Source: §7.1 · p. 18, Figure 7 · p. 19
  • Audio-text prompts alter target attributes while preserving structural markers and unrelated content, and require physically and logically plausible descriptions.Source: §7.1 · p. 18, Figure 8 · p. 20
  • Video-text prompts limit changes to 1–3 words, preserve grammatical structure, and require substitutions from the same semantic category; generated negatives are manually checked.Source: §7.1 · pp. 18–19, Figure 9 · p. 20

7.2 Detailed Taxonomy of Negative Samples

  • The taxonomy covers audio-video temporal and semantic alignment, video-text appearance and reasoning, and audio-text speech attributes and cultural cues; about 25% of samples emphasize temporal and semantic alignment.Source: §7.2 · p. 19, Table 3 · p. 21, Table 4 · p. 21

7.3 Negative Sample Examples

  • A video-text example replaces “woman” with “boy,” while an audio-text example changes a distressed male voice in silence to a calm voice with background music.The examples preserve grammatical structure for the video-text mismatch and physical plausibility for the audio-text mismatch.Source: §7.3 · p. 23

8 Extended Details on the Evaluation Dataset

The evaluation dataset separates normal and hard examples, with the hard subset emphasizing denser interactions, more difficult acoustic and visual settings, and varied human attributes.

8 Extended Details on the Evaluation Dataset

  • The evaluation dataset contains a Normal Subset of 350 samples and a Hard Subset of 120, with distribution shifts across six dimensions.Source: §8 · p. 23, Figure 12 · p. 25

8.1 Linguistic and Interaction Complexity

  • The Hard subset has more multi-speaker scenes (60.8% with at least three speakers) and frequent overlap or interruptions (47.5%), versus 48.3% single-speaker scenes and 90.3% without overlap in the Normal subset.The Hard subset also emphasizes multi-turn coherent interactions (36.7%) and complex multi-person overlap (25.8%).Source: §8.1 · p. 23

8.2 Acoustic and Visual Environmental Challenges

  • The Hard subset includes more fast or very fast speech (26.6%), night or evening scenes (53.3%), crowd noise (22.5%), and outdoor ambience (18.3%).Daytime scenes are more prevalent in the Normal set (42.0%), where clean audio accounts for 45.4%.Source: §8.2 · p. 25

8.3 Character Attributes and Emotional Breadth

  • The Hard subset emphasizes tense (25.8%) and excited (14.2%) emotions, while both subsets include varied human attributes and vocal textures.The Normal set has a broader distribution of serious and happy states.Source: §8.3 · p. 25

9 Additional Experimental Results

AVBench’s automated metrics agree with human preferences across seven objective dimensions, while its fine-tuned evaluator outperforms baselines on multimodal consistency. Qualitative cases illustrate its ability to detect audio-text, video-text, and audio-video mismatches that baseline models miss.

9.1 Human Preference Prediction Accuracy

  • For same-prompt video pairs, human consensus defines the preferred video, and a prediction is accurate only if it scores that video strictly higher.The section evaluates agreement with experts in direct side-by-side comparisons (2AFC).Source: §9.1 · p. 26
  • 85.4% average accuracy across seven objective dimensions, peaking at 98.1% for Speech Content.Accuracy measures how often the automated metric scores the human-preferred video higher in a pairwise comparison.Source: Figure 13 · p. 26, §9.1 · p. 26
  • 92.31% accuracy for the SFT-trained model on Video-Text consistency, versus 47.44% for the base Qwen model.The comparison covers Audio-Text, Video-Text, and Audio-Video consistency.Source: §9.1 · p. 26, Figure 14 · p. 27

9.2 Qualitative Case Studies

  • In the audio-text case, the SFT evaluator detects when a little girl speaks dialogue meant for two young women, unlike CLAP and base Qwen Audio2 7B.It penalizes Audio B and matches the human preference for Audio A.Source: §9.2 · p. 27
  • In the video-text evaluation, the prompt specifies “four people” in a vehicle, but Video A shows five; ViCLIP and base Qwen Omni2.5 7B overlook the mismatch, while the proposed model captures and penalizes it, matching human judgment.Source: §9.2 · p. 27
  • In the audio-video case, the SFT evaluator detects a noticeable delay and prefers synchronized Video B, while ImageBind and base Qwen Omni2.5 7B score the delayed video higher.Across the cases, the evaluator captures consistency errors that traditional metrics miss.Source: §9.2 · pp. 27–28

Related papers

Loading 2605.24652v1…