PaperlayerView exact revision on arXiv

Source-linked AI summary

VA-Judger: Reward Modeling from Human Preference Feedback for Joint Video-Audio Generation

Yinming Huang, Shuyuan Tu, Xi Yan, Zihan Yang, Jianhua Han, Xu Hang, Yu-Gang Jiang, Zuxuan Wu

arXiv:2608.18607v2cs.CV

TL;DR

Separate reward metrics miss the holistic coherence among text, video, and audio needed for human-aligned post-training. The paper introduces VAPref-10K, VA-Judger-Bench, and VA-Judger’s progressive human-grounded training pipeline, which improves preference alignment and generation quality.

  • Problem

    Separate audio, visual, and synchronization metrics do not capture the holistic semantic and temporal coherence underlying human preferences for joint video-audio generation.

  • Method

    The paper constructs human-preference data and trains VA-Judger progressively with easy-pair cold-starting, human-verified hard-pair alignment, and dimension-wise reinforcement learning.

  • Results

    VA-Judger better aligns with human judgments than existing omni-models in in-domain and out-of-domain evaluations, while its rewards improve generation quality.

  • Takeaways & Limitations

    Human-grounded rewards provide a practical signal for post-training joint video-audio generation beyond narrow expert metrics.

  • Takeaways & Limitations

    Metric-based evaluators remain limited because they observe isolated dimensions or global similarity rather than complete text-video-audio coherence.

Abstract

from arXiv · show

Using reinforcement learning to post-train joint video-audio generation models requires a reward signal. Existing methods construct this reward by combining metrics for individual quality dimensions, including audio quality, visual fidelity, and synchronization. However, these metrics evaluate perceptual dimensions separately and fail to capture the overall semantic and temporal coherence among the text prompt, video, and audio that shapes human preferences. Optimizing models against these metrics encourages reward hacking, generating video-audio content that achieves high scores on these metrics yet appears incoherent or unfaithful to human viewers. To address this problem, we first construct a large-scale human-preference dataset VAPref-10K for joint video-audio generation, comprising 9K prompts and 10.3K fine-grained paired comparisons from open-source generation models. We also introduce the VA-Judger-Bench benchmark with both in-domain and out-of-domain model comparisons to evaluate whether reward models truly align with human preferences. We further propose VA-Judger, a chain-of-thought omni-reward model for joint video-audio generation. In particular, VA-Judger first learns from pairs with clear quality gaps to establish structured output and coarse preference discrimination, then distills reliable preference explanations for harder near-quality comparisons via rejection sampling verified against human annotations, and finally performs dimension-wise reinforcement learning that decomposes human feedback into individual quality dimensions for denser reward signals than a single binary preference label. Experiments show that VA-Judger outperforms metric baselines in predicting human preferences on both in-domain and out-of-domain evaluations. Using its human-aligned rewards for post-training audio-video generation model also yields significant improvements in generation quality.

1 INTRODUCTION

Joint video-audio generation needs reward signals that reflect holistic human preferences, because separate modality metrics miss cross-modal coherence. The paper addresses this gap with human-preference data, VA-Judger, and a benchmark for evaluating alignment.

  • Separate audio, visual, and synchronization metrics fail to capture holistic text-video-audio coherence that shapes human preference.
  • VAPref-10K contains approximately 9K prompts and 10.3K fine-grained paired comparisons for joint video-audio generation.The comparisons use clips generated by open-source state-of-the-art models.
  • VA-Judger-Bench evaluates whether video-audio reward models align with human preferences across in-domain and out-of-domain comparisons.
  • VA-Judger is a chain-of-thought omni-reward model trained progressively from clear comparisons to harder human-verified preference explanations.The progressive strategy establishes structured comparison and preference discrimination before handling subtle quality differences.
  • VA-Judger better aligns with human judgments, and its rewards substantially improve human-perceived generation quality during post-training.

2 RELATED WORK

Prior multimodal reward models and reinforcement-learning methods extend human preference optimization across modalities, but joint video-audio generation requires explicit evaluation of audio, video, text, semantics, and synchronization together.

  • Joint video-audio generators increasingly combine visual motion and synchronized audio through architectures such as twin backbones, cross-modal fusion, and modality-balancing designs.
  • Existing multimodal reward models do not explicitly judge whether audio, video, and text form a coherent event.VA-Judger targets joint video-audio preference alignment across visual and audio quality, semantics, and cross-modal consistency.
  • OmniNFT applies diffusion reinforcement learning to multimodal generation but still optimizes separate evaluation metrics.The listed metrics include VideoAlign, HPSv3, Audiobox Aesthetics, CLAP, and DeSync/Synchformer.
  • VA-Judger is motivated by replacing narrow expert metrics with human-preference rewards for joint video-audio generation.

3 METHOD

The method combines difficulty-aware preference data with a three-stage reward-model pipeline: structured pairwise judgment, human-verified hard-pair alignment, and dimension-wise reinforcement learning.

  • 3.1 TASK FORMULATION: VA-Judger formulates reward modeling as structured pairwise comparison of two video-audio clips conditioned on a text prompt.It scores prompt alignment, video-audio consistency, audio quality, video quality, and completeness/coherence before producing a final preference.
  • 3.2 DATASET CONSTRUCTION: VAPref-10K derives realistic prompts from 10,173 web video-audio clips and organizes them across diverse scenarios.The source material includes YouTube, Bilibili, films, and television series.
  • 3.2 DATASET CONSTRUCTION: Difficulty-aware pairing uses 4.4K easy pairs with clear quality gaps to initialize comparison learning before near-quality cases.The design avoids instability from exclusively hard pairs and superficial cues from exclusively easy pairs.
  • 3.3 TRAINING PIPELINE: Stage 1 cold-starts structured preference discrimination on unambiguous cross-model pairs.The schedule teaches the response format before subtle perceptual differences are introduced.
  • 3.3 TRAINING PIPELINE: Stage 2 retains Gemini-generated rubric responses only when their final preferences agree with human labels, yielding 4.5K verified examples.This preserves structured responses while grounding hard-pair preferences in human feedback.
  • 3.4 DIMENSION-WISE REINFORCEMENT LEARNING: SFT alone can produce low loss while judgments contradict human preferences, motivating reinforcement learning on self-generated responses.
  • 3.4 DIMENSION-WISE REINFORCEMENT LEARNING: Dimension-wise GRPO rewards both the correct final preference and score orderings consistent with human-identified supporting dimensions.This supplies denser supervision than a single binary reward and discourages internally inconsistent reasoning.

4 EXPERIMENTS

Experiments show that separate evaluation metrics have limited agreement with holistic human preferences, while VA-Judger improves reward-model accuracy and post-training outcomes. Human evaluations further indicate that its post-trained model is preferred over OmniNFT and the base LTX-2.

  • Reward model evaluation: 50.43%–56.88% overall accuracy shows that representative single-dimension metrics only weakly recover holistic human preferences.VideoAlign and AudioBox degrade from 62.39% and 58.70% on easy pairs to 48.35% and 44.00% out of domain.
  • Reward model evaluation: VA-Judger’s three-stage training raises Qwen3-Omni Total Acc from 57.83% to 68.43%.Easy SFT reaches 62.35%, human-verified hard-pair training reaches 65.91%, and Dimension Wise GRPO reaches 68.43%.
  • Post-training LTX-2 with VA-Judger: VA-Judger post-training ranks first on 11 of 13 JavisBench metrics and six of seven complementary metrics.Relative to LTX-2, visual, motion, and audio quality increase from 2.248, 0.697, and 4.767 to 3.942, 1.183, and 5.610.
  • Post-training LTX-2 with VA-Judger: OmniNFT’s separate expert rewards can improve rewarded metrics without consistently improving overall human preference, whereas VA-Judger uses holistic human comparisons.This contrast is framed as reward hacking versus alignment with human preferences.
  • Post-training LTX-2 with VA-Judger: VA-Judger improves temporal instruction following for both the rabbit-motion and spacecraft-landing examples.The qualitative comparisons include LTX-2, OmniNFT, and the VA-Judger post-trained model.
  • Post-training LTX-2 with VA-Judger: 62.30% of human preferences favor the VA-Judger post-trained model, versus 27.63% for OmniNFT and 10.08% for base LTX-2.The evaluation uses 200 prompts and three-way comparisons among the three models.

5 CONCLUSION

The paper concludes that VA-Judger combines human-preference data, structured dimension-wise reasoning, and progressive training to improve reward modeling for joint video-audio generation. Its rewards improve LTX-2 quality and support more interpretable, human-aligned post-training.

  • 5 CONCLUSION: VA-Judger is a human-aligned omni-reward model that produces dimension-wise reasoning for joint video-audio generation.The paper constructs VAPref-10K and introduces VA-Judger-Bench to address preference-supervision scarcity and evaluate alignment.
  • 5 CONCLUSION: VA-Judger trains from easy comparisons, human-verified hard-pair explanations, and dimension-wise reinforcement learning.The pipeline refines both the reasoning process and final preference.
  • 5 CONCLUSION: Experiments show more accurate preference judgments than existing omni-models and automatic metrics, while post-training LTX-2 significantly improves video-audio quality.The conclusion presents these as the paper’s main empirical outcomes.

A QUALITATIVE RESULT OF VA-JUDGER JUDGMENT

VA-Judger evaluates two generated clips across fixed video-audio quality dimensions before producing a final preference. The associated materials show the judgment rubric and the datasets’ category and subset distributions.

  • Judgment protocol: The system prompt asks VA-Judger to compare two clips on five fixed dimensions before selecting a preference.The dimensions cover prompt match, audio-visual consistency, audio quality, video quality, and content coherence.
  • Judgment protocol: Figure 5 presents a complete VA-Judger input and response in which both clips are evaluated on every defined dimension.
  • Dataset context: The VAPref-10K construction overview uses captioned real clips, rewritten prompts, and candidate generations from LTX-2, OVI, and DaVinci-MagiHuman.
  • Dataset context: The source-video distribution spans six broad categories, while the JavisBench table reports category distributions for the full pool and evaluation subset.The supplied figure identifies six categories; the table caption states that each cell contains counts and percentages.

C JAVISBENCH EVALUATION SUBSET DISTRIBUTION

The JavisBench generation-model evaluation uses a uniformly random 200-prompt subset from the full 10,140-prompt pool, without category filtering or stratification.

  • The evaluation subset contains 200 prompts sampled uniformly at random from the full 10,140-prompt JavisBench pool.The sampling applies neither category-based filtering nor stratification.
  • Sound Type and Spatial Composition permit multiple labels, so their category percentages do not sum to 100%.

D.1 REWARD MODEL TRAINING AND EVALUATION

VA-Judger training uses supervised fine-tuning followed by dimension-wise GRPO, with standardized decoding and rubric-based accuracy evaluation.

  • Supervised fine-tuning: Supervised fine-tuning updates the language backbone while freezing the visual encoder and aligner.Training uses BF16, DeepSpeed ZeRO 3, a 5×10−6 learning rate, and an effective batch size of 128.
  • Dimension Wise GRPO: Dimension Wise GRPO samples eight responses per training pair and weights answer and dimension rewards equally.Responses omitting any required dimension score receive zero reward.
  • Evaluation: Evaluation uses greedy decoding and extracts the final choice from the <answer> tag.Invalid or missing choices count as incorrect for Total Acc and are excluded from Parsed Acc.

D.2 LTX-2 POST-TRAINING

The LTX-2 post-training setup applies VA-Judger-guided reinforcement learning with parameter-efficient adapters, fixed generation settings, and batched reward computation.

  • Model and optimization: LoRA adapters are inserted into LTX-2 video, audio, and cross-modal attention and feed-forward layers while the backbone remains frozen.Current and old policies use separate adapters, while disabling adapters recovers the fixed reference policy.
  • Generation settings: Each prompt produces eight candidates at 512×768 resolution with 121 frames at 24 FPS.Training rollouts use 20 denoising steps with video and audio classifier-free guidance scales of 1.5 and 3.0.
  • Model and optimization: Training randomly selects eight of 20 denoising timesteps for each update and clips advantages to [−5, 5].The reference regularization weight is 10−4.
  • Reward computation: Video loss is reweighted using video-to-audio attention, while two GPUs host VA-Judger inference servers for batched reward computation.All generation-model baselines use the same prompts, generation settings, and evaluation subset.

E FULL METRIC ALIGNMENT RESULTS

The paper evaluates metrics by converting scalar scores into pairwise preferences and measuring strict agreement with human labels across video, audio, cross-modal consistency, and synchronization dimensions.

  • Evaluation protocol: Strict pairwise accuracy compares each metric’s preferred clip with the human label, excluding exact metric ties.Parentheses indicate score coverage below 100%.
  • Metric evaluations: Video-quality metrics are evaluated for their agreement with human preferences.The results are reported in Table 4 as human preference accuracy percentages.
  • Metric evaluations: Audio-quality metrics are evaluated for their agreement with human preferences.The results are reported in Table 5 as human preference accuracy percentages.
  • Metric evaluations: Text-video and text-audio consistency metrics are evaluated using human preference accuracy.These evaluations are reported in Table 6.
  • Metric evaluations: Audio-video semantic-consistency metrics include three ImageBind-based measures that differ in preprocessing and feature aggregation.Audio-Video Alignment, AVHScore, and ImageBind A-V are compared in Table 7.
  • Metric evaluations: Synchronization and aggregate audio-video metrics are evaluated with human preference accuracy, with score coverage shown in parentheses.These results are reported in Table 8.
Loading 2608.18607v2…