PaperlayerView exact revision on arXiv

Source-linked AI summary

AV-GRPO: Modality-Anchored Decoupling Diffusion Reinforcement Learning for Joint Audio-Video Generation

Zhiyu Xu, Weilong Yan, Yufei Shi, Shiyang Li, Yihao Liu, Kin-Man Lam, Yuewen Cao

arXiv:2609.29816v1cs.CVcs.SD

TL;DR

Joint audio–video generation still struggles to combine stream quality, prompt alignment, and synchronization, and joint reward optimization makes comparisons and credit assignment difficult. AV-GRPO uses modality-anchored alternating updates, and experiments report consistent gains over LTX-2.3 and GDPO across two benchmarks and two fine-tuning settings.

  • Problem

    Joint audio–video generation still struggles with perceptual artifacts, text misalignment, and mismatched or out-of-sync events, while mixed rewards and changing counterpart samples complicate optimization and comparison.

    Source: §1 · pp. 1–2

  • Method

    AV-GRPO alternates modality-anchored rollouts and frozen-tower updates, with decoupled optimization settings, alongside the five-dimensionally decoupled 5DAV training set.

    Source: §1 · p. 2

  • Results

    Experiments on JavisBench and VABench show consistent improvements over LTX-2.3 and GDPO in perceptual quality, text alignment, and audio–video coherence and synchronization under LoRA and full fine-tuning.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The results support AV-GRPO as an effective and scalable approach to improving coupled audio–video generators.

    Source: §5 · p. 10

    The convergence analysis applies to an idealized counterpart under a common reference joint law and contraction assumptions.

    Source: §2.2 · p. 4

Abstract

from arXiv · show

Recent years have witnessed major progress in joint audio-video generation. Existing models still suffer from limited per-modality fidelity, insufficient text-modality alignment and weak cross-modal synchronization. While reinforcement-learning post-training offers a promising remedy, directly adapting it to joint audio-video generation is challenging. Heterogeneous multimodal rewards entangle learning signals and complicate credit assignment. Joint optimization of two modality towers is computationally expensive given their divergent dynamics. Moreover, synchronization evaluation difficulty depends on paired samples, preventing fair reward comparisons. We propose AV-GRPO, a modality-anchored online diffusion RL framework, and 5DAV, a decoupled, difficulty-controllable training dataset. AV-GRPO includes three key modules: (1) modality-anchored rollouts to disentangle learning signals and stabilize difficulty; (2) trajectory-locked frozen-tower optimization to reduce cost and reassign credit; (3) adaptive objectives and perturbation strengths tailored to modality-specific dynamics. This converts coupled multimodal preference learning into unimodal subproblems for precise reward attribution and better synchronization. Our 5DAV dataset decouples samples across five dimensions for systematic training. Experiments on JavisBench and VABench demonstrate AV-GRPO outperforms LTX-2.3 in generation quality, semantic alignment and cross-modal synchronization under LoRA and full fine-tuning. Ablations confirm our designs. Code and data: https://github.com/zhiyuxu03/AV-GRPO

1 INTRODUCTION

Joint audio–video generation still faces challenges in perceptual quality, text alignment, and synchronization, while joint reward optimization complicates comparison and credit assignment. AV-GRPO and 5DAV address these issues, with reported gains across two benchmarks.

1 INTRODUCTION

  • Joint audio–video post-training must improve both streams while preserving semantic and temporal dependence, but mixed rewards, paired-sample difficulty, joint-update costs, and differing modality dynamics complicate optimization.Synchronization scores can depend on the sampled counterpart, making comparisons across jointly varying pairs less controlled.Source: §1 · pp. 1–2
  • AV-GRPO alternates modality-anchored rollouts with frozen-tower updates and modality-specific objectives, noise strengths, and loss weightings to control comparisons, reduce costs, and direct credit.The framework alternates which modality is the target so both towers can improve under controlled cross-modal conditions.Source: §1 · p. 2
  • On JavisBench and VABench, AV-GRPO shows broad gains over LTX-2.3 and GDPO in quality, alignment, coherence, and synchronization under LoRA and full fine-tuning.Ablations test the dataset and alternating schedule.Source: §1 · p. 2
  • 5DAV is a training set decoupled across five dimensions, designed to support controlled post-training.The experiments use JavisBench and VABench.Source: §1 · p. 2

2 METHOD

AV-GRPO adapts online diffusion reinforcement learning to joint audio–video generation by conditioning each update on a fixed modality trajectory. It alternates frozen-tower optimization and uses adaptive noise clipping and modality-specific configurations for stability.

2.1 PRELIMINARIES

  • Flow-GRPO replaces deterministic sampling with a stochastic differential equation sampler that preserves the pretrained model’s marginal distributions and supports group-relative policy updates.The deterministic ODE does not provide stochastic samples or an inexpensive transition probability for this update.Source: §2.1 · p. 3

2.2 AV-GRPO

  • Each AV-GRPO phase fixes one modality’s complete trajectory, samples target-modality trajectories, and updates only the target tower; the phases alternate every n training steps.A shared anchor controls synchronization difficulty within the group, while refreshing anchors across groups exposes the target to varied counterparts.Source: §2.2 · p. 4
  • Because the anchor reward is constant across candidates, group centering removes it, leaving target-modality quality and alignment plus synchronization for reward attribution.Freezing the anchor avoids its backward activations and optimizer updates, while the target’s comparisons share the same counterpart.Source: §2.2 · pp. 4–5

2.3 ADAPTIVE NOISE CLIPPING FOR ROBUST SAMPLING

  • Adaptive Noise Clipping bounds the stochastic increment that can overwhelm the latent signal near t = 1; with 15 sampling steps, it stabilizes training at larger noise strengths.When clipping is inactive, the update reduces to the original one.Source: §2.3 · p. 5

2.4 HYPERPARAMETER DECOUPLING AND TRAINING STABILITY

  • AV-GRPO decouples noise strengths and objectives by modality; for LTX-2.3, it uses aV = 0.02 and aA = 0.8.The video tower can become over-exposed under the original objective, with weak gradients at late denoising stages; video policy and KL terms are reweighted, while audio retains the original weighting.Source: §2.4 · p. 5

3 EXPERIMENT

The experiments describe 5DAV, reward and evaluation setups, and results for AV-GRPO across benchmarks and ablations. AV-GRPO improves most reported measures, while the ablations assess dataset design and alternating schedules.

3.1 5DAV DATASET

  • 5DAV forms 288 categories from five dimensions, with 20 prompts per category for 5,760 samples and adjustable category sampling.The dimensions are semantic hierarchy, sound source type, synchronization difficulty, temporal complexity, and instruction granularity; the design supports traceable training and localization of model weaknesses.Source: §3.1 · pp. 5–6

3.2 EXPERIMENTAL SETUP

  • The experiments use one node of eight NVIDIA A800 80 GB GPUs, with the video tower updated first and modality switching every four steps.LTX-2.3 generates 544 × 960 clips of 97 frames at 24 FPS with 15 denoising steps.Source: §3.2 · p. 6

3.2.1 REWARD MODELS AND REWARD COMPOSITION

  • The reward composition combines modality-specific quality and text-alignment scores with reversed DeSync, then standardizes scores within each group.For audio, a CLAP guardrail uses only alignment advantage when the group’s average CLAP falls below a threshold.Source: §3.2.1 · pp. 6–7

3.2.2 EVALUATION BENCHMARKS

  • JavisBench assesses fidelity, text alignment, cross-modal semantic coherence, and synchronization, while VABench combines expert-model assessments with MLLM judgments.JavisBench includes VQ, AQ, ImageBind, CLIP, CLAP, AV-IB, AVH Score, JavisScore, and DeSync; VABench includes speech, audio-aesthetic, lip-sync, alignment, artistry, and expressiveness measures.Source: §3.2.2 · p. 7

3.3.1 QUANTITATIVE ANALYSIS

  • AV-GRPO consistently improves nearly all metrics over LTX-2.3 and GDPO, with full fine-tuning achieving the best overall performance.Examples include JavisBench AQ increasing from 5.097 to 5.798 and VABench Lip Sync increasing from 1.439 to 1.646 versus the stated baselines; LoRA and full fine-tuning retain distinct metric tradeoffs.Source: §3.3.1 · p. 7, Table 1 · p. 8, Table 2 · p. 8

3.3.2 QUALITATIVE ANALYSIS

  • Qualitative examples illustrate improvements in modality fidelity, text alignment, and synchronization across event-driven effects, music performance, object sounds, and natural ambience.The figure pairs video frames with corresponding audio waveforms to compare visual changes against audio timing and character.Source: Figure 2 · p. 6, §3.3.2 · p. 8

3.4.1 DATASET

  • The dataset ablations compare VGGSound with 5DAV and test removing easy or hard instruction-granularity and synchronization-difficulty categories.The reported changes in synchronization measures under difficulty removals show why an aggregate benchmark score alone is insufficient to assess the data design.Source: §3.4.1 · p. 8, Figure 3 · p. 9

3.4.2 ALTERNATING MODALITY-WISE OPTIMIZATION

  • Alternating optimization benefits both towers, and the four-step switching interval gives the strongest overall balance across the plotted measures.The comparison includes no alternation and intervals of two, four, and eight steps; individual metrics need not peak at the same interval.Source: §3.4.2 · p. 9, Figure 3 · p. 9

4 RELATED WORK

Prior joint audio–video work increasingly uses unified dual-stream generation, while this paper focuses on reward-guided post-training. Existing diffusion post-training methods address reward optimization, but simultaneous modality variation leaves cross-modal credit assignment unresolved.

4.1 JOINT AUDIO–VIDEO GENERATION

  • Unified audio–video models use interacting dual-stream architectures, whereas this work studies reward-guided post-training to improve both modalities without entangling their learning signals.Earlier asymmetric methods such as FoleyCrafter and MMAudio generate audio conditioned on video, limiting direct bidirectional co-generation.Source: §4.1 · p. 9

4.2 REINFORCEMENT LEARNING-BASED POST-TRAINING

  • GDPO independently normalizes multiple rewards before aggregation but does not resolve cross-modal credit assignment when both modalities vary simultaneously.DDPO and DPOK use policy gradients for reverse denoising, while Flow-GRPO and DiffusionNFT provide later online reward-optimization approaches.Source: §4.2 · p. 9

5 CONCLUSION

AV-GRPO combines anchored rollouts with alternating frozen-tower optimization for joint audio–video post-training. On JavisBench and VABench, it improves reported quality, alignment, coherence, and synchronization measures over LTX-2.3 and GDPO under both LoRA and full fine-tuning.

5 CONCLUSION

  • AV-GRPO combines anchored rollouts and alternating frozen-tower optimization to disentangle rewards, control comparisons, reduce overhead, and redirect credit assignment.The framework also adapts to modality-specific dynamics and sample difficulty.Source: §5 · pp. 9–10
  • Experiments show consistent improvements over LTX-2.3 and GDPO across perceptual quality, text alignment, audio–video coherence, and synchronization under LoRA and full fine-tuning.The conclusion characterizes AV-GRPO as an effective and scalable approach to improving coupled audio–video generators.Source: §5 · p. 10

A PROOF OF THE CONVERGENCE AND JOINT OPTIMALITY OF AV-GRPO

Under a common reference joint law and bounded reward, this proof relates alternating conditional optimization to a joint KL-regularized optimum. It establishes compatible optimal kernels and, under contraction, unique self-consistent marginals that match the optimum’s marginals.

A PROOF OF THE CONVERGENCE AND JOINT OPTIMALITY OF AV-GRPO

A.3 JOINT KL VARIATIONAL IDENTITY

  • The exponentially tilted joint distribution is the unique global optimum of the joint KL-regularized objective.Its optimal value is β log Z.Source: §A.3 · p. 13

A.4 OPTIMAL KERNELS OF THE CONDITIONAL SUBPROBLEMS

  • Each fixed-counterpart conditional subproblem has a unique optimal kernel, and the two kernels are conditional distributions of the same tilted joint optimum.The video and audio sides are treated symmetrically.Source: §A.4 · p. 13, §A.5 · p. 14

A.6.2 ONE COMPLETE GIBBS SWEEP

  • A complete alternating Gibbs sweep produces a joint law determined only by the input law’s audio marginal, and the tilted joint optimum is stationary for this chain.The stationarity result requires no ergodicity or convergence assumption.Source: §A.6.2 · p. 14, §A.6.3 · p. 14

A.8 EXISTENCE, UNIQUENESS, AND IDENTIFICATION OF THE SELF-CONSISTENT MARGINALS

  • When the product of the video and audio Dobrushin coefficients is below one, the alternating marginal map is a contraction with a unique fixed point.The contraction factor is ρ = δvδa < 1.Source: §A.7.3 · p. 15, §A.8 · p. 15
  • The unique self-consistent audio and video marginals are exactly the marginals of the jointly KL-optimal distribution.The fixed-point uniqueness identifies the two marginals with those of P ∗.Source: §A.8 · p. 15

A.9 THE UNIQUE STATIONARY JOINT DISTRIBUTION

This section distinguishes stationarity from uniqueness and convergence for the alternating chain. Under the contraction condition, the chain has a unique stationary law and both sampling phases converge geometrically to it in total variation.

A.9 THE UNIQUE STATIONARY JOINT DISTRIBUTION

  • Under ρ < 1, the alternating Gibbs chain has a unique stationary distribution, P ∗.P ∗ is stationary even without the contraction condition; contraction is needed for uniqueness.Source: §A.9 · p. 16
  • The joint optimum, compatible conditional kernels, and stationarity do not require ρ < 1, but uniqueness of stationarity and geometric convergence do.The distinction identifies which conclusions depend on strict contraction.Source: §A.9 · p. 16

A.10.2 TV ISOMETRIC EMBEDDING OF THE MIXING OPERATORS

  • For any initial joint law, both the video and audio sampling-phase laws converge geometrically in total variation when ρ < 1.The theorem gives explicit rates for both phases.Source: §A.10.2 · p. 16

A.11 MAIN THEOREM

For fixed text condition, Polish trajectory spaces, a common reference law, bounded reward, and ρ < 1, the theorem characterizes the idealized alternating Gibbs process. It establishes the joint optimum, conditional optima, unique stationary law and marginals, and geometric convergence.

A.11 MAIN THEOREM

  • P ∗ is the unique global optimum of the joint KL-regularized objective, with optimal value β log Z.The theorem assumes ρ = δvδa < 1 along with the stated reference-law and reward conditions.Source: §A.11 · p. 17
  • The two conditional KL subproblems have unique optimal kernels, which are regular conditional distributions of P ∗.Source: §A.11 · p. 17
  • The alternating Gibbs chain has P ∗ as its unique stationary distribution.Source: §A.11 · p. 17
  • The self-consistent marginal equations have a unique solution given by the marginals of P ∗.Source: §A.11 · p. 17
  • From any initial joint law, both sampling-phase laws converge to P ∗ in total variation at the explicit rates of Theorem 1.Source: §A.11 · p. 17

A.12 CONNECTION WITH ALTERNATING TRAINING

With exact conditional-kernel update oracles that do not interfere with the other tower’s realized kernel, the idealized training and inference processes share the alternating Gibbs convergence result. After one exact update per tower, their joint laws converge to the global optimum.

A.12 CONNECTION WITH ALTERNATING TRAINING

  • The consistency result assumes each tower can realize its selected optimal conditional kernel and that updating one tower leaves the other tower’s realized kernel unchanged.Source: §A.12 · p. 17
  • After each tower completes one exact update, both policy kernels are fixed at their selected conditional optima.Source: §A.12 · p. 17
  • The training-rollout phase laws and inference distribution using the same full-trajectory alternating protocol converge geometrically in total variation to P ∗, attaining the joint objective’s global optimum.This is an idealized training–inference consistency result under the exact-oracle assumptions.Source: §A.12 · p. 17

A.13 CONCLUSION

Under a common reference joint law, bounded reward, β > 0, and Dobrushin contraction ρ = δvδa < 1, the section establishes the unique joint KL-regularized optimum and convergence of alternating conditional Gibbs updates. The proof uses KL variational identity, compatibility from the shared reference law, and contraction for uniqueness and convergence.

A.13 CONCLUSION

  • The exponentially tilted joint law P* is the unique global optimum of the joint KL-regularized objective, with optimal value β log Z.The result assumes bounded reward, β > 0, and Dobrushin contraction ρ < 1.Source: §A.13 · p. 17
  • The conditional Gibbs kernels are unique optimal solutions to their subproblems and are compatible regular conditional distributions of P*.The self-consistent marginal equations also have a unique solution: the marginals of P*.Source: §A.13 · pp. 17–18
  • The systematic-scan Gibbs chain has unique stationary distribution P*, and its two sampling-phase laws converge geometrically in total variation from any initial joint law.The section also gives explicit convergence rates; exact-kernel-oracle implementations inherit the geometric convergence properties for training rollouts and inference.Source: §A.13 · pp. 17–18
  • The proof derives optimality from the KL variational identity, compatibility from the common reference joint law, and uniqueness and convergence from Dobrushin contraction.These claims hold under the section’s idealized assumptions.Source: §A.13 · pp. 17–18

B TRAINING DETAILS

The appendix gives training settings, Flow-GRPO and LongCat-Video framework details, and the reward composition used for each tower. It also describes a CLAP guardrail for audio-tower reward hacking.

B TRAINING DETAILS

  • AV-GRPO training uses one 8-GPU A800 node, with 8 samples per prompt and alternating tower updates; full fine-tuning switches towers every 4 steps on shared prompt batches.The LTX-2.3 clips use 544 × 960 resolution, 97 frames, 24 FPS, and 15-step denoising; LoRA and full fine-tuning use cosine-decayed learning rates over 480 steps.Source: B TRAINING DETAILS · p. 18

C.1 FLOW-GRPO

  • Flow-GRPO converts the deterministic flow ODE into an equivalent SDE, then uses Euler-Maruyama updates and group-relative reward normalization for policy optimization.The SDE is described as preserving the original model’s marginal distributions at all timesteps, with a Gaussian transition distribution and closed-form KL divergence.Source: §C.1 · pp. 18–19

C.2 LONGCAT-VIDEO

  • LongCat-Video’s policy and KL-loss gradients are reweighted to address scaling issues shared by both loss terms.The appendix proceeds from the reweighting coefficients to a revised GRPO objective and policy gradient.Source: §C.2 · pp. 19–21

D REWARD MODELS AND REWARD COMPOSITION

  • Video-tower rewards combine VideoAlign, CLIP, and DeSync; audio-tower rewards combine Audiobox Aesthetics, CLAP, and DeSync, with group-wise normalization used to form advantages.For audio, when average group CLAP is below its threshold, only CLAP supplies the learning signal; otherwise AQ and DeSync are incorporated.Source: D REWARD MODELS AND REWARD COMPOSITION · p. 21
  • Audio-tower training can reward-hack by sacrificing CLAP while boosting AQ and DeSync, so a threshold guardrail prioritizes semantic alignment when group-average CLAP is low.Below the threshold, the CLAP advantage directly becomes the final advantage.Source: D REWARD MODELS AND REWARD COMPOSITION · pp. 21–22

E MORE QUALITATIVE RESULTS

Figure 4 presents additional qualitative results for the full AV-GRPO model.

E MORE QUALITATIVE RESULTS

  • Figure 4 shows additional qualitative results for AV-GRPO (full).The supplied caption does not specify the compared examples or report an outcome.Source: Figure 4 · p. 22

Related papers

Loading 2609.29816v1…