Adaptive Reward Routing: Dynamic Multi-Reward Optimization for Joint Audio-Video Diffusion via Forward-Process RL

1 INTRODUCTION

• Joint generation must satisfy modality-specific visual and audio quality, cross-modal semantic alignment, and temporal synchronization, which are difficult to capture with one supervised objective.Reward-guided diffusion RL offers a multi-reward approach to these requirements.Source: §1 · p. 1
• Base- and trained-checkpoint probes show shifting synchronization-critical layers and cross-modal gradient peaks, making fixed update routing progressively stale.Figure 1 also reports reward conflicts and a vanishing AV-DeSync weight under geometry-only weighting.Source: §1 · p. 1, Figure 1 · p. 2
• Fixed reward weights leave conflicts unadapted, while gradient-geometry weights reflect compatibility rather than user-preference importance.The paper argues for conflict-aware adaptation anchored by user-defined priorities.Source: §1 · p. 2
• Cross-Modal Influence-Guided Routing uses bidirectional cross-attention responses to derive token weights and layer scales, while Preference-Preserving Modality-Aware Reweighting applies branch-specific conflict corrections after warm-up.The corrections are residuals to predefined reward weights, preserving user priorities as conflicts change.Source: §1 · p. 2
• Adaptive Reward Routing consistently improves modality quality, semantic alignment, and audio-video synchronization, with ablations and mechanism analyses supporting its adaptive components.Direct path interventions indicate the cross-attention response proxy identifies functionally important layers and tokens, while initialization-frozen routes become stale.Source: §1 · p. 2
• The paper frames joint audio-video diffusion RL as dynamic multi-reward optimization over update locations and reward coordination, and reports stable, effective post-training from adapting both.Its contributions unify cross-modal response-guided token/layer localization with preference-preserving, modality-aware reward coordination.Source: §1 · pp. 2–3

Joint Audio-Video Generation

• Joint audio-video systems connect pretrained experts, use unified diffusion Transformers, or couple separate streams through bidirectional cross-attention, making reward responsibility and gradient routing less obvious than in single-stream models.The passage cites Wang et al. (2025), Liu et al. (2026c;d), and HaCohen et al. (2026) for these approaches.Source: §2 · p. 3

Reinforcement Learning for Diffusion Models

• DiffusionNFT (Zheng et al., 2025) optimizes the forward process with implicit positive and negative policies; OmniNFT (Zhang et al., 2026) adds modality-wise credit assignment, while this work recomputes token- and layer-level routes.The related-work passage also describes GRPO (Guo et al., 2025), Flow-GRPO (Liu et al., 2026a), and DanceGRPO (Xue et al., 2025) as online optimization methods for flow-based generation.Source: §2 · p. 3

Multi-Reward Optimization

• GDPO (Liu et al., 2026e) uses predefined aggregation weights, while MARBLE (Zhao et al., 2026) uses gradient compatibility; this work estimates conflicts within each modality branch and corrects preference priors.Other cited multi-task approaches seek common descent directions, project conflicting gradients, or optimize local agreement.Source: §2 · p. 3

Problem Formulation and Preliminaries

• The setup studies joint audio-video diffusion post-training under DiffusionNFT (Zheng et al., 2025), using modality, sample, reward, token, block, and flow-matching-timestep indices.The modalities are video and audio.Source: §3 · p. 3

Joint Audio-Video Flow Matching

• LTX-2 uses separate audio and video streams under a shared timestep, predicts both velocity fields jointly, and exchanges information through bidirectional cross-attention.Gated audio-to-video and video-to-audio outputs are added to the video and audio streams, respectively.Source: §3 · p. 3

Diffusion Forward-Process Reinforcement Learning

• DiffusionNFT constructs implicit positive and negative policies from updated and trainable velocity predictors, then converts each prompt's group of generated samples into group-relative advantages.The advantage indicates whether a sample performs above or below its peers for the same prompt and requires no learned value function.Source: §3 · pp. 3–4

Multi-Reward Optimization

• The framework adapts reward reweighting, modality-branch assignment, token-level credit allocation, and layer-wise cross-modal gradient routing.These are the four optimization levels listed in the overview figure.Source: Figure 2 · p. 4
• GDPO combines independently computed reward advantages with predefined weights, whereas MARBLE chooses simplex weights to minimize the norm of the weighted normalized reward gradients.GDPO preserves explicit preferences but cannot adapt to conflicts; MARBLE adapts to gradient geometry but does not encode preference or modality responsibility.Source: §3 · p. 4

4.1 OVERVIEW

• The optimization generates samples, evaluates and normalizes rewards, refreshes branch-wise conflict estimates after warm-up, then applies routed losses and updates the policy.The refresh probes use uniform token weights before the routed update.Source: §4.1 · p. 5

4.2 CROSS-MODAL INFLUENCE-GUIDED ROUTING

• Pre-gate cross-attention responses collected during an intermediate-to-late denoising window guide token weights and layer-wise cross-modal gradient flow.Token weights average responses across selected timesteps and blocks; audio is normalized globally and video within each frame.Source: §4.1 · p. 4, §4.2 · p. 5
• Layer routing averages the same responses over tokens and timesteps, converting normalized layer scores into soft detachment so influential layers retain more gradient.The routed representation leaves forward values unchanged, and A2V and V2A paths are routed independently.Source: §4.2 · p. 5

4.3 PREFERENCE-PRESERVING MODALITY-AWARE REWEIGHTING

• Branch-specific reward-gradient interactions provide conflict-aware residual corrections after warm-up, while predefined weights remain preference priors with a nonzero floor.Rewards are probed through their responsible branches, with cross-modal rewards using both; token routing is disabled during probes.Source: §4.3 · pp. 5–6

4.4 TRAINING OBJECTIVE

• Adaptive reward weights produce separate modality-branch advantages, with cross-modal rewards included in both branches, before token routing forms modality losses for combination and reference-policy regularization.The detached mean absolute policy residual is averaged over each modality’s tokens and feature dimensions.Source: §4.4 · p. 6

5.1 EXPERIMENTAL SETUP

• Experiments use LTX-2 and LTX-2.3, 19,487 VGGSound-derived prompts, five reward signals, and full JavisBench evaluation on 10,140 prompts.Comparisons include GDPO, MARBLE, and OmniNFT, which represent fixed coordination, conflict-aware coordination, and static multimodal routing.Source: §5.1 · pp. 6–7

5.2 MAIN RESULTS AND TRAINING DYNAMICS

• The method achieves the best result on nine of ten metrics under each backbone, with consistent gains across modality quality, semantic consistency, cross-modal consistency, and synchronization.Qualitative examples also show more stable identities and scene structures than LTX-2 and OmniNFT.Source: Table 1 · p. 7, Figure 3 · p. 7, §5.2 · p. 8
• The method reaches the highest average reward while maintaining favorable trajectories across all five component rewards.The reported average is the arithmetic mean of the five normalized rewards.Source: §5.2 · p. 8, Figure 4 · p. 8

5.3 ABLATION STUDIES

• Token weighting improves local credit assignment, layer scaling further improves consistency and synchronization, and their combination is the strongest routing-only configuration.Branch-aware balancing, residual preference correction, and warm-up add a complementary reward-coordination path; the complete method performs best overall.Source: §5.3 · pp. 8–9, Figure 4 · p. 8, Table 2 · p. 9

5.4 VALIDATING THE CROSS-MODAL INFLUENCE PROXY

• Layer-proxy correlations are 0.98 for A2V and 0.97 for V2A; current-model agreement stays above 0.96, versus 0.56 and 0.38 for initialization-frozen routing.These intervention comparisons test layer rankings across training checkpoints.Source: §5.4 · p. 9, Figure 5 · p. 9
• Blocking the top-scoring 10% of tokens changes the final prediction 1.62× more for A2V and 1.74× more for V2A than blocking an equally sized random group.The comparison uses groups of identical size.Source: Figure 5 · p. 9, §5.4 · p. 10

• As optimization reshapes cross-modal functions, token/layer localization and relative reward strengths should evolve; Adaptive Reward Routing tracks both while preserving user-defined preferences.Source: §6 · p. 10
• The authors report consistent gains across backbones, metrics, and ablations, supporting adaptation of optimization alongside model change.Source: §6 · p. 10
• No established unified reward captures modality quality, semantic consistency, and temporal synchronization; human-preference models complement but do not replace fine-grained supervision.Source: §6 · p. 10
• The method adopts DiffusionNFT for stable direct supervision of sampled flow-matching timesteps, and can extend to other diffusion objectives through token losses and gradient paths.Reward coordination requires only reward-wise gradients.Source: §6 · p. 10
• Validation covers dual-stream LTX and unified single-stream JavisDiT++; routing needs identifiable modality tokens and isolatable directional responses, while reward coordination is architecture-independent.Models with inseparable modality representations or inaccessible interaction responses remain future work.Source: §6 · p. 10