PaperlayerView exact revision on arXiv

Source-linked AI summary

Real-Time Joint Audio-Video Generation by Parallel Adapter Composition

Jingyu Li, Xiaoxiao Xiang, Yiwen Guo

arXiv:2610.10343v2cs.CV

TL;DR

Real-time interactive joint audio–video generation needs incremental output and low per-chunk cost, but chained training can make later objectives undo earlier capabilities. This paper composes a frozen-backbone causal adapter with an off-the-shelf few-step adapter; the result tracks the teacher on appearance and compares favorably with chained baselines on most metrics.

  • Problem

    Bidirectional joint audio–video models require a fixed frame count and finish sampling before emitting output, limiting their fit for interactive applications that need bounded first-frame time and length-independent per-chunk cost.

    Source: §1 · p. 1

  • Method

    The paper trains a causal adapter against a frozen backbone and adds it at inference to an off-the-shelf few-step adapter, without joint or chained training.

    Source: §1 · p. 2

  • Results

    The composed model matches the teacher on appearance and single-frame distribution, beats CHAINED-DA on most held-out metrics, and trails CHAINED-AD on lip sync.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Parallel composition yields a streaming system that generates joint audio–video at ≈26 fps at 480×832 and sustains 30 s with stable image quality.

    Source: §1 · p. 2

    On distilled weights, plain flow-matching causal training can pull the few-step jump field toward instantaneous velocity, so four-step sampling may no longer suffice.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Deploying a joint audio-video diffusion transformer for real-time, interactive generation normally requires two essential modifications: block-autoregressive attention, so frames can be emitted before the whole clip is finished, and few-step sampling, so each block is cheap. Conventionally, the streaming video literature obtains both capabilities from a chained pipeline. It first distills a bidirectional teacher into a causal student, then into a few-step one, or proceeds in reverse order. Each stage of such a chain fine-tunes the weights the previous one produced, so a later objective can undo an earlier capability. Following the idea of model merging, we show that on a packed audio-video backbone the two capabilities can be acquired in parallel. A causal adapter is trained against the frozen backbone, and an off-the-shelf few-step adapter provides the few-step capability. As the two edit different functional axes, we predict, and then verify, that their weight-update directions are near-orthogonal, without any explicit orthogonality constraint during training. Orthogonal updates should combine without interfering, so parallel composition is a direct sum. The two adapters are simply added at inference, with no joint training, yielding few-step, streaming audio-video whose image quality tracks the bidirectional teacher. Compared to the chained baselines, the composed model matches or beats them on most metrics, making parallel composition a practical approach. The resulting streaming system generates joint audio-video in real time, $\approx$26 fps at $480\times832$ without quantization, and sustains 30 s of continuous generation with stable image quality. Demo Page: https://pac-demo-2027.github.io/demo/

1 INTRODUCTION

Real-time joint audio–video generation calls for incremental output and low per-chunk cost, but conventional recipes acquire causalization and few-step sampling through sequential training stages. This paper composes independently trained adapters in parallel, reporting teacher-like appearance and stronger performance than chained baselines on most metrics.

Motivation

  • Real-time interactive applications need bounded time-to-first-frame, per-chunk cost independent of session length, and no length ceiling.Most existing joint audio–video models are bidirectional, requiring a fixed frame count and delaying output until sampling finishes.Source: §1 · p. 1

Problem

  • Chained recipes learn causalization and few-step generation in sequence, so the first stage changes the base on which the second stage trains.Prior approaches either train a causal few-step student directly or split the objectives across stages in either order.Source: §1 · p. 1

Approach

  • The causal adapter is trained against the frozen backbone, while an off-the-shelf few-step adapter is added at inference without joint or chained training.Both capabilities are learned in parallel rather than by training one adapter on weights altered by the other.Source: §1 · p. 2

Contributions

  • The parallel composition matches the bidirectional teacher on appearance and single-frame distribution, beats CHAINED-DA on most held-out metrics, and trails CHAINED-AD on lip sync.It reaches ≈26 fps at 480×832 and sustains 30 s of generation with stable image quality.Source: §1 · p. 2

2 RELATED WORK

Prior work spans joint audio–video generation, streaming diffusion, and weight-space model merging. This paper applies parallel adapter composition to jointly generated audio–video, unlike related approaches that use chained stages or address audio-driven video.

Joint audio–video generation

  • Joint audio–video models generate both modalities simultaneously; LTX-2 uses separate towers linked by cross-attention, while Cosmos 3 and MiniMax-H3 process both modalities in one DiT sequence.Audio-driven portrait animation instead conditions video on a supplied audio track.Source: §2 · p. 2

Autoregressive and streaming diffusion

  • Streaming diffusion methods include causal few-step students and multi-stage chains; OmniForcing applies the latter to joint audio–video, while Causal Forcing distills after autoregressive training.The paper’s CHAINED-DA follows OmniForcing’s order, and CHAINED-AD follows the reverse-order recipe.Source: §2 · p. 3

Model merging and adapter composition

  • Model merging combines independently fine-tuned weights, but this paper applies it to adapters trained under different conditions for audio–video generation and analyzes when their sum is safe.Avatar-Forever also merges parallel branches, but generates video from given audio and does not analyze when the sum is safe.Source: §2 · p. 3

3 METHOD

The method trains a causal adapter against a frozen audio–video backbone and composes it with an existing few-step adapter. It uses clean-context teacher forcing for causalization and motivates direct composition through near-orthogonal updates, then fuses and serves the adapters in a streaming setup.

3 METHOD

  • The backbone is MiniMax-H3 in its first-frame-conditioned audio–video configuration, with an off-the-shelf few-step adapter and a causal LoRA trained against the frozen weights.Both LoRAs are added at inference.Source: §3 · p. 3

3.1 LIMITATIONS OF THE CHAINED BASELINES

  • Chained training can un-distill a few-step adapter when plain flow-matching regression trains causalization on distilled weights; the reverse-order chain requires three stages and three objectives.Both chains also propagate earlier-stage errors and tie each adapter to the base on which it was trained, whereas the parallel route trains both capabilities against the same frozen backbone.Source: §3.1 · p. 3

3.2 CLEAN-CONTEXT TEACHER FORCING

  • Clean-context teacher forcing trains each noisy chunk using clean history, with duplicated clean and noisy token axes and independently sampled noise levels for chunks.The mask prevents a noisy chunk from attending to its own clean copy, and the loss is applied to the noisy half.Source: §3.2 · p. 4

3.3 WHY THE TWO ADAPTERS COMPOSE: AN ORTHOGONALITY WE PREDICT

  • The two adapters edit different functional axes: few-step distillation changes the denoising map across noise levels, while causalization changes context routing.The updates are small perturbations of the frozen weights, motivating a prediction of near-orthogonal directions; the analysis finds overlapping inputs but near-orthogonal writes.Source: §3.3 · p. 4

3.4 WEIGHT FUSION AND AUTOREGRESSIVE INFERENCE

  • The few-step adapter is fused into the backbone offline, while the causal adapter is loaded at inference, so the few-step adapter adds no runtime overhead.Four-step sampling still holds when the distilled trajectory runs with a block-causal mask and cached history.Source: §3.4 · p. 5
  • Streaming processes one chunk at a time within a sliding window of at most N chunks, retaining a first-frame anchor and evicting the oldest chunk once the window is full.Each window starts at frame zero to preserve absolute rotary positions, keeping per-chunk cost independent of session length; the prompt is re-encoded for each window.Source: §3.4 · p. 5

4 EXPERIMENTS

The experiments compare parallel composition with chained and bidirectional systems across held-out quality, public benchmarks, 30-second streaming, and update geometry. DECOUPLED preserves image quality while exposing trade-offs in lip sync and long-horizon speaker identity; its independently trained updates write in near-orthogonal directions.

4.1 EXPERIMENTAL SETUP

  • The MiniMax-H3 backbone stays frozen; only the rank-128 causal LoRA is trained on 5–10-second clips, while public lightx2v Turbo supplies four-step distillation.Inference uses four denoising steps per audio-video chunk and a seven-chunk context window.Source: §4.1 · p. 5
  • Held-out scores measure fidelity to teacher generations, whereas public-benchmark scores use real sources and are not comparable.The held-out benchmark contains unseen clips, and all systems share the same first-frame-conditioned audio-video setting.Source: §4.1 · p. 5

4.2 MAIN RESULTS

  • DECOUPLED keeps WHOLE-SEG image quality (IQA 0.914 against 0.915), beats CHAINED-DA on most held-out metrics, but trails CHAINED-AD in lip sync (Sync-C 7.443 against 8.021).CHAINED-DA's FID is 70.31 against the teacher's 67.10, while DECOUPLED's is 66.15.Source: Table 1 · p. 6, §4.2 · p. 6
  • On AVSpeech, VidChatBench, and HDTF, DECOUPLED's IQA is 0.866/0.866/0.892, ahead of the 50-step teacher and autoregressive baselines; it also leads autoregressive AV-IB on each set.It has the lowest autoregressive FID and FVD on AVSpeech and VidChatBench.Source: §4.2 · p. 6
  • In Figure 2's 30-second example, CHAINED-DA stops articulating early while the composed routes speak throughout.Table 2 reports the same comparison over all 40 clips.Source: Figure 2 · p. 7, Table 2 · p. 7
  • At 30 seconds, DECOUPLED closes with identity 0.710 and image quality 0.871, the latter 2.6% above its opening; CHAINED-AD closes at 0.484 and 0.526.CHAINED-DA's face identity remains 0.683, but its lip sync falls to 3.75 (−43.7%).Source: §4.2 · p. 6, Table 2 · p. 7

4.3 THE PREDICTED ORTHOGONALITY, VERIFIED

  • Updates share inputs but write near-orthogonally: signed alignment is not separated from null (+0.0006 against +0.0000, p=0.26), while read interference exceeds null 5.7×.The measured comparisons cover shared projections the adapters both edit; the test excludes feed-forward projections untouched by the causal adapter.Source: Table 3 · p. 8, §4.3 · p. 8
  • The causal adapter also serves on undistilled 50-step weights (DECOUPLED-50: Sync-C 7.197, IQA 0.894), so it is reusable beyond four-step distillation.The same adapter is used with distilled and undistilled weights without retraining.Source: §4.3 · p. 8

5 LIMITATIONS

The evaluation is limited to talking-head footage, making the reported causalization costs and their metrics specific to that domain. Long rollouts also expose speaker-identity drift, while transfer beyond the tested backbone remains unestablished.

5 LIMITATIONS

  • All training and evaluation corpora are talking-head footage, so the attributed lip–speech coupling and audio-distribution costs are domain-specific.The metrics measuring those costs are specific to this domain as well.Source: §5 · p. 8
  • On long rollouts, speaker identity drifts furthest for DECOUPLED because its causal adapter never sees its own rollouts.The few-step adapter learned audio with whole-sequence attention, whereas the composed model must preserve timbre from cached history; self-rollout training is proposed as future work.Source: §5 · p. 9
  • Whether parallel composition transfers beyond MiniMax-H3 remains untested, and OmniForcing was not compared because it uses a different backbone.A same-backbone comparison would require porting one of the two pipelines.Source: §5 · p. 9

6 CONCLUSION

The conclusion presents parallel addition of a causal adapter and an off-the-shelf few-step adapter as a route to streaming audio–video without joint training. The appendix describes the data, backbone, chunk geometry, stale-cache choice, and attention-mask checks used in the experiments.

6 CONCLUSION

  • The composed model tracks the bidirectional teacher on appearance and holds quality over rollouts longer than the adaptation clips.The adapters are added at inference without joint training or an orthogonality regularizer, and the causal adapter can be served on undistilled or distilled weights.Source: §6 · p. 9

A TRAINING AND HELD-OUT DATA

  • The corpus contains around 4,000 generated 5–10 s clips, with 211 held out and training and test data balanced across Chinese and English.Videos are center-cropped and resized to 480×832 or 640×640.Source: A TRAINING AND HELD-OUT DATA · p. 11
  • Held-out clips use the 17n+5 frame grid and are compared frame by frame with the backbone’s 50-step reference, which sees both keyframes.Streamed systems see only the first keyframe, making the reference a strict upper bound.Source: A TRAINING AND HELD-OUT DATA · p. 11

B BACKBONE, CHUNK GEOMETRY AND THE STALE CACHE

  • MiniMax-H3 is adapted in its first-frame-conditioned FL2VA configuration, with video, audio, text, and conditioning packed into shared transformer blocks.The video VAE maps 17 frames to 5 latent tokens, audio is a 40 Hz stereo latent, and the backbone stays frozen.Source: B BACKBONE, CHUNK GEOMETRY AND THE… · p. 12
  • The system uses 34-frame chunks despite the conservative 51-frame alignment choice, deriving audio indices from absolute frame numbers to prevent accumulating rounding error.The video VAE requires chunk size to be a multiple of 17, while exact alignment with 40 Hz audio at 24 fps requires divisibility by 3.Source: B BACKBONE, CHUNK GEOMETRY AND THE… · p. 12
  • Streaming re-encodes cheap text at each window but accepts stale cached history keys instead of re-encoding the full history.Training encodes clean chunks with full-clip text context, while cached streaming keys were produced under earlier, shorter windows and earlier text context.Source: B BACKBONE, CHUNK GEOMETRY AND THE… · p. 12

C MASK VERIFICATION

  • The mask check ensures noisy rows cannot attend their own clean copy or any later chunk, while condition rows remain visible to every chunk.The check guards against lowering loss through access to the answer in the noisy chunk’s clean copy.Source: C MASK VERIFICATION · p. 12

D METRIC IMPLEMENTATION CONTROLS

The metric implementation section records choices that affect reported scores or their interpretation. It cautions against some cross-paper comparisons and distinguishes metrics’ intended uses and limitations.

D METRIC IMPLEMENTATION CONTROLS

  • IQA uses CLIP-IQA-style antonym prompting on ViT-L/14, so its scores are not comparable across papers.Sync-C and Sync-D use a 25 fps / 100 Hz MFCC grid with BGR input and int16-amplitude MFCCs.Source: D METRIC IMPLEMENTATION CONTROLS · p. 12
  • DeSync’s 200 ms class spacing suits cross-paper alignment comparisons but not offsets of a few tens of milliseconds.AV-IB distinguishes speech from non-speech but barely identifies whose speech it is, serving as a sanity check on this corpus.Source: D METRIC IMPLEMENTATION CONTROLS · p. 12
  • Table 1 and Figure 3 use FaceNet for CSIM, whereas Figure 5 uses ArcFace and its absolute cosine scores should not be compared.Speech is transcribed with Qwen3-ASR-1.7B; language is forced from the prompt, and early finishes count as deletions in WER/CER.Source: D METRIC IMPLEMENTATION CONTROLS · p. 12

E WHY ADDITION WORKS: A SECOND-ORDER VIEW

A second-order expansion identifies the interaction between the two adapter updates as the combination cost, which vanishes when they are orthogonal in the Hessian metric. The reported Frobenius-based measurements are an estimate rather than proof, and the paper distinguishes this requirement from the basin-sharing rationale for model soups.

E WHY ADDITION WORKS: A SECOND-ORDER VIEW

  • The second-order loss expansion makes the Hessian cross-term the entire interaction cost of adding the two updates, and it vanishes for H-orthogonal updates.The expansion is around the frozen weights W and includes the losses for each update separately.Source: E WHY ADDITION WORKS: A SECOND-ORDER… · p. 12
  • The Frobenius measurements estimate the cross-term rather than prove it: Frobenius orthogonality is necessary but not sufficient for H-orthogonality.The signed alignment is not separated from the null, and write interference exceeds it only modestly; read interference concerns overlapping inputs, not the Hessian cross-term.Source: E WHY ADDITION WORKS: A SECOND-ORDER… · pp. 12–13
  • Unlike model soups, the two updates use heterogeneous objectives and one adapter is off the shelf, so basin sharing is not the rationale for addition.The relevant quantity is the small cross-term measured in §4.3.Source: E WHY ADDITION WORKS: A SECOND-ORDER… · pp. 13–14

F HOW THE ADAPTER GEOMETRY IS COMPUTED

The appendix describes low-rank geometry measures for adapter updates and reports a depth-wise size asymmetry for the causal update trained on distilled weights.

F HOW THE ADAPTER GEOMETRY IS COMPUTED

  • The reported geometry quantities reduce to r × r matrices; subspace overlap is measured from orthonormal bases and has expectation r/n for random r-dimensional subspaces of R^n.The cited p-values use paired, two-sided Wilcoxon signed-rank tests against a null computed on the same block-projection pairs.Source: F HOW THE ADAPTER GEOMETRY IS COMPUTED · p. 14
  • ∆′c is 27% smaller than ∆c overall, and averages 44% of its size in the first block quarter versus 94% in the last, concentrating its training budget late in the network.The comparison uses Frobenius norms.Source: F HOW THE ADAPTER GEOMETRY IS COMPUTED · p. 14
  • The adapter comparison covers shared attention projections only; roughly half of ∆d’s parameters are in feed-forward layers, which ∆c does not touch and which are excluded.Norms are averaged after computing ∥∆∥F /∥W∥F per block and projection.Source: F HOW THE ADAPTER GEOMETRY IS COMPUTED · p. 14

G PUBLIC BENCHMARKS

Table 4 presents the systems from Table 1 on three public test sets, with scores evaluated against the real source video.

G PUBLIC BENCHMARKS

H LONG-HORIZON SAMPLE

The 30-second rollouts are examined using identity similarity to fixed references, complementing the main text’s endpoint-change summary. Face identity holds under composition, while speaker identity shows the reported drift.

H LONG-HORIZON SAMPLE

  • Figure 5 tracks identity drift across 40 thirty-second clips, probing every five seconds against fixed references.Face similarity uses an ArcFace reference portrait; speaker similarity uses the clip’s opening segment with CAM++.Source: Figure 5 · p. 14
  • Face identity is retained equally well by DECOUPLED, DECOUPLED-50, and CHAINED-DA, and worst by CHAINED-AD.Source: Figure 5 · p. 14
  • Speaker identity falls furthest for DECOUPLED, the long-session drift reported as a limitation.Source: Figure 5 · p. 14, H LONG-HORIZON SAMPLE · p. 14

I INFERENCE COST

Streaming inference adds key/value prefill overhead, while a sliding window keeps per-chunk cost constant across a session. The reverse-order CHAINED-AD baseline uses two further distillation stages after causalization.

I INFERENCE COST

  • Streaming’s extra wall-clock cost is key/value prefill, not denoising; subtracting it makes the two modes agree to within a few percent.Whole-segment latency grows from 11.2 to 54.3 seconds, while sliding-window session cost per chunk remains constant.Source: I INFERENCE COST · pp. 14–15
  • Table 4 compares Table 1 systems on three public test sets against the real source; FAD is omitted on HDTF and DeSync is undefined there.The caption attributes the FAD omission to clip-length mismatch and the DeSync omission to clips shorter than Synchformer’s 4.8-second context.Source: Table 4 · p. 15
  • About 26 fps at 480×832 is sustained on six GPUs without quantization, exceeding 24 fps playback; time-to-first-frame is 4.2–4.3 seconds across lengths.The same timings apply to DECOUPLED and CHAINED-DA because they share an adapter shape.Source: I INFERENCE COST · pp. 14–15

J THE REVERSE ORDER: CHAINED-AD’S TRAINING STAGES

  • CHAINED-AD follows causalization with consistency distillation on ground-truth cache and distribution matching on self-forced rollouts, anchored by a small supervised-chunk MSE.Without the anchor, the critic absorbs high-frequency noise and stops penalizing it; the two distillation increments are merged for Table 3.Source: J THE REVERSE ORDER: CHAINED-AD’S… · p. 15

Related papers

Loading 2610.10343v2…