CAVC: Convergence-Adaptive Audio-Visual Cache for Modality-Asymmetric Diffusion Generation

Motivation

• Existing acceleration methods can harm video detail and temporal dynamics by shortening refinement, while modality-agnostic caching treats audio and video uniformly despite their different recoverability.This motivates asking whether full joint audio-video computation is needed at every denoising step.Source: §1 · p. 1

Results

• 1.10–1.35× speedups across AVGen-Bench, VABench T2AV, and General-80 accompany improved visual dynamics and competitive audio fidelity and audio-visual alignment.The reported results also include improvements in several visual dynamics and temporal statistics.Source: §1 · p. 2

Motivation

• Audio drift from mid-stage caching can be substantially reduced by a few terminal joint steps, whereas video degradation can leave persistent losses in motion and temporal detail.Branch-freezing ablations show high raw speedups but severe video-dynamics degradation when the video branch is cached or frozen.Source: §1 · p. 2

Approach

• CAVC bypasses recoverable audio computation and cross-modal attention during the middle stage while preserving video refinement, then restores minimal joint denoising for synchronization.The framework reuses cached audio denoiser output and applies a few terminal joint steps.Source: §1 · p. 2

2.1 Audio-Visual Diffusion Generation

• Dual-branch audio-visual systems fuse audio and video latents through dense cross-modal attention, requiring repeated branch and interaction execution at every denoising step.Examples include MM-Diffusion [Ruan et al. (2023)] and CoDi [Tang et al. (2023)].Source: §2.1 · pp. 2–3

2.2 Acceleration of Diffusion Models

• Step-reduction methods such as solvers and distillation, and per-step methods such as token merging and pruning, can degrade visual dynamics when applied uniformly to audio-visual generation.CAVC instead preserves video sampling steps while selectively approximating recoverable audio computation.Source: §2.2 · p. 3

2.3 Modality-Agnostic Feature Caching

• Existing caching methods exploit temporal redundancy but generally cache audio and video uniformly; CAVC applies convergence-adaptive caching only to the recoverable audio branch.The related work names DeepCache [Ma, Fang, and Wang (2024)] as a feature-caching example.Source: §2.3 · p. 3

3.1 Asymmetric Problem Formulation

• CAVC exploits audio-side drift that a few terminal joint steps can reduce, whereas interrupted video refinement causes persistent motion and temporal degradation.This is an empirical recoverability distinction, not a claim that audio predictions numerically converge faster.Source: §3.1 · p. 4

3.2 Convergence-Adaptive Audio-Visual Cache

• After adaptive audio-cache triggering, CAVC reuses the audio denoiser output and bypasses audio denoising and cross-modal attention while video refinement continues.It then restores full joint denoising, which empirically reduces cross-modal discrepancies and improves synchronization.Source: §3.2 · p. 4

3.3 Theoretical Analysis

• Under local smoothness assumptions, the analysis bounds latent deviations by residual errors and cache duration, then characterizes output-error propagation through terminal refinement.The bounds characterize perturbation; they do not claim perfect recovery or require terminal updates to be contractive.Source: §3.3 · p. 5

3.4 Complexity Analysis

• 1.10–1.35× wall-clock acceleration is achieved while terminal refinement remains unchanged.Shared Phase-C and fixed costs remain in both speedup terms, making measured gains smaller than Stage-1 branch-level savings.Source: §3.4 · p. 6

4.1 Experimental Setup

• Experiments use AVGen-Bench, VABench T2AV, and General-80, comparing CAVC with Full AV, Cache-DiT, and TeaCache under matched generation specifications on one NVIDIA H20.Complete settings and evaluation protocols are provided in the supplement.Source: §4.1 · p. 6

4.2 Main Results: Quality-Efficiency Trade-off

• 1.10–1.35× wall-clock speedups accompany Dynamic Degree +34.7% (General-80), Flicker -16.3% (AVGen-Bench), DNSMOS -1.1% (AVGen-Bench), and onset-based AV-F1 +5.0%–+19.7%.Fine-grained AV Sync still has minor residual drops on some benchmarks.Source: §4.2 · p. 6

4.3 Ablation Studies

• Terminal recoupling closes 81.5–91.3% of the audio CLAP gap to Full AV, while video Dynamics recovery after caching ranges from −16.7% to 0.0%.The comparison supports preserving video refinement while allocating caching to audio.Source: §4.3 · p. 7
• Freezing the video branch or both branches yields ∼3× speedups but lowers Dyn. to 0.460/0.440; audio-only caching reaches 0.700 (+59.1%).Audio-only caching preserves video refinement.Source: §4.3 · p. 7
• Adaptive triggering reaches 1.19× speedup and onset-based AV-F1 0.674, while fixed 80% triggering yields 1.01×; θ = 0.30, W = 3 gives 1.18×.The stricter θ = 0.15 setting triggers at Audio@ kτ = 14.9 and achieves 1.13×; the default triggers at Audio@ kτ = 5.5.Source: §4.3 · p. 7

• CAVC preserves the full video branch, caches recoverable audio computation, bypasses AV attention mid-stage, and restores brief joint denoising at the end.The framework is training-free and uses sandwich-structured computation allocation.Source: §5 · p. 7
• Audio-side drift from mid-stage caching can be reduced by a few terminal joint steps, whereas video-side degradation from interrupted refinement is difficult to recover.Source: §5 · p. 7
• Experiments report 1.10–1.35× wall-clock acceleration, improved visual dynamics and temporal statistics, and competitive audio fidelity and AV alignment.Source: §5 · p. 7
• Maximum speedup is bounded by video-side computation; continuously shifting acoustic events delay caching, bringing efficiency toward the baseline sampler.Minor AV synchronization offsets in rapid multi-event scenes can often be reduced by increasing Phase C steps at marginal cost.Source: §5 · p. 8

• Table 5 records the generation settings shared by Full AV, CAVC, Cache-DiT, and TeaCache.Source: §A.1 · p. 10, Table 5 · p. 10
• Each sample is generated independently on one NVIDIA H20 GPU with 96 GB memory.Source: §A.1 · p. 10
• CPU offload is enabled for all four methods in the reported runs.Source: §A.1 · p. 10

A.2 Acceleration-Method Configuration

• Cache-DiT uses its official LTX2 implementation, while TeaCache uses a public implementation adapted to the LTX2.3 dual-stream pipeline.Frozen repository snapshots and the TeaCache adaptation patch are recorded in the anonymous code artifact manifest.Source: §A.2 · p. 10
• TeaCache reuses a residual while accumulated polynomial-mapped dt remains below 0.20; otherwise it runs a full transformer evaluation and resets the accumulator.Source: §A.2 · p. 10

B General-80 Prompt Set

• General-80 contains General-40 and AV-hard-40; General-40 spans natural scenes, cities, people, animals, sports, cooking, industry, transportation, performances, and dynamic physical events.Source: B General-80 Prompt Set · p. 10
• Prompts are selected for broad visual coverage, observable sound sources or event times, balanced AV-hard categories, and single- and multi-event scenes; selection is deterministic.The complete prompts and split identifiers are in the anonymous code artifact.Source: B General-80 Prompt Set · p. 10

C.1 VBench Visual Metrics

• VBench reports six visual dimensions and their evaluators; all scores are higher-is-better, but Dynamic degree measures motion rather than overall perceptual quality.The listed dimensions and evaluators appear in Table 8.Source: §C.1 · p. 10, Table 8 · p. 10

C.2 Audio Quality and MLLM Evaluators

• DNSMOS estimates calibrated overall audio MOS, CLAP compares audio with its prompt, and Qwen2.5-Omni scores audio reality, visual reality, and expressiveness.DNSMOS uses mono 16 kHz audio; CLAP uses mono 48 kHz audio, and Qwen2.5-Omni returns integer scores from 1 to 5.Source: §C.2 · pp. 10–11

C.3 Audio-Visual Alignment and Physics

• AV Sync maps temporal offset to a higher-is-better score, VideoPhy2 assesses local physical plausibility, and AV-F1 measures agreement between detected audio and visual event times.AV-F1 is a handcrafted no-reference diagnostic, not a learned semantic evaluator.Source: §C.3 · p. 11

C.4 Pure-Pixel No-Reference Diagnostics

• Pure-pixel diagnostics evaluate resized videos over at most 128 frames and, except for AV-F1 audio-onset extraction, require no pretrained weights.Their definitions are summarized in Table 9.Source: §C.4 · p. 11, Table 9 · p. 11

D Audio Cache Trigger Curves

• Caching activates online after the audio residual remains below threshold θ for W consecutive execution steps, making this an online cacheability criterion rather than a claim of faster numerical convergence than video.Table 10 reports observed trigger locations using forward execution index kτ for default θ = 0.30 and W = 3.Source: D Audio Cache Trigger Curves · p. 11, Table 10 · p. 11

E Validity of Cross-Noise-Level Prediction Reuse

• During Phase B, CAVC reuses the cached audio denoiser output at lower noise levels while the scheduler continues updating the audio latent.The audio latent and noise schedule are not frozen.Source: E Validity of Cross-Noise-Level… · p. 11
• The conditional deviation bound describes prediction-reuse effects but does not assert that the cached direction is exact; practical suitability is evaluated empirically.The evaluation uses residual curves and terminal-refinement ablations.Source: E Validity of Cross-Noise-Level… · p. 11

F.1 Problem Formulation and Assumptions

• The formulation compares exact joint-sampler latents with CAVC latents and defines cumulative deviation using their L2 distances.Figure 5 empirically measures residual behavior; the stated inequalities are assumptions over the evaluated trajectory, not consequences of the trigger threshold alone.Source: §F.1 · pp. 11–12, Figure 5 · p. 12

F.2 Single-Step Deviation Recurrence

• Video error amplification includes the factor (αt + βtLg), reflecting compounding from evaluating a drifted state through the approximated video network.The video discrepancy includes both removing audio conditioning and applying the altered network to a drifted state.Source: §F.2 · pp. 12–13

F.3 Multi-Step Accumulation

• The single-step recurrences are unrolled across the cached interval, with the Phase B audio deviation depending on the assumed scheduler constants and interval accumulation.The exact and approximate states agree at the beginning of Phase B because Phase A includes the full update at trigger step τ.Source: §F.3 · p. 13

F.4 Terminal Joint Propagation

• Terminal joint updates propagate product-state deviation under a local Lipschitz assumption, which allows either contraction or amplification.Table 4 empirically measures the behavior of the actual terminal updates.Source: §F.4 · p. 13

F Detailed Theoretical Derivation of the Output Deviation Bound

• The output-deviation bound depends on bounded prediction residuals and local Lipschitz assumptions, so it is not an unconditional perceptual-quality guarantee.The bound makes dependence on audio and video residuals, cache duration, and assumed local constants explicit.Source: F Detailed Theoretical Derivation of… · p. 11, §F.5 · p. 13

F.5 Output Deviation Bound

• The final scalar-output bound combines latent deviations with local assumptions on decoders and the evaluation function, but guarantees neither perceptual improvement nor absence of failure modes.Its dependence on residuals, caching duration, and local constants is explicit.Source: §F.5 · p. 13