PaperlayerView exact revision on arXiv

Source-linked AI summary

UniAVC: Unified Audio-Video Creation via Progressive Training and Asymmetric Scheduling

Kien T. Pham, Yuhao Liu, Chong Hou Choi, Zhenqi He, Yanghao Wang, Chenyang Lei, Rui Liu, Qifeng Chen, Long Chen

arXiv:2610.24791v2cs.CVcs.MMcs.SDeess.AS

TL;DR

Scarce paired data and shared noise schedules make unified audio-video creation challenging, especially for asymmetric cross-modal workflows. UniAVC trains unimodal capabilities before paired-data alignment and uses independently warped schedules with clean-branch supervision. It reports competitive performance across supported tasks, while remaining bounded by its noise-space coverage and computational demands.

  • Problem

    Paired audio-video corpora are scarce, and shared noise schedules overlook modality-specific denoising trajectories and do not train asymmetric conditioning workflows.

    Source: §1 · pp. 1, 3

  • Method

    UniAVC first trains unimodal generation and editing, then uses paired data for cross-modal bridges, with independently warped modality schedules and clean-branch subtasks.

    Source: §1 · p. 3

  • Results

    UniAVC achieves performance competitive with selected state-of-the-art baselines across all supported creation modes.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Intent-aware editing preserves one modality as a clean reference, while self-refinement selectively regenerates an unsatisfactory branch against an accepted one.

    Source: §5 · p. 9

    Inference precludes real-time deployment because requests require full denoising trajectories across both streams, and self-refinement adds a second pass over the target modality.

    Source: A2.4 LIMITATIONS · p. 18

Abstract

from arXiv · show

Unified audio-video creation requires a single model capable of text-conditioned joint synthesis, bidirectional cross-modal generation, and instruction-guided editing across both modalities. Existing methods rely heavily on paired corpora, which are scarce for joint generation and even rarer for editing. Consequently, they are spent learning single-branch generative and editing capabilities that abundant unimodal data could readily teach, rather than on the cross-modal coupling. Furthermore, prevailing systems apply a single shared noise schedule to both modalities, neglecting two critical limitations. First, due to the significant discrepancy in information density, video requires prolonged high noise to organize spatial and motion structure, whereas the sparser audio signal can settle earlier during the synthesis process, the dynamics a uniform schedule might compromise. Second, lockstep denoising leaves asymmetric workflows, such as conditioning on one modality to synthesize or edit another, untrained. We present UniAVC, which addresses both obstacles. It first trains each branch on unimodal dataset to full generative and editing competence, and subsequently dedicates paired data to exploit cross-modal correspondence. Modality-decoupled schedulers meanwhile warp the noise schedule independently from a shared timestep, while subtasks with one clean modality directly supervise asymmetric configurations. These settings respectively enable intent-aware editing, which preserves one modality intact while enlisting it to constrain the other, and post-inference self-refinement, which regenerates only an unsatisfactory branch against an accepted one. UniAVC thus spans multiple creation tasks from a single set of weights without task-specific adapters.

1 INTRODUCTION

UniAVC unifies audio-video generation and editing, addressing scarce paired supervision and shared noise schedules that fail to account for modality differences. It builds unimodal capabilities first, then uses paired data and asymmetric scheduling to support cross-modal creation and editing.

1 INTRODUCTION

  • Paired audio-video corpora are scarce, and using them to teach basic single-modality capabilities leaves less supervision for learning cross-modal coupling.Unimodal datasets already provide abundant examples for isolated visual motion and acoustic signals.Source: §1 · p. 2
  • A shared noise schedule mismatches video’s prolonged structural denoising with audio’s earlier convergence, while training only symmetric noise states leaves asymmetric conditioning workflows untrained.The resulting inference mismatch affects workflows such as editing video while preserving audio or generating video from clean audio.Source: §1 · pp. 2–3
  • UniAVC first trains each modality on unimodal data, then freezes both branches and uses paired data exclusively to train lightweight cross-modal bridges.Modality-decoupled schedulers warp noise trajectories independently from a shared timestep, while clean-branch subtasks supervise boundary configurations.Source: §1 · p. 3
  • The resulting asymmetric settings support intent-aware editing with one modality preserved as a clean reference and self-refinement that regenerates an unsatisfactory branch against an accepted one.These modes are enabled by warped noise trajectories and clean-branch supervision.Source: §1 · p. 3
  • UniAVC performs text-conditioned joint generation, bidirectional cross-modal synthesis, and instruction-guided editing of either or both modalities without task-specific adapters.The framework uses one set of weights for these creation modes.Source: Figure 1 · p. 2, §1 · p. 3
  • Experiments report competitive performance against selected state-of-the-art baselines across all supported creation modes.The introduction reports broad competitiveness without specifying numerical results.Source: §1 · p. 3

2 RELATED WORK

Prior work treats joint generation, cross-modal synthesis, and editing through largely distinct systems, while existing approaches to noise variation differ in how they couple modalities. UniAVC combines unimodal pretraining with paired-data alignment and schedules anchored to a shared step counter.

Joint Audio-Video Generation

  • Joint audio-video generation methods use shared architectures, align separate generators at inference, fuse modality backbones or experts, estimate priors, or train end-to-end models, but retain a shared noise schedule.Cross-modal models instead generate one stream conditioned on the other.Source: §2 · pp. 3–4

Joint Audio-Video Editing

  • Audio-video editing methods address scarce paired data through curation, self-supervised inpainting, joint editing, per-modality adapters, or shared representations; UniAVC learns single-modality capabilities from unimodal corpora.UniAVC reserves paired supervision for cross-modal coupling.Source: §2 · p. 4

Non-uniform Noise Schedules

  • Prior noise-schedule work varies noise across positions or modalities, but independent per-stream sampling can break the link between noise pairs and the inference step counter.UniAVC instead uses dedicated modality schedules anchored to a shared counter, with clean-branch subtasks supervising boundary states.Source: §2 · p. 4

3 METHOD

UniAVC unifies audio-video generation and editing by first building unimodal competence, then using paired data for cross-modal coupling. Modality-specific noise schedules and clean-branch training support asymmetric generation, editing, and refinement.

3.1 PRELIMINARY

  • Tasks are selected by reference presence and branch loss masks; independent noise levels corrupt target latents along rectified-flow paths.References are absent for generation from scratch, while one branch can be held clean as a conditioning signal.Source: §3.1 · p. 4

3.2 UNIFIED GENERATION AND EDITING

  • Separate video and audio transformer branches communicate through noise-conditioned cross-attention bridges, while masked in-context references unify generation and editing.Reference queries are restricted to uncorrupted reference spans rather than noisy targets.Source: §3.2 · p. 5, Figure 3 · p. 5

3.3 PROGRESSIVE TRAINING PARADIGM

  • Unimodal pretraining establishes each branch’s generation and editing competence, then paired finetuning updates only bridges and adaptive normalization parameters.Independent text/reference dropout supplies separate guidance axes, and null-edit reconstruction supervises source preservation.Source: §3.3 · p. 6

3.4 ASYMMETRIC MODALITY-DECOUPLED SCHEDULERS

  • A shared step index maps through modality-specific warped schedules; setting kv>ka lets audio resolve ahead of video, while clean-branch subtasks train asymmetric conditioning.The clean-branch regimes support conditioning on one modality while supervising the other; the unified index traverses continuous off-diagonal noise pairs.Source: §3.4 · pp. 6–7

3.5 SELECTIVE REGENERATION

  • Pinning one modality at σ=0 enables exact-preservation intent-aware edits; self-refinement instead pins an accepted first-pass branch while regenerating the other.Self-refinement re-noises the unsatisfactory generated branch and runs a second denoising trajectory.Source: §3.5 · p. 7

4 EXPERIMENTS

UniAVC is evaluated on joint editing, joint generation, and cross-modal generation benchmarks, with ablations examining progressive training, inference modes, and scheduler shifts.

4.1 EXPERIMENTAL SETUP

  • The experiments use JAVEditBench and held-out InsAVE-80K for joint editing, Verse-Bench for joint generation, and VGGSound for cross-modal generation.The implementation uses separate unimodal training followed by alignment finetuning on paired samples.Source: §4.1 · pp. 7–8

4.2 MAIN RESULTS

  • UniAVC performs competitively across supported tasks, outperforming prevailing methods on most joint audio-video editing metrics while achieving robust joint and cross-modal generation results.Qualitative comparisons report instruction following, perceptual fidelity, and preservation of character identity and unedited regions.Source: §4.2 · p. 8

4.3 ABLATION STUDY

  • Progressive training outperforms single-stage finetuning across all metrics, especially cross-modal AVQ and AQ metrics.The authors attribute this pattern to reserving paired data for the second stage to train cross-modal alignment.Source: §4.3 · p. 8
  • Intent-aware editing improves audio quality using uncorrupted source latents, while self-refinement yields the largest instruction-compliance gains by re-noising one branch against an accepted counterpart.Combining the modes achieves peak cross-modal synchronization and overall performance.Source: §4.3 · p. 9
  • Removing scheduler asymmetry notably reduces cross-modal AQ and AVQ, with minimal impact on visual quality; excessive shift strengths degrade most metrics.The reported ablation indicates schedule warping saturates beyond a moderate shift.Source: §4.3 · p. 9

5 CONCLUSION

UniAVC unifies audio-video generation and editing in one framework, using progressive training and asymmetric schedulers to support multiple creation modes. Its training coverage remains bounded, and cross-modal bridges do not repair unimodal deficits.

5 CONCLUSION

  • UniAVC supports text-conditioned joint generation, bidirectional cross-modal synthesis, and instruction-guided editing of either or both streams without task-specific adapters or secondary checkpoints.Progressive training builds unimodal competence before paired-data training of cross-modal bridges; asymmetric schedulers and clean-branch subtasks support off-diagonal and boundary configurations.Source: §5 · p. 9
  • Intent-aware editing preserves accepted modalities, while self-refinement selectively regenerates flawed modalities.These operational modes differ in conditioning and supervision rather than architecture.Source: §5 · pp. 9–10

A1 Additional Implementation Details 16

The appendix contents list training pools and sampling, subtask mixtures, and evaluation metrics.

A1 Additional Implementation Details 16

  • The listed additional implementation topics are training pools and sampling, subtask mixtures, and evaluation metrics.The contents assign these topics to appendix sections A1.1–A1.3.Source: A1 Additional Implementation Details 16 · p. 15

A2 Additional Experimental Results 17

The appendix contents list additional experimental results on qualitative and quantitative results, ablations, and limitations.

A2 Additional Experimental Results 17

A1.1 TRAINING POOLS AND SAMPLING

Training separates unimodal competence from cross-modal alignment: branches first learn generation and editing on modality-specific corpora, then paired samples train alignment.

A1.1 TRAINING POOLS AND SAMPLING

  • Training has two stages: independent unimodal pretraining establishes branch capabilities, and paired-data alignment finetuning establishes cross-modal alignment.Unimodal pretraining uses no paired audio-video samples; alignment finetuning uses paired data.Source: A1.1 TRAINING POOLS AND SAMPLING · p. 16
  • Video pretraining draws 34.3M samples from a 15.3M pool, retaining still-image tasks and balancing image and video sampling probabilities.Each image task has probability 0.33 and each video task 0.17 because image samples consume fewer tokens than 10-second video clips.Source: A1.1 TRAINING POOLS AND SAMPLING · p. 16
  • Audio pretraining draws 54.9M samples from a 14.9M pool and reweights sampling toward domain-level parity because speech samples outnumber environmental sounds roughly 5:1.The two audio tasks are sampled with equal probability of 0.50.Source: A1.1 TRAINING POOLS AND SAMPLING · p. 16
  • Alignment finetuning draws 0.67M paired samples from a 0.74M pool, allocating 0.30 to joint generation and 0.70 to joint editing.The draws complete roughly 0.90 of a single pass over the pool.Source: A1.1 TRAINING POOLS AND SAMPLING · p. 16

A1.2 SUBTASK MIXTURES

The appendix details task mixtures, benchmark measures, qualitative and quantitative outcomes, ablations, and limitations of UniAVC.

A1.2 SUBTASK MIXTURES

  • UniAVC's task mixture combines standard, reconstruction, and clean-branch objectives for joint generation, and standard and self-refinement objectives for joint editing.The joint-generation weights are 0.7 standard, 0.1 reconstruction, and 0.1 for each clean-branch subtask; joint editing uses 0.8 standard and 0.1 for each self-refinement subtask.Source: A1.2 SUBTASK MIXTURES · pp. 16–17

A1.3 EVALUATION METRICS

  • Benchmark measures differ: JAVEditBench assesses edit quality and preservation, InsAVE-80K spans modality and cross-modal alignment, VGGSound uses direction-specific FVD and FAD, and Verse-Bench includes motion, aesthetics, sound, lip-sync, and WER.The same abbreviation can denote different quantities across benchmarks.Source: A1.3 EVALUATION METRICS · p. 17

A2.1 QUALITATIVE RESULTS

  • Qualitative comparisons cover text-prompted joint generation and instruction-guided editing across five task families, including preservation of unmentioned modalities.The generation figure shows both streams produced from text prompts alone; editing comparisons use matched inputs.Source: A2.1 QUALITATIVE RESULTS · p. 17

A2.3 ABLATIONS

  • Intent-aware editing and self-refinement support high-quality target-branch editing while preserving the non-target counterpart and their alignment.Figure A3 illustrates both inference modes on matched inputs.Source: A2.3 ABLATIONS · p. 18, Figure A3 · p. 19

A2.4 LIMITATIONS

  • Training and inference incur substantial compute overhead; inference is not real-time, and few compositional requests in paired corpora constrain multi-goal editing.Self-refinement adds a second denoising pass over the target modality.Source: A2.4 LIMITATIONS · p. 18

A2.2 QUANTITATIVE RESULTS

Related papers

Loading 2610.24791v2…