Joint and Cross-Modal Video-Audio Generation and Editing: A Unified Formulation and Design Taxonomy

1 Introduction

• The overview addresses models that generate audio and video in isolation by treating the streams as a coupled pair whose outputs must cohere in time and semantics.It presents a unified formulation, a five-axis design taxonomy, and a taxonomy of nine joint-editing categories spanning 28 edit types.Source: §1 · p. 1

2 Problem Formulation and Notation

• Correspondence requires semantic agreement between visible and audible sources and temporal alignment of acoustic events with the frames of their visual causes.A high-quality video and audio pair can still feel broken when its correspondence score is low.Source: §2 · p. 2
• Joint generation produces both streams from a condition, cross-modal generation produces one stream given the other, and joint editing returns a modified pair.The formulations require high audio-visual correspondence; editing additionally applies the instruction across modalities while preserving content outside the target.Source: §2 · p. 2

3.1 Generative Modeling Foundations

• The background presents variational autoencoders, GANs, autoregressive models, and diffusion or flow-matching models as recurring generative families, using denoising as its running example.The example gradually corrupts a clean latent into noise and learns to reverse that process, optionally conditioned on c.Source: §3.1 · p. 3

3.2 Video Representations

• Video representations include pixels, continuous latents, and discrete tokens; latent models compress video, while tokenization enables autoregressive modeling.These choices trade off fidelity, computational cost, and compatibility with the accompanying audio representation.Source: §3.2 · p. 3

3.3 Audio Representations

• Audio representations include waveforms, spectrograms, and latent codecs, which encode audio as discrete or continuous tokens and decode it back to a waveform.Audio and video representations must be reconciled for temporal alignment and, in unified backbones, shared processing.Source: §3.3 · p. 3, JOINT AUDIO-VIDEO EDITING · p. 5

3 Background and Preliminaries

• The paper organizes methods along five axes: generation strategy, audio representation, video representation, alignment enforcement, and pretrained-weight reuse.The axes are complementary, not fully independent; for example, guidance-based strategies require two frozen unimodal models and dual pretraining.Source: §1 · p. 1, §3.3 · p. 4

3.4 Conditioning Mechanisms

• Conditioning mechanisms include cross-attention, adaptive normalization, concatenation, and classifier-free guidance, which vary in how they inject c into a model.The same mechanisms can let one modality condition the other in joint settings.Source: §3.4 · p. 5

4 A Five-Axis Design Taxonomy

• Methods are classified along five axes: generation strategy, audio representation, video representation, alignment enforcement, and pretraining reuse.Each axis constrains a different component, but choices are not fully independent; some combinations are empty or near-empty.Source: §4 · pp. 5–6

4.1 Generation Strategy

• Generation strategies include single-tower sharing, dual-tower cross-modal connections, cascaded conditioning, unified-token sequences, and inference-time guidance.Single-tower models share parameters across concatenated or interleaved streams, while dual-tower models retain modality-specific towers and exchange information through cross-modal connections. Cascaded models generate one modality conditioned on the other; unified-token modeling has no joint instance covered, and guidance couples frozen unimodal models at sampling time.Source: §4.1.1 · p. 6, §4.1.2 · p. 6, §4.1.3 · p. 7

4.2 Audio Representation

• Continuous audio latents are the dominant representation, used by 28 of the 29 methods in Table 1.An audio autoencoder compresses the waveform into a continuous latent for diffusion or flow modeling, followed by decoding.Source: §4.2.1 · p. 7
• Joint methods covered leave audio codec tokens and waveform generation unused, while mel spectrograms are modeled as image-like arrays requiring a vocoder.Codec tokens support autoregressive or masked modeling; mel-spectrogram generation requires a separate vocoder to recover the waveform.Source: §4.2.2 · p. 8, §4.2.3 · p. 8, §4.2.4 · p. 8

4.3 Video Representation

• Video representations include dominant 3D-VAE latents, per-frame 2D-VAE latents with a temporal module, discrete tokens, and pixels.Nineteen methods use 3D-VAE representations; no joint method covered uses discrete video tokens, and MM-Diffusion is the sole pixel-space instance.Source: §4.3.1 · p. 8, §4.3.2 · p. 8, §4.3.3 · p. 8

4.4 Alignment Enforcement

• Alignment is imposed through cross-attention, shared positional encoding, discriminator or classifier guidance, or an explicit spatio-temporal prior.Cross-attention is the dominant mechanism, used by nineteen methods; the other mechanisms apply correspondence pressure through position, sampling guidance, or a separately estimated prior.Source: §4.4.1 · p. 8, §4.4.2 · p. 8, §4.4.3 · p. 9

4.5 Pretraining Reuse

• Methods either train from scratch on paired data, reuse one pretrained backbone, or reuse and couple two pretrained backbones.The cited examples include MM-Diffusion and Movie Gen for from-scratch training, ALIVE for single-backbone reuse, and CoDi for dual-backbone reuse.Source: §4.5.1 · p. 9, §4.5.2 · p. 9, §4.5.3 · p. 9

5.1 The Space of Audio-Visual Edits

• The taxonomy spans nine edit categories and 28 edit types, with one-stream edits leaving the other modality unchanged.It also distinguishes genuinely joint edits from video- or audio-driven edits and edits primarily confined to one modality.Source: §5.1 · p. 9, Table 4 · p. 11

5.2 Synchronization

• Dubbing is the developed synchronization edit: methods re-synchronize lips, jointly generate translated audio and facial motion, or perform identity-preserving re-voicing.Alignment and re-timing have no dedicated methods.Source: §5.2 · p. 9

5.3 Joint Content

• Joint content edits insert, remove, or replace objects in both streams, but current methods remain object-scoped; scene swaps and ambience insertions are undemonstrated.AV-Edit gates a masked-autoencoder-plus-DiT pipeline by audio-visual correlation, while Object-AVEdit uses inversion and regeneration.Source: §5.3 · p. 9

5.4 Cross-Modal Transfer

• Cross-modal transfer runs generation within editing pipelines, but automatic propagation from one modality and bidirectional transfer remain unrealized.The taxonomy wheel arranges the nine categories and their edit types radially, with sector size proportional to edit-type count.Source: §5.4 · pp. 9–10, Figure 3 · p. 10

5.5 Identity & Performance; 5.6 Scene & Environment

• Face-and-voice swaps, performance transfer, and joint emotion edits are reached only through dubbing; scene and environment changes requiring proportional audio edits lack targeted methods.Source: §5.5 · p. 10, §5.6 · p. 10

5.7 Narrative

• Narrative edits such as cut-level splicing, reordering, and summarization are unsupported; current methods edit within shots rather than reasoning across shots and modalities.Source: §5.7 · p. 10

5.8 Generative; 5.9 Quality & Restoration

• Joint inpainting and outpainting have no dedicated methods, while restoration can break correspondence and has no learned joint treatment for subsequent synchronization repair.Source: §5.8 · p. 11, §5.9 · p. 11

5.10 Cross-Cutting

• Text is the most general editing instruction: existing approaches adapt a joint model to one example or use zero-shot delta-denoising under frozen unimodal models.Source: §5.10 · p. 12

6 Joint Audio-Visual Generation; 6.1 Unconditional Joint Generation

• Joint generation models output video and audio together, and unconditional generation exposes their dependency most directly.Coupled diffusion over a paired latent is the canonical unconditional instance, MM-Diffusion.Source: §6 · p. 12, §6.1 · p. 12

6.2 Text-to-Audio-Visual Generation

• Text-to-audio-visual methods generate both streams to depict a prompt, using varied architectures and coupling strategies including dual towers, context learning, and backbone fusion.The section also covers open systems scaling joint training and partially documented systems.Source: §6.2 · p. 12

6.3 Image-to-Audio-Visual Generation

• Image conditioning pins identity and layout, leaving motion and sound consistent with the fixed first frame as the central challenge.Animate-and-Sound shares an expert block between video and audio branches.Source: §6.3 · p. 12

6.4 Talking-Head and Speech-Driven Generation

• Talking-head generation pairs a speaking face with speech from an identity image and driving speech; streaming avatars can attend to future phonetic cues.Hallo-Live uses a short horizon of future phonetic cues.Source: §6.4 · p. 12

6.5 Long-Form Joint Generation

• Long-form generation must prevent identity, scene, and audio-visual relationships from drifting as errors accumulate over minutes.OmniForcing generates causal blocks while distilling from a bidirectional teacher.Source: §6.5 · p. 12

7 Cross-Modal Generation

• Cross-modal generation outputs exactly one modality conditioned on the other, with the observed input held fixed.Source: §7 · p. 12

7.1 Video-to-Audio Generation

• Video-to-audio is by far the most developed setting, spanning learned correspondence, flow matching, token models, controlled generators, and instruction-based control.Its timing requirement is exact placement of sound onsets at visual contact frames.Source: §7.1 · p. 12

7.2 Audio-to-Video Generation

• Audio-to-video is structurally harder because one audio track can license many videos, and dedicated attempts are isolated.Source: §7.2 · p. 12

7.3 Speech-to-Video Generation

• Speech-to-video mainly concerns portrait animation, where correspondence reduces to matching visemes and phonemes; beyond portraits, the mapping is largely untouched.Source: §7.3 · p. 15

7.4 Foley and Sound-Effect Generation from Video

• Foley focuses on diegetic sounds of visible actions, where timing errors of a few frames are wrong and practical control over sources, timing, and loudness matters.Source: §7.4 · p. 15

8 Alignment and Synchronization

• Alignment uses contrastively trained audio and video encoders whose embeddings are close under cosine similarity, globally or per time step.These are distinct from the generative encoders; temporal synchronization requires corresponding events within tolerance δ.Source: §8 · p. 15

8.1 Temporal Synchronization

• Temporal synchronization is enforced through pretrained representations, dedicated timing machinery, or architecture; cross-attention is the dominant architectural choice, used by 19 methods.Inference-only alignment by discriminator or classifier guidance is now the exception.Source: §8.1 · p. 15

8.2 Semantic Alignment

• Semantic alignment matches source identity globally but does not ensure correct event timing; language-mediated alignment offers explicit control at the cost of temporal precision.AVGen-Bench reports a gap between strong audio-visual aesthetics and unreliable semantic grounding.Source: §8.2 · pp. 15–16

8.3 Lip-Sync and Phoneme-Level Alignment

• Lip-sync requires roughly single-frame tolerance, constrains talking-head generation, speech-to-video, and dubbing, and has the most established automatic metrics.Human sensitivity to phoneme-to-viseme mismatch is unusually sharp, and available metrics agree best with human judgment in this subproblem.Source: §8.3 · p. 16

8.4 Rhythmic and Beat-Level Alignment

• Music alignment is rhythmic: motion peaks should coincide with beats, while generating video motion from music remains comparatively unexplored.Source: §8.4 · p. 16

8.5 Consistency Across Long Sequences

• Long-horizon alignment must be maintained because small errors accumulate into drift; current approaches use streaming blocks or extended generation windows.Both remain evaluated on horizons far shorter than the minutes-long content envisioned for applications.Source: §8.5 · p. 16

9 Architectures and Training Strategies

• Cascaded models generate one modality first and condition the other on it, while unified backbones learn the joint distribution through shared processing.Cascades are common in cross-modal generation; unified backbones are more common in joint generation.Source: §9 · p. 16

9.4 Autoregressive and Token-Based Approaches

• Autoregressive and masked-token models make streaming and variable-length generation natural, but joint discrete-token generation remains unoccupied in the surveyed design space.Input reordering also supports a single backbone across multiple tasks.Source: §9 · p. 17, §9.4 · p. 17

9.1 Two-Tower and Cascaded Models

• Two-tower and cascaded models reuse strong unimodal backbones with added coupling, gaining data efficiency at the cost of coordination between networks.They couple modalities through cross-modal connections or a generation order.Source: §9.1 · p. 17

9.2 Unified Multimodal Backbones

• Unified backbones process both modalities with shared parameters, capturing their dependency directly but requiring one representation to serve both.They may use a fused sequence or a single network with modality-specific heads.Source: §9.2 · p. 17

9.3 Diffusion-Based Approaches

• Joint diffusion and flow-matching models denoise paired latents using shared layers or cross-attention, while retaining classifier-free guidance.Source: §9.3 · p. 17

9.5 Training Objectives and Losses

• Joint methods add correspondence-targeting objectives beyond reconstruction or denoising losses, including contrastive alignment, adversarial realism, and synchronization preferences.These objectives are the training-time counterpart of the alignment-enforcement design axis.Source: §9.5 · p. 17

10.1 Audio-Visual Generation Datasets

• Generation datasets pair video and audio, increasingly with captions describing their relationship; dataset scale, domain, and caption quality bound what models can learn about correspondence.Table 6 lists representative datasets by domain, approximate scale, caption availability, and directly supported task.Source: §10.1 · p. 17, Table 6 · p. 18

10.2 Audio-Visual Generation Benchmarks

• Benchmarks fix prompts and protocols for comparison, but no shared joint audio-visual editing benchmark exists, leaving current editing results mutually incomparable.The surveyed editing sets lack supervision in the form of paired inputs, edit instructions, and target pairs.Source: §10.2 · p. 17, Table 7 · p. 18

11 Evaluation Metrics

• Evaluation requires per-modality quality and cross-modal consistency measures; edited pairs additionally require measures of instruction faithfulness and preservation of untouched content.Table 8 organizes metrics by what they measure.Source: §11 · pp. 17–18

11.1 Video Quality and Fidelity; 11.2 Audio Quality and Fidelity

• Video and audio quality metrics assess realism and prompt faithfulness, but neither stream's score alone establishes correspondence with the other.Source: §11.1 · p. 18, §11.2 · p. 18

11.3 Cross-Modal Alignment Metrics

• Alignment metrics assess semantic agreement through contrastive embeddings or temporal agreement through onset and beat distances.They distinguish joint and cross-modal evaluation from unimodal evaluation.Source: §11.3 · p. 18

11.4 Editing Faithfulness and Preservation

• Editing evaluation pairs instruction faithfulness with preservation of untouched regions, because the two requirements are in tension.Reporting either measure alone is misleading.Source: §11.4 · p. 18

11.5 Human Evaluation Protocols

• Human evaluation remains the reference standard for perceptual correspondence, typically using forced-choice comparisons of quality and synchronization.Automatic metrics are validated by agreement with human judgments.Source: §11.5 · p. 18

12.1 Film, Animation, and Content Creation

• Content creation prioritizes controllability and quality across new-clip generation and editing, often with soundtracks combining speech, effects, and music.Source: §12.1 · p. 19

12.2 Dubbing, Translation, and Accessibility

• Dubbing and accessibility require coordinated audio-video changes that preserve identity or source content; a wrong edit can be worse than no edit.JUST-DUB-IT jointly generates translated speech and matching facial motion while holding speaker identity fixed.Source: §12.2 · p. 19

12.3 Virtual Avatars and Telepresence

• Avatars and telepresence impose a hard latency constraint: generation must keep pace with speech, alongside roughly single-frame lip-sync tolerance.Streaming methods address this budget, and SoundReactor is the only method in Table 10 marked online.Source: §12.3 · p. 19

12.4 Personalization

• Personalization spans applications by applying identity, voice, or style consistency to generation or editing, while increasing the ethical exposure to impersonation.Source: §12.4 · pp. 19–20

13 Open Problems and Future Directions

• Open problems include scaling without multiplicative data and compute, maintaining long-horizon coherence, fine-grained event control, and physical plausibility.The paper also notes that synchronized clips can remain physically wrong and that joint models often fail physical-commonsense benchmarks.Source: §13 · p. 20
• Other gaps include underexplored task-design combinations, unreliable correspondence metrics, and the need to treat provenance and defenses across both streams.Source: §13 · p. 20

14 Conclusion

• The conclusion frames joint generation, cross-modal generation, and editing as problems over one audio-visual distribution, organized by a five-axis taxonomy.It identifies token-based joint generation, dedicated audio-to-video methods, and learned narrative and restoration editing as absent or nearly absent.Source: §14 · p. 20

Limitations

• The taxonomy covers documented methods through mid-2026 and excludes fully closed commercial systems whose representations and synchronization mechanisms are undisclosed.Its five design axes are complementary, and ambiguous assignments sometimes require judgment.Source: Limitations · p. 20

A Other Taxonomies

• The taxonomies organize methods by task, conditioning signal, audio target, controls, deployment or editing constraints, and dominant modeling mechanism.They cover unified task-level comparisons, fine-grained video-to-audio and Foley methods, and mechanisms including diffusion, autoregression, alignment losses, and MLLM/agent-style reasoning.Source: Table 9 · p. 30, Table 10 · p. 31, Table 11 · p. 32
• Video-editing methods are grouped into temporal-adaptation, training-modification, and conditioning-branch families, with controls, edit targets, and implementation strategies distinguished.A continuation classifies attention injection, motion-feature injection, latent manipulation, canonical representation, point/pose conditioning, human animation, and human-object animation.Source: Table 12 · p. 33, Table 13 · p. 34
• Table 14 includes commercial and foundation systems because they materially define current user-facing capabilities for video generation with audio and audio-for-video editing, even when model details or weights are not fully released.The listed examples label Movie Gen as a research model, Google V2A as a research system, and Veo 3/3.1 as a commercial/product.Source: Table 14 · p. 35, A Other Taxonomies · p. 35

• Sora 2 is listed as a commercial/product entry for synchronized dialogue and sound effects.Source: A Other Taxonomies · p. 35
• Adobe Firefly Audio is listed as a commercial creative tool for sound effects, soundtrack, and speech.Source: A Other Taxonomies · p. 35
• Seedance 2.0 (ByteDance Seed, 2026) is listed as a commercial/product entry supporting text, image, video, and audio prompts.Source: A Other Taxonomies · p. 35
• Table 15 reports code and artifact availability, with each method’s row checked individually in July 2026.Only functional official releases with a listed repository count as available; announcement placeholders and samples-only or dataset-only repositories count as unavailable. Availability changes quickly.Source: Table 15 · p. 36