PaperlayerView exact revision on arXiv

Source-linked AI summary

SepGen: Multi-Stem Audio-Video Separation and Generation in a Single Model

Aviad Dahan, Rajaei Khatib, Yonatan Bitton, Idan Szpektor, Lior Wolf, Raja Giryes

arXiv:2610.11361v1cs.SDcs.CV

TL;DR

Joint audio-video generators make scenes with a single mix, leaving sources unavailable as individual waveforms for novel-viewpoint rendering. SepGen jointly generates video, a mix, and captioned stems, or extracts stems from an observed scene. It outperforms language-conditioned separators on reported comparisons, with the lead persisting when spoken lines are removed from captions.

  • Problem

    Joint audio-video generators emit one audio-mix whose sources are not individually accessible, although novel-viewpoint rendering requires source waveforms that can be localized and propagated separately.

    Source: §1 · p. 1

  • Method

    SepGen extends a pretrained audio-video generator to produce video, an audio-mix, and captioned stems jointly, or to extract stems from an observed video and clean audio-mix.

    Source: §1 · pp. 1–2, §3 · p. 4

  • Results

    SepGen’s joint-run stems lead cascade baselines across reported generation metrics, and its separation lead persists when spoken lines are removed from captions.

    Source: §4.2 · p. 7, §4.3 · p. 8

  • Takeaways & Limitations

    Per-source waveforms provide the individual sources needed for spatial rendering from a new viewpoint.

    Source: §1 · p. 1

    The current layout is fixed at two sources.

    Source: §5 · p. 9

Abstract

from arXiv · show

A 4D audio-visual scene comprises a video, the dynamic geometry it depicts, and the sound sources that populate it, each with its own position and trajectory. Rendering such a scene from a novel viewpoint requires that every source be available as an individual waveform, so that it can be localized in the scene and propagated to the observer before the signals are mixed. Joint audio-video generators can synthesize the video and its soundtrack, but the soundtrack is emitted as a single audio-mix in which the sources are not individually accessible. We present SepGen, which extends a pretrained audio-video generator to emit the video, the mixed soundtrack, and one waveform per captioned source in one joint sampling run. SepGen supports two complementary modes: generation and separation. In generation mode, each source caption specifies what its stem contains. A two-speaker dialogue, for example, comes out as one stem per speaker in the original turn order. In separation mode, the input audio-mix remains clean while the captions specify what to extract, so the model can decompose a recording from a free-text description. We evaluate generation on scenes synthesized from text, and separation on scenes rendered by other generators and on real recordings of speech, music, and sound effects. Given an audio-mix and captions that carry the spoken lines, SepGen outperforms language-conditioned separators, most clearly on speech, and it keeps the lead when the lines are removed from the captions. Code, checkpoints, and datasets are available at https://sepgen.github.io/

1 INTRODUCTION

Joint audio-video generators produce a single mix whose sources cannot be moved or rerendered individually, limiting spatial audio when a scene is viewed from a new viewpoint. SepGen jointly produces video, the mix, and captioned source stems, and supports both generation and separation.

1 INTRODUCTION

  • A single mixed soundtrack cannot be repositioned source-by-source when a lifted video is rendered from a new viewpoint.Per-source waveforms give a spatial renderer the individual sources it needs to propagate sound to the observer.Source: §1 · p. 1
  • SepGen extends a pretrained audio-video generator to emit video, the audio-mix, and one waveform per captioned source in one sampling run.Predicting stems jointly from the shared audio-mix lets each follow its caption while sharing scene timing.Source: §1 · p. 1
  • A protected audio-mix channel bypasses the adapter, preserving the backbone’s soundtrack and video while stems use aligned positions and caption-specific routing.The stems attend to the audio-mix and one another; block-diagonal routing gives each stem only its own caption.Source: §1 · p. 1
  • The audio-mix noise level selects the mode: separation keeps it clean, while generation denoises it with the stems, which attend to its current estimate.Training starts with separation and briefly continues on noisy audio-mixes to produce one checkpoint for both modes.Source: §1 · p. 2
  • A brief continuation after separation training reduces leakage between generated stems, improves speech intelligibility, and stabilizes results across seeds while preserving separation.The paper evaluates both modes on generated, cross-generator, and real scenes using automatic metrics and a human A/B study.Source: §1 · p. 2

2 RELATED WORK

Prior work generates joint audio-video scenes, multiple audio stems, or query-conditioned separations, but the cited approaches differ in whether they provide captioned stems jointly with video. SepGen’s related-work framing emphasizes that combination and its distinction from query-by-query extraction.

Joint audio-video generation

  • Joint audio-video generators such as LTX-2 (HaCohen et al., 2026) and Ovi (Low et al., 2025) produce one audio-mix per scene, whereas SepGen outputs a separate waveform for each source in both modes.Video-to-audio systems likewise output one mix per scene (Polyak et al., 2024; Cheng et al., 2025a; Wang et al., 2025b; Shan et al., 2025; Viertola et al., 2025; Tao et al., 2026).Source: §2 · p. 2
  • SepGen’s extra audio tracks are captioned stems that neither video nor the audio-mix attends to, and the audio-mix bypasses the LoRA adapter.Its Cross-Stem Attention Guidance adapts Normalized Attention Guidance (Chen et al., 2025), using the sibling caption as the negative prompt.Source: §2 · p. 3

Multi-source audio generation

  • Multi-source audio models generate or separate several stems, but the closest cited model, MGE-LDM, is audio-only and extracts one source per run.The paper states that, to its knowledge, no method generates a video with several captioned stems in one denoising run.Source: §2 · p. 3

Language-queried source separation

  • Language-queried audio source separation (LASS) uses free-form text queries, with cited examples including AudioSep, FlowSep, and SAM Audio (Shi et al., 2025).Other cited systems extract several sources in one pass only from fixed categories.Source: §2 · p. 3

Audio-visual and generative separation

  • Query-based separators process prompts independently, so two speaker queries can both return the louder voice; SepGen predicts both stems jointly from the shared audio-mix.The same-category speaker example is also described in the introduction as a weakness of separate query runs.Source: §2 · p. 3

3 METHOD

SepGen extends a joint audio-video generator to produce captioned source stems alongside video and an audio-mix, supporting both generation and separation. Its architecture and training preserve the mix while allowing stems to attend to it and their own captions.

Problem Formulation

  • The audio stream concatenates source stems and the mix as aligned channels; stems attend to the mix and their own captions, while information flows only toward stems.A low-rank adapter updates stem tokens while the pretrained model weights remain frozen.Source: §3 · pp. 4–5
  • SepGen generates video, an audio-mix, and one caption-conditioned stem per source, or extracts captioned stems from an observed video and audio-mix.The modes differ in whether the model generates or receives the video and audio-mix.Source: §3 · p. 4

Training data

  • Training examples pair video, an audio-mix, two source waveforms, a scene caption, and per-source captions; the 4,843 examples come from four origins.Examples include simultaneous or alternating sources, and training randomly selects among up to three caption-verbosity levels.Source: §3 · p. 4

Objective

  • The stem-only loss trains predicted velocities toward flow-matching targets, while video and audio-mix tokens serve as unsupervised inputs describing the scene.The noisy latent is formed by interpolating the clean latent and noise according to σ.Source: §3 · p. 4

Training

  • Two-stage training first uses clean audio-mixes for separation, then changes only the mix-noise distribution to train generation while retaining separation.The generation stage starts from the separation checkpoint and uses the same data and loss.Source: §3 · pp. 4–5
  • The generation checkpoint also supports separation; relative to using the separation checkpoint as a generator, it raises J-swap from 1.95 to 2.07 and lowers own-line word error from 0.10 to 0.07.It reduces seed spread of J-swap from 0.13 to 0.01.Source: §3 · p. 5

Inference

  • Separation denoises stems against a clean observed mix, while generation denoises the mix with the stems and switches stem conditioning from noisy mix to its clean estimate.The estimated-clean-mix conditioning is used below threshold σg; Figure 3 depicts the separation and generation schedules.Source: §3 · pp. 5–6, Figure 3 · p. 6

4 EXPERIMENTS

Experiments assess caption adherence and cross-stem leakage in generation, separation on rendered and real scenes, listener preferences, and component ablations.

4.1 SETUP

  • Evaluation spans 194 generated prompts, rendered-scene benchmarks of 149 Veo and 64 LTX scenes, and 172 real SAM-Audio-bench recordings.Metrics compare stems with captions; a held-out set of 129 scenes with reference stems is also evaluated using envelope correlation and log-spectral distance.Source: §4.1 · pp. 6–7

4.2 GENERATION RESULTS

  • Joint-run stems lead every cascade on every metric, significantly except for CLAP against FlowSep and AudioSep; self-separation raises sounds scores but costs a second run.Joint-run stems keep lower own-line word error within seed variation, while summed stems match the audio-mix more closely than any cascade.Source: §4.2 · p. 7, Table 1 · p. 7

4.3 SEPARATION RESULTS

  • SepGen’s separation checkpoints surpass cascades on Veo Judge and swap margins, lead cascades across LTX metrics, and outperform the strongest SAM Audio checkpoint on real-recording Judge.On real recordings, SepGen also leads every cascade on both swap margins; the two checkpoints perform comparably across separation benchmarks.Source: §4.3 · pp. 7–8, Table 2 · p. 8, Table 3 · p. 8

4.4 USER STUDY

  • Listeners preferred SepGen in separation (0.58 vs 0.25 for SAM Audio) and generation (0.63 vs 0.28 for AudioSep).The study used forced choice with a tie option; it included 48 listeners for separation and 51 for generation.Source: §4.4 · p. 8, Figure 5 · p. 9

4.5 ABLATIONS

  • Separating each caption in a standalone pass produces near-copies of the audio-mix and collapses every swap margin, word error, and speaker-cosine measure.Generation collapses similarly without joint stems.Source: §4.5 · pp. 8–9
  • Replacing aligned rotary positions with a concatenated timeline lowers Judge scores and more than triples word error; blank or unrelated video leaves separation metrics unchanged.Enabling or disabling the adapter from identical inputs leaves audio-mix tokens and video bit-identical, though stem tokens differ.Source: §4.5 · p. 9

5 DISCUSSION AND CONCLUSION

SepGen produces per-source waveforms alongside video and soundtrack, using one model for generation and separation. Its current layout is limited to two sources, and stems retain scene acoustics that a renderer must account for before viewpoint propagation.

5 DISCUSSION AND CONCLUSION

  • The current layout is fixed at two sources, and stems retain reverberation and distance cues that a renderer must account for before propagation to a new viewpoint.The conclusion identifies extending beyond two stems and integrating a placement readout into a spatial renderer as future work.Source: §5 · p. 9
  • SepGen produces per-source waveforms with the video and soundtrack, switching between generation and separation through the audio-mix noise schedule alone.The conclusion describes a lightweight stem adapter trained on paired two-source data and a protected audio-mix that the backbone produces unmodified.Source: §5 · p. 9

Appendix

  • The appendix covers training and inference, benchmark construction, evaluation, generation and separation results, waveform-level evaluation, and placement readout.It points readers to audio-video examples and main results tables, and notes that automatic metrics are proxies for what listeners hear.Source: Appendix · p. 15

A TRAINING AND INFERENCE DETAILS

The appendix details the paired two-source training data, two-stage training and inference setup, and mechanisms for keeping stems caption-faithful while preserving the backbone's audio-mix and video outputs.

A TRAINING AND INFERENCE DETAILS

  • Training uses 4,843 paired two-source segments from CelebV-HQ, URMP, generated clips, and cross-video composites, with semantic, positional, and short source captions.Each source box marks its frame region in video latent frames and softly restricts that stem's video-to-audio attention during training; the bias is removed at inference.Source: A TRAINING AND INFERENCE DETAILS · p. 15
  • Training has 12,000 separation updates followed by 3,000 generation updates, with frozen, int8-quantized LTX-2.5 22B weights and 327M trainable parameters.LoRA updates are zeroed on audio-mix tokens, and loss is computed on stems alone; separation keeps the mix clean, while generation uses clean mixes on 30% of samples and otherwise noisy mixes.Source: A TRAINING AND INFERENCE DETAILS · p. 15
  • At inference, separation re-clamps the audio-mix to zero noise, while generation uses a time-local attention bias and Estimated Separation to condition stems on a clean mix estimate.Generation returns video at 1536 × 1024 and three 48 kHz waveforms; generation and separation use different sampling steps and guidance settings.Source: A TRAINING AND INFERENCE DETAILS · p. 15
  • Cross-Stem Attention Guidance contrasts each stem's own-caption branch with a sibling-caption branch to steer it away from the sibling's source.The audio-mix reads the scene caption in both branches and carries no contrast; Figure S1 depicts the branch routing and guidance operation.Source: A TRAINING AND INFERENCE DETAILS · pp. 15–16, Figure S1 · p. 16
  • With λ=2, τ=2.5, and α=0.5, removing guidance lowers the LTX-2.3 CLAP swap margin (Holm p=0.03), while Judge and J-swap changes are not significant.The caption identifies Table S1 as the LTX-2.3 separation-checkpoint ablation; λ≥3 lowers Judge.Source: A TRAINING AND INFERENCE DETAILS · p. 16, Table S1 · p. 16
  • With the adapter enabled or disabled, audio-mix and video activations are bit-identical at every transformer layer, while stem rows differ.Against a stock render, audio-mix and video differences are limited to floating-point rounding and, over a full render, the sampler's noise draw.Source: A TRAINING AND INFERENCE DETAILS · pp. 16–17
  • The first-stage checkpoint is separation-only; the second-stage checkpoint supports joint video-and-stem generation and separation with the same weights.Generation results use the second-stage checkpoint, while unnamed separation results use the separation checkpoint.Source: A TRAINING AND INFERENCE DETAILS · p. 17

B BENCHMARKS

The benchmarks cover generation on prompted scenes, separation on scenes rendered by other generators, and separation on in-the-wild recordings. Their construction specifies source captions, clip selection, and scoring scope.

B BENCHMARKS

  • Evaluation spans three generated-video benchmarks, scored separately for sounds and speech, plus one real-recording benchmark scored as a single pool.Language-queried methods receive identical per-source captions; empty-mask stems remain in the reported means.Source: B BENCHMARKS · p. 17
  • The generation benchmark contains 194 two-source prompts, including 60 speech prompts and 134 sounds-lane prompts, and SepGen generates video, mix, and stems in one run.Cascade baselines receive the same mix and captions; generated audio-mixes have mean word error 0.05 against the concatenated scripted lines.Source: B BENCHMARKS · p. 17
  • The LTX-2.3 benchmark has 64 two-source clips, while the screened Veo benchmark retains 149 clips across sounds and speech lanes.The Veo clips come from the generation prompt suite with an ultrawide camera cue; each method is scored on the same first 5.7 seconds.Source: B BENCHMARKS · p. 17
  • SAM-Audio-bench uses 172 in-the-wild recordings with one target description and no reference stems, and SepGen outputs a target stem plus residual-background stem.Its subsets are instrument-wild, soundeffects, and speaker; Judge and CLAP score the queried target, with a sibling-text swap margin also reported.Source: B BENCHMARKS · p. 17

C EVALUATION PROTOCOL

The evaluation protocol defines text-match and separation metrics, controls for scorer and decoder effects, and uses repeated seeds and listener preference studies. It reports uncertainty for the user-study preference shares, with clip resampling contributing most of the interval width.

C EVALUATION PROTOCOL

  • Identity scores compare each stem with its own description, while swap margins subtract the sibling-description score; a duplicated mix has zero swap margin.Speech evaluation pairs own-line word error with speaker cosine because silence can lower cosine while producing own-line word error near 1.0.Source: C EVALUATION PROTOCOL · p. 18
  • Separation results average across seeds {0, 1, 2}, while SAM-Audio-bench uses {0, . . . , 4}; significance tests are Holm-corrected paired Wilcoxon tests.Whisper transcription is nondeterministic, so word-error comparisons are made within a single recognizer pass.Source: C EVALUATION PROTOCOL · p. 18
  • Decoder requantization leaves the ranking unchanged, and the decoder accounts for at most 27% of a baseline's Judge margin; band-limiting preserves 96% of the Judge gap, 94% of J-swap, and 76% of C-swap over the best cascade baseline.SepGen remains ahead of every cascade baseline after band-limiting; WavLM speaker embeddings also distinguish each speech stem's own-region voice match from sibling-region matches.Source: C EVALUATION PROTOCOL · p. 18, Table S2 · p. 18
  • The separation user study uses four Veo and six SAM-Audio-bench clips, while the generation study uses ten generation-benchmark clips at seed 0.Listeners choose between anonymous candidate stems for each caption, with a tie option; the study retains 48 separation listeners and 51 generation listeners after screening.Source: C EVALUATION PROTOCOL · pp. 18–19
  • The user-study figure's preference-share intervals resample listeners and clips independently, and clip resampling accounts for most of their width.For overall separation share, listener-only resampling gives 0.55 to 0.61, while clip-only resampling gives 0.48 to 0.70; Table S3 reports 95% bootstrap intervals and majority-vote source counts.Source: C EVALUATION PROTOCOL · p. 19, Table S3 · p. 19

D GENERATION RESULTS

SepGen’s generation results examine category-level benchmark performance, the effect of joint sampling, and the schedule and checkpoint choices for its dual modes.

D GENERATION RESULTS

  • SepGen’s Judge and swap-margin leads hold across every scene category and are widest on music, where continuous instruments leave cascades no silent span to exploit.Table S4 adds seed spreads and paired-test stars; Table S5 reports the category breakdown.Source: D GENERATION RESULTS · p. 19, Table S4 · p. 19, Table S5 · p. 20
  • Without joint sampling, both benchmark lanes collapse, and Judge falls below every cascade baseline receiving the joint audio-mix.The control generates each source in a separate single-caption pass, without a shared audio-mix channel.Source: D GENERATION RESULTS · p. 20, Table S6 · p. 20
  • Starting Estimated Separation at every step lowers J-swap at equal Judge and word error, while using it at no step lowers J-swap but raises word error as Judge rises.The schedule threshold is the noise level below which stems condition on a clean audio-mix estimate rather than its noisy state.Source: D GENERATION RESULTS · pp. 20–21, Table S7 · p. 21
  • The separation checkpoint matches the generation checkpoint on Judge when generating but trails on J-swap and word error; the generation checkpoint separates the Veo benchmark at parity.Figure S2 illustrates generated stems, including singing with drums and two turn-taking speakers.Source: Figure S2 · p. 20, D GENERATION RESULTS · p. 21

E SEPARATION RESULTS

Separation results compare SepGen with cascade and speech-separation baselines, test candidate reranking, and examine whether stems depend on the observed audio-mix.

E SEPARATION RESULTS

  • Both SepGen checkpoints lead every cascade baseline on every column across the Veo, LTX-2.3, and SAM-Audio-bench benchmarks, except CLAP identity on Veo, where AudioSep ties the separation checkpoint.The Veo results include single-draw rows and a bottom block applying SAM Audio’s best-of-eight protocol to both methods.Source: E SEPARATION RESULTS · p. 21, Table S8 · p. 21, Table S9 · p. 22
  • After eight-candidate reranking, SepGen retains a significant lead over both SAM Audio checkpoints on every measure, especially own-line word error and speaker cosine.The reranker selects on Judge and CLAP; SepGen’s eight draws have Judge standard deviation 0.02.Source: E SEPARATION RESULTS · p. 21, Table S8 · p. 21
  • SepGen has lower own-line word error than MossFormer2 and SepFormer on all three benchmarks; on Veo, SepFormer’s speaker cosine is 0.59 against SepGen’s 0.61, not significant.The separators’ outputs are assigned to captions by matching their transcripts to the quoted lines.Source: E SEPARATION RESULTS · p. 21, Table S11 · p. 22
  • Replacing the observed audio-mix with silence collapses every stem, while a cross-clip donor makes stems separate and transcribe the donor scene rather than the caption’s script.The ablation uses the LTX-2.3 clip set and the separation checkpoint.Source: E SEPARATION RESULTS · p. 22, Table S12 · p. 22

F WAVEFORM-LEVEL EVALUATION

Waveform-level evaluation compares SepGen stems with held-out references and explains why decoder-resynthesized waveforms require phase-invariant measures.

F WAVEFORM-LEVEL EVALUATION

  • SepGen leads on envelope correlation and log-spectral distance across 258 held-out reference stems, including over SAM Audio at eight ranker-selected draws.The set includes 129 two-source scenes, 60 URMP duets, 57 CelebV-HQ dialogues, and 12 validation composites; LSD is 6.39 ± 0.05 over three seeds.Source: F WAVEFORM-LEVEL EVALUATION · p. 23, Table S13 · p. 23
  • On 57 dialogues, SepGen’s stems match the true stems’ own-line word error of 0.05, versus 0.29 for the strongest cascade baseline and 1.03 for the audio-mix.The evaluation level-matches outputs and searches for a ±50 ms lag before scoring.Source: F WAVEFORM-LEVEL EVALUATION · p. 23
  • For same-instrument URMP duets with identical captions on both stems, SepGen reads 11.7 dB LSD versus 6.3 dB on distinct-instrument duets; permutation-invariant LSD is 11.4 dB.The passage says the split is not directed by the caption and permutation does not fix it.Source: F WAVEFORM-LEVEL EVALUATION · p. 23
  • After the audio VAE decoder rewrites waveform phase, SI-SDR turns negative for every method despite high envelope correlation and low LSD, so phase-invariant measures are used.Restoring short-time phase alone makes SI-SDR positive again, while restoring magnitudes alone barely changes it.Source: F WAVEFORM-LEVEL EVALUATION · p. 23, Table S14 · p. 24

G PLACEMENT READOUT

SepGen uses each stem’s video attention to track its source, with a stability-based choice between attention maps and geometric-median smoothing.

G PLACEMENT READOUT

  • Each stem supplies video-to-caption and video-to-audio attention maps; SepGen averages them across separation steps and selected blocks, then keeps the more stable track.The maps are averaged over 30 separation steps and blocks 26 to 33 and 36.Source: G PLACEMENT READOUT · p. 24
  • For each frame, SepGen takes the weighted geometric median of the strongest 2% of attention and smooths the resulting track with a Kalman smoother.The method requires no extra training or detector.Source: G PLACEMENT READOUT · p. 24
  • Figure S3 displays two scenes, with caption-matched colors linking attention volumes, source waveforms, and pooled geometric-median plus Kalman tracks; voxels and tracks appear only where the source waveform is active.The scenes show a djembe/cello pair and a turn-taking tattoo artist/client dialogue.Source: Figure S3 · p. 24

Related papers

Loading 2610.11361v1…