PaperlayerView exact revision on arXiv

Source-linked AI summary

SAM Audio: Segment Anything in Audio

Bowen Shi, Andros Tjandra, John Hoffman, Helin Wang, Yi-Chiao Wu, Luya Gao, Julius Richter, Matt Le, Apoorv Vyas, Sanyuan Chen, Christoph Feichtenhofer, Piotr Dollár, Wei-Ning Hsu, Ann Lee

arXiv:2512.18099v1eess.AScs.CV

TL;DR

General audio separation remains limited by domain-specific models, fixed source categories, and restricted prompting. SAM Audio addresses this gap with a flow-matching diffusion transformer supporting text, visual, and temporal span prompts across diverse audio. It reports state-of-the-art separation performance, introduces a multimodal benchmark, and provides a reference-free evaluator that correlates strongly with human judgment.

  • Problem

    Existing audio separation models are often domain-specific, tied to fixed categories, or limited to a single prompting modality, while standard metrics can poorly reflect perceptual quality.

  • Method

    SAM Audio uses a flow-matching diffusion transformer to jointly generate target and residual stems from mixtures conditioned on text, visual, or temporal span prompts.

  • Results

    SAM Audio achieves state-of-the-art performance across general sound, speech, music, and instrument separation, including in-the-wild and professional audio.

  • Takeaways & Limitations

    The framework broadens controllable audio separation through multimodal prompting and adds SAM Audio-Bench and SAM Audio-Judge for multimodal, reference-free evaluation.

  • Takeaways & Limitations

    Visual prompting is less effective than text prompting, and general sound effects remain more challenging than specialized domains such as speech.

Abstract

from arXiv · show

General audio source separation is a key capability for multimodal AI systems that can perceive and reason about sound. Despite substantial progress in recent years, existing separation models are either domain-specific, designed for fixed categories such as speech or music, or limited in controllability, supporting only a single prompting modality such as text. In this work, we present SAM Audio, a foundation model for general audio separation that unifies text, visual, and temporal span prompting within a single framework. Built on a diffusion transformer architecture, SAM Audio is trained with flow matching on large-scale audio data spanning speech, music, and general sounds, and can flexibly separate target sources described by language, visual masks, or temporal spans. The model achieves state-of-the-art performance across a diverse suite of benchmarks, including general sound, speech, music, and musical instrument separation in both in-the-wild and professionally produced audios, substantially outperforming prior general-purpose and specialized systems. Furthermore, we introduce a new real-world separation benchmark with human-labeled multimodal prompts and a reference-free evaluation model that correlates strongly with human judgment.

1 Introduction

Audio separation decomposes mixtures into source tracks, but existing systems are constrained by fixed categories, domain specialization, or limited prompting. SAM Audio unifies text, visual, and temporal prompting for open-domain separation and introduces a multimodal benchmark with reference-free evaluation.

  • Motivation: Audio separation supports remixing, archival restoration, noise removal, accessibility, and multimodal audio understanding.
  • Research gap: Existing separation systems either decompose mixtures into fixed predefined sources or specialize in domains such as speech and music.These constraints limit adaptation to open-domain mixtures and source types.
  • Research gap: Prompted systems remain limited because text-based methods struggle in specialized domains, visual prompting is underexplored, and specialized models cannot generalize beyond fixed source types.
  • Core contribution: SAM Audio unifies text, visual, and temporal span prompting for open-domain separation across speech, music, and general sounds.Prompts can be used independently or jointly, while span prompting provides frame-level temporal control.
  • Benchmark contribution: SAM Audio-Bench covers speech, music, and sound effects with human-labeled multimodal prompts and a reference-free evaluator correlated with human perception.

2 Related Work

Prior separation research spans specialized discriminative and generative systems, promptable extraction, enhancement, music demixing, and multimodal approaches. Across these areas, fixed ontologies, narrow benchmarks, synthetic mixtures, and weak perceptual metrics motivate a more flexible evaluation and separation framework.

  • Separation models: Existing separation methods include spectrogram masking, waveform models, dual-path architectures, transformers, GANs, normalizing flows, diffusion, and flow matching.
  • Speaker separation: Promptable speaker extraction can identify an individual speaker using enrollment speech, but text-prompted speaker separation has received little attention.
  • Audio enhancement: Audio enhancement restores or denoises recordings, whereas faithful separation aims to extract the target source without altering intrinsic recording attributes.
  • Musical instrument separation: Music separation benchmarks and models commonly support fixed stems such as vocals, drums, bass, and other, limiting open-domain usability.
  • Benchmarks: Traditional and multimodal benchmarks often use narrow taxonomies, synthetic mixtures, or predefined stems, limiting realism and domain diversity.
  • Evaluation: SDR-like metrics quantify differences from reference signals but can poorly reflect perceptual quality, while listening tests are expensive and difficult to scale.

3 Approach

SAM Audio is a generative foundation model for open-domain audio separation that jointly supports text, visual, and temporal span prompts. It uses flow matching in a DAC-VAE latent space, jointly generates target and residual stems, and applies multi-diffusion for coherent long-form separation.

  • SAM Audio uses a flow-matching Diffusion Transformer in DAC-VAE latent space to jointly generate target and residual stems from an audio mixture.The target and residual DAC-VAE features are concatenated and denoised together in one pass.
  • Audio Representation: 25-Hz DAC-VAE features provide a compact audio representation intended to preserve fidelity while reducing the frame rate relative to Encodec features.The DAC-VAE uses a VAE bottleneck and produces T × C features with C = 128.
  • 3.1 Prompt Types: Three prompt types—text, visual, and temporal span—can be used independently or jointly to specify the target sound.Text provides semantic descriptions, visual prompts use masked video regions, and span prompts mark target-event intervals.
  • 3.1 Prompt Types: Span prompting offers precise temporal control and can disambiguate overlapping events, but is ambiguous when multiple sounds occur simultaneously.It performs particularly well for foreground sounds or when combined with text or visual prompts.
  • 3.1 Prompt Types: Text prompts enter through cross-attention, while visual and span prompts provide frame-aligned features concatenated with noisy latents.This combines global semantic information with frame-level conditioning.
  • 3.4 Longform audio separation with Multi-diffusion: Multi-diffusion processes overlapping windows in parallel and merges their predictions with normalized soft masks, propagating information across regions to avoid boundary artifacts.The iterative procedure produces smooth, globally coherent long-form separations.

4 Data

SAM Audio training data combines real stems, synthetic mixtures, and pseudo-labeled stems across speech, music, and general sounds. A filtering pipeline removes low-quality synthesized examples before training.

  • Data construction: SAM Audio uses real, synthetic, and pseudo-labeled training regimes that differ in how mixture, target, and residual signals are obtained.The ideal regime uses real isolated stems satisfying xmix = xtgt + xres, while other regimes synthesize mixtures or target/residual stems.
  • Fully-real data: Fully real supervision includes 536 hours of multi-track music and 21,910 hours of conversational speech with speaker-level target and residual stems.Music stems are mixed within compositions with ±5 SNR rescaling, while residual speakers receive randomly sampled ±15 dB SNR offsets.
  • Synthetic mixtures: Because real stems are scarce and domain-specific, SAM Audio also creates synthetic mixtures tailored to music, speech, and general-sound data.Synthetic examples mix clean or in-the-wild recordings while accounting for domain-specific properties such as vocal content, background noise, and production quality.
  • Pseudo-labeling: Random mixing can produce unrealistic combinations, so an intermediate SAM Audio checkpoint generates target and residual stems from natural recordings to bootstrap pseudo-labeled data.This procedure aims to create more realistic training tuples from unlabeled mixtures.
  • Pseudo-labeling: The pseudo-labeling pipeline captions mixtures with PLM-Audio, uses those prompts to generate target and residual stems, and filters candidates with text–audio, cleanliness, silence, and visual-quality criteria.Only high-quality pseudo-labeled stems are retained for subsequent training.

5 Evaluation

SAM Audio-Bench evaluates separation under realistic, multimodal, cross-domain conditions, while SAM Audio Judge adds human-aligned reference-free assessment. Together, they address gaps in benchmark realism, prompt coverage, domain breadth, and perceptual evaluation.

  • SAM Audio-Bench: Each 10-second test instance can include human-annotated language descriptions, frame-level visual masklets, and temporal presence or absence spans.Annotators enumerate sound events, mark visible sources, and identify temporal boundaries, enabling interchangeable prompting modalities.
  • SAM Audio-Bench: The benchmark spans speech cleaning, speaker separation, music cleaning, 37 instrument classes, and general sounds, with modality coverage summarized across tasks and datasets.Figure 5 reports task counts, supported prompt modalities, and source datasets.
  • SAM Audio-Bench: SAM Audio-Bench combines real in-the-wild audio/video, text and visual or temporal prompts, speech/music/instrument/general-sound tasks, and reference-free human evaluation.The benchmark is designed to unify realism, modality coverage, cross-domain scope, and perceptual assessment on shared items.
  • SAM Audio Judge: SAM Audio Judge defines nine perceptual dimensions for model performance and separation difficulty, addressing limitations of coarse relevance and quality scores.Performance dimensions include recall, precision, and faithfulness; difficulty dimensions include counting, overlap, loudness, confusion, and overall difficulty.
  • SAM Audio Judge: Human raters evaluate SAJ dimensions on five-point Likert scales, while side-by-side preference evaluation narrows confidence intervals by up to 20% and yields roughly 30% cost savings.The reported protocol also improves inter-annotator agreement and produces sharper preference-aligned differences.

6 Experimental Setup

The experiments evaluate three SAM Audio model sizes across text, visual, and temporal prompting tasks spanning general sounds, speech, music, and instruments. Comparisons include both open-domain prompted systems and specialized fixed-stem baselines.

  • Model configuration: SAM Audio is evaluated with 500M, 1B, and 3B parameter variants, and the largest model is used for key comparisons because it performs best.Parameter counts exclude the visual encoder, text encoder, and audio codec.
  • Training: Training uses large-scale pre-training followed by high-quality fine-tuning, with synthetic mixtures used for pre-training and pseudo-labeled data reserved for fine-tuning.Audio clips are capped at 30 seconds and randomly chunked when longer.
  • Inference: Inference uses a 16-step midpoint ODE solver, modality-specific candidate re-ranking with beam size 8, and default span prediction for text-only separation.Re-ranking combines SAM Audio Judge and CLAP for text, span IoU for spans, and ImageBind similarity for visual prompts.
  • Text prompting: Text-prompted evaluation covers general sound events, speech, speaker, music, in-the-wild instruments, and professional instrument separation.The tasks test open-vocabulary extraction across noisy, natural, and professionally recorded audio.
  • Visual and span prompting: Visual tasks extract highlighted sound events, instruments, or speakers, while span tasks specify when the target source is active.Evaluation filters ambiguous prompts and uses approximately 10-second samples, including chunks from longer MUSDB recordings.
  • Baselines: Baselines include open-domain text-prompted models and specialized systems such as Demucs and Spleeter, which support only limited fixed stem sets.Speech comparisons disable baseline enhancement or post-processing to focus on separation fidelity.

7 SAM Audio Results

SAM Audio outperforms prior general-purpose and specialized separation systems across text and visual prompting, while span conditioning improves separation when combined with text. The SAM Audio Judge and multi-diffusion strategy further support perceptual evaluation and long-form quality.

  • 7.1 Text-prompted separation: SAM Audio consistently outperforms prior models across nearly all categories, including specialized instrument and speaker separation.It achieves a 17.6% net win rate over Demucs, subjective scores of 4.45 versus 4.28 on professional instrument separation and 4.15 versus 3.51 on speaker separation.
  • 7.2 Visual-prompted separation: Visual prompting yields net win rates of 5%–48% over DAVIS and remains ahead of specialized baselines on speaker and instrument separation.SAM Audio surpasses specialized baselines by approximately 25% on speaker separation and 5% on instrument separation.
  • 7.3 Span-prompted separation: Combining text and span inputs improves performance across all domains, achieving net win rates between +12.9% and +39.0%.Span-only conditioning helps short, localized sounds but can degrade performance for long-duration or ambient speech and music.
  • 7.4 Span prediction boosts text-prompted separation: Predicted spans boost performance in most domains, with only minor degradation for professional instrument separation.The small gap between predicted and ground-truth spans indicates robustness to temporal estimation errors, while avoiding costly human annotations.
  • 7.5 Sound removal and latency: SAM Audio outperforms MoisesAI and AudioShake for explicit sound removal, while two ODE steps remain competitive for some tasks.Increasing ODE steps generally improves performance; the gap between two and sixteen steps is larger for speaker and instrument separation than for short, sparse sound effects.
  • 7.8.1 Subjective score correlation: The SAM Audio Judge model surpasses all evaluated metrics in correlation with human ratings across speech, music, and sound effects.Its PCC reaches 0.883, 0.815, and 0.815 for speech, music, and sound, respectively, while its SRCC reaches 0.817, 0.714, and 0.781.

8 Conclusion

SAM Audio is a general-purpose multimodal separation model that achieves state-of-the-art performance through scalable training and complementary prompting. The work also releases a multimodal benchmark and reference-free judge, while identifying visual prompting and general sound effects as remaining challenges.

  • 8 Conclusion: SAM Audio achieves state-of-the-art performance across diverse audio separation tasks with multimodal prompting.The model supports text, visual, and span prompts, used independently or jointly.
  • 8 Conclusion: Scalable synthetic mixing and pseudo-labeling provide broad training coverage without requiring manual stem annotation.The model scales data and capacity with flow matching and bootstraps stems from natural recordings using intermediate checkpoints.
  • 8 Conclusion: SAM Audio-Bench provides balanced human-annotated text, visual, and span prompts, while SAM Audio-Judge offers reference-free evaluation aligned with human perception.SAM Audio-Judge can also serve as a post-processing module to improve separation quality.
  • 8 Conclusion: Visual prompting is less effective than text prompting, and general sound effects remain more challenging than specialized domains such as speech.The authors identify stronger audio–visual grounding and better modeling of complex multi-source scenes as future directions.

A Subjective Evaluation

The subjective evaluation uses a side-by-side ACR protocol combining absolute quality ratings with forced pairwise preferences across modalities and domains. This design addresses limitations of single-stimulus and reference-based evaluations while supporting both deployment assessment and model comparison.

  • A.1.1 Why not MUSHRA, MOS, or CMOS?: The protocol targets ecological validity, sensitivity to small deltas, cross-modality comparability, and operational feasibility.It covers speech, music, instruments, and general sounds with text, visual masklet, and temporal-span prompts.
  • A.1.1 Why not MUSHRA, MOS, or CMOS?: MUSHRA is mismatched because trustworthy references are often unavailable, heterogeneous targets complicate anchors, and sessions can introduce drift and bias.Reference matching may replace perceived-quality judgments, while multi-stimulus sessions add cost and time overhead.
  • A.1.2 Evaluation protocol: Side-by-side ACR presents two outputs with the original mixture and target specification, then collects independent five-point ratings for Recall, Precision, Faithfulness, and Overall quality.A forced preference question is asked for every comparison, even when ACR scores differ.
  • A.1.2 Evaluation protocol: The hybrid protocol provides absolute quality signals for deployment readiness and relative signals for ablations and model selection.The same procedure supports controlled comparisons among text, visual, and temporal-span prompting.

A.1.3 Empirical Observations

The evaluation studies show that the side-by-side ACR protocol improves agreement and sensitivity while introducing longer handling time and persistent anchoring effects.

  • Protocol effects: Preference agreement and absolute-delta consistency improve with the always-on tie-breaker relative to ACR-without-tie-breaker and pairwise-only protocols.Agreement is measured using Gwet’s AC2.
  • Protocol effects: Approximately +0.15 points per item, the preference tie-breaker produces sharper ACR deltas that align absolute scores with expressed preferences.It increases sensitivity for near-tied comparisons relative to protocols without the tie-breaker.
  • Protocol effects: Up to ∼20% narrower confidence intervals yield roughly 30% cost savings for equivalent A/B sensitivity under side-by-side judgment.The comparison is against single-stimulus rating, with confidence-interval width measured by bootstrap.
  • Trade-offs: Side-by-side ACR roughly doubles per-item handling time compared with pairwise-only evaluation, but provides absolute and relative signals in one assignment.The stated handling times are 2 minutes versus 1 minute for CMOS.
  • Limitations: Absolute scores remain context-dependent because presenting an output beside a higher-quality counterpart can depress its ACR score.The authors therefore emphasize win rates and ACR deltas over raw absolute means.
  • Evaluation setup: The evaluation presents source audio or video, a prompt, and two model outputs, then assesses target presence, coverage, non-target leakage, and fidelity.Prompts may be text-based, video-based, or span-based.

B.1 Characterization and Statistics

SAM Audio-Bench characterizes instrument and speaker separation cases through source distributions, prompting availability, overlap, and structured human-judgment criteria.

  • Instrument and speaker statistics: Instrument samples are dominated by a few common sources, with a long tail of less frequent instruments, and most audios contain only two or three active instruments.This characterization is summarized for the instrument separation task in SAM Audio-Bench.
  • Instrument and speaker statistics: Speaker cases span multiple speaker types and prompting modalities while exhibiting substantial variation in overlap between target and interfering speakers.Some items lack visual or text prompts, and speaker-overlap distributions are explicitly summarized.
  • Prompt modalities: Prompting is represented through text descriptions, highlighted video regions, or highlighted audio spans indicating where target sounds are present or absent.These modalities correspond to the benchmark’s text-, video-, and span-based prompt types.
  • Judgment criteria: Human evaluation scores extracted audio on prompt faithfulness and similarity to the target sound in the source audio.The protocol separately considers whether requested sounds were followed and whether their acoustic character was preserved.
  • Judgment criteria: The annotation protocol separates task difficulty from model quality by asking about target presence, non-target sounds, overlap, loudness, confusion, and overall difficulty before output-quality questions.Q1–Q8 concern source content and separation difficulty, whereas Q9–Q12 require model outputs.

C.4 SAJ for Automatic and Fine-grained Performance analysis

The SAJ difficulty model enables automatic, fine-grained evaluation analysis by predicting task difficulty from mixture audio and a text prompt before separation.

  • Automatic evaluation: SAJ can automatically sample evaluation cases at different difficulty levels for any target-audio concept list.The dataset merges rare human-annotation levels 4 and 5 into a balanced four-level scale.
  • Difficulty analysis: The difficulty model uses only mixture audio and a text prompt, unlike standard SAJ, which also uses the separated output.This allows task difficulty to be estimated before running a separation system.
  • Difficulty analysis: Human-rated performance decreases monotonically from difficulty Level 1 to Level 4, with clear degradation at Level 4.The results support evaluating SAM Audio outputs across automatically curated difficulty subsets.

D.1 Effect of Model Scale

Scaling generally improves SAM Audio, especially for specialized separation, although the largest model is not uniformly better on every task.

  • Scaling results: Across 500M, 1B, and 3B models, the 3B model performs best on most tasks, while general SFX separation can favor or match smaller models.Span prediction is disabled for this scale comparison.
  • Scaling results: On instrument-in-the-wild separation, SAM Audio-Large exceeds SAM Audio-Base by 23% NWR and SAM Audio-Small by 20%.The gains indicate a stronger scaling benefit in specialized instrument separation.
  • Scaling results: Figure 17 reports net win rates for SAM Audio-Base and SAM Audio-Small against SAM Audio-Large.The figure provides a direct pairwise scale comparison complementing the tabulated text- and visual-prompting results.

D.2 Effect of Auxiliary Loss

The AED-based representation alignment loss improves both text- and visual-prompted separation during pre-training, with a larger gain for text prompting. The ablation suggests this benefit comes from learning semantic representations from noisier pre-training targets.

  • The ablation compares 3B checkpoints with and without the auxiliary loss on general SFX under text- and visual-prompted separation.The auxiliary loss is evaluated during pre-training, while the pre-trained model alone underperforms on specialized speech and music domains.
  • The authors hypothesize that alignment helps because noisier pre-training audio targets require stronger intermediate semantic representations for separation.The auxiliary loss is applied only during pre-training and not during fine-tuning.

D.3 Effect of fine-tuning

Fine-tuning consistently improves SAM Audio, with especially large gains on professionally supervised extraction and separation tasks. It also substantially improves audio cleanness, while visual alignment changes little for general sound effects.

  • Fine-tuning yields the largest text-prompted gains for instrument extraction, speech extraction, and speaker separation.These tasks benefit from high-quality datasets containing professionally recorded stems.
  • Visual alignment remains relatively stable for general SFX, while fine-tuning provides broader music and speech video coverage.Audio–visual pre-training already establishes strong region-to-audio correspondences for general sound events.
  • Fine-tuned models consistently produce cleaner separated audio with fewer artifacts across tasks, according to the PC metric.The improvement is attributed to clean audio targets used during fine-tuning.

D.4 Effect of using pseudo-labeled audio stem data

Adding pseudo-labeled audio stem data during fine-tuning improves both text- and visual-prompted separation. The largest gains occur in AES-PC for general sound, indicating cleaner separated stems.

  • Pseudo-labeled audio data produce consistent gains for both text- and visual-prompted separation.
  • The largest improvements occur in AES-PC for general sound.
Loading 2512.18099v1…