SAM Audio: Segment Anything in Audio

• Audio source separation supports creative production, archival restoration, noise removal, accessibility, and multimodal audio understanding by isolating individual sound events.Its broad utility makes separation both a practical capability and a testbed for identifying sources in audio.Source: §1 · p. 1
• Existing separation systems rely on fixed source taxonomies or specialized domains, while prompted methods still struggle with music, speech, real-world audiovisual ambiguity, and subtle sound differences.These limitations motivate a flexible framework integrating cues from multiple modalities.Source: §1 · pp. 1–2
• SAM Audio unifies text, visual, and temporal span prompting for open-domain separation across speech, music, and general sound events.A diffusion transformer trained with flow matching produces both the target and residual stems, while prompts can be used independently or jointly.Source: §1 · p. 2
• SAM Audio supports precise temporal control through span prompting, alongside text descriptions and visual masks with positive or negative clicks.The modalities specify what, where, and when to separate and can be combined for more precise isolation.Source: §1 · p. 2
• SAM Audio-Bench covers speech, music, and sound effects with human-labeled multimodal prompts and a reference-free evaluation model that correlates strongly with human perception.The benchmark addresses fragmented coverage, prompting incompatibilities, scarce clean references, and weak correlation between existing reference-free metrics and human judgment.Source: §1 · p. 2

2.1 Audio Separation Models

• Audio separation models have progressed from spectrogram and waveform masking to dual-path, Transformer, diffusion, and flow-matching formulations, while promptable systems add text, visual, or temporal conditioning.Speaker extraction commonly requires clean enrollment audio, text prompting remains limited, and span prompting addresses ambiguity without external reference audio.Source: §2.1 · pp. 3–4
• Most music separation systems support fixed stems—typically vocals, drums, bass, and other, with high-performing models often limited to five stems—constraining open-domain usability.General sound systems similarly rely on predefined ontologies, while text-prompted methods improve flexibility but can remain inferior in professional music demixing and multi-speaker scenes.Source: §2.1 · pp. 3–4
• Visual-prompted separation can associate target sounds with visible objects, regions, or faces, but remains underexplored for real-world open-domain audio and is usually evaluated on synthetic or small-scale data.Prior systems mainly target speech or instruments, despite advances using spatial cues, facial regions, multimodal attention, and diffusion-based conditioning.Source: §2.1 · p. 4

2.2 Audio Separation Benchmark and Evaluation

• Separation benchmarks largely use synthetic or studio stems, narrow taxonomies, and single prompt modalities, limiting realism, domain diversity, and multimodal evaluation.Traditional SDR-like metrics are easy to compute but correlate weakly with human judgments; listening tests are costly, while existing perceptual predictors generalize poorly beyond their intended domains.Source: §2.2 · pp. 4–5

3.1.1 Flow-Matching with Diffusion Transformer

• SAM Audio jointly generates target and residual stems in DAC-VAE latent space using a flow-matching Diffusion Transformer conditioned on multimodal prompts.The model transports Gaussian noise to the joint target-residual representation and predicts both components in one pass.Source: §3 · p. 5, §3.1.1 · p. 6

3.1.2 Prompt Types

• The model supports text, visual, and span prompts individually or jointly, with span intervals providing temporal disambiguation but remaining ambiguous for simultaneous events.Span prompting performs particularly well for foreground sounds or when combined with text or visual prompts.Source: §3.1.2 · pp. 6–7

3.1.3 Audio Mixture and Prompt Encoding

• Audio, visual, and span features are aligned frame-by-frame with noisy latents, while text is injected through cross-attention to combine temporal and global semantic cues.The mixture is encoded at 25 Hz; visual features come from the Perception Encoder, and span tokens explicitly indicate active or silent frames.Source: §3.1.3 · p. 7

3.2 Training Objective

• Training combines flow-matching loss with an auxiliary audio-event alignment loss that guides intermediate representations toward the target event’s semantics and temporal structure.The alignment projects DiT hidden states into an external AED embedding space and maximizes cosine similarity with the ground-truth target embedding.Source: §3.2 · pp. 7–8

3.3 Boosting text prompting with span prediction

• PEA-Frame improves text-only prompting by predicting frame-level activity spans from the mixture and text, then conditioning SAM Audio on both predicted spans and the original description.This approximates manually labeled temporal boundaries while retaining accessible text-based control.Source: §3.3 · p. 8

3.4 Longform audio separation with Multi-diffusion

• Multi-diffusion processes overlapping windows in parallel and merges their latent predictions with normalized soft masks, avoiding boundary artifacts in arbitrarily long audio.Information propagates across overlapping regions at every flow step, producing smooth, globally coherent separations.Source: §3.4 · pp. 8–9

4 Data

• SAM Audio combines large-scale audio–video data with high-quality datasets through three regimes that vary whether mixture, target, and residual signals are real, synthetic, or pseudo-labeled.Training tuples are formed as (xmix, xtgt, xres, c), with conditioning prompts c drawn from text, video, or span modalities.Source: §4 · p. 9, §4.1 · p. 9, Table 1 · p. 9

4.1.1 Fully-real triplets

• Fully-real triplets provide clean supervision for music and speech, using instrument stems from 10,610 compositions totaling 536 hours and conversational speech containing 21,910 hours of audio.Instrument mixtures use target-dependent residuals with ±5 SNR rescaling, while speech forms two speaker-target triplets with a randomly sampled ±15 dB residual SNR offset.Source: §4.1.1 · pp. 9–10

4.1.2 Synthetic audio mixtures

• Synthetic mixtures tailor source construction to each domain by combining clean music with non-music audio, speech with sampled non-speech clips, and in-the-wild or professional sound recordings.Synthetic examples use ±15 SNR for noisy music, preserve single- and multi-speaker speech segments, and mix sound clips according to their recording conditions.Source: §4.1.2 · p. 10

4.1.3 Pseudo-labeling Data Engine

• The pseudo-labeling engine uses an intermediate SAM Audio checkpoint to separate natural unlabeled recordings, then retains candidates after text–audio, cleanliness, silence, and audio–visual filtering.PLM-Audio supplies text prompts for the ∼1M-hour general-sound corpus, while visual samples require reliable mask coverage and audio–visual correspondence.Source: §4.1.3 · pp. 10–11

4.2.1 Text

• Text prompts use concise NP/VP sound-event descriptions generated by PLM-Audio, combined with metadata when available and filtered by CLAP similarity, with a 0.28 threshold corresponding to the 25th percentile.The pipeline also uses templates for fully-real music and speech, while PLM-Audio is trained through encoder alignment, synthetic-caption learning, and separation-oriented post-training.Source: §4.2.1 · pp. 12–13

4.2.2 Video / 4.2.3 Span

• Visual prompts use natural videos with paired audio, retain clips with strong audio–visual grounding, and generate pseudo-label masks via SAM3 from target-sound captions; span prompts target spiky, discrete events.Mixtures dominated by long, continuous ambience, such as rain or traffic, are excluded because span information is uninformative.Source: §4.2.2 · p. 13, §4.2.3 · p. 14

5.1.1 SAM Audio-Bench: Unifying Modalities, Domains, and Realism

• SAM Audio-Bench unifies real in-the-wild audio/video, text, visual-masklet, and temporal-span prompts across speech, music, instruments, and general sounds.Each 10-second instance supports interchangeable prompts, enabling controlled modality comparisons; dedicated tasks include speech cleaning, speaker separation, music cleaning, instrument stems, and general sounds.Source: §5.1.1 · pp. 14–15, §5.1.2 · p. 15

5.1.2 Summary Comparison to Existing Evaluation Sets

• Compared with representative benchmarks, SAM Audio-Bench uniquely combines real content, multimodal prompts on shared items, cross-domain coverage, and reference-free human evaluation.Table 5 summarizes the comparison across modality coverage, realism, domain scope, and reference availability.Source: §5.1.2 · p. 15, Table 5 · p. 16

5.2 SAM Audio Judge Model

• SAM Audio Judge defines nine perceptual dimensions covering separation performance and task difficulty, addressing ambiguity in coarse existing evaluation guidelines.Performance dimensions include recall, precision, faithfulness, and overall quality; difficulty dimensions include counting, overlap, loudness, confusion, and overall difficulty.Source: §5.2.1 · pp. 15, 17

5.2.2 Data Annotation

• The benchmark collects human annotations using detailed five-point Likert guidelines, calibrated audio examples, qualified raters, and three independent ratings per sample.Data span music, speech, and sound effects using both real and synthetic mixtures, with text prompts aligned to the model’s extraction tasks.Source: §5.2.2 · pp. 17–18

5.2.3 SAJ Model Training

• The SAJ model combines input audio, separated output, and text features through temporally aligned multimodal Transformer representations to predict perceptual scores.Pretraining on text–audio target-presence detection precedes fine-tuning for final SAJ score prediction.Source: §5.2.3 · p. 18

5.3 Subjective Evaluation Protocol

• Subjective evaluation presents source audio or video, prompts, and two model outputs, assessing target coverage, correctness, faithfulness, non-target sounds, overall quality, and tied-output preference.The procedure applies consistently across text-, visual-, and span-prompted conditions and reports overall OVR and NWR as primary results.Source: §5.3 · p. 19
• Side-by-side ACR with an always-on preference tie-breaker improves agreement and discrimination, narrows confidence intervals by up to 20%, and saves roughly 30% cost for equivalent A/B sensitivity.The protocol yields both absolute and relative judgments, though absolute scores remain context-dependent.Source: §5.3 · p. 19

5.4 Objective Metrics

• Objective metrics are prompt-specific: SAJ and CLAP evaluate text and span separation, SpanIoU measures temporal alignment, and ImageBind measures visual-audio consistency.SpanIoU derives predicted spans from VAD applied to separated waveforms, while ImageBind compares audio with the corresponding visual mask region.Source: §5.4 · p. 19

6.1 Training and inference configurations

• Three SAM Audio variants use 500M, 1B, and 3B parameter budgets, with two-stage pre-training and fine-tuning on capped 30-second audio clips.SAM Audio-Large performs best and is therefore used for key comparisons and analyses.Source: §6.1 · pp. 19–20

6.2 Tasks

• The benchmark evaluates text-prompted separation across general sound events, speech, speakers, music, and single-instrument extraction from clean studio and noisy in-the-wild mixtures.Speech extraction recovers complete speech regardless of speaker count or background noise, while speaker extraction primarily focuses on gender-based separation.Source: §6.2 · pp. 20–21

6.3 Baselines

• SAM Audio is compared with broad baseline sets across prompting modalities, evaluating each model on tasks it supports using its native input format when available.This protocol is intended to preserve fair comparisons across general-purpose and specialized systems.Source: §6.3 · p. 21

6.3.1 Text-prompting baselines

• Text-prompting comparisons include open-domain systems and specialized speech, music, speaker, and instrument separators, with domain-specific filtering and disabled enhancement modules where appropriate.Instrument comparisons account for each baseline’s supported stem vocabulary, while professional separation uses MUSDB18.Source: §6.3.1 · pp. 21–22

6.3.2 Visual-prompting baselines

• Visual-prompting comparisons use DAVIS-Flow, CLIPSep, IIANet, and AV-Mossformer2 across general sound, instrument, and speaker separation tasks.About 20% of evaluation videos fail AV-Mossformer2 preprocessing and are excluded from that comparison.Source: §6.3.2 · p. 22

6.3.3 Span-prompting baselines

• Because no public models support span prompting, the study compares span-only and joint text+span conditioning against text-only separation using SAM Audio.These comparisons quantify the benefit of temporal-span conditioning.Source: §6.3.3 · p. 23

7.1 Text-prompted separation

• SAM Audio consistently outperforms prior general-purpose, specialized, and proprietary systems, including 4.45 versus 4.28 OVR on MUSDB and 4.15 versus 3.51 in speaker separation.It achieves roughly 36% net win rate over SoloAudio, 17.6% NWR versus Demucs, and approximately 39% net win-rate improvement in speaker separation.Source: Table 10 · p. 23, §7.1 · p. 24

7.2 Visual-prompted separation

• SAM Audio outperforms DAVIS across visual-prompted tasks, with 5%–48% NWR gains, while surpassing specialized baselines by approximately 25% on speakers and 5% on instruments.Visual prompting has lower subjective scores than text because video supervision is noisier and visual masks provide less specific source cues, but masks disambiguate same-gender speakers.Source: Table 11 · p. 24, §7.2 · pp. 24–25

7.3 Span-prompted separation

• Combining text and span prompts consistently improves performance by 12.9%–39.0% NWR, whereas span-only prompting ranges from −49.6% to +26% relative to text-only.Span-only prompts degrade long-duration or ambient sounds but help short, localized sounds where temporal cues are discriminative.Source: Table 12 · p. 25, §7.3 · p. 25, Figure 10 · p. 26

7.4 Span prediction boosts text-prompted separation

• Predicted spans improve separation across most domains without human annotations, with only minor degradation on professional instrument separation.The small gap between predicted and ground-truth spans demonstrates robustness to temporal estimation inaccuracies.Source: Table 13 · p. 26, Figure 11 · p. 26, §7.4 · p. 26

7.5 Sound Removal

• SAM Audio outperforms MoisesAI and AudioShake on text-prompted music removal, and its high OVR indicates clean suppression of target sources.The removal trends mirror music extraction results, suggesting strong correspondence between the two modes.Source: §7.5 · pp. 26–27, Table 14 · p. 27, Figure 12 · p. 27

7.6 Latency

• With 16 midpoint-solver ODE steps, SAM Audio-Large separates a 10-second input in approximately 7.3 seconds on one A100 GPU.Reducing steps generally lowers quality, but two steps remain competitive for some tasks; fewer NFEs offer a favorable speed–quality trade-off, especially for sparse sound effects.Source: Figure 13 · p. 27, §7.6 · p. 27

7.7 Long-form Audio Separation

• On 1-minute recordings, multi-diffusion maintains high perceptual quality and achieves the best judge scores, avoiding the degradation of one-shot processing and discontinuities of chunk-wise separation.The method uses 20-second windows with 5-second context overlap and is evaluated on 50 internal samples.Source: §7.7 · pp. 27–28, Table 15 · p. 28

7.8.1 Subjective score correlation

• SAM Audio Judge outperforms all baselines across speech, music, and sound effects under both PCC and SRCC, reaching PCCs of 0.883, 0.815, and 0.815.Its SRCCs are 0.817, 0.714, and 0.781, while distortion-based SDR estimation often has low or negative perceptual correlations.Source: Table 16 · p. 28, §7.8.1 · pp. 28–29

8 Conclusion

• SAM Audio supports multimodal prompting and achieves state-of-the-art performance across various audio separation tasks.The model advances universal separation by scaling data and model capacity with flow matching.Source: §8 · p. 29
• Scalable synthetic mixing and pseudo-labeling pipelines provide broad training coverage without manual stem annotation.Pseudo-labeling bootstraps stems from natural recordings using intermediate SAM Audio checkpoints.Source: §8 · p. 29
• Visual and span prompting complement text prompting, with span prompting substantially improving text-based separation and enabling iterative refinement.Source: §8 · p. 29
• SAM Audio-Bench provides a balanced benchmark with human-annotated text, visual, and span prompts.Source: §8 · p. 29
• SAM Audio-Judge is a reference-free metric that correlates more strongly with human perception than existing alternatives and can improve separation as a post-processing module.Source: §8 · p. 30
• Visual prompting is less effective than text prompting, while general sound effects remain more challenging than specialized domains such as speech.The paper attributes these gaps to insufficient audio–visual grounding and difficulty modeling complex, multi-source acoustic scenes.Source: §8 · p. 30

A.1 Design

• The protocol is designed for complex real audio/video, small ablation deltas, multiple prompt modalities and domains, and scalable testing.Its criteria cover text, visual masklets, and temporal spans across speech, music, instruments, and general sounds.Source: §A.1 · p. 39

A.1.1 Why not MUSHRA, MOS, or CMOS?

• MUSHRA was rejected because real mixtures often lack trustworthy references, heterogeneous targets complicate anchors, and multi-stimulus sessions add bias, drift, cost, and time.Reference and anchor issues can shift judgments toward reference similarity rather than perceived quality Varadhan et al. (2024), while session effects are reported by Cooper and Yamagishi (2023).Source: §A.1.1 · p. 39
• Single-stimulus MOS/ACR is operationally simple but has limited sensitivity to small deltas, anchoring and range biases, and weak relative ranking without larger samples Sector (1996).Absolute scores can drift with session context Tversky and Kahneman (1974); Cooper and Yamagishi (2023).Source: §A.1.1 · p. 39
• Pure CMOS/CCR preference comparisons improve relative sensitivity but lack absolute deployment thresholds and can be brittle when error modes yield similar overall impressions Sector (1996).This motivates retaining absolute quality judgments alongside preference judgments.Source: §A.1.1 · p. 39

A.1.2 Our Hybrid: Side-by-Side ACR with Preference Tie-Breaker

• The hybrid presents two outputs for the same item side-by-side with the target specification and original mixture as context.The target specification may use text prompts, SAM masklets, temporal spans, or combinations thereof.Source: §A.1.2 · p. 39
• Evaluators independently score each output on 5-point Recall, Precision, Faithfulness, and Overall ACR dimensions aligned to separation goals.Recall measures target extraction, Precision measures non-target leakage, and Faithfulness measures similarity to the in-context target sound.Source: §A.1.2 · p. 39
• An always-on preference question asks which output is preferred, increasing inter-annotator agreement and sharpening delta estimates even when ACR scores differ.The same protocol supports controlled comparisons across text, visual SAM masklet, and temporal-span prompts, including text-only, text+visual, and visual-only ablations.Source: §A.1.2 · p. 39
• Together, absolute ratings assess deployment readiness while relative preferences support ablations and model selection.This combination provides complementary absolute quality and relative comparison signals.Source: §A.1.2 · p. 40

A.1.3 Empirical Observations

• The preference tie-breaker improves inter-annotator agreement and increases average ACR deltas by approximately +0.15 points per item, especially for near-tied comparisons.The protocol reports improved Gwet’s AC2 agreement and better alignment between absolute scores and expressed preferences.Source: §A.1.3 · p. 40
• Up to 20% narrower confidence intervals and approximately 30% lower testing cost make side-by-side judgment more sensitive than single-stimulus rating, despite roughly doubling per-item handling time.Side-by-side ACR takes about 2 minutes per item versus 1 minute for CMOS, while providing absolute and relative signals in one assignment.Source: §A.1.3 · p. 40

A.1.4 Operational Details

• Because anchoring persists, the protocol emphasizes win rates and ACR deltas over raw absolute means and cautions against interpreting cross-experiment shifts in absolute levels.Absolute scores can decline when an output is presented beside a higher-quality counterpart, so comparative statistics are preferred.Source: §A.1.3 · p. 40, §A.1.4 · p. 40, §A.1.5 · p. 41
• Evaluators are trained to follow text, masklet, and span prompts, distinguish precision from recall, and judge faithfulness to the target’s timbral and temporal character.Model identities are blinded, A/B order is randomized, items are batched across domains and modalities, and anomalous raters are removed using vendor checks and CLARA.Source: §A.1.4 · p. 40

A.1.5 Positioning vs. Standards and Literature

• The protocol combines ACR/MOS-style absolute interpretability with CMOS/CCR-like relative sensitivity while avoiding MUSHRA’s reference and multi-stimulus biases, though reference-free objective metrics remain an open area (Stöter et al., 2018).Its design is motivated by the reported disconnect between reference-based metrics and human perception in audio separation.Source: §A.1.5 · pp. 40–41

A.2 Subjective evaluation protocol

• The subjective evaluation presents source audio or video, one of three prompt types, and two model outputs, then assesses prompt faithfulness, target-sound faithfulness, and overall quality.Prompts may be text descriptions, highlighted video regions, or time spans marking where target sounds are present or absent.Source: Overview · pp. 41–42

What you will be annotating

• Annotators assess whether target sounds are present, their portion and similarity, non-target sounds and removal, then provide an overall model-performance score on a 1-to-5 scale.Source: What you will be annotating · pp. 42–43, Overall score for each model’s… · p. 43

Frequently asked questions

• SAM Audio-Bench covers general sound-separation samples characterized by target-sound categories and production complexity, with summary statistics reported for all samples and the human-evaluation subset.The benchmark’s evaluation guidance also defines how to count distinct instruments, speakers, and background noise in ambiguous mixtures.Source: Frequently asked questions · p. 44, Figure 14 · p. 45

B.1 Characterization and Statistics

• SAM Audio-Bench contains long-tailed instrument distributions, usually two or three active instruments, varied speaker types and prompting modalities, substantial speaker overlap, and diverse non-speech distractors.The benchmark’s mixtures range from environmental sounds to background music, reflecting real-world complexity.Source: §B.1 · p. 45, Figure 15 · p. 46, Figure 16 · p. 47

What is “distortion”?

• Annotators treat distortion as source-originating modification or added artifacts, including spectral changes, pitch differences, static, and garbling.They distinguish these changes from interference caused by other sounds in the mixture when possible.Source: What is “distortion”? · p. 48

Overview

• Model quality is judged by how much target audio is present, how closely it matches the source target, and how effectively non-target sounds are removed.The protocol evaluates target duration and number of sounds, audible distortions or artifacts, and residual interference.Source: Overview · p. 49, What you will be annotating · p. 51

What you will be annotating

• The annotation protocol first verifies prompt clarity, target presence, and non-target presence, then records task difficulty from non-target count, overlap, relative loudness, and acoustic confusion.Questions Q1–Q8 concern the source audio and task difficulty rather than model outputs.Source: What you will be annotating · p. 49

Q12. Overall score

• The overall human score ranges from 1 to 5 and should integrate the preceding judgments with the annotator’s overall judgment.A score of 5 denotes complete, source-faithful target extraction without non-target sounds, whereas 1 denotes incorrect or unfiltered output.Source: Q12. Overall score · p. 51, Frequently asked questions · p. 52

Frequently asked questions

• The protocol clarifies ambiguous counting cases, including whether speakers or instruments within a musical mixture should be treated as distinct target or non-target sounds.For specific instruments or song components, each instrument may be considered an individual sound.Source: Frequently asked questions · pp. 51–52
• Raters should maintain a single consistent speaker when prompts identify one speaker, treat ambiguity as unrated, and distinguish target distortion from valid non-target removal.Changing target volume or improving target quality counts as distortion, while reducing non-target volume is valid removal.Source: Frequently asked questions · p. 52

C.2 High quality raters selection in SAM Audio Judge

• A 40-sample expert-labeled golden set and Bayesian confusion-matrix modeling qualified annotators by expected overall-score MAD, yielding 128 raters.The selection procedure accounts for partial completion, item-score uncertainty, and imbalanced score distributions, with expert spot checks for consensus.Source: §C.2 · p. 52

C.3 Gemini-2.5-pro Prompts for Judge Evaluation

• The evaluation rates extracted audio on recall, precision, faithfulness, and overall performance from 1 (very poor) to 5 (excellent), compared with the source audio and target description.Recall measures target-sound coverage; precision measures similarity to the source; faithfulness measures non-target removal. Five reference examples illustrate what scores 1 through 5 mean.Source: §C.3 · p. 53

C.4 SAJ for Automatic and Fine-grained Performance analysis

• SAJ automatically samples evaluation cases at different difficulty levels for any target audio concept list.The resulting pipeline supports fine-grained, concept-specific robustness analysis.Source: §C.4 · pp. 53–54
• A text-prompted SAJ model predicts intrinsic separation-task difficulty using a balanced four-level scale after merging rare annotation levels 4 and 5.The dataset is balanced across levels 1–4 to support difficulty-based evaluation.Source: §C.4 · p. 53
• Unlike standard SAJ, the difficulty model uses only mixture audio and the text prompt, allowing difficulty estimation before running a separation system.Standard SAJ instead predicts separation performance from both mixture and output audio.Source: §C.4 · p. 53
• Predicted difficulty levels are applied to the SAJ test set, with human ratings computed for the corresponding SAM Audio outputs.This procedure links pre-separation difficulty estimates to observed human-rated performance.Source: §C.4 · p. 54
• The difficulty-stratified evaluation enables automatic curation of subsets for fine-grained, concept-specific robustness analysis.The subsets distinguish easier and harder separation cases according to predicted task difficulty.Source: §C.4 · pp. 53–54
• Human-rated separation quality decreases monotonically from difficulty Level 1 to Level 4, with Level 4 showing clear degradation.This result supports evaluating SAM Audio across automatically defined difficulty levels.Source: Table 18 · p. 54, §C.4 · p. 54

D.1 Effect of Model Scale

• The 3B model achieves the strongest performance across most tasks, though it is on par with or slightly below smaller models for general SFX separation.The comparison covers text and visual prompting across Tables 19 and 20 and Figure 17, with span prediction disabled.Source: §D.1 · p. 55, Table 19 · p. 55, Table 20 · p. 55
• On instrument-in-the-wild separation, SAM Audio-Large outperforms SAM Audio-Base by 23% NWR and SAM Audio-Small by 20%.Scaling capacity provides the greatest benefit in specialized domains, where larger models better capture fine-grained acoustic cues; smaller models remain competitive for broader sound categories.Source: §D.1 · p. 55

D.2 Effect of Auxiliary Loss

• The ablation evaluates two 3B pre-trained checkpoints on general SFX separation because the pre-trained model underperforms on specialized speech and music domains.The compared checkpoints differ only by whether representation alignment loss is included during pre-training.Source: §D.2 · p. 55
• Over 20% relative improvement in text alignment and ∼5% with visual prompting result from adding the AED-based alignment objective during pre-training.Table 21 compares 3B checkpoints with and without the auxiliary loss on text- and visual-prompted general SFX separation; the loss uses λ = 1.0.Source: Table 21 · p. 56, §D.2 · p. 56
• The alignment loss is applied only during pre-training, where noisier audio targets may make learned intermediate semantic representations more useful for separation.The auxiliary loss is not applied during fine-tuning; the semantic-representation explanation is presented as a hypothesis.Source: §D.2 · p. 56

Effect of fine-tuning

• The comparison evaluates the 3B model after pre-training alone against the same model further fine-tuned on curated separation data.The evaluation reports standard metrics alongside the Audiobox-Aesthetics PC score.Source: §D.3 · p. 56
• Fine-tuning consistently improves separation performance, with the largest text-prompted gains in instrument extraction, speech extraction, and speaker separation.Professionally recorded stems provide clean, reliable supervision, whereas general sound event and music separation improve less because pre-training already covers broad sound distributions and mixtures.Source: §D.3 · p. 56
• For visual-prompted separation, fine-tuning leaves general-SFX visual alignment relatively stable while benefiting music and speech videos more.Broader fine-tuning coverage for music and speech videos contrasts with strong audio–visual correspondences already established for general sound events during pre-training.Source: §D.3 · p. 56
• Fine-tuning substantially improves audio cleanness across tasks, producing fewer artifacts through clean audio targets.This improvement is measured by the PC score from Audiobox-Aesthetics, used as a proxy for audio cleanness.Source: §D.3 · p. 56

D.4 Effect of using pseudo-labeled audio stem data

• Pseudo-labeled audio data consistently improves both text- and visual-prompted separation, with the largest gain in general-sound AES-PC.The results suggest pseudo-labeled audio helps the model produce cleaner separation stems.Source: §D.4 · pp. 56–57, Table 23 · p. 57