PaperlayerView exact revision on arXiv

Source-linked AI summary

SCISSOR: Score-Conditioned Instrument Source Separation for Orchestral Recordings

Yiheng Lu, Hao-Wen Dong

arXiv:2609.33265v1cs.SDcs.AIeess.AS

TL;DR

Orchestral mixtures can obscure instrument identity when sections share pitches, timbres, and harmonics, while earlier score-concatenation methods had less consistent gains in dense mixtures and struggled to transfer from synthetic to real audio. SCISSOR matches score-derived, frame-wise source queries to shared audio features and jointly assigns mixture evidence; it leads evaluated systems on held-out real recordings and shows stronger performance across several transfer and score-corruption tests.

  • Problem

    Shared pitches, timbres, and harmonics obscure orchestral source identity, while earlier score-concatenation gains were less consistent in dense mixtures and score-informed transfer from synthetic to real audio was difficult.

    Source: §1 · p. 1

  • Method

    SCISSOR matches frame-wise queries formed from score pitches and learned instrument identities to shared audio features, then jointly assigns evidence across instrument and background slots.

    Source: §1 · p. 1

  • Results

    SCISSOR led evaluated systems on held-out real recordings, reaching 6.80 dB Avg. SDR on held-out URMP versus 5.62 dB for Tunturi et al.’s score-only model.

    Source: §5 · p. 4, §4 · p. 4

  • Takeaways & Limitations

    With SynthSOD-only training, SCISSOR led on SynthSOD and zero-shot PHENICX-Anechoic, surpassed its audio-only control on zero-shot URMP, and degraded less under representative score errors, though instrument-identity errors remained challenging.

    Source: §5 · p. 4

Abstract

from arXiv · show

Orchestral separation recovers instrument sections from mixtures in which shared pitches, harmonics, and timbres obscure source identity. An aligned score provides instrument labels, note pitches, and activity times. A score-informed approach appends piano rolls to audio features before mask prediction. We introduce SCISSOR (Score-Conditioned Instrument Source Separation for Orchestral Recordings), which uses the score to form a frame-wise query for each source. Each query matches a shared audio representation, and a softmax over instrument and background slots jointly assigns overlapping time-frequency evidence. The queries retain instrument identity even when notes are missing from the score. After training on SynthSOD and a small set of URMP and PHENICX-Anechoic recordings, SCISSOR achieves the highest average SDR on held-out real recordings. With SynthSOD-only training, it leads on SynthSOD and zero-shot PHENICX-Anechoic, and improves on its audio-only control on zero-shot URMP. SCISSOR also degrades less under score corruption than the evaluated score-based baselines.

1. INTRODUCTION

Orchestral mixtures make instrument identity difficult to infer when sections share pitches, timbres, and harmonics. SCISSOR uses score-derived source queries to jointly assign audio evidence, with reported gains in separation quality and robustness to score corruption.

1. INTRODUCTION

  • Shared pitches, similar timbres, and overlapping harmonics can obscure which orchestral section produced a sound in the mixture.Earlier score concatenation gains were less consistent in denser orchestral mixtures, and score-informed transfer from synthetic to real audio was difficult.Source: §1 · p. 1
  • SCISSOR forms frame-wise instrument queries from score cues and matches them with encoded audio to allocate mixture evidence jointly across sources.Instrument identity remains in a query even when the score omits a note, while acoustic evidence can temper inaccurate score cues.Source: §1 · p. 1
  • SCISSOR achieved the highest average SDR among evaluated systems on held-out real recordings and SynthSOD.The reported benefits also include performing well with a small real-recording training set and losing less SDR under score corruption than evaluated score-based baselines.Source: §1 · pp. 1–2
  • The evaluation covers held-out synthetic and real recordings, zero-shot transfer to real audio, and robustness to score corruption.Training uses aligned SynthSOD audio and scores plus a small set of URMP and PHENICX-Anechoic recordings.Source: §1 · p. 2

2. METHOD

SCISSOR forms score-conditioned, frame-wise queries that represent both active pitches and instrument identity. It matches these queries to shared audio features and jointly assigns each band–frame region to an instrument or background slot before refining the target stems.

2. METHOD

  • The model rasterizes score events on the audio frame grid and combines each slot’s active pitches with a learned instrument identity to form a source query.An empty pitch vector leaves the identity query intact, keeping the source available when a note is omitted or mistimed in the score.Source: §2 · p. 2
  • A shared identity-conditioned complex refiner uses the target allocations and audio features to estimate complex masks and reconstruct the stems.Training combines waveform, complex-STFT, band-ratio-mask, competition-region, and scale-invariant waveform losses.Source: §2 · p. 2
  • Separate projections place audio features and score queries in a shared, unit-normalized space for cosine matching.Separate projections accommodate the different input statistics of the two modalities.Source: §2 · p. 2
  • A softmax jointly assigns each band–frame region across four string slots and a background slot, so a larger allocation to one source leaves less for competitors.The background slot absorbs non-target energy, while soft allocations let acoustic evidence counter an absent or incorrect score event.Source: §2 · p. 2

3. EXPERIMENT

The experiments use SynthSOD for synthetic training and testing, with URMP and PHENICX-Anechoic providing real-recording data and transfer evaluations. The comparisons include matched audio-only and score-based systems, held-out recordings, zero-shot settings, and score-corruption conditions.

3. EXPERIMENT

  • SynthSOD provides synthesized ensembles and orchestras with isolated stems across 18 instrument tracks; the selected subset has 484 pieces, split into 351 training, 42 validation, and 91 test pieces.The four target sections are violin, viola, cello, and double bass; non-target stems remain in the mixture.Source: §3 · p. 2
  • URMP contributes 22 selected real recordings with string targets, while PHENICX-Anechoic contributes four orchestral excerpts; the real-data split uses 15 recordings for training and five for validation.The held-out real evaluation uses five URMP recordings and one PHENICX-Anechoic recording, and the two corpora are reported separately.Source: §3 · pp. 2–3
  • The baselines include HTDemucs, BS-RoFormer, Tunturi et al.’s audio-only, score-informed, and score-only X-UMX models, and a matched audio-only SCISSOR.Systems use aligned SynthSOD training and validation data; Table 1 comparisons additionally train on the same aligned URMP and PHENICX-Anechoic recordings.Source: §3 · p. 3
  • The score-corruption test compares SCISSOR with Tunturi et al.’s score-informed and score-only models across 33 score perturbation conditions.Perturbations include timing, pitch, note presence, instrument identity, and score structure; the checkpoint, mixture, references, and inference procedure stay fixed while only the score changes.Source: §3 · p. 3

4. RESULTS

SCISSOR leads the evaluated systems on held-out real recordings after training with a small real-recording set. With SynthSOD-only training, it leads on SynthSOD and zero-shot PHENICX-Anechoic, but not zero-shot URMP; it also loses less under score corruption than the compared score-based systems.

Score-corruption test results

  • With 75% of notes deleted, SCISSOR loses 0.41 dB, versus 1.41 dB for Tunturi et al.’s score-informed model and 2.19 dB for its score-only model.Under complete instrument relabelling, SCISSOR loses 1.53 dB but retains the highest SDR at 3.92 dB; across 33 conditions, it falls below its audio-only control in 8, versus 23 and 33 cases for the score-informed and score-only systems.Source: Table 3 · p. 4, §4 · p. 4

Real-recording separation test results

  • SCISSOR reaches 6.80 dB Avg. SDR on held-out URMP versus 5.62 dB for the strongest external baseline, and 1.84 versus 0.32 dB on held-out PHENICX-Anechoic.URMP confidence intervals overlap, so the gains do not establish statistical significance; the single PHENICX-Anechoic test piece has no across-recording confidence interval.Source: §4 · p. 4

Synthetic and zero-shot separation test results

  • 6.31 dB Avg. SDR makes SCISSOR the SynthSOD-only leader on 59 test pieces, versus 5.96 dB for the strongest external baseline.The estimates’ confidence intervals overlap.Source: §4 · p. 4, Table 2 · p. 4
  • On zero-shot PHENICX-Anechoic, SCISSOR has the highest point estimate at 1.85 dB, while on zero-shot URMP it beats its control 2.52 to 1.88 dB but trails the score-only model at 4.80 dB.The PHENICX-Anechoic interval overlaps the audio-only control’s; the URMP intervals for SCISSOR and the score-only model do not.Source: §4 · p. 4, Table 2 · p. 4

5. CONCLUSION

SCISSOR uses score-conditioned instrument queries to allocate mixture energy across source and background slots. It performs well on held-out real recordings and several SynthSOD-only tests, while retaining challenges under instrument-identity errors.

5. CONCLUSION

  • SCISSOR forms frame-wise source queries from score pitches and instrument identities, matches them to shared audio features, and jointly assigns mixture energy to instrument and background slots.Identity-only queries keep instruments available when notes are omitted from the score.Source: §5 · p. 4
  • With a small real-recording training set, SCISSOR leads evaluated systems on held-out URMP and PHENICX-Anechoic; with SynthSOD-only training, it leads on SynthSOD and zero-shot PHENICX-Anechoic and beats its control on zero-shot URMP.Tunturi et al.’s score-only model leads on zero-shot URMP.Source: §5 · p. 4
  • SCISSOR degrades less under representative score errors, although instrument-identity errors remain challenging.Source: §5 · p. 4

COMPLIANCE WITH ETHICAL STANDARDS

The study used publicly available music datasets and conducted no new human or animal subject experiments.

COMPLIANCE WITH ETHICAL STANDARDS

Loading 2609.33265v1…