On a Separate Note: Robust Score-Informed Note Separation with a Two-Stream TFC-TDF U-Net and Adaptive Set Ownership

1. INTRODUCTION

• Notes from the same instrument can share timbre and harmonic content, while vibrato, inharmonicity, room acoustics, and attacks further complicate separation.Existing music source separation usually targets instrument stems, while score-informed systems return voice- or part-level waveforms rather than individual notes.Source: §1 · p. 1
• Prior note-separation methods rely on filtering, NMF, or templates, with trade-offs between instrument-specific adjustment and general-purpose low fidelity.Their assumptions limit performance on octave overlaps, inharmonic partials, and broadband attacks.Source: §1 · p. 1
• NoteGrab conditions note extraction on aligned pitch, onset, and offset, while ASO compares concurrent estimates and reallocates shared mixture energy.The approach uses harmonic and percussive streams, with selective harmonic gating aimed at lower-octave collisions.Source: §1 · p. 1
• SCNS-Train contains 25,729 mixtures and 743,920 isolated-note targets, while SCNS-Eval uses recorded-note samples absent from training.The paper presents these datasets alongside its deep-learning approach to score-informed note separation.Source: §1 · p. 1
• NoteSep outperforms the evaluated Score-Informed NMF and Melodyne baselines and achieves competitive performance on real ensemble recordings.The real-recording evaluation aggregates note estimates into instrument parts for comparison with prior specialized systems.Source: §1 · p. 2

2.1. Task and system overview

• For each aligned score event, NoteGrab estimates the performed note waveform from a local mixture window; NoteSep processes each queried event before joint ASO reallocation.The score event supplies pitch, onset, and offset.Source: §2.1 · p. 2

2.2. Inputs

• Selective gating zeros low harmonic input only when a score-overlapping note lies one or two octaves lower, while leaving percussive input ungated.The gate is compared with an otherwise identical ungated variant; selective gating has higher SI-SDR and is less susceptible to octave confusion.Source: §2.2 · p. 2

2.3. Model architecture

• NoteGrab uses two TFC–TDF v3 U-Net streams, with bidirectional cross-attention exchanging intermediate features and pitch, onset, and offset conditioning applied through FiLM.The streams process harmonic and percussive inputs and sum their outputs in the waveform domain.Source: Figure 1 · p. 2, §2.3 · p. 2

2.4. Joint separation with Adaptive Set Ownership

• ASO gates raw note magnitudes, predicts log-gains using cross-note context, normalizes adjusted weights, and reallocates mixture STFT energy among estimates.Its fallback uses raw note magnitudes with mixture phase when all weights in a bin are zero; magnitude reallocation favors safer estimates with less leakage over perfect mixture consistency.Source: §2.4 · pp. 2–3
• For K queried notes, inference uses K shared-weight NoteGrab passes and one ASO pass; incorporating ASO for a single-note query requires labels for overlapping notes.The authors note that the ASO cost is relatively small when applications already require extracting every note.Source: §2.4 · p. 3

3.1. Experimental setup

• SCNS-Eval comprises 144 disjoint PDMX scores, 3,473 scored notes, 16 held-out instruments, and held-out note libraries, with no score or audio library shared with training.SCNS-Train contains 25,729 mixtures and 743,920 isolated-note targets; training data are rendered from six symbolic datasets and recorded-note libraries, with gain, detuning, pitch-shift, echo, room-response, and noise augmentations.Source: §3.1 · p. 3

3.3. Editing evaluation

• In Table 2’s 13,892-edit evaluation, ASO improves unedited and final-sum 2f scores and leads across metrics, while summed independent NoteGrab estimates have mixture-consistency errors.The evaluation separates requested-edit errors from errors in preserving the rest of the performance.Source: Table 2 · p. 4, §3.3 · p. 4

3.2. Note separation results

• 7.39 dB SI-SDR with ASO, which also reduces off-target energy relative to NoteGrab Selective; bidirectional cross-attention and selective gating each improve SI-SDR.The paired ablations show gains from both NoteGrab components, with ASO yielding the strongest overall improvement.Source: §3.2 · p. 4
• In one octave-overlapping chord, ASO shifts energy between shared partials, and A♭4’s SI-SDR rises 5.16 dB.Near 831 Hz, A♭4’s second harmonic coincides with A♭5’s fundamental; near 1.66 kHz, energy moves from A♭4’s fourth harmonic toward A♭5’s second.Source: §3.2 · p. 4, Figure 3 · p. 4

3.4. Instrument separation

• On PHENICX, NoteSep reaches 8.45 dB SI-SDRi and 1.53 dB SDR; on Bach10, it reaches 10.10 dB SI-SDRi and 6.83 dB SDR, improving over NoteGrab.NoteSep remains slightly below the task-specific Miron CNN in SI-SDRi but outperforms the Miron NMF model; both test datasets were unseen during training.Source: Table 3 · p. 4, §3.4 · p. 4

• On SCNS-Eval, NoteSep reaches 7.39 dB median SI-SDR, compared with 2.49 dB for Score-Informed NMF.Source: §4 · p. 4
• The paper presents the first deep learning-based method for note extraction and separation, alongside SCNS-Train and a held-out 16-instrument SCNS-Eval benchmark.Source: §4 · p. 4
• The accompanying artifact provides code, datasets, checkpoints, audio examples, and an interactive note-level visualization; proposed application areas include editing, performance analysis, and music education.Source: §4 · p. 4