Source-linked AI summary
MIDIBack: Harmony-Aware Singing Pitch Correction via Joint Vocal-Accompaniment Symbolic Modeling
Joaquim Cavalcante, Yicheng Gu, Adriel Trajano, Yuri Malheiros, Thais Gaudencio
TL;DR
Automatic pitch correction must distinguish unintended intonation errors from expressive pitch variation, while existing approaches can omit note-level accompaniment context. MIDIBack jointly models vocal and backing events in OctupleMIDI; it reports 78.6% overall RPA and case studies show vocal predictions adapting to backing-harmony shifts.
Problem
APC must infer intended vocal notes because acoustic pitch alone does not distinguish unintended intonation errors from expressive variation, and prior systems omit rich note-level accompaniment context.
Source: §1 · p. 1Method
MIDIBack jointly models vocal and accompaniment events in a shared OctupleMIDI sequence and trains a context-aware predictor to invert synthetic vocal-note corruptions.
Source: §1 · p. 1, §2.4 · p. 2Results
78.6% overall RPA; MIDIBack reports competitive results against baselines, with evaluation across six pitch-corruption scenarios.
Source: §1 · p. 1, §3.2 · p. 4Takeaways & Limitations
Controlled accompaniment modulation changes vocal pitch residuals in the modulation direction, with an amplified effect over longer context windows.
Source: §3.4 · p. 4
Abstract
from arXiv · showhide
Automatic pitch correction (APC) requires distinguishing the unintended intonation errors from expressive pitch variation. Existing systems either lack explicit harmonic modeling, as vocal-only methods do, or do not directly use the note-level polyphonic context. Therefore, we propose MIDIBack, a note-level APC framework that jointly models the vocal and accompaniment events in a shared OctupleMIDI sequence. We evaluate MIDIBack under 6 note corruption regimes, including global outshift, learned note-dependent detuning, uniform perturbations, and their combinations. The resulting model achieves 78.6% overall raw pitch accuracy (RPA), and 81.5% under combined global outshift and learned detuning. Removing the accompaniment conditioning reduces RPA from 81.5% to 35.8% in outshift, showing the effectiveness of accompaniment context. Case studies on accompaniment modulation further illustrate that vocal note predictions
1. INTRODUCTION
MIDIBack addresses the difficulty of separating unintended intonation errors from expressive pitch variation by using accompaniment context to predict intended vocal notes. It jointly models vocal and backing events in OctupleMIDI and evaluates the system under multiple corruption regimes.
1. INTRODUCTION
- Expressive deviations and unintended intonation errors are difficult to distinguish from acoustic pitch alone, so APC must predict intended notes from musical cues.Trained singers may deviate from target notes without sounding out of tune, and listeners accept wider pitch deviations in voice than in fixed-pitch instruments.Source: §1 · p. 1
- Existing APC approaches either rely on rigid scale quantization, require manually adjusted vocal MIDI targets, or omit note-level accompaniment context.DDPC conditions on acoustic CQT spectra, while BERT-APC uses vocal notes and coarse metrical and temporal metadata.Source: §1 · p. 1
- 78.6% overall RPA; MIDIBack jointly models vocal and accompaniment events in OctupleMIDI and reports competitive results against all baselines.It also reports 91.3% RPA on clean transcriptions and 81.5% under severe corruption settings.Source: §1 · p. 1
2. METHODOLOGY
MIDIBack estimates vocal notes from a shared sequence of symbolic vocal and backing events. Its pipeline transcribes both streams, represents musical events in OctupleMIDI, applies synthetic vocal corruption, and trains a context-aware predictor.
2.1. Multi-Track Symbolic Transcription
- Singing and backing audio are transcribed into note streams, then Madmom extracts note-level onset, duration, signature, and pitch attributes from the merged streams.GAME4 transcribes continuous vocal notes and MuScriptor transcribes backing notes.Source: §2.1 · p. 2
2.2. OctupleMIDI Representation
- Vocal and backing notes are merged into one chronological OctupleMIDI sequence, with each event represented by eight musical attributes.The attributes are bar, position, program, pitch, duration, velocity, time signature, and tempo; program 0 denotes vocals and nonzero programs denote backing tracks.Source: §2.2 · p. 2
- Continuous vocal pitches are encoded by interpolating adjacent semitone embeddings, retaining microtonal intonation nuance without expanding the vocabulary.The interpolation weight is the fractional part of the continuous pitch.Source: §2.2 · p. 2
2.3. Intonation Corruption
- Synthetic vocal corruption combines phrase-level key outshift, autoregressive detuner drift, and uniform noise, while accompaniment notes remain uncorrupted.The backing notes serve as an invariant musical anchor.Source: §2.3 · p. 2
2.4. Context-Aware Pitch Prediction
- Adversarial-MidiBERT is trained to invert synthetic vocal-note corruptions against rounded transcription centers, with backing tokens serving only as conditioning context.Loss gradients from backing-token positions are blocked.Source: §2.4 · p. 2
2. METHODOLOGY
- 78.6% overall RPA is reported for MIDIBack in the evaluation table’s stated scope of objective metrics across six corruption regimes.The provided table caption identifies RPA, MAE, Corr, and Corp as the reported measures; it does not supply cell values.Source: Table 1 · p. 3
3. EXPERIMENTS
Experiments compare MIDIBack with APC baselines across six pitch-corruption scenarios, evaluate training and architecture choices, and test whether vocal predictions respond to modulated accompaniment.
3.1. Experiment Setup
- Evaluation uses vocal-only and accompanied-singing datasets, compares against DDPC and BERT-APC, and reports RPA, MAE, Corr, and Corp across note-corruption regimes.The vocal correction training set comprises 2,356 multi-track songs; DDPC conditions on raw CQT spectra, whereas BERT-APC uses vocal context and metrical cues.Source: §3.1 · p. 3
3.2. Experimental Results
- MIDIBack matches DDPC on Uniform (73.1% vs. 74.6%) and Detune (80.8%) and improves on clean inputs (95.2% vs. 86.1%), reducing Corp from 13.9% to 4.8%.Against BERT-APC, it is comparable on learned detuning (92.3%) and clean inputs (97.1%), and performs better on Outshift (36.2%) and compound conditions (39.4% and 40.0%).Source: §3.2 · p. 4
3.3. Ablation Study
- 78.6% overall RPA and 62.0% on Outshift + Uniform are the highest results for training across all scenarios; Detune + Outshift leads compound evaluation at 82.8% RPA and 80.0% Corr.Single-corruption training tends to overfit its regime, while pairwise combinations improve stability.Source: §3.3 · p. 4
- Removing accompaniment conditioning hurts performance across all scenarios, especially global outshift, while replacing AdversarialMidiBERT with MusicBERT degrades almost all settings.Table 3 reports architecture ablations for the Uniform + Detune + Outshift regime.Source: §3.3 · p. 4, Table 3 · p. 4
3.4. Case Study
- With vocal notes fixed, predicted pitch residuals follow backing modulation of ±0.5 or ±1.0 semitones, with a stronger response over 8-bar than 4-bar windows.A +1.0-semitone example shows accompaniment shifts guiding vocal predictions toward harmonically modulated targets.Source: §3.4 · p. 4, Figure 2 · p. 4, Figure 3 · p. 4
4. CONCLUSION
MIDIBack predicts vocal pitch using symbolic backing-track context in a shared OctupleMIDI sequence. The complete model leads overall RPA across six controlled corruption regimes, while evaluation remains limited to transcription-derived targets and simulated corruptions.
4. CONCLUSION
- Across six controlled corruption regimes, the complete MIDIBack model achieves the highest overall RPA among the evaluated training regimes.Removing accompaniment conditioning degrades performance, especially under global pitch offsets, and vocal predictions adapt to modulated backing tracks.Source: §4 · p. 4
- The evaluation uses transcription-derived MIDI targets and simulated pitch corruptions, benchmarking corruption recovery rather than perceptual vocal tuning quality.The authors identify professionally annotated datasets and subjective tests on end-to-end audio resynthesis as future evaluation directions.Source: §4 · p. 4
- MIDIBack conditions vocal pitch prediction on symbolic backing tracks through a shared OctupleMIDI sequence.Source: §4 · p. 4