Source-linked AI summary
Revisiting Input Time-frequency Representations in Multi-pitch Estimation for Vocal Ensembles
Junyoung Koh, Hao-Wen Dong
TL;DR
Vocal-ensemble MPE is challenging because closely spaced fundamentals create overlapping harmonics, and commonly used HCQT features are costly to extract. The paper compares HCQT with directly supplied linear STFT inputs and finds that STFT outperforms HCQT while reducing feature-extraction cost.
Problem
Closely spaced fundamentals in vocal ensembles produce overlapping harmonics, while commonly used HCQT representations increase feature-extraction cost for on-the-fly mixtures.
Source: §1 · p. 1Method
The paper compares HCQT with a fixed-resolution linear STFT whose frequency bins are supplied directly to the model without a pitch-aligned input grid.
Source: §1 · p. 1Results
The linear STFT consistently outperforms HCQT-based inputs across five encoder architectures while substantially reducing feature-extraction time.
Source: §1 · p. 1Takeaways & Limitations
Longer analysis windows and spectral coverage beyond 7.7 kHz do not improve performance, while restricting input to the predicted pitch range reduces the STFT advantage.
Source: §5 · p. 4Reference F0 is obtained automatically, mostly with pYIN, so errors may propagate to evaluation; independently recorded stem mixtures also do not reflect interactions in jointly recorded ensembles.
Source: §4.3 · p. 4
Abstract
from arXiv · showhide
Multi-pitch estimation in vocal ensembles is challenging because singers occupy overlapping pitch ranges and often sing at closely spaced fundamental frequencies, causing their harmonics to overlap in time-frequency representations. Existing models commonly use harmonic constant-Q transform (HCQT)-based representations to provide frequency-adaptive resolution, at the cost of expensive feature extraction when training mixtures are generated on the fly. We revisit this design and compare HCQT with a linear short-time Fourier transform (STFT), whose frequency bins are directly provided as model inputs. Despite its fixed frequency resolution and the absence of a pitch-aligned input grid, the linear STFT outperforms HCQT while substantially reducing feature-extraction cost. Further analysis shows that a longer analysis window or broader spectral coverage provides no additional improvement, while restricting the input to the predicted pitch range reduces the advantage of the linear STFT. These results suggest that finer frequency resolution does not necessarily improve vocal-ensemble MPE, and that shorter analysis windows can be more effective for time-varying vocal pitches.
1. INTRODUCTION
Vocal-ensemble MPE is difficult because closely spaced fundamentals produce overlapping harmonics, while commonly used HCQT features are costly to extract for on-the-fly mixtures. This paper studies a linear STFT input and reports that it outperforms HCQT while reducing feature-extraction time.
1. INTRODUCTION
- Closely spaced fundamental frequencies in vocal ensembles produce overlapping harmonic components, making their separation in time-frequency representations difficult.Source: §1 · p. 1
- The linear STFT supplies fixed-resolution frequency bins directly, without frequency-dependent windows or a pitch-aligned input grid.HCQT uses frequency-dependent CQT windows and stacked harmonic-ratio channels, which align harmonics but increase feature-extraction cost, especially for on-the-fly mixtures.Source: §1 · p. 1
- The linear STFT consistently outperforms HCQT-based inputs across five encoder architectures while substantially reducing feature-extraction time.Increasing the analysis-window length or spectral range does not improve overall performance, while restricting input to the predicted pitch range reduces the STFT advantage.Source: §1 · p. 1
2. RELATED WORK
Prior vocal-ensemble transcription work includes salience-map methods, voice assignment, and ensemble separation. HCQT is widely used in these methods, while related music systems commonly use log-frequency representations and convolutional architectures that exploit harmonic alignment.
2. RELATED WORK
- Vocal-ensemble methods commonly use Deep Salience’s time-by-pitch activation formulation, including an HCQT-based CNN and later voice-assignment or ensemble-separation approaches.Source: §2 · p. 1
- The model architecture processes a spectrogram representation through a convolutional stem and encoder, then predicts frame-wise activations over 360 pitch bins from C1 to C7.Source: Figure 2 · p. 2
- HCQT-based inputs have been adopted for MPE and paired with convolutional architectures that preserve the pitch axis and exploit harmonic alignment.Basic Pitch constructs harmonic channels from frequency-shifted versions of a single CQT to reduce computational cost.Source: §2 · p. 2
3. METHOD
The method compares frequency-adaptive HCQT with fixed-resolution STFT inputs for vocal-ensemble MPE. It uses a 20-cent output grid and a model whose encoder operates over time to predict pitch-bin activations.
3.1. Preliminaries: Frequency Resolution
- HCQT’s frequency-adaptive analysis is compared with fixed-resolution STFT because vocal-ensemble MPE must distinguish closely spaced fundamentals whose absolute spacing in Hz increases with frequency.Source: §3.1 · p. 2
3.2. Pitch Quantization
- MPE is framed as frame-wise multilabel classification over 360 bins on a 20-cent grid spanning six octaves.Source: §3.2 · p. 2
- The fixed-resolution linear STFT uses a 2048-sample window, 256-sample hop, and 44.1 kHz sampling rate, and supplies the first K STFT bins directly to the model.The log-magnitude transform compresses dynamic range and keeps silent bins finite; the linear STFT spans 0–7.7 kHz in uniform 21.5 Hz intervals and extends beyond the predicted pitch range.Source: §3.2 · p. 2
3.3. Model
- A convolutional stem feeds an encoder that operates over time and a sigmoid head that predicts activations for 360 pitch bins.The evaluated encoders are ConvNeXt, TCN, Conformer, ConvNeXt V2, and InceptionNeXt; predictions are thresholded rather than peak-picked from a salience map.Source: §3.3 · p. 2
4. EXPERIMENTS
The experiments compare linear STFT and HCQT inputs across vocal-ensemble datasets and encoder architectures, finding that STFT performs better at lower computational cost. Window and spectral-range ablations qualify that advantage, while limitations include automatically estimated reference F0 and mixtures that omit interactions in jointly recorded ensembles.
4.1. Experimental Setup
- Training mixtures combine one to five vocal stems; evaluation uses held-out songs, while ablations use exact-bin F1 on 2,894 validation segments.Ablation thresholds are fixed at 0.5; each validation segment is 30 seconds.Source: §4.1 · pp. 2–3
4.2. Results
- Across encoders, linear STFT beats HCQT by 0.020 pooled validation F1 (95% interval 0.018–0.022), with lower computation and higher exact-bin accuracy on all five datasets.The exact-bin accuracy advantage generally decreases under a 50-cent tolerance.Source: Table 1 · p. 3, §4.2 · p. 4, Figure 3 · p. 4
- F1 peaks with a 46 ms STFT window; extending coverage beyond 7.7 kHz adds no benefit, while restricting input to C7 reduces the STFT advantage.HCQT reaches pitch error comparable to the best STFT configuration but has lower F1 at every tolerance.Source: §4.2 · p. 4, Figure 4 · p. 4
4.3. Limitations
- Reference F0 is estimated automatically, mostly with pYIN, and independently recorded stems do not reproduce the acoustic and tuning interactions of jointly recorded ensembles.Reference-estimation errors may propagate into evaluation.Source: §4.3 · p. 4
5. CONCLUSION
The conclusion finds that linear STFT is an effective, lower-cost input for vocal-ensemble MPE, outperforming HCQT across encoder comparisons. It leaves open whether these findings extend to instruments with different spectral and temporal characteristics.
- Under a common encoder, linear STFT outperforms HCQT at substantially lower computational cost, and every additional encoder also exceeds HCQT in pooled F1.Source: §5 · p. 4
- Longer analysis windows and spectral coverage beyond 7.7 kHz add no improvement, whereas restricting input to the predicted pitch range reduces the STFT advantage.Source: §5 · p. 4
- Future work will test whether these findings extend to musical instruments with different spectral and temporal characteristics.Source: §5 · p. 4