Source-linked AI summary
CREPE: A Convolutional Representation for Pitch Estimation
Jong Wook Kim, Justin Salamon, Peter Li, Juan Pablo Bello
TL;DR
Monophonic pitch tracking remains difficult for heuristic systems on challenging recordings despite its importance in music and speech processing. CREPE uses a deep convolutional neural network operating directly on the time-domain signal, and it achieves state-of-the-art performance with strong precision and noise robustness. The paper also provides a freely available pretrained Python implementation.
Problem
Existing pitch trackers rely mainly on DSP pipelines and heuristics, which can fail on challenging recordings despite pitch estimation’s importance in music and speech processing.
Method
CREPE is a data-driven deep convolutional neural network that estimates monophonic pitch directly from the time-domain audio signal.
Results
CREPE achieves state-of-the-art results, outperforming pYIN and SWIPE, while remaining highly precise and more robust to added noise in most cases.
Takeaways & Limitations
A pretrained CREPE model and Python implementation are made freely available for application and reproducibility.
Takeaways & Limitations
CREPE estimates each frame independently without temporal tracking, and its evaluation uses synthesized datasets to guarantee perfect f0 annotations.
Abstract
from arXiv · showhide
The task of estimating the fundamental frequency of a monophonic sound recording, also known as pitch tracking, is fundamental to audio processing with multiple applications in speech processing and music information retrieval. To date, the best performing techniques, such as the pYIN algorithm, are based on a combination of DSP pipelines and heuristics. While such techniques perform very well on average, there remain many cases in which they fail to correctly estimate the pitch. In this paper, we propose a data-driven pitch tracking algorithm, CREPE, which is based on a deep convolutional neural network that operates directly on the time-domain waveform. We show that the proposed model produces state-of-the-art results, performing equally or better than pYIN. Furthermore, we evaluate the model's generalizability in terms of noise robustness. A pre-trained version of CREPE is made freely available as an open-source Python module for easy application.
1. INTRODUCTION
Monophonic pitch estimation supports music and speech applications, but leading systems still rely on heuristics and can fail on challenging recordings. CREPE addresses this gap with a data-driven convolutional model operating directly on the time-domain signal.
- Pitch estimation supports music applications including dataset annotation and melody extraction, as well as speech analysis of prosody.
- pYIN is identified as the best-performing method to date among comparative studies of YIN-based approaches.
- Existing pitch trackers typically combine candidate-generating functions with heuristic pre- and post-processing rather than learning directly from data.
- Even pYIN can produce noisy estimates for uncommon instruments or rapidly fluctuating pitch curves.
- CREPE uses a deep convolutional neural network on the time-domain signal and reports state-of-the-art results, outperforming pYIN and SWIPE while remaining more noise-robust.
2. ARCHITECTURE
CREPE maps short time-domain audio excerpts through six convolutional layers to a 360-dimensional pitch representation. A deterministic weighted-average procedure converts the output activations into the final pitch estimate.
- The model processes a 1024-sample, 16 kHz waveform excerpt through six convolutional layers, producing a 2048-dimensional latent representation and 360 sigmoid outputs.
- Figure 1 depicts the convolutional architecture, whose output vector approximates a Gaussian curve before pitch estimation.
- Each output node represents a pitch value in cents relative to a reference frequency, with 360 values spanning six octaves at 20-cent intervals.
- The predicted pitch in cents is computed as a weighted average of the associated pitch values, then converted to frequency in hertz.
- Training targets use 360 frequency bins with the ground-truth bin activated and Gaussian blurring that decays with a 25-cent standard deviation.
- High activations indicate that the input likely has a pitch close to the corresponding output-node pitches.
- The network is trained by minimizing binary cross entropy between the target and predicted vectors.
- ADAM optimization uses a learning rate of 0.0002, while batch normalization and dropout with probability 0.25 are applied in the convolutional architecture.
3. EXPERIMENTS
CREPE is evaluated against pYIN and SWIPE on controlled synthetic datasets, varying pitch tolerance, additive noise, learned filters, and instrument coverage. It generally maintains stronger accuracy and robustness, while performance depends on timbre, frequency, and dataset representation.
- Datasets: The evaluation uses RWC-synth and MDB-stem-synth, with perfect f0 annotations; MDB-stem-synth provides more realistic timbral diversity than the homogeneous RWC-synth dataset.The MDB-stem-synth collection contains 230 monophonic tracks spanning 25 instruments.
- Pitch Accuracy: CREPE’s performance advantage becomes larger at stricter pitch tolerances, exceeding the baselines by over 8 percentage points at the 10-cent RPA threshold.The study also reports RPA at 25 and 50 cents for comparison.
- Noise Robustness: CREPE maintains the highest accuracy across SNR levels for pub and white noise, and across nearly all pink-noise levels, while brown noise favors pYIN.Below 10 dB SNR, CREPE performs better in all tested cases and shows consistently lower performance variance than the baselines.
- Model Analysis: The first-layer filters capture overtones and, for the more heterogeneous dataset, also overlap substantially with the ground-truth frequency distribution.The authors interpret this as evidence that the network adapts to dataset-specific timbre and frequency distributions.
- Performance by Instrument: On MDB-stem-synth, performance is worse for higher-frequency instruments and for the unseen dizi timbre, whereas well-represented timbres and frequency ranges yield better results.Dizi tracks came from one artist and were absent from the corresponding training and validation sets.
4. DISCUSSIONS AND CONCLUSION
CREPE achieves strong pitch-tracking performance, but its robustness and invariance remain bounded by transformations and temporal structure not explicitly modeled.
- CREPE obtains state-of-the-art results, outperforming pYIN and SWIPE across datasets with homogeneous and heterogeneous timbre.
- CREPE remains highly accurate under a strict evaluation threshold of 10 cents and is more robust to added noise in most cases.
- The architecture does not explicitly guarantee invariance to pitch-preserving transformations such as distortion and reverberation.Data augmentation is proposed as a way to help the model learn such invariance.
- CREPE estimates each frame independently without temporal tracking, unlike pYIN’s HMM-based enforcement of temporal smoothness.The authors propose adding recurrent layers through a jointly trained convolutional-recurrent neural network.