PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-domain Neural Pitch and Periodicity Estimation

Max Morrison, Caedon Hsieh, Nathan Pruyne, Bryan Pardo

arXiv:2301.12258v3eess.AScs.SD

TL;DR

Existing periodicity estimators have weak voiced/unvoiced accuracy, and neural pitch estimators have not shown good cross-domain generalization. This paper improves neural pitch and periodicity estimation, adds entropy-based periodicity decoding, and trains an estimator on speech and music. The proposed approach reports state-of-the-art performance, including cross-domain results, with neural CPU inference approaching DSP-based speeds.

  • Problem

    Existing periodicity estimators have low voiced/unvoiced accuracy, while neural pitch estimators have not shown good generalization across data domains.

    Source: §I · pp. 1–2

  • Method

    The paper improves neural pitch estimators and introduces entropy-based periodicity decoding, alongside training a neural estimator on both speech and music.

    Source: §I · p. 2

  • Results

    Cross-domain training reaches 99.62% music RPA and 95.12% speech RPA; proposed methods reduce baseline pitch estimation error by at least 28% and entropy-based periodicity estimation error by at least 46%.

    Source: §VI · pp. 6–7

  • Takeaways & Limitations

    Neural pitch and periodicity estimation approaches the CPU speed of state-of-the-art DSP methods, and the authors release penn with code and pretrained models.

    Source: §I · p. 2, §VII · p. 8

Abstract

from arXiv · show

Pitch is a foundational aspect of our perception of audio signals. Pitch contours are commonly used to analyze speech and music signals and as input features for many audio tasks, including music transcription, singing voice synthesis, and prosody editing. In this paper, we describe a set of techniques for improving the accuracy of widely-used neural pitch and periodicity estimators to achieve state-of-the-art performance on both speech and music. We also introduce a novel entropy-based method for extracting periodicity and per-frame voiced-unvoiced classifications from statistical inference-based pitch estimators (e.g., neural networks), and show how to train a neural pitch estimator to simultaneously handle both speech and music data (i.e., cross-domain estimation) without performance degradation. Our estimator implementations run 11.2x faster than real-time on a Intel i9-9820X 10-core 3.30 GHz CPU$\unicode{x2014}$approaching the speed of state-of-the-art DSP-based pitch estimators$\unicode{x2014}$or 408x faster than real-time on a NVIDIA GeForce RTX 3090 GPU. We release all of our code and models as Pitch-Estimating Neural Networks (penn), an open-source, pip-installable Python module for training, evaluating, and performing inference with pitch- and periodicity-estimating neural networks. The code for penn is available at https://github.com/interactiveaudiolab/penn.

I. INTRODUCTION

Pitch supports speech and music perception and many audio tasks, but existing periodicity estimators have weak voiced/unvoiced accuracy, while neural pitch estimators are slow and generalize poorly across domains. The paper improves neural pitch and periodicity estimation, introduces entropy-based periodicity decoding, and trains one estimator across speech and music.

Motivation

  • Pitch is related to fundamental frequency and supports speaker tracking, speech communication, and understanding chords and melody.Pitch contours are also used in music transcription, music information retrieval, speech analysis, synthesis, and prosody editing.Source: §I · p. 1
  • Existing periodicity estimators have low voiced/unvoiced accuracy, and some require silence post-processing that sets low-volume frames to zero.Periodicity is used to identify voiced speech and select meaningful pitch-contour regions.Source: §I · p. 1
  • Existing neural pitch estimators are noise-robust but slow relative to DSP methods and have not shown good generalization across data domains.Machine-learning pitch estimation is reported to outperform DSP methods in accuracy and noise robustness.Source: §I · pp. 1–2

Contributions and results

  • The authors train a neural pitch estimator to handle speech and music simultaneously without performance degradation.The resulting estimator is described as fast and accurate on both domains.Source: §I · p. 2
  • The paper introduces an entropy-based method for extracting per-frame periodicity and voiced/unvoiced classifications from statistical inference-based pitch estimators.The method is intended to improve framewise voiced/unvoiced classification of speech.Source: §I · p. 2
  • The proposed approach achieves state-of-the-art pitch and periodicity estimation, with neural inference approaching the CPU speed of DSP-based methods.The authors also release penn, a pip-installable library containing training, evaluation, and inference code and pretrained models.Source: §I · p. 2

II. BASELINE ARCHITECTURES

The paper follows a neural pitch-tracking paradigm in which a model estimates pitch distributions and a selection algorithm produces framewise pitch estimates. It uses CREPE and two CREPE-derived trackers as baselines.

Baseline paradigm

  • A statistical candidate-generating model maps each audio frame to a posterior distribution over possible pitches, then a candidate-selection algorithm chooses a pitch estimate at each time step.The paper identifies CREPE as perhaps the most widely used tracker of this kind.Source: §II · p. 2

Baseline systems

  • The authors use CREPE and two pitch trackers built upon CREPE to demonstrate the efficacy of their methods.The section introduces these three systems as baseline models.Source: §II · p. 2

Baseline paradigm

  • Candidate selection operates on the sequence of probability distributions produced from audio frames.The paper describes the selected output as the best pitch estimate at each time step.Source: §II · p. 2

A. CREPE

CREPE is a convolutional neural pitch estimator that processes normalized audio and predicts pitch-bin probabilities. Its training targets and local expected value decoding shape how it represents and extracts pitch.

A. CREPE

  • CREPE uses six convolutional blocks followed by a linear layer to estimate pitch from 1024 samples of 16 kHz audio.The waveform is preprocessed to have zero mean and unit standard deviation.Source: A. CREPE · p. 2
  • The network produces logits that become independent Bernoulli posterior probabilities for each pitch bin after sigmoid normalization.CREPE uses 360 bins spaced every 20 cents from 31 to 2006 Hz.Source: A. CREPE · p. 3
  • CREPE trains with sigmoid activation and binary cross-entropy on each bin, allowing simultaneous high confidence in multiple pitch bins.It blurs one-hot training targets with a 25-cent standard deviation, encouraging probability mass in adjacent bins.Source: A. CREPE · p. 3
  • Local expected value decoding removes probability outside nine bins centered on the argmax, then takes the remaining distribution's expected value.This produces pitch in Hz from the quantized output space.Source: A. CREPE · p. 3

B. DeepF0

The paper describes DeepF0 and FCNF0 as CREPE-related baselines, outlines their training procedure, and introduces a set of improvements applied to all three baseline neural estimators.

B. DeepF0

  • DeepF0 changes CREPE's architecture while retaining its data preprocessing, quantization, batch size, loss, early stopping, and sampling rates.It uses dilated convolutions to increase receptive field and residual connections to facilitate faster training.Source: B. DeepF0 · p. 3

C. FCNF0

  • FCNF0 modifies CREPE for speed by omitting convolutional zero-padding and dropout, replacing the output linear layer with a convolution, and using fully-convolutional inference.It uses 8 kHz input and 486 bins spanning 30–1000 Hz at 12.5-cent spacing.Source: C. FCNF0 · p. 3

D. Training Approach

  • CREPE, FCNF0, and DeepF0 use batch size 32 and stop after 16,000 steps without validation-accuracy improvement.The reimplementations use 64 validation batches without shuffling after each epoch.Source: D. Training Approach · p. 3

III. IMPROVING PITCH AND PERIODICITY ESTIMATION

  • The proposed improvements are applied to CREPE, DeepF0, and FCNF0, producing CREPE++, DeepF0++, and FCNF0++.The authors refer readers to Tables I and II for incremental and cumulative results.Source: §III · p. 3

A. Proposed best practices

The proposed best practices refine neural pitch models’ resolution, training, normalization, loss, and decoding, while entropy-based periodicity estimation derives a periodicity score from the pitch posterior. The section also describes training on unpitched audio and using an 8 kHz sample rate.

A. Proposed best practices

  • Reducing pitch-bin width to 5 cents lowers the expected quantization error to 1.25 cents.The models use 1440 pitch bins.Source: A. Proposed best practices · p. 3
  • Training with unpitched frames assigned random pitch bins promotes high-entropy posterior distributions in aperiodic regions for periodicity estimation.The proposed practices also include training to convergence, omitting input normalization, using categorical cross-entropy, increasing batch size to 128, and applying layer normalization.Source: A. Proposed best practices · p. 4

B. Proposed best practices from prior works

  • Removing dropout improves performance and training speed in the proposed models, which the authors attribute to using capacity previously constrained by dropout.The passage notes that this observation had also been made by the authors of DeepF0 and FCNF0.Source: B. Proposed best practices from prior… · p. 4
  • Local expected value decoding slightly improves pitch accuracy over argmax, and the models use 8 kHz sampling because its accuracy improvements outweigh lost harmonic information.The passage also says local expected value decoding addresses banding caused by requantizing an already quantized signal.Source: B. Proposed best practices from prior… · p. 4

IV. PERIODICITY ESTIMATION WITH STATISTICAL PITCH ESTIMATORS

  • Entropy-based periodicity is derived from the categorical pitch posterior by normalizing entropy by ln P and subtracting it from 1, so lower entropy means higher periodicity.Unlike the direct prediction method of Gfeller et al., it requires no additional loss or network-architecture modification; for two equal posterior peaks with P=1440, it gives 0.905 versus 0.5 for the max method.Source: §IV · pp. 4–5

A. Binary voicing decisions

Per-frame voiced/unvoiced classifications are obtained by thresholding estimated periodicity.

A. Binary voicing decisions

  • The method thresholds predicted pitch contours and corresponding periodicity estimates to produce per-frame binary voiced/unvoiced decisions, using α ∈ [0, 1] as a voicing-threshold hyperparameter.Source: A. Binary voicing decisions · p. 5

V. EVALUATION

The evaluation tests whether the proposed methods improve neural pitch and periodicity estimation and whether models generalize across speech and music. It uses annotated music and speech datasets and evaluates periodicity through voiced/unvoiced classification F1.

V. EVALUATION

  • The evaluation tests pitch and periodicity improvements, speech–music domain generalization, and CPU speed relative to DSP-based estimators.It also compares pitch-estimation results with HarmoF0, which requires a full audio file for processing.Source: §V · p. 5

A. Data

  • The study uses MDB-stem-synth for music and PTDB for speech, splitting files in each dataset randomly into 70% training, 15% validation, and 15% test.MDB-stem-synth contains 230 solo stems and 15.6 hours; PTDB contains 4,718 English speech and laryngograph recordings and 9.6 hours.Source: A. Data · p. 5

B. Evaluating periodicity and voicing decisions

  • Periodicity is evaluated using voiced/unvoiced classification F1, with decisions extracted from estimated periodicity using both max and entropy methods.The voicing threshold is searched on validation data from both datasets, including a fine-grained sequential search because models without the unvoiced training strategy exhibit a narrow optimum.Source: B. Evaluating periodicity and voicing… · p. 5

C. Evaluating pitch

Pitch evaluation reports error in cents, raw pitch accuracy, and raw chroma accuracy, alongside a table comparing model families on two datasets. The section also identifies FCNF0++ with local expected value decoding as the authors’ preferred model for most downstream applications.

C. Evaluating pitch

  • RPA and RCA use a 50-cent pitch threshold, and their gap indicates half- and double-frequency errors because RCA does not penalize octave shifts.Pitch error is measured in cents, with 100 cents corresponding to one semitone.Source: C. Evaluating pitch · p. 5
  • The table compares baselines, proposed models, and open-source models on PTDB and MDB-stem-synth for pitch error, RPA, RCA, voiced/unvoiced F1, and speed.Source: Table I · p. 6
  • FCNF0++ with local expected value decoding is considered most useful for most downstream applications, with competitive accuracy and speed.Source: C. Evaluating pitch · p. 6
  • PYIN uses the sum of peak-picked densities for periodicity decoding; the table’s arrows indicate whether higher or lower values are better.Source: C. Evaluating pitch · p. 6

D. Evaluating speed

The paper evaluates inference speed and compares the proposed estimators with neural and DSP-based alternatives. Its experiments also assess ablations and whether FCNF0++ generalizes across speech and music.

D. Evaluating speed

  • Inference speed is measured as average seconds per second of audio at a 10 ms hop, on an Intel i9-9820X CPU and an NVIDIA GeForce RTX 3090 GPU.GPU inference uses batches of up to 2048 frames; CPU batch size is not limited, and reported times include audio loading, device copying, and saving.Source: D. Evaluating speed · p. 6
  • The speed evaluation does not benchmark real-time streaming, though the methods can support streaming and its latency depends on batch size.GPU speeds are not reported for DSP baselines without GPU implementations.Source: D. Evaluating speed · p. 6

E. Experiments

  • FCNF0++ is the broadest-use model: its pitch accuracy is roughly equal to DeepF0++, its periodicity estimation is equal or better, and it runs an order of magnitude faster.The comparison also evaluates local expected value decoding with a window size of 19, or 95 cents, selected by hyperparameter search.Source: E. Experiments · p. 6
  • The experiments compare CREPE++, DeepF0++, and FCNF0++ with PYIN, DIO, and open-source CREPE, and test FCNF0++ trained on either dataset alone or both jointly.They also ablate each proposed method and examine training pitch-bin distributions to investigate cross-domain generalization.Source: E. Experiments · p. 6

VI. RESULTS

The proposed system improves pitch and periodicity estimation, with FCNF0++ performing well across speech and music. The results also identify unvoiced-frame training and entropy-based periodicity estimation among the most notable improvements.

Improved pitch and periodicity estimation

  • At least 28% lower pitch estimation error in cents and at least 46% lower entropy-based periodicity estimation error are reported across the baseline models.The paper reports these reductions in Table I and notes that, apart from AMP, it does not apply compute optimizations such as graph compilation or ONNX exporting.Source: §VI · p. 6
  • FCNF0++ combines pitch accuracy roughly equal to DeepF0++ with equal-or-better periodicity estimation and CPU inference speeds approaching DSP-based methods.The paper also reports GPU inference speeds capable of processing large audio datasets in minutes.Source: §VI · p. 7

Ablations of proposed improvements

  • Random-bin training on unpitched frames substantially improves periodicity accuracy, while the voiced/unvoiced threshold has a broader competitive-F1 region than without that strategy.The ablation results also identify unvoiced training, entropy-based periodicity, smaller quantization width, and layer normalization as notable improvements.Source: §VI · p. 7, Figure 2 · p. 7, Table II · p. 7

Cross-domain generalization

  • 99.62% music and 95.12% speech RPA with cross-domain training exceed HarmoF0’s domain-specific 98.46% and 94.59%, respectively, while permitting low-latency and streaming inference.The paper reports these values as state-of-the-art, noting possible test-partition variation in the comparison.Source: §VI · p. 7
  • Training on both datasets performs well on both test domains, while single-dataset models perform poorly where their training pitch-bin distributions fail to cover the test frequencies.The paper reports that CREPE++ and DeepF0++ show similar cross-domain generalization; it identifies universal estimation with high-entropy training data as future work.Source: §VI · p. 7

VII. CONCLUSION

The paper presents methods for improving neural pitch and periodicity estimation on speech and music, including entropy-based periodicity decoding. It reports performance gains and inference speeds approaching DSP-based CPU methods, while identifying several directions for further work.

VII. CONCLUSION

  • The work advances speech and music pitch and periodicity estimation with methods including unpitched-frame training, finer pitch quantization, and entropy-based periodicity decoding.The paper states that its entropy-based method improves periodicity estimation.Source: §VII · pp. 7–8
  • The neural estimators’ inference speeds approach those of state-of-the-art DSP-based CPU methods.The paper says its experiments are reproducible using configuration files in the open-source penn Python library.Source: §VII · p. 8
  • Future work includes generalization to unseen datasets, polyphonic estimation, higher-entropy training distributions, and faster CPU inference than DSP-based methods.The paper also proposes investigating whether periodicity’s correlation with signal energy is detrimental for downstream applications.Source: §VII · p. 8
Loading 2301.12258v3…