Unsupervised Instantaneous Phase and Frequency Tracking by Inverse Voice Synthesis

1. INTRODUCTION

• F0 is the rate of vocal-fold vibration and shapes intonation in speech and singing, as well as serving as a key control in parametric voice synthesis.Analysis-by-synthesis estimates input parameters and rebuilds the signal with a source-filter model.Source: §1 · p. 1
• Spectral losses provide weak supervision for periodicity and omit phase, while external F0 estimators are non-differentiable and can pass errors into synthesis.Prior systems use synthetic self-supervision, tracker labels, altered STFT settings, or optimal transport, whose effect on synthesis is unclear.Source: §1 · p. 1
• Unlike GOLF, which uses an external F0 to drive its oscillator, GOLF-inv aims to recover instantaneous phase and frequency through waveform matching.GOLF uses an all-pole filter and glottal model whose phase response follows vocal-tract physics.Source: §1 · p. 1
• GOLF-inv introduces an additive glottal source, phase correction, a state-space lattice filter, and unsupervised joint learning from spectral and harmonic-only waveform losses.Its closed-form pulse has analytic derivatives, and phase correction integrates frequency across frames without error accumulation.Source: §1 · p. 1

2. METHODOLOGY

• The synthesizer forms voice from a glottal source and shaped noise passed through a vocal-tract filter.The neural encoder estimates source and filter parameters from the input audio and trains through the synthesizer.Source: §2.1 · p. 1
• GOLF-inv builds its source additively, predicts frequency and phase per frame, and reads F0 from corrected phase, unlike GOLF's externally driven oscillator.It also replaces GOLF's direct-form filter with a state-space lattice.Source: §2.1 · p. 1
• Waveform MSE supervises the harmonic branch because its phase response is deterministic, whereas the stochastic noise branch has unpredictable waveform.The noise source is white-noise excitation passed through learned spectral shaping.Source: §2.1 · p. 1

2.2. Training criterion

• The full reconstruction combines deterministic harmonic and stochastic outputs, and the total objective sums harmonic-only waveform MSE with λ-weighted full-signal MSS.MSS learns both harmonic and stochastic magnitudes.Source: §2.2 · p. 1
• The stochastic noise branch is excluded from waveform MSE because its random excitation makes its waveform unpredictable; the authors note this works well likely because voiced regions dominate training.The loss routing is illustrated in Fig. 1.Source: §2.2 · p. 1, Figure 1 · p. 2
• Waveform MSE compares the deterministic harmonic output with the target, while MSS compares the spectrum of the full reconstruction.The routing lets MSE constrain phase without applying it to the unpredictable stochastic noise waveform.Source: §2.2 · p. 1, Figure 1 · p. 2

2.3. The glottal source G(z)

• Additive synthesis avoids the aliasing artifacts that can arise when resampling GOLF's non-band-limited LF glottal-pulse wavetable.Oversampling in later GOLF versions mitigates but does not remove those artifacts.Source: §2.3 · p. 2
• The model assumes one canonical pulse shape, with global, time-invariant harmonic amplitudes and phase offsets.The fundamental harmonic is set in phase with the instantaneous phase, and the harmonic amplitudes decrease with frequency so that phase tracks the fundamental rather than a sub-harmonic.Source: §2.3 · p. 2
• The encoder predicts framewise frequency and phase, upsamples frequency, and integrates it from the current frame-center phase to obtain uncorrected phase.The frequency estimate is interpolated with a cubic spline.Source: §2.3 · p. 2
• Smoothly distributing boundary phase mismatch produces the final instantaneous phase, while wrapping phase differences maps them into [−π, π).The phase correction can be significant for low-pitched voices at a 100 Hz frame rate.Source: §2.3 · p. 2
• At a 100 Hz frame rate, the maximum absolute frequency correction is roughly ±79 Hz; randomized integration windows divide that correction by the window length.Training samples window lengths from 1 to 4 frames to encourage reliance on predicted frequency for phase conversion.Source: §2.3 · p. 2

2.4. The noise source Q(z)

• Q(z) shapes white-noise excitation with 30 symmetric real cepstral coefficients, yielding a zero-phase response.The frame-wise frequency response is converted to a time-domain FIR using an FFT size of 512 and a 149-point Hamming window.Source: §2.4 · p. 2

2.5. The vocal tract filter H(z)

• Lattice reflection coefficients derived from positive tube areas stay between −1 and 1, bounding section weights and avoiding direct-form summation nodes’ dynamic-range problem.Each section represents wave scattering at a vocal-tract area discontinuity.Source: §2.5 · p. 2

2.5.1. State-space realisation of the lattice filter

• The state-space lattice enables parallel evaluation; the model uses order N = 14 and learns a zero-path component through C(t), with a time-varying pre-filter gain.N = 14 roughly corresponds to the length of a male vocal tract at 16 kHz.Source: §2.5.1 · p. 2

2.5.2. Parallel implementation

• Composing L updates and recursively applying the same reduction shortens sequential cost from O(T) to O(L logL T); L = 32 was selected empirically.The method computes block coefficients in parallel, then recovers the remaining states in parallel across blocks.Source: §2.5.2 · pp. 2–3

2.6. The analysis encoder

• A four-channel STFT feature passes through convolutional and Bi-LSTM layers before projections produce parameters that are linearly upsampled for time-varying synthesis.The encoder uses 16 kHz audio, a 512-point FFT, and a 100 Hz frame rate; the decoder remains the physical source-filter synthesizer.Source: §2.6 · p. 3

3.1. Datasets and training details

• M4Singer uses within-singer test splits, while LM-SSD tests held-out performances from three singers; training uses 16 kHz audio and a staged introduction of random integration windows.LM-SSD contains roughly two hours of selected studio recordings; its test performances are not unseen songs or singers.Source: §3.1 · p. 3

3.2. Model configurations

• GOLF-inv uses additive glottal-source synthesis, QHM phase correction, a state-space lattice filter, and unsupervised joint learning; comparisons include direct-form, QHM-only, and no-randomisation ablations.GOLF and DDSP+F0 use FCNF0++ F0; QHM uses FCNF0++ for initialisation.Source: §3.2 · p. 3

3.3. Copy-synthesis evaluation

• 8.10 dB SRER for GOLF-inv versus −2.67 dB for DDSP+F0, although GOLF has the lowest spectral distortion using external F0.The QHM-only source raises SRER to 10.68 dB but worsens MCD to 6.21 dB; the authors attribute QHM’s SRER lead mostly to its noise-model treatment of unvoiced parts.Source: §3.3 · p. 3, Table 1 · p. 4

3.4. Fundamental frequency estimation

• GOLF-inv achieves the highest overall accuracy on both LM-SSD splits, outperforms all neural baselines, and exceeds REAPER and SWIPE in raw pitch accuracy.Without random-window training, corrected test RPA is 92.28% versus 86.89% without correction; with the random window, the gap is only 0.50 points.Source: §3.4 · p. 4, Table 2 · p. 4

3.5. Glottal closure instant detection

• GOLF-inv tracks LM-SSD reference closure instants in both registers, trailing REAPER by 0.53 points in IDR and 0.01 ms in timing-error standard deviation.A hit is counted within a one-period window centered on the reference GCI; the standard deviation measures timing-error spread, not constant offset.Source: §3.5 · p. 4, Table 3 · p. 4

3.6. Discussion

• Waveform MSE is limited to the deterministic harmonic path, and the authors identify residual buzzing in unvoiced regions as the main limitation; only singing voice is tested.The discussion suggests voicing-dependent masking or spectral whitening before MSE may reduce the buzzing, and leaves speech for future work.Source: §3.6 · p. 4

• A physically constrained synthesiser and neural encoder recover F0 and glottal closure instants from waveforms on LM-SSD without F0 labels.Source: §4 · p. 4
• Waveform matching, including phase, rather than magnitude-spectrum matching alone, underlies the proposed recovery.Source: §4 · p. 4
• Within the discussed limits, inverse synthesis offers a practical route to label-free voice analysis.Source: §4 · p. 4

• The work is a non-commercial secondary analysis of publicly released M4Singer (CC BY-NC-SA 4.0) and LM-SSD (CC BY 4.0) recordings, requiring no ethical approval.Source: §6 · p. 5