PaperlayerView exact revision on arXiv

Source-linked AI summary

DDSP: Differentiable Digital Signal Processing

Jesse Engel, Lamtharn Hantrakul, Chenjie Gu, Adam Roberts

arXiv:2001.04643v1cs.LGcs.SDeess.ASeess.SPstat.ML

TL;DR

Audio generators commonly operate directly on waveforms or Fourier representations, leaving opportunities to exploit signal-processing knowledge underused. The paper introduces DDSP, which integrates differentiable DSP modules with neural networks and demonstrates high-fidelity, controllable synthesis without autoregressive or adversarial losses. Its modular design supports interpretable manipulation of synthesis and room-acoustics components.

  • Problem

    Waveform and Fourier audio models do not exploit important knowledge of sound generation and perception, while vocoder approaches have limited expressivity and are difficult to integrate with automatic differentiation.

  • Method

    DDSP integrates fully differentiable synthesizers and audio effects with neural networks, using interpretable signal-processing components in end-to-end training.

  • Results

    DDSP models generate high-fidelity audio without autoregressive or adversarial losses and support controllable synthesis, pitch extrapolation, dereverberation, room-acoustics transfer, and timbre transfer.

  • Takeaways & Limitations

    Differentiable DSP enables interpretable and modular generative audio models while retaining neural networks’ expressive power and end-to-end learning.

  • Takeaways & Limitations

    Earlier oscillator-model approaches required hand-tuned analysis parameters, and vocoder realism is limited by the expressivity of the chosen analysis/synthesis pair.

Abstract

from arXiv · show

Most generative models of audio directly generate samples in one of two domains: time or frequency. While sufficient to express any signal, these representations are inefficient, as they do not utilize existing knowledge of how sound is generated and perceived. A third approach (vocoders/synthesizers) successfully incorporates strong domain knowledge of signal processing and perception, but has been less actively researched due to limited expressivity and difficulty integrating with modern auto-differentiation-based machine learning methods. In this paper, we introduce the Differentiable Digital Signal Processing (DDSP) library, which enables direct integration of classic signal processing elements with deep learning methods. Focusing on audio synthesis, we achieve high-fidelity generation without the need for large autoregressive models or adversarial losses, demonstrating that DDSP enables utilizing strong inductive biases without losing the expressive power of neural networks. Further, we show that combining interpretable modules permits manipulation of each separate model component, with applications such as independent control of pitch and loudness, realistic extrapolation to pitches not seen during training, blind dereverberation of room acoustics, transfer of extracted room acoustics to new environments, and transformation of timbre between disparate sources. In short, DDSP enables an interpretable and modular approach to generative modeling, without sacrificing the benefits of deep learning. The library is publicly available at https://github.com/magenta/ddsp and we welcome further contributions from the community and domain experts.

1 INTRODUCTION

DDSP expands differentiable modeling beyond waveform and Fourier representations by integrating interpretable signal-processing components with neural networks. The paper uses audio synthesis to show that oscillator-based inductive biases can support high-fidelity, modular generation.

  • Motivation and contribution: DDSP integrates interpretable signal-processing elements into automatic-differentiation software, expanding the toolbox of differentiable functions for audio synthesis.The library is implemented in TensorFlow and is presented as broadly applicable, with audio synthesis as the focus.
  • Challenges of neural audio synthesis: Waveform and STFT-based models can represent arbitrary audio but impose wave-packet and phase-alignment biases that do not directly exploit oscillatory structure.Fourier models also face spectral leakage when basis frequencies do not match the audio sinusoid.
  • Challenges of neural audio synthesis: Autoregressive waveform models avoid wave-packet biases but require larger, more data-hungry networks and can suffer exposure bias during generation.They generate one sample at a time and therefore retain general waveform expressivity.
  • Oscillator models: Oscillator-based vocoders and synthesizers use interpretable parameters such as loudness and frequency, but prior neural approaches required hand-tuned analysis and lacked end-to-end gradients.The expressivity of these models is also limited by the chosen analysis/synthesis pair.
  • Contributions: DDSP implements fully differentiable synthesizers and audio effects, combining oscillator inductive biases with neural-network expressivity and end-to-end training.The paper demonstrates high-fidelity audio without autoregressive or adversarial losses.
  • Contributions: The modular models support independent pitch and loudness control, pitch extrapolation, blind dereverberation, room-acoustics transfer, timbre transfer, and smaller networks.The reported applications include converting singing voice into violin and extrapolating to unseen pitches.

2 RELATED WORK

Related work includes vocoders, synthesizer modeling, and differentiable speech synthesis. DDSP differs by providing differentiable DSP components for end-to-end learning rather than modeling a fixed synthesizer as a black box.

  • Vocoders: Vocoder families include source-filter models and sinusoidal/additive models, with additive synthesis offering greater expressivity at the cost of more time-varying parameters.DDSP builds a differentiable synthesizer from the Harmonic plus Noise model.
  • Synthesizers: Prior synthesizer research estimated commercial-synthesizer parameters with gradient-free methods, whereas DDSP supplies differentiable DSP components for end-to-end learning.The distinction is between modeling existing synthesizers as black boxes and constructing a differentiable toolkit.
  • Neural Source Filter: Neural Source Filter uses a differentiable waveshaping synthesizer, while DDSP explores additive synthesis and emphasizes a common family of differentiable audio-generation techniques.Both approaches generate audio in the time domain and use multiscale spectrogram losses in the frequency domain.

3 DDSP COMPONENTS

DDSP expresses core audio-synthesis and effects operations as differentiable, interpretable DSP components. The section develops harmonic-plus-noise synthesis, smooth parameter control, time-varying filtering, and efficient frequency-domain reverb.

  • DDSP expresses oscillators, envelopes, and LTV-FIR filters as feedforward functions suitable for parallel hardware and sample generation during training.These components provide the core differentiable DSP building blocks used in the experiments.
  • 3.1 SPECTRAL MODELING SYNTHESIS: Spectral Modeling Synthesis combines an additive synthesizer of many sinusoids with a subtractive synthesizer that filters white noise.The paper uses this expressive parametric model as an example DDSP implementation.
  • 3.2 HARMONIC OSCILLATOR / ADDITIVE SYNTHESIZER: A harmonic oscillator parameterizes sinusoidal frequencies as integer multiples of a time-varying fundamental frequency and uses harmonic amplitudes to shape the output.The oscillator phase is obtained by integrating instantaneous frequency, while the initial phase may be randomized, fixed, or learned.
  • 3.2 HARMONIC OSCILLATOR / ADDITIVE SYNTHESIZER: Harmonic amplitudes are factorized into global amplitude controlling loudness and a normalized harmonic distribution determining spectral variation.The harmonic distribution components are constrained to be nonnegative and sum to one.
  • 3.3 ENVELOPES: Amplitude and harmonic-distribution parameters are smoothed with overlapping windows; experiments use a 4ms hop and 8ms frames to reduce artifacts while preserving responsiveness.Bilinear interpolation is adequate for instantaneous-frequency upsampling, whereas amplitudes and harmonic distributions require smoothing.
  • 3.4 FILTER DESIGN: FREQUENCY SAMPLING METHOD: Time-varying FIR filters are applied frame-wise in the Fourier domain, while long room reverb uses frequency-domain convolution with O(n log n) scaling instead of O(n^3) matrix multiplication.The filter implementation predicts frame-specific transfer functions and reconstructs filtered audio through inverse transforms and overlap-add.

4 EXPERIMENTS

The experiments evaluate supervised and unsupervised DDSP autoencoders on NSynth and solo violin data, using factorized representations, differentiable synthesizers, effects, and multi-scale spectral reconstruction loss.

  • Datasets and models: Supervised and unsupervised DDSP autoencoders are evaluated on NSynth and solo violin datasets.The supervised model uses extracted F0 and loudness, while the unsupervised model learns F0 jointly with the network.
  • Architecture: The architecture combines neural components, latent representations, and deterministic synthesizers and effects.Some components vary by experiment: z is omitted for solo violin, while reverb is omitted for NSynth.
  • Encoders: The supervised encoder extracts loudness directly, uses pretrained CREPE for F0, and optionally encodes residual information z(t).The unsupervised encoder replaces CREPE with a jointly trained ResNet that estimates f(t) from a mel-scaled log spectrogram.
  • Architecture: The decoder maps f(t), l(t), and z(t) to control parameters for additive and filtered-noise synthesizers, which generate the reconstructed audio.Training minimizes reconstruction loss between synthesized and original audio.
  • Model size: DDSP models have up to 10 times fewer parameters than comparable neural synthesizers.Small 240k-parameter models are reported as promising for low-latency CPU or embedded-device applications, though less realistic than full models.
  • Reconstruction loss: The multi-scale spectral loss compares original and synthesized spectrogram magnitudes using L1 differences in both linear and logarithmic scales.Experiments use FFT sizes 2048, 1024, 512, 256, 128, and 64, with 75% STFT overlap; α is set to 1.0.

5 RESULTS

DDSP produces high-quality resynthesis and supports controllable manipulation of pitch, loudness, timbre, and room acoustics across musical sources and conditions.

  • High-fidelity synthesis: The supervised DDSP autoencoder outperforms a comparably conditioned WaveRNN baseline on NSynth resynthesis, especially on F0 L1.The additive synthesizer directly uses the conditioning frequency.
  • High-fidelity synthesis: The unsupervised DDSP autoencoder generates sounds with correct frequencies without supervision, although it is less accurate than the supervised DDSP model.It outperforms the supervised WaveRNN model.
  • Independent control: Varying factorized conditioning independently controls loudness and pitch, while interpolating z(t) produces smooth timbre changes.The experiments track spectral centroid as an indicator of the timbre transition.
  • Extrapolation: Shifting f(t) down an octave allows coherent solo-violin resynthesis outside the pitch range observed during training.The shifted output remains coherent and resembles a related instrument.
  • Dereverberation and acoustic transfer: Bypassing the learned reverb module produces dereverberated violin audio, while applying it to singing transfers the violin recording environment.The dereverberation quality is limited by the underlying generative model.
  • Timbre transfer: The model transfers singing-voice F0 and loudness patterns into violin timbre and room acoustics.The singing fundamental frequency is shifted up two octaves to fit the violin’s typical register before resynthesis.

6 CONCLUSION

DDSP combines differentiable DSP components with deep learning to retain strong signal-processing inductive biases and neural networks’ expressive power. Its modular design supports interpretable synthesis, component manipulation, and applications including dereverberation and extrapolation.

  • The DDSP library fuses classical DSP with deep learning while retaining end-to-end differentiability and neural-network expressivity.
  • The violin decomposition predicts structured amplitudes, harmonic distributions, and noise magnitudes from loudness and fundamental-frequency conditioning signals.
  • Its interpretable modules enable independent pitch and loudness control, pitch extrapolation, dereverberation, room-response transfer, and timbre transfer.

B.1 ENCODERS

The encoder–decoder architecture represents audio with fundamental frequency, loudness, and residual latents, then predicts synthesizer parameters from their combined sequence. Separate processing and shared decoding preserve distinct conditioning roles while producing harmonic and filtered-noise controls.

  • B.1 ENCODERS: Three encoders produce fundamental frequency f(t), loudness l(t), and residual vector z(t).
  • B.1 ENCODERS: The f-encoder extracts fundamental frequency using pretrained CREPE for supervised experiments and a jointly learned ResNet on log-mel spectrograms for unsupervised experiments.
  • B.1 ENCODERS: The l-encoder computes loudness through A-weighting, log scaling, and dataset-based centering.
  • B.1 ENCODERS: The z-encoder derives the first 30 MFCCs and processes them with normalization and a 512-unit GRU.
  • B.1 ENCODERS: The decoder receives the latent tuple (f(t), l(t), z(t)) over 250 timesteps and outputs harmonic amplitudes a(t) and filtered-noise FIR parameters H.
  • B.1 ENCODERS: A shared-bottom decoder computes a shared embedding and uses separate heads for the synthesizer outputs.

B.3 TRAINING

Training exploits DDSP’s differentiability while constraining synthesizer outputs to physically meaningful nonnegative parameters. The evaluated models use substantially fewer parameters than comparable GANSynth models, although the architectures were not optimized for size.

  • B.3 TRAINING: Because all DDSP components are differentiable, the model supports end-to-end training with SGD optimizers such as ADAM.
  • B.3 TRAINING: An additional perceptual loss from pretrained pitch and audio models guides f(t) prediction on NSynth.
  • B.3 TRAINING: The harmonic synthesizer uses 101 harmonics, with amplitude and distribution parameters upsampled using overlapping Hamming-window envelopes.
  • B.3 TRAINING: A modified sigmoid enforces nonnegative amplitudes, harmonic distributions, and filtered-noise magnitudes while slightly improving training stability.
  • B.3 TRAINING: The filtered-noise synthesizer uses 65 network output channels as FIR-filter magnitude inputs.
  • B.3 TRAINING: DDSP models require 2 to 3 times fewer parameters than GANSynth on the NSynth comparisons, despite not being optimized for size.

C.1 METRICS

The evaluation uses loudness and fundamental-frequency distances plus pitch outlier analysis to assess reconstruction fidelity and pitch tracking reliability. Each metric applies explicit processing or confidence criteria.

  • C.1 METRICS: Loudness L1 distance compares loudness extracted from synthesized audio with the input conditioning vector, with lower values indicating closer matches.
  • C.1 METRICS: F0 L1 distance is reported in MIDI space, where an average value of 1.0 corresponds to a semitone difference.
  • C.1 METRICS: F0 metrics use a confidence threshold of 0.85 to select regions with detectable pitch content and account for CREPE tracking imperfections.

C.2 INTERPOLATION METRICS

Table C.2 reports loudness and F0 L1 metrics for reconstruction and interpolation tasks, while the model uses harmonic synthesizers within a deliberately focused problem domain.

  • In reconstruction, the model receives standard f(t)A, l(t)A, and z(t)A inputs, with both metrics computed against ground-truth inputs.
  • Loudness interpolation supplies f(t)A, l(t)B, and z(t)A, evaluating Loudness (L1) against l(t)B.
  • F0 interpolation supplies f(t)B, l(t)A, and z(t)A, with F0 (L1) evaluated using f(t)B.
  • Table C.2 reports Loudness (L1) and F0 (L1) metrics for different interpolation tasks.The table compares reconstruction, loudness interpolation, and F0 interpolation settings.
  • The experiments restrict the synthesizer to harmonic oscillators to focus the problem domain, although inharmonicity can also be incorporated.
  • The modular synthesizer structure permits additional output- and parameter-based losses, such as SNR penalties for noisy outputs, though these engineered losses were not extensively tested.
Loading 2001.04643v1…