PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Scattering Spectrum

Joakim Andén, Stéphane Mallat

arXiv:1304.6763v2cs.SDcs.IT

TL;DR

The paper addresses the need for audio representations that capture multiple time scales while remaining stable to time-warping and selectively invariant to frequency transposition. It constructs scattering representations through cascaded wavelet convolutions and modulus operators, extending MFCC-like descriptors with modulation information. Time-frequency scattering achieves state-of-the-art classification results for musical genre and phone-segment tasks on GTZAN and TIMIT.

  • Problem

    Audio classification needs representations spanning multiple time scales, while time-warping and frequency transposition create variability that should be handled according to the task.

  • Method

    The paper builds locally translation-invariant and time-warping-stable scattering representations using cascaded wavelet transforms, modulus operators, and log-frequency scattering.

  • Results

    Time-frequency scattering yields state-of-the-art classification results for musical genre on GTZAN and phone-segment classification on TIMIT.

  • Takeaways & Limitations

    Second-order coefficients characterize amplitude modulations and transients, while log-frequency scattering provides frequency-transposition invariance for suitable classification tasks.

Abstract

from arXiv · show

A scattering transform defines a locally translation invariant representation which is stable to time-warping deformations. It extends MFCC representations by computing modulation spectrum coefficients of multiple orders, through cascades of wavelet convolutions and modulus operators. Second-order scattering coefficients characterize transient phenomena such as attacks and amplitude modulation. A frequency transposition invariant representation is obtained by applying a scattering transform along log-frequency. State-the-of-art classification results are obtained for musical genre and phone classification on GTZAN and TIMIT databases, respectively.

I. INTRODUCTION

Audio classification requires representations that capture information across multiple time scales while remaining stable to time shifts and time-warping. The paper introduces scattering representations to recover information lost by mel averaging and adds log-frequency scattering for transposition invariance.

  • Motivation: Audio signals contain pitch and timbre at milliseconds, rhythm at seconds, and musical progression over minutes or hours.MFCCs are efficient up to 25 ms, while many applications require structures up to 500 ms.
  • Motivation: Spectrogram coefficients at high frequencies are unstable to time-warping, whereas mel-frequency averaging improves stability but loses information.The mel-scale bandwidth grows with frequency, allowing warped high-frequency partials to overlap after averaging.
  • Approach: Scattering computes cascaded wavelet convolutions and modulus nonlinearities to produce locally translation-invariant, time-warping-stable coefficients over large time scales.Its computational structure resembles a convolutional network, but the filters are predefined and no learning is used.
  • Frequency invariance: A second scattering transform along log-frequency yields representations invariant to frequency transposition while remaining stable to frequency deformation.The averaging must be adapted because transposition invariance helps some tasks but removes information useful for speaker identification.
  • Deformation instability: Small dilations shift high-frequency harmonics by amounts that can exceed their bandwidth, so Fourier modulus representations fail deformation stability.The Euclidean distance need not decrease proportionally to the dilation size when high-frequency harmonic amplitudes are sufficiently large.

B. Mel-frequency Deformation Stability and Filter Banks

Mel-frequency averaging stabilizes spectrograms against time-warping by matching frequency bandwidth to high-frequency displacements, but its time averaging removes fine temporal structure. Wavelet filter banks and modulus envelopes provide the basis for recovering that information.

  • Mel-frequency stability: Mel-frequency spectrograms average spectrogram energy with mel-scale filters centered at frequency λ.Their bandwidth is approximately proportional to frequency, making the averaging compatible with frequency displacements caused by dilation.
  • Mel-frequency stability: Mel-scale averaging satisfies deformation stability, with relative error of order ϵQ and a bound whose constant is proportional to Q.The frequency partials of a signal and its dilated version overlap after mel-frequency averaging.
  • Information loss: Time averaging over T = 190 ms removes fine-scale information such as vibratos and attacks.Using short windows of about 25 ms reduces this loss but prevents capture of large-scale structures needed for classification.
  • Modulation structure: Modulation spectra can capture amplitude-envelope structure below T, but these spectra are themselves unstable to time-warping deformations.A time-warping of the input induces a time-warping of the wavelet modulus envelope.
  • Wavelet filter banks: The analytic wavelet transform uses constant-Q band-pass wavelets and a low-pass filter, with modulus coefficients retaining multiscale envelope variation.For suitable analytic wavelets, the modulus wavelet transform is invertible with a continuous inverse.

B. Deep Scattering Network

The deep scattering network replaces squared wavelet energies with modulus envelopes and recursively applies wavelet modulus transforms before low-pass averaging. This recovers high-frequency modulation information while retaining local time-shift invariance.

  • Network construction: Scattering replaces squared wavelet coefficients with modulus envelopes to avoid amplifying outliers, then recovers removed high frequencies with further wavelet modulus coefficients.Cascading this procedure defines the scattering transform.
  • Network construction: The zeroth-order coefficient is a low-pass time average, while first- and second-order coefficients are obtained by successive wavelet modulus transforms and averaging.The averaging filter of size T provides local invariance to time shifts at each order.
  • Network construction: Iterated modulus convolutions are represented as U_mx(t, λ_1, ..., λ_m) = | ||x ⋆ψ_λ1| ⋆...| ⋆ψ_λm(t)|.Scattering coefficients are the corresponding averaged outputs at each order.
  • Interpretation: The cascade resembles a convolutional network, but every layer produces averaged outputs, all filters are predefined wavelets, and no training-data learning occurs.The representation therefore relies on prior knowledge of the invariances to compute.
  • Parameter choices: Second-order wavelets with Q2 = 1 are adapted to transients and attacks, while musical modulation structures may require Q2 > 1.Orders m ≥ 3 use Qm = 1 and can often be neglected.

IV. SCATTERING PROPERTIES

The paper reviews scattering-transform stability to time-warping, energy conservation, and fast computation as core properties of the representation.

  • Scattering properties: The reviewed scattering properties include stability to time-warping deformation, energy conservation, and a fast computational algorithm.

A. Time-Warping Stability

Scattering representations are designed to remain stable under small time-warping deformations while preserving signal energy through cascaded wavelet modulus operators. Their energy shifts toward lower frequencies across orders, with higher orders becoming more relevant as the averaging scale increases.

  • Time-warping stability: Small time-warping deformations produce bounded representation changes when displacement is small relative to the averaging-filter support.The deformation model assumes |τ′(t)| < 1 and sup_t |τ(t)| ≪ T.
  • Time-warping stability: For Morlet wavelets, numerical experiments give the deformation-stability constant C0 ≈ 2, with overall dependence on the maximum wavelet quality factor across layers.The bound is inversely proportional to wavelet octave bandwidth.
  • Contraction and energy conservation: A scattering representation is contractive and can preserve signal energy because it cascades contractive wavelet modulus operators.The squared Euclidean norm aggregates coefficient energy across scattering orders.
  • Energy propagation: Iterated modulus operators progressively move energy toward lower frequencies, and under appropriate assumptions higher-order residual energy tends to zero.For infinite order, the scattering norm approaches the signal norm.
  • Energy across orders: As the averaging scale T increases, a progressively larger proportion of energy is absorbed by higher-order scattering coefficients.At small scales, first-order coefficients absorb most energy; higher orders become more important for larger T.
  • Representation size: The number of non-negligible mth-order coefficients is about Q1...Qm(log2 N)^m/m!, while subsampling uses half-overlapping windows to reduce computation.For second order, the count is about Q1Q2(log2 N)^2/2.

V. INVERSE SCATTERING

Inverse scattering reconstructs an approximate signal by deconvolving the highest-order averaged coefficients and successively inverting wavelet modulus operators. Second-order coefficients improve reconstruction of amplitude modulations, vibratos, and attacks, but reconstruction quality deteriorates at larger averaging scales.

  • Inversion procedure: Approximate inversion first estimates the highest-order modulus output by deconvolving its averaged scattering coefficients, then inverts each wavelet modulus operator.The deconvolution uses Richardson–Lucy, followed by approximate Griffin & Lim inversions.
  • Inversion limitations: The main reconstruction error arises because subsampling prevents exact recovery of the highest-order modulus output from averaged coefficients.Deconvolution instability is controlled by stopping Richardson–Lucy iterations after 30 steps.
  • Inversion procedure: The wavelet-modulus inversion is a non-convex optimization problem, so the paper uses an approximate fast algorithm motivated by sparse wavelet representations.Convex relaxation methods are described as too computationally and memory intensive for audio applications.
  • Reconstruction quality: First-order reconstructions become crude when T exceeds 100 ms because deconvolution loses too much information.This is illustrated with speech and music scalograms.
  • Reconstruction quality: Adding second-order coefficients restores amplitude modulations, vibratos, and attacks with greater precision and yields much better audio quality than first-order reconstruction.Even second-order reconstructions become crude for T ≥ 500 ms.
  • Information recovered: Normalized second-order coefficients provide high-resolution spectral information through interferences and characterize amplitude modulations.These properties motivate their use alongside first-order coefficients.

A. Normalized Scattering Transform

Normalization reduces redundancy and increases invariance in scattering representations. Normalized second-order coefficients measure frequency intervals through interference and remain nearly invariant to suitable filtering.

  • Amplitude invariance: First-order normalization makes coefficients insensitive to multiplicative constants, while the low-pass averaging scale controls which amplitude changes remain represented.A wider low-pass filter can retain local amplitude information while creating invariance to changes over larger intervals.
  • Normalized representation: Scattering coefficients are renormalized by coefficients from the previous order, with the representation usually limited to order l = 2.For second order, normalization divides S2x by S1x plus a silence threshold ϵ.
  • Filtering invariance: Normalized second-order coefficients are invariant to filtering by h(t) when the filter response is approximately constant over the first wavelet’s support.The paper states that this remains valid at every order m ≥ 2.
  • Frequency interval measurement: Second-order scattering coefficients accurately measure frequency intervals even though wavelets have poorer high-frequency resolution than windowed Fourier transforms.They detect intervals through interference within a first-wavelet frequency band.
  • Frequency interval measurement: For two components, normalized second-order coefficients are non-negligible when λ2 is of the order of |ξ2 − ξ1| and the component amplitudes have comparable magnitude.The coefficient magnitude depends on the second wavelet response at the frequency difference.
  • Multiple intervals: With multiple frequency components, second-order coefficients can measure multiple intervals, while unresolved interval interference can produce high-amplitude third-order coefficients.This extends interval detection beyond a single pair of components.

C. Amplitude Modulation Spectrum

Normalized second-order scattering coefficients characterize audio amplitude modulations while separating them from carrier or excitation structure. They capture chord interference, attack sharpness, and tremolo frequency, including in unvoiced sounds.

  • Amplitude modulation characterization: Second-order coefficients mainly depend on the amplitude modulation a(t), not the filter h and pitch ξ, when first-order scattering is non-negligible.This relation is stated for voiced and unvoiced sounds under the paper’s modeling conditions.
  • Normalization: Averaging and normalization remove the effect of different modulation amplitudes from first-order coefficients of the voiced examples.This leaves their first-order representation dominated by harmonic structure and the spectral envelope.
  • Attack transients: Smooth attacks produce large coefficients at low modulation frequencies, while sharper attacks produce larger coefficients at higher frequencies.The comparison uses the fourth partial λ1 = 4ξ in Figure 7(c).
  • Tremolo: For a tremolo a(t) = 1 + ϵ cos(ηt), second-order coefficients become large when λ2 = η.The tremolo frequency is therefore represented directly along the second-order modulation-frequency index.
  • Voiced and unvoiced sounds: Unvoiced sounds with matched filters and envelopes have second-order coefficients similar to voiced counterparts, with stochastic errors appearing as random low-amplitude fluctuations.Their first-order coefficients lack harmonic structure, while second-order coefficients retain the matched amplitude-modulation patterns.

VII. FREQUENCY TRANSPOSITION INVARIANCE

Frequency transposition can alter pitch and deform spectral envelopes, motivating representations invariant to log-frequency shifts while remaining stable to frequency deformations. The paper replaces MFCC-style averaging with frequency scattering and adapts the resulting invariance during supervised classification.

  • Motivation: Frequency transposition shifts and deforms spectral envelopes on a logarithmic frequency scale, creating a need for invariant and deformation-stable representations.The required invariance depends on the classification task because frequency location may itself carry useful information.
  • MFCC limitation: MFCC high-quefrency truncation increases transposition invariance through averaging along log-frequency, but can discard important information.The paper identifies this information loss as the motivation for replacing the DCT with wavelet-based frequency scattering.
  • Frequency scattering: Frequency scattering recovers information by iteratively applying wavelet transforms and modulus operators along the log-frequency variable γ.The frequency transform is formally analogous to time scattering, with convolutions performed along γ instead of time.
  • Representation: Cascading temporal and log-frequency scattering yields coefficients locally invariant to time and log-frequency shifts and stable to time and frequency deformations.The time-invariance interval is controlled by φ, while frequency-transposition invariance depends on φfr.
  • Adaptive invariance: The final frequency averaging is adaptively selected by a supervised classifier because the required transposition invariance is unknown for a given task.The implementation uses an SVM over the time-and-frequency scattering features.
  • Adaptive resolution: Temporal wavelet resolution can also be adapted by concatenating scattering features across Q1 values, balancing harmonic frequency resolution against transient localization.Q1 = 8 separates harmonic structures, whereas Q1 = 1 provides smaller temporal support; intermediate values add marginal improvements.

B. Musical Genre Classification

Scattering features improve musical genre classification on GTZAN by recovering longer-scale and frequency-transposition structure. Second-order time coefficients provide the main gain, while adaptive frequency resolution further improves the final error.

  • Evaluation: GTZAN contains 1000 thirty-second clips across 10 genres, with results reported as average error and standard deviation from ten-fold cross-validation.The dataset has 100 clips per genre.
  • Time scattering: At T = 740 ms, adding second-order scattering coefficients reduces classification error from 19.1% to 10.7%.Third-order coefficients reduce error only marginally to 10.6% while increasing computational and memory cost.
  • Frequency invariance: Log-frequency scattering reduces musical genre error by about 15%, while adding second-order frequency coefficients improves results only marginally.The frequency cascade supplies transposition invariance for a task that is partly invariant to frequency transposition.
  • Adaptive resolution: Computing coefficients with both Q1 = 1 and Q1 = 8 further reduces error by almost 10%.The paper relates this adaptivity to music containing both sharp transients and narrow-bandwidth frequency components.
  • Final result: The resulting error rate is 8.6%, compared with 9.4% for the non-scattering state-of-the-art.This result uses time and frequency scattering feature vectors with a Gaussian kernel SVM.
  • Caveat: GTZAN has significant statistical issues, limiting its suitability for evaluating further algorithmic refinements.More sophisticated classifiers can also reduce the second-order scattering error from 10.7% to 8.8%.

APPENDIX

The appendix estimates stochastic terms underlying scattering-coefficient analysis using assumptions about signal sparsity, stationarity, excitation, and filter supports. It explicitly presents the main magnitude arguments as non-rigorous and refers to a detailed argument elsewhere.

  • Assumptions: The derivation assumes locally constant amplitudes and frequency responses, non-sparse signals, and filter supports smaller than relevant signal periods or averaging windows.These conditions appear in the harmonic-excitation and local-constancy arguments.
  • Noise model: For Gaussian white-noise excitation, the appendix uses Gaussian and Rayleigh distribution properties to bound terms involving wavelet-filtered responses.It applies the lemma to stationary processes generated by filtered white noise.
  • Scope: The main arguments compute stochastic-term orders of magnitude but are explicitly not a rigorous proof.A detailed argument is cited separately.
  • Stochastic analysis: The appendix analyzes second-order scattering coefficients S2x(t, λ1, λ2) through stochastic-process bounds and a supporting lemma.The lemma concerns a zero-mean stationary process with power spectrum bRz(ω).
Loading 1304.6763v2…