Source-linked AI summary
LEAF: A Learnable Frontend for Audio Classification
Neil Zeghidour, Olivier Teboul, Félix de Chaumont Quitry, Marco Tagliasacchi
TL;DR
Mel-filterbanks are successful but fixed and potentially task-mismatched, motivating a general-purpose learnable alternative. The paper introduces LEAF, a lightweight frontend that learns filtering, pooling, compression, and normalization, and evaluates it across diverse audio tasks. LEAF consistently outperforms mel-filterbanks and prior learnable alternatives, including on AudioSet with far fewer frontend parameters.
Problem
Mel-filterbanks rely on fixed, hand-engineered operations whose perceptual biases and compression choices may not suit all audio-classification tasks.
Method
LEAF is a fully learnable frontend that replaces fixed filtering, pooling, compression, and normalization with trainable operations and can serve as a drop-in replacement.
Results
Across eight diverse audio-classification tasks, LEAF consistently outperforms mel-filterbanks and previous learnable alternatives, including on AudioSet with far fewer parameters.
Takeaways & Limitations
A single learned frontend can provide general-purpose audio classification across speech, music, audio events, and animal sounds.
Abstract
from arXiv · showhide
Mel-filterbanks are fixed, engineered audio features which emulate human perception and have been used through the history of audio understanding up to today. However, their undeniable qualities are counterbalanced by the fundamental limitations of handmade representations. In this work we show that we can train a single learnable frontend that outperforms mel-filterbanks on a wide range of audio signals, including speech, music, audio events and animal sounds, providing a general-purpose learned frontend for audio classification. To do so, we introduce a new principled, lightweight, fully learnable architecture that can be used as a drop-in replacement of mel-filterbanks. Our system learns all operations of audio features extraction, from filtering to pooling, compression and normalization, and can be integrated into any neural network at a negligible parameter cost. We perform multi-task training on eight diverse audio classification tasks, and show consistent improvements of our model over mel-filterbanks and previous learnable alternatives. Moreover, our system outperforms the current state-of-the-art learnable frontend on Audioset, with orders of magnitude fewer parameters.
1 INTRODUCTION
Mel-filterbanks have been historically successful in audio understanding, but their fixed, hand-engineered design embeds potentially flawed and task-mismatched biases. LEAF replaces these fixed operations with a lightweight, fully learnable frontend evaluated across diverse audio classification tasks.
- Mel-filterbanks compute spectrograms, apply logarithmically spaced triangular filters, and compress coefficients logarithmically to model human pitch and loudness perception.This human-auditory inspiration has historically supported audio understanding across speech, music, animal vocalizations, and audio events.
- Mel-filterbank biases include revised mel-scale designs, irreproducible original auditory experiments, and fixed compression choices despite proposed alternatives.
- Human-perception biases may help speech and music tasks but harm tasks requiring fine-grained resolution at high frequencies.
- LEAF decomposes the frontend into filtering, pooling, and compression/normalization, making all operations learnable with only a few hundred parameters.
- LEAF is evaluated across speech, music, audio-event, and animal-sound tasks, where one learned parameter set outperforms mel-filterbanks and prior learnable alternatives.The study uses eight datasets and also reports replication when training separate models for individual tasks.
2 RELATED WORK
Prior learnable audio frontends range from convolutional and structured filters to PCEN-based compression, but earlier approaches often used complex architectures or fixed mel-filterbanks. LEAF combines learnable filtering, pooling, compression, and normalization in a unified frontend.
- Earlier waveform frontends used multiple convolutional layers, making comparisons with mel-filterbanks difficult.
- Learnable filter approaches include mel-initialized filters, Gammatone filters, time-domain filterbanks, SincNet, Gabor filters, and spline filters.
- LEAF uses complex-valued Gabor filtering because Gabor filters are localized in both time and frequency without requiring a window function.
- PCEN previously learned compression over fixed mel-filterbanks, whereas LEAF jointly learns filters, pooling, compression, and normalization.
3 MODEL
LEAF decomposes audio feature extraction into learnable filtering, pooling, and compression/normalization, enabling end-to-end frontend optimization. Its Gabor-based design provides controlled frequency filters, lightweight pooling, and per-channel adaptive normalization.
- 3 MODEL: LEAF maps a waveform to a time-frequency representation through filtering, temporal downsampling, and dynamic-range compression or normalization.The frontend preserves the original temporal resolution during filtering before pooling reduces it.
- 3.1 FILTERING: The filtering block convolves the waveform with complex-valued filters and applies a squared modulus, producing real-valued channel responses at the input temporal resolution.The complex operation can be implemented with paired real-valued filters and squared ℓ2-pooling across adjacent channels.
- 3.1.2 GABOR 1D-CONVOLUTION: Gabor filters parameterize each channel by center frequency and bandwidth, yielding smooth, quasi-analytic bandpass filters with controllable frequency responses.Their bandwidths are constrained so the frequency-response full-width at half-maximum lies between 1/W and 1/2.
- 3.1.2 GABOR 1D-CONVOLUTION: Gabor filtering uses 2N parameters for N filters, compared with W·N for standard one-dimensional convolutions.At 16 kHz with a 25 ms window, W = 401, making Gabor filtering 200 times more parameter-efficient.
- 3 MODEL: Unlike ordered spectrogram features, learned filterbanks do not inherently constrain feature-axis ordering, which can affect subsequent frequency-dependent operations.Temporal ordering remains preserved because filtering is performed by convolution.
- 3.2 LEARNABLE LOWPASS POOLING: LEAF uses learnable Gaussian lowpass pooling, implemented as a zero-center-frequency Gabor special case with one learnable bandwidth per channel.Bandwidth initialization at 0.4 produces a response close to the Hann window used by mel-filterbanks.
- 3.3 LEARNING PER-CHANNEL COMPRESSION AND NORMALIZATION: The compression stage learns per-channel smoothing, normalization, offsets, and exponents through sPCEN instead of applying one fixed logarithmic transform.The final frontend cascades Gabor convolution, Gaussian lowpass pooling, and sPCEN.
4 EXPERIMENTS
LEAF is evaluated against mel-filterbanks and other learnable frontends across single-task, multi-task, and AudioSet multi-label classification. Across diverse datasets, LEAF generally matches or outperforms alternatives, with stronger compression results and competitive AudioSet performance using far fewer frontend parameters.
- Experimental setup: The experiments compare LEAF with log-compressed mel-filterbanks, learnable time-domain filterbanks, and SincNet using shared backbone architectures.The evaluation covers single-task classification, multi-task classification, and multi-label classification on AudioSet.
- Single-task classification: Eight single-task problems span acoustic scenes, birdsong, emotion, speaker, musical instrument and pitch, keywords, and language identification.The datasets include TUT, Birdsong, Crema-D, VoxCeleb, NSynth, Speech Commands, and VoxForge.
- Single-task classification: LEAF outperforms all alternatives on average and matches or exceeds them on individual tasks except for SincNet on VoxCeleb.Bootstrap tests give pMel < 10^-5, pTD-fbanks < 10^-5, and pSincNet = 0.059 for the dataset-resampled accuracy differences.
- Compression analysis: PCEN significantly improves over log compression on mel- and learned filterbanks, while sPCEN gives LEAF its best performance.For each compression choice, the learnable frontend matches or outperforms the corresponding mel-filterbank.
- Multi-task classification: In multi-task classification, LEAF achieves the best overall performance and matches or outperforms every other method on every task.The shared frontend and encoder use task-specific linear heads; bootstrap p-values are pMel = 0.048, pTD-fbanks < 10^-5, and pSincNet = 10^-5.
5 CONCLUSION
LEAF is introduced as a fully learnable alternative to handcrafted mel-filterbanks for audio classification. Across diverse tasks, it serves as a drop-in replacement and can learn general-purpose frontend parameters, while remaining subject to fixed convolutional design choices.
- LEAF is a fully learnable frontend designed to replace handcrafted mel-filterbanks in audio classification.
- Across a large range of tasks, LEAF operates as a drop-in replacement without task-specific adjustment.
- LEAF can learn a single parameter set for general-purpose audio classification while outperforming previously proposed learnable frontends.
- The current model retains fixed filter length and stride within its convolutional architecture, limiting adaptation across sampling rates and frequency content.
- Future work proposes learning these convolutional parameters directly and applying the filter, pool, and compress principle to seismic or physiological signals.
A APPENDIX
The appendix documents the experimental datasets and supplementary analyses, including frontend accuracy differences, learned filters, learned compression parameters, and noise robustness.
- Table A.1 lists the datasets used in the experiments and specifies that default train/test splits are adopted.
- Figures A.1 and A.2 show distributions of LEAF’s accuracy differences from other frontends in single-task and multi-task settings.
- Figure A.3 compares LEAF’s learned filters with the mel scale.
- Figure A.4 presents LEAF’s learned Gaussian lowpass filters on AudioSet against their shared initialization.
- Figure A.5 presents learned sPCEN parameter values on AudioSet against their shared initialization.
- Figure A.6 reports Speech Commands test accuracy across varying signal-to-noise ratios.