Source-linked AI summary
Learning Filterbanks from Raw Speech for Phone Recognition
Neil Zeghidour, Nicolas Usunier, Iasonas Kokkinos, Thomas Schatz, Gabriel Synnaeve, Emmanuel Dupoux
TL;DR
Fixed perceptual filterbanks provide useful speech-recognition priors, but may not be optimal for every task. This paper learns complex time-domain filters initialized to approximate MFSC and fine-tuned with the recognition network; on TIMIT, these front ends consistently outperform comparable MFSC inputs, with the best model learning all linear front-end operations.
Problem
Perceptually motivated features such as MFSC provide useful priors, but it is unknown whether they are optimal representations for every recognition task.
Source: §1 · p. 1Method
The paper replaces MFSC with a lightweight complex time-domain convolutional front end initialized with Gabor wavelets approximating MFSC and fine-tuned with the network.
Source: §1 · p. 1, §2.2 · p. 2Results
TD-filterbank models consistently outperform equivalent MFSC models across the tested architectures, and learning all linear MFSC operations gives the best model.
Source: §3.3 · p. 3, §5 · p. 4Takeaways & Limitations
With a parameter count comparable to standard MFSC, TD-filterbanks are consistently better in these experiments.
Source: §5 · p. 4
Abstract
from arXiv · showhide
We train a bank of complex filters that operates on the raw waveform and is fed into a convolutional neural network for end-to-end phone recognition. These time-domain filterbanks (TD-filterbanks) are initialized as an approximation of mel-filterbanks, and then fine-tuned jointly with the remaining convolutional architecture. We perform phone recognition experiments on TIMIT and show that for several architectures, models trained on TD-filterbanks consistently outperform their counterparts trained on comparable mel-filterbanks. We get our best performance by learning all front-end steps, from pre-emphasis up to averaging. Finally, we observe that the filters at convergence have an asymmetric impulse response, and that some of them remain almost analytic.
1. INTRODUCTION
The paper proposes a lightweight, learnable time-domain replacement for mel-filterbanks, initialized to approximate their computation. On TIMIT, the same end-to-end phone-recognition architectures perform better with the learnable front end than with MFSC.
1. INTRODUCTION
- Although perceptually motivated features such as gammatones and MFSC provide useful priors, the authors question whether fixed filterbanks are optimal for every recognition task.Source: §1 · p. 1
- Earlier raw-waveform approaches used convolutional networks configured for gammatones or pretrained to reproduce MFSC, whereas this work proposes a lightweight learnable replacement.Source: §1 · p. 1
- MFSC is a relevant target because it serves as the front end of state-of-the-art phone- and speech-recognition systems.Source: §1 · p. 1
- The proposed lightweight architecture uses complex-valued convolution, modulus, and low-pass filtering as a plug-in, learnable MFSC replacement, initialized with Gabor wavelets matched to MFSC center frequencies and bandwidths.Source: §1 · p. 1
- On TIMIT, replacing MFSC with the learnable architecture improves performance across competitive end-to-end phone-recognition models.Source: §1 · p. 1
2. TIME-DOMAIN MFSC
The section defines MFSC computation and develops a time-domain convolutional approximation that can replace MFSC in neural networks. The approximation uses complex wavelets initialized to match MFSC filters and has implementation conditions and design choices that shape the front end.
2.2. Approximating MFSC with convolutions in time
- TD-filterbanks replace the MFSC front end with time-domain convolution layers, whose complex wavelet filters approximate MFSC filters.The layer specification includes raw-waveform filtering and a grouped-convolution windowing stage.Source: §2 · p. 1, §2.2 · p. 2, Table 1 · p. 2
2.1. MFSC computation
- MFSC are computed by applying an STFT, then averaging frequency content with triangular filters whose centers and bandwidths increase linearly on a log scale.The standard setup uses 40 mel-spaced filters, 25 ms Hanning windows, and 10 ms frame steps.Source: §2.1 · p. 2, §2.2 · p. 2
2.2. Approximating MFSC with convolutions in time
- Complex wavelets approximate the triangular MFSC filters; this is a first-order scattering transform, valid when each wavelet's time support is shorter than the Hanning window.Gabor wavelets are set to match the target filter centers and widths.Source: §2.2 · p. 2
- The 40-dimensional Gabor-wavelet approximation produces spectrograms similar to MFSC on a TIMIT sentence after mean-variance normalization.The comparison is shown for a random sentence.Source: §2.2 · p. 2
- The grouped-convolution windowing stage uses a squared Hanning window, zero biases, and 10 ms decimation through stride 160.A log compression follows after adding 1 to the absolute output because learned weights are not constrained to be positive.Source: §2.2 · p. 2
3. EXPERIMENTS
On TIMIT, the experiments compare TD-filterbank and MFSC front ends across three model architectures and four TD-filterbank training settings. TD-filterbanks outperform equivalent MFSC models, while the best reported waveform result uses a learnable pre-emphasis layer.
3.1. Setting
- The TIMIT experiments use the standard train/dev/test split and 39 phonemes, with three convolutional architectures trained end-to-end using Autoseg.The models use either five convolutional layers with ReLU and dropout 0.5 or 0.7, or eight layers with PReLU and dropout 0.7; training runs for 2000 epochs with dev-set selection.Source: §3.1 · p. 3
3.2. Different types of TD-filterbanks
- Learn-filterbank is chosen for later experiments: random initialization performs considerably worse, while learning averaging with the filters gives no clear advantage over learning filters alone.The learned averaging filters remain almost identical to their initialization; fixed layers moderately worsen performance, which the authors hypothesize may relate to missing mean-variance normalization.Source: §3.2 · p. 3
3.3. Results
- TD-filterbanks systematically outperform equivalent MFSC models on TIMIT dev and test PER across the tested architectures.The reported gain from switching front ends adds 0.31% to the parameter count, even for the smallest architecture.Source: §3.3 · pp. 3–4
- Figure 2 shows example filters’ real parts in blue and imaginary parts in red; Figure 3 heat-maps initialization-filter and learned-filter frequency-response magnitudes.Source: Figure 2 · p. 4, Figure 3 · p. 4
- 18% test PER is reached by adding a learnable pre-emphasis layer; the best waveform model also exceeds Wavenet’s PER of 18.8.The Wavenet comparison is reported despite its use of phonetic alignment and an auxiliary prediction loss.Source: §3.3 · p. 4
4. ANALYSIS OF LEARNED FILTERS
The learned filters remain localized in time and frequency, but some become asymmetric and their bandwidths vary. The analysis also finds that some filters depart from analyticity, with an average negative-to-positive frequency energy ratio of 0.26.
4. ANALYSIS OF LEARNED FILTERS
- The analysis examines first-layer filters from the CNN-8L-PReLU-do0.7 + TD-filterbanks model; its visualizations show example filters and frequency-response magnitudes.Source: §4 · p. 4
- Some learned filters become asymmetric, with a sharp attack and slow decay, while retaining a mostly mel-scale frequency layout but showing substantial bandwidth variability.The authors note that this impulse-response asymmetry is also found in human and animal auditory filters estimated from behavioral and physiological data.Source: §4 · p. 4
- The analysis asks whether filters retain the initialization’s analyticity, noting that analytic filters have a sub-band Hilbert-envelope interpretation under squared-modulus convolution.Source: §4 · p. 4
- The learned filters’ average negative-to-positive frequency energy ratio is 0.26; filters with substantial negative-frequency energy are mostly in the 1000–3000 Hz range.Their negative-frequency spectrum appears to be a down-scaled version of the positive-frequency spectrum.Source: §4 · p. 4
5. CONCLUSION
The paper presents a lightweight TD-filterbank front end that approximates MFSC at initialization and can be fine-tuned end-to-end. In the reported experiments it consistently outperforms standard MFSC, and learning all linear operations from pre-emphasis through averaging gives the best model.
5. CONCLUSION
- A TD-filterbank front end with a parameter count comparable to standard MFSC is consistently better in the reported experiments.Learning all linear operations in the MFSC derivation, from pre-emphasis through averaging, provides the best model.Source: §5 · p. 4