Source-linked AI summary
Parallel Time-Band Mixing with Learned Observation-Adding for Robust ASR Front-Ends
Xingyu Shen, Runze Wang, Wei-Ping Zhu, Benoit Champagne
TL;DR
Robust ASR enhancement must avoid recognition-sensitive artifacts while reducing the sequential dependencies of recurrent front-ends. The paper proposes a sequence-parallel band-split front-end using PTBM and LOA, and reports consistent WER reductions on DNS Challenge and CHiME-4 with frozen Whisper back-ends. The front-end uses 0.96 M parameters and 0.58 GMAC/s.
Problem
Robust ASR front-ends must suppress interference without introducing recognition-degrading artifacts, while remaining computationally lightweight; recurrent band-split modules also limit parallel efficiency.
Method
The paper combines parallel intra-band temporal convolution and per-frame cross-band attention in PTBM, retaining mask-plus-residual reconstruction and adding LOA for adaptive artifact-aware blending.
Results
The proposed front-end consistently reduces WER relative to recurrent band-split baselines on DNS Challenge and CHiME-4 with frozen Whisper back-ends.
Takeaways & Limitations
Parallel time-band mixing is reported as a practical design solution for efficient speech-enhancement front-ends in robust ASR pipelines.
Takeaways & Limitations
LOA predicts one blending weight per test recording from statistics over all STFT frames, so the formulation is utterance-level rather than fully streaming.
Abstract
from arXiv · showhide
Speech enhancement is often used as a front-end for robust ASR, yet recurrent temporal and cross-band modules introduce sequential dependencies that reduce parallel efficiency. In this paper, we present a sequence-parallel band-split enhancement front-end built on a Parallel Time-Band Mixer (PTBM) block that eliminates within-block recurrent unrolling. PTBM integrates intra-band temporal mixing and per-frame cross-band attention within a unified parallel architecture, enabling efficient contextual modeling across both time and frequency dimensions. The system retains the mask-plus-residual reconstruction interface and introduces learned Observation-Adding (LOA) to suppress ASR-sensitive artifacts without development-set tuning. Experiments on DNS Challenge and CHiME-4 with frozen Whisper back-ends show that the proposed front-end consistently reduces word error rate relative to recurrent band-split baselines while requiring only 0.96 M parameters and 0.58 GMAC/s for the front-end network.
1. Introduction
Robust ASR front-ends must balance artifact suppression with lightweight inference, while recurrent band-split modeling limits parallel efficiency. The proposed system addresses these constraints with PTBM and LOA.
- Enhancement artifacts can degrade recognition even when speech quality improves, and front-ends must remain computationally lightweight for deployment.
- BSRNN-style front-ends improve reconstruction but retain recurrent temporal and cross-band computation, introducing sequential dependencies that reduce parallel efficiency.
- PTBM combines intra-band temporal convolutional mixing with per-frame cross-band self-attention in a unified parallel architecture.
- LOA adaptively predicts Observation-Adding coefficients, eliminating development-set tuning for artifact-aware signal blending.
- The front-end consistently reduces WER against recurrent band-split baselines on DNS Challenge and CHiME-4 with frozen Whisper back-ends.It requires 0.96 M parameters and 0.58 GMAC/s.
2. Proposed Method
The proposed front-end transforms a noisy waveform into a band-split representation, processes it through stacked PTBM blocks, and reconstructs an enhanced waveform for recognition.
- The system computes a complex STFT, partitions frequency bins into non-uniform sub-bands, and embeds each band into a shared feature representation.
- A stack of L PTBM blocks processes the band-split representation to model temporal and frequency interactions without recurrent unrolling.
2.2. Band Split and Reconstruction
The method partitions the STFT into non-overlapping bands, embeds each band separately, and reconstructs the enhanced spectrum through mask and residual predictions.
- The frequency axis is divided into K non-overlapping sub-bands whose sizes sum to F, following the predefined V4 configuration.
- Each band’s real and imaginary STFT components are concatenated and mapped by a band-specific linear layer into a C-channel embedding.
- The enhanced spectrum is reconstructed with the mask-plus-residual interface used in BSRNN-style front-ends.
- After PTBM processing, band-specific fully connected layers predict complex masks and residuals, which are concatenated into M and R matching the shape of X.
2.3. Parallel Time–Band Mixer Block (PTBM)
PTBM processes temporal and cross-band context in parallel, then fuses both cues and preserves a residual path to produce the next block representation.
- PTBM structure: Each PTBM block applies a Temporal ConvMixer and a Cross-Band Attention Mixer in parallel to the same input representation.
- Residual connection: A block-level residual connection adds the fused representation to the input and produces the next PTBM output.
- Temporal ConvMixer (TCM): The TCM branch uses gated dilated depthwise temporal convolutions independently within each sub-band to capture low-cost temporal context.
- Cross-Band Attention Mixer (CBA): The CBA branch applies multi-head self-attention across sub-band tokens independently at each frame, allowing parallel processing over time.
- Interaction: An interaction module combines temporal and cross-band outputs through element-wise gating and fusion.
2.4. Learned Observation-Adding (LOA)
LOA predicts an adaptive blending weight from energy and spectral mismatch statistics between the observed and enhanced signals, then blends them for ASR input. It is trained against oracle weights but operates utterance-level at inference.
- LOA uses a two-layer MLP with ReLU and sigmoid output to predict a blending weight from observed and enhanced waveform statistics.The input statistics are computed on training crops and full-length test recordings during inference.
- The MLP input combines an energy ratio with the mean and variance of the log-magnitude spectral difference.These statistics capture energy and spectral mismatch between the observed and enhanced signals.
- LOA is trained separately after freezing the SE network, regressing to oracle weights found by a 0.01-resolution grid search under the signal-level loss.This keeps the adaptive blending lightweight and recognizer-independent.
- At inference, LOA predicts one blending weight per test recording, making the formulation utterance-level rather than fully streaming.Statistics are computed over all STFT frames of the recording.
2.5. Training Objective
The enhancement network is trained with a fixed weighted combination of multi-resolution STFT magnitude loss and SI-SNR loss between enhanced and clean waveforms.
- The objective combines multi-resolution STFT magnitude loss and SI-SNR loss between the enhanced waveform ˆs and clean reference s.The weighted sum follows common speech enhancement and separation practice.
- λ is a fixed weight, and the objective remains compatible with the mask-plus-residual reconstruction interface.The SI-SNR term uses the negative SI-SNR objective.
3. Experimental Setup
The study evaluates separately trained SE front-ends on DNS Challenge and CHiME-4 using frozen Whisper back-ends, standardized decoding, and WER alongside front-end complexity.
- Datasets: The experiments use 16 kHz English DNS Challenge and CHiME-4 benchmarks, with a separately trained front-end for each benchmark.Results are reported on the corresponding official test sets.
- Datasets: CHiME-4 evaluation uses official real-recorded dt05 real and et05 real 1ch sets, with training mixtures synthesized from tr05 simu 1ch and DNS noise.Training SNRs are uniformly sampled from −5 to 20 dB.
- Datasets: DNS Challenge training follows the ESPnet-SE recipe with a 100-hour single-channel noisy-speech set and evaluation under reverberant and non-reverberant conditions.Model selection uses the official DNS validation set.
- Configuration: The default front-end uses V4 band partitioning with K=23, L=12 PTBM blocks, C=128, and a two-layer LOA MLP.The configuration also uses four TCM dilations and four CBA heads.
- Training: Each benchmark front-end is trained for 54k steps, selected by validation loss, then LOA is trained for 10k steps after the SE network is frozen.Training uses Adam and 4 s random crops with batch size 8.
- Decoding and metrics: Whisper Tiny, Medium, and Large serve as frozen ASR back-ends, with no external language model and shared decoding and normalization procedures.The primary ASR metric is WER, while Params and GMAC/s measure SE-network efficiency.
4. Results and Discussion
Across DNS Challenge and CHiME-4, the proposed front-end improves WER over band-split baselines while reducing computational cost. Ablations support combining temporal mixing, cross-band interaction, and LOA, while operator analysis attributes efficiency to sequence-parallel modules.
- Main results: The proposed front-end achieves lower WER than noisy input, BSRNN, and Zhao et al. across DNS Challenge and real-recorded CHiME-4 conditions.The comparisons use frozen Whisper back-ends of Tiny, Medium, and Large sizes.
- Main results: Compared with BSRNN, the model achieves lower WER with substantially fewer parameters and lower MACs; compared with Zhao et al., it improves WER at similar or lower MACs.This establishes an accuracy–efficiency trade-off relative to both band-split baselines.
- Ablation study: Removing cross-band interaction consistently degrades WER on DNS and CHiME-4, indicating that per-frame cross-band modeling supports recognition-relevant band structure.The ablation uses Whisper Large with LOA enabled unless otherwise noted.
- Ablation study: Removing temporal mixing causes a larger degradation, confirming that strong intra-band temporal context remains crucial for suppressing non-stationary interference.Together with the cross-band ablation, this supports combining both operators within PTBM blocks.
- Ablation study: LOA outperforms direct enhanced-waveform input and fixed OA, achieving the best performance without development-set coefficient tuning.Increasing PTBM depth improves WER with diminishing returns; L=12 is selected as the default balance between accuracy and complexity.
- Operator-level analysis: TCM and CBA are sequence-parallel and have lower per-invocation cost than GRU-based temporal and cross-band modules.The operator-level comparison aligns with the front-end MAC reduction reported in Table 1.
5. Conclusion
The paper presents a sequence-parallel band-split enhancement front-end using PTBM blocks and LOA. Across DNS Challenge and CHiME-4 with frozen Whisper back-ends, it improves WER over recurrent baselines while reducing model size and MACs.
- Conclusion: PTBM combines temporal convolution mixing and per-frame cross-band attention without recurrent unrolling.This forms the core sequence-parallel enhancement architecture.
- Conclusion: LOA provides artifact-aware waveform blending without development-set coefficient tuning.The mechanism is introduced as part of the proposed front-end.
- Conclusion: Across DNS Challenge and CHiME-4 with frozen Whisper back-ends, the front-end consistently improves WER over recurrent band-split baselines while reducing model size and MACs.The conclusion characterizes parallel time–band mixing as a practical design solution for efficient robust-ASR front-ends.
6. Generative AI Use Disclosure
The paper discloses that generative AI tools were used only for language editing and stylistic polishing.
- Generative AI Use Disclosure: Generative AI tools were not used to generate the paper’s scientific ideas, methods, experiments, figures, tables, or results.The authors state that they take full responsibility for the paper’s content.