Source-linked AI summary
FaSNet: Low-latency Adaptive Beamforming for Multi-microphone Audio Processing
Yi Luo, Enea Ceolini, Cong Han, Shih-Chii Liu, Nima Mesgarani
TL;DR
Low-latency applications need alternatives to neural beamformers that rely on non-causal processing and large context. FaSNet addresses this with a two-stage time-domain adaptive filter-and-sum architecture, achieving strong enhancement, separation, and ASR results, including 14.3% RWERR on CHiME-3. Its evaluation is bounded by simulated microphone-array conditions and direct-path targets.
Problem
Existing neural beamformers often require non-causal processing and large context for robust inter-channel feature estimation, limiting their suitability for low-latency applications.
Method
FaSNet estimates time-domain adaptive beamforming filters in two stages, first for a selected reference microphone and then for the remaining microphones.
Results
14.3% RWERR was achieved against the MVDR baseline on CHiME-3 with a mel-spectrogram target and SI-MSE objective, while FaSNet matched or surpassed oracle traditional beamformers across enhancement and separation tasks.
Takeaways & Limitations
FaSNet provides a flexible low-latency beamforming framework that can also be combined with single-channel enhancement systems.
Takeaways & Limitations
The evaluation assumes a circular omnidirectional array with at most four microphones and uses direct-path speech signals as targets.
Abstract
from arXiv · showhide
Beamforming has been extensively investigated for multi-channel audio processing tasks. Recently, learning-based beamforming methods, sometimes called \textit{neural beamformers}, have achieved significant improvements in both signal quality (e.g. signal-to-noise ratio (SNR)) and speech recognition (e.g. word error rate (WER)). Such systems are generally non-causal and require a large context for robust estimation of inter-channel features, which is impractical in applications requiring low-latency responses. In this paper, we propose filter-and-sum network (FaSNet), a time-domain, filter-based beamforming approach suitable for low-latency scenarios. FaSNet has a two-stage system design that first learns frame-level time-domain adaptive beamforming filters for a selected reference channel, and then calculate the filters for all remaining channels. The filtered outputs at all channels are summed to generate the final output. Experiments show that despite its small model size, FaSNet is able to outperform several traditional oracle beamformers with respect to scale-invariant signal-to-noise ratio (SI-SNR) in reverberant speech enhancement and separation tasks. Moreover, when trained with a frequency-domain objective function on the CHiME-3 dataset, FaSNet achieves 14.3\% relative word error rate reduction (RWERR) compared with the baseline model. These results show the efficacy of FaSNet particularly in reverberant and noisy signal conditions.
2. INTRODUCTION
Neural beamformers improve multi-channel audio processing but often conflict with low-latency requirements. FaSNet addresses this gap with a time-domain adaptive filter-and-sum design and flexible training objectives.
- Motivation: Neural beamformers have shown effectiveness for signal enhancement and ASR, but commonly require non-causal processing and extensive context.These requirements limit their suitability for applications demanding low-latency responses.
- Related approaches: Neural beamformers comprise filtering-based, masking-based, and regression-based approaches with different degrees of explicit beamforming.Filtering-based systems learn filters directly, masking-based systems estimate time-frequency masks for spatial covariance features, and regression-based systems learn the mapping implicitly.
- Proposed method: FaSNet is a time-domain adaptive filter-and-sum framework designed for realtime, low-latency applications.Its first stage estimates a filter for a selected reference channel, while its second stage estimates filters for the remaining channels.
- Proposed method: FaSNet uses normalized cross-correlation between channels as an inter-channel feature and can be trained with signal-level or ASR-level objectives.The framework uses temporal convolutional networks for low-resource, low-latency processing and supports objectives such as SNR or mel-spectrogram criteria.
3. FILTER-AND-SUM NETWORK (FASNET)
FaSNet is a two-stage time-domain adaptive filter-and-sum beamformer: it estimates filters for a reference microphone, then uses its cleaned output to estimate filters for the remaining channels. It combines NCC-based inter-channel features with channel-specific embeddings and supports time- or frequency-domain training objectives.
- 3.1. Problem definition: FaSNet frames each microphone signal and estimates time-domain filters whose filtered outputs are summed to reconstruct the reference signal.Context windows accommodate cross-microphone delays of ±L samples during filter estimation.
- 3.2. Reference channel processing: The first stage estimates reference-microphone filters from NCC features computed between the reference context window and other microphone frames.The NCC feature is implemented as cosine similarity and combines time-delay and signal-content information.
- 3.2. Reference channel processing: Mean-pooling combines the NCC features from non-reference microphones in a permutation-free representation.A linear projection of the reference center frame is concatenated with the pooled feature and passed to a TCN with a gated output layer.
- 3.3. Remaining channel processing: The second stage uses each source estimate from the reference stage as a cue for computing NCC features and filters for all remaining microphones.The remaining microphones share the same TCN and gated output layer; their filtered signals are summed with the reference-stage output.
- 3.4. Output and training: The complete FaSNet system performs overlap-and-add reconstruction and can be paired with a single-channel enhancement model for end-to-end training.Training can use SI-SNR for signal-quality tasks or mel-spectrogram SI-MSE for frequency-domain tasks such as ASR, with uPIT applied for output permutation.
4. EXPERIMENT CONFIGURATIONS
FaSNet is evaluated on simulated echoic enhancement and separation tasks, multichannel noisy ASR, and comparisons with classical time- and frequency-domain beamformers.
- Evaluation tasks: FaSNet is evaluated on echoic noisy speech enhancement, jointly addressing denoising and dereverberation.
- Evaluation tasks: FaSNet is evaluated on echoic noisy speech separation, separating the direct paths of two speakers in noise.
- Evaluation tasks: FaSNet is evaluated for multichannel noisy ASR using the third CHiME challenge dataset.
- Data and targets: The direct-path speech signals for all sources of interest serve as the evaluation targets.
- Data and targets: For ESE and ESS, experiments use circular omnidirectional arrays with up to 4 microphones and a fixed 10 cm diameter.
- System configuration: Each TCN uses R = 2 repeats and P = 5 blocks per repeat, with 64 input channels, 320 hidden channels, and 64-dimensional embeddings.Each TCN contains 0.76M parameters; the convolutional kernel size is 3.
- Beamformer comparisons: Classical baselines include time-domain TD-MWF and TD-MVDR, plus frequency-domain SDW-MWF and MVDR beamformers.The comparison includes both time- and frequency-domain beamformers, with time-domain comparisons described as fairer for FaSNet.
5. RESULTS AND DISCUSSION
Experiments evaluate FaSNet across oracle beamformer benchmarks, frame and segment latency trade-offs, enhancement and separation tasks, real recordings, and adaptive filter behavior.
- 5.1. Benchmarking oracle beamforming techniques on signal quality measurement: Oracle beamformers were evaluated on ESE and ESS using SI-SNR improvement, with time- and frequency-domain methods using full-utterance spatial covariance estimates.TD-MVDR performed better than TD-MWF on ESS, while TD-MWF was better on ESE; SDW-MWF outperformed MVDR among frequency-domain beamformers.
- 5.1. Benchmarking oracle beamforming techniques on signal quality measurement: Short-segment evaluation showed that MB-MVDR requires a large receptive field for reasonable performance, making rapid-changing conditions harder to handle.Spatial covariance estimation was tested on nonoverlapping 100, 250, and 500 ms segments.
- 5.2. Results of FaSNet on various tasks: FaSNet performance improved with longer frames, but its latency is 2L, requiring a performance–frame-size trade-off for strict low-latency applications.The selected configuration for subsequent experiments was L = 16.
- 5.2. Results of FaSNet on various tasks: In causal settings, FaSNet outperformed Conv-TasNet with two microphones, and its 4-channel tandem system exceeded segment-level oracle MB-MVDR.The tandem system also improved performance across almost all tested configurations and had potential for low-latency processing on shorter segments.
- 5.2. Results of FaSNet on various tasks: On CHiME-3, FaSNet significantly outperformed Conv-TasNet with both reverberant-clean and original-clean training targets, while achieving 14.3% RWERR with a mel-spectrogram objective versus MVDR.The beampattern visualization further shows content-dependent filters across frames and different regions within utterances.
6. CONCLUSION
FaSNet is a two-stage, time-domain adaptive beamforming method designed for online, low-latency applications. It matched or exceeded several oracle traditional beamformers and reduced CHiME-3 word error rate by 14.3% relative to a baseline.
- FaSNet uses a two-stage design to estimate a reference-microphone filter, then filters for the remaining microphones.The second stage uses the first stage’s output to calculate filters for all remaining microphones.
- FaSNet achieved better or on par performance than several oracle traditional beamformers on ESE and ESS tasks.The comparison covered echoic noisy speech enhancement and echoic noisy speech separation.
- 14.3% relative word error rate reduction was achieved on CHiME-3 compared with a baseline model.This result used a frequency-domain objective favored by backend speech-recognition systems.
- FaSNet generated content-dependent adaptive filters for speech and nonspeech regions.Beampattern visualizations showed different adaptive filtering behavior across regions.
- FaSNet can be concatenated with another single-channel system for further performance improvement.