Source-linked AI summary
GAN-based Joint Dereverberation and Directional Filtering
Weilong Huang, Shrishti Saha Shetu, Emanuël A. P. Habets
TL;DR
Strong reverberation makes spatial cues difficult to distinguish, limiting neural directional filtering for spatial sound capture. The paper proposes joint neural dereverberation and directional filtering, implemented with discriminative and GAN-based models, and finds consistent gains over cascaded baselines, with GAN-based NDDF strongest for high-order targets.
Problem
Strong reverberation degrades spatial cues and limits NDF-based spatial sound capture, especially when using compact microphone arrays.
Method
NDDF jointly reconstructs a dereverberated VDM from array input using discriminative and GAN-based neural models, with input–output-based directivity estimation.
Results
NDDF consistently outperforms cascaded dereverberation and directional-filtering baselines, while GAN-based NDDF outperforms the discriminative variant for high-order VDM targets.
Takeaways & Limitations
Input–output-based directivity estimation extends directivity analysis to signal-mapping spatial filters that synthesize outputs without explicit filtering or masking.
Takeaways & Limitations
The formulation assumes spatially uncorrelated sensor noise and training with sources and microphones lying in the x-y plane.
Abstract
from arXiv · showhide
Recently, neural directional filtering (NDF) enables reconstruction of a virtual directional microphone (VDM) with a desired directivity pattern, accurately rendering multi-source scenes by preserving spatial cues. In strongly reverberant environments, spatial cues become perceptually difficult to distinguish, limiting NDF-based spatial sound capture. This paper addresses this limitation with three contributions: First, we propose a neural dereverberation and directional filtering (NDDF) approach to reconstruct dereverberated VDM signals. Second, NDDF is implemented with discriminatively trained and generative adversarial network (GAN)-based models, compared with cascaded dereverberation and directional-filtering baselines. Experimental results indicate that the NDDF consistently surpasses the cascaded baselines. Additionally, the GAN-based NDDF outperforms the discriminative variant when addressing a high-order VDM target. Third, we introduce a method for directivity pattern estimation that relies solely on the input and output signals. This method is suitable for signal-mapping-based spatial filtering, which synthesizes the output signal directly without explicit filtering or masking.
1. INTRODUCTION
Reverberation degrades spatial cues in compact-array capture, motivating a joint method that combines dereverberation with flexible neural directional filtering. The paper proposes NDDF and reports that it outperforms cascaded baselines, with GAN-based NDDF especially effective for high-order VDM targets.
- Reverberant reflections overlap direct sound and degrade spatial cues such as interaural level difference in enclosed environments.
- Compact arrays with small apertures and few microphones obtain limited performance from conventional fixed beamforming.
- First-order cardioid patterns provide only approximately 4.8 dB directivity index, insufficient to suppress reverberant energy in strongly reverberant environments.
- The paper formulates dereverberation and directional filtering jointly because cascaded stages optimize independently and may not produce an optimal final output.
- NDDF reconstructs a dereverberated VDM directly from array input using discriminative and GAN-based models, and consistently outperforms cascaded baselines.
- For a high-order VDM target, GAN-based NDDF outperforms the discriminative variant and motivates input–output-only directivity estimation for signal-mapping filters.
2. PROBLEM FORMULATION
The problem formulation models reverberant multi-source mixtures captured by compact omnidirectional arrays and defines VDM reconstruction for a specified directivity pattern. NDDF targets the spatially coherent portion of the VDM response to reduce late-reflection effects.
- A compact array of Q omnidirectional microphones captures N sound sources in a reverberant room.
- In the STFT domain, Xq,n(f, t) denotes source n at microphone q, while Yq(f, t) denotes the microphone mixture.
- The mixture includes spatially uncorrelated sensor noise Vq(f, t), and each source contribution is modeled using an RTF Hq,n(f) and source signal Sn(f, t).
- NDF reconstructs a VDM signal at the reference microphone according to a specified directivity pattern Λ(θ, ϕ).
- The VDM transfer function accounts for propagation paths whose reflection contributions are weighted by directivity gains at their incident directions.
- NDDF decomposes the VDM transfer function into spatially coherent and diffuse components, targeting a dereverberated signal by reducing late reflections.
3. PROPOSED METHOD
The proposed method jointly reconstructs dereverberated virtual directional microphone signals using neural architectures, with targets based on direct sound and early reflections and experiments spanning first- and sixth-order cardioids.
- DNN Architecture and Training Loss: The GAN-based architecture uses a SEANet generator with a UNet-like time-frequency encoder–decoder and skip connections.Microphone magnitude-phase features are concatenated before encoder processing.
- DNN Architecture and Training Loss: The signal-based UNet estimates dereverberated VDM signals directly, while the mask-based UNet estimates a complex mask applied to the reference signal.The mask operation is bZ(f, t) = M(f, t)Y1(f, t).
- DNN Architecture and Training Loss: The generator loss combines temporal, spectral, adversarial, and discriminator-feature terms with weights λ1 through λ4.The objective is LGenerator = λ1 Ltemp + λ2 Lspec + λ3 Ladv + λ4 Lfeat.
- Training Strategy: Training targets use first-order and sixth-order Cardioid directivity patterns defined by a Jth-order Cardioid formulation.The target direction is specified by θs and ϕs, and null attenuation is set to −30 dB.
- Training Strategy: Training simulates random source-array configurations with up to three concurrent sources, random rooms, and generated room impulse responses.The room settings are summarized as ranges in Table 1.
- Training Strategy: The dereverberated target is constructed by windowing the VDM room impulse response to preserve direct sound and early reflections.The windowing approximates the coherent transfer-function component used for target computation.
4. EXPERIMENTAL SETUP
Experiments use a compact four-microphone array and simulated reverberant rooms, evaluate intrusive and non-intrusive speech metrics, and estimate directivity patterns from input–output signals.
- Experimental Configuration: The experimental array contains three 3-cm-diameter UCA microphones and one central reference microphone.The reference signal is the first NDDF input channel, while a selected UCA element supplies the target-direction channel.
- Datasets and Simulation: Training and validation speech come from LibriSpeech, while test speech comes from EARS; all signals are sampled at 16 kHz.The test samples use a minimum loudness threshold of −42 dBFS.
- Performance Measures: Intrusive evaluation uses frequency-weighted segmental SDR and PESQ, both requiring target references.Time-domain SDR is excluded because sample-level alignment is unsuitable for generative models.
- Directivity Pattern Estimation: The directivity estimator decomposes the reference signal into direct-path, early-reflection, and diffuse components before computing a source-direction power ratio.The decomposition is applied to the reference microphone signal and the target NDDF output.
- Directivity Pattern Estimation: A thresholded dominance test selects bins where a source direct path dominates coherent components and coherent energy exceeds diffuse energy.The decision threshold is δ = 0.4, and direction-wise estimates are averaged over test samples.
5. PERFORMANCE EVALUATION
NDDF is evaluated against cascaded dereverberation and directional-filtering baselines across reverberation conditions and target directivity patterns. GAN-based training shows the clearest advantages for the challenging 6th-order target and under stronger reverberation.
- Performance comparison: Table 2 compares NDDF variants with cascaded baselines across several RT60 values using fwSDRseg, C50, SRMR, and PESQ.All metrics are positively oriented; fwSDRseg, C50, and SRMR use dB, while PESQ uses the MOS scale.
- 1st-order target: For the 1st-order target, GAN-based training yields clearly higher C50, especially under severe reverberation at RT60 = 0.8 s.Differences between training paradigms are relatively small on fwSDRseg and PESQ.
- 6th-order target: For the 6th-order target, GAN-based methods achieve better overall fwSDRseg, PESQ, and C50 than discriminative methods, increasingly so as reverberation rises.The advantage is especially pronounced for models sharing the mask-based UNet backbone.
- Directivity patterns: Under low reverberation, FT-JNF NDDF and GAN NDDF produce nearly identical sixth-order directivity patterns.The compared variants are a discriminative FT-JNF model and a GAN model using a signal-based UNet.
- Directivity patterns: Under high reverberation, GAN NDDF more closely approximates the target mainlobe and exhibits less directivity-pattern distortion than FT-JNF NDDF.
6. CONCLUSIONS
The paper concludes that NDDF addresses NDF limitations in reverberant conditions through joint dereverberation and directional filtering. GAN-based NDDF achieves the best overall performance, while input–output directivity estimation supports directivity analysis for signal-mapping spatial filtering.
- NDDF reconstructs a dereverberated virtual directional microphone signal to address NDF limitations in reverberant conditions.
- Both discriminative and GAN-based NDDF variants were benchmarked against established cascaded baselines.
- GAN-based NDDF achieved the best overall performance across the experiments.
- Input–output-based directivity pattern estimation enables directivity analysis for signal-mapping spatial filtering approaches.