PaperlayerView exact revision on arXiv

Source-linked AI summary

GAN-based Joint Dereverberation and Directional Filtering

Weilong Huang, Shrishti Saha Shetu, Emanuël A. P. Habets

arXiv:2608.26403v1eess.AS

TL;DR

Strong reverberation makes spatial cues difficult to distinguish, limiting neural directional filtering for spatial sound capture. The paper proposes joint neural dereverberation and directional filtering, implemented with discriminative and GAN-based models, and finds consistent gains over cascaded baselines, with GAN-based NDDF strongest for high-order targets.

  • Problem

    Strong reverberation degrades spatial cues and limits NDF-based spatial sound capture, especially when using compact microphone arrays.

    Source: §1 · p. 1

  • Method

    NDDF jointly reconstructs a dereverberated VDM from array input using discriminative and GAN-based neural models, with input–output-based directivity estimation.

    Source: §1 · p. 1

  • Results

    NDDF consistently outperforms cascaded dereverberation and directional-filtering baselines, while GAN-based NDDF outperforms the discriminative variant for high-order VDM targets.

    Source: §1 · p. 1, §6 · p. 4

  • Takeaways & Limitations

    Input–output-based directivity estimation extends directivity analysis to signal-mapping spatial filters that synthesize outputs without explicit filtering or masking.

    Source: §1 · p. 1, §6 · p. 4

    The formulation assumes spatially uncorrelated sensor noise and training with sources and microphones lying in the x-y plane.

    Source: §2 · p. 1, §3.2 · p. 2

Abstract

from arXiv · show

Recently, neural directional filtering (NDF) enables reconstruction of a virtual directional microphone (VDM) with a desired directivity pattern, accurately rendering multi-source scenes by preserving spatial cues. In strongly reverberant environments, spatial cues become perceptually difficult to distinguish, limiting NDF-based spatial sound capture. This paper addresses this limitation with three contributions: First, we propose a neural dereverberation and directional filtering (NDDF) approach to reconstruct dereverberated VDM signals. Second, NDDF is implemented with discriminatively trained and generative adversarial network (GAN)-based models, compared with cascaded dereverberation and directional-filtering baselines. Experimental results indicate that the NDDF consistently surpasses the cascaded baselines. Additionally, the GAN-based NDDF outperforms the discriminative variant when addressing a high-order VDM target. Third, we introduce a method for directivity pattern estimation that relies solely on the input and output signals. This method is suitable for signal-mapping-based spatial filtering, which synthesizes the output signal directly without explicit filtering or masking.

1. INTRODUCTION

Reverberation degrades spatial cues in compact-array capture, motivating a joint method that combines dereverberation with flexible neural directional filtering. The paper proposes NDDF and reports that it outperforms cascaded baselines, with GAN-based NDDF especially effective for high-order VDM targets.

  • Reverberant reflections overlap direct sound and degrade spatial cues such as interaural level difference in enclosed environments.Source: §1 · p. 1
  • Compact arrays with small apertures and few microphones obtain limited performance from conventional fixed beamforming.Source: §1 · p. 1
  • First-order cardioid patterns provide only approximately 4.8 dB directivity index, insufficient to suppress reverberant energy in strongly reverberant environments.Source: §1 · p. 1
  • The paper formulates dereverberation and directional filtering jointly because cascaded stages optimize independently and may not produce an optimal final output.Source: §1 · p. 1
  • NDDF reconstructs a dereverberated VDM directly from array input using discriminative and GAN-based models, and consistently outperforms cascaded baselines.Source: §1 · p. 1
  • For a high-order VDM target, GAN-based NDDF outperforms the discriminative variant and motivates input–output-only directivity estimation for signal-mapping filters.Source: §1 · p. 1

2. PROBLEM FORMULATION

The problem formulation models reverberant multi-source mixtures captured by compact omnidirectional arrays and defines VDM reconstruction for a specified directivity pattern. NDDF targets the spatially coherent portion of the VDM response to reduce late-reflection effects.

  • A compact array of Q omnidirectional microphones captures N sound sources in a reverberant room.Source: §2 · p. 1
  • In the STFT domain, Xq,n(f, t) denotes source n at microphone q, while Yq(f, t) denotes the microphone mixture.Source: §2 · p. 1
  • The mixture includes spatially uncorrelated sensor noise Vq(f, t), and each source contribution is modeled using an RTF Hq,n(f) and source signal Sn(f, t).Source: §2 · p. 1
  • NDF reconstructs a VDM signal at the reference microphone according to a specified directivity pattern Λ(θ, ϕ).Source: §2 · pp. 1–2
  • The VDM transfer function accounts for propagation paths whose reflection contributions are weighted by directivity gains at their incident directions.Source: §2 · p. 2
  • NDDF decomposes the VDM transfer function into spatially coherent and diffuse components, targeting a dereverberated signal by reducing late reflections.Source: §2 · p. 2

3. PROPOSED METHOD

The proposed method jointly reconstructs dereverberated virtual directional microphone signals using neural architectures, with targets based on direct sound and early reflections and experiments spanning first- and sixth-order cardioids.

DNN Architecture and Training Loss

  • The GAN-based architecture uses a SEANet generator with a UNet-like time-frequency encoder–decoder and skip connections.Microphone magnitude-phase features are concatenated before encoder processing.Source: §3.1 · p. 2
  • The signal-based UNet estimates dereverberated VDM signals directly, while the mask-based UNet estimates a complex mask applied to the reference signal.The mask operation is bZ(f, t) = M(f, t)Y1(f, t).Source: §3.1 · p. 2
  • The generator loss combines temporal, spectral, adversarial, and discriminator-feature terms with weights λ1 through λ4.The objective is LGenerator = λ1 Ltemp + λ2 Lspec + λ3 Ladv + λ4 Lfeat.Source: §3.1 · p. 2

Training Strategy

  • Training targets use first-order and sixth-order Cardioid directivity patterns defined by a Jth-order Cardioid formulation.The target direction is specified by θs and ϕs, and null attenuation is set to −30 dB.Source: §3.2 · p. 2
  • Training simulates random source-array configurations with up to three concurrent sources, random rooms, and generated room impulse responses.The room settings are summarized as ranges in Table 1.Source: §3.2 · pp. 2–3, Table 1 · p. 3
  • The dereverberated target is constructed by windowing the VDM room impulse response to preserve direct sound and early reflections.The windowing approximates the coherent transfer-function component used for target computation.Source: Figure 2 · p. 2, §3.2 · p. 3

4. EXPERIMENTAL SETUP

Experiments use a compact four-microphone array and simulated reverberant rooms, evaluate intrusive and non-intrusive speech metrics, and estimate directivity patterns from input–output signals.

Experimental Configuration

  • The experimental array contains three 3-cm-diameter UCA microphones and one central reference microphone.The reference signal is the first NDDF input channel, while a selected UCA element supplies the target-direction channel.Source: §4.1 · p. 3

Datasets and Simulation

  • Training and validation speech come from LibriSpeech, while test speech comes from EARS; all signals are sampled at 16 kHz.The test samples use a minimum loudness threshold of −42 dBFS.Source: §4.1 · p. 3

Performance Measures

  • Intrusive evaluation uses frequency-weighted segmental SDR and PESQ, both requiring target references.Time-domain SDR is excluded because sample-level alignment is unsuitable for generative models.Source: §4.2 · p. 3

Directivity Pattern Estimation

  • The directivity estimator decomposes the reference signal into direct-path, early-reflection, and diffuse components before computing a source-direction power ratio.The decomposition is applied to the reference microphone signal and the target NDDF output.Source: §4.2 · p. 3
  • A thresholded dominance test selects bins where a source direct path dominates coherent components and coherent energy exceeds diffuse energy.The decision threshold is δ = 0.4, and direction-wise estimates are averaged over test samples.Source: §4.2 · p. 3

5. PERFORMANCE EVALUATION

NDDF is evaluated against cascaded dereverberation and directional-filtering baselines across reverberation conditions and target directivity patterns. GAN-based training shows the clearest advantages for the challenging 6th-order target and under stronger reverberation.

Performance comparison

  • Table 2 compares NDDF variants with cascaded baselines across several RT60 values using fwSDRseg, C50, SRMR, and PESQ.All metrics are positively oriented; fwSDRseg, C50, and SRMR use dB, while PESQ uses the MOS scale.Source: §5 · p. 3, Table 2 · p. 4

1st-order target

  • For the 1st-order target, GAN-based training yields clearly higher C50, especially under severe reverberation at RT60 = 0.8 s.Differences between training paradigms are relatively small on fwSDRseg and PESQ.Source: §5 · p. 4

6th-order target

  • For the 6th-order target, GAN-based methods achieve better overall fwSDRseg, PESQ, and C50 than discriminative methods, increasingly so as reverberation rises.The advantage is especially pronounced for models sharing the mask-based UNet backbone.Source: §5 · p. 4

Directivity patterns

  • Under low reverberation, FT-JNF NDDF and GAN NDDF produce nearly identical sixth-order directivity patterns.The compared variants are a discriminative FT-JNF model and a GAN model using a signal-based UNet.Source: §5 · p. 4, Figure 3 · p. 4
  • Under high reverberation, GAN NDDF more closely approximates the target mainlobe and exhibits less directivity-pattern distortion than FT-JNF NDDF.Source: §5 · p. 4

6. CONCLUSIONS

The paper concludes that NDDF addresses NDF limitations in reverberant conditions through joint dereverberation and directional filtering. GAN-based NDDF achieves the best overall performance, while input–output directivity estimation supports directivity analysis for signal-mapping spatial filtering.

  • NDDF reconstructs a dereverberated virtual directional microphone signal to address NDF limitations in reverberant conditions.Source: §6 · p. 4
  • Both discriminative and GAN-based NDDF variants were benchmarked against established cascaded baselines.Source: §6 · p. 4
  • GAN-based NDDF achieved the best overall performance across the experiments.Source: §6 · p. 4
  • Input–output-based directivity pattern estimation enables directivity analysis for signal-mapping spatial filtering approaches.Source: §6 · p. 4
Loading 2608.26403v1…