PaperlayerView exact revision on arXiv

Source-linked AI summary

SpatialNet: Extensively Learning Spatial Information for Multichannel Joint Speech Separation, Denoising and Dereverberation

Changsheng Quan, Xiaofei Li

arXiv:2307.16516v2cs.SDeess.AS

TL;DR

Real multichannel speech recordings are impaired by noise, reverberation, and interfering speakers, so this paper develops SpatialNet for joint speech separation, denoising, and dereverberation. It interleaves narrow-band and cross-band spatial processing, and reports state-of-the-art performance in almost all experiments across simulated and real datasets.

  • Problem

    Multichannel recordings contain ambient noise, room reverberation, and interfering speech, motivating joint separation, denoising, and dereverberation.

    Source: §I · p. 1

  • Method

    SpatialNet performs end-to-end multichannel enhancement with interleaved narrow-band and cross-band blocks that learn spatial information.

    Source: §I · p. 2

  • Results

    SpatialNet achieves state-of-the-art performance in almost all experiments on simulated and real datasets, in speech quality and ASR performance.

    Source: §I · p. 2

  • Takeaways & Limitations

    Across the reported tasks, results support spatial information as useful for speech separation, denoising, and dereverberation.

    Source: §I · p. 2

Abstract

from arXiv · show

This work proposes a neural network to extensively exploit spatial information for multichannel joint speech separation, denoising and dereverberation, named SpatialNet. In the short-time Fourier transform (STFT) domain, the proposed network performs end-to-end speech enhancement. It is mainly composed of interleaved narrow-band and cross-band blocks to respectively exploit narrow-band and cross-band spatial information. The narrow-band blocks process frequencies independently, and use self-attention mechanism and temporal convolutional layers to respectively perform spatial-feature-based speaker clustering and temporal smoothing/filtering. The cross-band blocks process frames independently, and use full-band linear layer and frequency convolutional layers to respectively learn the correlation between all frequencies and adjacent frequencies. Experiments are conducted on various simulated and real datasets, and the results show that 1) the proposed network achieves the state-of-the-art performance on almost all tasks; 2) the proposed network suffers little from the spectral generalization problem; and 3) the proposed network is indeed performing speaker clustering (demonstrated by attention maps).

I. INTRODUCTION

Real-world multichannel recordings combine noise, reverberation, and interfering speech, motivating joint speech separation, denoising, and dereverberation. SpatialNet combines narrow-band and cross-band processing, and the paper reports comparable or better performance than existing networks across tasks.

I. INTRODUCTION

  • SpatialNet combines frequency-independent narrow-band processing with frame-independent cross-band processing to learn both kinds of spatial information.Source: §I · p. 2
  • Narrow-band self-attention clusters spatial features across frames, while temporal convolution smooths signals and models convolutional reverberation.Source: §I · p. 2
  • Cross-band frequency convolutions learn correlations between adjacent frequencies, while a full-band linear module learns correlations across frequencies, including those associated with direct-path TDOA.Source: §I · p. 2
  • SpatialNet differs from related combined-band systems by using a Conformer narrow-band block and convolutional-linear cross-band block in an end-to-end pipeline.Source: §I · p. 2
  • SpatialNet achieves comparable or better performance than existing networks on all tasks, with state-of-the-art performance in almost all experiments.Source: §I · pp. 2–3

A. Time Domain Formulation

B. STFT Domain Formulation

The STFT formulation distinguishes desired speech from reverberation and noise through their time-frequency and spatial structure. Narrow-band models capture useful spatial cues, while cross-frequency relationships can improve modeling of desired speech.

Inter-frame and inter-frequency model

  • A sufficiently valid approximation of the inter-frame, inter-frequency convolution model uses neighboring frequencies, normally within [f − 4, f + 4].The full model sums over all frequencies; reports the neighboring-frequency approximation.Source: B. STFT Domain Formulation · p. 3

MTF approximation

  • The MTF approximation underlies common denoising and separation techniques but is unsuitable for highly reverberant scenarios when the STFT window is not sufficiently larger than the RIR.The fuller inter-frame, inter-frequency convolution model is accurate, but cross-band filtering makes it rarely used in practice.Source: B. STFT Domain Formulation · p. 3

Desired-speech spatial structure

  • Desired-speech relative transfer functions are highly correlated across frequencies, with frequency-invariant magnitude and phase linearly proportional to frequency.The passage notes that this relationship remains valid to a large extent outside free-field conditions.Source: B. STFT Domain Formulation · pp. 3–4

Spatial information for enhancement

  • Narrow-band spatial cues distinguish desired speech from other speakers, reverberation, and noise, while cross-frequency information can improve desired-speech modeling.Late reverberation and ambient noise are described as partially diffuse, unlike spatially coherent desired speech.Source: B. STFT Domain Formulation · p. 4

III. SPATIALNET

SpatialNet processes multichannel speech in the STFT domain with interleaved narrow-band and cross-band blocks, then predicts the target speech STFT coefficients. Its time-domain outputs are trained using negative SI-SDR with permutation invariant training.

III. SPATIALNET

  • SpatialNet concatenates the real and imaginary parts of multichannel microphone STFTs as input and predicts concatenated STFT coefficients for the desired speech signals.Inverse STFT converts the network output back to time-domain speech.Source: §III · p. 4
  • A convolutional input layer, interleaved narrow-band and cross-band blocks, and a linear output layer form the network architecture.The input layer uses a kernel size of 1 on frequency and 5 on time; the output layer maps the final hidden representation to target STFT coefficients.Source: §III · p. 4
  • Training minimizes negative SI-SDR on enhanced time-domain speech and uses permutation invariant training to address label permutation.Source: §III · p. 4

A. Narrow-band Block

The narrow-band block processes each STFT frequency independently with shared parameters. It uses self-attention to exploit spatial-vector similarity across frames and temporal convolutions for local smoothing and reverberation processing.

A. Narrow-band Block

  • Self-attention is used to compare spatial vectors across frames, reflecting the similarity-based principle of spatial-vector clustering.Frames dominated by different speakers or non-speech signals can be distinguished using their spatial vectors.Source: A. Narrow-band Block · p. 4
  • Temporal convolution layers perform local smoothing for signal-statistic estimation and model narrow-band reverberation processing.The passage connects this processing to convolution between the microphone signal and the CTF, and notes that the layers may imitate inverse filtering.Source: A. Narrow-band Block · p. 4
  • The narrow-band block processes a single STFT frequency at a time, with the frequency axis placed in the batch dimension.All frequencies share the same network parameters.Source: A. Narrow-band Block · pp. 4–5

1) Multi-head Self-attention Module:

The narrow-band block uses self-attention to collect or separate speaker components through spatial-vector similarity at each frequency. Its T-ConvFFN adds multiple temporal convolutions between feed-forward layers to support local smoothing and reverberation processing.

1) Multi-head Self-attention Module:

  • At one STFT frequency, multi-head self-attention computes spatial-vector similarity across time frames to collect or separate components from the same or different speakers.The module includes layer normalization, self-attention, dropout, and a residual connection.Source: 1) Multi-head Self-attention Module · p. 5

2) T-ConvFFN Module:

  • T-ConvFFN inserts three time-domain group-convolution layers between two linear layers, with group normalization after the second convolution.The convolutional layers operate on C′ channels split into G groups.Source: 2) T-ConvFFN Module · p. 5
  • Compared with the Conformer design [45], T-ConvFFN uses three convolutional layers between the feed-forward linear layers, operating on the expanded C′ channels for narrow-band smoothing and reverberation processing.Source: 2) T-ConvFFN Module · p. 5

B. Cross-band Block

SpatialNet’s cross-band block processes each STFT frame independently to exploit relationships among frequency bins. It combines frequency convolutions with full-band linear mapping.

B. Cross-band Block

  • The cross-band block processes each time frame independently, using frequency convolutions and a full-band linear module to learn cross-band spatial information.The motivation includes STFT-window leakage across neighboring frequencies and correlation of direct-path speech spatial features across frequencies.Source: B. Cross-band Block · p. 5

1) Frequency-convolutional Module:

  • Frequency convolutions model correlations between neighboring frequencies using group convolution along the frequency axis.The frequency-convolutional module also includes layer normalization and PReLU.Source: 1) Frequency-convolutional Module · p. 5

2) Full-band Linear Module:

  • The full-band linear module reduces hidden channels, maps across the frequency axis with channel-specific linear networks, then restores channels and adds the input.The full-band mapping networks are shared across repeated cross-band blocks; preliminary experiments found this sharing did not degrade performance.Source: 2) Full-band Linear Module · p. 6

C. Discussions

SpatialNet is designed to exploit narrow-band and cross-band spatial information, although its blocks may also learn spectral information. The discussion describes block ordering, network configurations, datasets, and evaluation measures.

C. Discussions

  • Spatial and spectral information are difficult to fully disentangle: narrow-band blocks may learn speech-noise spectral differences, while full-band mapping may learn full-band patterns.The authors argue full-band spectral learning is limited because cross-band blocks process frames independently and their mapping networks are shared across blocks.Source: C. Discussions · p. 6
  • Narrow-band blocks contribute more to performance and consume more computation, while cross-band blocks provide auxiliary information; the authors report somewhat better performance with cross-band blocks before narrow-band blocks.The discussion says the two block types are repeated many times, so their order does not matter much overall.Source: C. Discussions · p. 6

1) Network Configuration:

  • SpatialNet-small uses 8 blocks and hidden sizes C=96, C′=192, C′′=8; SpatialNet-large uses 12 blocks and C=192, C′=384, C′′=16.The networks use 4 self-attention heads; model sizes vary by sample rate and frequency count.Source: 1) Network Configuration · p. 6

2) Datasets:

  • Evaluation covers six simulated and real datasets, with dataset-specific array training and speech-quality and ASR measures.The datasets include SMS-WSJ, WHAMR!, Spatialized WSJ0-2mix, LibriCSS, Reverb Challenge, and CHiME 3/4; the measures include SDR, SI-SDR, PESQ, STOI/eSTOI, and WER.Source: 2) Datasets · pp. 6–7, 3) Evaluation Metrics · p. 7

4) Comparison Methods:

The paper compares against reported dataset-specific methods and uses SMS-WSJ-Plus ablations to examine SpatialNet’s components. The ablations identify narrow-band modules as major contributors, and attention maps support a speaker-clustering interpretation of MHSA.

4) Comparison Methods:

  • For each dataset, comparisons use methods that have reported results there, with results generally quoted from their original papers.The authors report searching for recently proposed state-of-the-art methods.Source: 4) Comparison Methods · p. 7

B. Ablation Studies

  • The SMS-WSJ-Plus ablations use six microphones and extend reverberation, speaker distance, interference, and added noise conditions.The setup extends T60 to [0.1, 1.0] s, speaker-to-microphone distance to m, SIR to [-5, 5] dB, and noise SNR to [0, 20] dB.Source: B. Ablation Studies · p. 7

1) Contribution of Sub-networks:

  • Each subnet contributes to enhancement, with narrow-band modules playing a more fundamental role than cross-band modules.MHSA and T-ConvFFN use 11.0 G/s and 10.1 G/s FLOPs, respectively, compared with 1.5 G/s for frequency-convolutional modules and 0.3 G/s for the full-band linear module.Source: 1) Contribution of Sub-networks · p. 7, Table I · p. 7

2) Attention Map Analysis:

  • Two representative attention heads attend differently: one models non-speech signals, while the other attends to speech and clusters frames by speaker.The authors interpret this as speaker and non-speech clustering based on narrow-band spatial features.Source: 2) Attention Map Analysis · pp. 7–8, Figure 2 · p. 8

3) Exploitation of Spectral Information :

SpatialNet relies more on spatial than spectral information for multichannel separation, with mild cross-language degradation. On SMS-WSJ, SpatialNet-large exceeds TFGridNet on speech enhancement and is comparable on ASR performance.

3) Exploitation of Spectral Information :

  • When two speakers share a position, SpatialNet’s separation degrades more than Beam-Guided TasNet’s, and the speakers almost cannot be separated.Noise and reverberation can still be largely suppressed; the authors interpret the separation result as evidence that SpatialNet uses little spectral information for separation.Source: 3) Exploitation of Spectral Information · p. 8, Table II · p. 8
  • Single-channel SpatialNet also performs well, but its separation degrades when speakers share a position, indicating use of their single-channel spatial difference.Its degradation is smaller than in the six-channel case, suggesting greater use of spectral information.Source: 3) Exploitation of Spectral Information · p. 8
  • SpatialNet’s cross-language performance degradation is mild and smaller than Beam-Guided TasNet’s, indicating dominant use of spatial over spectral information.On English test data, SpatialNet achieves similar ESTOI and SDR scores whether trained on English or Chinese data.Source: 3) Exploitation of Spectral Information · pp. 8–9, Table III · p. 9

C. Results on SMS-WSJ

  • SpatialNet-large achieves better speech enhancement and comparable ASR performance than TFGridNet on SMS-WSJ.SpatialNet-small also performs well with fewer parameters and computations.Source: C. Results on SMS-WSJ · p. 9, Table IV · p. 9

D. Results on WHAMR!

SpatialNet-large slightly outperforms TFGridNet on WHAMR!, while SpatialNet-small outperforms comparison methods on Spatialized WSJ0-2mix. On LibriCSS, SpatialNet-large substantially outperforms neural beamformers in ASR performance.

D. Results on WHAMR!

  • SpatialNet-large slightly outperforms TFGridNet on WHAMR!, a dataset with severe environmental noise and reverberation.The authors note that exploiting spatial information is more challenging here, especially with only two microphones.Source: D. Results on WHAMR! · p. 9

E. Results on Spatialized WSJ0-2mix

  • On four-channel Spatialized WSJ0-2mix, SpatialNet-small outperforms all comparison methods, while SpatialNet-large achieves almost perfect speech quality.The dataset’s varying array geometry challenges end-to-end generalization, yet SpatialNet and DasFormer show array-generalization capability.Source: E. Results on Spatialized WSJ0-2mix · p. 10, Table VI · p. 10

F. Results on LibriCSS

  • On LibriCSS, SpatialNet-large largely outperforms three neural beamformers in ASR performance.Because LibriCSS provides evaluation data only, the networks are trained on simulated signals.Source: F. Results on LibriCSS · p. 10, Table VII · p. 10

G. Results on Reverb Challenge Dataset

On Reverb Challenge, SpatialNet leads the compared methods on enhancement metrics and improves ASR on real recordings, especially at far distance. On simulated CHiME 3/4, it is comparable to and outperforms other comparison methods.

G. Results on Reverb Challenge Dataset

  • Reverb Challenge evaluation uses simulated and real recordings, an eight-channel circular array, and near (50 cm) and far (200 cm) speaker distances.Training uses 40,000 simulated RIRs because the training set contains only 24 real-recorded RIRs; both dry speech and direct-path signal serve as evaluation references.Source: G. Results on Reverb Challenge Dataset · pp. 10–11
  • The Reverb Challenge comparison includes WPE, WPE+BeamformIt, WPD, and deep-learning methods from.For WPD, prediction delay and tap count are set to 3 and 5, respectively.Source: G. Results on Reverb Challenge Dataset · p. 11
  • SpatialNet outperforms comparison methods by a large margin on CD, FWSSNR, and PESQ, and improves ASR over WPE- and beamforming-based methods on RealData, especially far data.On SimData, all methods achieve WERs close to clean speech.Source: G. Results on Reverb Challenge Dataset · p. 11, Table VIII · p. 11, Table IX · p. 11

H. Results on CHiME 3/4 Challenge Dataset

  • The CHiME 3/4 evaluation reports simulated-data enhancement only, where reverberation is very small and the task is denoising with real-recorded multichannel noise.The recordings use six tablet microphones, with channel five as reference.Source: H. Results on CHiME 3/4 Challenge… · p. 11
  • On CHiME 3/4, NBDF [23] outperforms MNMF beamforming and FasNet+TAC, while FT-JNF further improves performance by adding cross-band learning.McNet further improves performance by learning spectral differences between speech and noise; prominently outperforms these methods.Source: H. Results on CHiME 3/4 Challenge… · p. 11
  • SpatialNet achieves comparable results with and largely outperforms other comparison methods on simulated CHiME 3/4 data.SpatialNet-small slightly outperforms SpatialNet-large, possibly because low reverberation requires less information to learn.Source: H. Results on CHiME 3/4 Challenge… · p. 12, Table X · p. 12

I. Summary of Computational Complexity and Model Size

The complexity comparison covers FLOPs and parameter counts under specified channel, duration, and sampling-rate settings. SpatialNet-small is relatively compact among narrow-band methods, while SpatialNet-large has substantially higher FLOPs; the conclusion also notes offline-only processing and little spectral generalization difficulty.

I. Summary of Computational Complexity and Model Size

  • The complexity comparison measures FLOPs and parameter counts for six-channel, four-second, two-speaker utterances at 8 kHz and 16 kHz where code is available.STFT-domain methods use 32 ms windows and 16 ms hops, except TFGridNet, which uses a 32 ms window and 8 ms hop.Source: §I · p. 12
  • SpatialNet-small has relatively low FLOPs and model size among narrow-band methods while achieving outstanding speech enhancement performance.TFGridNet and SpatialNet-large have much larger FLOPs and perform better on most tasks than other methods.Source: §I · p. 12
  • Narrow-band STFT methods process frequencies independently, increasing computational complexity relative to full-band methods and doubling FLOPs when sampling rate rises from 8 kHz to 16 kHz.Their model sizes are usually small because parameters are shared across frequencies.Source: §I · p. 12

V. CONCLUSIONS

  • SpatialNet is designed for offline, noncausal processing; an online version is left for future development.The conclusion also reports little spectral generalization difficulty and good performance on a cross-language speech enhancement task.Source: §V · pp. 12–13

Related papers

Loading 2307.16516v2…