Source-linked AI summary
BASEN: Time-Domain Brain-Assisted Speech Enhancement Network with Convolutional Cross Attention in Multi-talker Conditions
Jie Zhang, Qing-Tian Xu, Qiu-Shi Zhu, Zhen-Hua Ling
TL;DR
Monaural speech enhancement in multi-talker conditions remains challenging when the target speaker is unknown. BASEN incorporates listener EEG into a time-domain enhancement network and fuses audio and EEG features with cross-attention. On the evaluated public dataset, it outperforms UBESD across reported metrics and subjects.
Problem
Monaural enhancement in multi-talker mixtures must identify the attended speaker without prior target-speaker information, a challenge compounded by the difficulty of non-stationary noise.
Source: §1 · p. 1Method
Results
BASEN outperforms UBESD across SI-SDR, PESQ, and STOI for all tested subjects, with less subject-to-subject variation.
Source: §4 · p. 4Takeaways & Limitations
The authors present BASEN as a candidate for realistic applications where no prior information about the attended speaker is available.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Time-domain single-channel speech enhancement (SE) still remains challenging to extract the target speaker without any prior information on multi-talker conditions. It has been shown via auditory attention decoding that the brain activity of the listener contains the auditory information of the attended speaker. In this paper, we thus propose a novel time-domain brain-assisted SE network (BASEN) incorporating electroencephalography (EEG) signals recorded from the listener for extracting the target speaker from monaural speech mixtures. The proposed BASEN is based on the fully-convolutional time-domain audio separation network. In order to fully leverage the complementary information contained in the EEG signals, we further propose a convolutional multi-layer cross attention module to fuse the dual-branch features. Experimental results on a public dataset show that the proposed model outperforms the state-of-the-art method in several evaluation metrics. The reproducible code is available at https://github.com/jzhangU/Basen.git.
1. Introduction
Monaural enhancement in multi-talker settings struggles when noise is non-stationary and the target speaker is unknown. BASEN uses listener EEG to guide time-domain speech extraction, with cross-attention intended to fuse audio and brain signals more fully.
1. Introduction
- Conventional statistical monaural enhancement can perform well in stationary noise but drops rapidly in non-stationary conditions such as cocktail parties.The paper motivates monaural enhancement as a more flexible and economical configuration than microphone arrays [4].Source: §1 · p. 1
- Time-domain enhancement predicts the clean waveform directly from mixtures, jointly optimizing magnitude and phase; such methods are described as smaller and lower-latency than alternatives.Conv-TasNet uses an encoder, speaker-separation masks, and a decoder.Source: §1 · p. 1
- Without prior target-speaker information, estimating the attended speech in multi-talker mixtures remains challenging; cascaded separation and auditory attention decoding also require substantial computation.Auditory attention decoding accuracy falls sharply with shorter decision windows.Source: §1 · p. 1
- BISS and BESD/UBESD use listener brain activity for attended-speech enhancement, but mainly concatenate modalities, leaving cross-modal complementary information underused.BISS estimates a speech envelope from EEG before fusing it with the mixture; BESD and UBESD use dual-branch end-to-end schemes.Source: §1 · p. 1
- BASEN combines speech-mixture and EEG encoders with convolutional multi-layer cross attention (CMCA), then decodes the fused features into speech and outperforms UBESD on a public dataset.The approach is built on the Conv-TasNet backbone.Source: §1 · p. 2
2. Methodology
BASEN extends an end-to-end time-domain separation architecture with an EEG branch and CMCA to fuse brain and audio features. The separator predicts source-specific masks, which are applied to the audio embedding to reconstruct speech.
2. Methodology
- BASEN encodes the noisy speech and EEG trials separately, fuses their embeddings in the separator, predicts source-specific masks, and decodes the masked audio embedding into speech.For speech enhancement, the source count T is 2: target speech and non-target additive noise.Source: §2 · p. 2
- The audio encoder and decoder use 1D convolutions for downsampling and upsampling, while the EEG encoder uses stride-8 downsampling and eight residual depth-wise convolution layers.Each depth-wise block factors convolution into depth-wise and pointwise operations to reduce model size.Source: §2 · p. 2
- The separation network uses three depth-wise convolution stacks around CMCA; its dilations grow exponentially, and skip connections feed the final target-specific mask prediction.The mask path applies PReLU and a 1 × 1 convolution to resolve the target-specific mask.Source: §2 · p. 2
- CMCA couples audio and EEG streams through cross-attention layers, then sums layer-wise features, concatenates them with the original embeddings, and applies a 1D convolution for fused features.The layer updates use residual cross-attention followed by group normalization.Source: §2 · pp. 2–3
- Each cross-attention block derives query, key, and value features with depth-wise 1D convolutions; query-key correlations are softmaxed, multiplied by values, and passed through residual connections and group normalization.The key and value come from one sequence, while the query comes from the other.Source: §2 · p. 3
- Training uses negative SI-SDR as the loss and the Adam optimizer.SI-SDR scales the target before comparing it with the reconstructed speech, making the measure invariant to reconstruction scale.Source: §2 · p. 3
3. Experimental setup
The study uses a two-speaker listening task with EEG recordings to evaluate speech enhancement under a fixed 0 dB mixture condition. The setup includes EEG processing, neural-feature extraction, and objective evaluation metrics.
Dataset and participants
- Data from 32 of 33 participants were analyzed; the excluded participant’s data were omitted for poor quality.Participants were native English, normal-hearing speakers with no history of neurological disease.Source: §3 · p. 3
Listening task and recordings
- Participants heard two male-read stories, one per ear, and attended to one ear throughout 30 trials of 60 seconds each.They answered questions about both stories after each trial; recordings used a 128-channel EEG cap at 512 Hz and were downsampled to 128 Hz.Source: §3 · p. 3
Mixture and data split
- Mixtures combined equal-RMS target and interfering speech at 0 dB, without added background noise or reverberation.Five trials were assigned to testing, two to validation, and the remainder to training; training and validation used 2-second segments, while testing used 20-second segments.Source: §3 · p. 3
EEG processing
- EEG preprocessing included filtering, noisy-channel repair, mastoid rereferencing, and ICA; a frequency-band coupling model then extracted cortical multiunit neural activity.The gamma-band amplitude and delta-band phase were combined with coefficients of 0.5 each.Source: §3 · p. 3
Evaluation
- Evaluation used SI-SDR, PESQ, and STOI; Figure 3 identifies these metrics for unknown-attended-speaker denoising.Figure 2 compares BASEN with UBESD and marks medians, interquartile ranges, and 1.5-IQR whiskers.Source: §3 · p. 3, Figure 3 · p. 4, Figure 2 · p. 4
Training
- Training used Adam for about 60 epochs with batch size 8 and a cosine-annealing learning-rate schedule with linear warmup.The maximum learning rate was 2 × 10^-4, with a 5% warmup ratio.Source: §3 · p. 4
4. Experimental results
BASEN outperformed UBESD across the reported evaluation metrics and subjects, with less variation across subjects. The experiments also found benefits from CMCA feature fusion and selected three cross-attention layers.
Feature-fusion experiments
- CMCA improved performance over simple audio–EEG concatenation, particularly on PESQ; concatenation was comparable to Conv-TasNet supplied with attended-speaker information.Without attended-speaker information, audio-only Conv-TasNet did not improve speech quality in this setting.Source: §4 · p. 4
CMCA depth
- Three CMCA layers gave the best performance across all metrics among the tested depths of one to five layers.The model had 0.57M parameters at one layer and 0.64M at three layers.Source: §4 · p. 4
BASEN versus UBESD
- BASEN outperformed UBESD in SI-SDR, PESQ, and STOI across subjects, with less subject-to-subject variation.Figure 2 presents the comparison with medians and violin-plot spread; the authors relate the lower variation to robustness against listening dynamics.Source: §4 · p. 4
5. Conclusion
The conclusion presents BASEN as a time-domain speech-enhancement approach that uses listener EEG and CMCA to fuse audio and EEG embeddings. The authors report that EEG trials help extract the attended speaker without prior target-speaker information.
Conclusion
- BASEN adds listener EEG to time-domain speech enhancement and uses CMCA to fuse audio and EEG embeddings.The network is based on a Conv-TasNet backbone.Source: §5 · p. 4
- The authors report that EEG trials help extract the attended speaker without prior information about the target speaker.They describe BASEN as a potential candidate for applications lacking that prior information.Source: §5 · p. 4
- Additional results are deferred to a future journal version because of page limits.Source: §5 · p. 4