PaperlayerView exact revision on arXiv

Source-linked AI summary

EEG-informed attended speaker extraction from recorded speech mixtures with application in neuro-steered hearing prostheses

Simon Van Eyndhoven, Tom Francart, Alexander Bertrand

arXiv:1602.05702v4cs.SDcs.SYeess.SYstat.ML

TL;DR

Prior AAD paradigms rely on separate clean speech sources, which are unavailable in realistic hearing-prosthesis microphone mixtures –. This paper uses microphone-derived speech envelopes and EEG to select and filter the attended speaker, reporting satisfactory extraction and denoising even with substantial noise.

  • Problem

    Prior AAD paradigms use separate clean speech sources, whereas hearing prostheses provide noisy speech mixtures at their microphones –.

    Source: §I · p. 1

  • Method

    The pipeline estimates speech envelopes from microphone mixtures, uses EEG-based AAD to select the attended envelope, and feeds its VAD track to a multi-channel Wiener filter.

    Source: §I · p. 2, A. Noise reduction problem · p. 3

  • Results

    The AAD-informed multi-channel Wiener filter extracts and denoises the attended speaker with satisfactory output SNR, including under substantial noise.

    Source: §VI · p. 10, §VII · p. 12

  • Takeaways & Limitations

    EEG-informed attended-speaker extraction from hearing-prosthesis microphone recordings is feasible as a proof of concept, without clean speech sources after initial training.

    Source: §VI · p. 10, §VII · p. 12

    Whether high AAD accuracy with only 15 EEG channels persists when only speech mixtures are available remains to be investigated.

    Source: §VI · p. 11

Abstract

from arXiv · show

OBJECTIVE: We aim to extract and denoise the attended speaker in a noisy, two-speaker acoustic scenario, relying on microphone array recordings from a binaural hearing aid, which are complemented with electroencephalography (EEG) recordings to infer the speaker of interest. METHODS: In this study, we propose a modular processing flow that first extracts the two speech envelopes from the microphone recordings, then selects the attended speech envelope based on the EEG, and finally uses this envelope to inform a multi-channel speech separation and denoising algorithm. RESULTS: Strong suppression of interfering (unattended) speech and background noise is achieved, while the attended speech is preserved. Furthermore, EEG-based auditory attention detection (AAD) is shown to be robust to the use of noisy speech signals. CONCLUSIONS: Our results show that AAD-based speaker extraction from microphone array recordings is feasible and robust, even in noisy acoustic environments, and without access to the clean speech signals to perform EEG-based AAD. SIGNIFICANCE: Current research on AAD always assumes the availability of the clean speech signals, which limits the applicability in real settings. We have extended this research to detect the attended speaker even when only microphone recordings with noisy speech mixtures are available. This is an enabling ingredient for new brain-computer interfaces and effective filtering schemes in neuro-steered hearing prostheses. Here, we provide a first proof of concept for EEG-informed attended speaker extraction and denoising.

I. Introduction

Selecting the attended speaker remains difficult in noisy multi-talker scenes, and prior AAD methods rely on clean speech sources unavailable to hearing prostheses. The proposed system uses microphone mixtures and EEG to identify and enhance the attended speaker, with envelope separation supporting the filtering process.

I. Introduction

  • Prior AAD paradigms use separate clean speech sources, whereas hearing prostheses provide noisy microphone mixtures; reliable operation in these conditions is important for neuro-steered devices.Energy- or direction-based heuristics can fail to select the attended speaker in real-life scenarios.Source: §I · pp. 1–2
  • The study combines EEG-based attention detection with acoustic noise reduction to suppress interfering sources in noisy recordings of two simultaneously active speakers.The processing chain is intended to operate on hearing-prosthesis microphone recordings without clean speech as sidechannel information.Source: §I · p. 2

A. Noise reduction problem

  • Each microphone records a convolutive mixture of both speech sources and background noise, and a frequency-dependent K-channel filter estimates the attended source at a reference microphone.The filter is designed to minimize mean-squared error against the attended speech component.Source: A. Noise reduction problem · p. 2
  • The multi-channel Wiener filter requires both the attended speaker's active times and an attended-speaker decision, while microphone envelopes alone contain crosstalk that can cause VAD misdetections.The proposed intermediate non-negative blind source separation step estimates cleaner speech envelopes for AAD and VAD.Source: A. Noise reduction problem · p. 3

B. Auditory attention detection (AAD) problem

Auditory attention detection uses EEG to reconstruct a speech envelope and determine which of two speech sources a listener attends to. In this study, the decoder is trained with clean speech envelopes, while subsequent attention detection can use microphone mixtures.

B. Auditory attention detection (AAD) problem

  • The AAD decoder is trained by linear regression to reconstruct the attended speech envelope from EEG measurements across channels and poststimulus lags.The EEG-to-envelope mapping differs for attended and unattended speech.Source: B. Auditory attention detection (AAD)… · p. 3
  • AAD succeeds when the reconstructed envelope correlates more strongly with the attended than the unattended speech envelope, or rA > rU.The correlations are Pearson correlation coefficients.Source: B. Auditory attention detection (AAD)… · p. 3
  • The study assumes access only to microphone recordings for AAD, except during initial EEG-decoder training, which uses the separate clean speech envelopes.Classical least squares estimates the decoder weights; regularization is omitted in this study.Source: B. Auditory attention detection (AAD)… · p. 3

III. Algorithm pipeline

The proposed pipeline separates envelope estimation and attention detection from final multichannel filtering, using EEG to select the speech envelope that guides enhancement.

III. Algorithm pipeline

  • The system computes energy envelopes from microphone mixtures and applies M-NICA to estimate the original speech-source envelopes.These estimated envelopes are then provided to the AAD processing block.Source: §III · p. 3
  • EEG-based AAD identifies which estimated envelope belongs to the attended speaker, and VAD is performed on the estimated envelopes.The attended-speaker decision selects the VAD track used by the filter.Source: §III · p. 3
  • The multi-channel Wiener filter uses the selected VAD track to filter microphone mixtures and produce an enhanced speech signal.Source: §III · p. 3

A. Conversion to energy domain (ENV)

The envelope-extraction stage works with microphone energy signals, enabling source separation at a lower sampling rate than the original audio. It models source and microphone energies as an approximate linear mixture.

A. Conversion to energy domain (ENV)

  • Energy-domain source separation operates at a much lower sampling rate than the original microphone signals, and nonnegative energy signals permit real-time separation using second-order statistics.These properties make the approach computationally efficient for battery-powered hearing prostheses.Source: A. Conversion to energy domain (ENV) · pp. 3–4
  • Microphone energy envelopes are calculated by squaring and low-pass filtering each microphone signal, then averaging over a window of T samples.The resulting short-time energy estimates the microphone's real energy.Source: A. Conversion to energy domain (ENV) · p. 4
  • The model relates stacked microphone and source short-time energies through a mixing matrix, but is only an approximation for finite HRIR durations and finite averaging windows.The authors report that it remains useful for estimating the original speech envelopes.Source: A. Conversion to energy domain (ENV) · p. 4

B. Speech envelope extraction from mixtures (M-NICA)

M-NICA separates microphone speech envelopes using non-negativity, supplying estimates for identifying the attended source. The multi-channel Wiener filter (MWF) then estimates that source at a selected reference microphone while suppressing other acoustic components.

B. Speech envelope extraction from mixtures (M-NICA)

  • M-NICA demixes observed mixtures into separate nonnegative sources using non-negativity and an iterative decorrelation and subspace-projection procedure.It relies on second-order statistics and operates on low-rate envelopes, supporting the target application’s computational constraints.Source: B. Speech envelope extraction from… · p. 4
  • M-NICA assumes independent, nonnegative, well-grounded sources; the number of speech sources must also be known a priori.The head’s angle-dependent attenuation creates different spatial energy patterns across microphones, yielding a well-conditioned mixing matrix.Source: B. Speech envelope extraction from… · p. 4

C. Multi-channel Wiener filter (MWF)

  • The MWF estimates the attended speech as observed at a selected reference microphone, using speech and undesired-signal autocorrelation matrices.The reference selector identifies the corresponding column of the speech autocorrelation matrix.Source: C. Multi-channel Wiener filter (MWF) · p. 5
  • The attended-speech autocorrelation is estimated by subtracting interference-only statistics from speech-plus-interference statistics, with periods identified through voice activity detection.The implementation uses a generalized eigenvalue decomposition for a rank-1 approximation because it targets one speech source.Source: C. Multi-channel Wiener filter (MWF) · p. 5

D. Voice activity detection (VAD)

The selected speech envelope is thresholded to identify voice activity, and its accuracy matters because VAD errors affect MWF denoising and speaker extraction. The experiment evaluates this processing flow with synthesized microphone mixtures, background noise, and varied speaker positions.

D. Voice activity detection (VAD)

  • AAD selects a demixed envelope as attended, and samplewise thresholding of its segments produces the VAD track used for MWF statistics.Occasional selection of the unattended envelope can cause VAD errors that affect MWF denoising and speaker extraction.Source: D. Voice activity detection (VAD) · p. 5

IV. Experiment

  • The experiment applies envelope extraction, attended-speech VAD selection, MWF computation, and WOLA filtering; a no-demixing comparison manually selects the microphone envelopes most correlated with each speaker.The manually selected envelopes are described as a best-case option that cannot be implemented in practice.Source: §IV · p. 5

A. Microphone recordings

  • Microphone mixtures combine two speech sources with five uncorrelated multi-talker noise sources at five azimuths, each noise source having one-tenth a speech source’s long-term power.The noise sources were absent from the AAD stimuli; 12 representative setups vary the two speakers’ spatial angles.Source: A. Microphone recordings · p. 6

B. AAD experiment

The AAD evaluation uses EEG from a prior study and trains subject-specific decoders with clean attended-speech envelopes. It tests attention detection on 30-second frames using envelopes from demixed or microphone signals.

B. AAD experiment

  • EEG data came from 16 normal-hearing subjects, with competing narrated stories mixed at both ears and attention detected on 30-second frames in leave-one-frame-out cross-validation.The selected subset used stimuli filtered with in-the-ear HRIRs for sources at −90° and 90°.Source: B. AAD experiment · p. 6
  • Subject-specific decoders were trained using EEG and clean attended-speech envelopes; energy envelopes were converted to amplitude envelopes for AAD correlations.The decoder is estimated once over the training frames, rather than separately per frame as in; the study follows the approach in.Source: B. AAD experiment · p. 6
  • AAD correlated decoder output with either the two demixed envelopes or the two microphone envelopes most correlated with the speech sources.The microphone-envelope option represents best-case performance without envelope demixing.Source: B. AAD experiment · p. 6

C. Preprocessing and parameter selection

Preprocessing sets the envelope and EEG rates and uses AAD decisions to assemble a hybrid VAD track. M-NICA and MWF operate in batch mode for this proof of concept, so reported performance is an upper limit for an adaptive system.

C. Preprocessing and parameter selection

  • Speech fragments are normalized to equal energy, and sources and HRIRs are resampled to 16 kHz before convolution.The paper notes that a real device would need only one final decoder after training.Source: C. Preprocessing and parameter selection · p. 6
  • Energy envelopes are sampled at 20 Hz, while 64 EEG channels are filtered from 1 to 9.5 Hz and downsampled to 20 Hz; the maximum EEG lag is 250 ms.Envelope signals are square-rooted and filtered between 1 and 9.5 Hz for AAD.Source: C. Preprocessing and parameter selection · p. 7
  • AAD decisions select and concatenate 30-second segments from demixed- or microphone-envelope VAD tracks to form the hybrid MWF input.The implementation uses non-overlapping 30-second windows; a smaller shift could increase the AAD decision rate.Source: C. Preprocessing and parameter selection · p. 7
  • M-NICA and MWF use batch-mode statistics measured over each full signal, making the reported results upper limits for adaptive implementations.An adaptive implementation is necessary in practice but beyond the paper’s scope; the study uses 100 M-NICA iterations.Source: C. Preprocessing and parameter selection · p. 7

A. Performance measures

The section defines measures for envelope separation, EEG-based attention decoding, and speech enhancement. These measures compare candidate envelopes with the speech sources and evaluate the multi-channel Wiener filter (MWF) against the best microphone input.

A. Performance measures

  • ∆rHL measures envelope separation as the difference between the highest and lowest Pearson correlations with a speech envelope.The value is averaged over speech fragments and source combinations for each angular separation; higher values indicate better separation.Source: A. Performance measures · p. 7
  • AAD decoding succeeds when the reconstructed envelope correlates more with the envelope associated with attended speech than with the other candidate envelope.A candidate is associated with attended speech when its correlation with attended speech exceeds its correlation with unattended speech.Source: A. Performance measures · p. 7
  • MWF performance is evaluated by comparing its output SNR with the input SNR of the microphone having the highest input SNR.Because the number of analyzed speaker setups varies by angular separation, median SNR values are mostly considered.Source: A. Performance measures · pp. 7–8
  • Fig. 2 contrasts an original speech envelope with a microphone envelope and with the corresponding demixed envelope.The caption distinguishes the microphone envelope in green from the demixed envelope in red, with the original speech envelope shown in black.Source: Figure 2 · p. 8

B. Speech envelope demixing

M-NICA substantially improves separation of speech envelopes across source positions, although background noise slightly reduces the separation measure, especially for nearby speakers. Demixing reduces crosstalk but does not improve average AAD accuracy; the accuracy measure does not establish how well a selected envelope represents attended speech.

B. Speech envelope demixing

  • M-NICA substantially improves ∆rHL across relative source positions, while background noise slightly lowers it, especially for nearby speakers.Greater angular separation tends to increase ∆rHL; the authors attribute this to more distinct head-related filtering and a better-conditioned separation problem.Source: B. Speech envelope demixing · p. 8, Figure 3 · p. 8

C. AAD performance

  • Three speaker setups—two at 30° separation and one at 60°—have substantially lower AAD accuracy and ∆rHL than the others.Fig. 4 plots average decoding accuracy against ∆rHL for twelve setups, comparing microphone and demixed envelopes from the noise-free case.Source: C. AAD performance · p. 8, Figure 4 · p. 9
  • Despite reducing crosstalk, demixing does not increase average AAD accuracy; that measure records envelope selection, not whether the selected envelope is a useful attended-speech representation.M-NICA also supplies as many candidate envelopes as there are speakers, avoiding the need to select optimal microphones.Source: C. AAD performance · p. 8
  • With demixed envelopes, decoder-output correlations more closely resemble those for clean speech envelopes, while microphone-envelope correlations cluster around the diagonal.The demixed point cloud has higher variance, consistent with imperfect demixing.Source: C. AAD performance · p. 9
  • Across subjects, decoding accuracy ranges from 52% to 98%; microphone- and demixed-envelope accuracy is generally below the clean-speech ideal, and demixing does not change it overall.The authors caution that this measure does not establish whether the chosen envelope supports an accurate attended-speaker VAD track.Source: C. AAD performance · p. 9

D. Denoising and speech extraction performance

Demixed envelopes let EEG-informed auditory attention detection guide multi-channel Wiener filtering to extract the attended speaker. The approach improves output SNR over microphone-envelope-based processing, including with background noise, while future validation is needed on unified noisy-scenario data.

D. Denoising and speech extraction performance

VI. Discussion

  • The substantial input-to-output SNR difference shows that EEG-based attention detection can guide MWF denoising to extract the attended speaker from microphone recordings.The authors present this as a first proof of concept for EEG-informed noise reduction in neuro-steered hearing prostheses.Source: §VI · p. 10
  • M-NICA demixes low-rate energy envelopes using second-order statistics, avoiding costly convolutive ICA on raw microphone signals before MWF filtering.Demixing strongly improves denoising; microphone envelopes can still yield comparable AAD accuracy when one source is already dominant, but M-NICA avoids selecting microphones with such favorable envelopes.Source: §VI · p. 10

D. Denoising and speech extraction performance

  • Output SNR is significantly higher with demixed than microphone envelopes in both noise-free and noisy conditions.Noisy-condition SNRs are lower but remain satisfactory given sub-zero input SNR; five uncorrelated noise sources at 10% of speech-source energy lower input SNR by approximately 3 dB.Source: D. Denoising and speech extraction… · p. 10, Figure 8 · p. 11

VI. Discussion

  • Extraction remains satisfactory for close speakers, and VAD is insensitive to thresholds across a wide range; longer-than-30-second windows may improve AAD accuracy but reduce time resolution.The discussion also notes that more advanced envelope or attention-detection methods may further improve accuracy.Source: §VI · p. 11
  • Future work should validate AAD and enhancement together on noisy multi-speaker data across angles and test whether 15-channel AAD performance holds when only mixtures are available.The discussion also calls for assessing EEG collection with miniature devices in real-life settings.Source: §VI · p. 11

VII. Conclusion

The proposed EEG-informed pipeline extracts and denoises attended speech with promising results in a noisy two-speaker setting, even as AAD performance decreases. Beyond initial training, it uses hearing-prosthesis microphone recordings without clean speech sources.

VII. Conclusion

  • Despite decreased AAD performance, the AAD-informed MWF extracts and denoises the attended speaker with satisfactory output SNR in substantial noise.The reported promising results are for a two-speaker environment.Source: §VII · p. 12
  • Except for an initial training phase, the algorithm operates solely on hearing-prosthesis microphone recordings, without knowledge of clean speech sources.The technique is described as extensible to multi-speaker scenarios.Source: §VII · p. 12
  • The pipeline’s elementary components are computationally inexpensive and implementable in real time, making them attractive for battery-powered hearing prostheses with severe energy constraints.Source: §VII · p. 12
  • The study bridges AAD research using clean speech envelopes and attended-speech enhancement in more realistic settings without access to those clean envelopes.Source: §VII · p. 12
Loading 1602.05702v4…