PaperlayerView exact revision on arXiv

Source-linked AI summary

NeuroCodec: EEG-Guided Latent-Domain Generative Model for High-Fidelity Target Speaker Extraction

Julian Silva, Jaliya Kodithuwakku, Navindu Sirimanna, Shaveen Herath, Wageesha N. Manamperi, Joshua P. Kulasingham

arXiv:2610.29815v1eess.AScs.LG

TL;DR

A key challenge for speech enhancement is not only isolating overlapping speech, but decoding the listener’s attention to identify which speaker to extract from the mixture. NeuroCodec predicts target speech in frozen audio-codec latent space, using graph-based EEG modeling and cross-modal attention with Mamba temporal modeling. NeuroCodec improves STOI from 0.812 (M3ANet) to 0.827 and PESQ from 2.23 to 2.31 on Cocktail Party, and from 0.859 to 0.868 STOI and 2.514 to 2.562 PESQ on KUL.

  • Problem

    A key challenge for speech enhancement is not only isolating overlapping speech, but decoding the listener’s attention to identify which speaker to extract from the mixture.

    Source: §I · p. 1

  • Method

    NeuroCodec predicts target speech in frozen audio-codec latent space, using graph-based EEG modeling and cross-modal attention with Mamba temporal modeling.

    Source: §I · p. 2, C. EEG Encoder · p. 3, D. Cross Modal Selective State Space… · p. 3

  • Results

    NeuroCodec improves STOI from 0.812 (M3ANet) to 0.827 and PESQ from 2.23 to 2.31 on Cocktail Party, and from 0.859 to 0.868 STOI and 2.514 to 2.562 PESQ on KUL.

    Source: A. Experimental Results · p. 5

  • Takeaways & Limitations

    NeuroCodec enables 44.1 kHz reconstruction with spectral content extending to approximately 20 kHz and is extended with a causal streaming formulation.

    Source: §I · p. 2, §VI · p. 7

Abstract

from arXiv · show

Driven by recent advancements in hearing assistive devices, target speaker extraction (TSE) in multi-talker environments has drawn significant attention. To address this, hearing technologies identify attention using neural signals recorded via electroencephalography (EEG) to guide TSE, enabling neuro-steered auditory attention decoding. However, most existing techniques rely on time-frequency masking, which suffers from low fidelity in spectro-temporally overlapping regions and is often restricted to narrowband audio. In contrast, we introduce NeuroCodec, which, to the best of our knowledge, is the first generative EEG-guided TSE framework operating in the latent space of a neural audio codec. Specifically, NeuroCodec models spatial EEG dependencies via graph convolutions and fuses cross-modal EEG and speech representations through attention mechanisms and Mamba-based temporal modeling. We further introduce a six-phase training curriculum that progressively incorporates latent reconstruction, multi-scale spectral, and adversarial objectives to ensure stable, high-fidelity speech generation. Experiments on the public Cocktail Party and KUL datasets show NeuroCodec outperforms baselines, across several perceptual metrics. A listening experiment with 20 participants validated that NeuroCodec offers greater perceptual accuracy, achieving a mean score of 82.24 compared with 49.63 for M3ANet. Additionally, the proposed method is seen to enable 44.1 kHz reconstruction with spectral content extending up to 20 kHz, further supporting the perceptual results. Finally, we extend NeuroCodec with a causal streaming formulation suited for real-time processing, providing a basis for hearing assistive devices.

I. INTRODUCTION

Neuro-steered TSE uses EEG to identify and extract the attended speaker, but existing systems rely on mixture-domain masks and have limited bandwidth. NeuroCodec predicts speech in a neural audio codec’s latent space, and the paper also introduces a causal streaming extension and evaluates the approach on two datasets.

I. INTRODUCTION

  • EEG can indicate the listener’s attended speaker without relying on external reference cues, while earlier AAD approaches detected attention without extracting speech.Integrated neuro-steered TSE frameworks use continuous EEG to reconstruct the attended waveform.Source: §I · p. 1
  • Offline neuro-steered TSE uses complete sequences, whereas online approaches use short causal windows; existing online systems generally perform below offline counterparts.This performance gap matters for hearing devices requiring continuous, low-latency operation.Source: §I · pp. 1–2
  • Existing EEG-guided TSE systems rely on mixture-domain mask prediction, which can retain interfering speech when target and interference overlap, and have been evaluated at limited sampling rates.The cited prior systems’ maximum output rates are 8 kHz or 14.7 kHz.Source: §I · p. 2
  • NeuroCodec predicts target speech in a frozen pretrained audio codec’s latent space rather than estimating time-frequency masks.The framework combines EEG and mixture-audio representations through cross-modal attention and Mamba-based temporal modeling.Source: §I · p. 2
  • The paper introduces OnlineNeuroCodec, a causal streaming extension that incorporates previous target-latent estimates through an auditory attractor for real-time extraction.Source: §I · p. 2
  • The evaluation covers subject-independent experiments on Cocktail Party and KUL, including objective and perceptual assessments, ablations, and streaming benchmarks.The reported evaluation also includes spectrogram analysis and a listening study.Source: §I · p. 2

II. NEUROCODEC FRAMEWORK

NeuroCodec maps a multi-speaker mixture and EEG signals to target-speaker codec latents, then decodes those latents into speech. A frozen DAC encoder provides compact mixture representations for subsequent EEG-audio fusion.

A. Model Overview

  • NeuroCodec maps mixture audio and multi-channel EEG to target-speaker latents, which a codec decoder converts into a speech waveform.The framework comprises audio and EEG encoders, a cross-modal selective state-space block, and a codec latent generator and decoder.Source: A. Model Overview · p. 2

B. Audio Encoder

  • A frozen pretrained DAC encoder maps the mixture waveform to compact latents with dimension D = 1024, reducing temporal resolution for later EEG-audio fusion.The encoder remains frozen to preserve its pretrained acoustic representation.Source: B. Audio Encoder · p. 2
  • The architecture illustration shows mixture latents and EEG embeddings entering stacked CM-S3 blocks, followed by target-latent prediction, quantization, and DAC waveform reconstruction.Source: Figure 1 · p. 3

C. EEG Encoder

The EEG encoder models electrode-space dependencies and temporal features; cross-modal blocks use acoustic queries to attend to EEG features before Mamba temporal modeling. Predicted target latents are decoded by the frozen DAC, with training objectives combining latent, spectral, and adversarial losses.

C. EEG Encoder

  • Third-order Chebyshev graph convolutions model spatial dependencies across EEG electrodes, followed by multi-kernel and temporal convolutions.The encoder operates at the 50 or 86 Hz codec latent rate, close to native 128 Hz EEG, avoiding initial upsampling.Source: C. EEG Encoder · p. 3

D. Cross Modal Selective State Space Block

  • Cross-modal attention uses acoustic tokens as queries and EEG features as keys and values, then Mamba models long-range temporal dependencies in codec-latent sequences.Mamba replaces self-attention with selective state-space modeling for linear-time temporal modeling.Source: D. Cross Modal Selective State Space… · p. 3

E. Codec Latent Generator and Decoder

F. Loss Functions

  • Training combines continuous-latent reconstruction, multi-resolution log-Mel spectral loss, and adversarial losses with feature matching.Spectral loss uses seven scales up to 20 kHz for Cocktail Party and four scales up to 4 kHz for KUL; MPD and MBMSD discriminators supervise waveform periodicity and multi-band spectral consistency.Source: F. Loss Functions · p. 3

G. Multi-phase Training Curriculum

A six-phase curriculum progressively adds spectral and adversarial objectives to stabilize training. OnlineNeuroCodec extends the framework to causal streaming, using current-window input and information from preceding windows.

G. Multi-phase Training Curriculum

  • A six-phase curriculum progressively introduces spectral and adversarial objectives to preserve training stability and mitigate mode collapse.Training moves from latent reconstruction, through Mel-loss ramp-up and discriminator warm-up, to full adversarial training.Source: G. Multi-phase Training Curriculum · p. 4
  • The six phases end with full adversarial training until validation performance plateaus, using a ReduceLROnPlateau scheduler.The scheduler has patience of 3 epochs and a reduction factor of 0.5.Source: G. Multi-phase Training Curriculum · p. 4

III. CAUSAL STREAMING EXTENSION OF NEUROCODEC

  • OnlineNeuroCodec restricts each prediction to the current input window and information accumulated from preceding windows.It preserves the offline formulation’s latent-domain processing.Source: §III · p. 4

A. Auditory Attractor via Running Speaker State

OnlineNeuroCodec carries a running latent representation of prior target estimates across hops and uses it to condition extraction for the attended speaker.

A. Auditory Attractor via Running Speaker State

  • The auditory attractor mean-pools the running state and projects it to FiLM scale and shift parameters that modulate the current mixture-latent query.The modulation occurs before EEG cross-attention.Source: A. Auditory Attractor via Running… · p. 4
  • A parameter-free EMA updates the running target-latent state from prior information and the latest hop estimate.The smoothing coefficient α is set to 0.7 based on validation performance.Source: A. Auditory Attractor via Running… · p. 4
  • Unlike NeuroHeed’s waveform re-encoding with a dedicated LSTM speaker encoder, OnlineNeuroCodec updates speaker state directly in codec latent space.The latent-space update avoids an additional recurrent speaker module.Source: A. Auditory Attractor via Running… · p. 4

B. Hopwise Training Strategy

Hopwise training applies reconstruction and spectral objectives to each streaming output, while stopping gradients through the running state between hops.

B. Hopwise Training Strategy

  • Training applies latent-domain reconstruction and decoded-segment spectral objectives to each output hop.The overall online objective averages the hop losses, with separate weights for reconstruction and Mel terms.Source: B. Hopwise Training Strategy · p. 4
  • Gradients are stopped at Zpast between consecutive hops, truncating backpropagation across hop boundaries while retaining the state for causal conditioning.Source: B. Hopwise Training Strategy · p. 4

IV. DATASETS AND EXPERIMENTS

The experiments use two public EEG-audio datasets and subject-independent evaluation, with dataset-specific audio rates, segmentation, and comparison settings.

A. Datasets

  • Cocktail Party includes 33 subjects with 128-channel EEG and 30 dichotic listening trials, while KUL includes 16 subjects and uses the first eight trials in these experiments.Both datasets involve attention to one of two concurrent narrative speech streams.Source: A. Datasets · p. 5

B. Implementation Details

  • Evaluation uses unseen test subjects; Cocktail Party uses a fixed 23/5/5 subject split and 2-second non-overlapping windows for comparison with prior models.Its native 44.1 kHz audio is encoded by pretrained 44.1 kHz DAC at an approximately 86 Hz latent frame rate.Source: B. Implementation Details · p. 5
  • KUL uses 16-fold subject-independent cross-validation and 4-second windows with a 1-second hop for comparison with the SI NeuroSpex+ setting.Its 8 kHz audio is upsampled to 16 kHz and encoded at a 50 Hz latent frame rate.Source: B. Implementation Details · p. 5

C. Evaluation Metrics

NeuroCodec improves selected objective metrics on both datasets, while qualitative spectrograms show wider-band reconstruction and cleaner separation in overlapping speech. Objective metrics were computed after resampling outputs to 8 kHz, and some models were omitted because they could not be re-evaluated under the same subject-independent setting.

C. Evaluation Metrics

  • STOI, ESTOI, and PESQ outputs were resampled to 8 kHz before objective evaluation to avoid giving NeuroCodec an inherent wider-bandwidth advantage.The resampling matched baselines that operate only at 8 kHz.Source: C. Evaluation Metrics · p. 5

A. Experimental Results

  • NeuroHeed+ and MSFNet were omitted because unavailable public code prevented re-evaluation under the same rigorous subject-independent setting.Source: A. Experimental Results · p. 5
  • STOI rose from 0.812 (M3ANet) to 0.827 and PESQ from 2.23 to 2.31 on Cocktail Party, while M3ANet retained a marginal ESTOI advantage.The authors report the highest STOI and PESQ values for NeuroCodec on this dataset.Source: Table I · p. 5, A. Experimental Results · p. 5
  • STOI increased from 0.859 (NeuroSpex+) to 0.868 and PESQ from 2.514 to 2.562 on KUL.The comparison uses prior-baseline metrics reported for the identical subject-independent configuration.Source: Table I · p. 5, A. Experimental Results · p. 5
  • NeuroCodec reconstructs spectral content up to 20 kHz with clearer harmonics, while M3ANet truncates high frequencies around 7 kHz and shows more leakage in overlap.The spectrogram comparison identifies preserved content above 7 kHz and reduced interference in overlapping-speech regions.Source: A. Experimental Results · pp. 5–6, Figure 2 · p. 6

B. Ablation Study

The ablations examine temporal modeling, EEG guidance, and EEG encoder components on KUL. Causal Mamba leads the tested sequence models, while noise substitution and component removal support the role of neural guidance and the encoder’s spatial and temporal features.

B. Ablation Study

  • The KUL ablations vary the fusion-block backbone, EEG encoder architecture, and training objectives.Source: B. Ablation Study · p. 6

1) Temporal modeling architecture:

  • Causal Mamba achieves the highest scores across all three metrics, outperforming a standard Transformer and a Snake-activation Transformer.The Snake-enhanced Transformer improves all three metrics over the standard Transformer.Source: 1) Temporal modeling architecture · p. 6

2) EEG encoder guidance and components:

3) Progressive training objectives:

The ablations favor progressively adding spectral and adversarial losses, while listening and streaming evaluations compare perceptual quality and extraction performance. Online results also quantify speed and context–latency tradeoffs.

3) Progressive training objectives:

  • PESQ 2.5623 and ESTOI 0.7544 are highest with the combined latent reconstruction, multi-scale log-Mel, and adversarial feature-matching objectives.Adding log-Mel loss improves perceptual quality over reconstruction alone, with further improvement after adding adversarial feature matching.Source: Table II · p. 6, 3) Progressive training objectives · p. 6

C. Perceptual Evaluation

  • 82.24 vs 49.63 mean MUSHRA scores for native NeuroCodec and M3ANet; bandwidth-matched NeuroCodec also scores 58.49 vs 49.63.The blinded listening test involved 20 participants and eight randomly selected 2-second samples from the Cocktail Party test set.Source: Figure 3 · p. 6, C. Perceptual Evaluation · pp. 6–7

D. OnlineNeuroCodec Evaluation

  • STOI rises from 0.612 to 0.803, ESTOI from 0.5716 to 0.7105, and PESQ from 1.290 to 1.662 for OnlineNeuroCodec versus NeuroHeed.NeuroHeed results came from a reimplementation because the public code provides only the offline model.Source: Table III · p. 7, D. OnlineNeuroCodec Evaluation · p. 7
  • OnlineNeuroCodec runs 2.1× faster than NeuroHeed without the attractor and 2.3× faster with it, while the attractor improves extraction accuracy.The corresponding RTFs are 0.13 versus 0.27 without the attractor and 0.18 versus 0.41 with it.Source: D. OnlineNeuroCodec Evaluation · p. 7
  • A 1.5-second buffered context with a 0.5-second hop is adopted: PESQ gains saturate beyond that context while RTF continues to rise.Increasing hop size lowers RTF but raises algorithmic latency.Source: Table IV · p. 7, D. OnlineNeuroCodec Evaluation · p. 7

VI. CONCLUSION

NeuroCodec predicts frozen audio-codec latents for EEG-guided target speaker extraction, improving reported objective and subjective speech quality. Its causal streaming extension performs well in the reported evaluation, but remains far from embedded hearing-aid deployment.

VI. CONCLUSION

  • NeuroCodec predicts latent representations of a frozen high-fidelity audio codec instead of estimating time-frequency masks.The framework combines graph-based EEG modeling, cross-modal attention, and Mamba-based temporal modeling.Source: §VI · p. 7
  • Experiments on Cocktail Party and KUL report improved intelligibility and perceptual quality; reconstruction reaches 44.1 kHz with spectral content to approximately 20 kHz.The conclusion also reports minimal speaker interference and a subjective listening study.Source: §VI · p. 7
  • OnlineNeuroCodec uses a causal auditory attractor from previous target estimates, but remains far from embedded hearing-aid deployment.Future work includes autoregressive codec-token prediction, reduced buffering, model compression, and low-power implementation.Source: §VI · p. 7

Related papers

Loading 2610.29815v1…