PaperlayerView exact revision on arXiv

Source-linked AI summary

SENSE: Semantic Neural Speech Synthesis from Brain Dynamics via Spatial Graph Encoding

Jisoo Park, Seonghak Lee, Hyojin Park, Junseok Kwon

arXiv:2609.37601v3eess.AS

TL;DR

EEG-to-speech has focused on acoustic fidelity, leaving semantic preservation and electrode relationships underused. SENSE combines graph-based spatial encoding with semantic conditioning from congruent N400 trials; on N400 it improves acoustic and semantic metrics, including in unseen-subject evaluation.

  • Problem

    EEG-to-speech methods optimize waveform fidelity, while whether generated speech preserves semantic content remains an open question and electrode relationships and N400 congruency structure are underused.

    Source: §1 · p. 2

  • Method

    SENSE combines a graph-based EEG encoder over electrode geometry with EEG Semantic Conditioning, aligning EEG to a pretrained semantic space using congruent trials.

    Source: §1 · p. 2

  • Results

    On N400, SENSE outperforms prior methods on acoustic and semantic metrics; with two subjects it surpasses the strongest all-18-subject baseline in word error rate.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    Channel attribution spans auditory, sensorimotor, and centro-parietal regions, consistent with speech-perception neuroscience.

    Source: §5 · p. 10

    Reliable word-level reconstruction remains challenging, with WER values near 1; full waveform synthesis has only been evaluated on the controlled N400 corpus.

    Source: §5 · p. 10

Abstract

from arXiv · show

Reconstructing speech from non-invasive brain signals offers a promising pathway for restoring communication in individuals who are cognitively intact but unable to speak. Existing EEG-to-speech approaches formulate this task as acoustic reconstruction, optimizing waveform fidelity while ignoring whether the generated speech preserves high-level semantic content. In this work, we revisit this formulation and argue that EEG signals carry not only acoustic but also semantic information. We identify two key limitations of prior methods: (1) the neglect of spatial relationships between EEG electrodes, and (2) the failure to exploit the semantic structure of the N400 paradigm, where congruent and incongruent trials reflect distinct semantic processing. We propose SENSE(Semantic-EEG Neural Speech SynthEsis), which combines a graph-based EEG encoder over electrode geometry with EEG Semantic Conditioning (ESC), aligning EEG to a pretrained semantic space using only congruent trials. On the N400 dataset, SENSE consistently outperforms prior methods on both acoustic and semantic metrics, and model-internal channel attribution suggests distributed reliance on auditory, sensorimotor, and centro-parietal regions, consistent with known speech-perception neuroscience. In the unseen-subject setting, SENSE trained on only two subjects already surpasses the strongest baseline trained on all eighteen subjects in word error rate, and matches it on acoustic metrics with as few as eight subjects.

1 Introduction

EEG-to-speech has been framed as acoustic reconstruction, although EEG carries semantic information and the N400 corpus provides a congruency structure that prior methods do not exploit. SENSE combines electrode-geometry modeling with semantic alignment to address both gaps.

1 Introduction

  • EEG is a portable, low-cost, millisecond-resolution non-invasive modality for speech restoration, relevant to people unable to speak despite intact cognition.Invasive or high-fidelity alternatives such as ECoG and fMRI carry surgical or temporal-resolution limitations.Source: §1 · p. 1
  • Prior EEG-to-speech methods optimize acoustic fidelity, leaving open whether reconstructed speech preserves semantic content.Source: §1 · p. 2
  • The N400 corpus distinguishes congruent from incongruent sentence endings, but prior models treat both trial types identically across training objectives.The N400 is an index of lexical and conceptual semantic processing.Source: §1 · p. 2
  • SENSE aligns EEG representations with a pretrained CLIP space using congruent trials and models electrode geometry with a graph-based encoder.The sentence-final word varies in the N400 paradigm, concentrating semantic differences there.Source: §1 · p. 2
  • With only two subjects, SENSE surpasses the strongest baseline trained on all 18 subjects in word error rate.Source: §1 · p. 3

2 Related Work

Related work spans neural speech and language decoding, semantic representations in brain activity, and EEG representation learning. SENSE differs by generating waveforms while combining semantic alignment with graph-based spatial modeling.

Neural Speech and Language Decoding

  • Invasive intracortical and ECoG systems achieve strong speech-decoding results, while non-invasive EEG-to-speech waveform methods [13] [17] use acoustic objectives and serve as SENSE's N400 baselines.Source: §2 · p. 3

Semantic Representations from Brain Activity

  • Brain activity carries semantic structure, and prior work aligns neural signals with pretrained embedding spaces for retrieval or identification; SENSE uses alignment to condition speech generation.Source: §2 · p. 3

EEG Representation Learning

  • EEG-to-speech encoders typically treat channels as feature dimensions, whereas graph-based electrode modeling has mainly been used for classification; SENSE applies it to generation.Source: §2 · p. 3

3 Method

SENSE combines a graph-based EEG encoder with phoneme supervision and semantic conditioning aligned to a pretrained space on congruent trials. At inference, EEG-derived representations condition the speech decoder without text.

3 Method

  • SENSE addresses unordered EEG channels with a graph-based encoder and handles N400 trial structure by restricting semantic alignment to congruent trials.Its latent is jointly shaped by reconstruction, phoneme, and semantic objectives.Source: §3 · p. 4, §3.1 · p. 4

3.1 Structure-Aware EEG Encoder

  • The spatial encoder gates channels, applies graph convolution over electrode geometry, and pairs it with a CNN skip branch to retain channel-specific interactions.The adjacency is fixed from anatomy because jointly learning it with only 7,200 training pairs risks overfitting.Source: §3.1 · p. 4

3.2 Phoneme Sequence Supervision

  • A training-only phoneme predictor uses CTC against transcript-tokenized phoneme sequences to encourage fine-grained phonetic structure in M.The predictor uses a Conformer block and LSTM attention decoder over a 51-symbol IPA vocabulary, then is discarded at inference.Source: §3.2 · p. 5

3.3 EEG Semantic Conditioning

  • ESC aligns EEG-derived conditioning to frozen CLIP text embeddings on congruent trials and trains the decoder on EEG, text, or null conditioning.Incongruent trials still contribute to acoustic and phonemic learning, but not semantic alignment.Source: §3.3 · p. 5

3.4 Training Objective and Inference

  • Training combines acoustic VITS, EEG reconstruction, phoneme, and semantic losses; inference uses EEG-derived conditioning and the VITS prior without text.The EEG decoder, phoneme predictor, and CLIP encoder are discarded at inference.Source: §3.4 · p. 6

4 Experiments

SENSE outperforms both baselines across the reported splits and metrics, with the largest gains on unseen subjects. Cross-subject results, ablations, and channel-attribution analyses examine the model’s generalization, components, and internal patterns.

4 Experiments

  • The N400 evaluation uses 7,200 training pairs from 18 subjects and tests unseen audio, unseen subjects, and unseen subjects with held-out sentences.Acoustic metrics use all trials, while semantic metrics use congruent trials only; results average five seeds.Source: §4 · p. 6

4.1 Main Results

  • SENSE outperforms both baselines across all splits and metrics, with its largest gains on the unseen-subject split.Absolute WER values near 1 indicate that intelligibility-level reconstruction remains challenging; Figure 3 compares generated mel-spectrograms and transcriptions.Source: §4.1 · p. 7, Figure 3 · p. 7

4.2 Cross-Subject Generalization

  • With two training subjects, SENSE already surpasses FE-Phoneme’s 18-subject WER; by eight subjects, it matches or exceeds that baseline on MCD, Mel-Correlation, and STOI.The fixed held-out pair is sub-23 and sub-24; varying held-out pairs also shows SENSE outperforming both baselines on every metric and subject.Source: §4.2 · pp. 7–8, Figure 4 · p. 7, Figure 5 · p. 7

4.3 Ablation Study

  • Removing ESC degrades semantic metrics, removing CTC increases WER more, and spatial components provide gains; shuffled CLIP targets perform around the no-ESC baseline.Adding ESC to baseline encoders improves them, but they remain below SENSE.Source: §4.3 · p. 8, Table 2 · p. 8, Table 3 · p. 8
  • The BioSemi anatomy-based adjacency outperforms learned, fully connected, and random alternatives; adding the CNN skip further improves performance.The reported comparison supports local distance-based connectivity as the relevant inductive bias.Source: §4.3 · p. 8, Table 4 · p. 8
  • CLIP outperforms CLAP and SBERT on all metrics as the ESC alignment target, with Figure 6 showing its brightest diagonal and largest diagonal-to-off-diagonal gap.The authors suggest CLIP’s image–text pretraining may better distinguish concrete, image-able N400 nouns.Source: Table 5 · p. 8, §4.3 · p. 9, Figure 6 · p. 9

4.4 Channel Attribution Analysis

  • SENSE’s ESC attribution spans temporal-auditory, motor, and centro-parietal regions, while a GNN without a skip shifts emphasis toward central hub electrodes.The authors treat gradient maps as model-internal diagnostics, not evidence of neural sources; the full pattern is consistent with known speech-perception findings.Source: §4.4 · p. 9, Figure 7 · p. 9

5 Conclusions

The conclusion presents SENSE as an acoustic-semantic EEG-to-speech approach with cross-subject gains, while noting limits in word-level reconstruction and evaluation scope. The broader-impact discussion emphasizes protections for neural data and deployment oversight.

5 Conclusions

  • On N400, SENSE outperforms prior methods on acoustic and semantic metrics, but WER near 1 leaves reliable word-level reconstruction challenging.Full waveform synthesis has been evaluated only on the controlled N400 corpus; channel attribution is model-internal rather than neural-source localization.Source: §5 · p. 10

5.1 Broader Impact

  • The study uses a previously released benchmark and should not be interpreted as unrestricted thought decoding; future applications warrant neural-data protections and deployment-specific oversight.The passage identifies on-device processing, explicit opt-in consent, access restrictions, and governance as possible measures.Source: §5.1 · p. 10

A Implementation Details

The appendix identifies implementation details omitted from the main method section and states that implementation materials and checkpoints are supplementary or planned for release.

A Implementation Details

  • The appendix collects omitted preprocessing, architecture, and training details; implementation materials are supplementary, with checkpoints planned for release after de-anonymization upon acceptance.Source: A Implementation Details · p. 15

A.1 Dataset Preprocessing

The dataset preprocessing section describes EEG filtering, artifact removal, resampling, and trial segmentation, alongside speech downsampling and mel-target extraction.

A.1 Dataset Preprocessing

  • EEG preprocessing filters 128-channel recordings, removes EOG-correlated ICA components, resamples from 512 to 256 Hz, and segments trials at sentence onsets with zero-padding as needed.Speech is downsampled to 22,050 Hz, and 80-band mel targets are extracted using the stated VITS configuration.Source: §A.1 · p. 15
  • Following prior work, four subjects with documented EEG-quality or behavioral issues are excluded, leaving 20 subjects, with 18 used for training and validation.Source: §A.1 · p. 15

A.2 Structure-Aware EEG Module

The EEG module encodes electrode geometry with a graph network, combines it with a gated CNN branch, and applies temporal modeling. A training-only EEG decoder and a connector to the VITS prior complete the pathway.

A.2 Structure-Aware EEG Module

  • The learnable channel gate starts with higher weights on six auditory-related electrodes than on the remaining channels.The six channels are T7, T8, FT7, FT8, TP7, and TP8; their initial weights are +2.0 versus 0.0 for other channels.Source: §A.2 · p. 15
  • Electrode positions from the BioSemi montage define a fixed, row-normalized adjacency matrix with a 4 cm threshold.The adjacency is registered as a buffer and is not updated during training.Source: §A.2 · p. 15
  • Two residual graph-convolution layers process 64-dimensional electrode features before node averaging and projection to the 192-dimensional latent space.A shared Linear-GELU block first projects each electrode’s scalar activation to 64 dimensions.Source: §A.2 · p. 15
  • A two-layer CNN skip branch maps 128 EEG channels to 192 dimensions, matches the GNN output’s empirical standard deviation, and uses a learnable mixing gate.The mixing gate starts at −2.0, corresponding to an initial sigmoid value of about 0.12.Source: §A.2 · pp. 15–16
  • Temporal encoding downsamples by a factor of 3 before four bidirectional S4 blocks process the 192-dimensional sequence.The S4 blocks use state dimension 64 and dropout 0.1.Source: §A.2 · p. 16
  • A training-only EEG decoder reconstructs EEG, while a six-layer attention connector projects the latent sequence to VITS-prior parameters with a stop-gradient at its input.The EEG decoder is discarded at inference; the stop-gradient isolates the EEG Module from speech-pathway gradients.Source: §A.2 · p. 16

A.3 Phoneme Sequence Supervision

Phoneme sequence supervision uses an autoregressive attention decoder and a CTC objective over phoneme alignments. The predictor follows the FE-Phoneme architecture and is discarded at inference.

A.3 Phoneme Sequence Supervision

  • The phoneme predictor uses one Conformer block followed by an LSTM attention decoder that predicts autoregressively over a 51-symbol vocabulary.The decoder uses location-based attention and has a maximum decode length of 50.Source: §A.3 · p. 16
  • The CTC objective marginalizes over alignments that map the predicted sequence of logit vectors to the target phoneme sequence.The standard CTC reduction maps alignments to the target sequence.Source: §A.3 · p. 16
  • The predictor is discarded at inference, and training uses PyTorch CTCLoss with blank 0 and actual phoneme sequence lengths.The lengths exclude the SOS token.Source: §A.3 · p. 16

A.4 EEG Semantic Conditioning

EEG Semantic Conditioning aligns a pooled EEG representation with pretrained CLIP text embeddings while using multiple conditioning sources during training. At inference, EEG-derived conditioning replaces text input.

A.4 EEG Semantic Conditioning

  • Frozen CLIP ViT-B/32 produces cached 512-dimensional text embeddings for dataset sentences.The CLIP encoder is not loaded during training.Source: §A.4 · p. 16
  • Separate learned projections map CLIP and EEG embeddings to 192 dimensions, with the EEG projection trained only by the ESC alignment loss.Gradients from the conditioning bridge are blocked at the EEG projection.Source: §A.4 · p. 16
  • The EEG conditioning vector is formed by a length-masked temporal mean of the EEG representation followed by a learned projection.The mask excludes zero-padded frames based on each sample’s actual EEG length.Source: §A.4 · p. 16
  • During training, the decoder receives EEG conditioning, CLIP text conditioning, or a null embedding; at inference it uses EEG conditioning without text guidance.The three training sources occur at approximately 33%, 33.5%, and 33.5%, respectively; inference sets CFG scale to 0.Source: §A.4 · p. 17

A.5 Training Configuration

The appendix specifies optimization, training duration, evaluation, and reproducibility settings. A matched-budget comparison reports that SENSE remains better than both baselines across four reconstruction metrics on the unseen-subject split.

A.5 Training Configuration

  • Two AdamW optimizers train the EEG module and phoneme predictor separately from the speech pathway, with a stop-gradient between them.Both use a learning rate of 2 × 10^-4 and exponential learning-rate decay.Source: §A.5 · p. 17
  • Table A.1 summarizes the loss-weight values used in the total training objective.The supplied passage does not specify the individual values.Source: §A.5 · p. 17, Table A.1 · p. 17
  • SENSE trains for 900 epochs and approximately 200,000 gradient steps on two NVIDIA RTX 3090 GPUs, with about two days of wall-clock training.The checkpoint evaluated at step 200,000 is selected after validation performance plateaus.Source: §A.5 · p. 17
  • SENSE remains better than both baselines across all four reconstruction metrics on the unseen-subject split after 100k training iterations.The comparison uses a matched training budget.Source: Table A.2 · p. 17, §A.5 · p. 17
  • Cross-subject scaling uses nested training subsets of 2, 4, 8, and 18 subjects while holding out subjects sub-23 and sub-24.All scaling experiments use the same configuration and checkpoint-selection protocol.Source: §A.5 · p. 18
  • Reported results average five random seeds, with standard deviations included when shown, and CUDA deterministic mode enabled.PyTorch, NumPy, and Python random generators are initialized identically for each seed.Source: §A.5 · p. 18

A.6 Evaluation Protocol

The evaluation protocol computes acoustic, semantic, and intelligibility metrics using specified models, preprocessing, and alignment procedures.

A.6 Evaluation Protocol

  • WER uses Whisper-large-v3 with temperature 0 and beam size 5; transcripts are normalized before comparison with ground truth.Normalization lowercases text, removes punctuation other than apostrophes, and collapses repeated whitespace.Source: §A.6 · p. 18
  • BERTScore uses roberta-large with baseline rescaling, and reports the F1 component.Source: §A.6 · p. 18
  • CLIP-Sim is the cosine similarity between CLIP ViT-B/32 text embeddings of the Whisper transcription and ground-truth sentence.Source: §A.6 · p. 18
  • MCD uses the first 25 mel-cepstral coefficients excluding coefficient 0 with dynamic time warping; Mel-Correlation is Pearson correlation after length truncation.Source: §A.6 · p. 18
  • STOI uses the pystoi implementation at 16 kHz, with both signals resampled from 22,050 Hz.Source: §A.6 · p. 18

A.7 Symbol Reference

This section presents supplementary analyses of EEG–text alignment, semantic conditioning, and transfer, and qualifies the findings with reconstruction limitations and checklist disclosures on experimental variability, compute, ethics, impact, release, and human subjects.

A.9.1 Trial-Specific EEG–Text Correspondence; A.9.2 Sentence Retrieval

  • Matched EEG–text similarity was significantly higher for all trials (p < 10−79) and congruent trials (p < 10−45), while sentence retrieval was above chance at every evaluated cutoff.Retrieval accuracy remained modest, providing evidence of sentence-specific correspondence rather than alignment only to a shared text-embedding distribution.Source: Table A.4 · p. 19, §A.9.1 · p. 19, Table A.5 · p. 19

A.9.3 Matched and Mismatched Semantic Conditioning

  • Matched conditioning produces the highest BERTScore, followed by EEG-derived and mismatched conditioning, but differences are small and acoustic metrics do not consistently separate the latter two.The findings indicate only a modest effect of conditioning identity on sentence-level semantic similarity and do not establish reliable recovery of sentence meaning in generated speech.Source: §A.9.3 · pp. 19–20, Table A.6 · p. 20

A.9.4 Training-Time and Inference-Time Effects

  • ESC has complementary effects: EEG-derived conditioning modestly improves semantic similarity at inference, while ESC training retains an advantage even under null conditioning.The authors attribute the training-time effect to ESC backpropagating through the semantic projection and EEG encoder; current reconstruction quality does not support reliable semantic speech decoding.Source: Table A.7 · p. 20, §A.9.4 · p. 20

A.10.1 ESC Without Congruency-Based Selection

  • Applying ESC to all trials improves all five metrics over removing ESC, while restricting supervision to congruent trials yields further improvements.This suggests congruency acts as a dataset-specific supervision filter, not an architectural requirement.Source: Table A.8 · p. 20, §A.10.1 · p. 21

A.11 Channel Attention Gate Visualization; A.12 Attribution Trajectory During Training

  • The channel gate retains auditory-prior activity while increasing central and frontal-central weights and suppressing occipital channels; attribution shifts toward centro-parietal and frontal-central regions during training.The largest reported gate gains are at C3 and C4 (∆σ ≈ +0.27 and +0.29).Source: §A.10.1 · p. 21, Figure A.2 · p. 22, §A.12 · p. 22

A.10.2 Transfer to Continuous Natural Speech

  • Across all 19 participants, matched EEG–text similarity exceeded randomly mismatched similarity (p < 2.0 × 10−4), demonstrating representation-level transfer to continuous listened-speech EEG.The N400-pretrained encoder and ESC objective adapted without congruency labels or artificially segmented sentences; aligned waveforms were unavailable, so speech reconstruction was not evaluated.Source: Table A.9 · p. 21, §A.10.2 · p. 21

A.14 Per-Subject Results

  • For cross-subject generalization, models were retrained from scratch on the remaining 18 subjects for each held-out pair, and results are reported per subject.Source: Table A.11 · p. 23

NeurIPS Paper Checklist

  • Main results report mean ± standard deviation over 5 random seeds, with variability measured across seeds under fixed train/test splits.Source: NeurIPS Paper Checklist · p. 32
  • Training one SENSE model takes approximately 2 days on 2 NVIDIA RTX 3090 GPUs, using FP32 precision and batch size 16 per GPU.Source: NeurIPS Paper Checklist · p. 32
  • The authors state that the research conforms to the NeurIPS Code of Ethics, uses a public EEG dataset, and involves no new human-subject data collection.They report that the original N400 dataset collection received institutional approval.Source: NeurIPS Paper Checklist · pp. 32, 34, 35
  • The checklist says broader impact is discussed, while safeguards are marked not applicable because the work does not release pretrained models or scraped datasets posing high misuse risk.Source: NeurIPS Paper Checklist · p. 33
  • The paper credits existing assets and reports their licenses, including CC BY-NC-ND 4.0 for the N400 corpus and MIT or Apache 2.0 for the cited implementations.The released code is under MIT.Source: NeurIPS Paper Checklist · p. 34
  • Anonymized supplementary code includes the model implementation, configuration files, training and inference scripts, README, and LICENSE; pretrained checkpoints are omitted until de-anonymized release.LLMs are marked not applicable as a core methodological component.Source: NeurIPS Paper Checklist · pp. 34–35

Related papers

Loading 2609.37601v3…