Source-linked AI summary
The Semantic Bottleneck: Leveraging Semantic Representations for Non-Invasive Speech Decoding
Gilad D. Landau, Dulhan Jayalath, Oiwi Parker Jones
TL;DR
Non-invasive speech decoding is limited by low signal-to-noise ratios that hinder fine-grained phoneme and word recovery. Brain2Semantics2Text maps sentence-level MEG into a pretrained semantic embedding space and inverts the predicted embeddings into text, achieving improved sentence-level results while avoiding word-level alignment.
Problem
Low signal-to-noise ratios make fine-grained phoneme or individual-word reconstruction difficult, motivating targets based on higher-level semantic representations.
Method
Brain2Semantics2Text maps sentence-level MEG responses into a pretrained semantic embedding space and reconstructs text by inverting the predicted embeddings.
Results
The method performs favorably against acoustic-based sentence-level decoding, with better BLEU-1, ROUGE-1, and BERTScore results than BrainECHO.
Takeaways & Limitations
Contextual semantic decoding from MEG provides a sentence-level target that can complement lower-level speech information in non-invasive reconstruction.
Takeaways & Limitations
The learned semantic structure was strongly biased toward the study corpus, indicating a need for greater topic and concept variability in training data.
Abstract
from arXiv · showhide
Non-invasive speech decoding remains constrained by the low signal-to-noise ratio of neural recordings, which makes fine-grained reconstruction of phonemes or individual words difficult. Motivated by neuroscientific evidence that high-level semantic representations are distributed across cortical regions and evolve over slower temporal scales, we hypothesize that semantic content may provide a more suitable target for non-invasive decoding than low-level acoustic or lexical features. We introduce Brain2Semantics2Text, a method that reconstructs text through an intermediate semantic embedding space. Our model maps sentence-level MEG responses into a semantic manifold and then inverts the predicted embeddings into natural language. This semantic bottleneck enables recovery of high-level meaning without word-level alignment. We describe the core principles of the approach, its implementation, and the strategies used to mitigate the challenges of learning a reliable neural-to-semantic mapping. Finally, we compare against prior non-invasive Brain2Text methods and show improved sentence-level results.
1 Introduction
Brain2Semantics2Text addresses the low signal-to-noise ratio of non-invasive speech decoding by targeting distributed, slow-changing semantic representations rather than fine-grained linguistic units. It maps sentence-level MEG into a pretrained semantic space and reconstructs text through semantic embedding inversion, avoiding word-level alignment.
- Motivation: Non-invasive speech decoding is safer and more accessible than invasive approaches but remains difficult because neural recordings have lower signal-to-noise ratios.This limits reliable recovery of phonemes and individual words.
- Motivation: Higher-order semantic representations are distributed, redundant, and slower-changing, making them potentially better matched to non-invasive neural modalities.The approach is motivated by their spatial and temporal properties across the cortical hierarchy.
- Approach: Brain2Semantics2Text maps sentence-length MEG responses into a pretrained semantic embedding space and subsequently reconstructs text.The semantic bottleneck shifts decoding toward sentence-level meaning.
- Approach: Semantic embedding inversion frames speech decoding as semantic reconstruction rather than direct word or phoneme prediction.This supports reconstruction from semantic embeddings produced by the neural decoder.
- Significance: Sentence-level decoding avoids dependence on precise word-level alignment and closed-vocabulary supervision, while aligning with temporally extended high-level language representations.The method therefore targets compositional semantic representations near the top of the speech-processing hierarchy.
2 Related Work
Prior work spans semantic decoding from fMRI and MEG, acoustic-based MEG decoding, and closed-vocabulary word classification. Brain2Semantics2Text instead maps an entire sentence-length MEG response directly into a single sentence-level semantic embedding before text generation.
- Semantic decoding: Earlier studies mapped fMRI responses into semantic embedding spaces, while later work recovered semantic information from temporally resolved MEG recordings.The MEG approach reconstructed temporally aligned contextual word embeddings at the word level.
- Auditory and speech-based MEG decoding: Acoustic-based MEG systems align neural activity with speech or auditory representations for subsequent decoding.These methods focus on representations closely tied to the acoustic structure of speech.
- Auditory and speech-based MEG decoding: BrainECHO is the closest sentence-level comparison: it uses a vector-quantized audio-spectrogram latent space, whereas Brain2Semantics2Text uses a sentence-level semantic embedding.Both methods avoid word-level alignment and generate text from an intermediate representation.
- Word-level classification: Closed-vocabulary word classification aligns MEG with lexical semantic embeddings but relies on exact word-level timing and candidate-word classification.Brain2Semantics2Text instead targets a single compositional sentence representation.
3 Method
Brain2Semantics2Text maps sentence-level MEG responses into a semantic embedding space and reconstructs text through embedding inversion. Its training and architecture are designed to learn a geometrically faithful, expressive, reversible semantic bottleneck from limited, noisy data.
- Pipeline: Sentence-level MEG responses are mapped into a pretrained semantic embedding space, then inverted into natural language.The two-stage pipeline frames speech decoding as semantic reconstruction rather than direct word or phoneme prediction.
- Backbone: A spatial-attention module and 1 × 1 convolution project MEG sensors into latent features, followed by dilated temporal convolutions with residual connections and GLUs.These temporal blocks are intended to capture long-range dependencies.
- Temporal aggregation: Temporal self-attention and masked mean pooling convert time-resolved features into a fixed-dimensional semantic representation while excluding padded-duration effects.The temporal Transformer uses four attention heads, and the mask is based on true segment lengths.
- Semantic embeddings: The selected embedding space must balance expressivity, soft reversibility, low length bias, and learnability from limited MEG data.ADA was chosen for high soft reversibility, low length bias, and good expressivity; lower effective rank is preferred when these properties remain sufficient.
- Manifold learning objectives: Because contrastive loss alone may optimize retrieval without preserving global manifold geometry, auxiliary losses constrain alignment, covariance, variance, and embedding structure.The combined objective includes contrastive, global cosine alignment, inversion, variance, and covariance terms, with individual components tested by ablation.
- Embedding inversion: Embedding inversion iteratively re-embeds generated hypotheses and refines them using embedding discrepancy, progressively reducing distance to the target embedding.The inversion model reconstructs text without directly optimizing in discrete token space.
4 Experiments
Experiments evaluate sentence-level semantic decoding through text-generation metrics, signal-dependent uplift, semantic embedding inversion, and data scaling. Brain2Semantics2Text compares favorably with acoustic-based sentence-level decoding, while semantic inversion requires sufficient data and is assessed with retrieval when full reconstruction is unreliable.
- Evaluation: The study reports standard text-generation metrics including WER, BLEU, ROUGE, and BERTScore, with BERTScore complementing lexical-overlap measures through sentence-level semantic similarity.Because the method may generate more words than the ground truth, recall-oriented ROUGE-1 shows a distinct signal-based improvement whereas WER is less informative.
- Method performance: Brain2Semantics2Text outperforms BrainECHO on BLEU-1, ROUGE-1, and BERTScore in absolute performance and neural-signal uplift.The comparison is against acoustic-based sentence-level decoding; word-level decoding uses a more constrained setting with exact alignment.
- Semantic embedding inversion: Semantic inversion is evaluated through controlled Gaussian noise, using unperturbed reconstruction for expressivity and degradation smoothness for soft reversibility.When full inversion is unreliable below a performance threshold, retrieval-based Discounted Cumulative Gain is used as a proxy for identifying the closest sentence in embedding space.
- Signal uplift: 1.2-point BERTScore uplift exceeds BrainECHO’s sentence-level uplift but remains below d’Ascoli et al.’s 3.2-point uplift.The d’Ascoli comparison uses word-level decoding with exact word-aligned supervision.
- Signal uplift: 6.0-point ADA cosine-similarity uplift is the largest among the compared methods, exceeding the corresponding uplift of the word-level method.The discrepancy between BERTScore and ADA cosine similarity indicates that the metrics capture different aspects of sentence-level decoding.
- Scaling behavior: Performance generally improves with more training data, with gains appearing to saturate around 55.6 hours.The scaling results support the value of larger MEG datasets while suggesting that future gains may also require greater data diversity.
5 Limitations and Future Work
The study identifies limitations in corpus diversity, semantic inversion, and subject variability, and outlines corresponding directions for future work.
- Corpus diversity: The training corpus produced semantic structure strongly biased toward its specific topics and concepts.Future data collection should emphasize greater semantic variability.
- Semantic inversion: Semantic inversion was treated as a black-box component and may not be optimally suited to semantic decoding.Future work should examine soft reversibility and optimize preservation of semantic content.
- Subject variability: The study does not address subject variability in semantic representations.Future work may model shared semantic structure alongside participant-specific profiles.
6 Conclusion
The conclusion frames contextual semantic decoding from brain activity as progress toward fully non-invasive speech decoding. It proposes combining semantic and lower-level speech information to compensate for limited non-invasive signal quality.
- Progress toward non-invasive decoding: Decoding contextual semantic content from brain activity represents an important step toward fully non-invasive speech decoding.The paper still treats fully non-invasive speech decoding as a long-term goal.
- Complementary signals: Combining contextual semantics with lower-level speech information could provide multiple reconstruction signals despite limited non-invasive signal quality.The proposed complementary targets span fast acoustic and lexical features and slower semantic information.
A Impact Statement
The paper discusses potential assistive applications while emphasizing that clinical deployment remains distant and that neural decoding raises unresolved privacy and governance concerns.
- Potential applications: The approach has potential applications in brain-computer interfaces for people who have lost the ability to speak.The paper presents this as a potential application rather than a current clinical capability.
- Clinical scope: Clinical deployment remains distant because current performance is below the requirements of communication aids.The paper states that substantial further work is needed.
- Privacy and governance: Improved neural decoding will require norms concerning consent, data ownership, and the boundary between assistive and surveillant applications.The paper identifies privacy concerns but does not resolve these governance questions.
B Hyperparameters
Table 5 lists the hyperparameters used to train the MEG-to-semantic embedding model.
- Hyperparameters: Table 5 reports hyperparameters for MEG-to-semantic embedding training.
- Training scope: The table concerns the training stage that maps MEG signals toward semantic embeddings.
- Training procedure: The listed settings belong to the MEG-to-semantic embedding training procedure.
C Compute Resources
The study reports the compute resources and runtime used for MEG-to-semantic embedding models, alongside qualitative visualizations of reconstructed sentence embeddings and decoded sentences.
- C Compute Resources: Models used one NVIDIA GPU, four CPU cores, and 64 GiB of memory; a full-data run took approximately 16–18 GPU-hours.Each run required about 6 minutes per epoch for 150–170 epochs.
- C Compute Resources: Training across seven random seeds used approximately 110–130 GPU-hours, excluding exploratory runs, failed jobs, and downstream evaluation or decoding.
- C Compute Resources: PCA projections compare ground-truth and reconstructed sentence embeddings to illustrate preservation of relative positions and global structure.The examples are intended for geometric comparison, while textual reconstructions may differ from the ground truth.
- C Compute Resources: Qualitative decoding examples illustrate semantic similarity between target sentences and reconstructed outputs.
E Neural Signal Ablations
The neural signal analysis examines how different MEG signal properties affect semantic mapping performance and identifies which response aspects contribute most to decoding.
- E Neural Signal Ablations: The analysis evaluates how different MEG signal properties affect semantic mapping performance.It provides an interpretable view of which aspects of the neural response contribute most to decoding.
F Semantic Embedding Analysis
The embedding analysis compares semantic spaces using sentence-length behavior and broader diagnostic measures, highlighting differences between ADA and SONAR representations.
- F Semantic Embedding Analysis: SONAR embeddings segregate sentences by length, whereas ADA embeddings remain more invariant to sentence length.The comparison suggests that ADA is less vulnerable to the sentence-length confound.
- F Semantic Embedding Analysis: Candidate embedding spaces are compared using expressivity, soft reversibility, length bias, and intrinsic dimensionality.The appendix provides the full explanation of how these diagnostic measures are computed.