PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Subject Generalization in Decoding Perceived Speech from Non-Invasive Brain Recordings

Aoke Zhang, Bo Wang, Xihong Wu, Heping Cheng, Jing Chen

arXiv:2608.22420v1cs.SDcs.AI

TL;DR

Existing cross-subject perceived speech decoding methods have limited generalizability and lack explicit mechanisms for extracting subject-consistent information, while requiring costly training procedures. CPSD addresses this gap through contrastive source pre-training, personal specialization, and PESA-based remapping of MEG/EEG data. Across three datasets, it outperforms baseline methods in Top-10 accuracy, with further analyses supporting its effectiveness, efficiency, and robustness.

  • Problem

    Existing cross-subject decoding methods have limited generalizability and lack explicit mechanisms for extracting subject-consistent information, resulting in high training costs and suboptimal decoding performance.

  • Method

    CPSD uses contrastive source model pre-training, personal specialization with target-subject data, and PESA to remap MEG/EEG data into a standardized reference space.

  • Results

    CPSD outperforms baseline methods across three datasets, achieving Top-10 accuracies of 61.3%, 43.0%, and 39.9%, respectively.

  • Takeaways & Limitations

    Analyses support the effectiveness, efficiency, and robustness of CPSD across modalities, languages, training settings, and evaluation conditions.

Abstract

from arXiv · show

Decoding perceived speech from non-invasive brain recordings has garnered significant attention in recent years due to its wide range of potential applications. However, existing methods face considerable challenges in cross-subject decoding, primarily due to limited generalizability and the absence of explicit mechanisms for extracting subject-consistent information. These limitations result in high training costs and suboptimal decoding performance. To address these challenges, we propose an innovative Cross-Subject Perceived Speech Decoding (CPSD) framework, which comprises two training stages: source model pre-training and personal specialization. In the source model pre-training stage, contrastive learning is employed to capture shared representations across multiple source subjects. Subsequently, personal specialization initializes the model for the target subject by extracting consistent components from the source model and fine-tuning it using target subject data. Additionally, we introduce the Positional Encoding-based Spatial Attention (PESA) module, which remaps MEG/EEG data into a standardized reference space, thereby enhancing cross-subject consistency and facilitating model training. We evaluate the proposed CPSD framework on three perceived speech neural datasets encompassing different modalities and languages. The results demonstrate that our framework outperforms baseline methods by more than 6.8%, 15.4%, and 15.8% in Top-10 accuracy on the Armeni 2022, PKUEEG 2025, and Broderick 2018 datasets, respectively. Further analyses confirm the effectiveness, efficiency, and robustness of the proposed approach.

I. INTRODUCTION

The paper introduces CPSD to address limited generalizability and subject inconsistency in cross-subject perceived speech decoding. It combines source model pre-training, personal specialization, and PESA, with evaluations and analyses supporting its performance, effectiveness, efficiency, and robustness.

  • Motivation: Cross-subject MEG/EEG decoding is hindered by low SNR, limited spatial resolution, and substantial variability in neural responses across subjects.Existing subject-layer approaches require simultaneous access to all subjects and retraining when new subjects are added.
  • Proposed Framework: PESA remaps data from different subjects into a standardized reference space to extract subject-consistent information and improve generalizability.The framework also uses subject-layer initialization to adapt consistent source components to individual target subjects.
  • Proposed Framework: CPSD is a cross-subject perceived speech decoding framework comprising source model pre-training and personal specialization.The framework is presented as specifically designed for perceived speech decoding.
  • Evaluation: The framework was evaluated on three MEG/EEG datasets spanning different modalities and languages, with consistent improvements over baseline methods across subjects.Ablation studies support the effectiveness of PESA and the personal specialization stage.
  • Evaluation: Extensive analyses support the proposed framework's effectiveness, efficiency, and robustness.The analyses include neural representation consistency, training time, zero-shot decoding, hyperparameter search, multi-subject comparisons, and PESA analysis.

II. RELATED WORK

Related work spans EEG classification, fMRI-based continuous perceived speech decoding, self-supervised transfer, adversarial subject-invariant representations, and contrastive learning across subjects. These approaches motivate transfer to target subjects while highlighting differences in task constraints, temporal resolution, and source-data requirements.

  • Perceived Speech Decoding: EEG category classification has achieved 61% accuracy on a binary task, but its strict experimental paradigm differs from natural daily language.The cited task is therefore distinct from continuous perceived speech decoding.
  • Perceived Speech Decoding: fMRI has been used for continuous perceived speech text generation, but its low temporal resolution is associated with generated texts having a high word error rate.
  • Cross-Subject Decoding: Self-supervised methods acquire transferable representations through time-frequency relationships or masked series modeling for cross-subject or cross-dataset transfer.
  • Cross-Subject Decoding: Adversarial optimization seeks subject-invariant representations, while contrastive methods maximize similarity between subjects exposed to the same stimulus.The contrastive approach targets MEG/EEG representations with higher subject consistency.
  • Problem Formulation: The cross-subject formulation uses source and target neural segments evoked by the same stimuli to fine-tune a source-pretrained model for target match-mismatch classification.Each subject can be iteratively selected as the target subject.

B. Model Overview

CPSD maps MEG/EEG data into a standardized reference space with positional encoding and spatial attention, then processes channel locations to construct latent representations. This design targets subject-consistent information and facilitates cross-subject model training.

  • The CPSD architecture maps input MEG/EEG data into a standardized reference space using the PESA module before convolutional processing.
  • 1) PESA: PESA uses positional encoding to provide unique position embeddings, emphasize relative offsets, and represent locations in the reference space.
  • 1) PESA: Sensor locations are obtained from MNE-Python and mapped by an MLP into a latent space with dimensions C × T.
  • 1) PESA: Similarity between mapped sensor locations and positional embeddings determines how input data contribute to different reference-space positions.
  • 1) PESA: The latent representation H is computed from the input X ∈ R^C×T after the spatial remapping operations.

2) Brain Encoder:

The brain encoder follows PESA and subject-layer processing to generate outputs aligned with wav2vec speech representations. CPSD trains this architecture through source-model pre-training and target-subject specialization.

  • 2) Brain Encoder:: The brain encoder uses five residual convolutional blocks followed by two convolutional layers to transform subject-layer outputs into final representations.
  • CPSD uses leave-one-subject-out training, treating all remaining subjects as source subjects and the held-out subject as the target.
  • Source-model pre-training feeds MEG/EEG segments, sensor locations, and subject IDs into the model and optimizes CLIP loss against wav2vec representations.
  • Personal specialization initializes the target subject layer and fine-tunes the resulting model on target-subject data.
  • The model outputs are aligned with wav2vec representations extracted from the corresponding speech stimuli.

1) Source Model Pre-Training:

Personal specialization adapts the pre-trained source model to a target subject by initializing its subject layer from components shared across source subjects. CorrCA identifies these components by maximizing inter-subject correlation.

  • 2) Personal Specialization:: CorrCA initializes the target subject layer from source-subject layers to capture information shared across subjects.
  • 2) Personal Specialization:: CorrCA optimizes a weight matrix to maximize correlation between two input subject data matrices.
  • 2) Personal Specialization:: The specialized model replaces subject layers with the average of consistent components across source subjects.
  • 2) Personal Specialization:: For target subject I, source subjects are indexed from 1 through I − 1, and model outputs are realigned with corresponding wav2vec representations.

D. Loss Function

The framework uses CLIP loss to align neural representations with speech features during training. The evaluation covers three speech-perception neural datasets spanning different modalities and languages.

  • CLIP loss is used during both source-model pre-training and personal specialization to align neural representations with corresponding speech features.
  • The loss maximizes similarity for positive neural–speech pairs while minimizing similarity for negative pairs.
  • The loss uses model outputs z_i, wav2vec representations y_i, batch size N, and a similarity function sim(x, y).
  • Evaluation uses the Armeni 2022, PKUEEG 2025, and Broderick 2018 neural datasets related to speech perception.

B. Data Preprocessing

The study preprocesses MEG/EEG data, controls data leakage through ordered splits, and evaluates decoding with Top-10 and rank accuracy.

  • Data Preprocessing: MEG/EEG trials were filtered, resampled to 100 Hz, artifact-cleaned with ICA, segmented into overlapping 3-second intervals, and normalized.Segments overlapped by 1.5 seconds and were normalized using RobustScaler.
  • Baseline Methods: Baseline comparisons included intra-subject and cross-subject methods, with hyperparameters taken from the original papers.BIOT, DAPE, and CL-CS were selected for cross-subject comparisons, while ATM-S, iTransformer, VLAAI, and Brainmagic represented intra-subject decoding.
  • Experimental Setup: Data were split into 70%, 10%, and 20% training, validation, and test sets by trial order, with non-overlapping stimuli across splits.The same partitioning method was applied to source and target subjects to avoid data leakage.
  • Evaluation Metrics: Top-10 accuracy measures whether the true segment appears among the ten most likely predictions, while rank accuracy evaluates the target sample’s predicted position.Both metrics are commonly used in speech decoding tasks for comparison with prior work.

F. Experimental Results

CPSD outperforms the evaluated cross-subject baselines across three datasets, and ablations attribute gains to PESA and personal specialization.

  • Main Results: 61.3%, 43.0%, and 39.9% Top-10 accuracy were achieved by CPSD on the three datasets, significantly exceeding cross-subject baselines.Pairwise t-tests found significant improvements for each dataset, with p < 0.001.
  • Main Results: Brainmagic performed best among the compared encoders when decoding exclusively from the target subject.This result motivated its selection as the brain encoder.
  • Ablation Study: PESA improved intra-subject decoding performance for the target subject in the ablation settings.The comparison included Base and Base+PESA configurations.
  • Ablation Study: The two-stage training approach significantly improved performance across all three datasets, while the complete CPSD framework achieved the highest decoding accuracies.The full framework combines PESA with personal specialization, and the reported improvements had p < 0.001.

3) Neural Representation Consistency:

CPSD’s subject-consistent representations are associated with decoding performance, while its specialization stage reduces training cost and supports limited zero-shot generalization.

  • Neural Representation Consistency: CPSD improved inter-subject correlation in nearly all evaluated representation settings compared with Brainmagic, with p < 0.001.ISC was computed using CorrCA for latent subject-layer representations and final outputs.
  • Neural Representation Consistency: corr = 0.66, p < 0.001 links model performance with neural representation consistency in Fig. 4.The Broderick 2018 dataset provided an additional correlation coefficient of 0.77.
  • Training Time Comparison: 7.5% and 15.6% of multi-subject training steps were required by specialization on PKUEEG 2025 and Broderick 2018, respectively.The training-step reduction was more pronounced for datasets with more subjects.
  • Zero-Shot Decoding: 44.7% (21/47) of subjects exceeded chance level in zero-shot decoding without personal specialization.The dataset-specific proportions were 33.3% (1/3), 48.0% (12/25), and 42.1% (8/19).

6) Hyperparameter Search:

The analyses examine CPSD’s stability, cross-subject performance, sensor-location dependence, and learned spatial weighting. Across these tests, the framework maintains high performance, preserves more performance than Brainmagic in cross-subject settings, and emphasizes speech-relevant temporal sensors.

  • Hyperparameter Search: CPSD is not sensitive to hyperparameter selection and maintains high performance under varying conditions on Broderick 2018.The tested hyperparameters are α, the number of MLP layers in PESA, and δ, the interval between adjacent position embeddings.
  • Comparison with Multi-Subject Settings: 1.7%, 2.5%, and 6.1% performance decreases separate CPSD’s cross-subject results from its multi-subject results across the three datasets.This degradation was significantly smaller than Brainmagic’s.
  • Comparison with Multi-Subject Settings: 10.3%, 6.4%, and 6.8% Top-10 accuracy improvements were obtained across the three datasets when CPSD and Brainmagic were applied to multi-subject decoding.Both models improved in the multi-subject setting through contrastive learning, but CPSD showed the smaller performance degradation relative to cross-subject decoding.
  • Sensor Location Validity Analysis: More than 11% Top-10 accuracy decreases followed sensor-location shuffling on both Armeni 2022 and Broderick 2018.The performance improvements of CPSD over the Shuffle setting were statistically significant at p < 0.01.
  • PESA Analysis: PESA assigns higher weights to sensors over bilateral temporal regions in perceived speech decoding.The channel-weight visualization is consistent with the role of these regions in auditory processing and indicates that PESA captures task-relevant brain regions.

V. CONCLUSION

CPSD combines source model pre-training with personal specialization and uses PESA to improve consistency across MEG/EEG subjects. Evaluations across three datasets support its effectiveness, robustness, and efficiency, while future work targets broader applications and improved zero-shot decoding.

  • V. CONCLUSION: CPSD uses source model pre-training and personal specialization to extract subject-consistent representations and adapt them to individual target subjects.Contrastive learning is used during pre-training, while consistent source-model components are adapted with target-subject data.
  • V. CONCLUSION: PESA remaps MEG/EEG data into a standardized reference space to improve neural-representation consistency, generalizability, and decoding performance.The framework was evaluated on three perceived speech neural datasets spanning different modalities and languages.
  • V. CONCLUSION: CPSD’s effectiveness, robustness, and efficiency were evaluated through ablations, hyperparameter searches, multi-subject and zero-shot decoding, consistency analysis, sensor-location tests, and training-time comparisons.PESA channel-weight visualizations also showed higher weights near the bilateral temporal lobes.
  • VI. FUTURE WORK: Future research will improve zero-shot decoding performance and explore additional applications, including transfer across experimental settings for non-invasive speech neuroprostheses.The paper links this direction to overlap between neural mechanisms underlying speech perception and speech production.
  • VI. FUTURE WORK: Perceived speech decoding remains insufficiently validated for tokenization-based cross-device training in brain foundation models.The paper proposes PESA’s channel-position-based dimensional decoupling as a feasible basis for unified-dimensional neural representations.
Loading 2608.22420v1…