Source-linked AI summary
Subject-Invariant Cross-Modal Decoding of Perceived Speech from Brain Recordings
Aoke Zhang, Jing Chen
TL;DR
Non-invasive perceived speech decoding faces the joint challenge of extracting rich spatiotemporal representations and generalizing across subjects. SICMD integrates MEG and fMRI with subject-consistency processing to learn spatiotemporally rich neural representations for cross-subject decoding. SICMD exceeded baseline methods by 10.6%, 10.1%, and 1.7% across the reported metrics.
Problem
Non-invasive perceived speech decoding faces the joint challenge of extracting rich spatiotemporal representations and generalizing across subjects.
Source: §1 · p. 1Method
Results
SICMD exceeded baseline methods by 10.6%, 10.1%, and 1.7% across the reported metrics.
Source: §3.3.1 · p. 4Takeaways & Limitations
The reported analyses support SICMD's effectiveness and efficiency for subject-invariant cross-modal perceived speech decoding.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Perceived speech decoding based on non-invasive brain-computer interface (BCI) signals has been extensively studied in recent years. Research in this field primarily faces two challenges: extracting neural representations with rich spatiotemporal information and achieving cross-subject generalization. Although separate studies have proposed methods to cope with these issues, a unified approach that simultaneously tackles both challenges remains lacking. To fill this gap, we propose the Subject-Invariant Cross-Modal Perceived Speech Decoding (SICMD) method, which integrates functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG). We conduct comprehensive analyses of the fusion method, fusion position, encoder architecture, and model inputs. Our results demonstrate that the proposed method improves Top-1, Top-10, and Rankacc by more than 10.6%, 10.1%, and 1.7%, respectively, compared to baseline methods in cross-subject perceived speech decoding tasks, while reducing training costs by 88.8% and 60.5% compared to multi-subject and intra-subject decoding settings. Further visualization experiments also confirm the effectiveness of our approach.
1. INTRODUCTION
Perceived speech decoding with non-invasive signals must capture rich spatiotemporal information while generalizing across subjects. SICMD combines MEG and fMRI with cross-subject decoding to address both challenges.
1. INTRODUCTION
- Non-invasive perceived speech decoding faces two challenges: extracting rich spatiotemporal representations and achieving cross-subject generalization.MEG offers whole-brain, non-invasive recording but has low spatial resolution, while fMRI has high spatial but seconds-scale temporal resolution.Source: §1 · p. 1
- Cross-modal methods mainly target intra-subject decoding, while CPSD remaps subjects into a reference space but uses only MEG/EEG.Individual differences limit intra-subject models' generalizability; subject-consistent responses to the same stimuli are described as important for cross-subject transfer.Source: §1 · p. 1
- SICMD unifies cross-modal and cross-subject decoding to learn spatiotemporally rich neural representations with transferability to target subjects.The method integrates MEG and fMRI and analyzes fusion method, fusion position, encoder architecture, and model inputs.Source: §1 · pp. 1–2
2. METHODS
SICMD combines MEG and fMRI encoders with subject-consistency processing and modality fusion. Its pipeline pretrains on source subjects, initializes a target subject layer, and realigns neural and speech representations.
2.1. Model Overview
- The MEG encoder uses PESA, a 1 × 1 CNN layer, a subject layer, and a brain encoder; the fMRI encoder uses linear encoding modules.The model overview identifies these components for the two modality encoders.Source: §2.1 · p. 2
2.1.1. PESA
- PESA [13] remaps MEG signals from different subjects into a standardized reference space, processing sensor locations with an MLP and applying channel-to-position similarity weights to input data to produce hidden output.The hidden dimension D is set to 270 in this experiment.Source: §2.1.1 · p. 2
2.1.2. Brain Encoder
- The MEG brain encoder uses ConvBlocks to extract and fuse neural response patterns at multiple hierarchical levels.SICMD retains the ConvConcatNet design and parameters; its fMRI encoder uses three MLP encoding modules followed by a linear layer.Source: §2.1.2 · p. 2
2.2. Training Pipeline
- Evaluation uses leave-one-subject-out splits, with each subject selected in turn as target and the remaining subjects used as sources.The training pipeline is illustrated in Figure 1.Source: §2.2 · p. 3
2.2.1. Source Model Pre-training
- During source-model pretraining, MEG segments are remapped using sensor locations, processed with subject IDs, and encoded before outputs are aligned with wav2vec2 representations.The pipeline passes remapped segments through a 1×1 convolutional layer, subject layer, and MEG brain encoder.Source: §2.2.1 · p. 3
2.2.2. Personal Specialization
- For personal specialization, CorrCA extracts consistent components from source subject layers, which are averaged to initialize the target subject layer before neural representations are realigned with speech representations.CorrCA is used before target-subject fine-tuning.Source: §2.2.2 · p. 3
2.3. Modality Fusion
- SICMD uses the fMMF fusion method to combine asynchronous modalities, leveraging their differing information-change rates and predictive capacities across speech levels.Source: §2.3 · p. 3
3. EXPERIMENTS
SICMD was evaluated on multimodal recordings from 12 subjects, with trial-based data splits and Top-k accuracy and Rankacc measures. It achieved the strongest reported cross-subject decoding results, required fewer training steps for personal specialization, and showed gains across fusion, input, and channel-weight analyses.
3.1. Dataset & Preprocessing
- The dataset contains fMRI and MEG recordings from 12 subjects listening to the same continuous Chinese speech across 60 trials.The data were split by trial indices into 70% training, 10% validation, and 20% test sets, with no trial overlap.Source: §3.1 · p. 3, §3.2 · p. 3
3.2. Implementation Details
- Top-k accuracy (k = 1, 10) and Rankacc measured decoding performance, and CLIP loss was used for training.Experiments used batch size 128 and a single NVIDIA RTX 3090 GPU.Source: §3.2 · p. 3
3.3.1. Perceived Speech Decoding
- 43.5% Top-1, 82.7% Top-10, and 93.4% Rankacc were SICMD’s highest decoding scores, exceeding baselines by 10.6%, 10.1%, and 1.7%, respectively.All improvements were statistically significant (p < 0.001).Source: Table 1 · p. 3, §3.3.1 · pp. 3–4
- Personal specialization required 88.8% fewer training steps than multi-subject decoding and 60.5% fewer than intra-subject decoding.The respective average step counts were 629.3, 5,595.5, and 1,593.4.Source: §3.3.1 · p. 4
3.3.2. Ablation Study
- The Section 2.3 fusion method outperformed Random, Cross-Attention, and Multiply fusion methods (p < 0.001).The comparison is reported in Table 2.Source: Table 2 · p. 4, §3.3.2 · p. 4
- Fusion at brain-encoder outputs significantly improved performance compared with fusion at PESA, CNN, or subject-layer outputs (p < 0.001).The comparison is reported in Table 3.Source: Table 3 · p. 4, §3.3.2 · p. 4
- Shuffling sensor-location order reduced Top-10 accuracy by 19.0% (p < 0.001), while the fMRI-encoder layer ablation supported SICMD’s selected hyperparameters.Figure 2 presents the layer-count ablation on the left and the shuffled-location condition on the right.Source: §3.3.2 · p. 4, Figure 2 · p. 4
3.3.3. PESA Analysis
- PESA channel weights were higher near the bilateral temporal lobes, indicating that the model learned effective information.The channel weights were visualized using the method described in [13].Source: Figure 3 · p. 4, §3.3.3 · p. 4
4. CONCLUSION
SICMD fuses MEG and fMRI to produce spatiotemporally rich representations and extract information consistent across subjects, aiming to improve transferability and performance. The conclusion cites analyses of training time, ablations, and PESA weight-matrix visualizations as evidence of effectiveness and efficiency.
4. CONCLUSION
- SICMD fuses MEG and fMRI and extracts information consistent across subjects to enhance transferability and performance.The method is described as producing representations rich in spatiotemporal information.Source: §4 · p. 4
- Training-time comparisons, ablation studies, and visualizations of the PESA weight matrix demonstrate the method’s effectiveness and efficiency.Source: §4 · p. 4
- Future work aims to extend existing methods to EEG-fNIRS systems with improved applicability.Source: §4 · p. 4