PaperlayerView exact revision on arXiv

Source-linked AI summary

Interpretable MEG Decoding of Perceived Speech: Cortical Sources and the Stimulus Features That Drive Retrieval

Ilia Semenkov, Daria Kleeva, Ivan Dakhtin, Zarina Maksudova, Alex Ossadtchi

arXiv:2608.01481v1cs.LGcs.SDq-bio.NC

TL;DR

Deep MEG speech-retrieval models remain difficult to interpret, leaving unclear which neural and stimulus features support retrieval. This paper redesigns the decoder for source-space interpretability and tests feature use through input interventions, achieving competitive retrieval while identifying sources and speech properties that contribute to decoding.

  • Problem

    MEG speech-retrieval weights do not map to named electrophysiological quantities, and the speech properties driving retrieval remain unclear.

  • Method

    The authors build a compact source-mappable decoder with spherical-harmonic spatial attention, matched spatial-temporal branches, and paired MEG feature-present/absent interventions.

  • Results

    39.75 ± 0.34% Top-1 accuracy among 1005 candidates was achieved across six seeds with approximately twenty times fewer decoder parameters, while source mapping and interventions identified retrieval-related neural and stimulus features.

  • Takeaways & Limitations

    Retrieval accuracy reflects combined information from localized speech-related MEG dynamics and selected acoustic, phonetic, and contextual features rather than a direct readout of linguistic content.

  • Takeaways & Limitations

    The study uses one 27-participant MEG corpus, so its feature-use pattern is not established for new narratives, architectures, corpora, or evaluation protocols.

Abstract

from arXiv · show

Short segments of perceived speech can be retrieved from non-invasive magnetoencephalographic (MEG) recordings by deep networks trained with a CLIP-style objective against wav2vec 2.0 audio embeddings. Yet their weights do not map onto electrophysiological quantities, and it remains unclear which speech properties drive retrieval. We build on a high-performing MEG-to-audio retrieval architecture but redesign both its front end and decoder. Its spatial attention operates on a flattened sensor layout; we replace it with spherical harmonics defined on the three-dimensional MEG helmet geometry. We reduce the subject-specific representation from 270 to 25 branches, add a temporal filter to each branch to match it to a neuronal source in space and time, and make the convolutional decoder shallower. Ocular and cardiac components are removed before training to reduce the risk of stimulus-locked shortcuts. On MEG-MASC, the model reaches 39.75 +/- 0.34% Top-1 accuracy among 1005 candidates across six trained solutions, with about 20 times fewer decoder parameters. Its weights map to source space, recovering generators consistent with the speech-perception network, while left-lateralized branches carry higher-frequency rhythmic components not evident on the right. Paired MEG occlusion shows that 15 of 19 stimulus features contribute, with the largest effects for silence, sound intensity, vowels, and acoustic onsets. Random word lists behave oppositely: substituting narrative MEG into them improves retrieval, indicating that activity without narrative structure carries less recoverable information than activity during coherent speech. The wav2vec target can be reduced to about twelve learned feature dimensions without loss of accuracy, whereas strong temporal compression causes a clear loss. Together, source mapping and input interventions reveal what drives retrieval.

1 Introduction

This section argues that speech-retrieval accuracy is insufficient without physiologically interpretable models, and introduces an architecture whose weights map to cortical sources and whose inputs can be intervened on to identify stimulus-linked information.

  • Motivation: Speech-retrieval networks achieve strong decoding but their learned weights do not correspond to named cortical locations, rhythms, or time courses.Accuracy therefore does not establish what electrophysiological information the network detected.
  • Method: The study redesigns a prior MEG-to-wav2vec architecture with spherical-harmonic spatial attention, 25 latent branches, branch-specific temporal filters, and artifact removal.The changes impose spatial and temporal physiological structure while reducing shortcut risk from ocular and cardiac components.
  • Results: 39.8% Top-1 and 70.4% Top-10 accuracy were obtained on MEG-MASC for retrieving three-second segments among 1005 candidates across six independently trained solutions.The complete MEG decoder used 486,619 trainable parameters, approximately 20× fewer than Défossez et al.’s brain decoder.
  • Interpretation: The analysis maps trained spatial and temporal weights to cortical source space and uses paired MEG substitutions to test which speech-stream properties support retrieval.Feature-marked intervals are replaced with matched real-MEG donor intervals to intervene on decoder inputs.
  • Contribution: The proposed architecture preserves decoding accuracy while exposing cortical sources, their dynamics, and stimulus-linked information used by the trained decoder.The authors frame such decoders as instruments for studying the brain rather than only benchmarks for decoding it.

2 Theoretical background

MEG decoding is framed as estimating task-related neural-source activity from sensor mixtures, while balancing interference rejection against sensitivity to target sources. The section motivates an interpretable, physiologically grounded deep-learning front end and applies Petrosyan et al.’s methodology to perceived-speech decoding.

  • MEG generative model: MEG sensors measure superposed magnetic fields generated by time-varying activations of task-related and task-unrelated neuronal sources.The measurements also include spatially white sensor noise.
  • MEG generative model: Each source maps to the sensors through a gain vector determined by its location and orientation, whose interpolation produces the source topography.These gain vectors are obtained by solving Maxwell’s equations for the head as a volume conductor.
  • Task-related subspace: All information about a behavioral target available from MEG lies in the subspace spanned by the topographies of task-related sources.This follows the assumption that the target depends only, possibly non-linearly, on task-related source activity.
  • Spatial filtering: Classical spatial filters must align with the target topography while suppressing other sources, creating a trade-off between interference rejection and target tuning.The trade-off reflects limited degrees of freedom and the limited rank of multichannel MEG measurements.
  • Deep learning and interpretability: Deep learning supports complex decoding through adaptive filters but makes it difficult to connect learned features to anatomical sources and neural dynamics.An interpretable, physiologically grounded front end addresses this shortcoming by relating network parameters to pivotal neural sources.
  • Application to perceived speech: The study applies Petrosyan et al.’s methodology to decode perceived speech from concurrently recorded MEG data.The effective number of source-representing branches can be inferred as the minimum number needed before performance plateaus or degrades.

3 Methods · 3.1 3D spatial attention layer · 3.2 Interpretable front-end

The model aligns MEG segments with wav2vec 2.0 audio embeddings using a CLIP-style objective, while redesigning spatial attention, temporal filtering, and the convolutional decoder. Its interpretable front end uses spherical-harmonic sensor mapping and subject-specific branches with trainable temporal filters targeting neuronal sources and frequency ranges.

  • 3 Methods: The architecture follows the prior retrieval block structure but replaces spatial attention, adds temporal filtering, and modifies the convolutional decoder.
  • 3.1 3D spatial attention layer: Spherical harmonics parameterize J = 270 virtual channels on the MEG sensors’ three-dimensional, approximately spherical arrangement.This replaces 2-D Fourier functions defined over a planar sensor-layout projection.
  • 3.1 3D spatial attention layer: The implementation uses L = 24 degrees, yielding L2 = 576 basis functions for each virtual channel, with coefficients normalized across sensors by softmax.
  • 3.1 3D spatial attention layer: The spatial coefficients are fixed after training and do not depend on the current input sample, although the term “spatial attention” is retained.
  • 3.2 Interpretable front-end: The subject block factorizes spatial and temporal processing into K branches, each expected to tune to a neuronal source with a frequency range and topography.
  • 3.2 Interpretable front-end: Each branch applies a shared 1 × 1 unmixing convolution after the softmax-normalized 3-D attention matrix, producing spatially filtered branch signals.
  • 3.2 Interpretable front-end: Each of the K branch signals receives one trainable 1-D depthwise temporal filter targeting specific frequency ranges.Filters contain 15 samples, corresponding to 150 ms at fs = 100 Hz, use same-length zero padding and K groups, and have no bias term.

3.3 Non-linear decoder · 3.4 Spatial weights interpretation and source localization

The model decodes 25 branch signals with a shallow non-linear temporal decoder into wav2vec features, while spatial filters are interpreted through covariance patterns and common-template cortical source estimates. Interpretation uses zero-mean temporal kernels, whereas training and retrieval retain the original kernels.

  • 3.3 Non-linear decoder: The decoder evaluates B ∈{0, . . . , 5} temporal convolutional blocks before a convolutional head; B = 0 passes branch signals directly to the head.Each temporal block contains three one-dimensional convolutions with residual connections, normalization, GELU activation, dilation, and gated channel reduction.
  • 3.3 Non-linear decoder: For 3 s inputs sampled at 100 Hz, the main model uses K = 25 branches and B = 2 blocks to produce a 768 × 149 MEG-derived embedding.The head projects K decoder channels to the F = 768 wav2vec feature channels using a stride-2 convolution.
  • 3.4 Spatial weights interpretation and source localization: Each branch’s effective spatial-filter matrix W(s) yields a corresponding spatial pattern ˆgs,k for interpretation.The branch-wise spatial pattern is computed from the effective spatial-filter matrix using Eq. (5).
  • 3.4 Spatial weights interpretation and source localization: For interpretation, each learned temporal kernel is replaced by its zero-mean version to avoid the DC component.This transformation applies to covariance-based patterns, temporal patterns, spectra, source estimates, SVD, and clusters, but not training, retrieval, or paired MEG occlusion.
  • 3.4 Spatial weights interpretation and source localization: Cortical localization applies the MNE inverse solution to branch-specific sensor topographies on fsaverage using an ico4 source space and regularization parameter of λ2 = 1/3.The forward model uses a boundary-element head model, common MEG-to-template coregistration, loose orientation constraint of 0.5, depth weighting of 0.5, and ad hoc sensor-noise covariance.
  • 3.4 Spatial weights interpretation and source localization: Absolute source amplitudes are normalized to unit L2 norm, producing common-template source estimates rather than participant-specific anatomical localizations.All participants are mapped using the same template anatomy and coregistration before visualization and clustering.

3.5 Clustering branch patterns across participants · 3.6 Paired MEG occlusion analysis of stimulus-feature use

Section 3.5 clusters participant–branch patterns using joint cortical-source and temporal-spectrum similarity. Section 3.6 tests stimulus-feature use through paired, matched MEG substitutions and stringent participant-level inference with feature-present validity checks.

  • 3.5 Clustering branch patterns across participants: For 27 participants and K = 25 branches, spatial patterns were projected to common cortical source space after temporal-kernel DC removal and normalized.The analysis used Cable Spool Fort recordings from the first session.
  • 3.5 Clustering branch patterns across participants: Branches were compared using Pearson correlations between cortical source-magnitude maps and log-magnitude temporal spectra, combined through a weakest-view distance.This makes two branches close only when both their cortical distributions and temporal frequency profiles are similar.
  • 3.5 Clustering branch patterns across participants: Patterns with roughness greater than 1 were excluded, and complete-linkage clustering used a minimum raw Pearson correlation of τ = 0.25 in both views.The corresponding distance threshold was (1 − τ)/2 = 0.375; each cluster was represented by its medoid.
  • 3.6 Paired MEG occlusion analysis of stimulus-feature use: The occlusion analysis evaluated 19 acoustic, phonetic, and linguistic features, including silence states, phoneme classes, word onsets, pseudowords, random word lists, and linguistic predictability.Features were defined on the 100 Hz audio timeline, with continuous variables converted into binary contrasts using quartiles where applicable.
  • 3.6 Paired MEG occlusion analysis of stimulus-feature use: For each feature, eligible intervals in 3 s test queries were replaced with matched real MEG from feature-absent or feature-present donor intervals.Donors matched participant, session, duration, and audio file; all 208 channels were copied with 20 ms raised-cosine tapering.
  • 3.6 Paired MEG occlusion analysis of stimulus-feature use: The feature effect contrasted rank changes after feature-absent versus matched feature-present substitution, where positive values indicated greater retrieval worsening for feature-absent MEG.The unchanged 1005-candidate retrieval bank was retained for every rank calculation despite feature-specific query exclusions.
  • 3.6 Paired MEG occlusion analysis of stimulus-feature use: Statistical testing used 100,000 participant sign-flip permutations with one-sided single-step max-T familywise-error correction.Signs were applied to each participant’s complete 19-feature vector, preserving within-participant feature dependencies.
  • 3.6 Paired MEG occlusion analysis of stimulus-feature use: A positive occlusion effect required a positive mean contrast, pFWER < 0.05, and a valid feature-present control with mean rank below 503.The control was required to preserve meaningful retrieval rather than merely reflect generic replacement disruption.

3.7 Dataset and preprocessing

The study uses MEG-MASC recordings of participants listening to fictional stories, with paired 3 s audio–MEG windows prepared through artifact removal, normalization, and latency correction. Development, validation, and test segments are separated by story pieces, with 1005 candidates in the test set and no word-onset alignment.

  • Dataset: MEG-MASC contains 49 approximately one-hour participant–session recordings from 27 English-speaking participants listening to four fictional stories.Twenty-two participants contributed two sessions and five contributed one.
  • Audio preprocessing: Audio was resampled to 16 kHz and segmented into 3 s windows with a 1 s stride, retaining windows above 10^-4 peak amplitude and covering at least 50% of an annotated word.Targets were wav2vec 2.0 Base representations formed by averaging the final four hidden layers at each model time step.
  • MEG preprocessing: Ocular and cardiac ICA components were removed before MEG resampling from 1000 to 100 Hz, baseline subtraction, robust scaling, standardization, and clipping to ±20 standard deviations.Normalization was performed separately for each participant–session–story recording.
  • Dataset splits: 2698 segments formed development, the fifth Black Willow piece was held out for validation, and the last seven pieces formed a test set of 1005 candidate segments.Validation segments were excluded from neural-network parameter fitting.
  • Alignment and leakage control: Each 3 s audio segment was paired with MEG beginning 150 ms later, while test segments were not aligned to word onsets, creating a less structured retrieval setting.Black Willow scaling parameters used only samples preceding the first test-piece onset to avoid test leakage.

3.8 Training and testing

Models used fixed AdamW training and validation-based early stopping, with seed 42 for the main analyses. Testing and ablations systematically varied paired-segment duration and temporal-filter support while controlling data splits and candidate counts where specified.

  • Training procedure: AdamW training used learning rates of 3 × 10−4 for network parameters and 10−3 for learned temperature, batch size 100, zero weight decay, and at most 50 epochs.Early stopping used seven-epoch patience, with checkpoints selected by lowest validation loss; the main analyses used seed 42.
  • Window-size evaluation: Paired MEG and audio segments were evaluated at W ∈{1.5, 2.25, 3, 4, 5} s using regenerated wav2vec 2.0 targets and MEG inputs shifted by 150 ms.Original 3 s segment start positions were retained rather than constructing new starts for each duration.
  • Window-size evaluation: 991 test candidates were used at every evaluated window duration, with common starts restricted to segments having a complete 5 s continuation.Separate K = 25 models with B ∈{0, 2, 5} convolutional blocks were trained for each duration.
  • Temporal-scale evaluation: Temporal-filter support was tested with kernel sizes of 1, 5, 9, 15, 19, 25, 29, and 49 samples, corresponding to 10–490 ms at 100 Hz.These were otherwise identical two-block, K = 25 models using seed 42, the same data split, optimizer, early-stopping rule, and validation checkpoint selection.

4 Results

Results show that a compact branch representation with sufficient convolutional processing supports high retrieval performance while mapping to speech-relevant cortical sources. Occlusion analyses identify broad speech-state, intensity, acoustic-change, and phonetic features as major contributors, whereas random word-list activity is less recoverable.

  • Model capacity: Retrieval rises sharply with branch count and reaches a broad high-accuracy plateau around K = 10–25, while larger K values flatten or mildly decline.This pattern indicates a compact task-related subspace; extra branches mainly add redundancy or harder-to-regularize degrees of freedom.
  • Model capacity: Convolutional blocks substantially improve retrieval once K is useful, but gains saturate with two or more blocks.Models without convolutional blocks are consistently weaker, indicating that compact branches still require temporal and contextual processing.
  • Spatial organization: Attention becomes more spatially structured and concentrated as decoder depth and branch count increase, highlighting auditory and anterior regions with stronger left-hemisphere concentration in the most complex configuration.The displayed maps reflect only the first shared spatial-attention stage; participant-specific filtering is defined by W(s).
  • Source structure: 93.7% of pooled within-cluster pairs correlate at 0.40 or above in both source-map and temporal-spectrum views, supporting reproducible retained patterns.Median within-cluster correlations range from 0.51 to 0.67 for cortical source maps and 0.63 to 0.72 for temporal spectra.
  • Source structure: Retained clusters occupy bilateral superior temporal regions, dorsal-stream supramarginal and central-sulcus areas, and left frontal-opercular, inferior-frontal, and anterior temporal sites.These locations align with speech-envelope tracking, phonological and sensorimotor functions, perceptual prediction, and auditory–conceptual integration.
  • Stimulus features: 15 of 19 tested features produced positive familywise-error-corrected paired rank contrasts, led by silence (∆r = 75.62), high loudness (60.77), vowels (38.70), and strong acoustic onset (36.19).Random word lists showed no corrected positive evidence (∆r = −17.77); narrative MEG substituted into word-list intervals improved retrieval relative to recorded word-list activity.

5 Discussion

The discussion presents a compact, source-localizable MEG decoder whose retrieval performance remains competitive while its architecture and weights become more interpretable. Interventions indicate that retrieval combines distributed acoustic, phonetic, and contextual speech information, with coherent narrative structure contributing beyond local word cues.

  • Compact localizable front end: 39.75 ± 0.34% Top-1 and 70.40 ± 0.31% Top-10 among 1005 candidates were achieved across six seeds with approximately twenty times fewer trainable parameters.The main configuration used K=25 and two convolutional blocks, compared with Défossez et al.’s 41.3% Top-1 and 70.7% Top-10 among 1363 candidates.
  • Compact localizable front end: The compact model outperformed the closest K = 270, five-block configuration by 3.60 percentage points in Top-1 and 3.14 points in Top-10 despite using 14.8 times fewer trainable parameters.Replacing spherical-harmonic attention with its 2D version or removing branch-wise temporal filters each cost about one percentage point.
  • Cortical mapping: Mapped branches formed distributed, lateralized sources concentrated along superior and middle temporal gyri, with additional peri-Sylvian foci consistent with auditory speech-perception networks.Low-frequency activity dominated, while some cluster medoids also showed alpha and lower-beta contributions.
  • Stimulus features driving retrieval: Occlusion results showed that retrieval uses broad acoustic state, rapid sound changes, phonetic content, and selected word-context properties rather than one privileged linguistic representation.The largest contrasts involved silence, high loudness, strong acoustic onsets, and vowels; positive effects also occurred for multiple other phoneme classes and high-surprisal words.
  • Feature-axis compression: About a dozen learned wav2vec feature dimensions preserved retrieval accuracy, whereas power-driven PCA degraded substantially earlier.Each reduced dimensionality was trained as a separate model, so the twelve-dimensional configuration was itself a working decoder rather than a post hoc projection.
  • What retrieval accuracy measures: Δr = −17.77 improved when random-word-list MEG was replaced with coherent-narrative MEG, showing that retrieval accuracy reflects composite information shaped by sentence and narrative context.The word-list intervals retained individual spoken words and local acoustic and phonetic cues, so the contrast was not simply additive.

6 Limitations

The study’s limitations concern generalization beyond one corpus and fixed narratives, simplified assumptions about cortical dynamics, non-independent feature interventions, and anatomical uncertainty in source localization.

  • Generalization: The single corpus with 27 participants supports generalization across listeners only for fixed held-out audio, not replication across new narratives.Repeating paired occlusion across six independently trained instances showed the feature-use pattern was not specific to the seed-42 solution.
  • Model interpretation: The front-end interpretation assumes linear space–time factorization and cannot establish the role of space–time inseparable cortical waves.A plurality of branches may represent some propagating activity, but this does not demonstrate cortical-wave involvement per se.
  • Intervention design: Paired substitutions do not isolate strictly independent causal contributions because annotations overlap or covary, masks differ, and donor replacement can alter correlated stimulus states.The feature-present arm controls generic replacement damage, but reported rank contrasts should therefore be interpreted as evidence that the train…
  • Source localization: RAP-MUSIC and clustered spatial-temporal profiles use a single generic forward model, so participant anatomy may shift estimated sites and broaden cortical maps.Individual forward models could nevertheless be built for six participants whose defaced volumes survived surface reconstruction.

Appendix · A Mapping the signal subspace to cortical sources via recursive subspace correlation scan

The appendix specifies a recursive subspace-correlation scan that maps a data-derived MEG signal subspace onto cortical source sites using a forward model. It iteratively selects sources by maximum subspace correlation, projects out selected contributions, and stops at an acceptance threshold or rank limit.

  • A Mapping the signal subspace to cortical sources via recursive subspace correlation scan: The procedure maps signal subspace S to cortical sources through a recursive subspace-correlation scan.It takes a data-subspace basis GD, forward-model matrix GM, and acceptance threshold θ as inputs.
  • A Mapping the signal subspace to cortical sources via recursive subspace correlation scan: Initialization sets k = 0, the selected-source matrix A0 empty, the projection P0 to identity, and projected bases GD,0 and GM,0 to their inputs.The scan begins with GD,0 = GD and GM,0 = GM.
  • A Mapping the signal subspace to cortical sources via recursive subspace correlation scan: At each recursion, the algorithm evaluates each candidate source i = 1, . . . , Q using the maximum subspace correlation between projected data and source subspaces.For each candidate, it computes ρk_1(i) = max subcorr(SD,k, SMi,k).
  • A Mapping the signal subspace to cortical sources via recursive subspace correlation scan: The next source is the candidate ik+1 with the largest correlation value ρk_1(i).The selection uses an argmax across all Q candidate sites.
  • A Mapping the signal subspace to cortical sources via recursive subspace correlation scan: The recursion terminates when the selected correlation does not exceed the acceptance threshold θ; otherwise, the selected source is appended and the recursion advances.The supplied algorithm specifies a break condition followed by k ← k + 1 when the threshold is passed.
  • A Mapping the signal subspace to cortical sources via recursive subspace correlation scan: After each selection, projection operators update GD,k and GM,k by removing the selected source contribution before the next scan.The updates are GD,k ← PkGD,k−1 and GM,k ← PkGM,k−1.
  • A Mapping the signal subspace to cortical sources via recursive subspace correlation scan: The output is the ordered set of selected source sites and their corresponding subspace correlations.The algorithm returns i1, . . . , ik together with the correlations accumulated during recursion.
  • A Mapping the signal subspace to cortical sources via recursive subspace correlation scan: The forward model represents Q cortical vertices with three oriented ECD topographies each, while the analyzed data subspace uses the leading R = 10 right singular vectors.GM is M × (Q × 3), and GD is an orthonormal basis of the analyzed spatial-pattern subspace derived from the row-L2-normalized spatial-pattern matrix.

B Robustness of paired occlusion effects across model initializations

Paired MEG occlusion findings were robust across six model initializations: all 15 seed-42 positive features replicated, and the strongest effects remained stable in magnitude. Negative contrasts for random word lists and rare words also generalized across models, while high-entropy and silence-onset effects did not robustly survive correction.

  • Cross-initialization robustness: 15 features met the corrected positive-effect criterion in all six trained models.The complete paired occlusion analysis was repeated across seeds 43–47 using the same architecture and procedures as seed 42.
  • Cross-initialization robustness: 79.89 rank positions was the across-seed median effect for silence, followed by 62.57 for high loudness, 38.10 for vowels, and 36.35 for strong acoustic onset.The corresponding ranges were 75.62–88.64, 60.77–65.92, 35.28–40.86, and 33.23–40.04, respectively.
  • Contrast robustness: −17.10 was the across-seed median random-word-list contrast, which was negative in every model.Rare words were also negative in all six models, whereas high-entropy did not pass correction in any model and silence-onset met the positive criterion in only two of six models.

C Model size and retrieval performance

The redesigned LISA decoder achieves retrieval with far fewer trainable parameters than the compared architecture, while controlled reductions in branch count and target dimensionality largely preserve performance.

  • Model size: 486,619 trainable parameters comprise the main LISA decoder, compared with 9,565,054 for Défossez et al.’s brain decoder in the 27-participant MEG-MASC setting.The LISA count includes 426,715 parameters in the spatial–temporal and nonlinear decoder core and 59,904 in the final wav2vec projection.
  • Retrieval performance: 7,210,224 parameters in the K = 270, five-block model are 14.8 times the main model’s size, yet yield 3.60 percentage points lower Top-1 and 3.14 points lower Top-10 accuracy.This controlled comparison retains the redesigned model’s other architectural, preprocessing, and training choices.
  • Compression: 380,769 parameters in the K = 15 model retain nearly the full retrieval score, while a learned 12-dimensional target retains the main performance with 427,651 parameters.These configurations test branch-count reduction and target-feature compression, respectively.
Loading 2608.01481v1…