PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Decoding as Cognitive Inference

Yi Guo, Changhong Jing, Yong Hu, Yan Liu, Michael K. P. Ng, Shanshan Wang, Shuqiang Wang

arXiv:2610.11923v1q-bio.NCcs.AI

TL;DR

Neural decoding must recover stable cognitive states from changing neural observations, while existing stimulus-response methods map observations to predefined labels. This paper uses brain-intrinsic priors to infer meta-neural semantic representations, supporting stable cross-session decoding despite neural drift and without requiring unchanged activity patterns.

  • Problem

    Neural decoding must resolve stable cognitive states from changing neural activity, while stimulus-response methods target predefined external labels.

    Source: Discussion · pp. 14–15

  • Method

    The framework incorporates brain-structural constraints as brain-intrinsic priors, instantiated with cortical geometric eigenmodes, to infer meta-neural semantic representations.

    Source: Discussion · pp. 14, 16

  • Results

    The framework supports stable cross-session decoding despite neural drift, consistent with stable recovery not requiring unchanged neural activity patterns.

    Source: Discussion · p. 15

  • Takeaways & Limitations

    The framework provides an analytical tool for examining how changes in neural activity relate to the stability of cognitive states.

    Source: Discussion · p. 14

    The framework uses structural priors; future studies could examine priors derived from neural dynamics.

    Source: Discussion · p. 16

Abstract

from arXiv · show

The brain maintains stable cognition despite continuously changing neural activity. How to extract stable cognitive states from variable neural observations remains a central problem in neural decoding. Existing neural decoding methods map neural observations to predefined external labels based on the stimulus-response principle, often capturing recording-specific spurious correlations. Inspired by how the brain infers the world, and specifically by Bayesian brain theory, we recast neural decoding as cognitive inference constrained by brain-intrinsic priors, yielding high-level meta-neural semantic representations. In decoding experiments spanning five neural recording modalities and three cognitive domains (motor, perception and internal mentation), our cognitive inference method reorganized the geometry of neural observation representations, yielding meta-neural semantic representations that exhibited consistent geometric relationships across cognitive tasks and enabled the recovery of stable cognitive states from variable neural observations. Our work provides an account of how the brain maintains relatively stable cognition despite continual changes in the external environment. Cognitive stability is sustained through cognitive inference from changing neural activity, without requiring fixed neural activity patterns.

Results

Cognitive inference yielded brain-wide meta-neural semantic representations with more consistent geometry across tasks and greater stability across sessions than low-level neural observations. The results also show improved decoding across diverse tasks and recording periods.

representations

  • 51.59% versus 28.28%: NSD cortical searchlight RSA showed broader correspondence for meta-neural semantic representations than for low-level image features.Semantic attribution was also more broadly distributed for high-level representations in SMN4Lang.Source: representations · p. 4, Figure 2 · p. 5
  • Semantic attribution concentrated in long-wavelength cortical eigenmodes, peaking near 84 mm; the whole-cortex semantic–connectivity coefficient was 0.33 versus 0.05 for CogReader.The framework also showed denser inter-network relations and more long-range connections than CogReader.Source: representations · p. 4

tasks

  • Across motor, perception and internal mentation, high-level representations showed more consistent state-difference directions and greater cross-condition generalization and parallelism.Complementary analyses found lower within-state dispersion, greater between-state centroid distance and higher category-structure RSA at the high level.Source: tasks · p. 6, Figure 3 · p. 7
  • The proposed framework achieved the highest participant-averaged within-participant and mean zero-shot decoding accuracy among four methods across all four datasets.Zero-shot models were applied to participants excluded from training without participant-specific adaptation.Source: tasks · p. 6, Figure 3 · p. 7

drift

  • Across sessions, high-level representations retained more consistent amplitude profiles, state separation and trajectory geometry, alongside higher mean cross-session decoding accuracy.The ECoG Speech recordings spanned 208 days, including follow-ups more than 200 days after day 0.Source: drift · p. 8
  • Representational displacement was 17.5 versus 3.7 in ECoG Speech, 12.7 versus 2.5 in AJILE12, and 17.9 versus 5.3 in SHU-MI for low- versus high-level representations.These values corresponded to reductions by factors of three to five across the datasets.Source: drift · p. 8

Cognitive states exhibit semantic

Across language and vision, cognitive inference recovered semantic content consistent with target states despite differences in sensory inputs and neural activity patterns. The framework also captured internally generated content and individual subjective states elicited by shared stimuli.

domains

  • The framework outperformed CogReader and a random-output control across four language datasets, with its advantage sustained throughout Chinese and French narratives.Source: domains · p. 10, Figure 5 · p. 11
  • In visual decoding, meta-neural semantics aligned more strongly with image-description semantics than low-level observations and outperformed NCC on retrieval and image reconstruction.Source: domains · p. 10, Figure 5 · p. 11
  • In language and vision, cognitive inference recovered semantic content consistent with target states despite differences in sensory inputs and neural activity patterns.Source: domains · p. 10

content to internal mentation

Discussion

The framework treats neural decoding as inference of cognitive states from changing observations, using brain-intrinsic priors to distinguish neural variation from state change. Its discussion connects stable cognition to preserved representational relationships across sessions and internally generated states, while noting that this implementation uses structural priors.

Discussion

  • The framework reframes neural decoding as inference of relatively stable cognitive states from changing observations, rather than mapping observations to predefined labels.It is also presented as an analytical tool for examining the relationship between neural activity changes and cognitive stability.Source: Discussion · p. 14
  • Cognitive inference reorganizes representational geometry so relationships among cognitive states become more stable despite variation in finite neural observations.The discussion characterizes cognitive stability at the geometric level as preservation of relational structure among states.Source: Discussion · pp. 14–15
  • Across days to months, meta-neural semantic representations retained consistent geometry and stable decoding despite neural drift, without requiring cross-session observation alignment.The discussion describes neural drift as compatible with stable cognitive states, rather than evidence that those states changed.Source: Discussion · p. 15
  • Internal-mentation experiments indicate that inferred representations can reflect internally generated states and subjective interpretations beyond stimulus content.The stated scope includes memory, imagination, judgment and subjective interpretation.Source: Discussion · p. 15
  • Individualized cortical geometric priors allow a common inference process to recover stable cognitive states under each person’s own structural constraints.The discussion relates this approach to inter-individual variation in brain structure and priors.Source: Discussion · pp. 15–16
  • The current implementation uses structural priors instantiated with cortical geometric eigenmodes; future work could examine priors derived from neural dynamics.The discussion suggests temporal priors as a way to exploit regularities in cognitive-state evolution over longer timescales.Source: Discussion · p. 16

framework

The framework infers a high-level meta-neural semantic representation from neural observations under brain-intrinsic structural constraints. It combines observation evidence and prior information through a temperature-scaled product of experts.

framework

  • The framework denotes the neural observation representation as o, the prior-constrained representation as u, and the inferred meta-neural semantic representation as z.The resulting z is used for subsequent decoding.Source: framework · pp. 16–17
  • The observation encoder maps modality-specific recordings into o, which represents evidence from the current neural observation.Grouped spatial-coordinate embeddings and factorized attention accommodate differences in recording length and channel configuration.Source: framework · p. 16
  • The prior encoder produces u by imposing structural constraints instantiated here with a fixed cortical geometric eigenmode basis.Eigenmode projection coefficients express the current observation in the basis, whose eigenvalues index spatial scale.Source: framework · pp. 16–17
  • Inference combines the current observation and prior-constrained representation as complementary factors in a temperature-scaled product of experts.The observation factor retains evidence from the recording, while the prior-constrained factor encodes brain-intrinsic structural constraints; temperatures control their relative contributions.Source: framework · p. 17
  • The neural observation and prior encoders were independently pretrained as masked autoencoders using unlabeled recordings from each modality.Fifty percent of temporal and spatial patches were masked, and reconstruction loss was evaluated only over masked positions.Source: framework · p. 17

Cortical geometric eigenmodes

Cortical geometric eigenmodes provide a common structural basis that is mapped into each recording modality’s observation space. In the framework, this basis supplies structural prior information for inference.

Cortical geometric eigenmodes

  • Cortical geometric eigenmodes instantiate intrinsic cortical geometry as the structural prior used by the current framework.The eigenfunctions form an orthogonal basis ordered by spatial scale and fixed across experimental tasks.Source: Cortical geometric eigenmodes · pp. 17–18
  • The non-negative eigenvalues are ordered from low to high, with smaller eigenvalues corresponding to larger spatial scales; the constant zeroth mode is removed.The remaining K modes are assembled into Φ.Source: Cortical geometric eigenmodes · p. 18
  • The cortical basis is mapped to each modality’s observation space according to its spatial sampling geometry to obtain eigenmode coefficient time series.For EEG and MEG, a lead-field matrix maps the cortical basis to sensor space.Source: Cortical geometric eigenmodes · p. 18
  • For fMRI, the analysis retained 1,000 modes per hemisphere, and least squares projected cortical time series to obtain coefficient time courses.The analysis used the standard 32k fs LR midthickness surface and removed the medial wall with a cortical atlas region-of-interest mask.Source: Cortical geometric eigenmodes · p. 18
  • For sensor-space mapping, the forward operator mixes cortical modes into correlated patterns; singular value decomposition processes the mapped basis before selected components estimate coefficient time series.Cortical spatial-scale information is incorporated into the sensor-space representation.Source: Cortical geometric eigenmodes · p. 18
  • For ECoG and fNIRS, the eigenmodes are mapped using contact locations or source-detector midpoints, while participant-specific cortical surfaces provide individualized bases.The fNIRS components are orthogonalized before separate HbO and HbR projection; participant-specific bases replace the standard cortical basis in modality-specific mapping.Source: Cortical geometric eigenmodes · pp. 18–19

meta-neural semantic representations

The framework derives high-level meta-neural semantic representations for downstream decoding. Open-ended language decoding aligns these representations with text before conditional generation, while image decoding maps them into an image-conditioning space.

meta-neural semantic representations

  • Open-ended language decoding aligns meta-neural semantic representations with BGE-M3 text states using a Q-Former, then proceeds to conditional generation.BGE-M3 token-level states are resampled to a fixed-length semantic sequence.Source: meta-neural semantic representations · p. 19
  • Language-model and alignment parameters are shared across participants, and test stories are held out from fitting for final generation and evaluation.The same story-level 9:1 split is applied to every participant; alignment is fitted before language adapters.Source: meta-neural semantic representations · p. 19
  • Image decoding maps the representation into normalized CLIP image-embedding space and uses a frozen latent-diffusion decoder to generate images.NOD trials are used only for visual pretraining; evaluations use participant-specific NSD models and assess viewed-image and mental-imagery reconstruction.Source: meta-neural semantic representations · pp. 19–20

Training and implementation details

The neural-observation and brain-intrinsic-prior encoders are pretrained separately, then frozen for inference and downstream language decoding.

Training and implementation details

  • Neural-observation and brain-intrinsic-prior encoders are independently pretrained as masked autoencoders with a reconstruction objective and 0.5 masking ratio.The supplied passage also names AdamW, but its learning-rate value is incomplete.Source: Training and implementation details · p. 20
  • The encoders are frozen after retaining the checkpoint with lowest validation reconstruction loss, and positive temperatures are calibrated separately by modality on validation data.The temperature pair with the lowest validation loss is fixed for held-out evaluations and controls the relative strength of factors in temperature-scaled PoE inference.Source: Training and implementation details · pp. 20–21
  • Language alignment uses subject-specific layers and a Q-Former for 200 epochs, followed by 15 epochs of joint linear-projector and PiSSA-LoRA optimization on frozen Phi-4-mini.The alignment stage uses batch size 256; the language adapters use rank 16 and dropout 0.05.Source: Training and implementation details · p. 21

Datasets

The study includes an internally acquired imagined-speech OPM-MEG cohort and public datasets across multiple recording modalities, with modality-specific preprocessing.

Datasets

  • The imagined-speech cohort comprised adults aged 25–29 years and used 5-second reading, 4-second speech-imagery and 4-second fixation periods; only imagery epochs entered the main analysis.Neural data were recorded with a 64-channel single-axis OPM-MEG system.Source: Datasets · p. 21
  • EEG and MEG recordings were filtered and resampled to 256 Hz, with ICA used for eligible continuous EEG recordings and OPM-MEG sensors co-registered to individual anatomy.EEG processing also used release-provided preprocessing for datasets distributed only as epochs.Source: Data preprocessing · p. 22
  • fNIRS preprocessing included intensity screening, optical-density conversion, motion correction and band-pass filtering before hemoglobin conversion; processed releases retained their provided preprocessing.Trial-wise baseline correction was applied where required.Source: Data preprocessing · p. 22
  • ECoG recordings were screened, line-noise filtered and resampled to 256 Hz, retaining the source release reference scheme where applicable.Dataset-supplied electrode coordinates supported cortical-surface sampling.Source: Data preprocessing · p. 22

Model comparisons

Model comparisons generally matched data partitions and evaluation samples, while using task-appropriate baselines across modalities and decoding settings.

Model comparisons

  • EEG and MEG comparisons used released LaBraM, BrainOmni and CBraMod weights with task-specific linear heads and matched within-participant partitions.Zero-shot cross-participant training excluded all samples from the evaluated participant; evaluation froze each backbone and fitted only its classification head.Source: Model comparisons · p. 22
  • Cross-session ECoG and EEG comparisons included SPaRCNet and used the same reference-session training data and subsequent-session test trials for every method.Shared cohorts used identical trial definitions, label vocabularies and test samples across methods.Source: Model comparisons · p. 23
  • Language decoding was compared with CogReader on the same story boundaries, while image retrieval was compared with NCC using identical NSD candidate sets and test trials.A random-control language condition replaced the neural soft prompt with independent standard-normal tokens while keeping the frozen language model and decoding procedure unchanged.Source: Model comparisons · p. 23

Quantitative analyses

The quantitative analyses define evaluation splits, representation metrics, attribution procedures and robustness tests across decoding, semantic alignment and cognitive-state analyses.

Quantitative analyses

  • Held-out evaluation separated data at task-appropriate levels, including disjoint participants for zero-shot transfer, shared story-level splits, and later sessions tested without refitting.Held-out stories were used only for final generation and evaluation.Source: Quantitative analyses · p. 23
  • Cortical attribution compared low-level vertex-to-semantic prediction with high-level eigenmode-to-semantic prediction, then related parcel maps to participant-specific connectivity and Yeo networks.The attribution maps were independently rank-normalized for visualization.Source: Quantitative analyses · pp. 23–25
  • Representational geometry analyses crossed two cognitive states with two conditions and quantified transfer and state-difference alignment using CCGP accuracy and cosine similarity.Representations were embedded separately; quantitative metrics were computed in full feature space.Source: Quantitative analyses · p. 25
  • Internal-state analyses used category-level RSA and content classification for imagined speech, while fMRI tasks tested self-generated-state labels and clarity, vividness and valence targets.Spacetop analyses also related neural representations to rating dissimilarities across and within participants using held-out observations.Source: Quantitative analyses · pp. 26–27
  • Structural controls compared geometric eigenmodes with Fourier, spherical-harmonic and graph-Laplacian bases; robustness tests varied noise, channel availability and training data.BrainOmni and the proposed model received matched perturbations and sample subsets.Source: Quantitative analyses · p. 27
  • The analyses used accuracy for discriminative decoding, BGEScore for language generation, CLIP similarity for image reconstruction, and top-k accuracy for retrieval.Matched-method comparisons used repeated-measures tests or paired tests, with stated corrections for multiple comparisons.Source: Quantitative analyses · pp. 27–28

LOW HIGH LOW HIGH

The supplied extended figures describe semantic maps, representation geometry, decoding comparisons, robustness analyses and the datasets used.

LOW HIGH LOW HIGH

  • Semantic-attribution maps compare low-level and high-level representations across 12 SMN4Lang participants, with percentages indicating cortical coverage.Source: Figure 1 · p. 34
  • Subjective-state figures show prediction–rating correlations and participant-specific residual predictions across Spacetop tasks, alongside RSA comparisons with NeuroSTORM.The figures identify rating dimensions, participant counts and statistical summaries.Source: Figure 2 · p. 35, Figure 4 · p. 37
  • Representational-geometry figures compare low-level and high-level representations across motor, perception and internal-mentation categories using MDS, dispersion, separation and RSA.Source: Figure 3 · p. 36
  • Ablation figures compare four cortical basis types within and across participants and plot accuracy against retained modes, marking the main-analysis dimensionalities.The marked settings are 14, 20, 30 and 28 modes.Source: Figure 5 · p. 38
  • Individual-geometry analyses report weaker spatial correspondence at higher eigenmode ranks but increasingly consistent eigenvalue scales, and compare group-level with participant-specific bases.Source: Figure 6 · p. 39
  • Additional figures describe decoding comparisons across inner-speech, somatomotor and fNIRS tasks, robustness to perturbations, and the study’s dataset and cohort inventory.Source: Figure 7 · p. 40, Figure 8 · p. 41, LOW HIGH LOW HIGH · p. 42
Loading 2610.11923v1…