Source-linked AI summary
Margin-Regularized Structured Semantic Alignment for Brain-Language Correspondence
Jiaqi Wang, Huawen Hu, Shu Zhang
TL;DR
Brain-language decoding remains difficult to interpret because decoded semantics may reflect language-model priors rather than neural representations. MD-SigLIP aligns brain and text embeddings with cluster-aware, margin-regularized ranking constraints, achieving state-of-the-art retrieval performance across two MEG-language datasets.
Problem
It remains unclear whether decoded semantic content genuinely originates from neural signals or is largely reconstructed by language-model priors, limiting interpretability of brain-language correspondence.
Method
MD-SigLIP aligns brain and text embeddings in a shared space while combining multi-positive semantic clusters with listwise margin regularization for structured ranking.
Results
MD-SigLIP achieves state-of-the-art retrieval performance across two MEG-language datasets, including 8.7% versus 6.3% Top 10 accuracy and median rank 138 versus 180 under full-vocabulary evaluation.
Takeaways & Limitations
The findings provide converging evidence that non-invasive neural signals encode retrievable language-related semantic structures aligned with text representations.
Abstract
from arXiv · showhide
With the rapid advancement of large language models, brain-language decoding has achieved remarkable progress. However, it remains unclear whether decoded content genuinely reflects neural representations or is largely reconstructed by the language model itself. This ambiguity limits interpretability and hinders the investigation of intrinsic brain-language correspondence. To address this challenge, we propose MD-SigLIP. This margin-regularized structured semantic alignment framework directly aligns brain embeddings with text embeddings in a shared semantic space, enabling retrieval-based decoding. This formulation enables explicit modeling of the correspondence between neural representations and language semantics. Building upon duplicate-aware sigmoid contrastive learning, we introduce a listwise margin-regularized term that enforces structured ranking constraints between positive semantic clusters and negative samples. By modeling multi-positive semantic structure and margin-based ordering simultaneously, the method captures the manifold organization of language embeddings reflected in neural signals. Experiments demonstrate state-of-the-art retrieval performance under both full-vocabulary and subset evaluation settings.
1 Introduction
The paper frames brain-language decoding as retrieval-based alignment to directly assess brain-text correspondence, addressing uncertainty about whether generative outputs originate from neural signals. It proposes MD-SigLIP, which adds listwise margin regularization for cluster-aware semantic modeling and explicit ranking constraints, and reports evidence of semantic and structural consistency.
- Motivation: Generative brain-language decoding can produce fluent outputs, but the extent to which decoded semantics originate from neural signals remains unclear.This ambiguity limits interpretability and investigation of intrinsic brain-language correspondence.
- Retrieval-based alignment: Retrieval-based approaches learn a shared embedding space and retrieve text candidates by neural-representation similarity, enabling explicit measurement of brain-text correspondence.They provide an alternative to relying on generative models for decoding.
- Method: MD-SigLIP extends D-SigLIP with listwise margin regularization that models semantically similar positives as clusters and enforces explicit ranking constraints against negatives.The method aggregates multiple similar positives into a cluster-level reference score and penalizes negatives ranked above that cluster.
- Evidence: Retrieval-based decoding and representational similarity analysis provide empirical evidence of semantic and structural consistency between brain and text representations.These analyses support the study of intrinsic brain-language correspondence.
- Implication: Non-invasive neural signals encode retrievable language-related semantic structures, offering an interpretable framework for studying brain-language correspondence.This claim follows from the proposed retrieval-based alignment framework and its reported empirical evidence.
2 Method
MD-SigLIP reformulates brain-to-text decoding as structured semantic alignment between neural and text embeddings in a shared space. It combines duplicate-aware multi-positive supervision with listwise margin regularization to enforce semantic ranking.
- Shared-space retrieval: A brain encoder maps MEG signals into the frozen T5 text-embedding space, and decoding retrieves candidate sentences by embedding similarity.For paired data, fθ maps neural signals to z_i while g produces text embeddings t_i.
- Duplicate-aware alignment: Duplicate-aware supervision treats semantically similar text embeddings as multiple positive targets rather than enforcing one-to-one matching.The multi-label sigmoid objective removes softmax competition and supports multiple positives per sample.
- Margin regularization: A listwise margin-regularized term aggregates positive similarities and encourages positive semantic clusters to rank above negative samples.The coefficient λ balances alignment and margin regularization.
- Similarity modeling: Similarity scores can use ℓ2-normalized embeddings with a learnable temperature τ and bias b within a mini-batch similarity matrix.The similarity matrix S ∈ R^B×B′ is defined over mini-batch brain and text embeddings.
3 Experiments and Results
Experiments on auditory and reading MEG datasets show that MD-SigLIP improves retrieval performance and produces semantically coherent decoding. RSA further indicates statistically significant structural alignment between brain and language embeddings.
- Datasets and preprocessing: MD-SigLIP is evaluated on Armeni2022 MEG-Audio and SchoffelenRead2019 MEG-Reading datasets after standardized filtering, downsampling, baseline correction, and word-level segmentation.Armeni2022 includes 3 subjects, approximately 10 recording hours per subject, and 7,282 distinct words; SchoffelenRead2019 includes 99 subjects and 2,223 distinct words.
- Retrieval performance: On SchoffelenRead2019, Top 10 accuracy rises from 8.2% to 10.3% and median rank falls from 86 to 65 under the Top 250 setting.Under full vocabulary, Top 10 accuracy increases from 6.3% to 8.7%, while median rank improves from 180 to 138.
- Qualitative decoding: Sentence-level retrieval captures core semantics and syntactically coherent structures, with semantically equivalent substitutions occurring when exact word matches fail.These qualitative results are presented as evidence that the model learns high-level semantic representations rather than merely memorizing training data.
- Semantic structure: Top 10 retrieved candidates cluster around contexts and concepts, indicating a smooth shared latent space with coherent semantic neighborhoods rather than isolated word matches.Similar neural activation patterns map to semantically related linguistic representations.
- Representational similarity analysis: RSA shows significant brain–text structural correspondence on full-vocabulary Armeni2022 (p < 0.001) and Top 15 SchoffelenRead2019 (p < 0.05).The findings indicate that MEG signals preserve relational semantic structure aligned with language embeddings, with extensive recordings per subject appearing particularly beneficial.
4 Conclusion
The conclusion frames brain-to-text decoding as structured semantic alignment and presents MD-SigLIP as a framework for interpretable brain-language correspondence. It combines cluster-aware, order-aware constraints with multi-positive supervision and listwise margin regularization in a shared embedding space.
- The work reformulates brain-to-text decoding as a structured semantic alignment problem.
- MD-SigLIP enforces cluster-aware and order-aware constraints in a shared embedding space.
- The method combines multi-positive supervision with listwise margin regularization to capture structured brain-language correspondence.