Source-linked AI summary
Towards semantic reconstruction of individual words from fnirs using clip loss
Santiago Posso-Murillo, Nathan Palladino, Ben Pyykkonen, Dong Y. Han, Luis G. Sanchez-Giraldo, Jihye Bae
TL;DR
fNIRS semantic decoding can map brain activity to word embeddings, but MSE-based approaches ignore relationships among embeddings, and CLIP had not been evaluated for this task. The paper compares CLIP and MSE with a Bi-LSTM across three fNIRS datasets and two embedding targets. CLIP is the most robust decoder across experiments, while T5 yields higher matching scores and GloVe-50 is associated with more reliable retrieval.
Problem
Method
The study compares CLIP and MSE for a Bi-LSTM mapping fNIRS signals to GloVe-50 or T5 word embeddings across three datasets.
Source: §1 · p. 2, §2.2 · p. 3Results
Bi-LSTM (CLIP) is the most robust decoder across experiments; T5 gives higher matching scores, while GloVe-50 gives more reliable retrieval accuracy.
Source: §4 · pp. 7–8Takeaways & Limitations
The authors conclude that CLIP suits word reconstruction because it directly optimizes whether predictions are closer to their own targets than to alternatives.
Source: §4 · p. 7The datasets were small and collected from healthy subjects, and the paradigm covers language perception of isolated words rather than speech production or natural speech.
Source: §4 · p. 8
Abstract
from arXiv · showhide
Semantic reconstruction maps neural activity to a word-embedding space, recovering the meaning of a perceived word instead of selecting it from a fixed vocabulary. Functional near-infrared spectroscopy (fNIRS) carries semantic information suitable for this mapping. However, most fNIRS decoders are trained with a squared-error objective that fits each word independently and ignores the geometry of the embedding space. To address this limitation, we evaluate a contrastive loss based on the contrastive-language-image-pretraining (CLIP) loss, as an alternative to mean-squared-error (MSE) for reconstructing perceived words from fNIRS. We compare the two objectives by training a bidirectional long short-term memory (Bi-LSTM) decoder to map fNIRS signals to word embeddings. We use GloVe-50 and T5 word embeddings as targets, across three fNIRS datasets recorded under a shared paradigm pairing each word image with its spoken name. Performance is measured with a pairwise matching score and open-vocabulary top-$k$ retrieval. The Bi-LSTM trained with CLIP is the most consistent decoder across experiments. T5 produces higher matching scores, whereas every significant retrieval result uses GloVe-50. These results support the use of contrastive objectives as a promising direction for fNIRS semantic decoding and motivate validation on larger datasets.
1 Introduction
fNIRS offers a portable alternative for semantic decoding, and word embeddings allow prediction beyond a fixed set of classes. This work evaluates CLIP as an alternative to MSE for fNIRS semantic decoding, where CLIP had not previously been evaluated.
1 Introduction
- fNIRS is a portable, motion-tolerant alternative to fMRI that measures the hemodynamic response through differential absorption by HbO and HbR.Prior studies found semantic information in fNIRS and classified speech-related signals in hybrid or fNIRS-only setups.Source: §1 · p. 2
- Word embeddings let a decoder predict a vector and identify a word by its nearest embedding, avoiding a fixed set of output classes.Prior fNIRS work mapped signals to GloVe with ridge regression and later used recurrent networks.Source: §1 · p. 2
- MSE fits each word independently and ignores relationships to other embeddings, while recurrent-model outputs required regularization to prevent collapse.These limitations motivate testing an objective that accounts for embedding-space structure.Source: §1 · p. 2
- CLIP aligns matched pairs and separates mismatched pairs; it had been used with other brain signals but had not been evaluated for fNIRS semantic decoding.Prior applications included fMRI with image embeddings, MEG and EEG with speech-audio embeddings, and EEG and MEG with word embeddings.Source: §1 · p. 2
- The study compares CLIP and MSE using GloVe-50 and pretrained T5 embeddings as targets for fNIRS semantic decoding.GloVe-50 is learned from word co-occurrence statistics; language-model representations had also been used as EEG and MEG decoding targets.Source: §1 · p. 2
2 METHODOLOGY
The study evaluates three fNIRS datasets collected from healthy participants under word-perception paradigms. Their vocabulary sizes, repetition schedules, and recording setups differ.
2.1 fNIRS Datasets
- The Cao dataset includes seven healthy subjects viewing 50 words from 10 semantic categories, each presented seven times for 3 seconds.Recordings used 22 channels over Broca’s and Wernicke’s areas at 7.8 Hz; the released data was preprocessed.Source: §2.1 · p. 3
- The Nathan dataset includes five healthy subjects perceiving six words from three categories, with each word presented 20 times for 5 seconds.It was recorded with 44 channels at approximately 7.63 Hz; raw intensities were converted to hemoglobin concentrations and band-pass filtered.Source: §2.1 · p. 3
- The Zinszer dataset includes 20 healthy subjects viewing eight words from two categories, with 12 presentations per word.It was recorded at 10 Hz using posterior and lateral channel arrays, and the study uses the available preprocessed data.Source: §2.1 · p. 3
2.2 Training Objectives
The decoder maps an fNIRS signal to a vector approximating the presented word’s embedding. The section introduces training objectives and defines quantities used in the batch-based objective.
2.2 Training Objectives
- The neural decoder maps fNIRS signals to vectors that approximate embeddings of the presented stimuli.The decoder is parameterized as a function from a T-by-n signal to a vector in R^d.Source: §2.2 · p. 3
- The training setup compares MSE with another objective for learning the decoder’s embedding predictions.The passage identifies MSE as one of the objectives under comparison.Source: §2.2 · p. 3
- For a training batch, the notation defines each prediction and target embedding, their cosine similarity, and a learnable temperature.The batch contains N samples; the temperature is t = e^τ.Source: §2.2 · p. 3
2.3 Word Embeddings
The study uses GloVe-50 and T5 representations as word-embedding targets. The targets differ in dimensionality and in how their representations are produced.
2.3 Word Embeddings
- GloVe-50 supplies 50-dimensional vectors learned from global word co-occurrence statistics.It is used following prior work for its widespread use and adequate vocabulary coverage.Source: §2.3 · p. 4
- T5 targets are 1024-dimensional representations extracted from the middle layer of the pretrained t5-large encoder.Each stimulus word is encoded in isolation and without context, following prior work.Source: §2.3 · p. 4
- For T5 words split into multiple tokens, the study averages their token embeddings.Source: §2.3 · p. 4
2.4 Evaluation Metrics
The evaluation uses pairwise matching to test whether two predictions align with their correct word embeddings, and open-vocabulary retrieval to test whether a trial’s word ranks among the category’s candidates. Statistical significance is assessed across subject-level scores against chance.
2.4.1 Matching Score
- Matching score is the fraction of subject-level trial pairs for which the summed cosine similarity of correct prediction–embedding pairings exceeds that of cross-pairings.The comparison is between the two ground-truth embeddings and their predictions.Source: §2.4.1 · p. 4
2.4.2 Open-Vocabulary Retrieval
- Open-vocabulary retrieval ranks the full vocabulary by cosine similarity and counts a trial correct when its true word appears in the top k.The chosen k equals the number of words per category, except Zinszer uses k = 2 for its two categories of four words.Source: §2.4.2 · p. 4
2.4.3 Statistical Analysis
- Significance is tested with a one-sided Wilcoxon signed-rank test of per-subject scores against chance, with p ≤ 0.05 significant and 0.05 < p < 0.10 marginal.The test reflects the small subject counts of 5 to 20.Source: §2.4.3 · p. 4
2.5 Experimental Settings
The experimental settings standardize the fNIRS input across datasets and compare MSE and CLIP losses using the same Bi-LSTM architecture and hyperparameters.
Input
- All datasets use the 1 s HbO window from 6.5 to 7.5 s after stimulus onset, excluding HbR to match Cao’s data and keep inputs identical.The window follows.Source: §2.5 · p. 4
Ridge-regression baseline
- The ridge-regression baseline fits one regressor per embedding dimension, selecting α by leave-one-out cross-validation over a log grid from 10^-2 to 10^8.The procedure follows.Source: §2.5 · p. 4
Bi-LSTM decoder; Training objectives
- The Bi-LSTM uses Adam with plateau-based learning-rate reduction and early stopping, while MSE and CLIP variants differ only in loss.MSE includes an uncentered-covariance regularizer weighted by γ; the architecture and hyperparameters follow the same pipeline for both objectives.Source: §2.5 · pp. 4–5
2.6 Average-Trial Pipeline
The average-trial pipeline reproduces prior neural-decoding tests on Cao and uses leave-two-words-out and leave-two-categories-out evaluations; those tests are restricted to Cao because they require many categories.
2.6 Average-Trial Pipeline
- The Cao benchmark averages seven trials per word into 50 word-vectors per subject and uses the Bi-LSTM configuration from.The configuration has 5 layers, hidden size 100, learning rate 4.74×10^-3, weight decay 2.72 × 10^-2, and γ = 2.59 for MSE.Source: §2.6 · p. 5
- Leave-two-words-out evaluation trains on 48 word-vectors and scores each held-out pair across 1,225 word pairs.The within-category and between-category scores average over 100 and 1,125 pairs, respectively.Source: §2.6 · p. 5
- Leave-two-categories-out evaluation holds out two of ten categories and averages scores across 45 pairs among the ten held-out words.These tests require many categories, so they are run only on Cao; Nathan and Zinszer use retrieval alone because they have three and two categories, respectively.Source: §2.6 · p. 5
2.7 Single-Trial Pipeline
The single-trial pipeline replaces repeated-trial pairwise evaluation with retrieval from the full vocabulary, addressing limitations in real-world use and scaling of the average-trial protocol.
2.7 Single-Trial Pipeline
- The average-trial pipeline presupposes repeated word presentations, while pairwise forced choice does not identify a word from the full vocabulary.Leave-two-words-out requires 15 models per subject for Nathan, 28 for Zinszer, and 1,225 for Cao, increasing to 61,075 for Cao without trial averaging.Source: §2.7 · p. 5
- All models are re-evaluated on single trials using retrieval from the full vocabulary, with each subject’s trials split word-stratified into 80/10/10 train, validation, and test sets.Source: §2.7 · p. 5
- The pipeline sweeps loss, depth, hidden size, learning rate, and embedding target, tunes γ for MSE, and tests the configuration with the best validation top-k accuracy once.Source: §2.7 · p. 5
3 RESULTS
Across the three result settings, Bi-LSTM with CLIP is the most consistent decoder, while retrieval significance is confined to GloVe-50. Average-trial matching favors T5 for ridge and shows a marginal zero-shot result; single-trial zero-shot decoding falls to chance.
3.1 Average-Trial Pipeline
- With average trials, T5 improves ridge matching scores across all three experiments, while Bi-LSTM (CLIP) with T5 has the highest leave-two-categories-out mean (0.59 ± 0.11), marginally above chance.Ridge with T5 is significantly above chance on within- and between-category tests; Bi-LSTM (MSE) with GloVe-50 is significant only within-category.Source: §3.1 · pp. 5–6, Figure 1 · p. 6
3.2 Single-Trial Pipeline
- With single trials, Bi-LSTM (CLIP) is the only configuration significantly above chance on within- and between-category tests with both embeddings; every configuration falls to chance on leave-two-categories-out.The reported comparison suggests trial averaging may raise the signal-to-noise ratio needed for zero-shot decoding.Source: §3.2 · p. 6, Figure 2 · p. 6
3.3 Open-Vocabulary Retrieval
- Bi-LSTM (CLIP) with GloVe-50 is the strongest retrieval decoder on two datasets, reaching the highest Cao accuracy and the only significant Nathan result; every significant retrieval result uses GloVe-50.On Zinszer, Bi-LSTM (MSE) with GloVe-50 is the only configuration above chance (0.27 ± 0.07, p = 0.028), although Bi-LSTM (CLIP) with GloVe-50 has the highest mean (0.28 ± 0.11).Source: §3.3 · pp. 6–7, Table 1 · p. 7, Table 2 · p. 7
4 Conclusion
Bi-LSTM with CLIP is the most robust decoder across the reported fNIRS experiments, while embedding performance differs by metric. The findings are preliminary given the small datasets and word-perception setting.
4 Conclusion
- Bi-LSTM (CLIP) is the only decoder above chance on single-trial matching, significant on retrieval for two of three datasets, and matches or exceeds the best single-subject retrieval score on every dataset.The authors describe the contrastive objective as suited to decoding because it optimizes whether a prediction is closer to its target than to alternatives.Source: §4 · p. 7
- T5 gives the higher matching score, whereas GloVe-50 gives more reliable retrieval despite its lower dimensionality and T5 token averaging for multi-token words.Source: §4 · p. 8
- The results are based on small datasets, and confirmation will require larger datasets.Source: §4 · p. 8
- The datasets use healthy subjects, language perception rather than production, and isolated words rather than natural speech, limiting clinical translation and task scope.Source: §4 · p. 8