PaperlayerView exact revision on arXiv

Source-linked AI summary

MindVLM: Neural-Grounded Visual Captioning via Subject-Aware Semantic Evidence Selection

Zixiang Yin, Yu-Ping Wang, Zhengming Ding

arXiv:2610.12673v1cs.CE

TL;DR

NSD pairs fMRI recorded from cropped images with captions describing full scenes, while existing approaches can overlook subject-specific neural evidence. MindVLM selects visual semantics supported by each subject’s brain activity and uses a frozen VLM to generate descriptions. It improves over base VLMs across three architectures and remains competitive with text-supervised captioning methods on semantic metrics.

  • Problem

    NSD pairs fMRI recorded from cropped image regions with full-scene MS-COCO captions, while existing methods can overlook subject-specific perceptual and semantic representations.

    Source: §1 · p. 2

  • Method

    MindVLM maps brain responses to subject-aligned visual tokens and uses a frozen VLM to generate descriptions grounded in neural-supported visual evidence.

    Source: §1 · p. 2

  • Results

    MindVLM consistently improves over base VLMs across three architectures and remains competitive with text-supervised methods on semantic metrics, with remaining gaps concentrated on lexical-overlap metrics.

    Source: §1 · p. 3, §4 · p. 9

  • Takeaways & Limitations

    Neural-guided visual grounding offers a text-free approach to personalized brain captioning that handles partial observability and produces interpretable subject-specific evidence maps.

    Source: §1 · p. 2

    MindVLM requires the test image at inference and a separate brain encoder for each subject, so its setting is not directly comparable to classic fMRI-to-text captioning.

    Source: §A.9 · p. 30

Abstract

from arXiv · show

Translating visual brain activity into natural language offers a powerful window into human cognition and neural semantic representation. Most prior brain-captioning approaches project fMRI signals into language space using paired supervision. However, the widely used Natural Scenes Dataset (NSD) was built based only on image-fMRI pairs, while existing methods rely on MS-COCO captions as indirect textual supervision. Critically, during fMRI acquisition, participants observe only cropped image regions, whereas the associated MS-COCO captions describe full scenes. This mismatch creates a fundamental gap between neural representations and textual supervision, introducing an inherent semantic loss in NSD-based brain captioning. To mitigate this limitation, we reformulate brain captioning as Neural-Grounded Visual Captioning, where language is generated directly from visually grounded neural evidence rather than text supervision. We propose MindVLM, a framework that maps fMRI signals into subject-aware visual tokens and leverages a frozen vision language model (VLM) to produce interpretable and personalized descriptions. Instead of point-wise regression, MindVLM focuses on extracting semantically relevant visual cues from neural activity, naturally accommodating partial observability and inter-subject variability. Extensive experiments on NSD show that MindVLM consistently improves caption quality across multiple VLM backbones, including LLaVA-v1.5, Qwen2.5-VL, and Gemma3, beyond intrinsic model priors. Despite requiring neither text supervision nor an LLM during training and operating under an image-grounded inference paradigm distinct from conventional captioning methods, MindVLM remains competitive with supervised approaches, suggesting that subject-specific neural grounding provides an efficient, scalable, and interpretable path toward faithful brain-to-language generation.

1 Introduction

NSD’s cropped-image fMRI signals are paired with full-scene captions, creating a mismatch between neural evidence and textual targets. MindVLM reframes captioning as selecting subject-specific visual semantics supported by neural activity, then generating descriptions with a pretrained VLM.

1 Introduction

  • NSD records fMRI while participants view cropped MS-COCO images [29], but its caption targets describe full scenes, creating semantic loss and risking unsupported content.The mismatch can push models toward captions that exceed information supported by the brain signals.Source: §1 · p. 2, Figure 1 · p. 2
  • Existing methods often overlook subject-specific attention and interpretation, while reliably collecting fMRI-aligned textual reports is challenging.The resulting descriptions can resemble generic image captions rather than language grounded in an individual’s neural evidence.Source: §1 · p. 2
  • Neural-Grounded Visual Captioning selects visual semantics supported by neural activity and composes them into language with a pretrained VLM, rather than directly regressing fMRI to text.The image-grounded approach is designed to handle partial visual observability without costly subject-specific text annotations or an LLM during training.Source: §1 · p. 2
  • MindVLM maps brain responses to subject-aligned visual tokens and passes the resulting evidence representation to a frozen VLM for personalized descriptions.The framework is presented as text-free and subject-aware, with interpretable visual evidence maps.Source: §1 · pp. 2–3
  • MindVLM improves caption quality over each tested VLM baseline and remains competitive with text-supervised methods on SPICE and CIDEr.The authors also report an order-of-magnitude lower training cost.Source: §1 · p. 3

2 Proposed Method

MindVLM generates language from image-grounded neural evidence by aligning subject-specific fMRI and visual representations, then using selected visual tokens to guide captioning.

2.1 Problem Formulation

  • Neural-Grounded Visual Captioning maps an image-fMRI pair to a description; 12,901 of 73,000 NSD images (17.7%) exhibit semantic loss from cropping.Semantic loss is the fraction of objects losing more than 50% of their pixels during cropping.Source: §2.1 · p. 3, Figure 2 · p. 3

2.2 Framework Overview

  • MindVLM aligns subject-specific neural and visual representations, then uses brain-activated visual tokens as evidence for a frozen VLM rather than directly regressing fMRI to text.The framework reflects semantic loss, individual perceptual differences, and the availability of structured visual tokens.Source: §2.2 · pp. 3–4, Figure 3 · p. 4, §2.4 · p. 6

2.3 Neural-Visual Mapping

  • A trainable ViT-based encoder and frozen VLM image encoder produce neural and visual tokens, whose pooled global representations are aligned with a symmetric CLIP-style contrastive loss.The loss averages fMRI-to-image and image-to-fMRI objectives.Source: §2.3 · pp. 4–5

2.4 Subject-Aware Captioning

  • At inference, local neural-visual similarities select the top q% of pairs and aggregate them into a visual evidence mask; a frozen VLM turns selected tokens into personalized descriptions.The similarity matrix represents fine-grained correspondence between localized neural and visual tokens.Source: §2.4 · pp. 5–6

3 Experiments

Experiments on four NSD subjects assess MindVLM across VLM architectures, conventional brain-captioning methods, token-selection thresholds, and personalized outputs. They also examine training efficiency against MindLLM.

3.1 Experimental Setup

  • Evaluation uses standardized NSD splits and per-subject models for subj01, subj02, subj05, and subj07, with 36,000 training and 4,000 test fMRI-image pairs.The dataset captures eight subjects, but data availability limits this evaluation to four.Source: §3.1 · p. 6

3.3 Semantic Decoding Quality: Reference to Brain Captioning Methods

  • MindVLM exceeds all compared methods on SPICE and leads on ROUGE, but trails text-supervised methods on n-gram metrics in the reference comparison.The authors frame this as a reference, not a direct benchmark, because MindVLM uses the test image at inference while conventional methods decode from fMRI alone.Source: Table 2 · p. 7, §3.3 · p. 7

3.2 Neural-Guided Visual Captioning across VLM Architectures

  • Across CLIP, ViT, and SigLIP backbones, MindVLM improves most metrics over base VLMs; Gemma3 BLEU@2 is the exception, differing marginally (49.16 vs. 49.13).The reported CIDEr gains are 60.61 to 62.06 for LLaVA and 46.13 to 47.38 for Qwen2.5-VL.Source: §3.2 · pp. 6–7

3.4 Role and Choice of Token Selection q

  • At top 40–55% selected pairs, subjects average over 90% of peak BLEU-3 and SPICE; scores then flatten or slightly decline.The analysis reports that selecting more pairs generally raises both metrics, while additional tokens beyond this range may add little or introduce noise.Source: Figure 4 · p. 8, §3.4 · p. 8

3.5 Captioning by Subject-Specific Perception

  • At q = 5, the same stimulus yields subject-specific captions: subj02 gives a broad description, subj05 adds action detail, and subj07 mentions frosting.The examples connect differences in retained tokens and spatial focus with distinct generated descriptions.Source: Figure 5 · p. 8, §3.5 · p. 8

3.6 Paradigm Efficiency

  • Training uses neural-visual contrastive alignment without an LLM; Figure 6 compares MindVLM-Online and MindVLM-Offline with MindLLM on resource metrics.The reported metrics include peak GPU memory, FLOPs, and trainable and frozen parameters.Source: §3.6 · p. 9, Figure 6 · p. 9

4 Conclusion

MindVLM reframes brain decoding as selecting visual semantics supported by neural activity and uses them to generate subject-aware captions. It improves over base VLMs across three architectures and remains competitive with text-supervised methods on semantic metrics, though lexical-overlap metrics favor caption-trained models.

4 Conclusion

  • MindVLM aligns neural activity with spatial visual tokens and uses a frozen VLM to caption only neural-supported evidence for each subject.The framework treats brain decoding as semantic evidence selection under partial visual observability.Source: §4 · p. 9
  • MindVLM, a subject-aware framework for Neural-Guided Visual Captioning, consistently improves over the base VLM across three architectures and remains competitive with text-supervised brain-captioning methods on semantic metrics despite using no text supervision.Source: §4 · p. 9
  • MindVLM remains competitive with text-supervised brain-captioning methods on semantic metrics, while lexical-overlap gaps favor models trained to reproduce captions.Source: §4 · p. 9
  • The results indicate neural-guided visual grounding as an efficient, interpretable, and cognitively faithful pathway for personalized decoding.Source: §4 · p. 9

A.1 Related Works

Brain captioning generates natural-language descriptions of visual stimuli viewed during scanning. NSD provides a setting for this work, with MS-COCO images and human-written captions available for evaluation.

A.1 Related Works

  • Brain captioning generates natural-language descriptions of the semantic content of visual stimuli viewed by subjects inside the scanner.Source: §A.1 · p. 15
  • Brain-to-language research also studies neural decoding during story listening and sentence-by-sentence reading of visually presented text.Source: §A.1 · p. 15
  • The Natural Scenes Dataset uses MS-COCO images and provides human-written captions for evaluation.Source: §A.1 · p. 15

A.1.1 Modality Alignment

Prior work aligns neural activity with visual or vision-language representations, with approaches spanning shared representations, image reconstruction, and representation learning. MindVLM instead selects neurally supported visual concepts under partial observability using local visual tokens and global semantic representations.

A.1.1 Modality Alignment

  • Prior neural-visual alignment work broadly aims to improve correspondence fidelity or robustness for downstream decoding.Source: §A.1.1 · p. 15
  • UMBRAE [56] learns a shared visual-token latent space across subjects, while BrainOT [58] matches neural and visual feature distributions using optimal transport.Source: §A.1.1 · p. 15
  • MindEye2 [43] prioritizes brain-to-image reconstruction, and VINDEX [57] uses multi-granularity image features to recover visual details.Source: §A.1.1 · p. 15
  • MindBridge [54], MindArtist, BrainCLIP, and MindTuner [20] use manifold learning, optimal transport, contrastive learning, or visual pivots to strengthen neural-visual correspondence.Source: §A.1.1 · p. 15
  • MindVLM targets brain captioning under partial visual observability, selecting neural-supported visual concepts rather than recovering full image representations.It derives global semantic representations from local visual tokens rather than enforcing point-wise matching to global image embeddings.Source: §A.1.1 · p. 15
  • Unlike methods often assuming complete visual information, MindVLM uses token-level evidence selection to preserve interpretability without over-constraining neural representations to unobserved details.Source: §A.1.1 · p. 15

A.1.2 Brain-to-LLM Integration

Brain-to-LLM approaches inject fMRI into pretrained language-model token spaces, while visually grounded alternatives align neural activity with visual representations. MindVLM removes heavy LLMs from training and uses flexible local-token-derived alignment for interpretable captioning.

A.1.2 Brain-to-LLM Integration

  • OneLLM [23] maps fMRI into frozen LLaMA2 [52] token embeddings, while MindLLM [39] uses fMRI prefix tokens and instruction tuning for language-model decoding.Source: §A.1.2 · p. 16
  • MindGPT grounds neural encoding in CLIP semantics, MindSA masks unattended visual regions, and a related brain-VQA framework aligns fMRI with multi-level visual features before LLM input.The related framework uses a 3D ViT [13] for fMRI feature extraction.Source: §A.1.2 · p. 16
  • Text-supervised LLM integration requires a frozen LLM during training and risks computational cost and textual-prior biases unrelated to a subject’s perceptual state.Source: §A.1.2 · p. 16
  • UMBRAE [56] avoids explicit text decoding during training by mapping fMRI into a shared space approximating multimodal-LLM visual tokens.Source: §A.1.2 · p. 16
  • MindVLM avoids exact matching to frozen image or text features and aligns local-token-derived global representations to support interpretable captioning.The framework aims to preserve flexibility while maintaining semantic fidelity and to trace which visual tokens contribute to caption content.Source: §A.1.2 · p. 16

A.2.1 NSD Dataset

NSD pairs high-resolution fMRI recordings with MS-COCO images and captions, but participants viewed center-cropped images, creating a mismatch with captions describing full scenes.

A.2.1 NSD Dataset

  • NSD contains whole-brain 7T fMRI recordings from eight participants at 1.8 mm isotropic resolution across 70,566 trials.Each participant viewed 10,000 images across 30–40 scanning sessions; 9,000 images were subject-specific and 1,000 were shared.Source: §A.2.1 · p. 16
  • Images shown during scanning were center-cropped and resized to 425×425, while their five MS-COCO captions describe the original full scenes.Consequently, some captioned objects may not appear in the stimulus that elicited the neural response.Source: §A.2.1 · p. 16
  • Each image was presented for 3 seconds with an object-recognition task, and the dataset’s preprocessing yields high-quality single-trial beta estimates.The task was intended to sustain attention and probe recognition and memory processes.Source: §A.2.1 · p. 16
  • NSD provides anatomical MRI, resting-state scans, behavioral measures, and data in subject-native and standardized spaces, alongside manually annotated visual-region ROIs.The paper uses the nsdgeneral atlas, with voxel counts varying by subject.Source: §A.2.1 · p. 17

A.2.2 Implementation Details

The experiments use frozen LLaVA-v1.5-7B components and train a subject-consistent fMRI encoder for neural–visual alignment, taking about 45 minutes per subject.

A.2.2 Implementation Details

  • LLaVA-v1.5-7B’s language model and CLIP ViT-L/14-336 visual encoder remain frozen, while a four-layer, four-head Vision Transformer fMRI encoder is trained from scratch.Visual features are precomputed for NSD images and loaded during training.Source: §A.2.2 · p. 17
  • Training runs for 150 epochs, including 10 warm-up epochs, with batch size 1000, AdamW, and a cosine-decayed learning rate initialized at 7.5 × 10−3.Weight decay is 0.05 and gradient clipping is 0.8; hyperparameters are consistent across subjects unless otherwise specified.Source: §A.2.2 · p. 17
  • Training takes about 45 minutes per subject on two NVIDIA RTX 6000 Ada GPUs, and inference caps VLM generation at 32 new tokens.All experiments use random seed 2025.Source: §A.2.2 · p. 17

A.3 Detailed Captioning Performance By Subject

The appendix provides subject-specific captioning tables across thresholds and plots selected-token counts against q, but the supplied passages do not state numerical results or winners.

A.3 Detailed Captioning Performance By Subject

  • The detailed subject analysis reports captioning performance at all q% thresholds and includes six additional metrics.No specific scores or threshold winner are stated in the supplied prose.Source: §A.3 · p. 17
  • Tables 3–7 report NSD fMRI-to-MS-COCO caption reconstruction performance for each of four subjects and their average.The supplied table captions identify scope but provide no metric values or comparisons.Source: Table 3 · p. 17, Table 4 · p. 18, Table 5 · p. 18
  • Figure 7 plots the number of selected visual tokens against threshold q.The supplied caption does not report the plotted values or trend.Source: Figure 7 · p. 20

A.4 Visual Token Selection

The appendix examines semantic selection, fMRI–image matching, token thresholds, visualizations, and limitations in inference, model transfer, and evaluation. It also documents asset attribution and misuse-risk assessments, and reports that participant-instruction and IRB-related disclosures are absent while the research poses no participant risks.

A.4 Visual Token Selection

  • At q=5, selection typically retains 150–250 high-confidence tokens; increasing q expands perceptual coverage while adding variability.Token counts increase monotonically with q, and distributions are broadly consistent across subjects, with some inter-subject variation such as higher counts for subj01.Source: §A.4 · p. 20

A.5 Brain-Vision Matching

  • Correct fMRI-image pairs had mean similarity above 0.4, while incorrect pairs were near zero; retrieval remained strong bidirectionally across candidate pools up to 1,000.The analysis covers 1,000 test samples and approximately one million fMRI-image pairs; retrieval uses Recall@K at candidate pool sizes of 100, 200, 500, and 1,000.Source: Figure 8 · p. 21, §A.5 · p. 21, Table 8 · p. 21

A.6 Semantic Loss & Decoding Consistency

  • At q=95, no-semantic-loss samples scored approximately 0.47 ROUGE and 0.28 SPICE versus 0.45 ROUGE and 0.26 SPICE with semantic loss.Across q thresholds, samples without semantic loss outperform those with semantic loss, and both groups’ curves improve monotonically as q increases.Source: Figure 9 · p. 22, §A.6 · p. 22

A.7.1 Caption performance v.s. q for all subjects

  • Figures 10 and 11 plot visual-token ratio against top q% selected fMRI-image token pairs for scaled semantic metrics METEOR, CIDEr, SPICE and scaled n-gram metrics BLEU-1, BLEU-2, BLEU-3, BLEU-4, and ROUGE.Source: Figure 10 · p. 23, Figure 11 · p. 24

A.7.2 Captions performance on samples with/without semantic loss

  • Subject-level comparisons of samples with and without semantic loss extend the analysis to all six remaining caption metrics.Figures 12 and 13 display comparisons using n-gram and semantic metrics, respectively.Source: Figure 12 · p. 25, §A.7.2 · p. 25, Figure 13 · p. 26

A.8 Captioning by subject-specific perception

  • Additional examples illustrate personalized language generation and visual perception when 5% of neural-visual activations are retained for all subjects.The appendix provides further examples beyond those shown in the main paper.Source: Figure 14 · p. 27, §A.8 · p. 27, Figure 15 · p. 28

A.9 Limitations

  • MindVLM requires test images at inference, separate encoders per subject and VLM visual backbone, and is limited by frozen-VLM concepts; evaluation is only on NSD.The authors note that cross-subject or multi-subject training and generalization to EEG, MEG, or datasets with different cropping or annotation protocols remain unvalidated.Source: §A.9 · p. 30

2. Limitations

  • The authors report no statistical-significance checks, citing limited direct comparability to conventional models and the time required for testing.The checklist gives the large dataset as an additional reason for not conducting these tests.Source: §2 · p. 33
  • The authors say releasing their data or models poses no high risk for misuse.Source: §2 · p. 35
  • The paper reports that existing data and pretrained models are described and their original creators credited.Source: §2 · p. 35
  • The checklist marks full participant instructions and compensation details as absent, while stating the neural-signal research poses no participant risks.Source: §2 · p. 36
  • The checklist reports no description of participant risks or IRB approval, with the justification that the neural-signal research poses no participant risks.Source: §2 · p. 36

Related papers

Loading 2610.12673v1…