PaperlayerView exact revision on arXiv

Source-linked AI summary

Brain-IT-VQA: From Brain Signals to Answers

Roman Beliy, Matias Cosarinsky, Oliver Heinimann, Navve Wasserman, Michal Irani

arXiv:2605.29588v2cs.CVcs.AIq-bio.NC

TL;DR

A longstanding neuroscience challenge is determining what visual information is represented in the brain. Brain-IT-VQA decodes fMRI into language for visual question answering, while NSD-VQA provides a controlled benchmark; the model outperforms prior methods across captioning and VQA.

  • Problem

    A longstanding neuroscience challenge is determining what visual information is represented in the brain, from broad semantic content to specific visual properties.

    Source: §1 · p. 1

  • Method

    Brain-IT-VQA decodes fMRI into language for captioning and visual question answering, while NSD-VQA provides about 20 question-answer pairs per image across 20 controlled categories.

    Source: §3.1 · p. 4, §7 · p. 9

  • Results

    Brain-IT-VQA outperforms prior methods across captioning and VQA; it gains +4.81 accuracy on VQA-v2 over MindLLM.

    Source: §5.2 · p. 6

  • Takeaways & Limitations

    NSD-VQA lets researchers evaluate which visual and semantic information is decodable from fMRI and use VQA to probe visual-representation organization.

    Source: §1 · pp. 2–3

    The masking-based analysis is a demonstration rather than definitive functional mapping, and its attributions depend on VQA model quality and NSD test-set coverage.

    Source: A Limitation · p. 15

Abstract

from arXiv · show

Decoding visual content from fMRI signals recorded while a person views images, and specifically answering questions about the seen images, is a long-standing challenge. While significant progress has been made in recent years in visual question answering (VQA) from fMRI, performance remains limited. Moreover, although recent models can make increasingly accurate predictions, they have rarely been used as tools for understanding the structure of visual representations in the brain. We present Brain-IT-VQA, a framework for visual question answering from fMRI. Building on the Brain Interaction Transformer (Brain-IT), our method decodes language tokens from brain activity and integrates them with a language model to answer visual questions. Our model substantially outperforms previous fMRI-based captioning and VQA approaches. We further introduce NSD-VQA, a new dataset and benchmark for visual question answering from fMRI. Unlike existing image-fMRI VQA datasets, which typically provide only a few broad and weakly controlled questions per image, NSD-VQA provides on average 20 question-answer pairs per image across 20 controlled question categories that disentangle multiple levels of visual understanding. This enables more reliable and interpretable evaluation despite limited fMRI test data. Together, Brain-IT-VQA and NSD-VQA provide both a strong predictive framework and a tool for studying brain representations. Using this benchmark, we quantify which forms of visual and semantic information can be reliably decoded from fMRI responses to natural images. We further analyze the contributions of different brain regions across question types.

1 Introduction

Brain-IT-VQA answers visual questions directly from fMRI, while NSD-VQA supports controlled evaluation of the visual and semantic information decoded from brain activity. Together, they make VQA a tool for studying how visual representations are organized in the brain.

1 Introduction

  • Brain-IT-VQA decodes language-conditioning representations from fMRI and combines them with a pretrained language model to answer questions without explicit image reconstruction, achieving state-of-the-art fMRI-based captioning and VQA performance.It builds on the Brain Interaction Transformer introduced in Brain-IT [1], which represents fMRI through shared functional voxel groups and integrates neural information across subjects.Source: §1 · p. 2
  • NSD-VQA provides an average of 20 question-answer pairs per image across controlled categories, enabling more informative and reliable evaluation despite small fMRI test sets.Categories cover objects, attributes, spatial relations, counting, actions, and scene-level information; existing benchmarks typically offer only a few broad or weakly controlled questions per image.Source: §1 · p. 2
  • The paper evaluates decodable visual and semantic information across question categories and relates predictions to learned functional voxel groups and established brain regions.It systematically studies which content can be reliably inferred from fMRI responses to natural images and how functional brain regions support different question types.Source: §1 · pp. 2–3
  • The framework positions visual question answering not only as brain decoding but also as a way to probe the organization of visual representations in the human brain.The broader challenge is determining how visual information, from semantic content to specific properties such as color, shape, and spatial layout, is represented across cortex.Source: §1 · pp. 1, 3

2 Related Work

The section positions Brain-IT-VQA among vision-language models and fMRI-decoding work, emphasizing its modular language-model interface, controlled evaluation dataset, and analysis of brain-region contributions. It identifies gaps in prior language-based decoding, VQA evaluation, and interpretability methods.

Vision-Language Models for VQA (on images)

  • Using InstructBLIP [25], Brain-IT-VQA injects fMRI-derived tokens through a modular frozen Q-Former without changing LLM weights, unlike direct-projection models [20] [21].Direct-projection models require LLM fine-tuning on image-specific features, making substitution of fMRI representations harder.Source: §2 · p. 3

Vision-Based Brain Decoding

  • Visual brain decoding has progressed from handcrafted and deep visual features to end-to-end methods, generative-model latents, diffusion reconstruction, and cross-subject generalization.Language-based approaches also align fMRI with visual or textual embeddings, use visual-language models downstream, or decode text end to end with language models.Source: §2 · p. 3

Multimodal and Language-Based Decoding

  • MindLLM [15] is the closest and strongest prior method, while Brain-IT-VQA uses functional clustering, dedicated cross-attention, and InstructBLIP [25] instead of anatomical parcellations and prepended Brain tokens.Functional clustering was shown to be superior for image decoding [1].Source: §2 · p. 3

Datasets and Evaluation for fMRI Decoding

  • NSD-VQA addresses the lack of controlled fMRI-VQA categories by probing distinct levels of visual understanding, unlike BrainHub [13] captioning and grounding or BrainChat broad COCO VQA questions.Its structured question types support fine-grained evaluation of what can and cannot be decoded from fMRI responses to natural images.Source: §2 · p. 4

Interpretability and Brain-Region Contributions

  • Brain-IT-VQA enables direct quantification of functionally organized voxel-cluster contributions across question categories, addressing prior methods’ lack of systematic brain-region-to-information analysis.Prior approaches often use global embeddings that obscure individual voxel or functional-region contributions.Source: §2 · p. 4

3 Method

Brain-IT-VQA decodes fMRI activity into captions or answers to textual image queries using BIT-L and InstructBLIP. Its two-pathway architecture extracts task-relevant representations, while two-stage training uses fMRI data and predicted responses for additional images.

3.1 Overview of our approach

  • BIT-L extends the Brain Interaction Transformer [1] to decode fMRI signals into captions or answers to textual image queries, integrating the predictions with InstructBLIP [25].Functionally similar voxels are clustered across subjects and summarized as Brain Tokens for attention-based processing.Source: §3.1 · p. 4

3.2 Model Architecture

  • Two complementary pathways extract CLIP-aligned representations and direct conditioning tokens from Brain Tokens; their averaged outputs and the textual query condition the language model.The CLIP-aligned pathway uses InstructBLIP’s Q-Former, adapted with LoRA and conditioned on the query; the direct pathway predicts task-specific soft prompts.Source: §3.2 · p. 4

3.3 Training Setup

  • Training first supervises BIT-L with separate MSE losses for CLIP visual tokens and Q-Former conditioning tokens, then jointly fine-tunes BIT-L and the Q-Former with LoRA for captioning and VQA.The rest of InstructBLIP’s LLM remains frozen in both stages.Source: §3.3 · p. 5
  • To augment limited subject-specific data, training uses predicted fMRI responses for additional COCO images, including approximately 120k unlabeled images during pretraining.In the second stage, all COCO images outside validation and test sets are used; responses are predicted with the Image-to-fMRI encoder [37].Source: §3.3 · p. 5

4 NSD-VQA Dataset

NSD-VQA provides approximately 20 question-answer pairs per image across 20 controlled categories for evaluating visual and semantic information decoded from fMRI. Its annotations and questions are automatically generated, verified, and designed to support targeted analysis.

Dataset overview

  • NSD-VQA pairs approximately 20 questions per image across 20 controlled categories, using NSD images with fMRI recordings to support targeted evaluation of visual and semantic understanding.The dataset is publicly available.Source: §4 · p. 5

Annotation pipeline

  • Structured annotations from Qwen3-VL-8B [43] capture salient objects and attributes including identity, counts, semantic categories, color, spatial position, actions, interactions, and scene descriptors.Source: §4 · p. 5

Filtering and verification

  • Object-count annotations are retained only when Qwen3-VL-8B and Gemma-4-31B-it [44] agree, while BGE merges semantically redundant labels into a unified vocabulary.Object presence is also checked across both model predictions.Source: §4 · p. 5

Question & Answer generation

  • Questions cover object properties, spatial cues, scenes, categories, and human attributes; conditional generation, negative examples, support thresholds, and answer balancing constrain evaluation.Question types with fewer than 50 instances are discarded, as are questions where one answer class exceeds 70% of instances; NSD-VQA-FS provides full-sentence responses rewritten with Llama-3.2-3B [46].Source: §4 · pp. 5–6

Summary

  • By grounding targeted question categories in explicit annotations, NSD-VQA enables systematic analysis of which visual and semantic information can be reliably inferred from fMRI signals.Source: §4 · p. 6

5 Experiments

Brain-IT-VQA outperforms prior methods across captioning and VQA benchmarks, while NSD-VQA analyses show that decoding accuracy varies with question type and model components. Experiments use NSD fMRI responses and evaluate both short-answer accuracy and text-generation metrics.

5.1 Experimental Setup

  • Experiments use the 1,000 NSD images shared across subjects as the test set and evaluate subjects 1, 2, 5, and 7.Short-answer tasks use accuracy; captioning and full-sentence generation use BLEU, METEOR, ROUGE-L, CIDEr, and SPICE. Results for VQA-v2, FSVQA, and captioning are for subject 1, while NSD-VQA variants average across the four subjects.Source: §5.1 · p. 6

5.2 Results

  • +3.57 BLEU-4 and +5.28 METEOR over MindLLM on COCO captioning, and +4.81 accuracy over MindLLM on VQA-v2.Brain-IT-VQA leads across all captioning metrics and VQA benchmarks; it also outperforms MindLLM across metrics on NSD-VQA and NSD-VQA-FS, averaged over subjects 1, 2, 5, and 7.Source: §5.2 · p. 6, Table 1 · p. 7, Table 2 · p. 7

5.3 Decoding Performance by Question Category

  • Binary questions reach 79–93% accuracy, while open-ended color (47.83%), food (54.02%), and action (66.35%) questions are harder.Scene questions score 93.00%, versus 73.56% for position and 71.56% for counting; animal Y/N reaches 89.62% compared with 62.26% for open-ended animal questions, indicating output-space complexity limits performance.Source: §5.3 · pp. 7–8, Table 4 · p. 8

5.4 Ablations

  • +1.35 VQA accuracy from the Q-Former and +3.79 from external data augmentation; removing BIT-L alignment or end-to-end training degrades performance by 16.93 and 33.6, respectively.Brain-IT-VQA also outperforms InstructBLIP applied to Brain-IT reconstructed images, although Brain-IT (Images) remains stronger than previous brain-to-VQA methods.Source: §5.4 · p. 8

6 Decoding Contribution Analysis

Brain-IT-VQA estimates brain-region contributions to decoding across NSD-VQA question categories. Food and holding questions show distinct spatial contribution patterns, suggesting that different visual and semantic information relies on partially different brain representations.

Overview

  • The framework estimates each region’s contribution across controlled NSD-VQA question categories to analyze which regions encode information relevant to visual and semantic understanding.Source: §6 · p. 8

Method

  • Randomized masking estimates marginal contributions across many region coalitions because single-region masking can underestimate contributions when brain representations are distributed and redundant.Source: §6 · p. 9

Technical Details

  • 10,000 randomized masking trials over 128 functional clusters and 200 NSD test stimuli provide data for ridge regression estimates of per-category cluster contributions.The regression controls for the other clusters’ contributions; the approach is inspired by occlusion sensitivity [63] and local surrogate modeling [64].Source: §6 · p. 9

Analysis

  • Holding questions show spatially concentrated contributions in a few regions, while food-question contributions are more distributed across ventral visual regions.The analysis covers subject 1; additional visualizations across question categories, subjects, and functional ROIs appear in Appendix F.Source: §6 · p. 9

7 Conclusion · Appendix

The conclusion presents Brain-IT-VQA as a state-of-the-art fMRI captioning and VQA framework and introduces NSD-VQA as a larger, controlled benchmark. It also reports initial evidence that brain regions contribute selectively to visual and semantic understanding, while calling for systematic validation of those attribution results.

7 Conclusion

  • Brain-IT-VQA performs fMRI-based visual captioning and VQA, outperforming previous methods by a large margin.The authors describe it as a state-of-the-art framework.Source: §7 · p. 9
  • NSD-VQA provides approximately 20 question-answer pairs per image for 73K NSD image-fMRI pairs.The dataset is automatically curated and described as extensive.Source: §7 · p. 9
  • NSD-VQA spans 20 controlled question categories, enabling more reliable and interpretable fMRI VQA evaluation.The authors describe this as an improvement over prior evaluation.Source: §7 · p. 9
  • The benchmark enables analysis of how brain-region contributions vary across question types.The conclusion frames this as analysis of brain representations.Source: §7 · p. 9
  • Initial masking-based attribution results indicate that different regions contribute selectively to visual and semantic understanding.These are presented as initial results for attributing information content to functional brain regions.Source: §7 · p. 9
  • Future work will systematically evaluate attribution results against functional neuroimaging literature and established region-function mappings.The evaluation is intended to assess how well identified contributions align with known mappings.Source: §7 · p. 9
  • That evaluation will also assess whether the attribution results discover new region-function mappings.The conclusion leaves open whether such discoveries will emerge.Source: §7 · p. 9

A Limitation · B Model Implementation Details · B.1 Architecture Details

The section qualifies the model’s assumptions and interpretability limits, then describes a Brain-IT-based architecture with separate visual-token and LLM-conditioning outputs. Its implementation uses a frozen InstructBLIP–Flan-T5-XL model.

A Limitation

  • The model assumes fMRI responses are memoryless and replicable, assumptions that may fail when effects such as representational drift occur.Replicability supports signal averaging to mitigate fMRI’s low SNR.Source: A Limitation · p. 15
  • Signal-quality variability makes regional contribution estimates less reliable for low-SNR subjects; interpretability is most reliable for high-SNR subjects.High SNR supports more accurate estimates of voxel functionality.Source: A Limitation · p. 15
  • The masking-based brain-region analysis is exploratory, not definitive, and depends on VQA model quality and NSD test-set coverage.The authors recommend interpreting its attribution estimates with caution.Source: A Limitation · p. 15

B.1 Architecture Details

  • Following BIT [1], the model uses 128 voxel clusters and a Brain Tokenizer that maps fMRI activations into 512-dimensional Brain Tokens through single-head graph attention.Source: §B.1 · p. 15
  • The model increases hidden dimensionality from 512 to 768 and uses 5 self-attention and 6 cross-attention blocks, each with 8 heads.The Cross-Transformer Module uses the self- and cross-attention blocks.Source: §B.1 · p. 15
  • A second independently learned cross-attention stack produces direct LLM-conditioning tokens of dimension 768 alongside visual CLIP tokens of dimension 1408.Both stacks share the same Brain Token representations, and all components use 768-dimensional hidden states to match the conditioning tokens.Source: §B.1 · p. 15
  • The implementation uses Salesforce/instructblip-flan-t5-xl [25], with a frozen EVA-CLIP ViT-g/14 encoder and Flan-T5-XL language model around a Q-Former.The image encoder produces 1408-dimensional features.Source: §B.1 · p. 15

B.2 Training Details … D.2 VQA Results Across Subjects

The supplementary sections specify training and compute procedures, describe NSD-VQA’s category structure and question-generation pipeline, and identify tables reporting captioning and VQA performance across subjects.

B.2 Training Details

  • Training reserves 10% of data for validation and uses AdamW at 5 × 10−4 for 50 epochs per stage, with 15-epoch stage-1 warmup and plateau-based decay.The learning rate drops by 0.1 after five validation epochs without improvement, and the best validation checkpoint is saved; stage-2 LoRA ranks and alphas are 2 for BIT-L and 4 for Q-Former.Source: §B.2 · p. 15

B.3 Compute Resources; C.3 Compute resources

  • A single H200 trains the joint-subject model in about 6 hours for stage 1 and 10 hours per dataset for stage 2; one image with 20 queries takes 0.1 seconds at inference.NSD-VQA construction required about 30 GPU hours, plus 8 GPU hours for its full-sentence variant.Source: §B.3 · p. 15, §C.3 · p. 18

C.1 Dataset Statistics

  • NSD-VQA’s 20 semantic question categories include open-ended and binary variants, yielding 23 displayed categories after filtering.Figure S1 shows the distribution of question-answer pairs across those displayed categories.Source: §C.1 · p. 16, Figure S1 · p. 16

C.2 Dataset Prompts

  • NSD-VQA prompts extract object- and scene-level annotations, verify counts through agreement between Qwen3-VL-8B and Gemma-4-31B-it, then generate question-answer pairs from templates.The annotations include object identity, attributes, actions, spatial position, and scene context; object presence is checked by requiring non-zero counts from both models.Source: §C.2 · p. 16

D.1 Captioning Across Subjects

  • Table S1 reports Brain-IT-VQA captioning results across subjects 1, 2, 5, and 7; the supplied passage gives no metric values.The table caption identifies its scope but does not provide results that support a performance comparison.Source: §D.1 · p. 19, Table S1 · p. 19

D.2 VQA Results Across Subjects

  • Table S2 reports Brain-IT-VQA performance across datasets and subjects 1, 2, 5, and 7; the supplied passage gives no metric values.The table caption identifies its scope but does not provide results that support a performance comparison.Source: §D.2 · p. 19, Table S2 · p. 19

D.3 NSD-VQA Results per category · D.4 NSD-VQA Question-Only Sanity Check

The section reports NSD-VQA performance by category across four subjects and compares Brain-IT-VQA with MindLLM for Subject 1. It also evaluates a question-only baseline and explains why some binary categories exceed chance without brain input.

D.3 NSD-VQA Results per category

D.4 NSD-VQA Question-Only Sanity Check

  • The question-only sanity check evaluates the model without brain input, conditioning only on the textual question.Source: §D.4 · p. 20, Table S5 · p. 20
  • Some binary (Y/N) categories achieve slightly above-chance performance because the evaluation split is not perfectly balanced between positive and negative answers.Source: §D.4 · p. 20
  • Although balancing is enforced at the full-dataset level, evaluation-split imbalance lets the question-only baseline partially exploit answer-frequency priors; Table S5 reports no-fMRI accuracy by category.Source: §D.4 · p. 20, Table S5 · p. 20

E Qualitative results … F.4 Functional ROI-Level Contributions

Supplementary results show additional fMRI-decoded descriptions and question answers, while contribution analyses reveal category- and subregion-dependent patterns, broad cross-subject consistency, and limitations of coarse ROI analysis.

E Qualitative results

F.1 Masking Procedure

  • BIT-L supports variable voxel counts because its graph neural network input can be masked by excluding selected voxels.Source: §F.1 · p. 23

F.2 Additional Brain Contribution Maps: Subject 1

  • Contribution patterns are concentrated in a few regions for vehicles, holding, and electronics, but more distributed for animals and actions.Source: §F.2 · p. 23
  • Different PPA subregions contribute to different question types, with distinct parts associated with location versus action questions.Source: §F.2 · p. 23

F.3 Voxel-Cluster Contributions: Additional Subjects

  • Across subjects, contributing regions are broadly consistent in general location but not identical in fsaverage space, as expected with individual variability in functional organization.Source: §F.3 · p. 26

F.4 Functional ROI-Level Contributions

  • At the ROI level, EBA dominates most question categories, but its large voxel count may explain this pattern rather than functional specificity.Source: §F.4 · p. 28
  • ROI analysis is coarser and less informative than cluster-level results, motivating finer-grained functional parcellations.Some ROIs contain thousands of voxels.Source: §F.4 · p. 28

Related papers

Loading 2605.29588v2…