PaperlayerView exact revision on arXiv

Source-linked AI summary

Brain-IT-VQA: From Brain Signals to Answers

Roman Beliy, Matias Cosarinsky, Oliver Heinimann, Navve Wasserman, Michal Irani

arXiv:2605.29588v2cs.CVcs.AIq-bio.NC

TL;DR

fMRI-based VQA remains limited and has rarely been used to study the organization of visual representations. The paper introduces Brain-IT-VQA, which decodes language tokens from brain activity for captioning and VQA, and NSD-VQA, a controlled benchmark; it reports stronger performance and region-sensitive analyses of decodable information.

  • Problem

    Existing fMRI VQA models have limited performance and provide insufficiently controlled evaluation for studying visual representations across question types and brain regions.

  • Method

    Brain-IT-VQA extends Brain-IT to predict language tokens from fMRI and integrates them with InstructBLIP, while NSD-VQA supplies controlled question categories for analysis.

  • Results

    Brain-IT-VQA achieves state-of-the-art captioning and VQA performance, including +3.57 BLEU-4 and +5.28 METEOR over MindLLM for captioning and +4.81 VQA-v2 accuracy over MindLLM.

  • Takeaways & Limitations

    Together, Brain-IT-VQA and NSD-VQA support both direct language decoding from fMRI and systematic study of visual and semantic brain representations.

  • Takeaways & Limitations

    Masking-based attribution is a demonstration rather than definitive functional mapping and depends on VQA-model quality and NSD test-set coverage.

Abstract

from arXiv · show

Decoding visual content from fMRI signals recorded while a person views images, and specifically answering questions about the seen images, is a long-standing challenge. While significant progress has been made in recent years in visual question answering (VQA) from fMRI, performance remains limited. Moreover, although recent models can make increasingly accurate predictions, they have rarely been used as tools for understanding the structure of visual representations in the brain. We present Brain-IT-VQA, a framework for visual question answering from fMRI. Building on the Brain Interaction Transformer (Brain-IT), our method decodes language tokens from brain activity and integrates them with a language model to answer visual questions. Our model substantially outperforms previous fMRI-based captioning and VQA approaches. We further introduce NSD-VQA, a new dataset and benchmark for visual question answering from fMRI. Unlike existing image-fMRI VQA datasets, which typically provide only a few broad and weakly controlled questions per image, NSD-VQA provides on average 20 question-answer pairs per image across 20 controlled question categories that disentangle multiple levels of visual understanding. This enables more reliable and interpretable evaluation despite limited fMRI test data. Together, Brain-IT-VQA and NSD-VQA provide both a strong predictive framework and a tool for studying brain representations. Using this benchmark, we quantify which forms of visual and semantic information can be reliably decoded from fMRI responses to natural images. We further analyze the contributions of different brain regions across question types.

1 Introduction

Brain-IT-VQA addresses limited and weakly controlled fMRI-based VQA by combining a predictive decoding framework with NSD-VQA, a benchmark for probing visual representations.

  • Decoding fMRI can reveal visual information ranging from broad semantic content to specific properties such as color, shape, spatial layout, and object attributes.
  • Existing fMRI captioning and VQA models remain limited and are not optimized for neuroscientifically informative question types.
  • Brain-IT-VQA provides end-to-end visual question answering from fMRI, while NSD-VQA enables controlled evaluation of visual and semantic information.
  • Together, the framework and benchmark use VQA to probe the organization of visual representations in the human brain.
  • The paper reports state-of-the-art fMRI VQA, a controlled benchmark, systematic decoding analyses, and interpretable brain-region contribution estimates.

2 Related Work

Vision-language models connect visual encoders with language models through several architectures, and Brain-IT-VQA builds on a modular frozen interface for fMRI-derived representations.

  • Vision-language models differ in how they connect visual and language modalities, including cross-attention gating, MLP projection, joint training, and query-token interfaces.
  • Brain-IT-VQA builds on InstructBLIP because its Q-Former offers a modular, frozen interface to the language model that can accept fMRI-derived token representations.
  • Prior fMRI question-answering work used broad COCO VQA questions, while controlled categories for disentangling visual understanding remained unavailable.

3 Method

Brain-IT-VQA converts fMRI activity into language-conditioning representations through Brain Tokens and integrates them with InstructBLIP for captioning and VQA.

  • 3.1 Overview of our approach: Brain-IT-VQA predicts language tokens from fMRI signals and uses InstructBLIP to generate captions or answers to textual image queries.
  • 3.1 Overview of our approach: BIT-L groups functionally similar voxels into shared clusters, summarizes each cluster as a Brain Token, and uses attention to produce task-relevant representations.
  • 3.1 Overview of our approach: The architecture extends Brain-IT with complementary prediction pathways that extract different brain-activity representations for conditioning the language model.
  • Training first predicts CLIP and InstructBLIP Q-Former targets from fMRI, then jointly fine-tunes BIT-L and the Q-Former end to end.
  • Additional natural images without fMRI recordings augment training by using an image-to-fMRI encoder to predict their brain responses.

4 NSD-VQA Dataset

NSD-VQA turns NSD image-fMRI pairs into a structured benchmark with approximately 20 verified question-answer pairs per image across controlled visual and semantic categories.

  • NSD-VQA is built from 73K NSD images with fMRI recordings and provides approximately 20 question-answer pairs per image across 20 controlled categories.
  • Vision-language models generate structured annotations covering object identity, counts, semantic categories, color, spatial position, actions, and interactions.
  • Counts are retained only when two vision-language models agree, with additional checks for consistent object presence.
  • Template queries instantiate questions conditionally from relevant annotations, covering object properties, spatial cues, scene understanding, categories, and human attributes.
  • By decomposing VQA into targeted categories grounded in explicit annotations, NSD-VQA supports systematic analysis of information inferred from fMRI.

5 Experiments

Experiments show strong performance across captioning and VQA benchmarks, while NSD-VQA reveals that decoding is easier for coarse and binary information than for fine-grained semantic alternatives.

  • Captioning: Brain-IT-VQA achieves state-of-the-art COCO captioning performance, improving BLEU-4 by +3.57 and METEOR by +5.28 over MindLLM.
  • Visual Question Answering: +4.81 accuracy on VQA-v2 separates Brain-IT-VQA from the strongest baseline, MindLLM, with consistent FSVQA gains across accuracy and generative metrics.
  • Visual Question Answering: Brain-IT-VQA outperforms MindLLM across all NSD-VQA and NSD-VQA-FS metrics, with gains in both short-form and full-sentence settings.Results are averaged across subjects for these benchmarks.
  • Decoding Performance by Question Category: 79–93% accuracy on binary questions contrasts with lower open-ended performance for color (47.83%), food (54.02%), and action (66.35%).The category analysis uses controlled NSD-VQA questions to evaluate distinct visual and semantic information.
  • Decoding Performance by Question Category: 93.00% scene-level accuracy exceeds position (73.56%) and counting (71.56%), while animal Y/N (89.62%) exceeds open-ended animal questions (62.26%).The reported pattern indicates higher performance for global contextual and binary distinctions than for more complex semantic alternatives.
  • Ablations: Removing BIT-L alignment or end-to-end training reduces VQA accuracy by -16.93 and -33.6, while removing the Q-Former or external data augmentation changes it by +1.35 and +3.79.
  • Ablations: Direct Brain-IT VQA outperforms InstructBLIP applied to Brain-IT reconstructed images, although Brain-IT (Images) remains stronger than previous brain-to-VQA methods.

6 Decoding Contribution Analysis

The contribution analysis estimates how voxel clusters affect decoding across question categories while accounting for distributed and redundant brain representations. It finds category-specific contribution patterns, including more concentrated contributions for holding questions than for food questions.

  • Method: Marginal contribution measures the decoding-performance change when a brain region is added to a coalition of other regions.
  • Method: Randomized masking across many configurations addresses compensation among redundant brain regions better than masking a single region.
  • Technical Details: The analysis masks random subsets of 128 functional clusters and predicts responses for 200 NSD test stimuli across 10,000 trials.
  • Analysis: Holding questions show spatially concentrated contributions in a small number of regions, whereas food contributions are more distributed across ventral regions.The reported patterns suggest partially different brain representations for these question categories.

7 Conclusion

Brain-IT-VQA combines strong fMRI-based captioning and VQA performance with NSD-VQA, enabling interpretable analysis of decodable information and brain-region contributions across question types.

  • Brain-IT-VQA outperforms previous methods for visual captioning and VQA directly from fMRI recordings.
  • NSD-VQA provides approximately 20 question-answer pairs per image across 20 controlled question categories for more reliable and interpretable evaluation.
  • The benchmark supports analysis of how different brain regions contribute selectively to visual and semantic understanding across question types.

A Limitation

The framework assumes memoryless, replicable fMRI responses and interprets masking-based regional attributions cautiously because reliability depends on subject signal quality, model quality, and test-set coverage.

  • The model assumes that fMRI responses are memoryless and replicable across repeated image presentations.These assumptions support signal averaging but may not hold when representational drift affects responses over time.
  • Regional interpretability estimates are less reliable for subjects with poor fMRI signal quality.Voxel functionality is estimated more accurately for subjects with high SNR.
  • Masking-based attributions are a demonstration of brain-exploration potential rather than a definitive functional mapping.Their interpretation depends on the underlying VQA model and NSD test-set coverage.

B.2 Training Details

Training uses a two-stage Brain-IT-VQA pipeline with validation-based optimization, while NSD-VQA is constructed through structured annotation, cross-model verification, and template-based question generation.

  • Training Details: Ten percent of the training data is reserved for validation, supporting hyperparameter selection and early stopping.
  • Training Details: Both training stages run for 50 epochs with AdamW, stage-1 warmup, and validation-plateau learning-rate reduction.The learning rate is 5 × 10^-4, with a 0.1 reduction after five validation-plateau epochs.
  • Training Details: Stage 2 uses LoRA ranks and scaling values of 2 for BIT-L and 4 for Q-Former.
  • Training Details: Inference processes one image with 20 queries in 0.1 seconds after training on a single H200 GPU.Stage 1 takes approximately 6 hours, while stage 2 requires approximately 10 hours per dataset.
  • Dataset Construction: NSD-VQA displays 23 categories because some of its 20 semantic categories are split into open-ended and binary variants.
  • Dataset Construction: NSD-VQA annotations extract object and scene attributes, verify counts and presence with two models, and convert retained annotations into template-based question-answer pairs.The annotation process uses Qwen3-VL-8B and the verification stage also uses Gemma-4-31B-it1.
  • Dataset Construction: Dataset construction required approximately 30 GPU hours, with 8 additional GPU hours for the full-sentence variant.

D.3 NSD-VQA Results per category

The supplementary results report category-level NSD-VQA performance, comparisons with MindLLM, and a question-only sanity check whose above-chance binary performance reflects answer-frequency imbalance.

  • Table S3 reports NSD-VQA performance by category across subjects 1, 2, 5, and 7.
  • Table S4 compares Brain-IT-VQA with MindLLM for Subject 1, marking statistically significant improvements using paired bootstrap testing.The test uses 10,000 bootstrap samples and a significance threshold of p < 0.05.
  • The question-only baseline evaluates NSD-VQA without brain input, using only the textual question.
  • Some binary categories exceed chance because the evaluation split is not perfectly balanced, allowing answer-frequency priors to influence predictions.

E Qualitative results

Brain-IT-VQA produces captions and answers to visual questions from fMRI signals, while analyses show that brain-region contributions vary across question categories and are more informative at the voxel-cluster level than at the ROI level.

  • Qualitative results: Brain-IT-VQA generates image descriptions and visual-question answers from fMRI signals, using voxel masking within its graph-based input processing.The qualitative examples use questions from NSD-VQA-FS, while the model processes variable voxel sets by excluding masked voxels from the input graph.
  • Voxel-cluster contributions: Different question categories show distinct voxel-cluster contribution patterns, with some concentrated in a few regions and others distributed more broadly.Vehicles, holding, and electronics show concentrated contributions, whereas animals and actions show more distributed patterns.
  • Voxel-cluster contributions: Subregions within the same functional area can contribute differently across question types, including distinct parts of the parahippocampal place area.The passage specifically links different PPA subregions to location and action questions.
  • Additional subjects: Contributing regions are broadly consistent in general location across subjects but are not identical in fsaverage space.The variation is consistent with inter-subject variability in functional organization.
  • Functional ROI-level contributions: ROI-level analyses are less informative than cluster-level analyses because ROIs coarsely partition activity, and EBA dominance may reflect its larger voxel count.These observations motivate using finer-grained functional parcellations for interpreting regional contributions.
  • Voxel-cluster contributions: Figure S11 summarizes how voxel-cluster importance changes across object, attribute, and relation questions, linking distinct regions to different visual and semantic processes.The figure presents category-dependent cluster contributions rather than a single uniform regional pattern.
Loading 2605.29588v2…