PaperlayerView exact revision on arXiv

Source-linked AI summary

MindLLM: A Subject-Agnostic and Versatile Model for fMRI-to-Text Decoding

Weikang Qiu, Zheng Huang, Haoyu Hu, Aosong Feng, Yujun Yan, Rex Ying

arXiv:2502.15786v2q-bio.NCcs.AIcs.LGeess.SP

TL;DR

Translating fMRI into text has academic and societal importance, but existing approaches can be limited to stimulus-image-related tasks. MindLLM pairs an fMRI encoder with an off-the-shelf LLM and neuroscience-informed attention; it outperforms baselines on downstream tasks and unseen-subject generalization.

  • Problem

    fMRI-to-text decoding has academic and societal importance, but existing approaches can be constrained to tasks tied to the current stimulus image and miss broader tasks such as recalling past visual experiences.

    Source: §1 · pp. 1–2

  • Method

    MindLLM pairs an fMRI encoder with an off-the-shelf LLM and uses neuroscience-informed attention to encode variable voxel inputs across subjects.

    Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 3

  • Results

    MindLLM outperforms baselines, with 12.0% average improvement in various downstream tasks and 24.5% improvement in generalization on unseen subjects.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Attention-weight visualizations provide interpretable insight into query patterns, including queries focused on PPA and FFA.

    Source: §4.8 · p. 9

    The authors leave applying domain-adaptation techniques as a direction for future work.

    Source: §4.4 · p. 7

Abstract

from arXiv · show

Decoding functional magnetic resonance imaging (fMRI) signals into text has been a key challenge in the neuroscience community, with the potential to advance brain-computer interfaces and uncover deeper insights into brain mechanisms. However, existing approaches often struggle with suboptimal predictive performance, limited task variety, and poor generalization across subjects. In response to this, we propose MindLLM, a model designed for subject-agnostic and versatile fMRI-to-text decoding. MindLLM consists of an fMRI encoder and an off-the-shelf LLM. The fMRI encoder employs a neuroscience-informed attention mechanism, which is capable of accommodating subjects with varying input shapes and thus achieves high-performance subject-agnostic decoding. Moreover, we introduce Brain Instruction Tuning (BIT), a novel approach that enhances the model's ability to capture diverse semantic representations from fMRI signals, facilitating more versatile decoding. We evaluate MindLLM on comprehensive fMRI-to-text benchmarks. Results demonstrate that our model outperforms the baselines, improving downstream tasks by 12.0%, unseen subject generalization by 24.5%, and novel task adaptation by 25.0%. Furthermore, the attention patterns in MindLLM provide interpretable insights into its decision-making process.

1. Introduction

MindLLM addresses the need for versatile fMRI-to-text decoding and unified architectures that handle subjects with different voxel inputs. It combines neuroscience-informed attention with Brain Instruction Tuning (BIT) and reports improvements on downstream tasks and unseen-subject generalization.

1. Introduction

  • Decoding needs vary from movement-intention control to interpreting abstract thoughts or memories, while stimulus-image mapping remains tied to the current image.The cited example, UMBRAE, cannot address broader tasks such as retrieving memories of past visual experiences.Source: §1 · p. 2
  • Subjects’ selected voxels vary in number and spatial distribution, while pooling or sampling can lose spatial information and unevenly represent brain areas.The paper identifies these issues as obstacles to a unified architecture.Source: §1 · p. 2
  • MindLLM pairs a subject-agnostic fMRI encoder with an off-the-shelf LLM, using neuroscience-informed attention and learnable queries to extract voxel features.Each query token attends to all voxels, aiming to minimize information loss from pooling or sampling.Source: §1 · p. 2, Figure 2 · p. 2
  • BIT trains on image-mediated tasks spanning perception, scene understanding, memory, knowledge retrieval, language, symbolic processing, and complex reasoning.The diverse tasks are intended to capture varied semantic information encoded in fMRI.Source: §1 · p. 2
  • 12.0% average improvement in various downstream tasks and 24.5% improvement in generalization on unseen subjects are reported over baselines.The paper also reports effective adaptation to novel tasks and says attention-weight analysis offers insights into the encoder’s workings.Source: §1 · p. 2

2. Related Works

Prior fMRI-to-text work includes stimulus-focused captioning, subject-specific or voxel-standardizing encoders, and multimodal or non-text decoding approaches. The related-work discussion positions MindLLM’s voxel-level encoder and BIT as targeting subject generalization and broader text-generation tasks.

Brain-Conditioned Text Generation

  • Prior fMRI-to-text systems use CLIP-guided encoders or embedding alignment, while UMBRAE maps fMRI to stimulus images and is consequently limited to image-related tasks.The passage contrasts these approaches with end-to-end BIT for tasks such as memory-related questions.Source: Brain-Conditioned Text Generation · p. 2
  • Voxel counts vary across subjects, and MLPs require fixed input sizes; pooling or sampling can overlook positional information or leave voxels unrepresented.MindLLM instead attends to every voxel while preserving positional information.Source: Brain-Conditioned Text Generation · pp. 2–3
  • OneLLM aligns eight modalities, including fMRI, but uses a separate convolution network per subject and relies only on captions as text annotations.The passage says these choices restrict new-subject applicability and versatile decoding.Source: Brain-Conditioned Text Generation · p. 3
  • Prior brain-decoding work targets images, videos, speech, music, and reverse-direction mappings; this paper focuses on voxel-level fMRI text generation.The related-work passage identifies its contributions as an encoder architecture and BIT objectives for text generation.Source: Brain-Conditioned Text Generation · p. 3

3. Method

MindLLM pairs an fMRI encoder with an off-the-shelf LLM, using neuroscience-informed attention for variable-sized voxel inputs and Brain Instruction Tuning (BIT) to support diverse fMRI-to-text tasks. Its encoder separates voxel signals from positional and brain-region information, while BIT trains on a large, multi-task conversation dataset.

3.1. Method Overview

  • MindLLM combines an fMRI encoder with an off-the-shelf LLM, and the encoder handles subject-specific voxel counts ranging from 12,682 to 17,907.Source: §3.1 · p. 3

3.2. fMRI Encoder

  • Learnable queries aggregate voxel signals as values, allowing the neuroscience-informed attention mechanism to accommodate varying voxel counts without pooling or sampling.The queries are intended to represent brain patterns.Source: §3.2 · p. 3
  • Voxel keys use positional coordinates rather than fMRI values, separating functional information from signal values to support generalization across subjects.The authors argue that including fMRI values in keys introduces noise because voxel positions carry functional information.Source: §3.2 · pp. 3–4
  • Multiple brain-region parcellations can be concatenated with positional encodings in the keys, combining coarse neuroscientific grounding with finer positional information.Source: §3.2 · p. 4

3.3. Brain Instruction Tuning (BIT)

  • BIT pairs fMRI recordings with annotations linked through MSCOCO images to represent visual perception, memory, language, symbolic processing, and reasoning tasks.The dataset uses prior-viewed-image captioning for memory tasks, text recognition and numerical reasoning for language and symbolic processing, and datasets requiring logical or inferential reasoning.Source: §3.3 · pp. 4–5
  • The BIT dataset contains 980,610 conversations associated with fMRI recordings from 15 datasets, including tasks beyond vision-related decoding.Source: §3.3 · p. 5
  • Training maximizes the assistant response probability while freezing the LLM weights and training only the fMRI encoder, with no additional complexity reported versus prior methods.Source: §3.3 · p. 5

4. Experiments

The experiments evaluate MindLLM on downstream decoding, unseen-subject generalization, and new-task adaptation, then examine scaling, key-embedding ablations, and attention patterns. The evaluation uses the Natural Scenes Dataset and includes brain captioning and diverse QA and captioning tasks.

4.3. Versatile Decoding

  • 12.0% average improvement over baselines across diverse downstream tasks; instruction tuning improves results by an average of 28.0%.Results are reported for subject 1; the table describes versatile decoding, with dashes marking tasks a model could not perform and a superscript ◦ marking training from scratch.Source: §4.3 · p. 6, Table 2 · p. 7

4.2. Brain Captioning

  • Brain captioning results show an average improvement of 3.32% over baselines on most metrics, despite using no subject-specific layers.The authors attribute the gains to the architecture and BIT.Source: Table 1 · p. 6, §4.2 · p. 6

4.4. Unseen Subject Generalization

  • 24.5% average improvement over the second-best subject-agnostic model when training on subjects 1–7 and evaluating on held-out subject 8.The authors identify domain adaptation as a possible direction for further gains.Source: §4.4 · p. 7, Table 3 · p. 8

4.5. Adapting to New Tasks

  • 13.5% average improvement on new-task adaptation, with a 25.0% absolute improvement from BIT.The evaluation includes sentiment understanding and utility/affordance tasks from TDIUC, described as particularly relevant to BCI applications.Source: §4.5 · pp. 7–8, Table 4 · p. 8

4.6. Performance scale with the number of subjects

  • Performance improves as the number of training subjects increases in both seen-subject and held-out-subject COCO captioning evaluations.The evaluations use subject 1 for the in-distribution setting and subject 8 for the held-out setting.Source: §4.6 · p. 8, Table 5 · p. 8, Table 6 · p. 8

4.7. Ablation Study

  • Combining positional and region encodings gives the best key-embedding design; region encodings alone speed early convergence but are eventually outperformed.Vanilla cross-attention and coordinate-to-embedding MLP variants perform poorly; the authors attribute the latter result to high-frequency spatial information in fMRI signals.Source: §4.7 · p. 8, Figure 6 · p. 9

4.8. Visualizations and Interpretations

  • Queries attend to single regions such as PPA and FFA or link early visual areas with higher-level regions such as PPA and IPS.The authors interpret these patterns as potentially reflecting visual-task brain dynamics, including transitions from seeing and thinking about an image to pressing a button for visual recall.Source: Figure 7 · p. 9, §4.8 · p. 9

5. Conclusion

MindLLM combines a subject-agnostic fMRI-to-text model with neuroscience-informed attention and brain instruction tuning; the authors report strong benchmark performance and interpretable attention patterns. The work is limited to static fMRI and identifies temporal information and video as future directions.

5. Conclusion

  • MindLLM is a subject-agnostic and versatile fMRI-to-text decoding model whose attention mechanism supports subject generalization and whose brain instruction tuning enhances versatility.The conclusion reports state-of-the-art benchmark performance and says visualizations offer insight into the patterns the model leverages.Source: §5 · p. 9
  • The work focuses on static fMRI; future work could integrate temporal information and investigate its relationship with video.Source: §5 · p. 9

A.1. Details of each dataset in brain instruction tuning

  • The appendix describes brain-instruction dataset sources and examples, including 12 task-directed question types and datasets marked for downstream-only use.It also lists FSVQA and VisDial, whose examples include multi-turn question answering.Source: §A.1 · pp. 15–16

A.2. Dataset statistics

The appendix gives examples of dataset conversations and identifies a table of subject voxel counts. It also describes how task instructions are provided.

A.2. Dataset statistics

  • Dataset examples include visual questions about scene contents, object attributes and counts, and multi-question prompts.Examples include comparing pedestrians with cars, identifying horses or a bird, and discussing oral hygiene in a public setting.Source: §A.2 · p. 16
  • Table 8 reports the number of voxels for each subject in the natural-scene dataset.The supplied passages identify the table's contents but do not provide its values.Source: Table 8 · p. 16, §A.2 · p. 16

A.3. Instructions

  • Dataset-specific instructions are listed, with question-answering instructions appended after the question on a new line.Some instructions are inspired by Han et al. (2024b).Source: §A.3 · p. 16

B. Implementation details

Implementation uses Vicuna-7b for the language model and specifies encoder, tuning, and downstream training settings. The BIT dataset statistics note that only a subset of conversations has matching fMRI recordings.

B. Implementation details

  • Brain instruction tuning uses Vicuna-7b, AdamW, a 1 × 10−3 learning rate, and batch size 64 for 8 days on 8× L40S GPUs.Downstream fine-tuning uses one L40S GPU, a 1 × 10−4 learning rate, and batch size 48; generation uses greedy decoding.Source: B. Implementation details · p. 16
  • The fMRI encoder uses 128 fMRI tokens, a 4-layer MLP, and 1024 queries with 128-dimensional embeddings.Source: B. Implementation details · p. 16
  • The usable BIT conversations are fewer than the total because only a subset of MSCOCO images were used as stimuli with fMRI recordings.Table 9 covers instruction templates and BIT dataset statistics; “−” marks instructions embedded in the conversation.Source: Table 9 · p. 17

C. Computational Complexity and Runtime Analysis

MindLLM's encoder adds only a marginal computational-cost increase over baselines despite its performance, while the shared LLM accounts for most inference time and complexity.

C. Computational Complexity and Runtime Analysis

D. Qualitative Analysis

Qualitative comparisons report stronger visual understanding than MindBridge and UniBrain on captioning and question-answering tasks. Examples highlight static-scene interpretation and action discrimination.

D. Qualitative Analysis

  • MindLLM better identifies static objects and spatial context than MindBridge and UniBrain, including an airplane on a runway rather than flying over a city.The example is from COCO captioning.Source: D. Qualitative Analysis · p. 17, Figure 8 · p. 18
  • On sports scenarios, MindLLM identifies “catch ball” where both baselines predict “serve,” indicating improved action-state discrimination.The example is from COCO-QA and OK-VQA qualitative analysis.Source: D. Qualitative Analysis · p. 17, Figure 9 · p. 19
  • The authors suggest these object-understanding and action-recognition improvements could contribute to neural prosthetics and assistive technologies.Source: D. Qualitative Analysis · p. 17

E. Ablation Studies on Downstream Tasks

The ablation results support the design of neuroscience-informed attention across downstream tasks.

E. Ablation Studies on Downstream Tasks

  • Table 11’s downstream-task ablations validate the design of neuroscience-informed attention.The reported evaluation follows brain instruction tuning without task-specific fine-tuning.Source: E. Ablation Studies on Downstream Tasks · p. 19

F. Architecture Comparison

MindLLM is the only compared model with all four listed properties. It shows gains over baselines on most tested scenarios for subjects 2, 5, and 7, while its latent representations become increasingly mixed across subjects through successive MLP layers.

F. Architecture Comparison

  • MindLLM is the only compared model with all four listed properties: cross-subject, unified architecture, shared parameters, and use of all voxels.The table defines these properties, including that a unified architecture can have subject-specific parameters.Source: F. Architecture Comparison · p. 19

G. Experiments on Other Subjects

  • On subjects 2, 5, and 7, MindLLM gains over baselines in the majority of scenarios.These subjects were chosen because they are commonly used in previous works.Source: G. Experiments on Other Subjects · p. 19

H. Latent Space Visualization

  • Across successive MLP layers, latent embeddings become increasingly mixed by subject, forming a subject-agnostic representation.The neuroscience-informed attention layer retains separable subject patterns; the paper connects the later mixing to zero-shot generalization on held-out subjects.Source: H. Latent Space Visualization · p. 19

I. Brain Attention Maps of Other Subjects

MindLLM’s brain attention maps show similar spatial patterns across subjects alongside moderate subject-specific variation.

I. Brain Attention Maps of Other Subjects

  • Attention maps show highly similar spatial patterns across subjects, indicating shared structural-functional correspondences.Figure 11’s caption likewise notes similar patterns across subjects, suggesting shared brain organization.Source: §I · p. 19, Figure 11 · p. 22
  • The maps also show moderate subject-specific variation, reflecting voxel-level functional-signal and precise voxel-location differences.The paper says flexible attention mechanisms accommodate these differences.Source: §I · p. 22

Related papers

Loading 2502.15786v2…