PaperlayerView exact revision on arXiv

Source-linked AI summary

UMBRAE: Unified Multimodal Brain Decoding

Weihao Xia, Raoul de Charette, Cengiz Öztireli, Jing-Hao Xue

arXiv:2404.07202v2cs.CVcs.AIcs.CL

TL;DR

Brain decoding often lacks accurate spatial information and commonly relies on subject-specific models. UMBRAE aligns brain signals with image features and trains across subjects; it performs better than or on par with state-of-the-art methods across several tasks, with direct brain grounding at least 10 times faster than natural baselines.

  • Problem

    Brain-decoding methods often fail to recover accurate spatial information, and distinctive brain activity patterns have typically led to subject-specific models.

    Source: §1 · p. 2

  • Method

    UMBRAE aligns neural signals with image features using a universal brain encoder and trains a shared representation across subjects for MLLM-based multimodal decoding.

    Source: §1 · p. 2, §3.1 · p. 5, §3.2 · p. 5

  • Results

    UMBRAE is better than or on par with state-of-the-art methods across brain captioning, retrieval, and visual decoding; direct brain grounding is on par with natural baselines and at least 10 times faster.

    Source: §1 · p. 2

  • Takeaways & Limitations

    BrainHub extends NSD with multimodal ground truth for evaluating semantic concepts and spatial localization in visual stimuli.

    Source: §1 · p. 2

    The method is constrained by NSD brain-response quality and experimental images sourced from COCO, limiting its scope to natural scenes similar to COCO.

    Source: §E.1 · p. 32

Abstract

from arXiv · show

We address prevailing challenges of the brain-powered research, departing from the observation that the literature hardly recover accurate spatial information and require subject-specific models. To address these challenges, we propose UMBRAE, a unified multimodal decoding of brain signals. First, to extract instance-level conceptual and spatial details from neural signals, we introduce an efficient universal brain encoder for multimodal-brain alignment and recover object descriptions at multiple levels of granularity from subsequent multimodal large language model (MLLM). Second, we introduce a cross-subject training strategy mapping subject-specific features to a common feature space. This allows a model to be trained on multiple subjects without extra resources, even yielding superior results compared to subject-specific models. Further, we demonstrate this supports weakly-supervised adaptation to new subjects, with only a fraction of the total training data. Experiments demonstrate that UMBRAE not only achieves superior results in the newly introduced tasks but also outperforms methods in well established tasks. To assess our method, we construct and share with the community a comprehensive brain understanding benchmark BrainHub. Our code and benchmark are available at https://weihaox.github.io/UMBRAE.

1 Introduction

Brain decoding often loses fine-grained semantic and spatial information, while subject-specific brain patterns have typically required separate models. UMBRAE proposes multimodal decoding and cross-subject training, evaluated with the BrainHub benchmark.

1 Introduction

  • Indirect communication channels such as language, gaze, facial expression, and action may be inaccurate for people with cognitive or physical disabilities or locked-in patients.Source: §1 · p. 1
  • Single-modality decoding can lose object appearance or location, and distinctive brain patterns have led prior methods to train separate models for each subject.Text may omit texture and spatial location, while visual decoding lacks scene structure and faces pixel-wise reconstruction challenges.Source: §1 · p. 2
  • UMBRAE aligns brain signals with images using a universal brain encoder and frozen MLLM, while cross-subject training learns a shared representation and supports adaptation with minimal data.The method aims to recover conceptual and spatial details at multiple levels of granularity through prompting.Source: §1 · p. 2
  • BrainHub extends NSD with multimodal ground truth for evaluating brain understanding.The benchmark pairs fMRI with semantic concepts and spatial localization in visual stimuli.Source: §1 · p. 2
  • At least 10 times faster, direct brain grounding performs on par with natural baselines, while UMBRAE is better than or on par with state-of-the-art methods across captioning, retrieval, and visual decoding.The grounding comparison is with natural baselines.Source: §1 · p. 2

2 Related Works

Prior brain-decoding work often uses unimodal outputs and subject-specific models; UMBRAE instead aligns brain features with image features and trains across subjects. Its encoder maps subject-specific inputs into a common representation for MLLM prompting.

2 Related Works

  • Prior approaches decode images, videos, or text, while brain alignment methods commonly use pretrained latent spaces and per-subject training; UMBRAE targets semantic and spatial cues through cross-subject image alignment.Related work also describes brain-conditioned image generation and multimodal models that require substantial data and computational resources.Source: §2 · p. 3, §3 · p. 4

3.1 Architecture

  • Subject-specific tokenizers map variable-length brain signals to fixed-length tokens, and a universal perceive encoder projects those tokens into a common space.The tokenizers capture subject-specific characteristics, while the universal encoder aims to capture shared knowledge across subjects.Source: §3.1 · p. 5

3.2 Cross-Subject Alignment

  • Cross-subject batches draw θ × B samples from a selected subject and the remaining examples uniformly from other subjects, exposing the model to user diversity without extra training time.The sampling strategy is intended to avoid subject preference and catastrophic forgetting.Source: §3.2 · p. 5

3.3 Multimodal Alignment

  • UMBRAE aligns brain representations with pretrained image features by element-wise reconstruction, rather than binding modalities through contrastive pretraining with images or text.The intermediate image features are intended to preserve semantic and spatial alignment and can be input directly to the MLLM.Source: §3.3 · p. 6

3.4 Brain Prompting Interface

  • At inference, brain features pass through the MLLM adapter as visual embeddings, then combine with an instruction prompt to produce an answer.The interface supports captioning and grounding prompts and can also be used for conversation, detail description, and complex reasoning.Source: §3.4 · p. 6

4 Experiments

The experiments evaluate UMBRAE across captioning, grounding, retrieval, and visual decoding using BrainHub, an extension of NSD. They also examine cross-subject training and adaptation to a new subject with limited data.

4.1 Implementation Details

  • One brain encoder is trained across four subjects; each subject has 24,980 training samples, and evaluation averages three repetitions across 982 test samples.Training uses a single A100 GPU for 240 epochs and takes about 12 hours.Source: Architecture · p. 7

4.2 BrainHub

  • BrainHub extends NSD with tasks for captioning, grounding, retrieval, and visual decoding, using fMRI responses paired with visual stimuli.Its tasks evaluate descriptions, spatial locations, retrieval accuracy, and reconstruction of visual stimuli.Source: §4.2 · pp. 7–8

4.3 Brain Captioning

  • UMBRAE outperforms all captioning baselines across metrics, and its cross-subject model exceeds the S1-only model.The comparison with the S1-only model demonstrates that training across subjects can improve captioning performance.Source: §4.3 · p. 8

4.4 Brain Grounding

  • UMBRAE performs roughly on par with constructed grounding baselines and is at least 10 times faster.Performance varies by category: UMBRAE is outperformed on Salient Creatures, but not on Salient Objects or Inconspicuous elements.Source: §4.4 · pp. 9–10

4.5 Brain Retrieval

  • 94.2% forward, 91.3% backward, and 93.8% exemplar retrieval accuracy; UMBRAE outperforms current methods.Exemplar retrieval searches for the exact original image among 982 test images.Source: §4.5 · pp. 10–11, Table 3 · p. 11

4.6 Visual Decoding

  • UMBRAE performs comparably or better than state-of-the-art methods on visual decoding without task-specific design.The visual reconstruction uses decoded text and CLIP image embeddings as cues for Versatile Diffusion.Source: Table 4 · p. 11, §4.6 · p. 11

4.7 Weakly-Supervised Adaptation

  • Finetuned S7 adaptation is comparable to the S7-only model with 30% of its data and often better above 50%.Training only a tokenizer while freezing the pretrained backbone generally performs worse than fine-tuning the backbone with it.Source: Figure 4 · p. 12, §4.7 · p. 12

5 Ablation Study

UMBRAE’s cross-subject design improves performance over single-subject training without requiring extra training resources, while image-feature alignment with an unchanged adapter and an MSE loss gives the strongest reported training-strategy results.

5.1 Architectural Improvements

  • Cross-subject UMBRAE surpasses its single-subject counterpart across almost all metrics while requiring basically the same training resources.The universal encoder adds subject-specific tokenizers and changes batch sampling; the authors attribute its gains to user diversity.Source: §5.1 · p. 13
  • The encoder uses 112.63 million parameters for one subject and 146.24 million for four, versus 1,003.64 million per subject for the MLP baseline.These correspond to reported parameter reductions of 88.78% for one subject and 96.36% for four subjects.Source: §5.1 · p. 13

5.2 Training Strategies

  • Finetuning the adapter or training it jointly with the brain encoder decreases performance, likely because updating well-trained parameters causes catastrophic forgetting.The adapter connects multimodal encoders to MLLM output spaces.Source: §5.2 · p. 14
  • Aligning brain responses with the pretrained LLM yields less desirable results than image-feature alignment across the tested variants.The authors say image-feature alignment performs best when only image–brain pairs are available for training.Source: §5.2 · p. 14
  • Applying MSE to image features while keeping the adapter unchanged gives the most favorable concept-recognition and object-localization performance; contrastive learning diminishes performance.Source: §5.2 · p. 14

6 Conclusion

UMBRAE combines multimodal brain decoding with cross-subject training and weakly supervised adaptation, and the paper presents BrainHub as an evaluation benchmark. The accompanying material describes supported prompts, benchmark construction, and the NSD data used.

6 Conclusion

  • UMBRAE decodes multimodal explanations from brain signals, uses cross-subject training, and supports weakly supervised adaptation to new subjects in a data-efficient manner.BrainHub is constructed to facilitate future research and evaluation.Source: §6 · p. 14

Appendix

  • The appendix outlines task prompts, BrainHub construction, further experiments and analyses, and discussion of limitations and potential negative impacts.Source: Appendix · p. 15

A UMBRAE: Tasks and Example Prompts

  • Users can switch tasks through natural-language prompts, but tasks requiring an object or scene to be located first—including grounding captioning and referring expression generation—are excluded.Brain signals, not images, are inputs; prompt embeddings are concatenated with predicted brain features.Source: A UMBRAE: Tasks and Example Prompts · p. 15

B Details on BrainHub

  • BrainHub extends NSD using COCO annotations.The benchmark is described as a multimodal brain-understanding benchmark.Source: B Details on BrainHub · p. 16

B.1 Natural Scenes Dataset

  • NSD records fMRI responses from eight participants who viewed images for up to 40 hours; the study uses the four participants who completed all scanning sessions.For each selected participant, the training set has 8,859 images and 24,980 trials; the common test set has 982 images and 2,770 trials.Source: §B.1 · p. 16
  • BrainHub’s test-set table reports images, voxels, captions, bounding boxes, fMRI regions of interest, subject references, and their dimensions.Source: Table 2 · p. 16

B.2 Benchmark Construction

BrainHub groups COCO classes by salience, and the accompanying captioning results report performance across subjects and qualitative comparisons with baselines. The benchmark’s categories are imbalanced, with inconspicuous classes most numerous.

B.2 Benchmark Construction

  • BrainHub groups 80 COCO classes into Salient, Salient Creatures, Salient Objects, and Inconspicuous; Inconspicuous is largest, and person instances stand out within Salient.The listed class counts are 11 for Salient Creatures, 17 for Salient Objects, and 52 for Inconspicuous; the figure illustrates category statistics and class mapping.Source: §B.2 · pp. 16–17, Figure 1 · p. 17

C.1 Brain Captioning

  • UMBRAE and UMBRAE-S1 produce the most fluent and relevant S1 captions, accurately describing scenes and counts where baseline captions diverge.SDRecon often produces incoherent or irrelevant descriptions; OneLLM is coherent but can diverge from the reference, while BrainCap faces content-similarity challenges.Source: §C.1 · pp. 17–18
  • The brain encoder is trained without ground-truth captions, unlike the cited brain-captioning state-of-the-art methods.Source: §C.1 · p. 18
  • Across S2, S5, and S7, UMBRAE and UMBRAE-Sx outperform the reported baselines on all metrics, and cross-subject UMBRAE generally exceeds single-subject models.Caption quality and relevance are also reported as comparable to COCO ground truth captions and image-based Shikra results.Source: §C.1 · p. 18

C.2 Brain Grounding

The grounding results generally favor cross-subject over single-subject models, and the paper describes both prompted localization and grounding without prior object knowledge. Additional figures illustrate grounding and retrieval procedures across subjects and tasks.

C.2 Brain Grounding

  • The cross-subject model generally outperforms single-subject models in grounding; reported results include acc@0.5 and use a prompt to locate a queried expression.The evaluation treats queried concepts as known and asks the model to detect them and return coordinates, a referring-expression-comprehension task.Source: §C.2 · p. 18
  • Spotting Captioning describes brain-response content and identifies mentioned objects or regions using points or boxes, without prior knowledge of the contents.The paper shows qualitative results from different prompts and subjects; box coordinates are textual outputs visualized on reference images.Source: §C.2 · p. 20

C.3 Brain Retrieval

  • Forward retrieval finds the paired CLIP image embedding from a brain embedding, while backward retrieval seeks the correct brain embedding from an image embedding.The figure shows the reference image alongside the top five retrievals for each direction.Source: §C.3 · pp. 20, 23
  • Figures 3–5 illustrate cross-subject captioning and grounding outputs across prompts and subjects; reference images are for visualization, while brain responses are the inputs.Source: Figure 3 · p. 21, Figure 4 · p. 22, Figure 5 · p. 23

C.4 Visual Decoding

UMBRAE’s predicted textual and visual outputs serve as cues for visual reconstruction with pretrained image-generation models, although its predicted latent codes and bounding boxes have important constraints.

C.4 Visual Decoding

  • UMBRAE is not specifically designed for fMRI-to-image reconstruction, but its predicted text and visual outputs can cue pretrained generators.The tested models include SD, SD-XL, Kandinsky, GlIGEN, and Versatile Diffusion.Source: §C.4 · p. 23, Table 5 · p. 25
  • Latent codes are crucial to visual-decoding performance; predicted text alone or text with a predicted image embedding is insufficient for reliable reconstruction.The comparison also tests ground-truth and Brain-Diffuser inputs as alternatives.Source: §C.4 · p. 23
  • Figure 6 distinguishes forward retrieval of an image embedding from a brain embedding from backward retrieval of a brain embedding from an image embedding.Source: Figure 6 · p. 24
  • Predicted bounding boxes are intended as low-level reconstruction constraints but perform poorly on low-level metrics because they are unreliable.Source: §C.4 · p. 24

D Additional Analyses

  • Additional analyses cover per-subject performance, weakly supervised adaptation, cross-subject sampling strategies, joint grounding and decoding, other MLLM tasks, and model-agnostic behavior.Source: D Additional Analyses · p. 27

D.1 Subject-Specific Analysis

UMBRAE models are trained either across subjects or for one subject, and the appendix examines subject combinations and adaptation. Training with multiple subjects generally improves performance, while adaptation can reach comparable performance using a fraction of a target subject’s data.

D.1 Subject-Specific Analysis

  • UMBRAE uses all training data from S1, S2, S5, and S7; UMBRAE-Sx uses data from one subject only.Source: §D.1 · p. 27
  • Training on more than one subject always improves performance, but using all available subjects does not always give the best result.For S7 evaluation, training on {S1, S5, S7} performs better than including S2.Source: §D.1 · p. 27, Table 7 · p. 28

D.2 Weakly-Supervised Adaptation

  • With only 30% of target-subject data, finetuned adaptation to S1, S2, and S5 achieves performance comparable to the single-subject model.The experiments pretrained on the other three subjects and finetuned with varying amounts of target-subject data.Source: §D.2 · p. 28, Figure 10 · p. 29
  • For adaptation, UMBRAE is first pretrained on the other subjects, then finetuned on varying proportions of the target subject’s data.Source: §D.2 · p. 28, Figure 10 · p. 29

D.3 Sampling Strategies in Cross-Subject Training

The cross-subject sampling strategy combines a dominant subject’s samples with samples from other subjects in each batch. The authors report that this strategy outperforms the alternatives they test.

D.3 Sampling Strategies in Cross-Subject Training

  • Each batch contains θ×B samples from a frequency-selected dominant subject, with the remaining samples drawn from other subjects.Source: §D.3 · p. 30
  • Ours outperforms Random, Stratified, and Ours-R in average metrics across users.Source: §D.3 · p. 30
  • The dominant-subject design is intended to capture within-subject variation while exposing the model to other subjects’ patterns to enhance discrimination and alleviate catastrophic forgetting.Source: §D.3 · p. 30

D.4 Joint Grounding-Decoding Evaluation

The joint evaluation presents grounding alongside image reconstruction and reports that the tasks are correlated. It also illustrates cases where grounding is correct but reconstruction remains inaccurate.

D.4 Joint Grounding-Decoding Evaluation

  • The grounding results are visualized on reference images to assess performance.Source: §D.4 · p. 30
  • Grounding and reconstruction are correlated, but correct grounding can accompany inaccurate reconstruction, as with a misoriented surfer, giraffe, or skier.Source: §D.4 · p. 30
  • Figure 11’s first row shows reference images, its second reconstructed images, and its third spotting-captioning results with object locations visualized in color.Source: §D.4 · p. 30

D.5 Other Supported Tasks

UMBRAE supports captioning, grounding, and question-answering tasks, and its model-agnostic design allows different encoders and language models. Replacing Shikra with LLaVA improves several instruction-following capabilities but does not provide spatial localization.

D.5 Other Supported Tasks

  • The supported task groups are captioning, grounding, and question answering; the section points to prior appendices for captioning and grounding results.This section presents additional QA examples.Source: §D.5 · pp. 30–31
  • Joint visualizations show reconstructions generated from decoded text and groundings, while warning that reconstruction may remain inaccurate despite correct bounding boxes.The figure shows reference images, reconstructions, and object coordinates represented by color in the reconstruction.Source: Figure 11 · p. 31
  • Additional supported question-answering tasks include Q→A, Q→CA, and Q→CBoxA.Example outputs for these tasks are shown in Fig. 12.Source: §D.5 · p. 31

D.6 Model-Agnostic Performance

  • UMBRAE can use different image encoders, LLMs, and MLLMs, rather than requiring a single model configuration.Examples include CLIP or DINO, LLaMA or Vicuna, and Shikra or LLaVA.Source: §D.6 · p. 31
  • LLaVA shows stronger instruction-following than Shikra in conversation, detailed description, and complex reasoning, but lacks spatial localization.The comparison uses LLaVA in place of UMBRAE's default Shikra MLLM.Source: §D.6 · p. 31

E.1 Method Limitations

UMBRAE's failures can involve semantic recognition, spatial localization, or both. The method also depends on the quality and scope of NSD brain responses and inherits risks associated with pretrained large models.

E.1 Method Limitations

  • NSD's non-invasive brain-response capture can be affected by participant movement or distraction, and COCO-sourced images constrain the method to similar natural scenes.The authors also note inherent inaccuracies introduced during data collection.Source: §E.1 · p. 32
  • Failure cases include correct semantics with inaccurate localization, accurate localization with semantic errors, and errors in both.Spotting captioning illustrates these cases because it outputs both concepts and locations.Source: §E.1 · pp. 31–32, Figure 14 · p. 33

E.2 Potential Negative Impact

  • Reliance on pretrained MLLMs brings potential biases, hallucinations, inappropriate content, and ethical concerns, while inaccurate interpretations may cause misunderstandings about brain signals.These are risks associated with the pretrained models underlying the method.Source: §E.2 · pp. 32–33

Related papers

Loading 2404.07202v2…