PaperlayerView exact revision on arXiv

Source-linked AI summary

BrainJanus: A Unified Model for Understanding and Generation across Brain, Vision, and Language

Haitao Wu, Qirui Zhang, Zhouheng Yao, Shangquan Sun, Qihao Zheng, Mianxin Liu, Chi Zhang, Wanli Ouyang, Chunfeng Song, Changqing Zhang, Jiamin Wu

arXiv:2606.30319v1cs.CVcs.LG

TL;DR

Existing brain models often separate encoding and decoding and align primarily with unimodal representations, despite the brain’s multimodal organization. BrainJanus unifies brain, vision, and language in one autoregressive model, achieving superior benchmark performance, zero-shot generalization, and interpretable cortical topography.

  • Problem

    Existing methods largely overlook the brain’s multimodal integration, treating encoding and decoding separately and relying primarily on unimodal brain alignment.

  • Method

    BrainJanus quantizes neural dynamics into tokens aligned with vision and language in a shared space, then uses one autoregressive model for any-to-any generation.

  • Results

    BrainJanus achieves superior performance across diverse encoding and decoding benchmarks, outperforms task-specific models under joint training, and demonstrates zero-shot generalization.

  • Takeaways & Limitations

    The framework integrates brain, vision, and language modeling while preserving interpretable cortical topography and individual variability in generated fMRI signals.

  • Takeaways & Limitations

    Semantic-level reconstruction evaluation can be exploited by padding ground-truth visual embeddings, potentially bypassing the biological mapping it is intended to assess.

Abstract

from arXiv · show

Modeling the bidirectional correspondence between external sensory stimuli and internal neural activity has emerged as a critical frontier in neuroscience. However, existing approaches predominantly treat brain encoding and decoding as isolated tasks, relying heavily on unimodal alignment and external priors while overlooking the brain's intrinsic nature as a multimodal integration system. To address these limitations, we propose BrainJanus, the first unified brain model that integrates brain, vision, and language within a single framework. Specifically, we introduce a Unified Brain Tokenizer to quantize continuous neural dynamics into discrete tokens aligned with visual and linguistic representations in a shared Omni space. Building on this, we utilize an All-in-One autoregressive architecture that leverages next-token prediction to enable seamless any-to-any generation, which encompasses image-to-brain and text-to-brain encoding, and brain-to-image and brain-to-text decoding. Extensive experiments demonstrate that BrainJanus achieves superior performance across diverse benchmarks. Furthermore, our framework exhibits zero-shot generalization and preserves interpretable biological topography, highlighting its potential as a general-purpose brain modeling paradigm. The code is available at \href{https://github.com/HaitaoWuTJU/BrainJanus}{GitHub}.

1. Introduction

The introduction frames brain encoding and decoding as complementary but insufficiently unified tasks, because the brain integrates visual and semantic information across modalities. BrainJanus addresses this gap with a shared discrete token space and unified autoregressive model spanning brain, vision, and language.

  • Motivation: Brain encoding predicts neural responses from visual inputs, whereas brain decoding infers stimulus-related information from neural activity.These directions correspond to image → brain encoding and brain → image decoding.
  • Problem: The brain’s intrinsic multimodal integration contrasts with task-specific approaches that rely on lossy neural modeling and large external priors.Examples include frozen CLIP, Stable Diffusion, LLaMA, and GIT.
  • Approach: BrainJanus is introduced as the first unified autoregressive framework bridging brain, vision, and language through a shared discrete token space.The framework enables seamless any-to-any generation within a single model.
  • Results: BrainJanus achieves competitive performance across encoding and decoding benchmarks and consistently outperforms task-specific models under joint training.The claim covers a wide range of encoding and decoding benchmarks.
  • Results: Unified multi-task learning enables zero-shot generalization with strong task-agnostic representations.The result is attributed to effective cross-modal knowledge transfer.
  • Results: Generated fMRI signals preserve interpretable cortical topography and biological variability, indicating meaningful neural representations.These properties suggest the model captures biologically relevant structure in neural activity.

2. Related Work

Prior work advances neural representation learning, brain–stimulus mapping, and unified multimodal generation, but does not yet provide a universal tokenizer that jointly models brain, vision, and language. Extending discrete multimodal tokenization to internal brain signals therefore remains an open frontier.

  • Neural Representation Pretraining: Neural representation pretraining uses large-scale data and transformer architectures across EEG, MEG, and fMRI, improving temporal modeling and cross-subject generalization.Examples include LaBraM, EEGPT, EEGformer, BrainLM, and MindEye2.
  • Research Gap: Prior work has not developed a universal brain tokenizer that converts continuous neural recordings into discrete tokens shared with text and vision for joint modeling.Extending unified multimodal understanding and generation to internal brain signals remains unexplored.
  • Bidirectional Brain–Stimulus Mapping: Brain–stimulus mapping methods pursue high-fidelity visual reconstruction, semantic alignment, cross-subject robustness, and zero-shot recognition under low signal-to-noise conditions.Approaches include shared-subject modeling, CLIP-guided training, reconstruction, and contrastive alignment of brain signals with visual-linguistic features.
  • Brain and Language Models: Recent models align brain encoders with multimodal large language models and build transformer-based foundations for diverse neural tasks.UMBRAE and MindLLM support grounding and open-ended instruction tuning, while MEG-GPT provides a transformer-based foundation model for MEG.
  • Unified Multimodal Modeling: Unified multimodal models such as Chameleon and Emu3 show that discrete tokenization enables joint autoregressive modeling of different modalities.Their unification spans modalities including text, image, video, and audio.

3. Method

BrainJanus uses a two-stage pipeline that first discretizes neural signals into tokens aligned in a shared Omni space, then jointly fine-tunes a Transformer across brain, vision, and language tasks. Its unified autoregressive model supports bidirectional generation between arbitrary modality pairs through interleaved token sequences.

  • Overall framework: The method first pretrains a Unified Brain Tokenizer, then applies supervised fine-tuning to a Transformer jointly trained on four mixed brain, vision, and language encoding-decoding tasks.The tokenizer maps brain signals into the shared Omni space before the multitask fine-tuning stage.
  • Unified Brain Tokenizer: The Unified Brain Tokenizer converts continuous neural signals into discrete token sequences compatible with the Omni space using a learned codebook.Its VQ-style objective combines reconstruction, codebook, and commitment terms.
  • Omni space: Brain, vision, and language inputs are mapped into finite token sequences with shared embedding dimensionality, enabling one model to process heterogeneous modalities.The modality-specific sequences may have different lengths while remaining in the same token space.
  • Unified Autoregressive Modeling: The unified autoregressive model predicts each next token from an interleaved multimodal context whose tokens can originate from any modality.This formulation allows arbitrary interleaving of brain, vision, and language tokens.
  • Unified Autoregressive Modeling: By autoregressively conditioning on any modality, BrainJanus enables translation, completion, and conditional generation across brain ↔ image, brain ↔ text, and image ↔ text.The same model also supports multi-step compositions such as brain → text → image without separate models.

4. Experiments

Experiments on NSD evaluate BrainJanus across brain-to-text, brain-to-image, and brain-encoding tasks using established quantitative and qualitative protocols. Results show strong decoding performance, while analyses expose vulnerabilities in conventional brain-encoding evaluation and motivate leakage-resistant training.

  • Evaluation Metrics: Brain-to-image evaluation measures low-level structural fidelity with PixCorr and SSIM and high-level semantic alignment with feature-based and CLIP metrics.Reported feature-based metrics include Inception, EfficientNet, and SwAV.
  • Brain Decoding: BrainJanus achieves a BERTScore of 38.12 and a CLIP score of 96.2%, surpassing prior state-of-the-art results by 7.21 and 1.5%, respectively.These results are reported for brain-to-text decoding.
  • Brain Decoding: 94.4% CLIP semantic similarity is achieved for brain-to-image decoding despite diffusion-free generation, exceeding diffusion-based baselines in high-level alignment.BrainJanus is the only autoregressive approach among the compared methods.
  • Brain Decoding: Qualitative comparisons show more accurate textual descriptions and visual reconstructions that better preserve object attributes, actions, and scene structure than prior methods.The visual comparison includes diffusion-based baselines.
  • Brain Encoding Evaluation: Voxel-level Pearson correlation and MSE can ignore global structure and overreact to trial-to-trial neural variability, limiting their validity for synthetic fMRI evaluation.The analysis also examines cosine similarity distributions across trials.
  • Brain Encoding Evaluation: Padding Hacking can achieve near-perfect semantic and visual reconstruction scores by leaking visual embeddings without learning a biological mapping, making image metrics insufficient alone.The authors therefore avoid direct alignment with pretrained visual embeddings and train with only MSE or CE loss.

5. Conclusion

BrainJanus unifies brain encoding and decoding with vision and language through a shared discrete token space and autoregressive Transformer, achieving strong benchmark performance and zero-shot generalization. It also preserves interpretable cortical topography, while remaining limited by visual-cortex fMRI coverage, possible hallucinations, computational cost, and untested robustness across broader settings.

  • Conclusion: BrainJanus integrates brain encoding and decoding with vision and language through a shared discrete token space and single autoregressive Transformer.This enables seamless any-to-any generation across modalities.
  • Conclusion: BrainJanus achieves superior performance across diverse encoding and decoding benchmarks, outperforms task-specific models under joint training, and demonstrates strong zero-shot generalization.
  • Conclusion: BrainJanus generates biologically plausible fMRI signals that preserve interpretable cortical topography and individual variability.
  • Limitation: The model is currently restricted to fMRI data from the visual cortex rather than whole-brain activity.
  • Limitation: Powerful generative priors may cause hallucinations by prioritizing visual quality over strict biological faithfulness.
  • Limitation: High computational cost and robustness across diverse neural modalities and subject populations remain to be fully explored.

A. Experimental details · A.1. Natural Scenes Dataset

Experiments use NSD’s high-resolution 7T fMRI recordings paired with natural images, with trial-specific responses analyzed across predefined visual-system ROIs. The supplied figure also compares image–caption semantic alignment across three caption sources.

  • A. Experimental details: Experiments use the Natural Scenes Dataset, pairing high-resolution fMRI recordings with natural image stimuli from COCO.NSD includes 7T fMRI data from eight subjects; images were shown for 3 seconds before a previously-seen judgment task.
  • A.1. Natural Scenes Dataset: The study follows prior work by using data from 8 subjects.
  • A.1. Natural Scenes Dataset: NSD data are preprocessed at 1.8 mm spatial resolution and represented as single-trial GLM-estimated beta weights.These beta weights capture trial-specific activation patterns.
  • A.1. Natural Scenes Dataset: Analyses are conducted within NSD-provided ROIs spanning early visual cortex and higher-level ventral visual areas.The selected ROIs include voxel counts reported for subj01, subj02, subj05, and subj07.
  • A.1. Natural Scenes Dataset: Figure 7 compares CLIP Score distributions for captions from Qwen3-VL-235B, GIT-large, and original COCO ground truth.The distributions assess semantic alignment between images and captions.
  • A.1. Natural Scenes Dataset: Qwen3-VL-235B captions show a distinct rightward-shifted CLIP Score distribution relative to GIT-large and original COCO annotations.The figure caption interprets this shift as superior image–text alignment.

A.2. Bounds on Voxel Response Consistency

Voxel-level inter-trial consistency analysis across eight subjects establishes empirical noise ceilings for image-to-voxel encoding, reflecting inherent biological variability. The achievable performance is bounded by an MSE lower bound of approximately 0.55 and upper bounds below 0.65 for both Cosine Similarity and Pearson Correlation.

  • A.2. Bounds on Voxel Response Consistency: Inter-trial consistency metrics establish a noise ceiling, an empirical upper bound for any image-to-voxel encoding model.The analysis uses Mean Squared Error, Cosine Similarity, and Pearson Correlation across voxel responses to the same stimulus.
  • A.2. Bounds on Voxel Response Consistency: Approximately 0.55 is the empirical lower bound for MSE because of inherent biological variability.This bound represents the optimal achievable floor for mean squared error under inter-trial response variability.
  • A.2. Bounds on Voxel Response Consistency: Below 0.65 are the upper bounds for both Cosine Similarity and Pearson Correlation.These bounds quantify the maximum achievable consistency for the two similarity and correlation metrics.

A.3. Implementation details · B. Detailed Results

The implementation uses a specified Python/PyTorch environment with distributed training and BF16 precision, while the brain tokenizer is a VQ-VAE configured with discrete compression and explicitly stated optimization settings.

  • A.3. Implementation details: BrainJanus is implemented with Python 3.12.11, CUDA 12.8, PyTorch 2.8.0, transformers 4.57.1, and flash attn 2.8.1 on Ubuntu 22.04.05 LTS.
  • A.3. Implementation details: Training uses accelerate 1.10.1 with ZeRO Stage 2 and bfloat16 (BF16) precision.
  • A.3. Implementation details: Experiments run on 96 Intel Xeon vCPUs at 2.90 GHz, eight NVIDIA A100 GPUs with 80 GB memory each, and 1024 GB of RAM.
  • A.3. Implementation details: The brain tokenizer uses a VQ-VAE architecture with codebook size 128, compression ratio 128, and embedding dimension 32.
  • A.3. Implementation details: The tokenizer is trained with AdamW, initial learning rate 1 × 10−4, batch size 256, and 100 epochs.
  • A.3. Implementation details: Its commitment loss coefficient is 0.25 and entropy loss ratio is 0.1.

B.1. Detailed Results for Each Subject · B.2. Brain Decoding Cases · B.3. Brain Encoding Cases

The appendix reports quantitative evaluations for brain-to-text and brain-to-image generation, alongside qualitative evidence for brain decoding and image-to-fMRI encoding. Results indicate coherent multimodal decoding, structurally consistent predicted brain activity, and preserved semantic content in reconstructed images.

  • B.1. Detailed Results for Each Subject: Quantitative brain-to-text caption-generation results across all subjects are reported in Table 8.The table evaluates brain-to-text caption generation for all subjects.
  • B.1. Detailed Results for Each Subject: Quantitative brain-to-image generation results across all subjects are reported in Table 9.The table evaluates brain-to-image generation for all subjects.
  • B.2. Brain Decoding Cases: Brain-to-text decoding generates coherent descriptions capturing salient objects and high-level semantics.These qualitative results are presented in Figure 9 and Figure 10.
  • B.2. Brain Decoding Cases: Brain-to-image reconstructions preserve reasonable global structure and semantic consistency despite degradation in fine-grained details.The reconstructions correctly reflect object categories and spatial layouts.
  • B.3. Brain Encoding Cases: Figure 11 evaluates image-to-fMRI prediction and subsequent reconstruction, with synthesized fMRI responses structurally consistent with real brain activity.The predicted signals support reconstruction of images with recognizable semantic content and visual characteristics.
  • B.3. Brain Encoding Cases: Images reconstructed from predicted fMRI signals retain recognizable semantic content and visual characteristics.These findings support bidirectional mappings between visual stimuli and brain activity within the unified framework.

B.4. Visual-to-fMRI Hacking Analyse

The section argues that voxel-level encoding metrics overlook cortical topography and neural-response variability, while semantic-level image-reconstruction evaluation can be trivially hacked. It introduces Padding Hacking, which zero-pads ground-truth visual embeddings to the voxel dimension instead of learning a biological mapping.

  • Voxel-level variability: Voxel-wise Pearson correlation and MSE neglect global cortical topography and are sensitive to trial-to-trial neural-response variability.These limitations can fail to penalize structurally incoherent predictions and penalize plausible signals.
  • Semantic-level Hacking: Semantic-level evaluation measures the fidelity of images reconstructed through an Image → Visual Embedding → Syn-fMRI → Reconstructed Image pipeline.Recent studies established this protocol to address low-level variability in fMRI.
  • Semantic-level Hacking: Padding Hacking zero-pads a ground-truth visual embedding, such as VQ-VAE or CLIP, to match voxel dimension N rather than learning a biological mapping.The strategy treats the embedding as voxel data, exposing a trivial vulnerability in the encoding-decoding evaluation protocol.
Loading 2606.30319v1…