PaperlayerView exact revision on arXiv

Source-linked AI summary

Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decoding

Zijiao Chen, Jiaxin Qing, Tiange Xiang, Wan Lin Yue, Juan Helen Zhou

arXiv:2211.06956v3cs.CV

TL;DR

Reconstructing semantically accurate visual stimuli from fMRI is difficult because brain representations are complex, individual variability complicates decoding, and paired annotations are scarce. MinD-Vis addresses this with masked fMRI pre-training and a double-conditioned latent diffusion model, outperforming prior methods in semantic identification and image quality, including an FID of 1.67.

  • Problem

    Reconstructing clear, semantically meaningful visual stimuli from fMRI is difficult because neural activity is complex, individual representations vary, and fMRI-image pairs are scarce.

  • Method

    MinD-Vis uses masked modeling on large-scale unlabeled fMRI data, inspired by sparse coding, followed by a double-conditioned latent diffusion model for image generation.

  • Results

    MinD-Vis generated more plausible and semantically similar images than previous methods, achieving an FID of 1.67 and higher 100-way top-1 identification accuracy than Ozcelik et al.

  • Takeaways & Limitations

    The results support MinD-Vis as a framework for decoding visual stimuli from brain recordings using only a few paired fMRI-image annotations across multiple datasets.

  • Takeaways & Limitations

    MinD-Vis lacks strong pixel-level guidance and interpretation analysis, limiting pixel-level performance and biological understanding of learned features; it also remains focused on individual decoding using the visual cortex.

Abstract

from arXiv · show

Decoding visual stimuli from brain recordings aims to deepen our understanding of the human visual system and build a solid foundation for bridging human and computer vision through the Brain-Computer Interface. However, reconstructing high-quality images with correct semantics from brain recordings is a challenging problem due to the complex underlying representations of brain signals and the scarcity of data annotations. In this work, we present MinD-Vis: Sparse Masked Brain Modeling with Double-Conditioned Latent Diffusion Model for Human Vision Decoding. Firstly, we learn an effective self-supervised representation of fMRI data using mask modeling in a large latent space inspired by the sparse coding of information in the primary visual cortex. Then by augmenting a latent diffusion model with double-conditioning, we show that MinD-Vis can reconstruct highly plausible images with semantically matching details from brain recordings using very few paired annotations. We benchmarked our model qualitatively and quantitatively; the experimental results indicate that our method outperformed state-of-the-art in both semantic mapping (100-way semantic classification) and generation quality (FID) by 66% and 41% respectively. An exhaustive ablation study was also conducted to analyze our framework.

1. Introduction

Visual decoding from fMRI is difficult because brain signals are complex, paired annotations are scarce, and individuals differ in their activation patterns. MinD-Vis addresses these challenges with biologically guided representation learning and double-conditioned diffusion synthesis.

  • Motivation: fMRI-based recovery of original visual stimuli is difficult because paired fMRI-image data and biological guidance are limited.Existing reconstructions are often blurry and semantically meaningless.
  • Motivation: Individual subjects show distinct brain activation patterns for the same visual stimulus, requiring representations that capture shared features while retaining perceptual variation.The framework therefore uses large-scale, demographically rich data and conditional synthesis.
  • Motivation: Masked signal modeling can learn contextual representations from large datasets, while diffusion models provide strong content-generation capability for faithful visual decoding.The motivation combines self-supervised pretraining with probabilistic image synthesis.
  • Contributions: MinD-Vis introduces Sparse-Coded Masked Brain Modeling as a biologically guided feature learner for vision decoding.Its large representation-to-data-space ratio is intended to increase the information capacity of learned representations.
  • Contributions: Double-conditioned latent diffusion strengthens decoding consistency while allowing variance among images with the same semantics.This design is integrated with SC-MBM to generate plausible images while preserving semantic information.
  • Evaluation: Quantitative and qualitative tests span multiple datasets, including a new dataset not previously used for this evaluation task.The experiments assess both semantic decoding and image-generation quality.

2. Related Work

Prior visual-decoding methods use fMRI-linked image features, autoencoders, regression, or generative models, while newer work also uses masked modeling and diffusion. MinD-Vis combines sparse masked fMRI modeling with a latent diffusion pipeline.

  • Visual Decoding Methods: Conventional decoding predicts hierarchical VGG image features from fMRI for classification or generation, while later methods use unpaired learning, regression, or pretrained generative models.These approaches generally train and decode directly from fMRI representations.
  • Individual Variability: Figure 2 illustrates that regions of interest responding to the same visual task differ in location and size across subjects.The primary visual cortex is shown in the left and right hemispheres.
  • Masked Signal Modeling: Masked signal modeling recovers masked data from the remaining signal and has been applied to language and vision representation learning.Typical mask ratios differ by modality, with 75% used for visual signals and 25% for natural language.
  • Diffusion Models: Diffusion models generate samples by gradually adding noise and learning a reverse process that recovers data from corrupted states.The reverse conditional probability is modeled across a fixed-length Markov chain.
  • MinD-Vis: Figure 3 shows MinD-Vis pretraining masked fMRI patches with an autoencoder before sending the learned latent through cross-attention and time-step conditioning in an LDM.The LDM operates on a low-dimensional image latent rather than directly on the original image.
  • Latent Diffusion: Latent diffusion performs generation in compressed latent feature space, reducing computational cost and spatial downsampling compared with pixel-space diffusion.Its pipeline uses an autoencoder to compress and reconstruct images.

3. Methodology

MinD-Vis learns sparse fMRI representations with masked modeling, then uses double-conditioned latent diffusion to synthesize semantically consistent visual reconstructions. Its design addresses fMRI redundancy, dimensionality differences, individual variation, and the need for stronger conditioning.

  • Motivation: fMRI contains spatial redundancy, dimensionality mismatch with images, and domain shifts across datasets and individuals.Neighboring voxels can have similar amplitudes, ROI voxel counts are much smaller than image pixel counts, and ROI size and location vary.
  • Framework: MinD-Vis uses two sequential stages: masked fMRI representation learning followed by conditional image synthesis with a latent diffusion model.Stage A learns fMRI representations with masked signal modeling; Stage B integrates the encoder with an LDM through cross-attention and time-step conditioning.
  • Stage A: SC-MBM: SC-MBM masks a large portion of fMRI patches and uses a large representation-to-patch-size ratio to increase information capacity in an over-complete space.The approach is motivated by spatial redundancy in fMRI and sparse coding of sensory information in the visual cortex.
  • Stage B: DC-LDM: The fMRI encoder conditions the LDM through cross-attention while an additional projected conditioning signal is added to the UNet time-step embeddings.This double conditioning is intended to provide stronger guidance because fMRI-to-image decoding emphasizes semantic consistency across similar brain activities.
  • Decoding outcome: MinD-Vis is reported to produce more accurate and plausible reconstructions with better-preserved semantic information than previous methods.The qualitative comparison is performed on the GOD test set against three other reconstruction methods.

4. Experiments

Experiments use HCP, GOD, and BOLD5000 to pre-train, fine-tune, and validate MinD-Vis. Evaluation emphasizes semantic correctness and generation quality, while FID is treated as a reference because GOD has few test images.

  • Datasets: Three public datasets—HCP, GOD, and BOLD5000—support pre-training, fine-tuning, and validation.The combined HCP and GOD pre-training data contain 136,000 fMRI segments from 340 hours of scans; GOD pairs are used for fine-tuning and BOLD5000 for validation.
  • Datasets: The GOD dataset contains 1,250 images from 200 classes, with 1,200 training images and 50 test images whose classes do not overlap.BOLD5000 contains 5,254 fMRI-image pairs from 4,916 images, including 113 test images.
  • Implementation: The full model uses a 16-voxel patch size, 1024-dimensional embeddings, 24 encoder layers, and a 0.75 mask ratio.Images are generated at 256 × 256 resolution with 250 PLMS steps.
  • Evaluation: Semantic correctness is evaluated with 1000-trial n-way top-1 and top-5 classification accuracy rather than pixel-level metrics.A pre-trained ImageNet1K classifier is used to assess semantic correctness.
  • Evaluation: FID compares ground-truth and generated test images, but is used only as a reference because the GOD test set is small and may underestimate the distribution.

5. Results

MinD-Vis produced semantically plausible reconstructions, with stronger identification accuracy and lower FID than prior methods, while remaining consistent across samplings and datasets. Ablations indicate that SC-MBM pre-training, large patch embeddings, high masking, and jointly optimized double conditioning materially affect performance.

  • GOD benchmark: FID reached 1.67 with MinD-Vis best samples, compared with 2.36 or more for Ozcelik et al. and other methods.Prior conventional methods had higher pixel similarity but fewer semantically meaningful details.
  • Generation consistency: Across five samplings, average success rates were 0.2385±0.030 for 50-way and 0.1736±0.029 for 100-way identification, with average FID 2.22±0.3.These samplings shared shape, color, texture, and semantic details, indicating stable semantic categories across trials.
  • SC-MBM ablation: Without SC-MBM pre-training, both directly trained models underperformed the Full model and failed to generate sensible images.The comparison used an untrained full-depth encoder and a shallower untrained encoder with fewer parameters.
  • SC-MBM ablation: Accuracy peaked at 23.9% with a 1024-dimensional patch embedding, then decreased at larger dimensions.Larger embeddings increase representation capacity but also make the model more data-hungry.
  • SC-MBM ablation: A 0.75 mask ratio achieved the highest average accuracy while reducing memory consumption because the encoder processed only unmasked patches.The high ratio was motivated by spatial redundancy in fMRI data.

6. Discussion and Conclusion

MinD-Vis decodes visual stimuli from sparse paired fMRI-image annotations through a two-stage framework and produces more plausible, semantically similar images than previous methods. Its current limitations concern pixel-level guidance, biological interpretation, and reliance on individual visual-cortex decoding.

  • Limitations: Limited pixel-level guidance constrains pixel-level performance and limits biological understanding of the features learned by MBM.The authors identify both pixel-level performance and interpretation analysis as limitations of the current framework.
  • Future work: The framework currently focuses on individual decoding using only the visual cortex, leaving cross-subject generalization and other brain regions for future work.The authors propose extending the method beyond visual-cortex-only individual decoding.
  • Framework: MinD-Vis uses masked fMRI pre-training followed by double-conditioned latent diffusion to decode visual stimuli from few paired annotations.Stage A learns generalizable context knowledge from large-scale unlabeled fMRI; Stage B generates plausible seen images from the learned representations.
  • Results: MinD-Vis generates more plausible and semantically similar images than previous methods across multiple datasets.The conclusion reports that the framework pushes the state of the art forward in decoding visual stimuli.

A. More Generation Samples

The appendix presents additional generated samples, ablation examples, failure cases, datasets, and implementation details for MinD-Vis. Samples are generated at 256×256×3 resolution with 250 PLMS steps.

  • Generation settings: Samples are generated at 256×256×3 resolution using 250 PLMS steps.The paper states that more samples can be found and generated in the code base.
  • Ablation study: Figure A.4 compares the full model against variants removing SC-MBM pre-training, changing the fMRI encoder, limiting optimization, or using single conditioning.The full model uses SC-MBM pre-training, jointly optimizes the fMRI encoder and cross-attention heads, applies double conditioning, and uses an ImageNet-pre-trained LDM.
  • Failure cases: Figure A.5 attributes typical failures to disjoint GOD training and testing classes and stimulus-unrelated thoughts captured by fMRI.The stated boundary is that geometric information may be learned from training while unseen test classes cannot be inferred.
  • Datasets: The HCP pre-training data comprise approximately 2000×15-minute 3T resting-state fMRI runs from 1091 subjects, using visual-cortex V1-V4 with approximately 4000 voxels.The HCP release supplies the large-scale magnetic-resonance dataset used for pre-training.
  • Datasets: BOLD5000 provides 5,254 fMRI-image pairs from 4,916 natural images and is introduced as a reconstruction-task evaluation dataset.Its manually annotated ROIs contain approximately 1,500 voxels per subject, and the pre-trained encoder is directly applied despite ROI differences.
  • Pre-training data: The combined HCP and GOD pre-training set contains 136,000 pure fMRI samples and is described as the largest pre-training fMRI dataset used for this task.HCP contributes 130,000 averaged fMRI time points, while GOD contributes training and testing fMRI.

C. Results on Different Subjects

Results vary across subjects and datasets, with signal-to-noise ratio associated with performance differences. The appendix defines evaluation as an N-trial, n-way top-1 semantic classification test using a pre-trained ImageNet classifier.

  • Subject differences: Higher signal-to-noise ratio leads to better performance in the reported experiments.BOLD5000 has much higher SNR than GOD, and GOD Subject 3 has significantly higher SNR than the other GOD subjects.
  • Evaluation: The evaluation uses a 1000-trials 50-way top-1 semantic classification test on the best-generated samples.Table C.1 identifies voxel-wise mean SNR as the reported signal-quality measure.
  • Evaluation: The N-trial, n-way top-1 test measures the semantic accuracy of generated images by comparing generated and corresponding ground-truth image classes.The procedure denotes the generated image as x, the ground truth as x̂, and the class label as y.
  • Evaluation: A pre-trained ImageNet classifier determines whether generated and ground-truth images belong to the same class rather than relying on handcrafted class features.The implementation uses a pre-trained ResNet, and other model-based pre-trained classifiers do not change the metric result.
  • Evaluation: The algorithm outputs a success rate r∈[0,1] after N classification trials.Each trial obtains the ground-truth class, samples n−1 random classes, and tests whether the generated image is classified correctly among the n candidates.

D.2. SC-MBM Pre-training

SC-MBM pre-training masks and reconstructs fMRI patches in an enlarged latent representation, using asymmetric encoding and decoding with additional sparsification. The resulting encoder is then integrated with a double-conditioned latent diffusion model for fine-tuning.

  • Architecture: fMRI voxels are divided into patches and transformed into embeddings by a one-dimensional convolution with stride equal to patch size.This adapts masked-image-modeling practice to fMRI data.
  • Representation: The Full model uses patch size 16 and embedding dimension 1024, giving an embedding-to-patch dimension ratio of 64.The enlarged representation dimension is intended to boost the information capacity of fMRI representations.
  • Masked modeling: SC-MBM uses an asymmetric architecture with a much smaller decoder, random masking, mask tokens, and positional embeddings.The masked patch embeddings are transformed into the decoder’s embedding space for reconstruction.
  • Training: Random sparsification sets 20% of voxels in each fMRI sample to zero as data augmentation.The procedure is applied to train the data-hungry ViT-like model.
  • Training setup: SC-MBM pre-training is performed on eight RTX3090ti GPUs until convergence, while DC-LDM fine-tuning uses one RTX3090ti GPU for 500 epochs.The paper provides separate hyperparameter tables for the two stages.
  • Fine-tuning: After pre-training, the fMRI encoder is integrated with a latent diffusion model through cross-attention and time-step conditioning.During fine-tuning, the encoder and cross-attention modules are optimized while other LDM components remain intact.
  • Reconstruction: Masked brain prediction examples use a 0.75 mask ratio and compare original, masked, and decoder-recovered visual-cortex fMRI data.The figure also displays correlations between original and recovered fMRI.

E. Other Ablation Studies

The ablations examine masking, architecture, preprocessing, augmentation, and representation capacity choices for MinD-Vis. Results support random masking, crop-ratio tuning, and a larger representation space, while image-feature guidance and some alternatives do not improve performance.

  • Patch Sizes: Smaller patch sizes improve representation accuracy as patch size increases from 16 to 64, while the minimal tested patch size is not fully specified here.The analysis attributes the trend to using more words to describe the same voxel data.
  • Mask Strategy: Random masking outperforms the biologically motivated focus masking strategy that masks more patches in the V1 region.The focus strategy was designed around seven visual-cortex networks, including V1, but did not outperform random masking.
  • Pretext Tasks: Adding VGG-derived image-feature reconstruction to masked-modeling pretraining does not outperform masked modeling alone.The auxiliary image-feature decoder uses second-layer pretrained VGG features and adds its reconstruction loss to the masked-modeling loss.
  • Crop Ratio: A crop ratio of 0.2 gives the best finetuning performance among the tested ratios from 0 to 0.4.Random cropping accommodates differences in the image regions emphasized by subjects.
  • Evaluation Protocol: The ablation protocol reports 1000-trial 50-way top-1 semantic classification accuracy after 500-epoch pretraining and 500-epoch GOD finetuning.The full-model settings are marked in gray in Table E.4.
  • Information Capacity: The framework uses a representation space much larger than the data space to increase fMRI information capacity in analogy with sparse coding.Information capacity is defined through representation-space dimensionality relative to data-space dimensionality, R = ˜L/L.

G. Pixel-level metrics

Pixel-level metrics provide an additional evaluation perspective alongside semantic metrics. MinD-Vis performs better on semantic measures than on pixel-level measures, reflecting a fidelity–meaningfulness trade-off.

  • Semantic versus Pixel-level Evaluation: MinD-Vis and Ozelik outperform the other methods on semantic metrics but not on pixel-level MSE and LPIPS metrics.The paper prioritizes semantic recovery because it is considered crucial for understanding the content represented in fMRI.
  • Pixel-level Evaluation: MSE and LPIPS are used as additional pixel-level benchmarks for comparing reconstructed images.The corresponding comparison is reported in Table G.5.

H. 2-way and 5-way metrics

Additional 2-way and 5-way classification evaluations further compare the reconstruction methods. MinD-Vis outperforms the other methods on both metrics.

  • Classification Metrics: Table H.6 compares methods using 2-way and 5-way classification metrics.
Loading 2211.06956v3…