Cinematic Mindscapes: High-quality Video Reconstruction from Brain Activity

• Earlier deep-learning and representation-learning work advanced visual class detection and static-image reconstruction from fMRI.Source: §1 · p. 1
• A two-second fMRI scan can span about 60 video frames, making recovery of high-frame-rate visual experience a complex decoding task.Source: §1 · pp. 1–2
• Hemodynamic response delays vary across subjects and brain regions, making fixed temporal shifts of fMRI suboptimal.Source: §1 · p. 2
• MinD-Video combines a progressively trained fMRI encoder with an augmented Stable Diffusion model, trained separately and then fine-tuned together.Source: §1 · p. 2
• 85% semantic accuracy and 0.19 SSIM are reported, outperforming previous state-of-the-art approaches by 45%.Source: §1 · p. 2

• Earlier video reconstruction treated videos as multiple image reconstructions, while later conditional video GANs improved quality and frame rate but faced data scarcity.Source: §2 · p. 2
• A related self-supervised approach reported better results than the conditional video GAN, but its generated videos had low visual fidelity and semantic meaning.Source: §2 · pp. 2–3
• Masked brain modeling learns general fMRI features by reconstructing masked data in a large dataset, supporting downstream transfer with limited annotated data.Source: §2 · p. 3
• CLIP learns a shared image-and-language latent space through contrastive learning, with semantic information represented across both modalities.Source: §2 · p. 3
• Stable Diffusion generates compressed data latents rather than data directly, reducing computational requirements while supporting detailed image generation.Source: §2 · p. 3

3.1 Motivation and Overview

• A decoupled fMRI encoder and video generator are trained separately, then finetuned together to support adapting either module to new architectures.The encoder supplies embeddings as conditions for video generation.Source: §3.1 · p. 3

3.2 The fMRI Pre-processing

• Visual-cortex regions are defined by parcellation for pretraining, while the target dataset uses reproducibility-based voxel selection, retaining the top 50% of statistically significant voxels.The target-dataset selection uses a one-sample t-test with P<0.01 and Bonferroni correction.Source: §3.2 · pp. 3–4

3.3 Progressive Learning and fMRI Encoder

• The encoder progresses from general features to semantic-related features through finetuning, with multimodal contrastive learning bringing fMRI embeddings toward a shared text-image space.The contrastive stage trains on fMRI, video, and caption triplets.Source: §3.3 · pp. 4–5
• Masked brain modeling learns general visual-cortex features by masking about 75% of fMRI tokens and training an autoencoder to recover them.The pretraining uses visual-cortex data from the Human Connectome Project.Source: §3.3 · p. 4
• A sliding-window spatiotemporal attention layer processes multiple fMRI frames to account for hemodynamic-response lag; spatial attention captures token correlations and temporal attention captures correlations across the window.The paper notes that HRF varies across individuals and sessions, motivating participant-adaptive algorithms rather than personalized HRF models.Source: §3.3 · pp. 4–5

3.4 Video Generative Module

• The video generator adapts Stable Diffusion with sparse causal attention over the previous two frames, allowing scene dynamics while maintaining smoothness.Adversarial guidance uses the average test-set fMRI as a negative condition to help distinguish generated videos for different fMRI inputs.Source: §3.4 · p. 5

3.5 Learning from the Brain - Interpretability

• Attention maps across encoder layers and learning stages are projected onto brain surfaces to examine shifts from local relations toward global, abstract features and show regional contributions.The visualizations use average maps across testing samples.Source: §3.5 · p. 6

4.1 Datasets

• Pretraining uses 600,000 HCP fMRI segments, while the paired benchmark contains three subjects, 4,320 training examples, and 1,200 test fMRIs.The benchmark videos span animals, people, and natural scenery.Source: §4.1 · p. 6

4.2 Implementation Details

• Videos are downsampled from 30 to 3 FPS, yielding six frames per fMRI frame; reconstruction can span longer videos when multiple fMRI frames and more GPU memory are available.The augmented Stable Diffusion model is tuned at 256×256 resolution with 3 FPS.Source: §4.2 · p. 6

4.3 Evaluation Metrics

• Evaluation combines frame-level SSIM and ImageNet N-way top-K classification with video-level classification using a VideoMAE-based classifier.The frame-level classification success rate is averaged over 100 trials.Source: §4.3 · pp. 6–7
• Figure 4 compares generated samples with prior-work examples and reports that Mind-Video outputs are more semantically meaningful and better match ground truth.Source: Figure 4 · p. 7

5 Results

• 0.186 SSIM on Subject 1’s first test video exceeded the previous state of the art by 45%.Across all subjects and test videos, the method outperformed Kupershmidt et al. (2022) by 35% on average.Source: §5 · p. 7
• 0.849 video-classifier 2-way accuracy and 0.2 50-way accuracy exceeded chance; image-classifier accuracies were 0.795 and 0.19, respectively.The corresponding chance levels were 0.5 for 2-way and 0.02 for 50-way classification.Source: §5 · p. 7
• Results were consistent across subjects, whose regions of interest contained 6,016, 6,224, and 3,744 voxels, respectively.Subject 3 had about half as many voxels as the other subjects, which may allow a larger contrastive-learning batch size.Source: §5 · p. 7
• Reconstructions recovered objects, people, animals, scenes, motions, and scene dynamics, including transitions shown in Figure 5.The paper defines video semantics to include both video contents and motion or scene dynamics.Source: §5 · p. 7, Figure 5 · p. 8
• Window size 2 performed best in general, while removing multimodal contrastive learning or adversarial guidance reduced generation quality or evaluation performance.Without adversarial guidance, videos could become highly similar, reducing the distinguishability of fMRI embeddings; Table 1 covers the ablations and subject evaluations.Source: Figure 6 · p. 8, Table 1 · p. 8

5.1 Interpretation Results

• The visual cortex, including central and peripheral visual fields, received the highest attention, while dorsal-attention and default-mode networks also contributed.The visual-cortex dominance held across layers and training stages.Source: Figure 7 · p. 9
• Early encoder layers focused attention on brain-region structure, whereas deeper layers showed more dispersed attention associated with holistic, abstract visual features.The analysis averages attention across testing samples and heads and displays network-level and voxel-wise patterns in Figure 7.Source: §5.1 · pp. 8–9
• As learning progressed, first-layer attention increased in higher cognitive networks and decreased in the visual cortex, which the authors interpret as increasing semantic learning.The stages compared were before contrastive learning, after contrastive learning, and after co-training with the video-generation model.Source: §5.1 · p. 9

6 Discussion and Conclusion

• The co-trained pipeline recovers video semantics, motions, and scene dynamics compared with ground truth, establishing a new state-of-the-art in this domain.It combines progressive fMRI encoder learning with an augmented Stable Diffusion model and fMRI adversarial guidance.Source: §6 · p. 9
• The method remains intra-subject; inter-subject generalization is unexplored, and reconstructions use less than 10% of cortex voxels.The authors also leave use of whole-brain data unexploited.Source: §6 · p. 9
• The authors identify privacy protection and prevention of malicious use as requirements for prospective applications in neuroscience and brain-computer interfaces.Source: §6 · p. 9

A Q&A Session

• The performance gain is attributed to both the fMRI encoder and Stable Diffusion, rather than to the diffusion model alone.The encoder captures spatiotemporal fMRI information; Stable Diffusion uses those representations to generate videos.Source: A Q&A Session · p. 12
• The encoder targets nuanced continuous semantics, such as texture and depth, which discrete object labels may not adequately represent.The authors describe discrete classification as a trade-off between classification complexity and a restricted solution space; reports reasonable performance only for tasks below 50-way.Source: A Q&A Session · p. 12
• The authors argue that continuous semantic representations better reflect the complexity and diversity of neural processes and provide more interpretable decoding insights.Source: A Q&A Session · p. 12
• The fMRI-to-video task is not merely missing-frame imputation on an fMRI-to-image pipeline; the authors characterize the tasks and their complexities as fundamentally different.Source: A Q&A Session · p. 12
• Video reconstruction must capture changing brain activations and their relation to moving video elements, unlike static-image decoding focused on spatial information.The hemodynamic response delays and blurs neural timing, while fMRI's low temporal resolution challenges fast-change capture.Source: A Q&A Session · p. 12
• Stable Diffusion's probabilistic generation can produce slight differences between frames, while video generation also requires frame consistency.Source: A Q&A Session · p. 13

Large-Scale Pre-training

• The encoder is pre-trained on 600,000 HCP fMRI samples, then tuned with target-dataset fMRI using masked brain modeling.The ViT-large-based training takes around three days on eight RTX3090 GPUs; Tab. B.1 is identified as the hyperparameter table.Source: B More Implementation Details · p. 13, Table B.1 · p. 13

Multimodal Contrastive Learning

• The pre-trained encoder gains temporal attention heads and is contrastively trained on fMRI-image-text triplets, using a randomly selected video frame from each fMRI window.The dataset has about 4,000 samples; training uses dropout 0.6 and runs for 12,000 steps.Source: B More Implementation Details · p. 13

Training of Augmented Stable Diffusion

• The augmented Stable Diffusion model is trained on target-dataset videos downsampled from 30 FPS to 3 FPS at 256×256 resolution.The authors say it can work at full time resolution; downsampling is used because of GPU-memory limits and to adapt the 512×512-pretrained model.Source: B More Implementation Details · p. 13
• Before co-training, Stable Diffusion's self-attention, cross-attention, and temporal-attention heads are trained with text conditioning for 800 steps.The training takes around two hours on one RTX3090.Source: B More Implementation Details · p. 13

Co-training

• During co-training, 77×768 fMRI-encoder embeddings condition the augmented Stable Diffusion model; the encoder and part of Stable Diffusion are updated.Co-training runs for 15,000 steps; inference uses 200 diffusion steps and an average fMRI as negative guidance at scale 12.5.Source: B More Implementation Details · p. 13

C Analysis of Visual Results

• Across all three Wen (2018) subjects, generations are consistent; Subject 3's higher numeric results may reflect its larger training batch.Subjects 1 and 2 use around 6,000 ROI voxels each, while Subject 3 uses around 3,000; samples are shown in Fig. C.1.Source: C Analysis of Visual Results · p. 13, Figure C.1 · p. 14
• Incomplete-modality contrastive learning looks worse than the full three-modality model, but still outperforms inference without adversarial guidance, which produces visually meaningless results.The full model uses full-modality contrastive learning with adversarial guidance; reconstruction samples appear in Fig. C.2.Source: C Analysis of Visual Results · p. 14, Figure C.2 · p. 14
• Some failures misidentify animals or objects, yet motions, colors, and scene dynamics can still be reconstructed correctly.The cited example misses an airplane and a flying bird but recovers similar fast-motion scenes; Fig. C.3 shows fail cases.Source: C Analysis of Visual Results · p. 15, Figure C.3 · p. 15