Source-linked AI summary
Future Video Generation Better Aligns with the Human Visual Cortex than Observed Video
Chang-Bae Bang, Hyungjin Chung, Byung-Hoon Kim
TL;DR
Studies of vision-model alignment have largely examined representations of observed stimuli, despite the brain also predicting upcoming input. This paper compares AR and non-AR video-model representations with visual-cortex fMRI responses and finds stronger alignment for future generation, alongside a hierarchy-specific pattern and a human preference for outputs from aligned layers.
Problem
Vision-model alignment studies have largely focused on observed visual stimuli, although human visual processing also predicts upcoming stimuli.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
The findings may provide indirect evidence that the visual cortex performs future-oriented computation, especially in higher-order visual cortex.
Source: §6 · p. 9The behavioral experiment reflects neural-alignment layer profiles in human preference but does not establish how neural alignment gives rise to that preference.
Source: §6 · p. 10
Abstract
from arXiv · showhide
Studying the alignment between the internal representations of vision models and the responses of the visual cortex to the same observed visual stimuli has enabled us to better understand human visual processing. However, studies so far have largely overlooked the fact that the human brain not only processes observed visual stimuli, but also predicts upcoming stimuli based on what has been observed. Accordingly, we hypothesize that internal representations for generating future video frames are better aligned with the predictive nature of human visual processing than representations of the observed video itself. To this end, we compare the alignment between human video-watching fMRI responses in the visual cortex and the internal representations from two types of video diffusion models, an autoregressive (AR) model and its non-AR base model. We first conduct a within-model analysis of the AR video diffusion model and show that the representations for future video generation align better with the visual cortex than the representations of the observed video. We then compare the internal representations of the AR model with those of its non-AR base model and again show that the representations for future video generation align better with the visual cortex than the representations for observed video reconstruction by the base model. Specifically, the alignment of observed video reconstruction is concentrated in lower-order visual cortex, whereas that of future video generation is concentrated in higher-order visual cortex. Finally, we show in a human behavioral experiment that humans prefer videos generated by amplifying the contributions of individual layers that align better with the visual cortex.
1 INTRODUCTION
Prior alignment studies have largely focused on observed stimuli, although the brain also predicts upcoming visual input. This study compares video-model representations with visual-cortex responses and reports stronger alignment for future generation, a hierarchy-specific pattern, and a related human preference.
1 INTRODUCTION
- The study compares an autoregressive model and its non-AR base model using fMRI responses recorded as people watch naturalistic videos.Source: §1 · p. 2
- Future video generation aligns better with the visual cortex than the observed video represented by both the AR model and the human brain.The study compares internal video-diffusion representations with human video-watching fMRI responses.Source: §1 · p. 3
- Observed video reconstruction aligns more with lower-order visual cortex, whereas future video generation aligns more with higher-order visual cortex.Source: §1 · p. 3
- Humans prefer future videos generated by amplifying individual layers that align better with the visual cortex.Source: §1 · p. 3
2 RELATED WORK
Related work covers representation alignment, predictive visual processing, and video-diffusion representations. Together, these areas motivate comparing future-video generation with observed-video representations.
2.1 BRAIN–VISION MODEL REPRESENTATION ALIGNMENT
- Vision-model alignment studies use neural responses to examine how visual information may be processed, with prior work spanning object recognition, video models, and image diffusion models.Source: §2.1 · p. 3
2.2 PREDICTIVE NATURE OF HUMAN VISUAL PROCESSING
- The visual cortex represents upcoming stimuli as well as observed input, with evidence for prediction in both lower- and higher-order visual areas.Source: §2.2 · p. 3
2.3 INTERNAL REPRESENTATION OF DIFFUSION VIDEO MODELS
- Non-AR diffusion models denoise all video frames together, whereas AR diffusion models sequentially generate future video conditioned on observed video.Source: §2.3 · pp. 3–4
3 PRELIMINARIES
The preliminaries describe the fMRI dataset and notation, then introduce latent-space video diffusion as denoising along a noise-to-data path.
3 PRELIMINARIES
- The BOLD Moments Dataset contains fMRI responses from ten subjects who watched 1,102 three-second naturalistic videos, with 1,000 videos for training and 102 for testing.Source: §3 · p. 4
- In the response notation, Y indexes responses across videos and cortical vertices, m indexes subjects, p indexes vertices, and r denotes a region of vertices.Source: §3 · p. 4
- Video diffusion maps a video to a latent representation and generates it by iteratively denoising along a flow-matching path between noise and data.At noise level σ, the latent interpolates the clean latent with Gaussian noise; the schedule runs from the noisiest step to the cleanest.Source: §3 · p. 4
4 METHODS AND EXPERIMENT SETUP
The methods compare observed-video and future-generation representations from AR models with observed-video reconstruction from their non-AR base models. They map these representations to fMRI responses using vertex-wise encoding and variance partitioning.
4 METHODS AND EXPERIMENT SETUP
- The analysis focuses on 7,894 HCP-MMP-atlas Visual System vertices across five divisions and excludes vertices with noise ceilings at or below 10%.Source: §4 · p. 4
- The main analysis uses 40-layer Helios fine-tuned from Wan-2.1 T2V-14B and replicates the analysis with 30-layer Self-Forcing and CausVid models.Videos are resampled or padded and sampled at model-specific resolutions and frame rates.Source: §4 · p. 4
- The AR model receives the observed video and generates its future, while the non-AR base model denoises the observed video to reconstruct it.Source: §4 · p. 5
- Vertex-wise ridge-regression encoding models predict fMRI responses from flattened observed-video, future-generation, and reconstruction representations, with test videos held out from fitting and selection.Source: §4 · p. 5
- Variance partitioning separates unique from shared response contributions for observed video versus future generation and reconstruction versus future generation.The selected future-generation and reconstruction features use best-aligned denoising steps and layers chosen on training folds.Source: §4 · pp. 5–6
5 RESULTS
Across within- and cross-model analyses, future-video representations align more strongly with the visual cortex, with their contribution increasing in higher-order regions. Layer-wise amplification also produced a human preference pattern correlated with neural alignment.
5.1 WITHIN-MODEL ANALYSIS
- 0.222 vs −0.006: future video generation has a positive unique contribution in the Visual System, while the observed video’s unique contribution is negative.Across the cortical surface, future video generation’s unique contribution grows toward higher-order visual cortex, while the observed video’s shared and unique contributions concentrate in lower-order cortex.Source: §5.1 · pp. 6–7
5.2 CROSS-MODEL ANALYSIS
- 0.334 vs 0.301: future video generation’s best encoding accuracy exceeds observed video reconstruction’s, with significant advantages in Dorsal Stream, Ventral Stream, and MT+ Complex.The three higher-order divisions each have p = 0.002.Source: Figure 3 · p. 7, §5.2 · pp. 7–8
5.3 NEURAL ALIGNMENT TO BEHAVIORAL VALIDATION
- Across eight STG-guided layers, neural alignment and human preference correlate at r = 0.75 (p = 0.029).The experiment amplified one layer at a time during future-video generation; encoding accuracy peaked at layer 11 and preference at layer 16.Source: Figure 4 · p. 8, §5.3 · pp. 8–9
5.2 CROSS-MODEL ANALYSIS
- Unique contributions shift across the hierarchy: observed video reconstruction is concentrated in lower-order cortex, while future video generation grows toward higher-order cortex.The cortical-surface plot shows the same spatial pattern.Source: Figure 3 · p. 7, §5.2 · p. 8
- Future-generation alignment is highest at the noisiest step and declines with denoising, whereas reconstruction alignment peaks early and then declines.Both representations follow an inverted-U profile across layers, peaking at an early-to-middle layer.Source: Figure 3 · p. 7, §5.2 · p. 8
6 DISCUSSION
The discussion interprets future-video alignment as consistent with predictive visual processing, especially in higher-order cortex, and relates layer-wise alignment to human preference. It also notes limits in temporal scope and causal interpretation.
Interpretation
- The absence of a unique contribution from observed tokens, alongside stronger future-video alignment, is indirect evidence for future-oriented visual-cortex computation.The discussion connects this interpretation to predictive signals previously reported in higher-order visual cortex.Source: §6 · p. 9
Why the noisiest step and an early-to-middle layer
- Future-video alignment peaks at the noisiest step, a result the authors relate to Bayesian accounts in which neural populations encode a range of possible stimuli.They contrast this with observed-video reconstruction, whose alignment peaks at a less noisy step and whose unique contribution is concentrated in lower-order cortex.Source: §6 · p. 9
Comparison with a wide range of vision models
- Self-supervised video models aligned best overall, but their advantage over future-video representations narrowed toward higher-order visual cortex.The authors describe this narrowing as possible indirect evidence that higher-order cortex predicts future visual stimuli; prior work had reported strong alignment for self-supervised video models (Höfling et al., 2026; Tang et al., 2025).Source: §6 · p. 9
From neural alignment to human preference
- Humans preferred videos generated by amplifying layers that aligned better with cortex; the authors suggest richer visual representations may make generated videos more natural.They relate this interpretation to prior diffusion-model studies linking richer internal representations with improved generated content (Zhang et al., 2025).Source: §6 · p. 9
Limitations
- The study tests only 3 s observed-to-1 s future videos, maps one cortical response per video, and does not establish that neural alignment causes preference.The response time course is not compared with the model’s internal representations.Source: §6 · pp. 9–10
7 CONCLUSION
The paper concludes that future-video representations align better with visual-cortex responses than observed video or its reconstruction, with the advantage growing along the visual hierarchy. Layer amplification also yielded a human preference pattern consistent with cortical alignment.
Conclusion
- Future-video generation aligns better than both the observed video and its reconstruction, with its advantage increasing along the visual hierarchy.Unique contributions from the observed video and its reconstruction concentrate in lower-order visual cortex; people also preferred videos generated from layers that align well with cortex.Source: §7 · p. 10
Research transparency
- The authors used generative AI for research feedback, implementation, translation, result interpretation, coding, drafting, literature work, brainstorming, editing, and reference formatting, and reviewed the assisted work.They state that AI was not used to generate synthetic datasets, develop theoretical models or hypotheses, or support qualitative data analysis.Source: AI USE STATEMENT · p. 10
Data and ethics
- The fMRI data came from the anonymized BOLD Moments Dataset, and adult volunteers gave informed consent for the no-risk video-watching preference experiment.No identifying information about the observers was collected or reported.Source: ETHICS STATEMENT · p. 10
A.1 REPLICATION ON OTHER AUTOREGRESSIVE VIDEO MODELS
The appendix repeats the within-model and cross-model analyses on CausVid and Self-Forcing, two autoregressive models fine-tuned from the same non-AR base model.
A.1 REPLICATION ON OTHER AUTOREGRESSIVE VIDEO MODELS
- The appendix repeats the within-model and cross-model analyses on CausVid and Self-Forcing using the same encoding and variance-partitioning procedures.Representations were extracted across all four denoising steps and all 30 layers.Source: §A.1 · p. 14
- Both replicated models are fine-tuned from Wan-2.1 T2V-1.3B and generate future video in four denoising steps.The models are CausVid and Self-Forcing.Source: §A.1 · p. 14
- Figure 5 presents the within-model analysis for CausVid.The broader replication includes analyses of both future-video generation and observed-video reconstruction.Source: Figure 5 · p. 14, §A.1 · p. 14
A.2 ABLATION STUDY
The ablation compares future and observed token readouts within the autoregressive model. Future tokens show stronger encoding accuracy and a larger unique contribution across visual divisions.
A.2 ABLATION STUDY
- The AR activation is divided into observed and future tokens; the analysis reads out future tokens and also tests observed and concatenated readouts using the same projection and ridge regression.The observed-token representations are also variance-partitioned, with each representation cell selected on training folds for each region and subject.Source: §A.2 · pp. 14–15
- Visual System encoding accuracy was 0.334 for future tokens, 0.211 for observed tokens, and 0.309 for concatenated tokens.The figure caption identifies the comparison as future-video-generation accuracy under different token readouts.Source: §A.2 · p. 15, Figure 9 · p. 16
- Future video generation’s unique contribution exceeded that of observed tokens in every division (p = 0.002), increasing from 0.055 in Primary Visual to 0.189 in MT+ Complex.In the Visual System, the shared contribution was 0.213, while the unique contributions were 0.121 for future generation and −0.001 for observed tokens.Source: §A.2 · p. 16
A.3 COMPARISON WITH A WIDE RANGE OF VISION MODELS
The section compares the future-video-generation representation with a broad set of image and video vision models. It describes how model activations are extracted and evaluated.
A.3 COMPARISON WITH A WIDE RANGE OF VISION MODELS
- The comparison includes 80 image and video vision models alongside the autoregressive model’s future-video-generation representation.The passage describes this as a comparison of internal representations.Source: §A.3 · p. 16
- For video models, repeated passes cover the clip using windows of n frames spaced δ frames apart; for image models, each pass is a single frame.The number of video-model passes is K = ⌈T/(nδ)⌉, while image models use K = T.Source: §A.3 · p. 16
- The reported encoding accuracies cover every model in the Visual System and identify the best model of each group in each division.Vision-model activations are flattened over tokens and channels, averaged over passes, and evaluated using the same sparse projection and nested cross-validation.Source: §A.3 · p. 17
A.4 FMRI DATASET AND PREPROCESSING
This appendix specifies the fMRI data and analysis pipeline, layer-guidance manipulation, human-preference evaluation, and statistical tests.
A.4 FMRI DATASET AND PREPROCESSING
- Ten subjects watched 1,102 naturalistic 3-second videos; normalized responses were averaged across repeats and organized into five visual divisions.Source: §A.4 · pp. 17–19
A.5 FEATURE CONSTRUCTION AND ENCODING MODEL
- Features were sparsely projected to 6,004 dimensions, and vertex-wise ridge penalties were selected by nested cross-validation on the 1,000 training videos.Source: §A.5 · p. 19
- Each subject’s region used the grid cell with the highest mean out-of-fold score; test videos were excluded from model selection.Source: §A.5 · p. 19
A.6 VARIANCE PARTITIONING
- Variance partitioning combines two feature spaces with convex weights fit to training responses, separating their unique and shared contributions.Source: §A.6 · p. 19
A.7 SPATIOTEMPORAL SKIP GUIDANCE
- STG-Attention skips one layer’s self-attention to form a perturbed weak model, then pushes the generation path away from that model.The guidance used w = 1.5 and classifier-free guidance scale s = 5.0.Source: §A.7 · p. 20
A.8 HUMAN PREFERENCE EVALUATION
- The preference test compared nine generation arms across 720 trials by five observers, using paired video preferences to rank the arms.Eight arms applied STG at individual layers; the baseline used classifier-free guidance alone.Source: §A.8 · p. 20
A.9 STATISTICAL TESTING
- Paired comparisons across ten subjects used an exact two-sided sign-flip test, and the layer-profile correlation used an exact permutation test.Log-worth intervals were 95% bootstrap intervals from 2,000 observer resamples.Source: §A.9 · p. 20