Do Vision Model See Like the Brain? A Comparison Across EEG Encoding Model

1. Introduction

• The study fills a gap by comparing CNNs and transformers in one EEG analysis, using millisecond resolution to locate where their depth profiles begin to differ.Prior work had largely tested CNNs with fMRI or invasive recordings, while transformer comparisons had not combined broad model coverage with layer-by-layer EEG analysis.Source: §1 · pp. 1–2
• Across six CNNs and two vision transformers, EEG correspondence is similar at early and intermediate depths but diverges at deeper layers, with a growing transformer advantage at final layers.The study evaluates Pearson correlation at every individual layer or transformer block.Source: §1 · pp. 2–3
• To disentangle architecture from training objective, the study compares MoCo-v1 with same-architecture supervised CNNs and CLIP-RN50 with same-objective CLIP-ViT-B/32.These controlled comparisons vary training objective or architecture while holding the other factor fixed.Source: §1 · pp. 2–3
• The introduction proposes that classification compresses brain-relevant information, while self-attention keeps spatial tokens accessible; CORnet-S’s anatomically labelled IT layer still has low correlation.The study presents the classification bottleneck and self-attention as candidate explanations for the deeper-layer divergence.Source: §1 · pp. 2–3
• The study also examines transformer token representations, arguing that token-preserving self-attention helps sustain brain-like representations at the deepest layers associated with inferotemporal cortex.Inferotemporal cortex is described as the brain’s semantic object-recognition stage.Source: §1 · pp. 2–3

2.1. Biological Inspiration and Modern Optimization Divergence

• Hierarchical V1 findings inspired CNNs, but classification-optimized networks can diverge from human perception, and higher ImageNet accuracy need not improve perceptual alignment.This gap motivates testing whether alternative architectures and training paradigms preserve brain-like visual representations.Source: §2.1 · p. 3

2.2. CNN-Brain Correspondence via Spatial Neuroimaging

• Primate and human fMRI studies linked CNN layers to visual hierarchy, but relied on few CNNs, invasive or single-model data, or best-layer summaries rather than broad layerwise comparisons.Brain-Score showed that ImageNet accuracy does not necessarily predict brain-likeness, but summarized each cortical region using its best-matching layer.Source: §2.2 · p. 4

2.3. CNN-Brain Correspondence via Temporal Neuroimaging

• Temporal studies largely tested CNNs alone; EEG captures millisecond visual dynamics, yet THINGS-EEG2 had not been used for unified layerwise comparisons across CNN and transformer architectures.The dataset provides responses from 10 participants across 16,540 training images and 200 test images.Source: §2.3 · p. 4

2.4. Transformer-Brain Comparisons

• Prior transformer comparisons used fMRI: Oota et al. compared architecture families, while Conwell et al. found matched CNNs and ViTs comparable and objective and visual diet more influential, but reported best-layer summaries.Because these studies lacked fine-grained temporal measurements, when and how the architectures diverge across processing stages remains unclear.Source: §2.4 · p. 5

3.1. Dataset

• The Gifford et al. (2022) dataset provides EEG from 10 participants viewing 16,540 training images and 200 test images, with 80 repetitions per test image.Analysis uses 17 posterior visual channels, epochs from −200 to 800 ms, and 100 time points; averaging repeated responses yields high signal-to-noise test responses.Source: §3.1 · p. 5

3.2. Choice of Vision Models

• The study tests eight models: six CNNs—AlexNet, VGG16, ResNet50, CORnet-S, MoCo-v1, and CLIP-RN50—and two vision transformers.The transformers are DINOv2-ViT-B/14 and CLIP-ViT-B/32.Source: §3.2 · p. 5

3.2.1. Supervised CNNs

• AlexNet, VGG16 (Simonyan and Zisserman, 2014), ResNet50 (He, Zhang, Ren and Sun, 2015), and CORnet-S (Kubilius, Schrimpf, Nayebi, Bear, Yamins and DiCarlo, 2018) use end-to-end ImageNet classification.Features were extracted at every named layer, from the first convolutional output through the final classification layer.Source: §3.2.1 · p. 5

3.2.2. Self-supervised CNN

• MoCo-v1 (He, Fan, Wu, Xie and Girshick, 2020) uses a ResNet-50 backbone trained with contrastive instance discrimination without class labels.Its layers were extracted in the same way as those of the supervised CNNs.Source: §3.2.2 · p. 5

3.2.3. Vision Transformers

• DINOv2-ViT-B/14 (Oquab et al., 2023) uses self-distillation without labels or language supervision, while CLIP-ViT-B/32 aligns images with natural-language captions.Features were extracted at the output of each of the 12 transformer blocks.Source: §3.2.3 · p. 5

3.2.4. Architecture and Objective Controlled Comparisons

• MoCo-v1 and CLIP-RN50 provide controlled comparisons that vary objective at fixed ResNet-50 architecture or architecture at fixed CLIP objective.All eight models are frozen feature extractors, with no fine-tuning.Source: §3.2.4 · p. 6

3.3.1. Kernel PCA Dimensionality Reduction

• Features are standardized, then reduced to 3,000 components with polynomial Kernel PCA; the degree-4 kernel is fit on training features and projects test features out of sample.The kernel uses γ=1/D and coefficient 1; fitting once on training data avoids recomputing the decomposition on test data.Source: §3.3.1 · p. 6
• 3,000 Kernel PCA components were sufficient: six models changed by at most ±0.003 in Pearson r from 2,000 to 3,000, while VGG-16 and CLIP-RN50 gained under 0.01.The component sweep covered all eight base models and used the full-time-window peak metric.Source: §3.3.1 · p. 6, Figure 2 · p. 7

3.3.2. Ordinary Least-Squares Encoding

• OLS weights are estimated separately for each subject, using training EEG responses averaged across available repetitions as targets.The EEG training targets are regressed on Kernel-PCA-projected features.Source: §3.3.2 · p. 6

3.3.3. Layer-wise Encoding

• Layer-wise encoding reduces each layer independently with Kernel PCA, fits a separate OLS model, and reports that layer’s encoding score.This produces the per-layer scores reported in Section 4.Source: §3.3.3 · p. 6

3.3.4. All Layer Concatenated Encoding

• All-layer concatenation yields the whole-model score: concatenate layer features, apply one Kernel PCA reduction and OLS model, then evaluate predicted EEG.This score is used for the overall model ranking in Table 2.Source: §3.3.4 · p. 7

3.4.1. Test set Prediction

• After estimating W, the model’s test-set EEG prediction is computed once from projected test features and held fixed for subsequent evaluation.Source: §3.4.1 · p. 8

3.4.2. Split half Correlation

• Following Gifford et al. (2022), test repetitions are split into halves, and correlations between predicted responses and half-averaged EEG are averaged over 100 random splits.Correlations are computed independently for each channel and timepoint rather than against one fixed average of the test repetitions.Source: §3.4.2 · p. 8

3.4.3. Temporal and Layer-wise Peak Encoding

• For each layer and subject, the channel-averaged correlation’s peak across the full −200 to 800 ms window is used, without restricting the search to a subwindow.This avoids favoring any processing stage through the choice of time range.Source: §3.4.3 · p. 8

3.4.4. Noise Ceiling

• The noise ceiling is bounded by correlating one half’s EEG average with the average across all test repetitions, with both quantities averaged over the splits.The bound accounts for trial-to-trial EEG variability limiting attainable model correlations.Source: §3.4.4 · p. 8

3.4.5. Subject level Aggregation

• Per-layer, per-subject peak correlations are averaged across subjects to produce one encoding score for each layer or model.Source: §3.4.5 · p. 8, §3.4.3 · p. 8

• The evaluation tracks encoding accuracy layer by layer to examine how brain correspondence changes across network depth.Source: §4 · p. 8
• It tests how transformer token aggregation strategies—CLS, patch, and combined representations—influence encoding performance.Source: §4 · p. 8
• It compares models at an aggregate level to assess correspondence with brain activity.Source: §4 · p. 8
• It analyzes encoding accuracy across EEG channels to identify where effects are strongest.Source: §4 · p. 8

4.1. Layer-wise Encoding Across CNN Models

• r=0.654 for AlexNet maxpool2, 0.653 for ResNet-50 block2, and 0.646 for MoCo-v1 block2; CORnet-S V2 was lowest at r=0.636.Across CNN architectures, peak layer-wise accuracy occurred within the 60–150 ms post-stimulus window.Source: Table 1 · p. 9, §4.1 · p. 9
• Deeper CNN representations generally showed weaker EEG correspondence than early layers, particularly after the initial visual response.The layer-by-time heatmaps show strongest correlations concentrated in early layers shortly after stimulus onset, while deepest layers remain weaker throughout the window.Source: §4.1 · p. 9, Figure 4 · p. 10
• The weakening toward deeper representations appeared across supervised CNNs, self-supervised MoCo-v1, CLIP-RN50, and CORnet-S, whose final IT representation was also weaker than its earlier layers.CORnet-S’s IT layer was explicitly designed to correspond to a high-level visual processing stage, yet it did not maintain the strong EEG correspondence of earlier layers.Source: §4.1 · pp. 9–10

4.2.1. Transformer Token Representation Analysis

• The four tested strategies were CLS alone, mean patch tokens, CLS concatenated with mean patches, and CLS concatenated with all individual patch tokens; only the last preserves unpooled spatial detail.The first three strategies compress the spatial patch sequence into a global or pooled representation.Source: §4.2.1 · p. 11

4.2.2. Temporal Encoding Across Token Representations

• Across both ViTs, CLS+all patches produces stronger encoding than the three pooled variants throughout the post-stimulus period, including later responses and early processing stages.The three pooled representations have broadly similar temporal profiles, while retaining individual patch tokens strengthens correspondence at early processing stages.Source: Figure 5 · p. 11, §4.2.2 · pp. 11, 14, Figure 6 · p. 12

4.2. Block-wise Encoding of Vision Transformers

• With default CLS, both ViTs peak at block_3 (r=0.448), below strongest CNN layers (r=0.636–0.654), yet retain strong EEG correspondence after 200ms in their deepest blocks.The block-by-time profiles show this depth-dependent pattern, with deepest-block correspondence persisting later in the time window.Source: §4.2 · pp. 10–11, Figure 7 · p. 13, Figure 8 · p. 13

4.2.3. Effect of Token Aggregation on Encoding Accuracy

• For DINOv2, pooled variants peak at approximately r=0.48–0.51 versus approximately r=0.66 for CLS+all patches; CLIP’s pooled variants stay near r=0.48 versus r=0.64.Across both models, the CLS+all-patches representation consistently outperforms the pooled alternatives.Source: §4.2.3 · p. 14, Figure 11 · p. 15

Layer-wise and token-representation findings

• CNN correspondence was strongest in early layers and declined with depth, whereas transformers maintained stronger correspondence in deeper blocks, particularly beyond approximately 200 ms.For both transformers, CLS + all patches concatenated produced the strongest correspondence; representations discarding individual spatial patch tokens showed substantially weaker encoding.Source: §4.3 · p. 16

Model-level performance

• r=0.677 ± 0.052 for AlexNet was the highest peak, followed by MoCo-v1 (0.673 ± 0.050), CORnet-S (0.672 ± 0.052), and ResNet-50 (0.670 ± 0.049).Other reported peaks were VGG-16 (r=0.667 ± 0.049), CLIP-RN50 (r=0.659 ± 0.043), DINOv2 with CLS + all patches (r=0.656 ± 0.051), and CLIP-ViT with CLS + all patches (r=0.640 ± 0.048). Combining representations across depths allowed early and intermediate encoding to contribute substantially to peak accuracy, masking deeper-layer differences.Source: §4.3 · pp. 16–17, Table 2 · p. 17, Figure 13 · p. 17

Architecture and training-objective controls

• MoCo-v1 (r=0.673 ± 0.050) and ResNet-50 (r=0.670 ± 0.049), which share a ResNet-50 backbone but differ in training objective, had highly similar temporal profiles and peak accuracy.This comparison suggests that switching from supervised classification to MoCo-v1’s contrastive objective did not substantially alter encoding performance across the complete model representations.Source: §4.3 · pp. 17–18
• r=0.476 ± 0.049 for standard CLIP-ViT was below CLIP-RN50’s r=0.659 ± 0.043, but retaining all patch tokens raised CLIP-ViT to r=0.640 ± 0.048.With patch tokens retained, CLIP-ViT approached CLIP-RN50’s peak while preserving a stronger late temporal profile; the models share a contrastive language-image objective but differ in architecture.Source: §4.3 · p. 18

4.4. Spatial Topography of Encoding Accuracy Across Model Categories

• CNNs peaked sharply at occipital channels, with onset at 60–70 ms; this topography was similar across classification-trained and contrastively trained models.CLIP-RN50 reproduced the classification-trained CNN pattern despite its non-classification objective, supporting the reported architecture-specific spatial signature.Source: §4.4 · p. 18, Figure 14 · p. 18
• For both transformers, CLS+all-patches concatenation sustained correlation to roughly 400–500 ms in parieto-occipital channels, unlike weaker, shorter-lived pooled variants.All four token variants retained the same occipital-dominant topography; the difference was response strength and duration.Source: §4.4 · p. 19, Figure 15 · p. 19
• Across all tested models and variants, encoding was occipital-dominant; architecture, objective, and token choice affected signal strength and duration rather than spatial location.The common pattern peaked over occipital channels, while the study’s reported differences concerned temporal extent and encoding strength within that shared map.Source: Figure 14 · p. 18, §4.4 · p. 19, Figure 15 · p. 19

4.5. Discussion

• CNN correspondence is strongest in early layers and weakens with depth, while transformer blocks sustain stronger late correspondence; neither architecture uniformly outperforms the other.Both are about equally aligned with the earliest visual response, but their correspondence diverges as processing deepens and extends beyond the first feedforward pass.Source: §4.5 · pp. 19–20
• With patch tokens retained, peak correlations reached r=0.640 for CLIP-ViT and r=0.656 for DINOv2, versus approximately r=0.48 to 0.51 with pooled or CLS representations.Preserving patch tokens also maintained a more robust, prolonged response than pooling.Source: §4.5 · pp. 19–20
• MoCo-v1 and ResNet-50 peaked at r=0.673 and r=0.670, while default CLIP-ViT-B/32 reached r=0.476 versus CLIP-RN50 at r=0.659.These controlled comparisons indicate that changing the objective alone had little effect for a fixed CNN architecture, while CLIP-ViT’s peak rose to r=0.640 with patch tokens retained and its stronger late temporal profile persisted.Source: §4.5 · p. 20
• All model and token variants shared an occipital-dominant spatial pattern, while patch-preserving transformers sustained responses for approximately 400 to 500 ms.Thus, model differences reflected signal strength and duration within a common spatial map; pooled representations and deeper CNN layers showed weaker, shorter-lasting responses.Source: §4.5 · p. 20

5. Conclusion

• Transformers retain stronger brain correspondence than CNNs in deeper layers during the later post-stimulus response, despite similar early-layer alignment.This later response phase is linked to higher-level visual areas such as V4 and inferotemporal cortex.Source: §5 · p. 20
• Keeping all transformer patch tokens considerably improves encoding accuracy, yielding stronger peak correlations and a longer-lasting response.Source: §5 · p. 20
• Conventional CNN classification bottlenecks may reduce brain-predictive information at depth as representations compress toward fixed categorical outputs.Source: §5 · p. 20
• Both CNNs and transformers, potentially including hybrids, may offer better representations throughout the visual hierarchy without privileging one architecture.The findings motivate vision models inspired by and aligned with the brain.Source: §5 · p. 20