Primate vision reveals a missing principle for robust dynamic AI

Introduction

• Temporal integration can exploit appearance cues without yielding robust motion representations, so strong video-benchmark performance alone does not establish dynamic understanding.The introduction notes that fMRI’s comparatively slow hemodynamic response also limits access to rapid temporal transformations in neural processing.Source: Introduction · p. 3
• The study asks whether motion information in primate IT becomes accessible across appearance changes, and whether predictive learning can capture this transformation.Predictive video world models use temporal structure to predict missing or future scene states, potentially encouraging representations of persistent spatiotemporal structure across changes in instantaneous appearance [17].Source: Introduction · pp. 3–4
• The study tests humans, macaque IT, and image- and video-based models on identity, motion direction, and speed, using appearance disruption that preserves object motion.It extends the comparison to segmentation systems, explicit optic-flow models, and predictive video world models, and examines IT population activity as well as individual neural and model-unit dynamics.Source: Introduction · p. 4
• Video-recognition models benefit from temporal integration but rarely transfer motion across appearance changes, unlike humans and macaque IT; predictive world models best matched IT but still missed its temporal transformation.Predictive world models provided the strongest correspondence with IT among tested video-modeling approaches, but no model reproduced IT’s progressive shift from appearance-dominated responses toward motion coding that generalizes across appearance changes.Source: Introduction · p. 4
• IT representations progressively shift from appearance-dominated responses toward motion information that is increasingly accessible across changes in appearance.The study identifies this progressive integration of appearance and motion into appearance-robust object representations as a computational target for dynamic AI.Source: Introduction · p. 4

Humans accurately discriminate moving objects

• Humans discriminated object identity, motion direction, and speed with mean accuracies of 0.94 ± 0.02 SD, 0.91 ± 0.07 SD, and 0.82 ± 0.06 SD, respectively.Participants made these judgments from 300 ms videos.Source: Humans accurately discriminate moving… · p. 7

Temporal information improves dynamic object representations in video models

• Video-recognition ANNs decoded identity, motion direction, and speed more accurately than image-based ANNs, and temporal-input ablation reduced decodability for all three.The video-versus-image mean accuracy differences were 0.07 for identity, 0.07 for direction, and 0.04 for speed; removing temporal variation reduced mean accuracy by 0.11, 0.10, and 0.06, respectively.Source: Temporal information improves dynamic… · p. 8

Video recognition models encode motion but show limited generalization beyond appearance

• Human motion-direction judgments remained high when appearance was removed, approximately matching performance on appearance-based stimuli.Appearance-free videos preserved pixel-wise motion from naturalistic clips while replacing their appearance with random patterns.Source: Video recognition models encode motion… · p. 10
• Conventional video-recognition ANNs generalized poorly across appearance: their motion-direction decoding fell far below human performance, while image-based ANNs fell to chance.Decoders were trained on naturalistic features and tested without refitting on appearance-free features.Source: Video recognition models encode motion… · p. 10, Figure 3 · p. 11

Explicit motion processing and predictive world modeling improve cross-appearance generalization

• 0.85 ± 0.04 SD: V-JEPA2 world models decoded motion direction from appearance-free videos, versus 0.79 ± 0.01 SD for optic-flow models and 0.28 ± 0.03 SD for SAM2.On naturalistic videos, the corresponding accuracies were 0.92 ± 0.02 SD, 0.91 ± 0.01 SD, and 0.88 ± 0.03 SD, respectively.Source: Explicit motion processing and… · pp. 12–13
• Cross-appearance motion generalization was not a generic consequence of temporal processing: explicit motion inputs and predictive video learning both supported more robust representations than conventional video-recognition objectives.The optic-flow models combined inter-frame displacement signals with RGB appearance, whereas V-JEPA2 models learned by predicting masked or future visual content.Source: Explicit motion processing and… · pp. 11–12

Macaque IT contains motion information that generalizes across appearance disruption

• A motion decoder trained on naturalistic IT responses decoded appearance-free responses above chance (0.58 ± 0.02 SE), statistically similar to held-out naturalistic responses (0.61 ± 0.02 SE; p = 0.27).Source: Macaque IT contains motion information… · p. 13

Temporal processing improves late IT alignment, with predictive world models performing best

• Early-IT CKA was similar for spatial and spatiotemporal ANNs (0.35 ± 0.06 SD vs 0.34 ± 0.06 SD), while late-IT CKA favored spatiotemporal features (0.30 ± 0.05 SD vs 0.28 ± 0.04 SD).Source: Temporal processing improves late IT… · p. 14, Figure 5 · p. 15
• Late-IT CKA was 0.36 ± 0.0 SD for spatiotemporal predictive world models versus 0.30 ± 0.02 SD for spatial versions, the highest scores among tested models.Source: Temporal processing improves late IT… · p. 14, Figure 5 · p. 15

Current video models miss the temporal transformation of IT representations

• IT temporal change fell from 0.27 ± 0.006 SE near 120 ms to 0.11 ± 0.007 SE near 330 ms (Δ median early-late = 0.16, p < 0.001), indicating greater late stability.Source: Current video models miss the temporal… · p. 16

IT progressively develops appearance-robust motion representations that remain weak in video models

• Initially, IT and spatial video ANNs showed similar appearance-over-motion differences (0.85 ± 0.06 MAD vs 0.86 ± 0.06 SD; p = 0.15), but later IT incorporated appearance-invariant motion while ANN appearance dominance persisted (video-ANN Δ = 0.70 ± 0.16 SD).At later frames, spatiotemporal ANNs reduced the appearance–motion gap relative to spatial counterparts (p < 0.001), but this remained insufficient to match IT.Source: IT progressively develops… · pp. 19–20, Figure 8 · p. 20
• IT progressively incorporates motion information that generalizes across appearance, whereas current video models remain appearance-dominated; predictive world models most closely approach this computation.Source: IT progressively develops… · p. 20

Discussion

• Humans remained accurate when appearance was removed but motion preserved, whereas early video-recognition models dropped substantially and tracking models lacked comparable transfer.Naturalistic motion decoding alone does not establish an appearance-robust motion representation, because natural videos correlate object appearance and movement.Source: Discussion · p. 21
• Predictive world models paired strong cross-appearance generalization with the highest IT correspondence among tested artificial systems, whereas optic-flow models generalized behaviorally but aligned less with late IT.Optic-flow and predictive world-modeling ANNs approached human-level generalization, but optic-flow models showed relatively limited correspondence with late IT representations.Source: Discussion · pp. 21–22
• Macaque IT represented motion direction and supported decoding across appearance manipulations, indicating that high-level object representations encode how objects move as well as what they are.Biological vision integrates motion with object information while becoming progressively less dependent on the appearance through which motion was observed.Source: Discussion · p. 22
• Video ANNs corresponded better to late IT activity than spatial models, but the improvement was modest and even the strongest-aligned models left substantial representational differences.Video world-modeling ANNs showed the strongest IT alignment among the compared models.Source: Discussion · p. 22
• IT responses evolved from rapidly changing, appearance-dominated early activity toward more stable motion coding, but video ANNs showed weaker reorganization and remained more appearance-dominated.Spatiotemporal architectures were more dynamic than frame-based counterparts, yet did not reproduce IT’s progressive transformation toward appearance-generalizing motion responses.Source: Discussion · p. 22
• Robust dynamic AI should progressively add appearance-generalizing motion while retaining identity-relevant appearance; predictive modeling is promising, but scale, data, and implementation differences preclude attributing its advantage to prediction alone.The remaining gap with IT motivates controlled tests of predictive objectives as a route toward more biologically faithful representations.Source: Discussion · pp. 22–23
• Naturalistic benchmarks can obscure whether models exploit correlated visual cues, so controlled manipulations that preserve one factor while disrupting another should complement behavioral evaluation with neural measurements.This helps distinguish models that reproduce behavior from those that approach biological representational strategies.Source: Discussion · p. 23
• The study tested identity, motion direction, and speed in controlled videos; future work should examine other dynamics and how IT interacts with MT, parietal cortex, and motion-sensitive pathways.The analysis covered only IT, one component of the broader dynamic-vision system.Source: Discussion · p. 23

• Human experiments were approved by York University’s Ethics Review Committee (Human Participant Review Subcommittee).Source: Ethics Approval · p. 24
• Participants were recruited through Amazon Mechanical Turk, consented on the first page before proceeding, and received $15 CAD/hour.Source: Ethics Approval · p. 24
• Nonhuman-primate data collection and procedures followed NIH, MIT, and Canadian Council on Animal Care guidelines and were approved by York University’s Animal Care Committee.Source: Ethics Approval · p. 24

Human subjects

• 107 participants recruited through MTurk completed the online experiments without prior training, for $15 CAD per hour, under York University ethics approval.Procedures followed institutional guidelines.Source: Human subjects · p. 24
• Participants with accuracy below 0.6 in the first 30 trials of each session were excluded; chance performance was 0.5.Source: Human subjects · p. 24
• Video-level behavioral metric reliability increased with additional repetitions, reaching approximately 0.8 and approaching asymptote after about 60 repetitions.Source: Human subjects · p. 24

Non-human primates

• Five adult male rhesus monkeys contributed inferior temporal cortex recordings, with site counts varying across the figure analyses.M1 and M2 each contributed 61 sites for Figs. 3, 4, and 8; M3 contributed 192 sites, and M4 and M5 each contributed 288 sites for Figs. 5–7. Procedures followed NIH and animal-care guidelines and were approved by the York University Animal Care Committee.Source: Non-human primates · p. 24

Visual Stimuli

• Videos showed 10 object categories across identity-preserving changes in position, rotation, and size, rendered as 18 achromatic 256 × 256 frames over 300 ms at 60 Hz.Objects were rendered against random backgrounds. The viewing parameters were position (x and y), rotation (x, y, and z), and size.Source: Visual Stimuli · p. 25
• Behavioral motion stimuli translated objects over stationary backgrounds at selected speeds and in eight directions.Speed was derived from pixel displacement per second for 256 × 256 frames spanning 8° of visual angle; directions included the four cardinal and four diagonal directions.Source: Visual Stimuli · p. 25

Generation of appearance-free videos

• Appearance-free videos were created by advecting random pixel noise with optic flow estimated from the original video, then resampling warped frames by nearest-neighbor interpolation.Resampling realigned subpixel flow displacements to the discrete pixel lattice.Source: Generation of appearance-free videos · p. 25

Generation of coherent-incoherent motion videos

• Incoherent-motion videos randomly permuted the coherent videos’ 18 frames, preserving their frames and visual content while disrupting smooth trajectories and consistent motion direction and speed.Duration and frame rate remained unchanged.Source: Generation of coherent-incoherent… · p. 25

Human behavioral tasks

• On Mechanical Turk, observers completed two-choice object-category, motion-direction, and speed discriminations after 100-ms fixation and 300-ms video presentation.Responses were allowed up to 1500 ms; although stimulus sets contained ten categories and eight directions, each trial was binary, making chance performance 0.5. For speed, participants chose the faster of two videos.Source: Human behavioral tasks · p. 26

Human behavioral metric

• Video-level I1 scores averaged accuracy across ground-truth-versus-distractor pairings and videos, with 0.5 denoting chance performance.The metric adapts I14 to the two-alternative forced-choice responses and accounts for differences in distractor difficulty.Source: Human behavioral metric · p. 26

Surgical Procedures

• Rhesus macaques received bilateral Utah arrays in IT, each comprising 96 electrodes positioned using visible sulcal landmarks.Array locations were anterior to the posterior middle temporal sulcus and inferior to the superior temporal sulcus.Source: Surgical Procedures · p. 26

Eye Tracking

• Eye tracking monitored fixation during video trials, which were aborted if gaze deviated more than ±2° from the central fixation point.During passive viewing, videos lasted 300 ms and were preceded by 100 ms of fixation; trials included a 100 ms gray blank, reward, and 500 ms inter-trial interval.Source: Eye Tracking · p. 27

Neural Recordings

• IT activity was continuously sampled at 20 kHz, with all sites used for population analyses and reliable, visually responsive sites selected for neural-alignment analyses.Sites were treated as random samples from pooled IT; reliability was estimated using 100 split-half resamples with Spearman–Brown correction. The reliability threshold was greater than 0.4, primarily to reduce computational load.Source: Neural Recordings · p. 27

Prediction of behavior from neural data

• IT motion-direction decoding was tested for transfer from naturalistic to paired appearance-free videos using cross-validated LDA at each 30-ms bin.Paired versions stayed in the same fold; the naturalistic decoder was applied without refitting, and the peak naturalistic time bin selected the appearance-free score. Performance used video-averaged I1, with variability reported as standard error.Source: Prediction of behavior from neural data · pp. 27–28

Temporal change of neurons

• Temporal stability was measured from correlations between each IT neuron's stimulus-wise response vectors at time offsets of 30 and 60 ms, corrected for response reliability.Reliability was estimated with split-half correlations and Spearman–Brown correction; population analyses used median aggregation, response windows up to 390 ms, and sites with reliability above 0.4.Source: Temporal change of neurons · p. 28

Selectivity of neurons to appearance and motion factors

• Neuron-level appearance and motion sensitivity were quantified by noise-corrected correlations across coherent naturalistic, incoherent-motion, and appearance-free video conditions.Spearman correlations compared coherent with incoherent videos for appearance sensitivity and coherent with appearance-free videos for motion sensitivity; population trajectories used median corrected correlations and variability used median absolute deviation.Source: Selectivity of neurons to appearance… · p. 29

Model Types

• The study compared 15 video-based methods spanning computer-vision tasks with 14 image-based architectures benchmarked on Brain-Score.The video models included recognition, segmentation, optic-flow, and predictive world-modeling approaches.Source: Model Types · p. 29

Model Training

• All ANNs were publicly available pretrained models used off the shelf, without additional training or fine-tuning.For initialization-based video models such as SAM2, the first-frame object bounding box was manually annotated and supplied as a prompt.Source: Model Training · p. 30

Model Feature Extraction

• Image-based models processed video frames independently, including recurrent models treated as feedforward, with features taken from their reported IT-like layers.Source: Model Feature Extraction · p. 30
• For video models with temporal buffers, features were extracted frame by frame using the current and preceding frames, repeating the first frame when history was unavailable.The IT-like layer was selected using the layer with highest neural predictivity on HVM640 images.Source: Model Feature Extraction · p. 30
• Spatial-only variants removed temporal information by repeating frames in buffers, eliminating optic flow, or running SAM2 independently on each frame.Optic-flow fields for flow-based models were computed with RAFT.Source: Model Feature Extraction · p. 30

Prediction of behavior from model features

• Per-frame LDA decoded motion direction and object identity, while PLS predicted speed; motion transfer tested naturalistic-trained decoders on paired appearance-free videos without refitting.Paired naturalistic and appearance-free videos were assigned to the same cross-validation fold.Source: Prediction of behavior from model… · pp. 30–31

Model–neural representational similarity

• Model–IT similarity was quantified with noise-normalized linear CKA, comparing ANN features with early and late IT responses to the same naturalistic videos.ANN features were averaged over the first and last five video frames; IT responses covered 90–180 ms and 300–390 ms after onset.Source: Model–neural representational similarity · p. 31

Temporal change of ANN representations

• Temporal analyses measured ANN response correlations across frame offsets and compared early with late activations; appearance- and motion-factor selectivity was also assessed without reliability correction.For correlations, early and late responses were defined at video frames 1 and 14; for factor selectivity, they were defined at the second and second-to-last frames.Source: Temporal change of ANN representations · p. 32

Statistical Analyses

• Continuous-variable relationships were assessed with Spearman rank or Pearson correlations unless otherwise specified.Analyses were performed in Python using scipy.stats.Source: Statistical Analyses · p. 32
• Group tests were selected by Shapiro–Wilk normality checks and pairing: paired or independent-samples t-tests for normal data, and Wilcoxon rank-sum tests for unpaired comparisons when normality failed.All tests were two-tailed, with exact p-values and test statistics reported.Source: Statistical Analyses · p. 33
• Distinct hypotheses were analyzed within separate statistical families without global multiple-comparison correction; nonsignificant trends were not used for inferential conclusions.Exact p-values and test statistics were reported.Source: Statistical Analyses · p. 33

Temporal integration and decoding

• Across the 15 video models, spatiotemporal architectures outperform spatial models in decoding object identity, motion direction, and speed.The reported mean accuracies are 0.72 ± 0.07 versus 0.63 ± 0.07 for identity, 0.71 ± 0.11 versus 0.56 ± 0.03 for direction, and 0.70 ± 0.06 versus 0.62 ± 0.03 for speed (spatiotemporal versus spatial).Source: Figure S1 · p. 41

IT cross-appearance decoding

• IT population motion-direction decoding generalizes from naturalistic to appearance-free videos, with accuracy increasing as more neural sites are included.The decoder was trained on naturalistic-video responses and tested on held-out naturalistic and paired appearance-free responses.Source: Figure S2 · p. 41, Supplementary Figures · p. 42

World-model and optic-flow comparisons

• Video world models maintain high motion-direction decoding on appearance-free videos and show the strongest correspondence with late IT motion responses, unlike optic-flow models with weak neural alignment.Optic-flow models generalize behaviorally despite their weaker alignment with IT dynamics.Source: Figure S3 · p. 42, Supplementary Figures · p. 43

Appearance and motion factors

• IT shifts from early appearance dominance toward later motion coding, but image- and video-based ANNs retain appearance-biased representations over time.In the late period, the appearance–motion gap is 0.34 for IT, compared with 0.86 for image-based ANNs.Source: Figure S4 · p. 43, Supplementary Figures · pp. 43–44

Model specifications

• The study compares image-based recognition ANNs and video-based models spanning action recognition and segmentation, using designated IT-like layers for neural analyses.Architectures include 2D and 3D convolutions, recurrent and transformer networks, and state-space models, trained on ImageNet or specialized video datasets.Source: Figure S5 · p. 44