PaperlayerView exact revision on arXiv

Source-linked AI summary

Zero-shot World Models Are Developmentally Efficient Learners

Khai Loong Aw, Klemen Kotar, Wanhee Lee, Seungwoo Kim, Khaled Jedoui, Rahul Venkatesh, Lilian Naing Chen, Michael C. Frank, Daniel L. K. Yamins

arXiv:2604.10333v1cs.AIcs.CV

TL;DR

Young children develop broad physical understanding from very limited experience, a challenge for visual-learning systems that remain data-inefficient. The paper proposes a zero-shot visual world model learned from one child’s first-person experience and reports competence across physical-understanding benchmarks, alongside developmental behavioral signatures and brain-like representations.

  • Problem

    Visual-learning algorithms remain data-inefficient on children’s real-world visual streams, and task-specific labeled readouts limit the range of feasible downstream tasks.

    Source: Discussion · p. 11

  • Method

    ZWM learns a sparse temporally factored visual predictor and uses approximate causal inference through minimal perturbations to extract visual-cognitive quantities zero-shot.

    Source: The ZWM framework · pp. 3–4

  • Results

    ZWM yields zero-shot optical flow, depth, segmentation, and physical reasoning, demonstrating diverse visual-cognitive capacities from a child’s naturalistic video experience.

    Source: ZWM performs diverse visual-cognitive… · pp. 6–7, Discussion · p. 11

  • Takeaways & Limitations

    The findings provide proof-of-concept for a hybrid innateness account in which structural priors support capacities and representations learned from naturalistic visual experience.

    Source: Discussion · p. 12

    The study leaves the developmental origins of semantic concepts, such as named object categories and relationships, unaddressed.

    Source: Discussion · p. 12

Abstract

from arXiv · show

Young children demonstrate early abilities to understand their physical world, estimating depth, motion, object coherence, interactions, and many other aspects of physical scene understanding. Children are both data-efficient and flexible cognitive systems, creating competence despite extremely limited training data, while generalizing to myriad untrained tasks -- a major challenge even for today's best AI systems. Here we introduce a novel computational hypothesis for these abilities, the Zero-shot Visual World Model (ZWM). ZWM is based on three principles: a sparse temporally-factored predictor that decouples appearance from dynamics; zero-shot estimation through approximate causal inference; and composition of inferences to build more complex abilities. We show that ZWM can be learned from the first-person experience of a single child, rapidly generating competence across multiple physical understanding benchmarks. It also broadly recapitulates behavioral signatures of child development and builds brain-like internal representations. Our work presents a blueprint for efficient and flexible learning from human-scale data, advancing both a computational account for children's early physical understanding and a path toward data-efficient AI systems.

The ZWM framework

ZWM combines sparse temporally factored prediction, zero-shot extraction through approximate causal inference, and composition of simple prompts to build increasingly complex visual inferences. Together, these components form a data-driven world model that forecasts scene changes under proxy actions.

Sparse temporally-factored prediction

  • A sparse temporally factored predictor learns to separate object appearance from object and camera motion using frame pairs and sparse, temporally biased masks.It predicts the second frame from a dense first frame plus at most 10% of the second frame’s patches, minimizing average L2 loss. The generic masking constraint leads the predictor to encode low-dimensional motion in compact, interpretable visual tokens.Source: The ZWM framework · pp. 3–4

Zero-shot extraction via approximate causal inference

  • ZWM extracts visual quantities by minimally perturbing an input, comparing the predictor’s outputs before and after, and aggregating the differences as approximate causal inference.For object segmentation, a hypothetical motion applied to one object patch propagates across that object; thresholding the resulting optical-flow difference identifies its pixels.Source: The ZWM framework · p. 4

Compositional prompting

  • ZWM composes prompts to extract and integrate increasingly abstract visual structures, progressing from optical flow and relative depth to object segmentation and intuitive physics.This sequence builds a computational graph of visual intermediates rather than operating only on RGB pixels.Source: The ZWM framework · p. 4

Model implementation

  • The predictor is a ViT-based neural network trained by stochastic gradient descent on RGB frame pairs from real-world videos.Models have 170 million or 1 billion parameters; frame gaps are sampled uniformly from 150ms to 450ms, inputs are 256x256 pixels patchified into 8x8 patches, and 10% of second-frame patches are revealed.Source: The ZWM framework · pp. 4–5

Results

Across zero-shot visual-cognitive tasks, BabyZWM performs competitively with specialized supervised systems while retaining most performance with single-child data. Its training trajectories broadly parallel reported developmental patterns, and its internal representations align hierarchically with neural responses.

ZWM performs diverse visual-cognitive tasks zero-shot

  • BabyZWM is competitive with supervised flow systems on TAP-Vid-DAVIS, and both ZWM and BabyZWM exceed 90% UniQA-3D depth accuracy, trailing only supervised binocular depth.BabyZWM is also strong but slightly below supervised models on TAP-Vid-Kubric; the depth models surpass large vision–language models and are comparable to supervised and self-supervised monocular estimators.Source: Figure 2 · p. 6, ZWM performs diverse visual-cognitive… · pp. 6–7
  • On SpelkeBench, BabyZWM rivals supervised Mask2Former variants but trails SAM2; ZWM, BabyZWM, and V-JEPA2 approach 100% across physical-reasoning categories.BabyZWM attention heads consistently follow the hand, the causal agent, when predicting the target object's motion.Source: Figure 3 · p. 7, ZWM performs diverse visual-cognitive… · p. 7, Figure 4 · p. 8

ZWM achieves data efficiency and continual learning

  • BabyZWM retains most performance versus models trained on Kinetics-400 and BVD, while single-child and age-ordered single-child versions perform similarly across most or all tasks.The single-child model used 132 hours from one individual; age-ordered variants used a single pass through the data with clips ordered by the child's age.Source: ZWM achieves data efficiency and… · pp. 7, 9
  • Symmetric masking variants perform substantially worse than standard asymmetric masking, indicating that emphasizing motion information benefits data efficiency and zero-shot abstraction.Standard BabyZWM fully reveals the first frame and masks 90% of the second frame; the tested symmetric variants mask 45%-45% or 90%-90%.Source: ZWM achieves data efficiency and… · p. 9

BabyZWM’s developmental curves broadly parallel children’s learning

  • BabyZWM's optical-flow accuracy rises and plateaus, relative-depth performance rises steeply and stays high, and segmentation and intuitive-physics capabilities continue improving during training.These trajectories broadly parallel reported child development, but benchmark-specific task difficulty, metrics, and ceiling effects limit direct comparisons and motivate more systematic benchmarking.Source: BabyZWM’s developmental curves broadly… · pp. 9–10

ZWM representations align with neural responses

  • BabyZWM's neural predictivity develops early for visual cortex and more gradually for higher regions, while model layers align hierarchically with cortical areas.This pattern appears across human fMRI and macaque electrophysiology datasets and is consistent with hierarchical visual organization.Source: Figure 6 · p. 10, ZWM representations align with neural… · p. 11

Discussion

The discussion presents ZWM as a data-efficient, zero-shot alternative to task-specific visual readouts and as a hybrid account of learned representations and structural priors. It also identifies unresolved questions about readouts, semantic learning, behavioral validation, and long-horizon prediction.

Discussion

  • ZWM replaces task-specific labeled readouts with a universal zero-shot predictor, extracting flow, depth, segmentation, and physical reasoning from hundreds of hours of one child’s video.The discussion argues this data-driven approach could matter in areas such as robotics, medical imaging, and embodied AI where large-scale labeled data is unavailable.Source: Discussion · pp. 11–12
  • ZWM combines minimal learned-network bias with structural priors, proposing that architecture, learning algorithm, and task-specific readout programs may be innate while representations are learned.The authors frame this as proof-of-concept evidence that naturalistic visual experience can support visual-cognitive capacities and object- and physics-like representations, challenging strong nativist accounts.Source: Discussion · p. 12
  • The discussion leaves open whether zero-shot readouts are hard-wired neural circuits, developmentally learned adapters, or online query-like inference routines.Source: Discussion · p. 12
  • ZWM treats cheap data-patch operations as proxies for actions in a data-driven world model, enabling hypotheticals from learned dynamics; future work could learn interactive action policies.The discussion suggests comparing such policies with children’s hand and head motions in BabyView.Source: Discussion · p. 12
  • The work leaves semantic concept development and detailed behavioral and neural comparisons unaddressed, while deterministic regression can produce blurry predictions under uncertainty and limit long-horizon prediction and control.Future work could integrate children’s linguistic and auditory experience and extend training to multiple frames with richer temporal memory and long-horizon tasks.Source: Discussion · pp. 12–13
  • Future work could integrate zero-shot task extractions into the predictor as intermediate targets, creating a bootstrapping cycle for learning richer predictors and more sophisticated visual abstractions.The proposed cycle would let each additional intermediate enrich the predictor and support increasingly efficient learning.Source: Discussion · p. 13

Methods

The methods describe how ZWM predicts masked future-frame pixels from temporally asymmetric inputs and uses perturbation-based prompts to extract physical and visual quantities. They also evaluate models trained on varied video datasets across visual-cognitive and neural benchmarks, comparing neural predictivity against representation-based and task-specific models and using custom zero-shot probes for visual-cognitive tasks.

Model architecture

  • Symmetric masking performed substantially worse across zero-shot tasks than ZWM’s temporally asymmetric masking, which reveals the first frame and 10% of the second-frame patches.The study also evaluates ZWM-170M (∼170 million parameters) and ZWM-1B (∼1 billion parameters).Source: Model architecture · p. 33

Training procedure

  • Training uses raw RGB frame pairs sampled from real-world video with uniformly random 150–450ms gaps (5–14 frames at 30 fps) and no data augmentation; optimization uses AdamW.The learning rate peaks at 3e-4 and follows cosine decay with 2000 warmup steps; gradient clipping uses a maximum norm of 1.0.Source: Training procedure · p. 34

Training datasets

  • The training comparison includes BabyView (868 hours from 34 children, plus ∼100 preschool hours), single-child and random 132-hour subsets, Kinetics-400 (∼670 hours), and BVD (∼7,000 hours).The 132-hour subsets comprise recordings from one child and a random sample across all 34 children. Kinetics-400 consists of 10-second clips across 400 action categories, while BVD combines computer-vision datasets and Internet videos.Source: Training datasets · pp. 34–36

Zero-shot prompt design

  • Zero-shot prompts perturb a latent visual cause, compare predictions with an unperturbed prediction, and aggregate differences; prompts can compose into more complex inferences.Optical flow traces a point’s displacement, and relative depth is inferred from binocular disparity by composing optical-flow prompts.Source: Zero-shot prompt design · pp. 36–37
  • Hypothetical motion displaces object patches to generate a scene, enabling segmentation through coherent flow; physics prompts reveal hand motion and compare predicted consequences with the target.Segmentation aggregates masks across eight displacement directions. Physics evaluations also compose flow and segmentation prompts to assess what moved and whether force transferred between objects.Source: Zero-shot prompt design · pp. 37–38

Evaluation benchmarks

  • Evaluation covers point tracking, relative depth, class-agnostic segmentation, intuitive physics, developmental trajectories, and neural predictivity across fMRI and macaque spiking data.The physics benchmark tests five interaction categories using 100 image pairs, eight mask configurations per pair, and MSE and LPIPS comparisons; neural predictivity uses cross-validated ridge regression and noise-corrected Pearson correlations.Source: Evaluation benchmarks · pp. 38–39
  • Both NSD and TVSD use 10-fold cross-validation and target-specific RidgeCV regularization across 21 α values; noise ceilings and evaluated regions differ by dataset.NSD uses Allen et al. reliability estimates and evaluates V1, V2, V4, and anterior ventral visual regions; TVSD uses split-half correlations and evaluates V1, V4, and IT cortex.Source: Evaluation benchmarks · pp. 39–40

Baselines

  • Representation-based comparators were ImageNet-supervised ResNet50 [109] and BabyView-trained DINOv3 [47] and V-JEPA2.DINOv3 and V-JEPA2 were trained on 868 hours of BabyView data; V-JEPA2 had 300 million parameters.Source: Baselines · p. 40
  • For representation models, optical flow and binocular depth use cosine-similarity patch correspondence, while segmentation groups patches similar to a seed.For depth, relative depth is inferred from the magnitude of the correspondence disparity.Source: Baselines · p. 40
  • For V-JEPA2 intuitive physics, revealed hand and background patches condition masked-patch predictions, which score correct when closer to target-frame than context-frame features.ResNet50 and DINOv3 are excluded from this temporal benchmark because they are single-image models.Source: Baselines · pp. 40–41
  • Task-specific comparisons use supervised flow, monocular and binocular depth, and segmentation systems; the novel intuitive-physics benchmark has no established baseline.Flow baselines are CoTracker3 [50], DPFlow [51], and SeaRAFT [52]; depth baselines include MiDaS-CNN [57], MonoDepth2 [58], FoundationStereo [59], and Gemini-1.5 [54], GPT-4-Turbo [55], and GPT-4o [56]; segmentation baselines are Mask2Former [61] and SAM2 [63].Source: Baselines · p. 41

Supplementary Text

Supplementary analyses find hand-directed attention in deeper BabyZWM layers during intuitive-physics prediction, alongside hierarchical alignment between model layers and visual-cortex regions across human fMRI and macaque electrophysiology benchmarks.

Attention head analysis for intuitive physics

  • The attention analysis compares moved-object queries with hand, background-grounding, and random patches, averaging profiles across categories and eight random seeds.Background attention is normalized by the ratio of background to hand patches to compare groups of different sizes.Source: Attention head analysis for intuitive… · pp. 42–43
  • In deeper transformer layers, moved-object queries disproportionately attend to hand patches across intuitive-physics categories, consistent with—but not proof of—causal attention.Attention weights are indirect evidence; further mechanistic work is needed to establish whether these heads causally contribute to physical predictions.Source: Figure S1 · p. 42, Attention head analysis for intuitive… · p. 43

Neural predictivity results

  • BabyZWM’s earlier layers best predict earlier cortical regions and deeper layers higher visual areas across NSD human fMRI and TVSD macaque electrophysiology, with alignment comparable to larger-data ZWM variants.TVSD supports the hierarchy at single-neuron resolution; Baby V-JEPA2 shows lower neural alignment than its larger-data counterpart.Source: Neural predictivity results · p. 44, Figure S2 · p. 44

Related papers

Loading 2604.10333v1…