Source-linked AI summary
Zero-shot World Models Are Developmentally Efficient Learners
Khai Loong Aw, Klemen Kotar, Wanhee Lee, Seungwoo Kim, Khaled Jedoui, Rahul Venkatesh, Lilian Naing Chen, Michael C. Frank, Daniel L. K. Yamins
TL;DR
The paper addresses how children acquire broad physical understanding from limited first-person experience without task-specific examples. It proposes ZWM, which combines temporally factored prediction, approximate causal inference, and compositional inference, and reports broad zero-shot visual cognition, brain-like representations, and efficient learning from naturalistic data. The paper limits its scope to physically grounded infant-learned quantities and leaves semantic concept development unresolved.
Problem
The paper asks how data-efficient and flexible visual-cognitive abilities can arise from limited first-person experience without task-specific examples.
Method
ZWM uses a sparse temporally factored predictor, approximate causal inference through minimal perturbations, and compositional prompting to extract visual-cognitive quantities zero-shot.
Results
ZWM performs diverse visual-cognitive tasks zero-shot, including optical flow, depth, segmentation, and physical reasoning, while showing brain-like hierarchical neural alignment.
Takeaways & Limitations
The results provide proof-of-concept support for acquiring visual-cognitive capacities and object- and physics-like representations from naturalistic visual experience.
Takeaways & Limitations
ZWM leaves unresolved how semantic concepts such as named objects, relationships, and activities arise developmentally.
Abstract
from arXiv · showhide
Young children demonstrate early abilities to understand their physical world, estimating depth, motion, object coherence, interactions, and many other aspects of physical scene understanding. Children are both data-efficient and flexible cognitive systems, creating competence despite extremely limited training data, while generalizing to myriad untrained tasks -- a major challenge even for today's best AI systems. Here we introduce a novel computational hypothesis for these abilities, the Zero-shot Visual World Model (ZWM). ZWM is based on three principles: a sparse temporally-factored predictor that decouples appearance from dynamics; zero-shot estimation through approximate causal inference; and composition of inferences to build more complex abilities. We show that ZWM can be learned from the first-person experience of a single child, rapidly generating competence across multiple physical understanding benchmarks. It also broadly recapitulates behavioral signatures of child development and builds brain-like internal representations. Our work presents a blueprint for efficient and flexible learning from human-scale data, advancing both a computational account for children's early physical understanding and a path toward data-efficient AI systems.
The ZWM framework
ZWM combines temporally factored prediction, approximate causal inference, and compositional prompting to extract increasingly complex visual-cognitive structures without task-specific supervision.
- Sparse temporally-factored prediction: ZWM separates appearance from dynamics using a sparse temporally factored predictor trained to reconstruct a future frame from a full prior frame and sparse future patches.The predictor implicitly infers appearance, motion, and object structure from this asymmetric masked-prediction problem.
- Zero-shot extraction via approximate causal inference: Zero-shot extraction perturbs an input minimally, compares the predictor’s outputs before and after perturbation, and aggregates the difference into a visual quantity.This procedure makes the predictor’s compressed motion representations explicitly available without task-specific training examples.
- Zero-shot extraction via approximate causal inference: ZWM uses approximate causal inference because its temporally factored predictor supports minimal perturbations that expose latent physical structure, such as pixels belonging to one moving object.Object segmentation uses hypothetical motion to reveal groups of pixels that move together.
- Compositional prompting: Compositional prompting builds complex queries by progressively integrating inferred structures, including optical flow, binocular relative depth, object segmentation, and intuitive physics.The sequence estimates flow, derives depth, segments objects through simulated motion, and combines flow and segments for physical reasoning.
- Model implementation: The predictor is implemented as a Vision Transformer trained by stochastic gradient descent on RGB frame pairs sampled from real-world videos.The implementation includes 170-million- and 1-billion-parameter variants, with frames sampled using inter-frame gaps of 150–450 ms.
Results
BabyZWM performs a broad suite of visual-cognitive tasks zero-shot from naturalistic child-view video, often matching or competing with supervised systems. Its capacities develop across training in trajectories that broadly parallel children’s learning, while its representations align with hierarchical neural responses.
- Task performance: BabyZWM is competitive with supervised models across optical flow, relative depth, object segmentation, and intuitive physical reasoning without task-specific labels.It is strong on established real-world benchmarks and is evaluated alongside representation-based and task-specific alternatives.
- Optical flow: BabyZWM matches supervised flow baselines on challenging real-world videos, while remaining slightly below supervised models trained on synthetic data for Kubric flow.It also matches supervised baselines for occlusion detection and outperforms DINOv3 and V-JEPA2.
- Relative depth estimation: Both ZWM and BabyZWM exceed 90% accuracy on relative depth, comparable to supervised and self-supervised monocular estimators but below a supervised binocular model.Depth is extracted zero-shot by computing optical flow between stereo images.
- Object segmentation: BabyZWM rivals supervised Mask2Former variants for class-agnostic object discovery but performs slightly below SAM2, which uses large-scale human annotations.The segmentation procedure uses hypothetical motion and groups pixels that move together as a latent physical object.
- Intuitive physical understanding: ZWM, BabyZWM, and V-JEPA2 approach 100% across five intuitive-physics categories, whereas Baby V-JEPA2 does not.The benchmark tests cohesion, support, force transfer, and force separation in short-timescale hand–object interactions; BabyZWM attention heads follow the hand when predicting object motion.
- Developmental trajectories: BabyZWM’s optical flow, depth, segmentation, and intuitive-physics abilities improve across training in trajectories that broadly echo reported developmental patterns.The authors caution that benchmark difficulty, metrics, and ceiling effects prevent these curves from establishing a clean ordering of capability development.
- Neural representations: BabyZWM’s neural predictivity shows an early-first trajectory, with earlier model layers aligning to earlier cortical regions and deeper layers to later regions.This alignment appears across human fMRI and macaque electrophysiology datasets and is consistent with hierarchical visual organization.
Discussion
ZWM is presented as a unified zero-shot world model that addresses visual learning’s data-efficiency and task-flexibility gaps while supporting a hybrid account of learning and structure. The discussion also identifies unresolved scope boundaries, including semantic concepts, developmental comparisons, prediction uncertainty, and longer-horizon control.
- Implications for visual learning: ZWM replaces task-specific labeled readouts with a unified predictor that supports zero-shot generalization across challenging real-world, synthetic, and transformed scenes.The model can acquire competence from limited, single-child naturalistic video presented online in a single epoch.
- Implications for visual learning: A single learned predictor yields optical flow, depth, segmentation, and physical reasoning without task-specific labeled readouts.The authors frame this as a universal interface for vision, relevant where large-scale labeled data is unavailable.
- Cognitive-science implications: ZWM combines minimal learning machinery with explicit structural priors, supporting a hybrid position between pure learning and strong innate world knowledge.Its architecture, learning algorithm, and readout programs are treated as potentially innate, while representational content and parameters are learned from experience.
- Cognitive-science implications: The interpretation of zero-shot readouts remains open: they may be hard-wired circuits, developmental adapters, or online query-like inference routines.The passage presents these alternatives as future directions rather than settled explanations.
- Limitations: The current work does not address how semantic concepts arise, provides few detailed developmental behavioral or neural comparisons, and can produce blurry predictions under uncertainty.The deterministic predictor’s mode collapse limits predictions when multiple futures are possible.
- Limitations: Extending ZWM to multi-frame training, richer temporal memory, and long-horizon prediction and control remains an important next step.The present discussion identifies longer-horizon prediction and control as outside the current scope.
Supplementary Materials for Zero-shot World Models Are Developmentally Efficient Learners
The supplementary materials list the paper’s authors.
- The supplementary materials identify Khai Loong Aw, Klemen Kotar, Wanhee Lee, Seungwoo Kim, Khaled Jedoui, Rahul Venkatesh, Lilian Naing Chen, Michael C. Frank, and Daniel L.K. Yamins as authors.
Methods
ZWM uses a temporally asymmetric masked-prediction architecture and zero-shot prompts that extract and compose visual-cognitive quantities from natural video. Models are trained across diverse visual diets and restricted developmental curricula, then evaluated on physical reasoning and other tasks.
- Model architecture: ZWM-170M uses 24 transformer layers and 12 attention heads, whereas ZWM-1B uses 48 layers and 16 heads.The models total approximately 170 million and 1 billion parameters, respectively.
- Model architecture: The predictor fully observes f1 but reveals only 10% of f2 patches, encouraging temporal factorization of appearance and motion.Both frames are patchified into 1024 tokens, with the second frame’s remaining patches replaced by a shared learnable mask token.
- Model architecture: Symmetric 45%-45% and 90%-90% masking variants perform substantially worse across all zero-shot visual-cognitive tasks, indicating that temporal bias is critical.The comparison tests whether asymmetric masking, rather than masking alone, supports flexible zero-shot extraction.
- Training: Training uses masked-patch mean squared error on predicted target frames, with frame pairs sampled from real-world video at 150–450 ms temporal gaps.Models are trained with AdamW, cosine learning-rate decay, gradient clipping, and no data augmentation.
- Zero-shot prompt design: Zero-shot prompts perturb inputs, compare perturbed and unperturbed predictions, and aggregate differences to extract quantities such as optical flow and intuitive physics.Later prompts compose earlier intermediates, including flow, relative depth, and object segments, into more complex inferences.
- Evaluation benchmarks: The intuitive physics benchmark tests five categories of tabletop reasoning involving a hand and one or two objects.It evaluates object cohesion, support, force transfer, and force separation within a short-timescale physical reasoning setting.
Supplementary Text
Supplementary analyses indicate that BabyZWM attends to causal agents during intuitive physics and develops hierarchical representations aligned with visual cortex across species and measurement modalities.
- Attention analysis: In deeper layers, moved-object query patches allocate disproportionately more attention to hand patches than to background or random patches across intuitive physics categories.The hand is treated as the causal agent of object motion in the attention analysis.
- Neural predictivity: BabyZWM shows hierarchical layer-area correspondence in both human fMRI and macaque single-neuron benchmarks.Earlier layers best predict earlier cortical regions, while deeper layers align with higher visual areas.
- Neural predictivity: BabyZWM achieves neural predictivity comparable to ZWM variants trained on substantially larger and more diverse datasets.The cross-species and cross-modality results support the reported hierarchy beyond the scale of fMRI measurements.