Source-linked AI summary
Video Generation Models are General-Purpose Vision Learners
Letian Wang, Chuhan Zhang, Rishabh Kabra, Jasper Uijlings, Steven Waslander, Andrew Zisserman, Joao Carreira, Kaiming He, Misha Andriluka, Eduard Gabriel Bazavan, Andrei Zanfir, Cristian Sminchisescu
TL;DR
Computer vision lacks a unified, general-purpose foundation model with spatiotemporal understanding, language alignment, and scalable pre-training. GenCeption repurposes a pre-trained text-to-video diffusion backbone into a text-steered feed-forward perception model, achieving state-of-the-art performance across diverse tasks and strong data efficiency and generalization.
Problem
Computer vision remains dominated by task-specific models and lacks a universal pre-training objective combining spatiotemporal evolution, vision-language alignment, and scale.
Method
GenCeption fine-tunes a pre-trained video diffusion backbone across diverse visual tasks and reformulates iterative diffusion as a text-steered, single-step feed-forward model.
Results
GenCeption achieves state-of-the-art performance across multiple perception tasks, matches or surpasses specialized models, and shows scaling, data efficiency, and emergent sim-to-real and out-of-distribution generalization.
Takeaways & Limitations
Large-scale video generation provides a foundational pre-training paradigm whose learned spatiotemporal priors and vision-language alignment transfer to unified visual perception.
Takeaways & Limitations
The proposed paradigm requires pre-training that is scaled in both data and compute to enable emergent vision intelligence.
Abstract
from arXiv · showhide
Driven by next-token prediction, NLP shifted from task-specific models into powerful generalist foundation models. What, then, is the equivalent catalyst needed to achieve a general-purpose model in computer vision? In this paper, we contend that large-scale text-to-video generation serves as a strong pre-training paradigm for computer vision, providing the necessary spatiotemporal priors, vision-language alignment, and scalability required for general visual intelligence. We introduce GenCeption, which leverages a pre-trained video generative diffusion backbone to define a feed-forward perception model, capable of performing various vision tasks steered by text instructions. Empirical results demonstrate that GenCeption achieves state-of-the-art performance across a diverse suite of tasks, including depth, surface normal, and camera pose estimation, expression-referring segmentation, and 3D keypoint prediction, often matching or surpassing specialized models (e.g. DepthAnything3, SAM3, D4RT, VGGT-Omega, Sapiens, David, Genmo, and Lotus-2). Furthermore, the video generative pretrained backbone outperforms alternative pretraining paradigms (e.g., V-JEPA, and Video MAE) under comparable settings. Importantly, GenCeption exhibits preliminary data and model scaling properties along with exceptional data efficiency, where it achieves comparable performance with leading models like D4RT and VGGT-Omega with 7 to 500 less training data. Finally, GenCeption also exhibits intriguing emergent behaviors: a model trained exclusively on synthetic human videos generalizes to real-world footage and out-of-distribution object categories (e.g., animals and robots). These findings suggest that video generation is not merely a synthesis tool, but a foundational path toward generalist vision intelligence for the physical world. Project page: https://genception.github.io
1. Introduction
The paper argues that large-scale text-to-video generation can provide a universal pre-training paradigm for generalist vision, and introduces GenCeption to repurpose a video diffusion backbone for diverse perception tasks. GenCeption performs competitively with specialized models while showing scaling, data-efficiency, and emergent generalization behaviors.
- Computer vision remains dominated by task-specific models, motivating a unified architecture and universal pre-training objective analogous to language-model pre-training.
- Text-to-video generation is proposed to satisfy spatiotemporal, vision-language alignment, and scale requirements through high-fidelity video prediction and large datasets and compute.
- GenCeption fine-tunes a pre-trained video diffusion backbone across diverse visual tasks, using text instructions and a feed-forward architecture within a unified task-agnostic model.
- GenCeption achieves state-of-the-art performance across multiple perception tasks, matching or surpassing specialized models including DepthAnything V3, SAM3, D4RT, VGGT-Ω, Sapiens, David, and Genmo.
- The video-generative backbone outperforms alternative pre-training paradigms under comparable settings and improves with more data and larger model size.
- GenCeption achieves comparable performance with leading models using 7× to 500× less training data and generalizes from synthetic videos to real footage and unseen object categories.
2. Related Work
Related work spans image and video perception foundations, self-supervised representation learning, and diffusion-model reuse. GenCeption differs by using native video diffusion, post-training, and efficient feed-forward inference for temporally consistent, multi-task perception.
- Existing perception foundations such as Segment Anything and Depth Anything achieve robust performance but remain primarily task-specific or image-domain systems.
- Masked autoencoders and related video extensions learn representations from reconstructing missing regions, while DINO uses feature-level self-distillation across image views.
- Vision-only representation methods lack explicit vision-language alignment, whereas contrastive vision-language learning supports semantic tasks such as open-vocabulary classification and segmentation.
- Single-image diffusion models can be repurposed for prediction, but their video applications require additional mechanisms to address temporal consistency.
- GenCeption shares the hypothesis that video generative models learn powerful visual priors but adds dedicated post-training and quantitative evaluation across standardized benchmarks.
- Unlike image-space approaches and multi-step generation, GenCeption operates natively on video and uses an efficient feed-forward architecture across a broader task spectrum.
3. GenCeption
GenCeption repurposes a pretrained text-to-video diffusion model into a unified, feed-forward perception model. It supports dense and sparse vision tasks through text-conditioned outputs and shared representations, with primarily synthetic training data.
- Core premise: GenCeption treats a pretrained text-to-video diffusion backbone as a universal visual representation learner for diverse perception tasks.The approach builds on spatiotemporal, geometric, physical, and vision-language representations learned during generative pretraining.
- Feed-forward reformulation: The model converts multi-step diffusion generation into single-step, feed-forward prediction by directly processing clean input-video latents.The DiT is used as a feature extractor rather than an iterative sampler.
- Unified task representation: Text prompts steer one shared architecture across dense tasks such as depth, normals, segmentation, raymaps, and sparse 2D/3D keypoint prediction.Dense outputs use RGB-space representations, while sparse outputs use learnable tokens decoded into per-frame coordinates.
- Unified task representation: The Rothko Raymap compresses camera rotation and translation information into a standard three-channel representation.This adapts high-dimensional camera-ray data to the RGB ambient space used for dense task outputs.
- Synthetic data: Synthetic data addresses the scarcity of real videos containing ground truth for depth, normals, segmentation, pose, and keypoints.The workflow uses RenderPeople assets, CMU motions, varied scenes and cameras, and rendered supervision.
- Training objectives: A unified loss and data-representation strategy reduces the need for task-specific loss functions and manual balancing across heterogeneous objectives.Depth normalization is used as an example of resolving a task-specific constraint at the data level.
4. Experiments
GenCeption is evaluated across depth, normals, segmentation, 3D keypoints, and camera pose, using challenging video and image benchmarks. It generally matches or surpasses specialized models, while showing efficiency, scaling, pretraining, and generalization behaviors alongside task-dependent joint-training effects.
- Evaluation setting: The evaluation covers depth, surface normals, segmentation, 3D keypoints, and camera pose across challenging real-world benchmarks without using their associated training sets.Training is entirely synthetic except for expression-referring segmentation, which incorporates real-world data.
- Efficiency: 5.92s at 13.6 FPS: the 1.3B model performs feed-forward inference on 81-frame 480x832 videos without WAN’s standard 50-step diffusion.The 14B model takes 10.03s at 8.0 FPS under the same video setting.
- Joint-training ablation: Joint training benefits foreground segmentation but can regress depth and camera pose, leave expression-referring segmentation largely unchanged, and severely degrade 3D keypoint estimation.The authors associate the 3D-keypoint degradation with token-based coordinate regression and disruption of pretrained DiT attention mechanics.
- Pretraining and scaling: The generative WAN 2.1 backbone outperforms V-JEPA and VideoMAE V2 under comparable fine-tuning settings, while performance improves with more pretrained layers.Training a randomly initialized DiT from scratch produces a nearly flat learning curve.
- Emergent generalization: A model trained purely on synthetic videos generalizes to real-world footage, including videos with multiple objects and out-of-distribution articulated-object classes.The reported classes include humans, animals, and anthropomorphic characters.
5. Conclusion
GenCeption presents large-scale video generation as foundational pre-training for unified visual perception. Repurposing a video diffusion backbone yields efficient feed-forward perception, specialized-model-level performance, and emergent behaviors.
- GenCeption repurposes a pre-trained video diffusion backbone into a high-efficiency, feed-forward perception model without costly iterative sampling.
- Across diverse vision tasks, GenCeption matches specialized state-of-the-art models.
- The framework provides evidence that generative video backbones contain a universal “world model” supporting generalist visual intelligence for the physical world.