Source-linked AI summary
Sequential Modeling Enables Scalable Learning for Large Vision Models
Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan Yuille, Trevor Darrell, Jitendra Malik, Alexei A Efros
TL;DR
Large vision models lack the unified, diverse visual data structure available to language models, motivating the question of how far pixels alone can support scalable learning. The paper represents images, videos, and annotations as visual sentences, tokenizes them, and trains an autoregressive Transformer on next-token prediction. Across model sizes and tasks, the approach scales effectively and supports prompt-based vision tasks, while its generalization and failure boundaries remain open.
Problem
The paper asks how far a Large Vision Model can go from pixels alone, given the need for diverse visual data and flexible task specification.
Method
The paper creates purely visual sentences from diverse images, videos, annotations, and 3D data, then trains autoregressively on their token sequences.
Results
The models show appropriate scaling with model and data size, address many vision tasks through visual prompts, and achieve 81.2 PCK on unseen Pascal 3D+ keypoint detection without dataset training.
Takeaways & Limitations
A single vision model can address many tasks through prompting while benefiting from large-scale visual data.
Takeaways & Limitations
Visual prompts are often under-constrained, tokenizer issues and limited high-quality video data cause failures, and emergence and true generalization remain open questions.
Abstract
from arXiv · showhide
We introduce a novel sequential modeling approach which enables learning a Large Vision Model (LVM) without making use of any linguistic data. To do this, we define a common format, "visual sentences", in which we can represent raw images and videos as well as annotated data sources such as semantic segmentations and depth reconstructions without needing any meta-knowledge beyond the pixels. Once this wide variety of visual data (comprising 420 billion tokens) is represented as sequences, the model can be trained to minimize a cross-entropy loss for next token prediction. By training across various scales of model architecture and data diversity, we provide empirical evidence that our models scale effectively. Many different vision tasks can be solved by designing suitable visual prompts at test time.
1. Introduction
The paper asks how far a large vision model can go from pixels alone, while retaining scalable training and prompt-based task flexibility. It unifies diverse visual data into sequences and trains on next-token prediction.
- The central question is how far vision competence can be developed from pixels alone, without relying on linguistic data.
- The model aims to scale with large data and support flexible task specification through prompting.
- Visual sentences represent raw images, videos, and annotations such as segmentations, depth, keypoints, and multiview objects without pixel-external meta-knowledge.The training dataset contains 1.64 billion images or frames.
- A 3-billion-parameter Transformer processes visual data as token sequences, with each image mapped to 256 vector-quantized tokens.
- Cross-entropy next-token prediction trains the model after images, videos, and annotated images are represented as sequences.
- The model shows scaling with model and data size, supports many prompted vision tasks, benefits from unsupervised data, and hints at visual reasoning, though further study is needed.
2. Related Work
Prior work spans pretrained vision models, multitask and in-context learning, and autoregressive visual generation. The paper builds on these directions while targeting broader scale and task flexibility.
- Pretrained Vision Models: Pretrained vision models evolved from CNN-based methods to higher-capacity Transformer approaches, but vision-only models have struggled with very large datasets.
- Multi-task Learning and In-context Learning: Traditional multitask learning supports multiple tasks but usually fixes the task set in advance.
- Multi-task Learning and In-context Learning: In-context approaches instead let a model infer the task from a prompt containing task information and a query image.
- Auto-regressive Visual Models: Autoregressive visual modeling has been applied to pixels, patches, video frames, and motion data, with newer work replacing N-grams by neural architectures.
3. Data
UVDv1 assembles diverse purely visual sources into a large dataset, and visual sentences provide a common sequential format for images, videos, annotations, and synthetic 3D views.
- Dataset composition: UVDv1 combines unlabelled images, annotated images, unlabelled and annotated videos, and synthetic 3D objects into 1.64 billion images.The dataset contains no non-visual metadata such as text.
- Dataset composition: Unlabelled images provide over 80% of the data and broad visual diversity, while annotations, videos, and synthetic objects contribute quality, temporal information, or 3D structure.
- Visual sentences: Single images, sampled video clips, multiview synthetic objects, and same-category image groups are organized as visual sentences.
- Image sequences: Video visual sentences use 16 frames sampled at strides of 10, 20, or 30, while Objaverse contributes rendered multiview sequences.The paper rendered 42,000 training sequences and 8,000 testing sequences.
- Annotated data: Annotations are represented as images, allowing segmentation, edges, depth, normals, detection, and other annotation types to share a visual-sequence format.
- Annotated data: Annotated videos are serialized either by alternating each frame with its annotation or by grouping frames before their corresponding annotations.
4. Approach
The approach tokenizes each image, concatenates tokens across a visual sentence, and trains an autoregressive Transformer to predict subsequent tokens. At test time, partial visual sentences act as task prompts.
- Architecture: The model uses a two-stage design: a visual tokenizer converts images to token sequences, then an autoregressive Transformer models visual sentences.
- Image tokenization: A 256 × 256 image produces 256 tokens, with each token selected from an 8192-entry codebook.
- Image tokenization: VQGAN encodes each image into discrete visual tokens, and its decoder converts predicted tokens back into images.
- Sequence modeling: Tokens from multiple images are concatenated into one-dimensional sequences, which the causal Transformer trains with cross-entropy next-token prediction.
- Training: The model is trained on 420 billion tokens using one epoch across 300M, 600M, 1B, and 3B parameter sizes.
- Prompting: Downstream tasks are specified by constructing a partial visual sentence and sampling generated tokens to complete the requested output.
5. Experimental Results and Analysis
The experiments show that LVM performance scales with model size, training data size, and dataset diversity, while visual prompts support diverse sequential, analogy, and compositional tasks. Longer temporal context improves frame prediction until performance stabilizes, and the model generalizes to an unseen keypoint-detection dataset.
- Scalability: LVM training loss decreases throughout training, with larger models reducing loss faster as they process more data.All models were trained for one epoch, making training loss similar to validation loss.
- Scalability: Larger models attain lower perplexity across semantic segmentation, depth estimation, surface normal estimation, and edge detection tasks.Evaluation uses five-shot prompts on ImageNet validation images and measures perplexity of the ground-truth annotation prediction.
- Dataset ablation: Each image, video, and annotation data component contributes positively to downstream performance, showing benefits from dataset diversity.The ablation trains several 3B-parameter models on different dataset subsets.
- Sequential prompting: 49.8 perplexity is reported for next-frame prediction on the Kinetics-700 validation set, with predictions reflecting spatial positioning, viewpoint, and object understanding.The model is prompted with seven frames; examples also use 15-frame contexts and predict four frames.
- Sequential prompting: 62.1 → 48.4 perplexity is observed as context increases from 1 to 11 frames, after which performance stabilizes.This context-length analysis uses frame-generation perplexity on the Kinetics-700 validation set with contexts ranging from 1 to 15 frames.
- Analogy Prompting: Analogy prompts use image sequences followed by a query to produce task outputs, including tasks unseen during training.The paper also demonstrates task compositing, where rotation and keypoint correspondence are combined in one prompt.
- Analogy Prompting: 81.2 PCK is achieved on Pascal 3D+ keypoint detection without training on that dataset, exceeding the listed task-specific baselines.The listed baselines score 68.0, 68.9, and 78.6 PCK.
6. Limitations
The current LVM has several limitations, including under-constrained visual prompts, capability boundaries, tokenizer issues, insufficient video data, and constrained experimentation.
- Visual prompts can be under-constrained because images are more high-dimensional than language.
- Some requested tasks may exceed the capabilities of the current system.
- Tokenizer issues and a lack of high-quality video training data cause additional failures.
- Limited computing resources prevented exploration of different datasets and detailed ablation studies.
- Although among the biggest vision models to date, the model remains small compared with modern Large Language Models.
- The emergence and true generalization of Large Vision Models remain open questions.
Appendix B: Unified Vision Dataset (UVD) In-Depth Analysis
The supplementary analysis examines UVD, including its composition, data distribution, and additional dataset details.
- UVD integrates diverse datasets with distinct characteristics and annotation types.
- The dataset analysis covers composition and data distribution.
- The appendix provides further details about UVD beyond its headline description.
Appendix C: Additional Results
The supplementary appendix provides extended results and visual evidence, including additional figures and quantitative assessments.
- The appendix extends the study’s reported results.
- It includes supplementary figures as visual evidence.
- It also provides quantitative assessments.
A.1. Model Architectures.
The appendix presents the LVM model architecture configurations and identifies them in Table 2.
- The architecture uses the Transformer variant of LLaMA.
- Different model sizes vary hidden dimension, MLP intermediate dimension, number of heads, and number of layers.
- The appendix describes the model architecture configurations of the LVMs.
A.2. Training and optimizer details.
The training setup uses AdamW and a large TPU-based transformer implementation, with UVD providing diverse visual data sources for pretraining.
- Training and optimizer details: The models use AdamW with shared optimizer hyperparameters across model sizes.Training runs use TPU-v3 pods on Google Cloud.
- Hyperparameters: Table 3 presents the hyperparameters used for LVM pretraining.The supplied passage identifies the table but does not enumerate its values.
- Training data: UVD combines unpaired images, annotated images, videos, annotated videos, and synthetic 3D views.The dataset integrates visual data spanning multiple content and annotation types.
B.2. Summary of Dataset Distribution in UVD
UVDv1 aggregates more than 430 billion tokens across predominantly unannotated images, alongside annotated images, videos, annotated videos, and synthetic 3D views. Its video data is tokenized into varied frame sequences, supporting sequential prediction experiments.
- Dataset composition: 88.49% of UVDv1 tokens come from single images, making unannotated image data the dominant category.Figure 14 likewise states that most training data comes from LAION single images.
- Dataset composition: Synthetic 3D views account for 0.05% of the dataset, supporting advanced 3D vision tasks.The category contains datasets such as Objaverse.
- Video tokenization: Video datasets use tailored tokenization strategies with dataset-specific strides, sequence lengths, and epoch schedules.Examples include 16-frame sequences for YouCook, CharadesEgo, YouTube VOS, and several other video datasets.
- Sequential prompting: Sequential prompting evaluates sketch understanding, arbitrary-axis 3D rotation, and frame prediction by predicting future images or frames.The model predicts one subsequent sketch image, four rotated views, or four future frames depending on the task.
C.2. Analogy Prompting
Analogy prompting casts diverse vision tasks as visual input-output transformations, including pose, depth, surface normals, segmentation, edges, inpainting, colorization, and deraining. Sequential prompts likewise support sketch, 3D rotation, and frame prediction.
- Dense prediction: Analogy prompts map input images to task-specific outputs such as joints, depth, surface normals, segmentation, or edges.The prompts represent annotations as visual images and ask LVM to predict the corresponding output map.
- 3D understanding: Surface-normal prompting shows robustness and generalization despite inaccuracies in some prompt normal images.This result concerns understanding 3D structures from 2D data.
- Image restoration: MSE 0.106 is reported for image inpainting under partially masked image-to-image prompting.The evaluation varies mask ratios and reports semantic understanding through reconstruction error.
- Image restoration: MSE 0.51 is reported for image colorization using grayscale image-to-image analogy prompting.The model is prompted to colorize an input grayscale image.