Multimodal Autoregressive Pre-training of Large Vision Encoders

1. Introduction

• AIMV2 is motivated by the gap between generative methods’ simplicity and scalability and discriminative methods’ parameter efficiency, amid pressure to rethink vision pre-training for LLM-era applications.The introduction notes that generative vision pre-training still lags discriminative methods, while contrastive approaches can be challenging to train and scale.Source: §1 · p. 1
• AIMV2 pairs a vision encoder with a joint multimodal decoder that autoregressively reconstructs image patches and text tokens, using a straightforward training objective.Visual representations are concatenated with corresponding caption embeddings before the decoder processes the sequence.Source: §1 · p. 1, Figure 1 · p. 2
• The method is presented as requiring neither excessively large batch sizes nor specialized interbatch communication, while extracting training signal from every image patch and text token.The authors also describe its architecture and objective as aligning with LLM-powered multimodal applications.Source: §1 · p. 1
• AIMV2 performs favorably against state-of-the-art vision-language methods [35], outperforms DINOv2 on open-vocabulary detection and referring-expression comprehension, and improves consistently with data or parameter scaling.The introduction also reports strong frozen-trunk recognition and compatibility with native-resolution and zero-shot recognition tools.Source: §1 · pp. 1–2

2.1. Pre-training

• AIMV2 concatenates image patches before text tokens, then uses a multimodal decoder to predict the next token with image and text losses.The image loss is pixel-level regression; the text loss is cross-entropy, combined as L_text + α * L_img.Source: §2.1 · p. 2

2.2. Architecture

• At fixed pre-training data size, validation loss improves as parameter count increases, while the compute budget changes the optimal model size.Larger models can perform worse when severely undertrained, but improve consistently as compute increases.Source: Figure 2 · p. 3
• AIMV2 uses ViT vision encoders and a shared-capacity multimodal decoder, with separate output heads for image and text predictions.The model family spans 300M to 3B encoder parameters.Source: §2.2 · pp. 2–3, Table 1 · p. 3
• AIMV2 pre-training uses 12 billion image-text samples drawn from public and private datasets, with alt-text and synthetic captions.The stated sources include DFN-2B, COYO, and a proprietary high-quality image-text-pair dataset.Source: §2.3 · p. 3, Table 2 · p. 3

2.4. Post-Training

• High-resolution adaptation fine-tunes models from 224px pre-training resolution to 336px or 448px using 2 billion image-text pairs.This stage excludes synthetic captions; the paper reports that zero weight decay is important for stable optimization.Source: §2.4 · p. 3
• Frozen-trunk IN-1k top-1 accuracy scales strongly with model parameters, and the trend persists when image resolution increases.Figure 3 describes the measurement as frozen-trunk top-1 accuracy for IN-1k.Source: Figure 3 · p. 4
• AIMV2 outperforms a captioning-only baseline across dataset and model sizes and shows fewer signs of saturation as data scales.The comparison investigates the image-level objective against language-only supervision.Source: Figure 4 · p. 4
• Native-resolution fine-tuning varies image area and batch size so each mini-batch contains a fixed total number of patches.The approach preserves aspect ratios and avoids sequence-packing heuristics, attention masking, and custom pooling operations.Source: §2.4 · p. 4

3. Analysis

• The analysis examines scaling data and model capacity, then compares AIMV2 with captioning-only training to assess the image-level objective.The study’s scaling tests use intermediate checkpoints from runs trained to convergence.Source: §3 · p. 4, §3.1 · p. 4, §3.2 · p. 4

3.1. Scaling AIMv2

• Performance improves consistently with more data or parameters, but larger models can underperform smaller ones at low compute budgets because they are undertrained.The scaling study trained models from 300 million to 3 billion parameters and varied exposure from 500 million to 6.4 billion image-text pairs.Source: §3.1 · p. 4

3.2. AIMv2 vs. Captioning

• The image-level objective improves frozen-trunk ImageNet-1k accuracy over captioning-only training across model capacities and data sizes, without the captioning baselines’ observed data-scaling plateau.Both approaches improve with more data or capacity; plateauing with additional data is observed for captioning baselines but not AIMV2.Source: §3.2 · p. 4

4. Results

• The paper evaluates AIMV2 as an off-the-shelf generalist vision encoder across recognition, detection, captioning, and multimodal benchmarks.The multimodal evaluations include instruction tuning and few-shot in-context learning.Source: §4 · p. 5, §4.3 · p. 6

4.1. Image Recognition

• 89.5% top-1 accuracy on ImageNet-1k is achieved by the AIMV2-3B variant finetuned for 448px images, with a frozen trunk.At ViT-Large capacity, AIMV2 beats OAI CLIP on most recognition benchmarks and exceeds DFN-CLIP and SigLIP on several, despite using 12B versus 40B training pairs.Source: §4.1 · p. 5
• AIMV2-Lnative attains 87.3% ImageNet-1k accuracy on original native-resolution validation images, with only minor degradation versus dedicated-resolution models.With LiT tuning, AIMV2 outperforms OAI CLIP in zero-shot ImageNet-1k but trails longer-trained SigLIP.Source: Table 6 · p. 5, Table 5 · p. 5, §4.1 · p. 6

4.2. Object Detection and Grounding

• AIMV2 outperforms DINOv2 and other vision-language models on all but one open-vocabulary detection and referring-expression benchmark, with particularly strong LVIS performance.The reported metrics are mean average precision for open-vocabulary detection and Precision @1 for referring expression comprehension.Source: Table 4 · p. 5, §4.2 · p. 6

4.3.1. Multimodal Instruction Tuning

• AIMV2-L outperforms OAI CLIP, SigLIP, and DINOv2 on most instruction-tuned benchmarks; AIMV2-3B performs best on all but one.The evaluations use a frozen vision encoder connected to the LLM through a 2-layer MLP; the connector and LLM are jointly trained.Source: Table 7 · p. 6, §4.3.1 · p. 7
• AIMV2 matches or exceeds OAI CLIP and SigLIP across instruction-tuning settings that vary the LLM decoder and data mixture.The evaluated combinations include Llama 3.0 with Cambrian data and Vicuna 1.5 with the Llava SFT mixture.Source: §4.3.1 · p. 7, Figure 6 · p. 7
• AIMV2 retains its advantage over OAI CLIP and SigLIP when high-resolution inputs are processed by tiling.All methods improve at higher resolutions, with a significant improvement for TextVQA; the study uses 2×2 crops at 672px and 3×3 crops at 1008px.Source: Figure 5 · p. 6, §4.3.1 · p. 7

4.3.2. Multimodal In-Context Learning

• AIMV2 achieves the best average performance in the 4-shot and 8-shot multimodal in-context settings, surpassing the higher-capacity DFN-CLIP used by MM1.The reported averages cover eight benchmarks, including COCO, TextVQA, VQAv2, and OK-VQA.Source: Table 8 · p. 6, §4.3.2 · p. 7

• Combining image-level and text-level objectives improves performance, particularly on TextVQA; AIMV2 also outperforms CLIP and CapPa under comparable training settings.The study reports CLIP results at both 8k and 16k batch sizes for comparison.Source: §5 · p. 8
• Captioning-only training performs better with prefix attention than fully bidirectional attention in the vision encoder; the authors frame the proposed explanation as a hypothesis for further study.The hypothesis is that prefix attention enables informative contexts from partial images for subsequent visual and textual tokens.Source: §5 · p. 8
• A single joint decoder achieves results comparable to separate decoders while offering greater simplicity and pre-training efficiency as data and model capacity scale.Increasing decoder capacity beyond a threshold can reduce performance, with width changes having a greater effect than depth changes.Source: §5 · p. 8
• Performance varies only slightly around an objective-balancing value of α = 0.4 across the three benchmarks.The ablation uses a ViT-Large encoder, 2 billion image-text pairs, and 224 × 224 images.Source: §5 · p. 8

• Earlier autoregressive vision models discretized pixels or patches, while AIM [33] used continuous reconstruction but lagged state-of-the-art performance and needed high-capacity models.AIMV2 addresses these limitations with multimodal pre-training; concurrent multimodal autoregressive work often prioritizes generation quality over representation quality [113] [124] [125] [131].Source: §6 · p. 9
• Vision pre-training has moved from supervised checkpoints to self-supervision and web-scale contrastive vision-language learning, each with noted scalability or training trade-offs.Contrastive models require large batches and dataset filtering, while AIMV2 does not perform explicit feature-space alignment.Source: §6 · p. 9
• Captioning methods have supported visual pre-training, but AIMV2 differs by treating vision and text tokens symmetrically without cross-attention and adding autoregressive image modeling.The related work compares this design with captioning approaches such as CapPa and multimodal systems such as LLaVA and MM1 [85].Source: §6 · p. 9

7. Conclusion

• AIMV2's unified objective supports image recognition, grounding, and multimodal understanding, with the conclusion reporting strong performance and improved sample efficiency.The paper attributes this to leveraging signals from all input tokens and patches.Source: §7 · p. 9
• The conclusion reports that AIMV2 consistently outperforms or matches existing self-supervised and vision-language models, and describes its pretraining as straightforward to scale.It also identifies scalability as a basis for further vision-model scaling.Source: §7 · p. 9

A. Hyperparamters

• Captions use the SigLIP tokenizer, and text longer than 77 tokens is truncated.Source: A. Hyperparamters · p. 15
• Attentive probing uses shared hyperparameters across AIMV2 capacities and baselines, with learning rate and weight decay swept to report each model's strongest result.Source: A. Hyperparamters · p. 15

B. Image Recognition

• Recognition benchmarks span fine-grained, medical, satellite, natural-environment, and infographic imagery, with dataset splits and sizes listed in Table B1.Source: B. Image Recognition · p. 15, Table B1 · p. 15
• Scaling model capacity and image resolution leads to consistent recognition improvements across most tasks.The appendix compares 224px, 336px, and 448px inputs; Table B2 reports frozen-backbone attentive-probing results.Source: B. Image Recognition · p. 15, Table A2 · p. 15
• Attentive probing sweeps learning rate and weight decay and reports each model's strongest result.The same probing hyperparameters are used for AIMV2 and baselines.Source: Table A2 · p. 15

C.1. Instruction Tuning Setup

• Instruction-tuning hyperparameters are held constant across language decoders, AIMV2 capacities, and baselines.Source: §C.1 · p. 15

C.2. Additional Results

• Table B2 evaluates high-resolution AIMV2 recognition models against 224px models using attentive probing with a frozen backbone.Source: Table B2 · p. 16
• With Cambrian tuning and tiling, increasing resolution consistently improves AIMV2 on text-rich benchmarks including InfoVQA, ChartQA, DocVQA, and TextVQA.The evaluation compares 336px with tiled 672px and 1008px images.Source: §C.2 · pp. 15–16
• COCO, NoCaps, TextCaps, and MMEp show no significant gains from increased resolution.Source: §C.2 · p. 16
• AIMV2 consistently outperforms the baselines with both large and small language decoders, remaining the preferred choice for multimodal understanding with a DCLM-1B decoder.Source: §C.2 · p. 16
• The appendix lists multimodal evaluation benchmarks with their splits, prompts, and metrics, and supplies instruction-tuning hyperparameters for the Llava SFT and Cambrian mixtures.Source: Table C2 · p. 16, Table C1 · p. 16

C.3. Qualitative Results

• After Cambrian instruction tuning, AIMV2 correctly reads and localizes text in cluttered or blurry scenes and gives a more plausible calorie estimate than SigLIP and OAI CLIP.Examples include reading a license plate and luggage weight despite visual distractions.Source: §C.3 · p. 16, Figure C1 · p. 17
• Table C3 compares AIMV2 multimodal evaluations across Llava [73] and Cambrian [115] SFT mixtures and resolutions of 336px, 672px, and 1008px.Source: Table C3 · p. 17
• Table D1 reports COCO and LVIS mean average precision, including object-size subsets for COCO and frequency subsets for LVIS.Source: Table D1 · p. 17

D.1. Open Vocabulary Detection and Grounding

• AIMV2 exceeds DFN-CLIP by +0.5 AP on COCO small classes and by +1.3 AP on the LVIS frequent subset.The passage notes that LVIS objects are more likely to be small.Source: §D.1 · p. 17
• Across window sizes, AIMV2 outperforms DINOv2 on all OVD benchmarks and on two of three referring-comprehension benchmarks.The ablation varies AIMV2's pre-training resolution to match the fine-tuning window size, while also evaluating DINOv2 with matched and pre-training-sized windows.Source: §D.1 · p. 17, Table D2 · p. 17
• With a DCLM 1B decoder, AIMV2 performs better on most of 12 multimodal benchmarks than OAI CLIP and SigLIP.The comparison uses the Llava SFT mixture and a constrained-capacity decoder.Source: Figure C2 · p. 18
• AIMV2 strongly outperforms the best DINOv2 on LVIS Val and outperforms it on all but one benchmark.The comparison covers the best-performing AIMV2 and DINOv2 variants across benchmarks.Source: §D.1 · p. 18

D.2. Detection and Segmentation via ViTDet Mask-RCNN

• AIMV2 consistently outperforms contrastive-pretrained encoders on COCO17 detection and segmentation, while DINOv2 provides the strongest performance.All compared detectors use a ViTDet formulation of Mask-RCNN.Source: §D.2 · p. 18
• The detectors are finetuned on COCO17 train for 100 epochs with batch size 64, using shared hyperparameters and 1024-pixel inputs.The recipe includes windowed attention size 16, AdamW, cosine learning-rate decay, layer-wise learning rates, and weight decay.Source: §D.2 · p. 18
• The evaluation reports detection and segmentation results on COCO17-val using Mask-RCNN backbones.The backbone forward passes are matched to the respective ViT-L implementations before training.Source: §D.2 · p. 18