PaperlayerView exact revision on arXiv

Source-linked AI summary

RADIOv2.5: Improved Baselines for Agglomerative Vision Foundation Models

Greg Heinrich, Mike Ranzinger, Hongxu, Yin, Yao Lu, Jan Kautz, Andrew Tao, Bryan Catanzaro, Pavlo Molchanov

arXiv:2412.07679v2cs.CVcs.AI

TL;DR

Agglomerative models need to combine diverse teacher representations, but resolution-dependent mode shifts and the growing token counts of high-resolution VLM inputs present challenges. RADIOv2.5 uses multi-resolution training, teacher-loss balancing, mosaic augmentation, and token compression; reported results include gains across dense tasks and VLMs.

  • Problem

    Agglomerative models must reconcile teacher features across resolutions and distributions, while high-resolution VLM inputs increase token counts and attention complexity.

    Source: §1 · p. 1

  • Method

    RADIOv2.5 combines multi-resolution training, mosaic augmentation, balanced teacher losses, and token compression for VLM integration.

    Source: §1 · p. 2, §4.3 · p. 5, §4.4 · p. 7

  • Results

    Config ℬ improves most dense tasks (+2.8 on ADE20k) and average VLM results (+3.4), with more consistent results at higher resolutions.

    Source: §5.2 · p. 10

  • Takeaways & Limitations

    The framework preserves accuracy across a broad range of resolutions and supports VLM integration while retaining critical visual information under high compression.

    Source: §7 · pp. 11–12

    Knowledge agglomeration assumes that multiple foundation models offer diverse, meaningful image representations and that their knowledge can be distilled into one model.

    Source: §2.1 · p. 2

Abstract

from arXiv · show

Agglomerative models have recently emerged as a powerful approach to training vision foundation models, leveraging multi-teacher distillation from existing models such as CLIP, DINO, and SAM. This strategy enables the efficient creation of robust models, combining the strengths of individual teachers while significantly reducing computational and resource demands. In this paper, we thoroughly analyze state-of-the-art agglomerative models, identifying critical challenges including resolution mode shifts, teacher imbalance, idiosyncratic teacher artifacts, and an excessive number of output tokens. To address these issues, we propose several novel solutions: multi-resolution training, mosaic augmentation, and improved balancing of teacher loss functions. Specifically, in the context of Vision Language Models, we introduce a token compression technique to maintain high-resolution information within a fixed token count. We release our top-performing variants at multiple scales (-B, -L, -H, and -g), along with inference code and pretrained weights

1. Introduction

Knowledge agglomeration must reconcile teachers with different feature resolutions and distributions while producing features suited to diverse tasks. RADIOv2.5 addresses these challenges with multi-resolution training and token merging for high-resolution VLM use.

1. Introduction

  • Teacher distribution balancing and flexible-resolution features remain challenges for combining teachers and serving diverse downstream applications.Applications include image captioning and dense segmentation, which require different feature resolutions.Source: §1 · p. 1
  • Multi-resolution training stabilizes resolution-dependent feature shifts in AM-RADIO and supports accuracy across resolutions.The paper traces the mode shift to students learning from different teachers at different resolutions.Source: §1 · p. 1, Figure 1 · p. 2
  • High-resolution VLM inputs increase output-token counts and therefore the quadratic complexity of attention.The paper proposes ToMeSD [4] to decouple vision-encoder resolution from the number of patches used by the VLM.Source: §1 · pp. 1–2
  • The paper proposes token merging to retain fine-grained details while merging uninformative tokens.This is presented as a way to preserve high-resolution information for VLM integration.Source: Figure 1 · p. 2, §1 · p. 2
  • RADIOv2.5 combines a multi-resolution strategy, feature-selection study, and visual-feature compression for language-model integration.The paper states that its vision encoder disproportionately benefits from the proposed approach.Source: §1 · p. 2

2. Background

The background frames knowledge agglomeration as distilling diverse teacher representations into one student, using teacher-specific adaptors and losses. It introduces the RADIOv2.1-L baseline and an evaluation framework spanning image-level, dense, 3D, and vision-language tasks.

2.1. Knowledge Agglomeration

  • Knowledge agglomeration assumes multiple foundation models provide diverse image representations that can be distilled into one model.Teacher-specific adaptors project student summary and patch features for distillation, while losses align student and teacher features.Source: §2.1 · pp. 2–3
  • Table 1 reports ablations; Probe3D depth and surface-normal metrics are bucket-averaged, and SAM COCO uses the “instance all” bucket.The caption states that each incremental change improves metrics.Source: Table 1 · p. 3
  • The training objective aggregates per-teacher losses, with weighting factors balancing each teacher’s contribution.Summary and patch loss objectives need not use the same similarity or distance metric.Source: §2.1 · p. 3

2.2. Baseline Model

  • RADIOv2.1-L is the paper’s ViT-L replication of AM-RADIO, used for faster iterations.Its baseline uses a ViT backbone with CPE for multi-resolution support.Source: §2.2 · p. 3

2.3. Evaluation Framework

  • The evaluation framework probes image-level reasoning, semantic and instance segmentation, 3D understanding, vision-language reasoning, and additional dense tasks.Image-level evaluation includes zero-shot and k-NN ImageNet1k classification; dense multi-task probing includes Pascal Context and NYUDv2.Source: §2.3 · p. 3

3. Challenges

The section identifies resolution-dependent mode switching in AM-RADIO and excessive visual-token counts as challenges. It describes how the former appears across resolutions and why the latter constrains vision-language models.

3.1. Achieving Multi-Resolution Robustness

  • Config ℬ fixes mode switching and stays strong across more resolutions than Stage 2.Stage 2 avoids mode switching because its teachers were trained only at low resolution, but its quality quickly degrades at high resolution.Source: Figure 2 · p. 4
  • Above roughly 512px, AM-RADIO’s zero-shot Top-1 accuracy drops sharply as its features shift from DINOv2-like to SAM-like.The paper traces the shift to the student learning from different teachers at different resolutions; at high resolution it sees only SAM features.Source: §3.1 · pp. 3–4

3.2. Token Count

  • In VLMs, excessive visual tokens can hurt performance or overflow sequences; 2 × 2 pixel unshuffling reduces token counts but cannot adapt to image-specific information density.Source: §3.2 · p. 4

4. Method

The method addresses scale instability and teacher imbalance through multi-resolution training, mosaic augmentation, and PHI-S, then examines teacher choice, partitioning, token compression, and feature selection. Its findings include better resolution consistency, improved efficiency and quality, and task-dependent benefits from selected features.

4.1.1. A Measure of Scale Equivariance; 4.1.2. Tiling

  • The baseline is less scale-equivariant than DINOv2, while tiling harms scale consistency and loses global context.SigLIP features also vary across tiling arrangements; tiling can increase vision tokens by up to 12×.Source: §4.1.1 · p. 4, §4.1.2 · p. 5, Figure 4 · p. 5

4.2. Multi-Resolution Training

  • Matching all teachers at all resolutions, with two resolutions trained simultaneously in the final stage, is sufficient for consistent accuracy across resolutions.The setup downsamples high-resolution student features for low-resolution teachers and uses padding and cropping for SAM, whose features deteriorate with interpolation.Source: §4.2 · p. 5

4.3. Mosaic Augmentation

  • Mosaic augmentation reduces high-resolution teacher training cost, eliminates feature interpolation, and improves student quality.It runs a high-resolution teacher on mosaics of smaller images, then crops the resulting feature maps for student training.Source: §4.3 · pp. 5–6, Figure 5 · p. 6

4.4. Teacher Loss Balancing

  • PHI-S rotates and rescales teacher activations to balance their losses; the baseline otherwise spends too much energy matching SAM.PHI-S reduces errors relative to teacher variance, and the authors find it improves training stability and overall benchmark performance.Source: §4.4 · pp. 6–7

4.5. Is SAM a good teacher?

  • With mosaic augmentation and PHI-S, including SAM has negligible but positive classification impact and strongly benefits dense tasks.The authors also note that SAM inclusion supports downstream applications such as VideoSAM.Source: §4.5 · p. 7, Table 4 · p. 7

4.6. Partitioning; 4.7. SigLIP Teacher

  • Fewer partitions favor summarization when teacher overhead is affordable, while partitioning reduces overhead under compute constraints; the authors also replace OpenAI-CLIP with SigLIP for VLM gains.The partition-count effect is less clear for dense tasks.Source: §4.6 · p. 7, §4.7 · p. 8

4.8. Token Compression

  • +1.4 average improvement on VLM benchmarks follows output-stage token merging; token merging also retains diverse information under high compression.The method uses strided bipartite matching of similar tokens, with features rather than attention keys for matching.Source: §4.8 · p. 8, Figure 8 · p. 8

4.9. Feature Selection

  • Intermediate activations benefit downstream tasks when a nonlinear transformation is used, but dense features do not improve VLM results in this study.A linear probe alone does not leverage the added intermediate information, and the authors note they cannot disentangle that information’s contribution from the DPT head’s extra parameters.Source: §4.9 · p. 9, Table 7 · p. 9

5. Experiments

Experiments compare RADIOv2.5 with other vision encoders and agglomerative models across dense and vision-language tasks. Ablations examine multi-resolution training, teacher and backbone choices, and token compression.

5.1. Evaluation Benchmarks

  • Evaluation covers image classification, instance and semantic segmentation, 3D understanding, and VLM benchmarks.The protocols use ImageNet1k, COCO, ADE20k, Navi, and several visual-question-answering and OCR benchmarks.Source: §5.1 · p. 9

5.2. Ablation Studies

  • Adding multi-resolution training in config ℬ improves most dense tasks, including ADE20k by 2.8 points and VLMs by 3.4 points on average.Config ℬ also shows more consistent results at higher resolutions.Source: §5.2 · p. 10
  • Replacing OpenAI CLIP with SigLIP in config 𝒞 yields a 6.7-point improvement, while the ViT-H backbone in config 𝒟 improves all metrics.Config 𝒟 replaces the 320M-parameter ViT-L with a 553M-parameter ViT-H.Source: §5.2 · p. 10
  • Pixel unshuffling with Token Merging reduces vision tokens from 576 to 256 and improves VLM scores by 5.9 points on average.The appendix analysis compares token-count and accuracy trade-offs with SigLIP.Source: §5.2 · p. 10

5.3.1. Dense Task Evaluation

  • RADIOv2.5 models consistently improve over previous baselines on dense-task benchmarks.Table 8 compares dense-task benchmarks against other agglomerative models.Source: Table 8 · p. 10, §5.3.1 · p. 11

5.3.2. VLM Evaluation

  • RADIOv2.5 models consistently exceed other vision encoders in VILA accuracy when visual features are compressed to 196 tokens.The comparison uses MN-Minitron-8B and the LLaVA1.6 data mixture.Source: Table 9 · p. 10, §5.3.2 · p. 11

6. Related Work

Related work spans multi-teacher vision distillation, flexible-resolution vision transformers, and multi-encoder integration into VLMs. RADIOv2.5 is distinguished by developing one versatile vision encoder through teacher balancing and resolution-focused training.

6.1. Agglomerative Models

  • Prior agglomerative models use different strategies, while RADIOv2.5 standardizes teacher features and targets accuracy across scales with multi-resolution training.AM-RADIO has inconsistent accuracy across resolutions; SAM-CLIP lacks a strong dense model such as DINO, while UNIC uses expandable projectors and teacher-dropping regularization.Source: §6.1 · p. 11

6.2. Multi-Resolution Support

  • Unlike FlexiViT and NaViT, RADIOv2.5 distills resolution-robust students from teachers that cannot flexibly change resolution.It also uses Cropped Position Embeddings for inference across scales.Source: §6.2 · p. 11

6.3. Multi-Encoder VLMs

  • Where Cambrian-1, Eagle, and BRAVE combine multiple encoders within VLM architectures, RADIOv2.5 aggregates them first into a single vision encoder.The cited approaches respectively use latent-query cross-attention, channel-wise token concatenation, and a fixed-length representation.Source: §6.3 · p. 11

6.4. Applications of Agglomerative Models

  • The paper situates agglomerative models in applications including robotic-hand semantic keypoint tracking and multi-image correspondence.It expresses a broader interest in VLMs as computer-vision generalists.Source: §6.4 · p. 11

7. Conclusion

The conclusion presents RADIOv2.5 as a multi-teacher distillation framework designed to preserve accuracy across resolutions. It highlights teacher balancing, token compression, and feature selection as complementary contributions.

7. Conclusion

  • RADIOv2.5 accommodates differences in teacher resolution, activation magnitude, and noise patterns while targeting accuracy across resolutions.Its multi-resolution training mitigates resolution-dependent performance degradation, including mode switching.Source: §7 · p. 11
  • Mosaic augmentation and PHI-S balance computational load and the loss contributions of individual teachers.Source: §7 · p. 12
  • Token compression supports VLM integration while preserving critical visual information, and intermediate activations benefit dense tasks, particularly with nonlinear transformations.Source: §7 · p. 12

A.1.1. More Vision Encoder Comparisons

The appendix comparison reports VILA results for RADIOv2.5-H under matched and increased token budgets. RADIOv2.5-H improves over the SigLIP baseline at the matched token count, with further benefits at a larger count.

A.1.1. More Vision Encoder Comparisons

  • RADIOv2.5-H significantly improves results over the SigLIP baseline at the same 196-token count, with additional benefits at 512 tokens.The comparison also includes OpenAI-CLIP-336 and AM-RADIO-H.Source: §A.1.1 · p. 16, Table A1 · p. 16
  • The comparison holds the Qwen2-7B-Instruct LLM, training data, and hyperparameters constant across configurations.The reported VILA metrics include TextVQA, ChartQA, DocVQA, InfoVQA, OCRBench, GQA, POPE, MME, SEED, and AI2D.Source: §A.1.1 · p. 16, Table A1 · p. 16

A.1.2. Scaling up to More Data

With the improved 9.8M-sample SFT mixture, RADIOv2.5-H outperforms SigLIP on the reported VILA benchmarks except AI2D, where they tie.

A.1.2. Scaling up to More Data

  • The VILA benchmark table reports text, chart, document, OCR, general question answering, perception, and image-understanding metrics.Its average rescales MME by 20 and OCRBench by 10 before averaging.Source: Table A2 · p. 16
  • RADIOv2.5-H outperforms SigLIP on every reported benchmark except AI2D, where the models tie.The comparison uses Qwen2-7B-Instruct and an improved data mixture.Source: §A.1.2 · p. 16, Table A2 · p. 16
  • The improved SFT mixture totals 9.8M samples and includes ShareGPT4v, LLaVA Instruct, Cambrian, VFLAN, and some benchmark training sets.Source: §A.1.2 · p. 16

A.1.3. Effect of the Compression Method

The appendix compares token compression and high-resolution inference, reports segmentation and dense-task results, and provides scale-variance and feature visualizations.

A.1.3. Effect of the Compression Method

  • Token Merging significantly improves RADIOv2.5-H, while increasing its token count from 256 to 512 adds a modest gain; it does not improve SigLIP.Table A3 compares encoders and compression methods across VILA metrics.Source: §A.1.3 · p. 16, Table A3 · p. 17

A.1.4. Block-wise vs Output Token Merging

  • One-shot token merging at the ViT output yields improved results over incremental block-wise merging in this setup.The comparison uses RADIOv2.5-H and contrasts merging criteria and token layouts.Source: §A.1.4 · p. 17, Table A4 · p. 17

A.1.5. High-Resolution Inference using Tiling

  • RADIOv2.5 shows slightly improved accuracy with fewer output tokens than SigLIP under the reported tiling comparison.The comparison is not matched in pixels or tokens: RADIOv2.5 processes 3.9M pixels and outputs 1,313 tokens on average, versus 1.8M pixels and 2,410 tokens for SigLIP.Source: §A.1.5 · p. 17, Table A5 · p. 17

A.2. Semantic Segmentation

  • RADIOv2.5 favorably scales to higher resolutions, whereas AM-RADIO accuracy falls above 512 × 512 on semantic segmentation.The evaluation reports mIoU on ADE20k and VOC; the table also states RADIOv2.5-B exceeds DINOv2-g-reg despite being one-tenth its size.Source: §A.2 · p. 18, Table A6 · p. 18

A.3. Additional Benchmarks

  • At the largest scale, RADIOv2.5-H is extremely competitive with DINOv2-g at half the parameters; ViT-B and ViT-L are the closest same-size agglomerative models to DINOv2.The reported dense-task evaluations use a frozen backbone and MLoRE’s conv head; hyperparameters were not tuned.Source: §A.3 · p. 18

A.5. A Measure of Scale Equivariance

  • The scale-variance procedure standardizes features per channel, resizes them to a common spatial size, then averages variance across scales and spatial positions.Source: §A.5 · pp. 19–20

A.7. Visualizations of Native vs Emulated High-Resolution Inference

A.6. Mode-Switch PCA Visualizations

  • RADIO’s baseline features shift from DINO-like at low resolution to SAM-like at high resolution, as illustrated by PCA visualizations.Source: §A.6 · p. 21, Figure A1 · p. 22
Loading 2412.07679v2…