AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into One

1. Introduction

• AM-RADIO students can exceed any teacher on important axes while supporting CLIP zero-shot applications and Segment-Anything tasks through compatible language models and mask decoders.The student can replace the vision encoder for already-trained mask decoders.Source: §1 · p. 3
• Evaluating more than 10 architectures under one training recipe showed that CNN-like models are faster but struggle to distill ViT VFMs, motivating hybrid E-RADIO, at least 6x faster than teachers at matched resolution.E-RADIO exceeds the performance of its predecessors.Source: §1 · p. 3
• The method distills multiple distinct foundation models, including models with incompatible input resolutions, into one student that can replace teachers or provide visual encodings for LLaVA.The introduction identifies CLIP, DINOv2, and SAM as teachers with complementary capabilities in zero-shot performance, dense spatial features, and open-vocabulary instance segmentation.Source: §1 · pp. 2–3

Knowledge Distillation

• The method matches teacher feature representations before task-specific processing, ignoring data labels and teacher logits rather than using soft output targets.Knowledge distillation can use teacher logits as soft labels or intermediate activations; AM-RADIO chooses feature matching for its heterogeneous teachers.Source: §2 · p. 3

Multi-Teacher Distillation

• For multiple heterogeneous teachers, AM-RADIO maps the unified student representation to each teacher independently through teacher-specific projection heads instead of marginalizing teachers into one label.It omits feature normalization in the loss to preserve drop-in compatibility with teacher frameworks.Source: §2 · p. 3

Distilling Foundation Models

• AM-RADIO distills high-quality CLIP [51] via OpenCLIP, DINOv2, and SAM, rather than focusing on one teacher.Prior approaches include self-distillation into smaller variants and distillation from a CLIP teacher.Source: §2 · p. 3
• Unlike concurrent work merging CLIP into pretrained SAM via distillation, AM-RADIO adds DINOv2, uses straightforward feature matching, and permits a student architecture distinct from any teacher.The student need not be pretrained.Source: §2 · p. 3

3. Knowledge Agglomeration

• Multi-teacher distillation trains a vision foundation model from scratch that agglomerates the distinctive attributes of its teachers.The framework is illustrated in Figure 2.Source: §3 · p. 3

3.1. Overview

• CLIP [51], DINOv2, and SAM contribute complementary strengths, and training treats datasets as image sources without supplemental ground-truth guidance.Evaluation spans ImageNet k-NN and zero-shot accuracy, ADE20K and Pascal VOC linear-probe mIoU, LLaVA-1.5 tasks, and SAM-style COCO instance segmentation.Source: §3.1 · pp. 3–4

3.2. Adaptor Heads

• Each teacher uses two simple two-layer MLP adaptor heads, one for summary vectors and one for spatial features.Each head includes LayerNorm and GELU; its intermediate dimension is the maximum teacher embedding dimension, and its output matches the corresponding teacher.Source: §3.2 · p. 4

3.3. Distillation Dataset Choice

• ImageNet-1K training yields the highest image-classification metrics, but DataComp-1B is chosen to avoid directly training on the zero-shot evaluation domain.Source: §3.3 · p. 4

3.4. Loss Formulation

• Cosine distance outperformed L1, MSE, and Smooth-L1 for summary matching, while spatial-feature supervision improved dense-task performance and overall model quality.Spatial matching combines cosine similarity with smooth L1 to match teacher-vector magnitudes for drop-in compatibility; the chosen weights are α = 0.9 and β = 0.1.Source: §3.4 · pp. 4–5

3.4.1 Loss Balancing

• Naive loss balancing is used throughout: more complex schemes offer little advantage, and AMTML-KD collapses weight onto CLIP and performs worse.AdaLoss gives the best ImageNet-task results but the worst ADE20K result among the compared balancing methods.Source: §3.4.1 · p. 5, Table 4 · p. 5

4. Implementation Details

• Training used AdamW, batch size 1,024, a 0.001 base learning rate, cosine annealing, and 600k steps (614M examples) on DataComp-1B in two stages.The best student used DFN CLIP ViT-H/14 at 378px, OpenAI CLIP ViT-L/14 at 336px, DINOv2 ViT-g/14 at 224px, and SAM ViTDet-H at 1024px.Source: §4 · p. 5
• Teacher groups were partitioned across GPUs by batch size and student resolution; SAM training used 64 GPUs and an effective batch size of 1,152.CLIP+DINOv2-only training used 32 GPUs and batch size 1,024.Source: §4 · p. 5
• Cropped Position Embedding (CPE) lets ViTs handle arbitrary resolutions and regularizes position embeddings; a 224²-trained student generalized to 518², outperforming non-CPE and DINOv2-g.CPE also improved semantic-segmentation linear-probing mIOU, with negligible drop in summary metrics at 224 resolution.Source: §4 · p. 5, Table 5 · p. 6, Figure 3 · p. 6
• For high-resolution ViT students, variable window-size training augmentation reduces SAM-training compute and enables inference throughput scaling; spectral reparametrization [73] and 0.02 weight decay address instability.The stabilization measures were applied because high-resolution training was unstable and attention entropy collapsed.Source: §4 · p. 6
• When student and teacher feature grids differ, outputs are bilinearly interpolated to the larger resolution; students use separate CLS tokens because average pooling harms feature loss despite improving summary loss.The choice prioritizes feature loss, which the passage describes as more important.Source: §4 · p. 6, Table 6 · p. 6

5.1. Efficient Students

• At 10 times faster with a 20% improvement over teacher models, E-RADIO leads the efficiency results; ViT-H/16 matches teacher speed and outperforms them on 6 of 9 tasks.The full ViT-H/16 model is reported as fast as the teachers, while E-RADIO’s efficiency comparison uses a 224px resolution and window size of 7.Source: Figure 4 · p. 7, §5.2 · p. 7
• E-RADIO combines YOLOv8 C2f convolution blocks with windowed transformer stages and 16x output stride, improving dense tasks while being only 10% slower.Its design adds local and downsampled global windowed attention, then upsamples the final feature maps and adds them to third-stage features.Source: §5.1 · p. 7

5.2. Comparison with teachers

• ERADIO-L is significantly faster than all ViT models and outperforms MetaCLIP on most metrics at matched throughput, while enabling zero-shot capability absent in DINOv2 and SAM.The paper also reports that DINOv2 provides important properties for ADE20k and VOC dense tasks.Source: §5.2 · p. 7

5.3. Semantic Segmentation Linear Probing

• Linear probing evaluates frozen teacher and student features on ADE20k and PascalVOC2012 with an MMSeg-trained linear head and mIoU.Crop sizes are 512 for RADIO, 518 for DINOv2, and native resolution for other models; evaluation uses slide mode.Source: §5.3 · p. 8

5.4. Visual Question Answering

• RADIO’s flexible input resolution reduces LLaVA visual tokens by 20%: 462 tokens on average versus 576 for 336px patch-14 encoders.Images are resized with the long edge at 432px and padded to the nearest multiple of the patch size.Source: §5.4 · p. 8

5.5. 3D Awareness Probing

• RADIO performs on par with the larger DINOv2-g-14-reg on depth, surface normals, and multi-view keypoint correspondence, and significantly better than other comparably sized teachers.The models are evaluated on NAVI across monocular depth, surface normals, and multi-view correspondence tasks.Source: §5.5 · p. 8

6. Conclusion and Key Insights

• Multi-teacher distillation unites complementary VFM strengths—CLIP language grounding, DINOv2 dense correspondences, and SAM segmentation—and often outperforms teachers; better teachers also yield better students.Source: §6 · p. 8
• Full feature distillation brings an 18% relative improvement on ADE20K dense image understanding tasks.Source: §6 · p. 8
• DINOv2 outperforms SAM on zero- and few-shot tasks: ADE20K linear-probing segmentation is 1.7x better, and ImageNet1k k-NN is 4x better.SAM detects edges and segments objects well but performs poorly at high-level object description and combining multiple-object semantics.Source: §6 · p. 8
• Because most model designs are overly tailored to supervised ImageNet1K training and scale poorly to VFM settings, the authors designed E-RADIO, a hybrid CNN-Transformer backbone that improves the Pareto frontier.Source: §6 · pp. 8–9
• RADIO produces high-resolution, low-noise features, but its DINOv2-head features abruptly switch modes at 720px, possibly reflecting partitioned CLIP+DINO and SAM training.The authors intend to address this apparent latent low- versus high-resolution mode in future work.Source: §6 · p. 8, Figure 5 · p. 9

• E-RADIO comprises a stem, two YOLOv8 convolutional blocks, and two transformer blocks with multi-resolution windowed self-attention.Source: Figure 6 · p. 14
• A separate figure presents E-RADIO’s multi-resolution attention.Source: Figure 7 · p. 14

A. E-RADIO architecture details

• The first two E-RADIO stages use YOLOv8 C2f convolutions, while the final two use Transformers with windowed attention and multi-resolution attention (MRA).The architecture is illustrated in Figure 6.Source: A. E-RADIO architecture details · p. 14
• Every E-RADIO stage except the last is followed by a downsampling block implemented as a 3x3 stride-2 convolution and batch normalization.Source: A. E-RADIO architecture details · p. 14

A.1. Multi-Resolution Attention

• Multi-resolution attention addresses communication between windows without SWIN’s non-negligible window-shifting compute cost.The approach reuses windowed attention while avoiding window shifting.Source: §A.1 · p. 14
• Each transformer layer applies local windowed attention, optionally subsampling features by convolution and then restoring their original resolution by deconvolution.Without subsampling, this is standard windowed attention; FasterViT2 interleaves normal attention with attention subsampled at ratio 2.Source: §A.1 · p. 14

A.2. Configurations

• Models share the same configuration apart from embedding dimension, which scales with model size; inputs are 224 pixels and the stem uses two stride-2 3x3 convolutions.The architecture has two convolutional stages followed by two transformer stages; the first two stages use three C2f layers each.Source: §A.2 · p. 14
• Variants use embedding dimensions XT 64, T 80, S 96, B 128, and L 192; outputs are 14x14, formed by upsampling stage 4 and adding stage 3 features.Stages 3 and 4 use five multi-resolution or windowed attention blocks, respectively, with window size 7; XT and T use [1] [5] stage layers.Source: §A.2 · p. 15

B. PCA Visualizations

• PCA maps spatial features to RGB; while most models accept only square inputs at fixed resolutions, DINOv2 and RADIO also support arbitrary resolutions and aspect ratios.DINOv2 and RADIO are visualized in both square-input and flexible-input settings.Source: B. PCA Visualizations · p. 15

C. ViTDet Augmentation

• ViTDet augmentation reorders patches so contiguous chunks form windows, exploiting transformer permutation invariance after positional encodings to switch between windowed and global attention.The procedure absorbs windows into the batch dimension for windowed attention and restores the original shape for global attention.Source: C. ViTDet Augmentation · pp. 18–19
• The ViTDet procedure samples a window size and alternates windowed and global attention, while always applying global attention in the final transformer layer.After processing, patches are returned to their input order.Source: C. ViTDet Augmentation · p. 19

D. Comparison with SAM-CLIP [60]

• The authors report overlapping SAM-CLIP metrics for zero-shot ImageNet-1k and ADE20k linear-probe segmentation, but say differences prevent concluding which model is superior.They argue that DINOv2 performs better than SAM on ADE20k linear probing and infer its inclusion likely explains their higher quality on that metric.Source: D. Comparison with SAM-CLIP [60] · p. 19

E.1. Uncertainty

• Following, the base formulation weights each task loss by 1/(2σ_k^2) and adds log σ_k.Source: §E.1 · p. 19
• The student predicts b_k = log σ_k^2 to avoid division by zero and regress unconstrained scalar values.Source: §E.1 · p. 20
• The modified uncertainty loss replaces manual λ scalars with learned weights and adds a penalty for large uncertainties.It is given as L(x) = Σ_k λ_k L_k(x) + b_k/2, with λ_k = e^-b_k/2.Source: §E.1 · p. 20
• For each teacher, a learned function predicts separate balance parameters for summary and feature-vector losses.The resulting loss sums teacher- and output-specific weighted losses with a ψ penalty on the predicted parameters.Source: §E.1 · p. 20
• The modified formulation uses softplus ψ on uncertainty parameters to preserve the prior behavior while improving stability for loss weights above 1.The authors note ψ(x) ≈ x for large x, ψ′(x) ≈ 1/2 near zero, and ψ′(x) → 0 when x < 0; its positive range keeps the loss positive.Source: §E.1 · p. 20

E.2. AdaLoss

• In addition to uncertainty auto-balancing, the authors explored AdaLoss.Source: §E.2 · p. 20

F. Visual Question Answering Samples

• Figures 9–13 show GQA and TextVQA question-answer samples from LLaVA models using various image encoders, with answers colored by whether they match ground truth.The accompanying examples include questions about a book’s author or a box’s company, and about coin weight or beer flavor.Source: F. Visual Question Answering Samples · p. 20, Figure 9 · p. 22, Figure 10 · p. 23
• In the GQA helmet example, attention shifts toward the relevant image region across language-model layers, and the model answers “Blue.”The visualization averages visual-token attention over heads at each layer and overlays heat maps on the image.Source: Figure 8 · p. 21
• TextVQA examples include a coin-weight question explicitly noted as not requiring reading text in the image.Other displayed questions ask for a book’s author or a company on a box.Source: F. Visual Question Answering Samples · p. 26