Source-linked AI summary
GRAFT: Growing Agglomerative Foundation Models via Continual Teacher Distillation
Zhenghao Zhao, Chi Zhang, Qingshuang Chen, Yelin Kim
TL;DR
Existing agglomerative models rely on a fixed teacher set, so adding a capability requires joint re-distillation over the enlarged set. GRAFT incrementally distills from the previous student and an incoming teacher, using distinct readouts and relational alignment for heterogeneous representations. The resulting backbone spans five capability families while retaining prior capabilities without full joint re-distillation.
Problem
Existing agglomerative models assume a fixed teacher set, so incorporating a new foundation model generally requires repeating joint distillation over the enlarged set.
Source: §1 · pp. 1–2Method
GRAFT warm-starts from the previous student, freezes it as a retention anchor, and jointly distills from it and the incoming teacher, with teacher-specific readouts and relational image-text alignment.
Source: §1 · p. 2, §3.2 · p. 5Results
GRAFT spans five capability families and acquires new capabilities while retaining previously distilled ones without full joint re-distillation over the accumulated teacher set.
Source: §1 · p. 3Takeaways & Limitations
Each incoming teacher can be added through a single incremental distillation stage rather than revisiting all previously integrated teachers.
Source: §5 · p. 9Table 1 is not a controlled ranking because baselines differ in teacher scale and training-corpus size and composition.
Source: §4.2 · p. 8
Abstract
from arXiv · showhide
Vision foundation models such as DINOv2, SigLIP2, and MASt3R develop complementary capabilities from different pretraining objectives, yet their knowledge remains distributed across separate, specialized models. Multi-teacher knowledge distillation offers a path toward consolidating these capabilities into a single agglomerative backbone, but existing approaches assume a fixed set of teachers, and incorporating a new teacher requires repeating expensive joint distillation over the entire teacher set. We introduce GRAFT, a continual multi-teacher distillation framework that enables a unified backbone to progressively acquire capabilities from an open-ended sequence of foundation models. When a new teacher arrives, GRAFT treats the previously distilled model as a teacher for preserving learned capabilities, while the current student jointly learns from both the previous model and the incoming teacher. Furthermore, to reconcile the incompatible representation geometries of heterogeneous teachers, we introduce Teacher Specific Readout Tokens, which grant each teacher an independent read-out of the shared encoder, together with Geometry Agnostic Relational Loss that aligns a vision-language teacher by matching image-text similarity structures rather than raw feature values. We provide GRAFT model, which is a single, continually extensible backbone that unifies five domains, including image understanding, 2D dense prediction, 3D human pose estimation, 3D vision, and vision-language, delivering strong performance across all of them while acquiring each new capability at the cost of a single distillation rather than a full re-distillation.
1 INTRODUCTION
Foundation models develop complementary capabilities, but existing agglomerative models assume a fixed teacher set, making later additions require renewed joint distillation. GRAFT incrementally adds capabilities while preserving prior ones in a shared backbone, using teacher-specific readouts and a relational objective to accommodate heterogeneous representations.
Motivation
- Existing agglomerative models assume a fixed teacher set, so adding a teacher generally requires joint distillation again over the enlarged set.The student must be distilled from scratch against all teachers, making each addition a full distillation run rather than one teacher’s worth of work.Source: §1 · pp. 1–2
GRAFT
- GRAFT keeps prior capabilities by using the previous student as a frozen anchor while jointly distilling from it and the incoming teacher.Each step needs only those two reference models, regardless of the number of capabilities already accumulated.Source: §1 · p. 2
- Teacher Specific Readout Tokens give each teacher an independent readout, while Geometry Agnostic Relational Loss transfers vision-language knowledge by matching image-text similarity structures.The relational loss avoids directly matching feature values, and the readouts let heterogeneous teachers extract different global summaries from the shared encoder.Source: §1 · pp. 2–3
Contribution
- The paper formalizes continual multi-teacher distillation for vision foundation models, where heterogeneous teachers arrive over time and prior teachers are unavailable.GRAFT is presented as turning capability consolidation into a continual process.Source: §1 · p. 2
Results
- GRAFT unifies five capability families—image understanding, 2D dense prediction, 3D human pose estimation, 3D vision, and vision-language—in one model.It acquires new capabilities while retaining previously distilled ones without full joint re-distillation over the accumulated teacher set.Source: §1 · p. 3
2 RELATED WORK
Vision foundation models develop complementary strengths from different pretraining objectives, motivating consolidation into a shared encoder. Prior agglomerative and continual-distillation work informs GRAFT, while existing agglomerative formulations largely optimize over a fixed teacher collection.
2.1 VISION FOUNDATION MODELS
- Different pretraining objectives produce complementary strengths: self-supervised models support spatial tasks, image-text models emphasize semantic alignment, and other models target specialized capabilities.The paper argues that no single representation uniformly dominates across task families.Source: §2.1 · p. 3
2.2 AGGLOMERATIVE MODELS AND MULTI-TEACHER DISTILLATION
- Agglomerative methods consolidate multiple teachers into a shared backbone using teacher-specific heads, projectors, standardization, or separate adaptation paths.AM-RADIO introduced the paradigm; PHI-S, DUNE, and SAK address differing aspects of heterogeneous representations.Source: §2.2 · p. 3
- Related work also extends multi-teacher distillation to vision-language models, proxy datasets, heterogeneous objectives, and domain-specific applications, but generally uses teachers available during distillation.Existing agglomerative methods are designed for a fixed teacher set, so adding a teacher requires revisiting earlier teachers or another joint consolidation stage.Source: §2.2 · p. 4
2.3 CONTINUAL LEARNING
- Continual-learning approaches use regularization, output distillation, or replay to prevent degradation, and SE2D studies sequential teacher distillation without retaining the full teacher history.GRAFT differs by consolidating representation-level capabilities across image understanding, dense prediction, 3D pose and geometry, and vision-language rather than matching a common logit space.Source: §2.3 · p. 4
3 METHOD
Continual multi-teacher distillation extends a student as teachers arrive, even though past teachers are unavailable. GRAFT uses the previous student to preserve earlier task functions while accommodating heterogeneous teacher readouts and objectives.
3.1 PROBLEM DEFINITION: CONTINUAL MULTI-TEACHER DISTILLATION
- Past teachers are unavailable at each round, so the student must acquire the incoming task while retaining all earlier tasks.The learner receives the prior student, an image corpus, and the incoming teacher; performance is evaluated by acquisition and retention.Source: §3.1 · p. 4
3.2 THE GRAFT PIPELINE
- GRAFT warm-starts the active student from the previous student and uses that frozen model as an anchor while distilling from the incoming teacher.The historical student serves as a compact proxy for past teachers, so retention does not require keeping the full earlier teacher set.Source: §3.2 · pp. 5–6, Figure 3 · p. 5
- The retention objective constrains each previous task’s current output to reproduce its corresponding frozen-anchor output, regularizing the student in function space.The complete objective combines this preservation term with acquisition from the incoming teacher; λ controls historical preservation strength.Source: §3.2 · pp. 5–6
3.3 ADDRESSING REPRESENTATION HETEROGENEITY
- A shared global summary can bottleneck heterogeneous teachers, so Teacher Specific Readout Tokens give each task an independent readout alongside shared patch representations.Each task’s projector receives its dedicated contextualized readout token together with common contextualized patch features.Source: §3.3 · p. 6
- For vision-language teachers, Geometry Agnostic Relational Loss matches image-text similarity relationships rather than directly aligning image embeddings.This transfers relative semantic relationships without requiring the student to reproduce the teacher’s absolute feature coordinates.Source: §3.3 · pp. 6–7
4 EXPERIMENTS
Experiments evaluate GRAFT across five capability families and test continual growth, retention, heterogeneous-teacher integration, and qualitative outputs. The results report broad coverage alongside competitive performance, with comparisons across several design choices.
4.1 DATASETS AND METRICS
- The experiments use four ViT-L teachers and a ViT-B/14 student, with datasets spanning teacher domains and standard protocols across the five evaluated capability families.Downstream evaluation includes classification, segmentation and depth, human pose, 3D geometry, and vision-language retrieval.Source: §4.1 · p. 7
4.2 MAIN RESULTS
- GRAFT reports results across all five capability families, leading agglomerative baselines on image understanding and matching or approaching the strongest specialized baseline on the others.It is the only model in Table 1 with a result in every column; the table is not a controlled ranking because teacher scale and training corpora differ.Source: §4.2 · p. 8
- A student built entirely through successive grafts retains earlier capabilities through later additions and reaches performance comparable to the DUNE-warm-started GRAFT.The sequence starts from DINOv2 and adds Multi-HMR, MASt3R, then SigLIP2; the fully grafted model trails slightly on 3D pose and geometry.Source: §4.2 · p. 8
4.3 ABLATIONS
- Comparisons of distillation strategies show that direct distillation acquires vision-language capability but degrades prior dense and geometric capabilities, while GRAFT components improve the acquisition-retention balance.LoRA preserves prior behavior when its parameters are removed but yields limited new capability; LPFT and EWC improve balance but still show interference.Source: §4.3 · pp. 8–9
- A residual MLP projector achieves acquisition and retention comparable to or better than Transformer alternatives, while increasing Transformer depth does not consistently help.The results suggest that projector capacity alone does not resolve the student-teacher representation mismatch.Source: §4.3 · p. 9
- GRAFT shows similar acquisition and retention behavior when the vision-language teacher is SigLIP2 or comparable-scale DFN5B.This comparison suggests the grafting mechanism does not rely on properties unique to SigLIP2.Source: §4.3 · p. 9
4.4 QUALITATIVE RESULTS
- Qualitative comparisons report competitive or superior GRAFT performance against corresponding teachers in depth estimation, semantic segmentation, multi-human reconstruction, and 3D reconstruction.Figure 4 presents the four task comparisons.Source: §4.4 · p. 9, Figure 4 · p. 9
5 CONCLUSION
GRAFT grows an agglomerative vision backbone through continual distillation, using the previous student alongside each incoming teacher. It integrates five capability families and addresses heterogeneous representations with teacher-specific readouts and image-text similarity matching.
5 CONCLUSION
- GRAFT uses a frozen previous student as a retention teacher and distills jointly from it and each incoming foundation model, adding capability in a single incremental stage.This avoids revisiting all previously integrated teachers.Source: §5 · p. 9
- A single ViT-B backbone progressively integrates image understanding, 2D dense prediction, 3D human pose estimation, 3D vision, and vision-language capabilities.Teacher Specific Readout Tokens and Geometry Agnostic Relational Loss accommodate heterogeneous representation geometries, with the latter matching image-text similarity structure rather than direct feature values.Source: §5 · pp. 9–10
A COMPARISON CONDITIONS
The Table 1 comparison is not like-for-like because methods use different teachers and training corpora, which affect reported metrics. Table 3 collects these conditions and notes undisclosed data sizes and differences in reporting.
A COMPARISON CONDITIONS
- Different teacher sets and training corpora make Table 1’s comparison not strictly like-for-like and directly affect reported metrics.Table 3 collects these conditions.Source: A COMPARISON CONDITIONS · p. 14, Table 3 · p. 14
- Table 3 reports teacher scale per model and teacher counts, marks undisclosed entries unavailable, and notes SigLino reports image tokens rather than images.EUPE also trains a from-scratch 1.9B proxy before distillation.Source: Table 3 · p. 14
- GRAFT’s data subtotal counts DataComp-12M and ImageNet-19k but excludes Mapillary and GLDv2 because their dataset sizes are undisclosed.Its teachers are DUNE’s three, inherited through the warm start, plus the grafted vision-language teacher.Source: Table 3 · p. 14
B EXTENDED RESULTS
On map-free relocalization, GRAFT ranks second only to its MASt3R teacher and ahead of dedicated matching and two-view baselines on the reported AUC and reprojection-error metrics. Evaluation substitutes the GRAFT encoder into MASt3R, fine-tunes only the decoder, and keeps the backbone frozen.
B EXTENDED RESULTS
- GRAFT ranks second only to the MASt3R teacher, ahead of dedicated matching and two-view baselines on virtual-correspondence AUC, reprojection error, and relative-pose AUC.The evaluation uses the official leaderboard test split, substitutes the GRAFT encoder into MASt3R, fine-tunes the decoder, and freezes the backbone.Source: B EXTENDED RESULTS · p. 14, Table 4 · p. 14
- The residual gap to the MASt3R teacher concentrates in fine-grained pose precision.The passage attributes this to compressing a ViT-L geometry specialist into a ViT-B trunk shared with four other capabilities and adapting no backbone weights at evaluation.Source: B EXTENDED RESULTS · p. 14
- Table 4 reports map-free visual relocalization on the official leaderboard test split, with AUC and precision as percentages and metric directions marked in the header.Baseline rows are quoted from Sarıyıldız et al. (2025), Table 4.Source: Table 4 · p. 14