PaperlayerView exact revision on arXiv

Source-linked AI summary

Training a Student Expert via Semi-Supervised Foundation Model Distillation

Pardis Taghavi, Tian Liu, Renjie Li, Reza Langari, Zhengzhong Tu

arXiv:2604.03841v1cs.CV

TL;DR

Deploying vision foundation models for instance segmentation is costly, while adapting them requires expensive pixel-level labels. This paper presents semi-supervised distillation into compact students, which surpass adapted teachers by +3.4 maskAP on Cityscapes and +1.5 maskAP on ADE20K despite being approximately 11× smaller.

  • Problem

    Vision foundation models are costly to deploy and insufficiently specialized, while instance segmentation requires expensive pixel-level annotations.

  • Method

    A three-stage semi-supervised distillation framework adapts foundation teachers, transfers knowledge to compact students, and refines students using instance-aware pixel-wise contrastive learning.

  • Results

    +3.4 maskAP on Cityscapes and +1.5 maskAP on ADE20K over adapted teachers are achieved by a student approximately 11× smaller than the teacher.

  • Takeaways & Limitations

    Pixel-level contrastive regularization improves pseudo-label quality and supports efficient low-label adaptation of vision foundation models.

  • Takeaways & Limitations

    Increasing the pixel-contrastive loss weight raises training cost and can trade inter-instance separation against intra-instance cohesion.

Abstract

from arXiv · show

Foundation models deliver strong perception but are often too computationally heavy to deploy, and adapting them typically requires costly annotations. We introduce a semi-supervised knowledge distillation (SSKD) framework that compresses pre-trained vision foundation models (VFMs) into compact experts using limited labeled and abundant unlabeled data, and instantiate it for instance segmentation where per-pixel labels are particularly expensive. The framework unfolds in three stages: (1) domain adaptation of the VFM(s) via self-training with contrastive calibration, (2) knowledge transfer through a unified multi-objective loss, and (3) student refinement to mitigate residual pseudo-label bias. Central to our approach is an instance-aware pixel-wise contrastive loss that fuses mask and class scores to extract informative negatives and enforce clear inter-instance margins. By maintaining this contrastive signal across both adaptation and distillation, we align teacher and student embeddings and more effectively leverage unlabeled images. On Cityscapes and ADE20K, our $\approx 11\times$ smaller student improves over its zero-shot VFM teacher(s) by +11.9 and +8.6 AP, surpasses adapted teacher(s) by +3.4 and +1.5 AP, and outperforms state-of-the-art SSKD methods on benchmarks.

1. Introduction

The introduction motivates compact, specialized instance-segmentation models by highlighting foundation models’ deployment cost and limited downstream specialization. It presents an SSKD framework with an instance-aware contrastive loss and three-stage teacher-to-student training.

  • Motivation: VFMs are costly to deploy because their scale creates substantial inference overhead under latency, memory, and energy constraints.This issue is especially relevant to resource-constrained real-world applications.
  • Motivation: Generic foundation-model training can underperform on specialized downstream tasks, motivating efficient models tailored to specific domains.The introduction emphasizes task and domain specialization as a recurring limitation for instance segmentation.
  • Contributions: The proposed instance-aware pixel-wise contrastive loss combines mask and class predictions to identify informative negatives and strengthen inter-instance separation.It is designed for dense prediction settings.
  • Contributions: The proposed three-stage SSKD framework adapts the foundation teacher, distills it into a compact student, and trains compact student experts.The stages include self-training with contrastive calibration and distillation using a unified objective.

2. Related work

Related work highlights VFMs’ strong transferability and open-set recognition alongside their computational cost, while contrastive distillation aligns teacher–student representations for dense prediction. The proposed method instead uses an instance-aware pixel-wise contrastive objective as self-supervision rather than explicit teacher–student contrastive matching.

  • Vision Foundation Models: VFMs advance computer vision through large-scale pre-training and transferability across diverse tasks, while remaining computationally demanding for resource-constrained deployment.Recent efforts also combine complementary VFMs to extend their capabilities.
  • Contrastive Knowledge Distillation: Contrastive knowledge distillation aligns teacher and student representations through contrastive objectives, with dense-prediction extensions operating at ROI or pixel level.These extensions have been explored for object detection and semantic segmentation.
  • Contrastive Knowledge Distillation: Unlike explicit teacher–student contrastive matching, the method uses an instance-aware pixel-wise contrastive objective as a self-supervised signal.

3. Method

The method presents a three-stage semi-supervised distillation pipeline that adapts a pretrained VFM, transfers knowledge to a lightweight student, and refines it using labeled data. Pixel-wise contrastive calibration and instance-aware negative sampling exploit unlabeled images while reducing pseudo-label bias.

  • Pipeline: The three stages are teacher adaptation, knowledge transfer, and student refinement, targeting comparable or better accuracy with substantially lower computational cost.Teacher adaptation uses labeled and pseudo-labeled data; transfer freezes the adapted teacher; refinement uses labeled data only.
  • Teacher Adaptation: Teacher adaptation combines self-training with pixel-wise contrastive regularization to sharpen mask boundaries and improve pseudo-label quality.The adapted teacher generates pseudo-labels for subsequent training on labeled and unlabeled data.
  • Student Refinement: Student refinement fine-tunes the distilled student on labeled data only to reduce residual pseudo-label drift and sharpen target-domain decision boundaries.Excessively large λpxl can increase training cost and weaken intra-instance cohesion by overemphasizing separation.
  • Contrastive Calibration: Instance-aware negative sampling fuses mask and class predictions to prioritize pixels likely to belong to different instances without quadratic pairwise comparisons.Dissimilar pixels are sampled proportionally as negatives for the NT-Xent pixel-wise contrastive loss.
  • Contrastive Calibration: Under a negative-sampling guarantee with different-instance probability p > 0.5, one contrastive update increases the expected inter-instance margin Δemp.The expectation holds even with imperfect pseudo-labels when negatives follow the instance-aware strategy.
  • Knowledge Transfer: The unified distillation objective combines supervised, pseudo-label, and pixel-wise contrastive terms across labeled and unlabeled data.The coefficients λsemi and λpxl balance pseudo-label transfer and contrastive regularization.

4. Experiments

Experiments on Cityscapes and ADE20K show that contrastive teacher adaptation, pseudo-label distillation, and student refinement contribute complementary gains. Ablations further validate the sampling strategy and the framework’s three-stage design.

  • Main results: On Cityscapes and ADE20K, pixel-level contrastive adaptation improves teacher maskAP from 29.8 to 30.5 and 14.8 to 15.2, respectively.The corresponding maskAP50 gains are 54.9→56.6 on Cityscapes and 23.7→24.5 on ADE20K.
  • Contrastive-loss validation: Throughout training, the true-negative sampling probability remains above 0.9, while empirical margin increases approximately linearly with λpxl.This behavior is consistent with Proposition 3.1.
  • Loss ablations: 32.2 maskAP is achieved when supervised, pseudo-label, and pixel-level contrastive losses are combined, versus 21.1 with supervised loss alone.Adding pseudo-label loss raises performance to 30.7 maskAP before contrastive regularization reaches 32.2.
  • Negative-sampling ablations: 32.2 maskAP and 56.5 maskAP50 are obtained by fusing mask and class predictions for negative sampling, outperforming uniform, mask-only, and class-only variants.The fusion sampler indicates that mask and class information provide complementary sampling signals.

5. Conclusion

The paper presents a semi-supervised distillation pipeline combining self-training, instance-aware pixel-wise contrastive learning, and supervised refinement to produce compact student experts that outperform adapted teachers. Its analysis supports contrastive negative sampling for improving pseudo-label quality and increasing expected inter-instance margins under mild assumptions.

  • Conclusion: The pipeline combines self-training, instance-aware pixel-wise contrastive learning, and final supervised refinement to transfer knowledge from large vision foundation models into compact student experts.It targets efficient low-label adaptation of strong foundation models.
  • Conclusion: +3.4 maskAP on Cityscapes and +1.5 maskAP on ADE20K are achieved over adapted teachers by students approximately 11× smaller than the teachers.These results demonstrate compact students can surpass adapted foundation-model teachers.
  • Conclusion: Pixel-level contrastive regularization improves pseudo-label quality and enables efficient low-label adaptation of strong foundation models.The contrastive signal is central to leveraging limited labeled data with abundant unlabeled data.
  • Conclusion: Under mild assumptions, theoretical analysis shows the negative sampling strategy increases the expected inter-instance margin.The analysis provides theoretical support for the proposed contrastive sampling strategy.

Supplementary Material

The supplementary material provides additional methodological and experimental details supporting the main paper, including datasets, hyperparameters, proofs, training protocols, and ablations.

  • The supplementary material includes dataset statistics, full hyperparameter settings, a formal proof, extended training protocols, and additional ablation studies.

7. Hyperparameters

The section summarizes key teacher and student hyperparameters and reports results averaged over three independent runs with different random seeds.

  • 7. Hyperparameters: Key teacher and student hyperparameters are summarized in Table 7, with results averaged over three independent runs using different random seeds.This section focuses on implementation settings and evaluation averaging.

8. Proof Sketch of Proposition 3.1

The proof analyzes unit-norm anchor, positive, and negative pixel embeddings under the pixel-wise contrastive loss, showing that one gradient step increases the expected inter-instance margin by ε under near-orthogonality assumptions.

  • Proof setup: The proof defines unit-norm embeddings for an anchor pixel, its positive, and R negatives, then introduces the pixel-wise contrastive loss.The setup uses an anchor-positive-negative construction for the subsequent gradient analysis.
  • Gradient update: A gradient computation characterizes the loss update, followed by one gradient descent step with step size λpxl.These expressions establish how the contrastive objective changes the relevant similarities.
  • Margin increase: ε: one update on Lpxl increases the expected inter-instance margin by ε.The proof argues that inter-instance contributions vanish in expectation while intra-instance cases determine the margin increase.
  • Near-orthogonality assumption: The approximation ⟨z+, z−⟩≈0 is justified by converged negative weights and concentration of random high-dimensional unit-vector inner products near zero.Contrastive training further pushes negative similarities into a tight, small-magnitude distribution, supporting the approximation up to O(1/…).

9. More Training Details

Teacher models are fine-tuned on labeled data, adapted through pseudo-label self-training, and followed by dataset-specific student training and supervised refinement. The schedules use 90k student iterations on Cityscapes and 80k on ADE20K, plus 2k supervised finetuning iterations for both.

  • Teacher training schedule: Teachers undergo 1k iterations of fine-tuning on the labeled set before pseudo-label self-training.The initial fine-tuning precedes all teacher self-training.
  • Teacher training schedule: Teachers then perform 5k iterations of self-training with pseudo-labels.This stage follows labeled-set fine-tuning.
  • Student training schedule: 90k student-training iterations are used on Cityscapes, compared with 80k on ADE20K.The Cityscapes schedule follows prior works, while ADE20K uses a shorter schedule.
  • Student training schedule: Both datasets receive an additional 2k iterations of supervised finetuning.This supervised phase follows the main student-training schedule.

10. Additional Ablation Studies

The ablations identify robust contrastive-learning settings, quantify trade-offs among loss variants and negative-sampling scopes, and show that teacher adaptation substantially strengthens student distillation. Additional studies compare debias-score formulations and teacher backbones.

  • Contrastive Hyperparameters: The adopted Cityscapes settings are λpxl = 0.2, K = 256, and T = 0.2, with T = 0.2 consistently best for teacher and student models.Increasing λpxl from 0 to 0.2 steadily improves performance, while larger-negative sampling gains are marginal relative to overhead.
  • Loss Variants: Replacing asymmetric InfoNCE with a marginbased hinge loss preserves maskAP at 32.2% and raises maskAP50 by +0.6, but takes 1.6× longer to train.The hinge variant uses margin = 0.2.
  • Debias Score Formulation: The debias-score ablation evaluates original mask-and-class confidence fusion, squared scores, and square-root scores to respectively preserve, amplify, or temper high-confidence negative bias.These formulations are evaluated as alternatives for sdeb (§3.2).
  • Negative Sampling: Memory-bank sampling with 10 k pixel embeddings improves performance by +0.5 maskAP and +0.8 maskAP50 over in-batch sampling, while requiring approximately 2.2× longer training.The additional cost comes from maintaining and querying the memory bank.
  • Teacher Adaptation: Distilling from the adapted teacher T2 instead of the zero-shot VFM teacher T1 yields a +8.4 AP improvement for the student.Teacher adaptation ablations compare finetuning only, self-training, and self-training with the proposed contrastive loss.
  • Backbone Comparison: The proposed DINOv2-S student is compared with Guided Distillation baselines using ResNet-50, DINOv2-B, and DINOv2-L teacher backbones.The comparison is reported as an extended backbone study.
Loading 2604.03841v1…