Training a Student Expert via Semi-Supervised Foundation Model Distillation

Motivation

• VFMs are costly to deploy because their scale creates substantial inference overhead under latency, memory, and energy constraints.This issue is especially relevant to resource-constrained real-world applications.Source: §1 · p. 1
• Generic foundation-model training can underperform on specialized downstream tasks, motivating efficient models tailored to specific domains.The introduction emphasizes task and domain specialization as a recurring limitation for instance segmentation.Source: §1 · p. 1

Contributions

• The proposed instance-aware pixel-wise contrastive loss combines mask and class predictions to identify informative negatives and strengthen inter-instance separation.It is designed for dense prediction settings.Source: §1 · p. 2
• The proposed three-stage SSKD framework adapts the foundation teacher, distills it into a compact student, and trains compact student experts.The stages include self-training with contrastive calibration and distillation using a unified objective.Source: §1 · p. 2

Vision Foundation Models

• VFMs advance computer vision through large-scale pre-training and transferability across diverse tasks, while remaining computationally demanding for resource-constrained deployment.Recent efforts also combine complementary VFMs to extend their capabilities.Source: §2 · p. 2

Contrastive Knowledge Distillation

• Contrastive knowledge distillation aligns teacher and student representations through contrastive objectives, with dense-prediction extensions operating at ROI or pixel level.These extensions have been explored for object detection and semantic segmentation.Source: §2 · p. 2
• Unlike explicit teacher–student contrastive matching, the method uses an instance-aware pixel-wise contrastive objective as a self-supervised signal.Source: §2 · p. 2

Pipeline

• The three stages are teacher adaptation, knowledge transfer, and student refinement, targeting comparable or better accuracy with substantially lower computational cost.Teacher adaptation uses labeled and pseudo-labeled data; transfer freezes the adapted teacher; refinement uses labeled data only.Source: §3.1 · pp. 2–3, Figure 1 · p. 3

Teacher Adaptation

• Teacher adaptation combines self-training with pixel-wise contrastive regularization to sharpen mask boundaries and improve pseudo-label quality.The adapted teacher generates pseudo-labels for subsequent training on labeled and unlabeled data.Source: §3.1 · p. 3, §3.3 · p. 4

Student Refinement

• Student refinement fine-tunes the distilled student on labeled data only to reduce residual pseudo-label drift and sharpen target-domain decision boundaries.Excessively large λpxl can increase training cost and weaken intra-instance cohesion by overemphasizing separation.Source: §3.1 · p. 3, §3.2 · p. 4, §3.3 · p. 5

Contrastive Calibration

• Instance-aware negative sampling fuses mask and class predictions to prioritize pixels likely to belong to different instances without quadratic pairwise comparisons.Dissimilar pixels are sampled proportionally as negatives for the NT-Xent pixel-wise contrastive loss.Source: §3.2 · p. 4
• Under a negative-sampling guarantee with different-instance probability p > 0.5, one contrastive update increases the expected inter-instance margin Δemp.The expectation holds even with imperfect pseudo-labels when negatives follow the instance-aware strategy.Source: §3.2 · p. 4

Knowledge Transfer

• The unified distillation objective combines supervised, pseudo-label, and pixel-wise contrastive terms across labeled and unlabeled data.The coefficients λsemi and λpxl balance pseudo-label transfer and contrastive regularization.Source: §3.3 · pp. 4–5

Main results

• On Cityscapes and ADE20K, pixel-level contrastive adaptation improves teacher maskAP from 29.8 to 30.5 and 14.8 to 15.2, respectively.The corresponding maskAP50 gains are 54.9→56.6 on Cityscapes and 23.7→24.5 on ADE20K.Source: §4.2 · p. 5

Contrastive-loss validation

• Throughout training, the true-negative sampling probability remains above 0.9, while empirical margin increases approximately linearly with λpxl.This behavior is consistent with Proposition 3.1.Source: §4.3 · p. 6

Loss ablations

• 32.2 maskAP is achieved when supervised, pseudo-label, and pixel-level contrastive losses are combined, versus 21.1 with supervised loss alone.Adding pseudo-label loss raises performance to 30.7 maskAP before contrastive regularization reaches 32.2.Source: §4.4 · p. 6

Negative-sampling ablations

• 32.2 maskAP and 56.5 maskAP50 are obtained by fusing mask and class predictions for negative sampling, outperforming uniform, mask-only, and class-only variants.The fusion sampler indicates that mask and class information provide complementary sampling signals.Source: §4.4 · p. 7

Conclusion

• The pipeline combines self-training, instance-aware pixel-wise contrastive learning, and final supervised refinement to transfer knowledge from large vision foundation models into compact student experts.It targets efficient low-label adaptation of strong foundation models.Source: §5 · p. 8
• +3.4 maskAP on Cityscapes and +1.5 maskAP on ADE20K are achieved over adapted teachers by students approximately 11× smaller than the teachers.These results demonstrate compact students can surpass adapted foundation-model teachers.Source: §5 · p. 8
• Pixel-level contrastive regularization improves pseudo-label quality and enables efficient low-label adaptation of strong foundation models.The contrastive signal is central to leveraging limited labeled data with abundant unlabeled data.Source: §5 · p. 8
• Under mild assumptions, theoretical analysis shows the negative sampling strategy increases the expected inter-instance margin.The analysis provides theoretical support for the proposed contrastive sampling strategy.Source: §5 · p. 8

• The supplementary material includes dataset statistics, full hyperparameter settings, a formal proof, extended training protocols, and additional ablation studies.Source: Supplementary Material · p. 9

7. Hyperparameters

• Key teacher and student hyperparameters are summarized in Table 7, with results averaged over three independent runs using different random seeds.This section focuses on implementation settings and evaluation averaging.Source: §7 · p. 9

Proof setup

• The proof defines unit-norm embeddings for an anchor pixel, its positive, and R negatives, then introduces the pixel-wise contrastive loss.The setup uses an anchor-positive-negative construction for the subsequent gradient analysis.Source: §8 · p. 9

Gradient update

• A gradient computation characterizes the loss update, followed by one gradient descent step with step size λpxl.These expressions establish how the contrastive objective changes the relevant similarities.Source: §8 · p. 9

Margin increase

• ε: one update on Lpxl increases the expected inter-instance margin by ε.The proof argues that inter-instance contributions vanish in expectation while intra-instance cases determine the margin increase.Source: §8 · p. 9

Near-orthogonality assumption

• The approximation ⟨z+, z−⟩≈0 is justified by converged negative weights and concentration of random high-dimensional unit-vector inner products near zero.Contrastive training further pushes negative similarities into a tight, small-magnitude distribution, supporting the approximation up to O(1/…).Source: §8 · p. 9

Teacher training schedule

• Teachers undergo 1k iterations of fine-tuning on the labeled set before pseudo-label self-training.The initial fine-tuning precedes all teacher self-training.Source: §9 · p. 9
• Teachers then perform 5k iterations of self-training with pseudo-labels.This stage follows labeled-set fine-tuning.Source: §9 · p. 9

Student training schedule

• 90k student-training iterations are used on Cityscapes, compared with 80k on ADE20K.The Cityscapes schedule follows prior works, while ADE20K uses a shorter schedule.Source: §9 · p. 9
• Both datasets receive an additional 2k iterations of supervised finetuning.This supervised phase follows the main student-training schedule.Source: §9 · p. 9

Contrastive Hyperparameters

• The adopted Cityscapes settings are λpxl = 0.2, K = 256, and T = 0.2, with T = 0.2 consistently best for teacher and student models.Increasing λpxl from 0 to 0.2 steadily improves performance, while larger-negative sampling gains are marginal relative to overhead.Source: §10.1 · pp. 9–10, Table 8 · p. 10

Loss Variants

• Replacing asymmetric InfoNCE with a marginbased hinge loss preserves maskAP at 32.2% and raises maskAP50 by +0.6, but takes 1.6× longer to train.The hinge variant uses margin = 0.2.Source: §10.3 · p. 10, Table 10 · p. 10

Debias Score Formulation

• The debias-score ablation evaluates original mask-and-class confidence fusion, squared scores, and square-root scores to respectively preserve, amplify, or temper high-confidence negative bias.These formulations are evaluated as alternatives for sdeb (§3.2).Source: §10.4 · p. 10, Table 11 · p. 10

Negative Sampling

• Memory-bank sampling with 10 k pixel embeddings improves performance by +0.5 maskAP and +0.8 maskAP50 over in-batch sampling, while requiring approximately 2.2× longer training.The additional cost comes from maintaining and querying the memory bank.Source: §10.5 · pp. 10–11, Table 13 · p. 11

Teacher Adaptation

• Distilling from the adapted teacher T2 instead of the zero-shot VFM teacher T1 yields a +8.4 AP improvement for the student.Teacher adaptation ablations compare finetuning only, self-training, and self-training with the proposed contrastive loss.Source: §10.2 · p. 10, Table 9 · p. 10, Table 12 · p. 10

Backbone Comparison

• The proposed DINOv2-S student is compared with Guided Distillation baselines using ResNet-50, DINOv2-B, and DINOv2-L teacher backbones.The comparison is reported as an extended backbone study.Source: §10.7 · p. 11, Table 14 · p. 11