PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Lightweight Object Detectors via Multi-Teacher Progressive Distillation

Shengcao Cao, Mengtian Li, James Hays, Deva Ramanan, Yi-Xiong Wang, Liang-Yan Gui

arXiv:2308.09105v1cs.CVcs.LG

TL;DR

Object detection distillation is complicated by structured outputs and teacher-student capacity and architecture gaps. MTPD progressively transfers knowledge through an automatically ordered teacher curriculum. The paper reports state-of-the-art performance on COCO and successful distillation from Transformer teachers to convolutional students.

  • Problem

    Detection distillation must handle variable-length outputs and multi-task heads, while teacher-student capacity and architectural gaps and teacher selection remain challenges.

    Source: §1 · pp. 1–2

  • Method

    MTPD progressively distills a lightweight detector using an automatically arranged sequence of teachers as a curriculum.

    Source: §1 · p. 2, §5 · p. 9

  • Results

    The authors report state-of-the-art performance on COCO and successful distillation from Transformer-based teachers to convolution-based students.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The authors report that distillation improves generalization rather than optimization.

    Source: §1 · p. 2, §5 · p. 9

    The work is mainly empirical and leaves theoretical understanding of representation-based adaptation cost and distillation’s improved generalization for future work.

    Source: §5 · p. 9

Abstract

from arXiv · show

Resource-constrained perception systems such as edge computing and vision-for-robotics require vision models to be both accurate and lightweight in computation and memory usage. While knowledge distillation is a proven strategy to enhance the performance of lightweight classification models, its application to structured outputs like object detection and instance segmentation remains a complicated task, due to the variability in outputs and complex internal network modules involved in the distillation process. In this paper, we propose a simple yet surprisingly effective sequential approach to knowledge distillation that progressively transfers the knowledge of a set of teacher detectors to a given lightweight student. To distill knowledge from a highly accurate but complex teacher model, we construct a sequence of teachers to help the student gradually adapt. Our progressive strategy can be easily combined with existing detection distillation mechanisms to consistently maximize student performance in various settings. To the best of our knowledge, we are the first to successfully distill knowledge from Transformer-based teacher detectors to convolution-based students, and unprecedentedly boost the performance of ResNet-50 based RetinaNet from 36.5% to 42.0% AP and Mask R-CNN from 38.2% to 42.5% AP on the MS COCO benchmark.

1. Introduction

Resource-constrained applications need accurate, lightweight detectors, but distilling knowledge for detection is complicated by its structured outputs. MTPD addresses capacity and teacher-selection challenges with progressive distillation and reports improved generalization.

1. Introduction

  • Safety-critical real-time systems, including self-driving cars and virtual or augmented reality devices, face computational and memory constraints that motivate lightweight models.Knowledge distillation transfers learned information from a large model into a small one.Source: §1 · p. 1
  • Detection distillation is challenging because multi-task heads produce variable-length outputs, so classification methods are often not directly applicable.Detection heads handle classification and box or mask regression.Source: §1 · p. 1
  • A capacity gap can limit student performance even with a strong teacher, while architectural differences and unoptimized teacher selection pose additional challenges.The paper highlights Transformer-teacher and convolution-student pairs as one such architectural mismatch.Source: §1 · p. 2
  • MTPD progressively distills from a curriculum of teachers and uses representation similarity to automatically determine their order for a given student.The framework is presented as compatible with existing detection-distillation mechanisms and as effective across heterogeneous settings.Source: §1 · p. 2
  • The authors report that multi-teacher distillation improves generalization rather than optimization, with transferred knowledge leading to a flatter minimum.The paper also investigates progressive distillation from Transformer-based teachers to convolution-based students.Source: §1 · p. 2

2. Related Work

Prior work covers distillation for classification, detection, and segmentation, with detector methods often transferring features. This paper distinguishes sequential multi-teacher distillation from ensembles and applies simple feature matching.

2. Related Work

3. Approach

MTPD treats detector distillation as a curriculum: a student learns progressively from an ordered sequence of teachers. The approach combines feature matching with detection supervision and uses representation-based costs to select a teacher order.

3. Approach

  • MTPD is a general meta-strategy that progressively trains a student detector using a sequence of teachers.The approach section introduces single-teacher feature matching before describing progressive distillation with multiple teachers.Source: §3 · p. 3

3.1. Preliminary: Single Teacher Distillation via Simple Feature Matching

  • Single-teacher distillation minimizes L2 discrepancy between teacher and student neck features while retaining the ground-truth detection loss.A mapping r(·) aligns feature dimensions; it may use an identity mapping, 1×1 convolution, upsampling, or both, and is needed only during training. The feature-matching loss is simpler than more complex alternatives and does not require running the teacher heads.Source: §3.1 · p. 3

3.2. Progressive Distillation with Multiple Teachers

  • Progressive distillation uses one teacher at a time rather than an ensemble, treating the ordered teacher sequence as a curriculum for the student.Figure 2 illustrates ResNet-50 RetinaNet COCO validation AP improving from 36.5% to 37.9% with HTC, then from 37.9% to 39.9% with DetectoRS.Source: §3.2 · p. 3, Figure 2 · p. 4
  • Teacher ordering is framed as finding a sequence that bridges the student’s capacity gap without exhaustively searching all teacher permutations.The proposed approach uses model-representation correlation analysis as a basis for ordering.Source: §3.2 · p. 3
  • Adaptation cost C(A, B) is the validation distillation loss after fitting a linear mapping from model A’s frozen features to model B’s, and it is non-symmetric.A larger cost indicates greater difficulty adapting A’s representation to B’s; adapting high-capacity representations to low-capacity ones is relatively easier than the reverse.Source: §3.2 · p. 4
  • Backward Greedy Selection builds the order backward from the best-performing final teacher, adding preceding teachers by smallest adaptation cost until a stopping condition or teacher limit is reached.The maximum number of selected teachers k can be set according to desired training time.Source: §3.2 · p. 4
  • MTPD can be paired with more advanced distillation mechanisms by selecting and ordering teachers first, then training the student sequentially with each teacher.The teacher-order design requires about 3 GPU hours per student model; adding candidates can increase pairwise-mapping computation near-linearly.Source: §3.2 · p. 4

4. Experiments

The experiments evaluate MTPD across teacher-order selection, homogeneous and heterogeneous detectors, and established distillation mechanisms. They report stronger student performance, including transfer from Transformer teachers to convolutional students, and attribute gains to better generalization rather than optimization.

4.1. Searching for Near-Optimal Teacher Orders

  • BGS produces consistently near-optimal teacher orders and outperforms other heuristics, including global optimization algorithms.The experiments use BGS-selected orders in later sections rather than exhaustively testing all possible orders.Source: Table 2 · p. 5, §4.1 · p. 6

4.2. Distillation with Homogeneous Teachers

  • 36.5% to 39.9% box AP for RetinaNet and 38.2% to 41.4% box AP plus 34.7% to 37.3% mask AP for Mask R-CNN follow MTPD.The same-budget teacher curriculum outperforms direct distillation and teacher ensembling; large-object APL improves about 6% for both models.Source: §4.2 · pp. 6–7, Table 3 · p. 6

4.3. Distillation with Heterogeneous Teachers

  • 37.0% box AP for the ResNet-18 student and 31.5% box AP plus 28.2% mask AP for the low-resolution student follow heterogeneous-teacher distillation.The ResNet-18 student’s APL rises from 43.6% to 50.0%, while the 0.25× student is distilled through teachers at 0.5×, 0.75×, and 1× resolution.Source: §4.3 · p. 7, Table 4 · p. 7

4.4. Generalizability to State-of-the-Art Distillation Mechanisms

  • MTPD improves direct-distillation baselines by 0.2% to 0.8% AP across state-of-the-art mechanisms without increasing training time.For FGD-distilled RetinaNet, AP rises from 40.7% to 41.5%, a gain larger than the 0.3% gain from FGD to MGD.Source: §4.4 · p. 7, Figure 4 · p. 8
  • 42.0% AP for ResNet-50 RetinaNet and 42.5% AP for Mask R-CNN are achieved with progressive distillation from Transformer-based teachers.The reported approach uses an intermediate Swin-Tiny teacher to bridge the gap from Swin-Small to the students.Source: §4.4 · p. 8, Table 5 · p. 8

4.5. Unpacking the Performance Gain: Generalization or Optimization?

  • Distilled students achieve higher validation accuracy despite higher training detection loss, supporting better generalization rather than improved optimization.Their loss landscapes are flatter than the original student’s; the authors report a similar observation in other students.Source: §4.5 · pp. 8–9, Figure 5 · p. 9

5. Conclusion

The paper presents MTPD as a teacher-curriculum approach for lightweight object detection, reporting successful Transformer-to-convolutional distillation and improved generalization. Its analysis is empirical, with theoretical understanding left for future work.

5. Conclusion

  • MTPD progressively transfers knowledge from an automatically arranged sequence of teachers to train a lightweight object detector.The authors describe the approach as mitigating the capacity gap between teacher and student.Source: §5 · p. 9
  • The authors report successful distillation from Transformer-based teachers to convolution-based students and improved generalization rather than optimization on COCO.Source: §5 · p. 9
  • The work is focused on empirical results and analysis; theoretical understanding of adaptation cost and distillation-related generalization remains future work.The authors also note potential misuse involving surveillance, privacy, or biased outcomes.Source: §5 · p. 9

Appendix

The appendix outlines additional analyses, ablations, generalization experiments, comparisons, and implementation details for MTPD.

Appendix

  • Section A adds results and analysis of teacher-order selection.Source: Appendix · p. 12
  • Sections B and C provide ablations for homogeneous and heterogeneous teacher distillation.Source: Appendix · p. 12
  • Sections D–G cover Argoverse-HD streaming accuracy, prior-method comparisons, further distillation-mechanism experiments, and implementation details.Source: Appendix · p. 12

A. Additional Results on Searching for Near-Optimal Teacher Orders

The section compares Backward Greedy Selection (BGS) with other heuristics for ordering teachers by adaptation cost and teacher performance. BGS produces competitive orders in mini-budget and full-budget settings, while performance ranking can falter with diverse teacher architectures.

A. Additional Results on Searching for Near-Optimal Teacher Orders

  • BGS starts with the best-performing teacher, then adds eligible teachers with the lowest adaptation cost to the next teacher in the sequence.The candidate pool excludes teachers whose adaptation cost to the next teacher is not lower than the student's cost to that teacher.Source: A. Additional Results on Searching for… · p. 12
  • The comparison includes shortest paths minimizing summed or maximum adaptation cost, forward nearest-teacher construction, and top-k performance ranking.These alternatives respectively optimize path cost, choose each next teacher from the current one, or order top-performing teachers by performance.Source: A. Additional Results on Searching for… · p. 14
  • BGS produces teacher orders that are competitive with or better than the alternative heuristics in both mini-budget and full-budget settings.Top-k performance is comparable to BGS in the mini-budget setting, but BGS outperforms it with a full budget.Source: A. Additional Results on Searching for… · p. 14
  • With diverse teachers, top-k ranking selects a sequence yielding 42.0 AP, while BGS selects one yielding 42.5 AP.The authors attribute the difference to BGS accounting for representation similarity, which performance ranking alone does not.Source: A. Additional Results on Searching for… · p. 14
  • BGS prioritizes the final target teacher, then selects the preceding teacher with the smallest adaptation cost to that target.The authors identify this backward selection behavior as the reason BGS works best in their setting.Source: A. Additional Results on Searching for… · p. 14

B. Ablation Study on Distillation with Homogeneous Teachers

For homogeneous COCO detectors, sequential distillation improves the student, and a curriculum-like teacher order performs best. Distilling first from Teacher V and then Teacher IV improves both box and mask AP over the off-the-shelf student.

Simultaneous vs. progressive distillation

Impact of individual teachers

  • Distillation improves the off-the-shelf student across the tested teacher types, but a more powerful teacher does not necessarily yield a better distilled student.The authors also report that distillation is easier from teachers with the same head.Source: B. Ablation Study on Distillation with… · p. 15

Simultaneous vs. progressive distillation

  • A curriculum using the lower-adaptation-cost teacher first and the stronger, higher-cost teacher later achieves the best performance.The authors report that sequential distillation improves performance regardless of teacher order, but the curriculum-like order performs best.Source: B. Ablation Study on Distillation with… · p. 15

Overall performance

  • 41.4% box AP (from 38.2%) and 37.3% mask AP (from 34.7%) are achieved by distilling Student II with Teachers V→IV.The resulting Mask R-CNN has performance comparable to HTC with much smaller runtime.Source: B. Ablation Study on Distillation with… · p. 15

C. Ablation Study on Distillation with Heterogeneous Teachers

In heterogeneous distillation, MTPD improves COCO detector AP, and teacher order is especially important when student and teacher architectures differ. The section also compares distillation with longer student training and with teachers trained at different input resolutions.

Overall performance

Impact of individual teachers

  • A smaller capacity gap is more important in heterogeneous distillation, where a less powerful but more similar teacher can outperform the strongest teacher.Teacher I, which shares the student's neck and head and has a similar but deeper backbone, gives the best single-teacher result, improving APs by 2%.Source: C. Ablation Study on Distillation with… · p. 16

Sequential order of teachers

  • Progressive distillation outperforms simultaneous distillation, and the heterogeneous setting requires starting with the teacher closest to the student.Teacher I differs from Student III only in the depth of its ResNet backbone; the best reported progression proceeds from Teacher I to stronger teachers.Source: C. Ablation Study on Distillation with… · p. 16, Table 12 · p. 17

Training a student longer vs. distilling a student

  • Twelve additional training epochs improve APs by 1%, but longer training without distillation yields no significant further improvements.The authors contrast this with the gains from progressive distillation.Source: C. Ablation Study on Distillation with… · p. 16

Distillation with different model resolutions

  • For different-resolution models, the student feature maps are upsampled for teacher supervision, and the best sequence uses Teachers I-3, I-2, then I-1.The setup uses a 333 × 200 student and teacher variants trained at 666 × 400, 1,000 × 600, and 1,333 × 800 input resolutions.Source: C. Ablation Study on Distillation with… · p. 17, Table 13 · p. 17

D. Generalizability to Other Datasets and Evaluation Protocols

The authors test MTPD beyond COCO using Argoverse-HD, including evaluation on a novel test set, streaming accuracy, and direct distillation on Argoverse-HD. They report gains in each setting, with direct distillation improving box AP from 27.1% to 29.2%.

Generalization to Argoverse-HD

  • COCO-trained distilled models gain 2% AP on Argoverse-HD without retraining, and MTPD also improves streaming detection accuracy over prior results.Streaming accuracy evaluates accuracy and latency together in an online real-time setting; the table caption reports that a lightweight model with full-resolution input is more helpful than a complex model that downsizes input.Source: Table 14 · p. 18, D. Generalizability to Other Datasets… · p. 18

Direct distillation on Argoverse-HD

  • 29.2% box AP (from 27.1%) is achieved by directly distilling on Argoverse-HD with teachers ordered from the most similar to the more powerful.The reported progression is Teacher A, then B, C, and D.Source: D. Generalizability to Other Datasets… · p. 18
  • Direct Argoverse-HD distillation improves box AP by 2.1%, compared with 1.7% for the distilled models evaluated there after COCO training.The directly distilled setup uses Faster R-CNN models with ResNet-18 and ResNet-50 backbones.Source: D. Generalizability to Other Datasets… · p. 18

E. Additional Related Work: Detailed Comparison with Prior Knowledge Distillation Methods for Image Classification

The comparison distinguishes MTPD from classification-oriented distillation and other multi-teacher strategies by its detection-specific, sequential transfer and teacher ordering. The discussion also contrasts it with online distillation and reports that detector-specific methods outperform VID.

E. Additional Related Work: Detailed Comparison with Prior Knowledge Distillation Methods for Image Classification

  • Object detection’s multi-task heads and variable-length outputs make classification distillation methods often unsuitable without detection-specific mechanisms.Detection includes classification and box or mask regression, unlike single-output classification.Source: E. Additional Related Work: Detailed… · pp. 19–20
  • MTPD differs from TAKD by addressing detection features or structured predictions and automatically ordering diverse teachers by representation similarity through BGS.Unlike TAKD’s depth-ordered assistants, MTPD trains its teachers independently, making teacher generation parallelizable.Source: E. Additional Related Work: Detailed… · p. 19
  • Sequential progressive distillation outperforms simultaneous teacher-ensemble distillation, which may give conflicting intermediate-feature supervision signals.The comparison is reported for experiments 7–9 versus 10–12 in Table 12.Source: E. Additional Related Work: Detailed… · p. 19
  • Unlike online distillation’s single, updating teacher, MTPD switches among independently trained, frozen teachers with different architectures and intermediate capacities.Online methods commonly use teacher and student models with similar or identical architectures.Source: E. Additional Related Work: Detailed… · pp. 19–20
  • VID performs worse than detector-specific CWD, FGD, and MGD, and worse still than MGD combined with MTPD.The comparison concerns RetinaNet detectors.Source: E. Additional Related Work: Detailed… · p. 20, Table 16 · p. 20

F. Additional Results on Generalizability to State-of-the-Art Distillation Mechanisms

The additional experiments test MTPD with state-of-the-art distillation mechanisms and across teacher–student architecture directions. They report gains in convolution-to-Transformer distillation and show that sequential use of intermediate and final teachers outperforms keeping either teacher throughout.

F. Additional Results on Generalizability to State-of-the-Art Distillation Mechanisms

  • The experiments target the practically motivated direction of high-capacity Transformer teachers teaching lightweight convolution-based students, which the paper describes as challenging.The authors favor convolution-based students because commonly used vision Transformers are still slower than convolutional counterparts in their stated setting.Source: F. Additional Results on… · pp. 20–21
  • 0.9% AP from direct MGD distillation rises by a further 0.6% AP with MTPD, yielding 45.2% AP for RetinaNet/Swin-Tiny on COCO.The teachers use ConvNeXt-Small and intermediate ConvNeXt-Tiny backbones; Table 17 identifies MTPD as the best-accuracy student.Source: F. Additional Results on… · p. 21, Table 17 · p. 21
  • Using only the smaller-gap intermediate teacher remains suboptimal, even though extending its training improves performance from 40.2% AP to 40.7% AP.The student is RetinaNet/ResNet-50, the intermediate teacher is RetinaNet/ResNet-101, and the final teacher is RetinaNet/ResNeXt-101.Source: F. Additional Results on… · pp. 21–22
  • 40.7% AP from longer intermediate-teacher distillation remains below final-teacher distillation, while sequential intermediate-then-final distillation reaches 41.4% AP.Each scheme uses the same total 2× training schedule; Table 18 reports MTPD outperforming both direct schemes.Source: Table 18 · p. 21, F. Additional Results on… · p. 22

G. Implementation Details

The implementation specifies detector training settings, integration choices for existing distillation losses, and feature alignment for Transformer-based teachers.

G. Implementation Details

  • Detectors are trained on eight GPUs for 12 epochs, using 1,333 × 800 COCO inputs or 1,920 × 1,200 Argoverse-HD inputs.Each GPU hosts two COCO images or one Argoverse-HD image; SGD uses momentum 0.9.Source: G. Implementation Details · p. 22
  • For RetinaNet, the feature-matching loss uses λ = 0.5; for Mask R-CNN, λ = 0.8, with performance not very sensitive from 0.3 to 0.8.For CWD, FGD, and MGD integrations, the authors follow public implementations and apply the inheriting strategy only for the first teacher.Source: G. Implementation Details · p. 22
  • Swin Transformer teachers use an FPN neck with the student’s channel count, so feature-map matching is unnecessary.The Swin backbone shares feature-map resolutions with typical convolutional networks such as ResNet-50.Source: G. Implementation Details · p. 22
Loading 2308.09105v1…