PaperlayerView exact revision on arXiv

Source-linked AI summary

Continual Detection Transformer for Incremental Object Detection

Yaoyao Liu, Bernt Schiele, Andrea Vedaldi, Christian Rupprecht

arXiv:2304.03110v1cs.CV

TL;DR

IOD suffers catastrophic forgetting, and direct KD and ER are ineffective for transformer-based detectors. CL-DETR adapts both techniques through reliable label-level distillation and distribution-calibrated exemplar replay, achieving state-of-the-art IOD results and a 4.2-point AP improvement over direct KD and ER.

  • Problem

    IOD trains detectors in phases with new-category annotations, but catastrophic forgetting and mismatched training statistics make direct KD and ER ineffective for transformer-based detectors.

    Source: §1 · pp. 1–2

  • Method

    CL-DETR uses Detector Knowledge Distillation to select reliable old predictions and reconcile them with new ground truth, while calibrating exemplars to match the training-set category distribution.

    Source: §1 · p. 2

  • Results

    CL-DETR improves AP by 4.2 percentage points over direct KD and ER on transformer-based detectors in COCO 2017 experiments.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The method enables effective KD and ER across transformer-based detectors including Deformable DETR [61] and UP-DETR [10].

    Source: §1 · p. 2, §5 · p. 8

    The pseudo-label strategy may fail to generate boxes for all objects when images contain too many objects, favoring correct but potentially incomplete annotations.

    Source: §4.2 · p. 8

Abstract

from arXiv · show

Incremental object detection (IOD) aims to train an object detector in phases, each with annotations for new object categories. As other incremental settings, IOD is subject to catastrophic forgetting, which is often addressed by techniques such as knowledge distillation (KD) and exemplar replay (ER). However, KD and ER do not work well if applied directly to state-of-the-art transformer-based object detectors such as Deformable DETR and UP-DETR. In this paper, we solve these issues by proposing a ContinuaL DEtection TRansformer (CL-DETR), a new method for transformer-based IOD which enables effective usage of KD and ER in this context. First, we introduce a Detector Knowledge Distillation (DKD) loss, focusing on the most informative and reliable predictions from old versions of the model, ignoring redundant background predictions, and ensuring compatibility with the available ground-truth labels. We also improve ER by proposing a calibration strategy to preserve the label distribution of the training set, therefore better matching training and testing statistics. We conduct extensive experiments on COCO 2017 and demonstrate that CL-DETR achieves state-of-the-art results in the IOD setting.

1. Introduction

IOD applies incremental learning to object detection, where phased training can cause catastrophic forgetting and direct KD or ER performs poorly for transformer-based detectors. CL-DETR addresses these issues with detector-specific distillation, calibrated replay, a revised protocol, and strong COCO 2017 results.

Motivation

  • Directly applying KD and ER to transformer-based detectors performs much worse than joint training because parallel hypotheses create unbalanced distillation and conflicting old/new evidence.ER also mismatches testing statistics when it samples equal numbers of exemplars despite skewed object-category distributions.Source: §1 · pp. 1–2

CL-DETR

  • CL-DETR selects reliable old-model predictions, merges them with new-category ground truth, resolves conflicts, and distills only correctly matched foreground predictions.This Detector Knowledge Distillation strategy ignores redundant background detections while retaining standard joint bipartite matching.Source: §1 · p. 2
  • CL-DETR calibrates exemplar replay to preserve the training-set category distribution, reducing the mismatch between training and testing statistics.Each phase also includes a smaller calibration-focused training step.Source: §1 · p. 2

Benchmark protocol

  • The revised IOD protocol prevents repeated images across phases except for deliberately stored exemplars, addressing incompatibility with standard incremental-learning definitions.Earlier protocols could expose the same mixed-category images in multiple phases.Source: §1 · p. 2

Results

  • CL-DETR improves AP by 4.2 percentage points over direct KD and ER on transformer-based detectors in COCO 2017 experiments.The study evaluates Deformable DETR [61] and UP-DETR [10], with ablations supporting the modelling choices.Source: §1 · p. 2

2. Related Work

Related work frames CL-DETR as an incremental object-detection method that adapts KD and ER, which were developed largely around conventional detectors, to transformer-based architectures. The paper argues that these adaptations are needed because direct KD and ER transfer poorly to Deformable DETR and UP-DETR.

Incremental learning

  • Incremental learning commonly preserves prior knowledge through KD or replays stored exemplars through ER.KD may match logits or feature maps, while ER maintains a reservoir of samples from earlier training rounds.Source: §2 · p. 2

Incremental object detection

  • IOD is harder than incremental image classification because images contain multiple old and new object types while only new types are annotated in each phase.Prior detection methods extended KD to features, proposal networks, or detector outputs, while others used exemplar fine-tuning or adaptive sampling.Source: §2 · p. 2
  • Existing IOD methods mainly target conventional detectors, whereas direct KD and ER do not work well on state-of-the-art transformer detectors such as Deformable DETR [61] and UP-DETR [10].CL-DETR proposes fixes specifically for this mismatch.Source: §2 · pp. 2–3

Transformer-based object detection

  • DETR uses bipartite matching to assign each object hypothesis to one target or background, while Deformable DETR [61] adds sparse multilevel attention and UP-DETR [10] uses unsupervised pre-training.These architectures provide the detector context in which CL-DETR applies its broadly applicable KD and ER changes.Source: §2 · pp. 2–3

3. Methodology

CL-DETR adapts incremental object detection for transformer detectors by combining structured detector knowledge distillation with exemplar replay under a stricter non-overlapping phase protocol.

3.1. Incremental object detection

  • CL-DETR defines incremental phases with distinct image subsets and category annotations, while allowing stored exemplars from previous phases during replay.Images may contain objects from any category, but only current-phase categories are annotated; exemplar replay trains on current data plus prior exemplars.Source: §3.1 · p. 3
  • The revised protocol prevents the same non-exemplar images from appearing in multiple phases, unlike prior IOD protocols that permit overlapping image subsets.This preserves the standard incremental-learning property that samples observed in different phases do not repeat, except for deliberately stored exemplars.Source: §3.1 · p. 3

3.2. Transformer-based detectors

  • Transformer detectors produce fixed-query object predictions and train them with Hungarian matching, pairing hypotheses with object or background targets while supporting one-to-one assignments.The DETR loss combines classification with bounding-box loss for non-background targets, and the matching is obtained with the Hungarian algorithm.Source: §3.2 · p. 3

3.3. Detector knowledge distillation

  • Classical KD fails because background-heavy query outputs create unbalanced supervision and old-category knowledge can conflict with current labels for new categories.The standard L_DETR + L_KD formulation also ignores the detector’s one-to-one matching structure, which is central to avoiding duplicate detections.Source: §3.3 · pp. 4–5
  • DKD selects confident foreground predictions from the old model as pseudo-labels, filters overlaps with new-category ground truth, and jointly trains merged labels through bipartite matching.Background predictions are ignored because they are imbalanced and may contradict new-category labels; the resulting labels are padded with background targets and optimized using the DETR loss.Source: §3.3 · p. 5

3.4. Distribution-preserving calibration

  • Distribution-preserving calibration selects exemplars to approximate each phase’s category distribution, then fine-tunes with the new exemplar set using the DETR loss.The selection minimizes category-marginal divergence, while training first uses DKD on current data plus old exemplars and then uses only the calibrated exemplar set.Source: §3.4 · pp. 5–6

4. Experiments

CL-DETR is evaluated on COCO 2017 with Deformable DETR and UP-DETR across two- and multiple-phase IOD settings, using standard detection metrics and repeated randomized splits. It consistently improves over existing IOD methods and direct KD/ER baselines, while ablations examine distillation, replay, pseudo-label selection, and visual behavior.

4. Experiments

  • 4.2 percentage points of AP improvement over direct KD and ER are reported on COCO 2017 for CL-DETR across transformer-based detectors.Experiments use Deformable DETR and UP-DETR as baselines.Source: §4 · p. 6

4.2. Results and analyses

  • 40.1% and 37.5% AP are achieved by Deformable DETR with CL-DETR in the 70+10 and 40+40 settings, respectively, attaining the highest AP.The method consistently outperforms the state-of-the-art and other IOD methods in both two-phase settings.Source: §4.2 · p. 7
  • 7.4 percentage points higher AP than [13] is obtained in 40+10×4, compared with a 2.9-point gain in 40+20×2.CL-DETR’s relative advantage increases with the number of phases.Source: §4.2 · p. 7, Figure 3 · p. 7
  • Joint bipartite matching boosts AP by 5.8 percentage points over conventional KD, while pseudo-label selection further improves AP and reduces forgetting.The ablation attributes these gains to one-to-one object–hypothesis matching, reduced duplicate detections, and rejection of redundant background information.Source: §4.2 · pp. 7–8
  • Calibration improves all-category AP by 1.8 points, old-category AP by 2.1 points, and reduces AP forgetting by 2.1 points versus conventional ER.The strategy preserves the training-set category distribution rather than balancing exemplar categories.Source: §4.2 · p. 8
  • Top-K pseudo-label selection reaches peak AP at K=10, with at most a 0.4-point difference across K values, indicating robustness to this hyperparameter.The experiments compare top-K confident non-background predictions with score-threshold selection.Source: §4.2 · p. 8, Table 3 · p. 8
  • The visualizations show accurate pseudo boxes can match ground truth, but dense images may leave some objects uncovered while retaining foreground focus.The trade-off favors correct but possibly incomplete annotations over contradictory or noisy labels.Source: Figure 4 · p. 8, §4.2 · p. 8

5. Conclusions

The paper concludes that CL-DETR enables effective KD and ER for transformer-based IOD. It combines detector-specific distillation, distribution-preserving replay, and a benchmark protocol without duplicated images across phases.

5. Conclusions

  • CL-DETR effectively applies KD and ER to transformer-based detectors, including Deformable DETR and UP-DETR.The method selects informative old predictions, rejects redundant background predictions, and aligns exemplars with the training distribution.Source: §5 · p. 8
  • The revised IOD protocol avoids using duplicated images in different training phases, making the benchmark more realistic.The paper identifies online learning as a future extension.Source: §5 · p. 8

A. Traditional IOD protocol and results

The traditional IOD protocol allows phases to observe overlapping images, unlike the standard incremental-learning assumption. Under this protocol, CL-DETR still outperforms existing methods, including without exemplars.

A. Traditional IOD protocol and results

  • The traditional protocol permits the same images to appear in multiple phases when they contain objects from different category subsets.This differs from the standard assumption that non-exemplar samples do not repeat across phases.Source: A. Traditional IOD protocol and results · p. 11
  • 2.3 percentage points higher AP than ERD [13] is achieved by CL-DETR without ER in the 40+40 setting.Under the traditional protocol, CL-DETR consistently outperforms the state-of-the-art and other IOD methods.Source: A. Traditional IOD protocol and results · p. 11

B. More ablation results

Supplementary ablations examine hyperparameter robustness, validation-set consistency, curriculum learning, fine-grained performance, and exemplar replay strategies. They generally support the robustness and distribution-preserving design of CL-DETR.

B. More ablation results

  • Peak AP occurs at λ=0.7, with only a 1.0-point maximum difference across tested values, indicating robustness to λ.The ablation uses COCO 2017 in the 70+10 setting.Source: B. More ablation results · p. 11
  • Top-K selection performs best on a separate validation set, consistent with the main-paper ablation.The validation uses COCO 2017 in the 70+10 setting.Source: B. More ablation results · p. 11
  • Curriculum learning for p slightly improves final performance by 0.3 AP by gradually including lower-confidence objects during training.The schedule decreases p from 0.5 to 0.1.Source: B. More ablation results · p. 11
  • CL-DETR outperforms LwF and iCaRL in AP, old-category AP, and FPP in fine-grained Deformable DETR results.The comparison is conducted on COCO 2017 in the 70+10 setting.Source: B. More ablation results · p. 11
  • Distribution-persevering exemplar replay yields higher AP and lower FPP than existing replay strategies [43] [57].Preserving COCO 2017’s natural data distribution outperforms category-balanced exemplar selection.Source: B. More ablation results · p. 11

C. More visualization results

Figure S1 visualizes one-to-one matching between merged bounding boxes and new-model predictions, showing complementary old-category pseudo-labels and new-category ground truth. The visualization indicates that the method resolves conflicts and helps ignore background predictions.

  • Figure S1 shows one-to-one matching between merged bounding boxes and new-model predictions in COCO 2017’s 70 + 10 setting.Merged boxes combine old-category pseudo-labels with new-category ground-truth boxes.Source: C. More visualization results · p. 11
  • The old-category pseudo-labels and new-category ground truth are complementary, representing old and new objects respectively.Source: C. More visualization results · p. 11
  • The visualization indicates that the method resolves pseudo-label and ground-truth conflicts while ignoring background predictions.Source: C. More visualization results · p. 11

D. Source Code in PyTorch

The supplementary materials provide installation and experiment-running instructions, define evaluated incremental settings, and list supplementary tables covering results and ablations. They also include visualizations of matching and describe the exemplar-memory conditions used in supplementary comparisons.

Supplementary results

  • Table S1 reports traditional-protocol COCO 2017 IOD results for CL-DETR with and without exemplar replay, using a 10% dataset-size memory budget when replay is enabled.Related-method results are taken from [13].Source: Table S1 · p. 12

Experimental settings

  • The A + B protocol observes fractions of training samples across two phases, with experiments reported for 40 + 40 and 70 + 10 settings.Results average AP across three category orders and data orders with 95% confidence intervals.Source: D. Source Code in PyTorch · p. 12

Ablations

  • Supplementary tables evaluate λ, pseudo-label selection, curriculum learning, and exemplar-replay strategies, including fine-grained KD and ER ablations.The ablations use COCO 2017, primarily in the 70 + 10 setting, with Deformable DETR for the KD and ER analyses.Source: Table S2 · p. 12, Table S3 · p. 12, Table S4 · p. 12

Installation

  • The project requires Python 3.7 with Anaconda, PyTorch, torchvision, other dependencies, and compiled CUDA operators.The instructions also include environment activation and dependency-installation steps.Source: D. Source Code in PyTorch · p. 12

Running experiments

Visualization

  • Figure S1 visualizes matching between merged old-category pseudo-labels, new-category ground truth, and new-model predictions in the 70 + 10 setting.Source: Figure S1 · p. 13

Related papers

Loading 2304.03110v1…