PaperlayerView exact revision on arXiv

Source-linked AI summary

Overcoming Catastrophic Forgetting in Incremental Object Detection via Elastic Response Distillation

Tao Feng, Mang Wang, Hangjie Yuan

arXiv:2204.02136v1cs.CV

TL;DR

Incremental object detection must learn new classes without catastrophically forgetting old ones, while prior IOD methods under-explore detector responses. The paper proposes Elastic Response Distillation with statistical response selection for classification and regression heads, and reports state-of-the-art MS COCO performance that narrows the gap toward full training.

  • Problem

    Incremental object detection suffers catastrophic forgetting when detectors are fine-tuned on new data, while prior IOD work under-explores response information.

  • Method

    Elastic Response Distillation separately distills classification and regression responses, adds incremental localization distillation, and uses Elastic Response Selection to choose valuable locations.

  • Results

    The method achieves state-of-the-art performance on MS COCO and substantially narrows the performance gap toward full training.

  • Takeaways & Limitations

    Selective response distillation provides a response-based incremental detection paradigm that preserves category and localization knowledge across detector heads.

  • Takeaways & Limitations

    Incrementally trained models may introduce privacy issues, which the paper suggests mitigating by limiting access to trained models.

Abstract

from arXiv · show

Traditional object detectors are ill-equipped for incremental learning. However, fine-tuning directly on a well-trained detection model with only new data will lead to catastrophic forgetting. Knowledge distillation is a flexible way to mitigate catastrophic forgetting. In Incremental Object Detection (IOD), previous work mainly focuses on distilling for the combination of features and responses. However, they under-explore the information that contains in responses. In this paper, we propose a response-based incremental distillation method, dubbed Elastic Response Distillation (ERD), which focuses on elastically learning responses from the classification head and the regression head. Firstly, our method transfers category knowledge while equipping student detector with the ability to retain localization information during incremental learning. In addition, we further evaluate the quality of all locations and provide valuable responses by the Elastic Response Selection (ERS) strategy. Finally, we elucidate that the knowledge from different responses should be assigned with different importance during incremental distillation. Extensive experiments conducted on MS COCO demonstrate our method achieves state-of-the-art result, which substantially narrows the performance gap towards full training.

1. Introduction

Incremental object detection must learn from changing class distributions without losing earlier knowledge. The paper proposes Elastic Response Distillation to selectively preserve important classification and localization responses, narrowing the gap to full training.

  • Motivation: Class-incremental object detection is especially challenging because new knowledge from non-stationary data can interfere with previously learned knowledge.The paper focuses on class incremental object detection as the most intractable IOD scenario.
  • Motivation: Response-based distillation targets classification predictions and bounding-box responses, whose information is under-explored in prior IOD work.Responses contain logits and bounding-box offsets, motivating direct response learning.
  • Motivation: Not all response locations are equally important, so selecting an appropriate subset is necessary for retaining old-object behavior.The paper frames response selection as analogous to constraining important parameters during continual learning.
  • Proposed approach: Elastic Response Distillation learns classification and regression responses separately, adding incremental localization distillation to preserve location information.Elastic Response Selection automatically chooses valuable distillation nodes using statistical characteristics.
  • Outcome: MS COCO experiments report state-of-the-art performance and substantially narrow the gap between incremental learning and full training.The method is presented as alleviating catastrophic forgetting through selective response distillation.

2. Related work

Prior work addresses incremental learning and object-detection distillation through parameter constraints, knowledge transfer, and feature or component imitation. The paper positions IOD response distillation as insufficiently attentive to detector-head information.

  • Incremental Learning: Incremental learning methods use parameter constraints or knowledge distillation to protect old knowledge while learning new tasks.EWC and MAS regularize parameters, whereas LwF, iCaRL, and BiC transfer or correct prediction knowledge.
  • Incremental Object Detection: Incremental object detection remains less explored than incremental classification because detection combines multiple outputs and has greater task complexity.Existing systems span Fast R-CNN, FCOS, CenterNet, Faster R-CNN, RetinaNet, and open-world detection settings.
  • Incremental Object Detection: Existing IOD distillation frameworks do not sufficiently address the detector head, although head responses carry important information for incremental detection.This motivates explicitly studying response-based distillation in the head.
  • Knowledge Distillation for Object Detection: Object-detection distillation has imitated features, all detector components, foreground and background responses, or regression localization information.These approaches establish knowledge distillation as a broad transfer mechanism beyond incremental detection.

3. Method

ERD trains the student with detector losses plus separate classification and regression response distillation. ERS selects statistically valuable response locations instead of treating every response equally.

  • Overall Structure: The framework distills responses from the teacher’s classification and regression heads while adding incremental localization distillation for the regression branch.The student learns from selected teacher responses rather than all head outputs.
  • Overall Structure: The total objective combines detector-specific training loss with weighted ERD classification and bounding-box distillation losses.The default weights are λ1 = λ2 = 1, and teacher and student denote the two detector models.
  • ERD at Classification Head: Classification response distillation converts teacher and student logits into softened probability distributions before selectively matching category responses.Selected responses are obtained from teacher outputs on new data and transferred through incremental distillation.
  • ERD at Regression Head: Regression response distillation transfers localization information from selected teacher bounding boxes using distributional edge representations and KL-divergence loss.The regression branch is challenging because it can predict low-confidence boxes even without old-category objects.
  • Elastic Response Selection: Selecting all responses can hurt performance, making response-node selection important for preventing catastrophic forgetting.ERS is designed to maintain fairness across responses while retaining valuable distillation nodes.
  • Elastic Response Selection: ERS replaces fixed confidence or top-k rules with statistical selection for classification and regression responses.Classification uses confidence statistics, while regression uses Top-1 distribution values followed by thresholding and NMS.

4. Experiments and Discussions

Experiments on MS COCO evaluate ERD under one-step, multi-step, ablation, generalization, and feature-versus-response settings. ERD consistently mitigates catastrophic forgetting, with selective response distillation producing strong performance across detectors and incremental scenarios.

  • One-step: 17.8% AP under fine-tuning versus 40.2% with full-data training shows the severity of catastrophic forgetting in the one-step setting.ERD improves AP by 19.1%, 28.6%, and 20.3% at IoU 0.5, 0.75, and 0.95, respectively, while remaining 3.3% below the upper bound.
  • Class-wise analysis: ERD preserves information for most old classes while learning newly introduced classes, unlike direct fine-tuning.Figure 3 compares full-data training, fine-tuning, and ERD using per-class AP; Figure 4 compares first-40 and last-40 class performance.
  • Multi-step: ERD outperforms fine-tuning at every incremental step in both two-step and four-step settings.The authors attribute this to ERD providing valuable responses during each update to alleviate forgetting from dynamic data flow.
  • Ablation Study: Regression-only distillation reaches 13.0% AP, while adding ERS to classification and regression responses raises performance to 36.9%.Applying ERS to the regression branch adds 5.4% AP over the baseline.
  • Ablation Study: ERS is nearly parameter-free because varying α produces a maximum performance gap of only 0.4%.The tested combinations vary α1 and α2 over [1], [1], [1], and.
  • Generalization: ERD generalizes to FCOS with stable gains and narrows the upper-bound gap to 2.7% under the last-40-plus-first-40 scenario.Adapting ERD to FCOS requires replacing LD loss with GIoU loss and adjusting response handling for regression and centerness branches.
  • Feature versus response distillation: ERS-based response distillation reaches 36.9% AP versus 32.5% for the corresponding all-response configuration, while adding feature layers reduces AP by 0.4%.The authors suggest feature layers and selected head responses may induce different optimization directions.

5. Conclusion

The paper presents ERD as a response-based incremental object detection paradigm that separately learns classification and regression responses. ERS selects suitable responses, while incremental localization distillation preserves localization information during incremental learning.

  • 5. Conclusion: ERD learns classification and regression responses and introduces incremental localization distillation for regression responses.The method is designed to alleviate catastrophic forgetting in incremental object detection.
  • 5. Conclusion: Elastic response selection provides suitable responses for different detection heads during incremental distillation.The conclusion identifies selective response learning as a central design component.

Broader Impact

The study of incremental object detection may clarify neural-network formation and support lifelong-learning mechanisms, while incremental models may raise privacy concerns.

  • Incremental object detection research may provide a technical basis for developing lifelong-learning mechanisms.
  • Models after incremental learning may create privacy issues, which limiting access to trained models can mitigate.

Supplementary materials

The response-selection analysis shows that confidence distributions vary substantially across images, motivating elastic selection of informative responses.

  • Classification responses on the P3 level contain mostly low-confidence background positions and relatively few high-confidence positive responses.The P3 classification head produces 100 ∗152 responses.
  • Maximum response confidence varies across images, making fixed-number selection prone to redundant background or missing foreground responses.The reported maxima are approximately 0.04 and 0.14 for two samples.
  • Elastic Response Selection provides fairer response selection across images with different confidence distributions.

B. Detailed results of various responses for IOD.

The detailed response-count analysis supports selecting responses elastically rather than using a single exhaustively tuned global number.

  • Exhaustive searches over response counts from 5 to all responses can identify an adequate global number, but still trail the elastic method.

C. Detailed results of base classes and new classes.

The base-versus-new-class analysis shows that ERD substantially recovers base-class performance in incremental settings, with a slight decrease for new classes relative to the upper-bound comparison.

  • ERD substantially improves base-class performance, which catastrophic forgetting drives close to zero in the incremental setting.
  • Table 10 reports performance percentages for base and new classes under different scenarios, with the Upper Bound using full-data standard training.
  • The base-class improvement comes at the cost of a slight decrease in new-class performance.

D. Ablation study of LD loss.

Removing LD loss leaves the method with high performance despite losing extra localization information, while LD loss produces more accurate detection boxes.

  • 36.4% performance is maintained when LD loss is replaced with general L2 distillation loss.The result supports generalization beyond detectors for which LD distillation loss is applicable.
  • LD loss yields more accurate detection boxes than omitting LD loss in incremental scenes.

E. Additional Visualizations of ERD.

Visualizations show that ERD improves incremental detection over direct fine-tuning and SID, approaching the upper-bound detector, while ERS selects responses using confidence-based thresholds.

  • Visual comparison: ERD detects substantially more objects than direct fine-tuning and SID in the first 40 classes plus last 40 classes scenario.Direct fine-tuning barely detects objects, while SID misses most objects despite reaching 34% AP.
  • Visual comparison: ERD performs most closely to the upper-bound method in the illustrated incremental-learning comparison.
  • Elastic Response Selection: ERS samples classification responses by computing confidence statistics and retaining candidates above a threshold defined by the mean plus α1 standard deviations.
  • Elastic Response Selection: ERS samples regression responses using analogous statistics and a threshold defined by the mean plus α2 standard deviations.
Loading 2204.02136v1…