PaperlayerView exact revision on arXiv

Source-linked AI summary

Incremental Object Detection via Meta-Learning

K J Joseph, Jathushan Rajasegaran, Salman Khan, Fahad Shahbaz Khan, Vineeth N Balasubramanian

arXiv:2003.08798v3cs.CVcs.LGeess.IVstat.ML

TL;DR

Incremental object detectors can forget old classes as new classes arrive, while distillation can hinder adaptation to new tasks. This paper meta-learns gradient preconditioning to share information across incremental tasks. It reports outperforming current best methods on two benchmark datasets.

  • Problem

    Incremental object detectors may forget old classes when trained on new tasks, while distillation-based retention can obstruct learning novel classes.

    Source: §1 · p. 1

  • Method

    The method meta-learns gradient preconditioning that reshapes updates to share information across incremental tasks.

    Source: §1 · p. 1

  • Results

    The meta-learned detector outperforms current best methods across two benchmark datasets and a variety of class-incremental settings.

    Source: §4 · p. 5, §6 · p. 10

  • Takeaways & Limitations

    The authors conclude that preconditioned gradient updates help retain old knowledge while adapting flexibly to new tasks.

    Source: §6 · p. 10

Abstract

from arXiv · show

In a real-world setting, object instances from new classes can be continuously encountered by object detectors. When existing object detectors are applied to such scenarios, their performance on old classes deteriorates significantly. A few efforts have been reported to address this limitation, all of which apply variants of knowledge distillation to avoid catastrophic forgetting. We note that although distillation helps to retain previous learning, it obstructs fast adaptability to new tasks, which is a critical requirement for incremental learning. In this pursuit, we propose a meta-learning approach that learns to reshape model gradients, such that information across incremental tasks is optimally shared. This ensures a seamless information transfer via a meta-learned gradient preconditioning that minimizes forgetting and maximizes knowledge transfer. In comparison to existing meta-learning methods, our approach is task-agnostic, allows incremental addition of new-classes and scales to high-capacity models for object detection. We evaluate our approach on a variety of incremental learning settings defined on PASCAL-VOC and MS COCO datasets, where our approach performs favourably well against state-of-the-art methods.

1 INTRODUCTION

Incremental object detection must accommodate new classes without forgetting old ones, but distillation-based approaches can hinder learning novel classes. The paper proposes meta-learned gradient preconditioning to share information across tasks and support both retention and adaptation.

1 INTRODUCTION

  • We address incremental object detection, where learning new classes can erase knowledge of old ones and distillation can hinder adaptation. Our method meta-learns gradient preconditioning to share information across tasks. The paper reports outperforming current best methods on two benchmark datasets.Source: §1 · p. 1
  • Existing detectors can forget old classes when trained on new tasks, while distillation can impede learning novel classes.The introduction identifies optimal plasticity—adapting quickly without losing previously acquired knowledge—as an essential characteristic of incremental object detectors.Source: §1 · p. 1
  • Conventional meta-learning methods face incremental-detection challenges including fixed class counts, task-overfit task design, task-specific inference, and small architectures.Object-detection images may contain multiple classes, and detectors combine subnetworks for features, proposals, classification, and localization.Source: §1 · p. 1
  • The proposed approach learns gradient preconditioning for incremental object detection, aiming to reshape updates for both old and new tasks.Its meta-training captures properties of incrementally presented task distributions to alleviate forgetting and prepare networks for quick adaptation.Source: §1 · p. 1
  • The authors report evaluations on two benchmark datasets against three competitive baseline methods.The introduction describes these evaluations as demonstrating the utility of the proposed meta-learning methodology.Source: §1 · p. 1

2 RELATED WORK

The section reviews incremental learning and meta-learning, contrasting distillation-based object detection methods with a gradient-based meta-learning approach for incremental detection.

Incremental Learning

  • Most earlier class-incremental object detection methods use knowledge distillation to address forgetting, varying the detector or network components distilled.Prior approaches distill outputs, intermediate features, or both; some also expand proposal-network capacity, while one setting uses precomputed proposals rather than learning the proposal network.Source: §2 · p. 2
  • The review reports that earlier methods failed to improve benchmark on standard evaluation criteria, while later work improved the state of the art using Faster R-CNN and distillation.The authors hypothesize that distillation's restraint may hinder learning new tasks and argue it should be modulated for generalizability.Source: §2 · p. 2

Meta Learning

  • Meta-learning is grouped into optimization-based, black-box adaptation, and non-parametric methods; prior k-shot detection work predicts or reweights detector components or features.The authors note that adapting meta-learning methods from few-shot image classification to detection is not straightforward.Source: §2 · p. 2
  • The authors propose meta-learning a gradient preconditioning matrix to encode information across tasks and support continual adaptation to new classes.They distinguish this gradient-based approach from prior meta-learning methods for k-shot object detection.Source: §2 · p. 2

3 METHODOLOGY

The methodology frames class-incremental detection as adapting a two-stage detector to disjoint arriving class groups without access to all prior data. It meta-learns gradient-warping layers and combines their training with distillation and exemplar-based stores.

3.1 Problem Formulation

  • A task is a disjoint group of classes introduced at time t; the detector must learn each new task without prior datasets while maintaining performance on earlier classes.Images may contain multiple object classes, but annotations are provided only for instances belonging to the current task.Source: §3.1 · pp. 2–3
  • The detector follows a two-stage R-CNN pipeline: a backbone produces features, the RPN proposes regions, and the RoI Head classifies proposals and regresses boxes.The stated formulation focuses on R-CNN-family two-stage detectors.Source: §3.1 · p. 3

3.2 Meta-Learning the Gradient Preconditioning

  • Gradient preconditioning matrices warp task gradients; nonlinear warp layers are meta-learned across tasks, inducing richer gradient relationships through their Jacobians than block-diagonal preconditioning.The authors state that this approach models the joint task distribution and supports generalization to new tasks, faster convergence, and alleviated forgetting.Source: §3.2 · p. 3

3.3 Incremental Object Detector

  • The detector combines classification and localization losses with distillation from a frozen previous model, using a convex combination weighted by α.Distillation covers backbone features and RoI Head targets; the RPN and RoI Head contribute detection losses.Source: §3.3.1 · p. 4
  • Per-class exemplar-image queues feed a feature store of fixed-size per-class RoI-feature queues, giving warp-layer training equal priority across classes despite uneven instance counts.The warp loss accumulates classification and localization losses over stored features and labels; the image store also carries information from previously seen classes.Source: §3.3.2 · p. 4
  • Task parameters update on current-task batches, while warp parameters update every γ iterations using the warp loss and image store; inference fine-tunes task parameters with stored images.The authors report superior performance with just 10 examples per task.Source: §3.3.3 · p. 5

4 EXPERIMENTS AND RESULTS

The experiments evaluate incremental object detection on PASCAL VOC and MS COCO under multiple class-incremental settings. The method outperforms and Faster ILOD in mAP across the reported settings, while ORE is slightly better in VOC 15+5.

4.1 Datasets and Evaluation Metrics

  • PASCAL VOC 2007 and MS COCO 2014 are evaluated using mAP@50; COCO also reports AP and recall across IoUs, scales, and detection counts.The COCO 40+40 results cover the full validation set, with minival also reported for comparison with and.Source: §4.1 · p. 5, §4.4 · p. 6

4.2 Experimental Settings

  • VOC settings range from adding classes 16–20 one at a time to two-task splits, while COCO uses a 40+40 split.The VOC two-task settings are 15+5, 10+10, and 19+1; task data includes images containing a task class, leaving co-occurring out-of-task instances unlabelled.Source: §4.2 · p. 5, §4.4 · p. 6

4.3 Implementation Details

  • The detector is Faster R-CNN with a ResNet-50 backbone and frozen batch normalization; unlike 's Fast R-CNN, it also learns the RPN.Unseen-class logits are set to −1010, and gradients are selectively zeroed to keep task and warp parameters fixed during their respective updates.Source: §4.3 · p. 5

4.4 Results

  • Across settings, the method outperforms and Faster ILOD in mAP, while ORE is slightly better in VOC 15+5; Table 1 shows consistent gains over for sequential VOC additions.With standard training, adding T2 significantly reduces performance on T1 classes.Source: Table 1 · p. 6, §4.4 · p. 6
  • Table 1 reports per-class AP and overall mAP as five VOC classes are added sequentially to a model initially trained on 15 classes.The mAP-old column covers base classes, while mAP covers all classes seen so far.Source: Table 1 · p. 6, §4.4 · p. 6

5 DISCUSSIONS AND ANALYSIS

Ablations show the strongest 15+5 result when distillation, gradient preconditioning, and fine-tuning are combined. The analysis also examines where to place preconditioning, distillation and update sensitivities, alternative methods, auxiliary data, and compute costs.

Choice of Preconditioning Layers

  • For the distilled detector, overall mAP is similar at 58.5% with a trained RPN versus 58.1% without a trained RPN; preconditioning layers are added in the FRoI Head.The FRoI Head is category-specific and most forgetful; each RoI-pooled feature corresponds to one object, supporting class-balanced meta-learning with the FStore.Source: §5 · p. 7

Ablation Experiment

  • 67.8% mAP is the best 15+5 ablation result with distillation, gradient preconditioning, and fine-tuning; distillation alone reaches 58.5%.Using distillation and gradient preconditioning without fine-tuning gives 54.3% mAP.Source: §5 · p. 6, Table 6 · p. 8

Choice of Feature Distillation Methodology

  • NST and attention transfer perform significantly worse than L2 feature distillation, so the paper uses L2 for its reported results.The compared alternatives are Neuron Selectivity Transfer (NST) and Activation-based attention Transfer (AT).Source: §5 · p. 8, Table 8 · p. 8

Using a Moving Average Teacher

  • Across 10+10, 15+5, and 19+1 settings, performance worsens as the moving-average teacher is updated more aggressively from the student.The authors attribute this to the forgetful student degrading the teacher's knowledge of previous tasks.Source: §5 · p. 8, Table 10 · p. 9

Sensitivity of Warp Interval

  • Less frequent warp-layer updates cause slight performance degradation in the 15+5 setting.The authors attribute the trend to more frequent updates preconditioning gradients more effectively.Source: §5 · p. 8, Table 9 · p. 8

Sensitivity Analysis

  • Across γ = 20/200/2000 iterations, higher distillation weighting raises T1 accuracy while lowering T2 accuracy.T1 is the first 15 classes and T2 the last 5 in the 15+5 setting.Source: §5 · p. 8, Table 9 · p. 8

Comparison with DMC

  • Using auxiliary data further improves the proposed method's performance in the reported 10+10 and 19+1 comparisons with DMC.The base methodology needs only the base and incremental models and no auxiliary data; the 15+5 DMC comparison is unavailable.Source: §5 · p. 9

Time and Memory Requirements

  • 5.71 hours is the reported total compute time for the 10+10 setting, including 18,000 training iterations and 3,000 fine-tuning iterations.The timings were measured on one server with four NVIDIA GTX 1080-Ti GPUs.Source: §5 · pp. 9–10, Table 13 · p. 10

6 CONCLUSION

The meta-learned incremental detector outperforms current best methods on two benchmark datasets. It preconditions gradient updates to share information across tasks, supporting retention of old knowledge and flexible adaptation to new tasks.

6 CONCLUSION

  • The meta-learned incremental object detector outperforms current best methods on two benchmark datasets.Source: §6 · p. 10
  • Meta-learned gradient preconditioning shares information across incremental tasks to retain old knowledge and adapt flexibly to new tasks.Source: §6 · p. 10
  • Extensive ablation experiments examine the contributions of the methodology’s constituent components.Source: §6 · p. 10
  • The authors identify extensions to single-stage detectors and incremental action recognition, object counting, and instance segmentation as future research directions.Source: §6 · p. 10

Related papers

Loading 2003.08798v3…