PaperlayerView exact revision on arXiv

Source-linked AI summary

Relation Distillation Networks for Video Object Detection

Jiajun Deng, Yingwei Pan, Ting Yao, Wengang Zhou, Houqiang Li, Tao Mei

arXiv:1908.09511v1cs.CV

TL;DR

Video object detection needs reliable object relations from both the current frame and supportive frames, but modeling these interactions is costly and vulnerable to invalid proposals. RDN uses multi-stage reasoning with progressive relation distillation to augment reference-object features, achieving superior ImageNet VID results, including 83.8% and 84.7% mAP with linking and rescoring.

  • Problem

    Reliable video object relations must incorporate present-frame and long-range supportive objects, while directly modeling all interactions increases computation, memory demand, and invalid proposals.

  • Method

    RDN aggregates and propagates relations from supportive proposals to reference proposals through stacked multi-stage reasoning and progressive distillation of high-objectness supportive proposals.

  • Results

    RDN improves video object detection and box linking on ImageNet VID, reaching 83.8% and 84.7% mAP with linking and rescoring using ResNet-101 and ResNeXt-101-64×4d, respectively.

  • Takeaways & Limitations

    Modeling spatio-temporal object relations with progressive distillation supports both per-frame detection and box linking across frames.

Abstract

from arXiv · show

It has been well recognized that modeling object-to-object relations would be helpful for object detection. Nevertheless, the problem is not trivial especially when exploring the interactions between objects to boost video object detectors. The difficulty originates from the aspect that reliable object relations in a video should depend on not only the objects in the present frame but also all the supportive objects extracted over a long range span of the video. In this paper, we introduce a new design to capture the interactions across the objects in spatio-temporal context. Specifically, we present Relation Distillation Networks (RDN) --- a new architecture that novelly aggregates and propagates object relation to augment object features for detection. Technically, object proposals are first generated via Region Proposal Networks (RPN). RDN then, on one hand, models object relation via multi-stage reasoning, and on the other, progressively distills relation through refining supportive object proposals with high objectness scores in a cascaded manner. The learnt relation verifies the efficacy on both improving object detection in each frame and box linking across frames. Extensive experiments are conducted on ImageNet VID dataset, and superior results are reported when comparing to state-of-the-art methods. More remarkably, our RDN achieves 81.8% and 83.2% mAP with ResNet-101 and ResNeXt-101, respectively. When further equipped with linking and rescoring, we obtain to-date the best reported mAP of 83.8% and 84.7%.

1. Introduction

Video object detection commonly improves per-frame features or associates boxes across frames, but object relations across video spatio-temporal context remain insufficiently exploited. RDN addresses this by progressively modeling and distilling relations among reference and supportive proposals.

  • 1. Introduction: Video object detection enhances detectors through box-level association or feature aggregation across nearby frames.Box association generates tubelets, while feature aggregation improves per-frame features.
  • 1. Introduction: Object relations capture interactions or geometric positions between objects and have supported recognition, detection, cross-domain detection, and image captioning.Image relation modeling can enhance an object using weighted appearance features from other objects, with weights reflecting object dependency.
  • 1. Introduction: Extending image relation modeling to video must account for reference-frame objects and supportive objects from nearby frames, increasing computation, memory demand, and invalid proposals.These issues can affect the stability of relation learning.
  • 1. Introduction: RDN augments reference-proposal features by aggregating relation features over proposals from support frames through basic and advanced multi-stage reasoning.The basic stage uses all supportive proposals, while the advanced stage selects high-objectness supportive proposals and further refines relations.

2. Related Work

Prior video detectors primarily aggregate features or associate boxes, while cross-frame object relations remain underexploited. RDN extends relation modeling across multiple frames and progressively distills supportive proposals to improve efficiency and relation quality.

  • 2. Related Work: Image object detection uses proposal-based two-stage or proposal-free one-stage detectors, with some methods modeling relations among regions or CNN features.The cited image relation module measures interactions among region proposals.
  • 2. Related Work: Unlike image-only relation modeling, RDN explores object interactions across multiple frames and progressively schedules relation distillation to reduce relation-measurement cost.The design targets complex spatio-temporal context and the large supportive-proposal set in video.
  • 2. Related Work: Video object detection is difficult because spatial and temporal variations can deteriorate object appearances through motion blur or occlusion.Feature aggregation is a common approach for enhancing per-frame features with nearby-frame information.
  • 2. Related Work: Existing feature-level and box-level methods improve quantitative scores, but object relations are not fully exploited across frames for video detection.RDN instead aggregates and propagates object relations across frames to augment object features in a cascaded manner.

3. RDN for Video Object Detection

RDN models object interactions across reference and support frames, progressively distilling relations to augment reference proposals for video object detection and box linking.

  • Proposal construction: RDN generates reference and support-frame proposals with RPN, organizing support proposals into a pool and a high-objectness advanced pool.The advanced pool samples r% of supportive proposals with high objectness scores.
  • Detection: The resulting relation-augmented reference proposals are passed to detection for classification and regression in the reference frame.RDN inference iterates over frames while maintaining a proposal feature buffer.
  • Spatio-temporal relation modeling: RDN extends image relation modeling to video by handling interactions across multiple frames while addressing the computational cost of using all supportive proposals.Directly applying image-based relation modeling would increase computation and memory as support proposals grow.
  • Basic stage: The basic stage stacks relation modules to augment each reference proposal using relation features aggregated from all supportive proposals.Stacking is used because a single relation module is insufficient for complex spatio-temporal interactions.
  • Advanced stage: The advanced stage refines selected supportive proposals against the full supportive pool, then uses them to further strengthen basic-stage reference proposals.Progressive distillation reduces computation and filters invalid supportive proposals for relation reasoning.
  • Box linking with relations: Learnt object relations are also incorporated into box linking to propagate confidence among related detections across frames and improve rescored tube results.The optimal linking path is found with the Viterbi algorithm, followed by rescoring using the average top-50% classification score.

4. Network Architecture

The architecture uses modified backbones, RPN-based region feature extraction, and multi-stage relation modules to process and augment video object proposals.

  • Backbone: ResNet-101 and ResNeXt-101-64×4d backbones use stride 16 feature maps and atrous convolution in the last stage.The final-stage stride changes from 32 to 16 pixels while atrous convolution preserves receptive fields.
  • Region Feature Extraction: RPN region extraction uses 12 anchors across four scales and three aspect ratios before selecting 300 proposals per frame with NMS.The process selects 6,000 highest-objectness proposals and applies NMS at a 0.7 IoU threshold.
  • Relation Distillation Networks: Each relation module produces 16 relation features of dimension 64, yielding a 1,024-dimensional relation-augmented feature.The basic stage stacks two relation modules, while the advanced stage first enhances proposals in the advanced supportive pool.

5. Experiments

Experiments on ImageNet VID validate RDN’s video object detection gains, including improvements from spatio-temporal relation modeling, cascaded distillation, and relation-aware box linking. Additional analyses examine temporal span, module count, and proposal sampling trade-offs.

  • 5.1. Dataset and Evaluation: The evaluation uses ImageNet VID validation mAP, with RDN trained using both VID and overlapping ImageNet DET classes.ImageNet VID contains 3,862 training videos and 555 validation videos across 30 classes; the official test annotations are unavailable.
  • 5.3. Performance Comparison: 81.8% mAP with ResNet-101 surpasses the best end-to-end competitor by 2.9% absolute, while ResNeXt-101-64×4d reaches 83.2%.These comparisons use end-to-end methods without post-processing.
  • 5.3. Performance Comparison: 83.8% and 84.7% mAP are obtained with ResNet-101 and ResNeXt-101-64×4d, respectively, after Box Linking with Relations.BLR integrates learned object relations into Tube Linking by propagating confidence scores among highly related detection boxes.
  • 5.4. Experimental Analysis: 81.8% mAP is achieved by the full RDN, compared with 75.4% for single-frame Faster R-CNN and 80.9% for the basic relation stage.The ablation attributes the additional gain to cascaded relation distillation in the advanced stage.
  • 5.4. Experimental Analysis: T = 18 provides the best temporal-span performance and a practical performance–runtime trade-off.Performance changes little once the span exceeds 12, whereas inference runtime generally increases with span.
  • 5.4. Experimental Analysis: Two basic-stage relation modules reach 80.9% mAP, while proposal sampling is set to K = 75 and r = 20% for favorable performance–runtime trade-offs.More basic-stage modules slightly reduce performance; sampling changes affect performance only slightly but increase runtime, especially in the advanced stage.

6. Conclusions

RDN models object relations across video frames through multi-stage reasoning and progressive relation distillation, improving video object detection. On ImageNet VID, it achieves a best reported mAP of 84.7% after linking and rescoring.

  • RDN models object relations across reference and support frames to augment proposal features for video object detection.It uses RPN proposals, a supportive pool from support frames, and relation-based feature augmentation.
  • RDN combines multi-stage reasoning with progressively scheduled relation distillation.Supportive proposals are refined against other supportive proposals before their relations are distilled into reference-frame proposals.
  • 84.7% mAP is the best reported result achieved by RDN after linking and rescoring on ImageNet VID.
Loading 1908.09511v1…