PaperlayerView exact revision on arXiv

Source-linked AI summary

Optimizing Video Object Detection via a Scale-Time Lattice

Kai Chen, Jiaqi Wang, Shuo Yang, Xingcheng Zhang, Yuanjun Xiong, Chen Change Loy, Dahua Lin

arXiv:1804.05472v1cs.CV

TL;DR

Video object detection must balance accuracy with the computational demands of real-time use, and cost-effective ways to combine detection with propagation and refinement remain an open question. The paper introduces a Scale-Time Lattice that reallocates computation across time and scale. It reports 79.6 mAP at 20 fps on ImageNet VID, with a 79.0 mAP at 62 fps tradeoff.

  • Problem

    Combining image detection, temporal propagation, and coarse-to-fine refinement cost-effectively remains an open question in video object detection.

    Source: §1 · p. 1

  • Method

    The Scale-Time Lattice unifies sparse detection with temporal propagation and across-scale refinement as operations over scale and time.

    Source: §1 · pp. 1–2, §3 · p. 3

  • Results

    79.6 mAP at 20 fps on ImageNet VID, or 79.0 mAP at 62 fps as a performance-speed tradeoff.

    Source: §5.2 · p. 6

  • Takeaways & Limitations

    The framework offers a flexible design space for balancing video-detection performance and computational cost.

    Source: §6 · p. 8

    The fast D&T result is evaluated on sparse key frames; its full all-frame performance is expected to be lower if other frames are interpolated.

    Source: §5.2 · p. 6, §5.3 · p. 6

Abstract

from arXiv · show

High-performance object detection relies on expensive convolutional networks to compute features, often leading to significant challenges in applications, e.g. those that require detecting objects from video streams in real time. The key to this problem is to trade accuracy for efficiency in an effective way, i.e. reducing the computing cost while maintaining competitive performance. To seek a good balance, previous efforts usually focus on optimizing the model architectures. This paper explores an alternative approach, that is, to reallocate the computation over a scale-time space. The basic idea is to perform expensive detection sparsely and propagate the results across both scales and time with substantially cheaper networks, by exploiting the strong correlations among them. Specifically, we present a unified framework that integrates detection, temporal propagation, and across-scale refinement on a Scale-Time Lattice. On this framework, one can explore various strategies to balance performance and cost. Taking advantage of this flexibility, we further develop an adaptive scheme with the detector invoked on demand and thus obtain improved tradeoff. On ImageNet VID dataset, the proposed method can achieve a competitive mAP 79.6% at 20 fps, or 79.0% at 62 fps as a performance/speed tradeoff.

1. Introduction

Video object detection must balance detection quality with real-time constraints, but combining detection, propagation, and refinement cost-effectively remains an open question. The paper proposes a flexible Scale-Time Lattice that sparsely applies expensive detection and uses cheaper networks to propagate and refine results.

1. Introduction

  • Expensive CNN-based per-frame detection and temporal post-processing can be too slow and resource-demanding for real-time systems.The introduction gives 7 fps on a Titan X as an example for a competitive Faster R-CNN-based detector.Source: §1 · p. 1
  • Prior work explored model compression and temporal propagation, but the performance-cost tradeoff remained limited.The paper cites bounding-box tracking and feature warping as examples of temporal propagation approaches.Source: §1 · p. 1
  • The open problem is how to combine image-based detection, temporal propagation, and coarse-to-fine refinement cost-effectively.Earlier work improved individual steps, but the paper frames their cost-effective combination as unresolved.Source: §1 · p. 1
  • The Scale-Time Lattice provides a unified view of detection steps and a flexible design space for reallocating computation on demand.It represents the steps as directed links between scale-time positions, making cost distribution across them visible.Source: §1 · pp. 1–2, Figure 1 · p. 1
  • The framework combines sparse key-frame detection with cheaper propagation and refinement, reporting competitive ImageNet VID performance at 20 fps against the 2017 challenge-winning method [32] [33].The introduction describes adaptive key-frame selection and motion-history-based propagation as components of its design.Source: §1 · p. 2

2. Related Work

Related work spans image detectors, video-specific detection and propagation, and coarse-to-fine refinement. The paper distinguishes its approach by combining temporal and spatial refinement in a scale-time lattice.

Object detection in images

  • Image detectors use two-stage or single-stage paradigms, with single-stage methods often more efficient but less accurate.Examples include Faster R-CNN [26] among two-stage methods and SSD [21], YOLO [24], and RetinaNet [20] among single-stage methods.Source: §2 · p. 2

Object detection in videos

  • Video methods include post-processing, tracking, and sparse-frame feature propagation, with reported tradeoffs in runtime and accuracy.Optical-flow propagation in [33] achieved 10× speedup with a 4.4% mAP drop; feature aggregation in [32] ran at around 1 fps.Source: §2 · p. 2

Coarse-to-fine approaches

  • Earlier coarse-to-fine approaches mainly refine results from low to high resolution through cascaded structures.The paper says its approach extends coarse-to-fine behavior across both spatial and temporal dimensions.Source: §2 · p. 2

3. Scale-Time Lattice

The Scale-Time Lattice unifies sparse image detection and dense video detection as operations across spatial scales and time. Detection results travel along chosen paths to produce fine-scale outputs covering every time point.

3. Scale-Time Lattice

  • The video detection objective is precise localization in each frame while meeting runtime requirements.The proposed framework unifies sparse image-based detection with dense video-result construction.Source: §3 · pp. 2–3
  • The lattice is a directed acyclic graph whose nodes hold detection results at a spatial resolution and time, and whose edges encode operations between nodes.Temporal propagation links adjacent time steps at the same scale; spatial refinement links neighboring scales at the same time.Source: §3 · p. 3, Figure 2 · p. 3
  • The pipeline generates detections on sparse key frames, plans paths from those inputs to dense output nodes, then propagates and refines results.Detection accuracy is measured at the output nodes.Source: §3 · p. 3
  • The framework seeks a cost-performance balance by allocating computation across lattice components; more key frames can improve performance but add substantial cost.Cheaper propagation and refinement networks can carry results over long paths while maintaining competitive accuracy.Source: §3 · p. 3
  • The lattice proceeds from coarse to fine in both time and space, narrowing temporal intervals while increasing spatial resolution.Its refinement process forms a two-dimensional scale-time space for gradually denser and more precise boxes.Source: §3 · p. 3

4. Technical Design

The framework connects sparse detector outputs through repeated propagation-and-refinement units, then adds adaptive key-frame selection and tube-level rescoring.

4. Technical Design

  • Propagation and Refinement Units connect key-frame detections across time and scales, and their outputs are integrated into tubes for tube-level classification.The repeated units propagate results across time and refine them across scales before output-node detections are integrated.Source: §4 · p. 4

4.1. Propagation and Refinement Unit (PRU)

  • Each PRU propagates detections from two key frames toward the intermediate frame, merges the estimates, and refines boxes across scales from coarse to fine.Propagation uses motion-history images and grayscale frames with an RoIAlign-based regressor; refinement regresses box offsets against the intermediate image.Source: §4.1 · p. 4

4.2. Key Frame Selection and Path Planning

  • Key-frame density adapts to object size and motion, adding a midpoint when propagation easiness is low; paths use two propagation/refinement stages before linear interpolation.The easiness measure uses matched detections, and the selection procedure is conducted for one pass in the reported experiments.Source: §4.2 · p. 5

4.3. Tube Rescoring

  • A tube-level classifier reclassifies linked box sequences, then adjusts box scores so hard positives can be boosted and false positives suppressed.The classifier’s tube score and label are used in the box-score adjustment.Source: §4.3 · p. 5

5. Experiments

On ImageNet VID, the method balances detection performance and runtime through sparse key-frame detection and propagation. Ablations compare propagation, refinement, and key-frame selection choices within that design.

5.1. Experimental Setting

  • The ImageNet VID evaluation uses 3,862 training videos and 555 validation videos, measuring detection performance with mAP.The base detector is Faster R-CNN with a ResNet-101 backbone; its reported mAP is 74.5.Source: §5.1 · pp. 5–6

5.2. Results

  • 79.6 mAP at 20 fps is competitive with D&T’s 79.8 at about 5 fps, while the 62 fps configuration achieves 79.0 mAP.The cost/performance comparison also reports ST-Lattice outperforming naїve interpolation, DFF, D&T, TPN+LSTM, and FGFA.Source: Figure 4 · p. 6, §5.2 · p. 6

5.3. Ablation Study

  • At a small key-frame interval, the MHI-based propagation method surpasses per-frame detection; its main gains are on fast-moving objects.The ablation fixes the key-frame interval at 24 unless otherwise indicated and evaluates propagated intermediate-frame results without refinement or rescoring.Source: §5.3 · pp. 6–7, Figure 7 · p. 7
  • The two-stage PRU that separately regresses motion and offset outperforms overall-offset and single-step variants, while joint training improves results over separate training.Joint training propagates refinement-component gradients back to the propagation component, improving first-step mAP.Source: §5.3 · p. 7, Table 1 · p. 8, Figure 8 · p. 8
  • Performance rises with deeper propagation and refinement networks, but added computation yields more mAP in spatial refinement than in temporal propagation.With the other network medium, scaling spatial refinement from small to large gains 1.2 mAP (72.5 →73.7); the same added cost on FT gains 0.8 mAP (72.7 →73.5).Source: §5.3 · pp. 7–8, Table 2 · p. 8
  • Adaptive key-frame selection outperforms uniform sampling for ST-Lattice, especially at larger intervals, by assigning harder samples to per-frame detection.Detections on adaptively selected versus uniformly sampled key frames have mAP values of 73.3 versus 74.1, respectively.Source: §5.3 · p. 8, Figure 9 · p. 8

6. Conclusion

The Scale-Time Lattice unifies detection, temporal propagation, and across-scale refinement while offering a flexible space for balancing performance and cost. The paper reports competitive performance against state-of-the-art video detectors at much faster speed and notes potential applications beyond detection.

6. Conclusion

  • The framework achieves competitive performance against state-of-the-art video object detectors at much faster speed.Source: §6 · p. 8
  • The Scale-Time Lattice integrates detection, temporal propagation, and across-scale refinement in a flexible framework for balancing performance and cost.Source: §6 · p. 8
  • The framework may also apply to video object segmentation and tracking.Source: §6 · p. 8

Related papers

Loading 1804.05472v1…