PaperlayerView exact revision on arXiv

Source-linked AI summary

Video Object Detection with an Aligned Spatial-Temporal Memory

Fanyi Xiao, Yong Jae Lee

arXiv:1712.06317v3cs.CV

TL;DR

Video object detection must use temporal and motion information while coping with limited diversity in redundant video datasets. STMN introduces a pretrained-compatible spatial-temporal recurrent memory and MatchTrans alignment, outperforming ConvGRU and achieving state-of-the-art ImageNet VID results.

  • Problem

    Video detectors must handle temporal and motion variation, while contemporary video datasets have limited intra-category diversity because their frames are highly redundant.

  • Method

    STMN uses an STMM recurrent unit that integrates pretrained image-classification weights and preserves spatial memory, with MatchTrans aligning memory across frames.

  • Results

    STMN outperforms ConvGRU and achieves state-of-the-art results on ImageNet VID across different base detectors and backbone networks.

  • Takeaways & Limitations

    The STMM and MatchTrans design choices provide benefits in video object detection, including pretrained-weight integration and spatial alignment across time.

Abstract

from arXiv · show

We introduce Spatial-Temporal Memory Networks for video object detection. At its core, a novel Spatial-Temporal Memory module (STMM) serves as the recurrent computation unit to model long-term temporal appearance and motion dynamics. The STMM's design enables full integration of pretrained backbone CNN weights, which we find to be critical for accurate detection. Furthermore, in order to tackle object motion in videos, we propose a novel MatchTrans module to align the spatial-temporal memory from frame to frame. Our method produces state-of-the-art results on the benchmark ImageNet VID dataset, and our ablative studies clearly demonstrate the contribution of our different design choices. We release our code and models at http://fanyix.cs.ucdavis.edu/project/stmn/project.html.

1 Introduction

Video object detection must handle challenges absent or amplified in static images while using temporal and motion information during learning. STMN addresses these limitations with spatial-temporal memory, pretrained image-detector integration, and explicit frame-to-frame alignment.

  • Video detectors are important because robots, surveillance systems, vehicles, and wearable devices receive videos and must recognize objects and interactions accurately.
  • Static image detectors struggle with motion blur, low resolution, compression artifacts, occlusion, and extreme viewpoints, while ignoring useful temporal information.
  • Post-processing approaches link detections or refine results after static detection, leaving temporal information unused during detector training and making consecutive failures difficult to overcome.
  • Fixed-window methods have difficulty modeling variable, long-term information, while vector memories lose spatial information and require region-level compensation.
  • STMN jointly models and aligns long-term appearance and motion dynamics end-to-end using the convolutional recurrent STMM, which integrates pretrained static-image weights.
  • STMN outperforms ConvGRU and achieves state-of-the-art results on ImageNet VID across different base detectors and backbone networks.
  • The STMM preserves spatial information and uses MatchTrans to align convolutional features across frames, enabling localized multi-frame region features through ROI pooling.

2 Related work

Related video-detection methods use tracking, fixed-window feature aggregation, box displacement prediction, or vector memories, whereas STMN learns spatial-temporal memory for long and variable sequences.

  • Static-image detection established strong end-to-end deep-learning frameworks, but this work targets the less-studied problem of detecting objects in videos.
  • Many video detectors use two-stage pipelines that track or refine per-frame static detections, requiring heuristic post-processing choices during testing.
  • Feed-forward methods combine neighboring-frame features during training but aggregate information over only small, fixed numbers of frames.
  • STMN carries spatial-temporal information across long and variable numbers of frames, unlike fixed-window alternatives.
  • Vector-form recurrent memories lose spatial information and can require separate region-tube memories, making such approaches potentially slow.
  • Unlike Detect and Track, STMN aggregates information across more than two frames and warps entire feature maps for coarse pixel-level alignment.
  • STMN differs from prior ConvGRU applications by classifying bounding boxes, using pretrained static-image weights, and aligning memory frame-to-frame with MatchTrans.
  • The architecture forwards consecutive frames through convolutional stacks, feeds spatial-preserving maps into STMM, and sends the center-frame output to classification and box regression.

3 Approach

STMN uses recurrent spatial-temporal memory to aggregate appearance across frames while preserving spatial information, and MatchTrans aligns that memory as objects move. The design integrates pretrained image-detector weights and supports bidirectional temporal context for video detection.

  • STMN models changing object appearance and motion over time using a Spatial-Temporal Memory Network for video object detection.
  • Spatial-temporal memory module: The STMM recurrent unit receives each frame’s convolutional features and prior memory, then updates memory carrying information from earlier frames.
  • Spatial-temporal memory module: Two STMMs process frames in opposite directions, and their memories are concatenated to provide temporally modulated memory for each frame.
  • Spatial-temporal memory module: The concatenated memory preserves spatial information and feeds classification and bounding-box regression using current and temporally neighboring frames.
  • Spatial-temporal memory module: STMM gates mask prior memory for forgetting and combine it with candidate memory; BN* constrains gate values to [0, 1].The gates are generated from affine transformations of current features and prior memory followed by ReLU.
  • Spatial-temporal memory alignment: MatchTrans matches current-frame features to a local neighborhood in the previous frame to transform and align prior spatial-temporal memory.Without alignment, moved objects leave saliency trails from overlapping unaligned maps; aligned memory is cleaner and produces better detections.
  • Approach summary: The method aggregates and aligns useful information from temporally nearby frames while integrating pretrained static-image detector weights into the memory processing unit.This addresses limited intra-category diversity in contemporary video datasets and enables transfer from large-scale image datasets.

4 Results

The STMN detector achieves strong ImageNet VID results across base detectors and backbones, while ablations show benefits from spatial alignment, pretrained weights, and recurrent design. Qualitative examples indicate improved robustness to blur, difficult viewpoints, occlusion, and crowded scenes.

  • Comparison to state-of-the-art: +7.1% mAP over the static-image R-FCN detector demonstrates the effectiveness of STMN’s spatial-temporal memory.STMN also achieves the best performance among existing video object detection methods in the reported comparison.
  • Comparison to state-of-the-art: STMN outperforms all existing methods using the same base detector and backbone in both R-FCN+ResNet-101 and Fast-RCNN+VGG-16 settings.The comparison is reported on ImageNet VID mAP.
  • Comparison to state-of-the-art: 61.7% val mAP with Fast-RCNN+VGG-16 exceeds the corresponding static-image detector’s 52.2% val mAP.This result demonstrates generalization across different base detectors and backbone networks.
  • Qualitative results: STMN detections are more robust to motion blur, difficult viewpoints, occlusion, and crowded scenes than static-image R-FCN detections.The examples also show more consistent detections across frames and use neighboring easier frames for difficult cases.
  • Contribution of STMN components: 1.7% test mAP improvement over STMN-No-MatchTrans shows the value of spatial alignment across frames.The ablation uses Fast-RCNN with VGG-16 and omits data augmentation.
  • Contribution of STMN components: 44.8% test mAP for ConvGRU-FreshFC rises to 48.0% with pretrained FC weights and 50.7% for the full STMN.The results support using pretrained weights throughout the feature stack and prediction head together with an appropriate recurrent computation.
  • Temporal window analysis: Test mAP differences relative to window size 7 are -1.9%, 0.0%, +0.7%, and +1.0% for window sizes 3, 7, 11, and 15.Performance tends to increase as the test window becomes longer.
  • Computational overhead: Processing 11 frames of size 337x600 takes 0.83 seconds for STMN versus 0.52 seconds for R-FCN, adding 0.028 seconds per frame.The added time includes STMM computation and MatchTrans alignment on a Titan X GPU.

5 Conclusion

The paper concludes that STMN combines pretrained-weight integration with spatial memory alignment to achieve state-of-the-art ImageNet VID results. It also suggests potential utility for video tasks requiring accurate spatial information.

  • Conclusion: STMN combines a recurrent unit that integrates pretrained image-classification weights with an in-network module that aligns memory spatially across time.Together, these design choices lead to state-of-the-art results on ImageNet VID.
  • Conclusion: STMN may also be useful for action detection and keypoint detection, which require accurate spatial information.This is presented as a potential application beyond video object detection.
Loading 1712.06317v3…