PaperlayerView exact revision on arXiv

Source-linked AI summary

Context R-CNN: Long Term Temporal Context for Per-Camera Object Detection

Sara Beery, Guanhang Wu, Vivek Rathod, Ronny Votel, Jonathan Huang

arXiv:1912.03538v3cs.CVcs.LGeess.IVq-bio.PE

TL;DR

Sparse, irregularly sampled static-camera footage presents object-detection challenges, even as repeated behavior and stable backgrounds provide potential context. Context R-CNN retrieves per-camera temporal features through attention. It improves over single-frame baselines across camera-trap and traffic-camera data, with a reported 17.9% mAP gain on a camera-trap dataset.

  • Problem

    Passive-monitoring images can be sparse and lack guarantees of framing, focus, lighting, or scale, creating challenges for single-frame detection.

    Source: §1 · p. 1

  • Method

    Context R-CNN uses attention to index a per-camera long-term memory of contextual features, alongside a separate short-term attention mechanism.

    Source: §1 · p. 2, §3.1 · pp. 4–5

  • Results

    Context R-CNN improves detection over single-frame baselines on camera-trap and traffic-camera data; it improves mAP at 0.5 IoU by 17.9% on a commonly used camera-trap dataset.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The approach improves detection across static-camera domains and is described as adaptive to low, irregular frame rates, using context from up to a month.

    Source: §6 · p. 8

    Memory contents and size are domain-specific design concerns, and reducing storage and computational overhead while maintaining performance gains remains future work.

    Source: §6 · p. 8

Abstract

from arXiv · show

In static monitoring cameras, useful contextual information can stretch far beyond the few seconds typical video understanding models might see: subjects may exhibit similar behavior over multiple days, and background objects remain static. Due to power and storage constraints, sampling frequencies are low, often no faster than one frame per second, and sometimes are irregular due to the use of a motion trigger. In order to perform well in this setting, models must be robust to irregular sampling rates. In this paper we propose a method that leverages temporal context from the unlabeled frames of a novel camera to improve performance at that camera. Specifically, we propose an attention-based approach that allows our model, Context R-CNN, to index into a long term memory bank constructed on a per-camera basis and aggregate contextual features from other frames to boost object detection performance on the current frame. We apply Context R-CNN to two settings: (1) species detection using camera traps, and (2) vehicle detection in traffic cameras, showing in both settings that Context R-CNN leads to performance gains over strong baselines. Moreover, we show that increasing the contextual time horizon leads to improved results. When applied to camera trap data from the Snapshot Serengeti dataset, Context R-CNN with context from up to a month of images outperforms a single-frame baseline by 17.9% mAP, and outperforms S3D (a 3d convolution based baseline) by 11.2% mAP.

1. Introduction

Passive monitoring cameras collect sparse, sometimes difficult-to-interpret images, while their static scenes and recurring subject behavior offer useful temporal context. Context R-CNN uses context from the same camera to improve detection, including with irregular sampling and horizons up to a month.

1. Introduction

  • Passive-monitoring images lack guarantees of framing, focus, lighting, and scale, and can contain occluded objects, poor image quality, and salient background distractors.These conditions can cause failures in single-frame detection networks.Source: §1 · p. 1
  • Similar behavior across days and static backgrounds make other frames from the same camera potentially useful context for difficult detections.Subjects may appear days or weeks apart, including along habitual routes.Source: §1 · pp. 1–2
  • Context R-CNN differentiably indexes a long-term memory bank and uses unlabeled examples from a novel camera to improve generalization.The approach is framed as non-parametric estimation on top of a parametric detector.Source: §1 · p. 2
  • The paper applies Context R-CNN to camera traps and traffic cameras, where within-camera context can help detect recurring subjects despite irregular sampling.Examples include animals returning to a trail and buses returning along a route.Source: §1 · p. 2
  • 17.9% improvement in mAP at 0.5 IoU is reported on a commonly used camera-trap dataset, with context extending up to a month.The method also improves over strong single-frame baselines and is described as robust to frame-rate and sampling irregularity.Source: §1 · p. 2

2. Related Work

Prior video-detection methods often assume dense, regular sampling or have limited effective temporal range, while object-level attention offers a route to longer-range context. Context R-CNN adapts that approach to static cameras and extends its use to camera-trap and traffic-monitoring tasks.

2. Related Work

  • Flow-based, 3D-convolution, and recurrent video methods are often poorly suited to sparse, irregular sampling; recurrent models may also have short effective temporal receptive fields.Attention-based architectures have been explored for longer-range dependencies [3] [12] [28] [38] [39].Source: §2 · p. 3
  • Unlike prior object-level attention work, Context R-CNN combines a single-frame detector with long-term memory and dedicated short-term attention, and can use negative examples to learn background classes.The cited methods include object-level aggregation across time; Wu et al. use feature banks extending up to 60 seconds in each direction.Source: §2 · p. 3
  • Earlier camera-trap methods use temporal information but do not provide per-image species identification and localization for multi-species bursts, limiting per-image counts for biologists.Prior work also reports that detection can improve generalization to new camera locations.Source: §2 · p. 3
  • Prior traffic-camera temporal methods focus on short-term horizons rather than long-term context.Other traffic-camera research has focused on counting vehicles or people in images.Source: §2 · p. 3

3. Method

Context R-CNN extends single-frame detection with short- and long-term attention over features from the same camera. Its memory banks represent nearby proposals and longer-horizon detections, which attention aggregates into current-frame features.

3. Method

  • Context R-CNN routes current-frame proposals through sequential, modular short- and long-term attention before Faster R-CNN’s second stage uses the context-informed features.The architecture is built on single-frame detection because sparse, irregular frame rates make typical temporal architectures poorly suited to this setting.Source: Figure 3 · p. 4, §3 · p. 4

3.1. Building a memory bank from context features

  • The long-term bank stores pooled, spatiotemporally encoded detection features from a chosen camera-specific time horizon, using curated proposals and a frozen pretrained detector.The described strategies allow up to 8,500 contextual features, sufficient for a month of context in these datasets.Source: §3.1 · pp. 4–5
  • Short-term memory contains pooled proposal features from all boxes in a small window around the current frame, typically up to five frames.Unlike the curated long-term bank, short-term features use the trained first-stage extractor and are not curated for the small windows used.Source: §3.1 · p. 5

3.2. Attention module architecture

  • Each attention block uses current-frame proposal features to attend over context features, forms a projected weighted sum, and adds that context back to the input features.The model uses separate parameters for short- and long-term attention.Source: §3.2 · p. 5

4. Data

The evaluation covers species detection in camera traps and vehicle detection in traffic cameras, using held-out camera locations for testing.

  • Snapshot Serengeti uses seasons 1–6 across 225 cameras, 3.2M images, and 48 classes; Caltech Camera Traps has 140 cameras, 243K images, and 18 classes.Source: §4 · p. 6
  • Camera-trap bursts contain 1–10 frames at 1 fps, and the datasets include false motion triggers in 75% of Snapshot Serengeti images and 50% of Caltech Camera Traps images.Evaluation uses human-labeled validation images from held-out locations: 64K images across 45 locations for Snapshot Serengeti and 62K across 40 for Caltech Camera Traps.Source: §4 · p. 6
  • CityCam covers 10 vehicle classes, around 60K frames, and 900K annotated objects from 17 cameras; 13 camera locations train the model and four are held out for testing.Source: §4 · p. 6

5. Experiments

Across three camera datasets, Context R-CNN improves on single-frame detection, while experiments on Snapshot Serengeti examine temporal attention, memory horizons, and memory features. Longer temporal horizons improve performance, and empty-frame representations reduce false positives.

5. Experiments

  • Experiments evaluate held-out camera locations with mAP@0.5 and AR, focusing mainly on SS and testing short- and long-term attention, feature extraction, memory horizons, and sampling.The study also compares three datasets and explores multiple stored features per frame on CityCam.Source: §5 · p. 6

5.1. Main Results

  • Absolute mAP at 0.5 IoU improves 19.5% on CCT, 17.9% on SS, and 4.5% on CC; AR@1 improves 8.5% on CCT, 11.8% on SS, and 2% on CC.Results use Faster R-CNN with a ResNet-101 backbone; AR is reported for the top prediction.Source: §5.1 · p. 6, Table 1 · p. 6
  • S3D exceeds the single-frame baseline by 6.8% mAP, while spatial feature averaging gains 1.9% and majority voting does not improve it.These short-term comparisons use a three-frame input window.Source: §5.1 · p. 6
  • 55.9% mAP is the best SS result, from combining short- and long-term attention; the model improves performance for all classes.The single-frame non-local attention variant gains 7% mAP, three-frame short-term attention adds 1.5%, and month-long attention improves 9.2% over short-term attention.Source: §5.1 · p. 7, Figure 5 · p. 7

5.2. Changing the Time Horizon (Table 1(b))

  • Performance increases as the long-term memory horizon grows, and attention spans the available horizon while tending to favor temporally nearby images.Week- and month-long memories show day/night periodicity in attended frames.Source: §5.2 · p. 7
  • SS memory features fine-tuned on its training set achieve 5.3% higher mAP than COCO-only features, while COCO-only memory features still improve on the single-frame model by 12.4%.At a one-month horizon, halving memory with stride 2 drops mAP@0.5 by 3.1%; stride 4 causes an additional 1.7% drop.Source: §5.3 · p. 7
  • Adding empty-image representations reduces false positives across confidence thresholds; 97 of the 100 highest-confidence detections on labeled-empty images were missed animals.The authors note that these missed labels make estimated performance improvements likely conservative.Source: §5.3 · p. 8
  • For SS, storing only the top-scoring detection is suited to mostly single-object images, whereas CityCam performs best when memory stores eight detections per frame.The paper also identifies a possible hallucination failure case: one strongly misclassified image may influence other detections at that camera.Source: §5.3 · p. 8

6. Conclusions and Future Work

Context R-CNN uses per-camera temporal context for detection across static-camera domains and irregular sampling. The authors identify memory curation as future work because memory contents and size matter by domain.

6. Conclusions and Future Work

  • Context R-CNN uses up to a month of per-camera temporal context and improves detection over single-frame baselines on camera-trap and traffic-camera data.The method is described as adaptive and robust to low, irregular frame rates.Source: §6 · p. 8
  • Attention-based temporal context is particularly beneficial in the static-camera setting.Source: §6 · p. 8
  • Memory contents and size are domain-specific, motivating future work on banks optimized for accuracy and size.The stated goal is to reduce computational and storage overhead at training and inference while maintaining performance gains.Source: §6 · p. 8

7. Acknowlegdements

The authors acknowledge colleagues and the Wildlife Insights Team for their valuable insight and note NSFGRFP support.

7. Acknowlegdements

  • The authors thank Pietro Perona, David Ross, Zhichao Lu, Ting Yu, Tanya Birch, the Wildlife Insights Team, Joe Marino, and Oisin MacAodha for valuable insight.Source: §7 · p. 8
  • The work was supported by NSFGRFP Grant No. 1745301.Source: §7 · p. 8
  • The authors state that the views are theirs and do not necessarily reflect those of the NSF.Source: §7 · p. 8

A. Implementation Details

The experiments use a Faster R-CNN with a ResNet 101 backbone and attention modules, trained on TPUs with clips of four adjacent frames.

A. Implementation Details

  • The attention modules were implemented in the TensorFlow Object Detection API Faster R-CNN architecture with a ResNet 101 backbone.Faster R-CNN optimization and parameters were held constant against the single-frame baseline, with hyperparameter sweeps for robust baselines.Source: A. Implementation Details · p. 12
  • Training batches used 32 clips, each drawing four frames spaced one frame apart, resized to 640 × 640, and distributed across eight TPU cores.Frames from the same clip were colocated, and random flipping was applied consistently to memory banks and current frames.Source: A. Implementation Details · p. 12
  • Training used MomentumSGD with weight decay 0.0004, momentum 0.9, and attention softmax temperature T = .01.The authors report that T = .01 outperformed .1 and 1 in early experiments.Source: A. Implementation Details · p. 12

B. Dataset Statistics and Per-Class Performance

The three datasets have long-tailed class distributions, and per-class results generally improve with Context R-CNN, with one stated CityCam exception. The spatiotemporal encodings normalize date and object-position information.

B. Dataset Statistics and Per-Class Performance

  • Snapshot Serengeti, Caltech Camera Traps, and CityCam have long-tailed class distributions, and rare classes are those not well-represented during training.Rare classes frequently prove difficult to recognize.Source: B. Dataset Statistics and Per-Class… · p. 12
  • Per-class performance improves for all Caltech Camera Traps classes and universally for Snapshot Serengeti, while CityCam's “Middle Truck” class performs slightly worse.The authors describe “Middle Truck” as relatively ambiguous because “middle” size is not well-defined; Figure 8 reports per-class mAP averaged across IoU thresholds.Source: B. Dataset Statistics and Per-Class… · p. 12, Figure 8 · p. 12

C. Spatiotemptoral Encodings

  • The encodings represent year, month, day, hour, and minute as normalized values, using 1990–2030 for the year and fixed calendar divisors for the other fields.The month, day, hour, and minute divisors are 12, 31, 24, and 60, respectively.Source: C. Spatiotemptoral Encodings · p. 12
  • Object center coordinates, width, and height are normalized using the corresponding image dimensions.Source: C. Spatiotemptoral Encodings · p. 12

D. Camera Movement

Context R-CNN leverages the static-camera assumption through its memory bank without requiring the camera to remain fixed. The system can still use context after a camera shift.

D. Camera Movement

  • The system has no hard requirement that a camera remain static, but leverages the static-camera assumption through its memory bank.Its memory bank supplies appropriate and relevant context.Source: D. Camera Movement · pp. 12–13
  • After a camera shift, Context R-CNN can still use contextual information across images from the same camera.Figure 9 shows before-and-after images from August 8 and 9, 2010.Source: Figure 9 · p. 13, D. Camera Movement · p. 13
  • The authors report robustness to moved static cameras, unlike traditional background modeling approaches.Source: D. Camera Movement · p. 13

E. Attention Visualization

The visualizations show attention adapting across time to relevant examples, including objects of the same class and unlabeled background categories. Attention can also group visually similar background objects that are not the same instance.

E. Attention Visualization

  • Attention adapts across time to relevant information and consistently focuses on objects of the same class.The visualizations show examples from Snapshot Serengeti and Caltech Camera Traps, with the highest-weight images and boxes highlighted.Source: E. Attention Visualization · p. 13, Figure 11 · p. 14
  • On unseen camera locations, the model attends to unlabeled background classes such as bushes and rocks, learning when to ignore them.Examples include a bush in Snapshot Serengeti and a rock in Caltech Camera Traps.Source: E. Attention Visualization · pp. 13–14, Figure 12 · p. 14
  • Background attention can span different instances of a visually similar class, rather than only the same object instance.The authors connect this pattern to geographic clustering of plant species and rock types.Source: E. Attention Visualization · p. 13
Loading 1912.03538v3…