Source-linked AI summary
Mobile Video Object Detection with Temporally-Aware Feature Maps
Mason Liu, Menglong Zhu
TL;DR
Efficient video object detection must use temporal information while remaining practical for low-powered mobile and embedded devices. The paper integrates convolutional LSTMs into fast single-image detectors, including efficient Bottleneck-LSTM modifications, and reports favorable performance with mobile-oriented computational costs. Its scope is online detection, using only current and preceding frames.
Problem
The paper investigates how to add temporal awareness to video detection while preserving the speed and low resource consumption of efficient single-frame models.
Method
The model interweaves convolutional detection layers with recurrent layers that refine and propagate feature maps across frames, using efficiency-focused Bottleneck-LSTM modifications.
Results
The model compares favorably with efficient single-frame baselines on Imagenet VID 2015 across variants using 200M–1100M multiply-adds and 1–3.5 million parameters.
Takeaways & Limitations
The architecture demonstrates real-time mobile video object detection while incorporating temporal cues and maintaining a lightweight design.
Takeaways & Limitations
The method is evaluated in an online setting, where each detection uses only frames up to the current frame, and optical-flow comparisons involve distinct computational constraints.
Abstract
from arXiv · showhide
This paper introduces an online model for object detection in videos designed to run in real-time on low-powered mobile and embedded devices. Our approach combines fast single-image object detection with convolutional long short term memory (LSTM) layers to create an interweaved recurrent-convolutional architecture. Additionally, we propose an efficient Bottleneck-LSTM layer that significantly reduces computational cost compared to regular LSTMs. Our network achieves temporal awareness by using Bottleneck-LSTMs to refine and propagate feature maps across frames. This approach is substantially faster than existing detection methods in video, outperforming the fastest single-frame models in model size and computational cost while attaining accuracy comparable to much more expensive single-frame models on the Imagenet VID 2015 dataset. Our model reaches a real-time inference speed of up to 15 FPS on a mobile CPU.
1. Introduction
The paper addresses efficient video object detection by adding temporal awareness to fast single-image detectors while preserving mobile and embedded deployment constraints. It uses recurrent layers to refine feature maps across frames and evaluates the resulting architecture on Imagenet VID 2015.
- Motivation: Efficient video detection remains unclear because single-image models are resource-intensive, while video models must additionally exploit temporal cues.The paper targets low computational budgets for mobile and embedded platforms.
- Motivation: Temporal continuity lets earlier frames refine current predictions, while correlated intermediate feature maps provide richer temporal information.The model conditions each frame’s feature maps on corresponding maps from previous frames.
- Method: The architecture combines convolutional layers with convolutional LSTMs to fuse feature-map hypotheses with temporal context and produce refined representations.Recurrent layers propagate temporal cues across frames as the video is processed.
- Evaluation: On Imagenet VID 2015, model variants span 200M–1100M multiply-adds and 1–3.5 million parameters while comparing favorably with efficient single-frame baselines.The variants are described as deployable across mobile and embedded platforms.
- Contributions: The paper introduces a unified online video-detection architecture requiring minimal computational resources and proposes recurrent layers for temporal feature-map refinement.These contributions extend efficiency-focused single-image detectors with temporal awareness.
- Contributions: The authors modify convolutional LSTMs to make recurrent processing compatible with efficiency-focused networks and real-time mobile object detection.This supports the paper’s efficiency-oriented architecture.
2. Related Work
Prior video-detection systems use postprocessing, optical flow, or expensive base networks, whereas this paper integrates temporal context directly into an efficient detector. Its recurrent feature-level design avoids postprocessing and optical-flow requirements while targeting mobile deployment.
- Single-image detection: SSD provides the paper’s base framework because it offers efficiency and competitive single-image detection accuracy.SSD is a single-shot detector that generates predictions in one pass at fixed anchor positions.
- Video detection: Seq-NMS and TCNN organize detections across frames using sequence linking, optical flow, suppression, or tracking, but neither performs online inference or focuses on efficiency.These methods operate through postprocessing of detections.
- Feature-level temporal modeling: The proposed method incorporates temporal context at the feature level without postprocessing or online learning, and existing result-processing methods can still be applied afterward.This makes comparisons closer to the underlying single-frame detector.
- Tracking-based methods: D&T combines tracking and detection but relies on an expensive Resnet-101 base network that is unsuitable for mobile devices.Its experiments focus on expensive high-accuracy models rather than mobile-focused systems.
- Optical-flow methods: Optical-flow approaches warp or aggregate neighboring feature maps, but flow computation and concurrent key-frame detection increase computation, memory, and storage requirements.The paper reports that even the smallest DFF flow network costs approximately twice as much as its larger model and 10× as much as its smaller one.
- Optical-flow methods: Unlike optical-flow systems, the proposed network has fewer parameters than its efficient baseline detector and no additional memory overhead.This supports its mobile-oriented design.
- Recurrent methods: Convolutional LSTMs encode spatial and temporal information, but this paper modifies them for efficiency and integrates them into the base convolutional architecture.Earlier LSTM-based video methods commonly apply recurrent processing after network outputs.
3. Approach
The approach integrates convolutional LSTMs into an SSD-MobileNet detector to propagate and refine feature maps over time for efficient online video detection. A Bottleneck-LSTM, depthwise separable convolutions, and constrained LSTM placement reduce the added computational cost.
- Architecture: The model injects convolutional LSTM layers directly into an SSD-MobileNet detector to encode spatial and temporal information jointly.The network processes frames sequentially and uses recurrent states to represent temporal context.
- Feature Refinement: Each recurrent layer receives a feature map hypothesis, fuses it with temporal context from previous frames, and outputs a refined feature map.The refined output replaces the original feature map in subsequent computations.
- LSTM Placement: LSTMs are placed only after low-spatial-dimension feature maps, because earlier placements create larger input volumes and prohibitive computational cost.The considered locations include the Conv13 layer and SSD feature maps, with several single- and multi-LSTM arrangements.
- Extended Width Multiplier: The architecture uses separate width multipliers for the base network, SSD feature maps, and LSTM layers to control channel dimensions independently.The supplied design sets αbase = α, αssd = 0.5α, and αlstm = 0.25α, making each LSTM output one-fourth the input size.
- Efficient Bottleneck-LSTM: In one architecture, four gate computations are combined into one convolution that computes 1024 gate channels but outputs a 256-channel recurrent state.The table describes a single-LSTM LSTM-SSD architecture with a 256-channel state.
- Efficient Bottleneck-LSTM: The Bottleneck-LSTM first computes an N-channel bottleneck feature map and reuses it in the remaining gates, reducing computation while producing a deeper recurrent design.Depthwise separable convolutions reduce required computation by 8 to 9 times compared with previous convolutional-LSTM definitions.
4. Experiments
Experiments evaluate training, ablations, robustness, final accuracy, and mobile runtime for the proposed video detector. Results show that feature-map recurrence, Bottleneck-LSTMs, and multi-scale placement support efficient and stable detection.
- Experiment Setup: The model is trained on all 3,862 Imagenet VID training videos using 10-frame sequences and evaluated on 11,080 minival frames with mean average precision @0.5 IOU.Efficiency is reported using parameter count and multiply-adds (MAC).
- Ablation Study: LSTM placement after feature maps improves performance, with Conv13 providing the greatest improvement.This result supports adding temporal awareness in the feature space.
- Ablation Study: Bottleneck-LSTMs are an order of magnitude more efficient than other recurrent layers while attaining comparable performance.The comparison includes averaging, LSTMs, and GRUs, with recurrent-layer parameters and multiply-adds measured separately.
- Ablation Study: Accuracy remains near-constant up to α_lstm = 0.25α, then drops as the LSTM output channel dimension is reduced further.Each experiment places a single Bottleneck-LSTM after Conv13.
- Ablation Study: Placing LSTMs after feature maps at different scales is beneficial, whereas stacking two LSTMs after the same feature map is not.The final model places LSTMs after all feature maps and does not further bottleneck the outputs for FM3 and FM4.
- Comparison With Other Architectures: 15 FPS is achieved by the α = 0.5 model on the big core of a Qualcomm Snapdragon 835 in a Pixel 2.The evaluation uses single-threaded inference with a custom on-device TensorFlow implementation.
5. Conclusion
The paper presents a temporally aware framework that unifies mobile SSD models and recurrent networks for efficient video object detection. Its modifications improve efficiency with minimal performance loss, enable real-time mobile inference, and outperform comparable state-of-the-art single-frame models.
- The framework unifies mobile SSD models and recurrent networks into a single temporally aware architecture for mobile video object detection.
- The introduced modifications make the network more efficient with a minimal decrease in performance.
- The network is sufficiently fast to run in real time on mobile devices.
- The method outperforms comparable state-of-the-art single-frame models, indicating benefit from temporal cues in videos.