PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Feature Flow for Video Recognition

Xizhou Zhu, Yuwen Xiong, Jifeng Dai, Lu Yuan, Yichen Wei

arXiv:1611.07715v2cs.CV

TL;DR

Per-frame evaluation of image-recognition CNNs is too slow and costly for video recognition. Deep feature flow runs the expensive network only on sparse key frames and propagates features through a flow field, achieving substantial speedups while retaining moderate accuracy loss. The framework is evaluated on video detection and segmentation and supports end-to-end joint training.

  • Problem

    Applying image-recognition networks independently to every video frame creates unaffordable computational cost, despite the need for fast and accurate video recognition.

  • Method

    Deep feature flow evaluates the recognition network on sparse key frames and propagates deep feature maps to other frames through a flow field.

  • Results

    The approach achieves up to 10× faster, real-time frame rates than per-frame evaluation, with moderate accuracy loss of a few percent.

  • Takeaways & Limitations

    Deep feature flow provides a fast, accurate, general, and end-to-end framework for practical video recognition across object detection and semantic segmentation.

  • Takeaways & Limitations

    The practical recommendations are heuristic rather than general because they were observed on only two tasks and limited data.

Abstract

from arXiv · show

Deep convolutional neutral networks have achieved great success on image recognition tasks. Yet, it is non-trivial to transfer the state-of-the-art image recognition networks to videos as per-frame evaluation is too slow and unaffordable. We present deep feature flow, a fast and accurate framework for video recognition. It runs the expensive convolutional sub-network only on sparse key frames and propagates their deep feature maps to other frames via a flow field. It achieves significant speedup as flow computation is relatively fast. The end-to-end training of the whole architecture significantly boosts the recognition accuracy. Deep feature flow is flexible and general. It is validated on two recent large scale video datasets. It makes a large step towards practical video recognition.

1. Introduction

Deep feature flow addresses the high cost of per-frame video recognition by applying convolutional networks sparsely and propagating deep features between nearby frames. The framework uses flow-based propagation and end-to-end training to pursue faster recognition with moderate accuracy loss.

  • Motivation: Per-frame application of image-recognition CNNs makes video recognition computationally unaffordable for many applications.Fast and accurate video recognition is important for autonomous driving and video surveillance.
  • Approach: Deep feature flow applies the recognition network on sparse key frames and propagates their deep feature maps to other frames through a flow field.Nearby-frame feature maps can remain similar, while deep features preserve spatial correspondence with image content.
  • Efficiency and training: Flow estimation and feature propagation are much faster than computing convolutional features, avoiding the main computational bottleneck.When flow is estimated by a network, the flow and recognition components are optimized jointly for recognition.
  • Efficiency and training: End-to-end training significantly boosts recognition accuracy while retaining the framework’s fast and general design.The approach can adopt most state-of-the-art image-recognition networks in the video domain.
  • Results: Up to 10× faster, real-time frame rates are achieved compared with per-frame evaluation, with moderate accuracy loss of a few percent.The framework was validated on video object detection and semantic segmentation using large-scale video datasets.

2. Related Work

Related work addresses video recognition through feature reuse, network acceleration, optical flow, or temporal modeling. Deep feature flow instead uses temporal coherence and spatial feature correspondence to reduce computation while retaining single-frame network operation.

  • Image recognition: Image-recognition networks remain computationally unaffordable when applied to every video frame, motivating an efficient video-domain solution.Deep feature flow is presented as an effective and efficient alternative for video recognition.
  • Network acceleration: Single-image acceleration methods use matrix factorization or weight quantization and are generic and complementary to deep feature flow.These approaches reduce computation within network layers rather than exploiting temporal redundancy.
  • Optical flow: Optical-flow research spans handcrafted variational and matching methods as well as deep networks for motion estimation.Deep feature flow exploits optical flow to speed up general video recognition tasks.
  • Temporal information: Temporal video-recognition methods use volume data or long-range spatial-temporal modeling, often improving accuracy while greatly increasing computational cost.Deep feature flow instead exploits temporal coherence while continuing to run the network on single frames.
  • Temporal coherence: Slow Feature Analysis motivates exploiting the smoother evolution of high-level semantic features across consecutive video frames.The paper conjectures that deep feature flow may benefit from this temporal smoothness.
  • Feature reuse: Clockwork Convnets reuse previous features by disabling selected layers on some video frames, whereas deep feature flow propagates features using frame correspondence.The paper describes Clockwork as simpler and less effective, with lower speedup and narrower applicability.

3. Deep Feature Flow

Deep feature flow accelerates video recognition by computing deep features on sparse key frames and propagating them to other frames. Its differentiable flow-and-scale design supports end-to-end training and sparse-frame annotations.

  • Deep Feature Flow: Deep feature flow decomposes recognition into feature and task subnetworks, applying the feature network to key frames and the task network to propagated maps.The feature network is fully convolutional, while the task network performs recognition over intermediate feature maps.
  • Feature Propagation: Non-key-frame features are obtained by warping the preceding key-frame maps with a flow field, using bilinear interpolation at fractional offsets.The flow projects current-frame locations back to key-frame locations, and the warping kernel is bilinear.
  • Feature Propagation: A scale field modulates propagated feature amplitudes to compensate for inaccuracies from flow estimation and object occlusion.The scale field is produced from the current and key frames by a scale function and applied element-wise during propagation.
  • Training: The propagation function is parameter-free and differentiable, allowing gradients to reach the source features, scale field, and flow field during training.The flow function can be hand-crafted or learned; when learned, flow and recognition components are jointly optimized.
  • Training: DFF training can use all frames associated with annotated targets, unlike per-frame training that can use only annotated frames when labels are sparse.This allows the method to exploit unannotated neighboring frames in datasets with sparse ground-truth annotations.
  • Inference Complexity: The non-key-frame cost is dominated by the flow network because task-network, warping, and scale-function complexities are much smaller.Overall speedup also depends on key-frame sparsity, with one key frame scheduled every l consecutive frames.

4. Network Architectures

The architecture combines ResNet feature networks, FlowNet-based flow estimation, and task-specific heads for semantic segmentation and object detection. These components are configured to preserve dense feature maps while controlling computational complexity.

  • Flow Network: The default flow component is FlowNet Simple, with FlowNet Half and FlowNet Inception providing lower-complexity alternatives.FlowNet Half halves convolutional kernels, while FlowNet Inception reduces complexity to 1/8 of FlowNet.
  • Flow Network: Flow networks are pretrained on Flying Chairs, operate at output stride 4, and are down-sized to match the feature-map resolution.Bilinear interpolation aligns the flow and scale fields with feature maps whose stride is 16.
  • Feature Network: The default feature network uses ImageNet-pretrained ResNet-101 or ResNet-50 with feature stride reduced from 32 to 16.The modified network uses dilation and an appended convolution to produce 1024-dimensional intermediate feature maps.
  • Semantic Segmentation: Semantic segmentation applies a 1 × 1 convolution to intermediate maps to produce C+1 score maps followed by per-pixel softmax probabilities.The task network therefore contains one learnable weight layer in this configuration.
  • Object Detection: Object detection uses R-FCN with separate fully convolutional branches for region proposal and detection over the intermediate feature maps.The proposal branch uses RPN, anchors, non-maximum suppression, and position-sensitive ROI pooling in the detection branch.

5. Experiments

Experiments evaluate deep feature flow on Cityscapes segmentation and ImageNet VID detection, comparing variants, implementation choices, and accuracy–speed tradeoffs. DFF achieves substantial speedups with moderate accuracy loss, while joint training and design choices materially affect performance.

  • Datasets and setup: Experiments evaluate DFF on Cityscapes semantic segmentation and ImageNet VID object detection using recent large-scale video datasets.Cityscapes provides sparse annotations, while ImageNet VID uses fully annotated snippets and mAP evaluation.
  • Baselines and variants: DFF achieves 3.7× and 5.0× speedups over per-frame evaluation for segmentation and detection, respectively, with slightly lower accuracy.The method is compared against Frame and several DFF and SFF variants.
  • Ablations: Joint end-to-end training, especially of the flow network, is crucial because fixing or separately training components produces worse accuracy.The accuracy drop is significant when the flow network is fixed.
  • Ablations: Removing scaled feature modulation reduces accuracy by less than one percent on both tasks, indicating that scaling is slightly helpful.The experiment evaluates a DFF variant with the scale function removed.
  • Accuracy–speed tradeoff: 10× faster detection increases throughput from 4.05 fps to 41.26 fps while accuracy decreases from 73.9% to 69.5%.For segmentation, throughput increases from 2.24 fps to 17.48 fps while accuracy decreases from 69.7% to 62.4%.
  • Implementation choices: FlowNet Inception is usually faster than its two counterparts at the same accuracy, while the preferred feature network depends on task, speed regime, and video dynamics.ResNet-101 is preferred in high-accuracy settings; for faster segmentation, ResNet-50 becomes better above roughly 6.35 fps.

6. Future Work

The paper identifies flow estimation and key-frame scheduling as important directions for further exploration. It also notes that current practical recommendations are heuristic because they were observed on only two tasks and limited data.

  • Flow estimation: Flow quality remains difficult to assess because ground-truth optical flow is unavailable, while existing datasets are limited.Current optical-flow work relies on synthetic data or small real datasets, which the paper considers insufficient for deep learning.
  • Flow estimation: Faster and more accurate flow networks could further improve the method.The paper adopted FlowNet mainly because few alternatives were available.
  • Key-frame scheduling: Adaptive key-frame scheduling could improve the speed–accuracy trade-off when image content changes at different rates.The current scheduler uses a fixed key-frame duration, whereas drastic content changes may warrant a new key frame.

A. FlowNet Inception Architecture

The paper describes three FlowNet variants used for flow estimation: FlowNet, FlowNet Half, and FlowNet Inception. Their architectures follow the Simple design or an Inception-based design and are detailed in separate tables.

  • FlowNet variants: FlowNet and FlowNet Half follow the Simple FlowNet architecture.Their detailed layer configurations are provided in Tables 6 and 7, respectively.
  • FlowNet variants: FlowNet Inception follows an Inception-based architecture.Its detailed configuration is provided in Table 8.
  • Architecture details: The three variants are documented through separate architecture tables.FlowNet and FlowNet Half are associated with Tables 6 and 7, while FlowNet Inception is associated with Table 8.
Loading 1611.07715v2…