PaperlayerView exact revision on arXiv

Source-linked AI summary

DMC-Net: Generating Discriminative Motion Cues for Fast Compressed Video Action Recognition

Zheng Shou, Xudong Lin, Yannis Kalantidis, Laura Sevilla-Lara, Marcus Rohrbach, Shih-Fu Chang, Zhicheng Yan

arXiv:1901.03460v3cs.CV

TL;DR

Compressed-video methods avoid slow optical-flow computation, but noisy, low-resolution motion vectors provide less discriminative motion information. DMC-Net trains a lightweight generator to refine those cues using flow supervision and action-classification learning. Across three benchmarks, it narrows the accuracy gap with flow-based methods while running two orders of magnitude faster at inference.

  • Problem

    Optical flow is time-consuming, while compressed-video motion vectors are noisy and low-resolution, limiting their discriminative motion representation.

    Source: §1 · p. 1, §3.2 · p. 3

  • Method

    A lightweight generator refines compressed-video motion vectors using optical-flow reconstruction and adversarial losses, jointly with downstream action-classification supervision.

    Source: §1 · p. 2

  • Results

    DMC-Net significantly shortens the performance gap between compressed-video methods with and without optical flow across three benchmarks and runs two orders of magnitude faster than flow-based methods at inference.

    Source: §1 · p. 2

  • Takeaways & Limitations

    DMC-Net achieves high accuracy without requiring optical-flow estimation at inference, operating exclusively in the compressed domain.

    Source: §1 · p. 2

Abstract

from arXiv · show

Motion has shown to be useful for video understanding, where motion is typically represented by optical flow. However, computing flow from video frames is very time-consuming. Recent works directly leverage the motion vectors and residuals readily available in the compressed video to represent motion at no cost. While this avoids flow computation, it also hurts accuracy since the motion vector is noisy and has substantially reduced resolution, which makes it a less discriminative motion representation. To remedy these issues, we propose a lightweight generator network, which reduces noises in motion vectors and captures fine motion details, achieving a more Discriminative Motion Cue (DMC) representation. Since optical flow is a more accurate motion representation, we train the DMC generator to approximate flow using a reconstruction loss and a generative adversarial loss, jointly with the downstream action classification task. Extensive evaluations on three action recognition benchmarks (HMDB-51, UCF-101, and a subset of Kinetics) confirm the effectiveness of our method. Our full system, consisting of the generator and the classifier, is coined as DMC-Net which obtains high accuracy close to that of using flow and runs two orders of magnitude faster than using optical flow at inference time.

1. Introduction

DMC-Net targets the accuracy gap between fast compressed-video methods and methods that compute optical flow. It learns discriminative motion cues from compressed-video motion vectors and residuals, then runs without flow estimation at inference.

1. Introduction

  • CoViAR uses compressed-video modalities quickly, but reaching state-of-the-art accuracy requires adding optical flow; motion vectors are coarse and less discriminative than flow.Motion-vector resolution is reduced 16x, losing fine details important for distinguishing actions; separately processing motion vectors and residuals ignores their interaction.Source: §1 · p. 2
  • A lightweight generator refines noisy motion vectors using stacked motion vectors and residuals, with optical-flow reconstruction and adversarial losses plus downstream action-classification supervision.Source: §1 · p. 2
  • Two orders of magnitude faster than methods using optical flow, DMC-Net operates exclusively in the compressed domain at inference while achieving high accuracy.Figure 1 compares inference time and accuracy on HMDB-51.Source: Figure 1 · p. 1, §1 · p. 2
  • On HMDB-51, UCF101, and a Kinetics subset, DMC-Net significantly shortens the performance gap between compressed-video methods with and without optical flow.Source: §1 · p. 2

2. Related Work

Prior work models motion using optical flow or compressed-video modalities, but compressed-video methods have struggled to match two-stream accuracy. DMC-Net instead refines motion vectors in the compressed domain to generate discriminative motion cues.

2. Related Work

  • Two-stream networks process RGB frames and optical flow in separate ConvNets, while compressed-video methods seek to avoid costly flow computation [41] [5] [49].Source: §2 · p. 3
  • CoViAR uses I-frames, motion vectors, and residuals without RGB-frame decoding, but low-resolution vectors and separate processing limit its accuracy [55].Source: §2 · p. 3
  • Unlike prior motion-representation methods that take decoded RGB frames as input, DMC-Net refines compressed-domain motion vectors with less model capacity.Source: §2 · p. 3

3. Approach

DMC-Net uses motion vectors and residuals from compressed video to generate a finer, more discriminative motion cue, then trains that generator with flow-based and action-classification objectives. At inference, the generated cue feeds an action classifier.

3.1. Basics and Notations of Compressed Video

  • 16×16 macroblocks make motion-vector values identical within each block, while I-frames and residuals have shape (3, H, W) and motion vectors (2, H, W).Motion vectors record macroblock displacement; residuals store the RGB difference after motion compensation.Source: §3.1 · p. 3

3.2. The Discriminative Motion Cue Generator

  • The DMC generator takes stacked motion vectors and residuals to reduce motion-vector noise and recover fine motion details for a more discriminative representation.The inputs combine coarse-scale motion vectors with residuals that often align with moving-object boundaries.Source: §3.2 · pp. 3–4, Figure 3 · p. 4
  • 0.63% of the PWC-Net flow estimator's GFLOPs are required by the DMC generator, which uses reduced filters and a shortcut to predict refinements added to motion vectors.The shortcut connects the input motion vector to the output DMC; the architecture search balances accuracy and complexity.Source: Table 1 · p. 4, §3.2 · p. 4

3.3. Flow-guided, Discriminative Motion Cues

  • The generator is designed around three flow-guided losses: pixel-level flow reconstruction, adversarial learning, and downstream action classification.Optical flow is used as guidance because it offers higher resolution and more accurate motion information than motion vectors.Source: §3.3 · p. 5, §3.3.3 · p. 5

3.3.1 Optical Flow Reconstruction Loss

  • The MSE reconstruction loss minimizes per-pixel differences between generated DMC and corresponding optical flow over training P-frames; weighting salient flow regions did not improve accuracy.The loss averages squared differences over the sampled training P-frames.Source: §3.3.1 · p. 5

3.3.2 Adversarial Loss

  • Because MSE tends to blur motion boundaries, alternating discriminator training encourages generated DMC to resemble the flow distribution and retain finer details.The discriminator distinguishes real flow from generated DMC; the generator is trained to make its output indistinguishable from flow.Source: §3.3.2 · p. 5

3.3.3 The Full Training Objective Function

  • The generator and action classifier are jointly trained end-to-end with reconstruction, generator adversarial, and classification losses, while the discriminator is optimized in alternating turns.The full objective includes balancing weights for the reconstruction and generator adversarial terms.Source: §3.3.3 · pp. 5–6

3.4. Inference

  • At inference, only the generator and classifier run in the compressed domain, and DMC-Net is two orders of magnitude faster than methods using optical flow.The generator outputs DMC, which the classification network uses to predict the action class.Source: §3.4 · p. 6, Figure 4 · p. 6

4. Experiments

The experiments evaluate DMC-Net on three action-recognition datasets and compare its accuracy and speed with compressed- and decoded-video methods. They also examine the effects of the proposed losses and report inference-speed comparisons.

4.1. Datasets and Evaluation

  • Evaluation uses top-1 video-level accuracy on UCF-101, HMDB-51, and Kinetics-n50, a 400-class subset sampled from Kinetics-400.Kinetics-n50 has 30 training and 20 testing videos sampled per class; the paper says it also evaluates on the full set in supplementary material.Source: §4.1 · p. 6

4.2. Implementation Details

  • For DMC, the evaluation uses a ResNet-18 classifier and omits cropping and flipping, unlike the other modalities' testing setup.The paper uses TV-L1 flow to guide DMC-Net training and fuses prediction scores across I, MV, R, and DMC.Source: §4.2 · p. 6

How much gain DMC-Net can improve over CoViAR?

  • On HMDB-51, adding flow reconstruction to classification raises accuracy from 60.5% to 61.5%, and adding adversarial training raises it to 61.8%.The authors attribute the accuracy gain over CoViAR to reduced noisy signals and sharper motion-boundary details in DMC.Source: How much gain DMC-Net can improve over… · p. 6, Figure 5 · p. 7

4.4. Inference Speed

  • DMC-Net adds little computational overhead to CoViAR and remains significantly faster than conventional two-stream methods; its generator is faster than the compared flow-estimation and motion-representation methods.The speed comparisons use batch size 1; the table caption distinguishes sequential and concurrent CNN forwarding for one comparison.Source: §4.4 · pp. 7–8, Table 3 · p. 7

4.5. Comparisons with Compressed Video Methods

  • DMC-Net outperforms the other evaluated compressed-video methods; using an I3D classifier yields higher accuracy than the ResNet-18 version.The paper also reports that DMC-Net (I3D) outperforms a number of methods requiring video decoding.Source: §4.5 · p. 8

4.6. Comparisons with Decoded Video Methods

  • DMC-Net (I3D) improves I3D RGB when their predictions are combined, while the state-of-the-art accuracy comes from slower two-stream RGB-plus-flow methods.TVNet is better than DMC-Net (ResNet-18) but worse than DMC-Net (I3D).Source: §4.6 · p. 8

5. Conclusion

The conclusion presents DMC-Net as an efficient compressed-domain action-recognition model, reporting substantial accuracy gains over prior work on three benchmarks without computationally expensive flow.

5. Conclusion

  • DMC-Net achieves substantial accuracy gains over prior work on three action-recognition benchmarks without computationally expensive optical flow.Source: §5 · p. 8

7. Appendix

The appendix describes the compressed-video inputs used by DMC-Net and provides additional model-training details.

7.1. Data Modalities in the Compressed Domain

  • The compressed domain provides I-frame RGB values and, for P-frames, motion vectors and residual errors.P-frames are reconstructed by warping a reference frame with motion vectors and adding residual errors.Source: §7.1 · p. 9

7.2. More implementation details

  • Training first uses flow reconstruction with the classifier fixed, then trains the generator and classifier end-to-end before initializing full training with the adversarial loss.The first stage uses one generator epoch for flow reconstruction followed by 49 epochs with classification; the full model then includes a discriminator.Source: §7.2 · p. 9

7.3. Other early fusion possibilities

The appendix compares early-fusion choices and tests attention-weighted flow reconstruction as an alternative and complement to the GAN loss.

7.3. Other early fusion possibilities

  • On HMDB-51, stacking motion vectors and residuals directly reaches 61.80%, above Add at 61.32% and Concat at 61.36%.The authors suggest direct stacking performs better because motion vectors and residuals are strongly correlated in pixel space before convolution.Source: §7.3 · p. 9

7.4.1 Approach

  • Attention-weighted reconstruction assigns location-specific optical-flow weights to the DMC-to-flow MSE, aiming to emphasize action-discriminative regions.The weights are obtained from a trained flow classifier using interpretation methods such as CAM or GBP.Source: §7.4.1 · pp. 10–11
  • CAM uses the ground-truth class activation map as the attention map, while GBP backpropagates class-specific contributions to flow values.CAM upsamples its map to match DMC size; GBP sets negative gradients to zero when backpropagating through ReLU layers.Source: §7.4.1 · p. 11

7.4.2 Experimental results

  • Attention-weighted reconstruction is comparable to the unweighted loss without GAN, but adding it to DMC-Net with GAN hurts accuracy.The authors conclude it offers no benefit beyond GAN and omit it from the main model.Source: Table 5 · p. 11, §7.4.2 · pp. 11–12

7.5. More discussions about the speed

DMC-Net’s accuracy depends on the classifier, and I3D’s clip-level prediction makes its total action-prediction time much longer than its per-frame time suggests.

7.5.1 Speed of DMC-Net (I3D)

  • DMC-Net (I3D) achieves much better accuracy than DMC-Net (ResNet-18), but their speeds are not directly comparable because one is clip-level and the other frame-level.DMC-Net (I3D) processes 250 frames for one prediction, while ResNet-18 uses one frame.Source: §7.5.1 · p. 12
  • I3D takes 250 frames concurrently to produce one action-class prediction, so its 0.79 ms per-frame time does not represent total prediction time.Source: §7.5.1 · p. 12
  • 197.5 ms per action prediction for DMC-Net (I3D) (0.79 ms × 250 frames), versus 0.76 ms for DMC-Net (ResNet-18) using one frame.The per-frame times are 0.79 ms for I3D and 0.76 ms for ResNet-18.Source: §7.5.1 · p. 12

7.6. Generalize DMC-Net to different compressed video standards

The experiments use MPEG-4 compressed video, while handling other encoding formats is identified as a practical direction for further work.

  • The method follows CoViAR [55] in specifically using MPEG-4 video.Source: §7.6 · p. 12
  • For arbitrary input formats, the authors note that conversion to MPEG-4 is possible, while developing support for other formats remains an interesting direction.Source: §7.6 · p. 12
  • On HMDB-51, FFmpeg conversion averages 1.13 ms per frame, compared with 0.76 ms per frame for DMC-Net and more than 75 ms for the two-stream method.The conversion timing is for sequential processing of each video.Source: §7.6 · p. 12

7.7. More ablation studies

The supplementary ablations examine alternatives to the main paper’s end-to-end training and single-stage DMC generation design.

7.7.1 End-to-end learning

  • The alternative training strategy first trains the generator without the classifier or classification loss, then fixes it while training the classifier on generated DMC.Source: §7.7.1 · pp. 12–13

7.7.2 Decomposed two-stage DMC generation

  • The main generator jointly corrects noisy motion vectors and creates fine details while upsampling the 14×14 representation to 224×224.Motion vectors are 224×224 but comprise 16×16 macroblocks with identical values per block; downsampling by 16 preserves the same motion information.Source: §7.7.2 · p. 13
  • The proposed two-stage alternative rectifies noise in downsampled motion vectors, then upsamples them to generate DMC; it also applies flow reconstruction loss to the intermediate and generated representations.Source: §7.7.2 · p. 13

7.8. Smoothing Motion Vector via bilinear interpolation before fed into the DMC generator

The ablations compare the main single-stage, end-to-end setup using blocky motion vectors with alternatives that separate training, split generation into stages, or smooth the input first.

7.8.1 Experimental results

  • End-to-end learning is effective: the authors compare the main setup with one that separates generator and classifier training, and retain end-to-end training.Source: §7.8.1 · p. 13, Table 6 · p. 13
  • Two-stage DMC generation offers no accuracy benefit, so the authors retain a single network that corrects errors and generates fine motion details together.Source: §7.8.1 · p. 13, Table 6 · p. 13
  • Bilinear smoothing before generation produces DMC with comparably good motion cues and similar accuracy to feeding blocky motion vectors directly.The main setup uses end-to-end training, single-stage generation, and blocky motion-vector input.Source: §7.8.1 · p. 13, Table 6 · p. 13

7.9. Results on the full Kinetics dataset

On the full Kinetics dataset, DMC-Net reaches accuracy close to CoViAR + TV-L1 Flow and improves on CoViAR. Figure 9 contrasts the main paper’s single one-stage DMC generation with a decomposed two-stage strategy.

7.9. Results on the full Kinetics dataset

  • 65.42% accuracy for DMC-Net (ours) is close to CoViAR + TV-L1 Flow at 65.43% and improves on CoViAR at 65.37%.The authors reused Kinetics-n50 training hyper-parameters because training on full Kinetics took extremely long on one GPU.Source: §7.9 · p. 13
  • The authors conjecture that I-frame and Residual models seeing high-variance training data leaves motion with little complementary information for distinguishing action categories.Source: §7.9 · p. 13
  • Figure 9 illustrates the main paper’s single one-stage DMC generation strategy alongside a decomposed two-stage strategy.Source: Figure 9 · p. 14
Loading 1901.03460v3…