PaperlayerView exact revision on arXiv

Source-linked AI summary

A robust and efficient video representation for action recognition

Heng Wang, Dan Oneata, Jakob Verbeek, Cordelia Schmid

arXiv:1504.05524v1cs.CV

TL;DR

Video action recognition must cope with camera motion and the variability of realistic videos. The paper improves dense trajectories through camera-motion estimation and explores Fisher-vector pooling, evaluating the approach across three recognition and localization tasks. It reports state-of-the-art performance on all three tasks and substantial improvements over prior results.

  • Problem

    Realistic video recognition faces camera motion, motion clutter, occlusion, background clutter, and substantial intra-class variability.

  • Method

    The paper combines camera-aware dense trajectories, homography-based optical-flow correction and trajectory pruning, Fisher-vector encoding, and localization-specific rescoring and suppression.

  • Results

    The proposed pipeline achieves state-of-the-art performance on action classification, action localization, and complex-event recognition.

  • Takeaways & Limitations

    Improved trajectories and Fisher-vector pooling provide a general video-recognition pipeline across the evaluated tasks.

  • Takeaways & Limitations

    Human detection can fail with complex poses, self-occlusion, and motion blur, requiring tracked detections to compensate for missing detections.

Abstract

from arXiv · show

This paper introduces a state-of-the-art video representation and applies it to efficient action recognition and detection. We first propose to improve the popular dense trajectory features by explicit camera motion estimation. More specifically, we extract feature point matches between frames using SURF descriptors and dense optical flow. The matches are used to estimate a homography with RANSAC. To improve the robustness of homography estimation, a human detector is employed to remove outlier matches from the human body as human motion is not constrained by the camera. Trajectories consistent with the homography are considered as due to camera motion, and thus removed. We also use the homography to cancel out camera motion from the optical flow. This results in significant improvement on motion-based HOF and MBH descriptors. We further explore the recent Fisher vector as an alternative feature encoding approach to the standard bag-of-words histogram, and consider different ways to include spatial layout information in these encodings. We present a large and varied set of evaluations, considering (i) classification of short basic actions on six datasets, (ii) localization of such actions in feature-length movies, and (iii) large-scale recognition of complex events. We find that our improved trajectory features significantly outperform previous dense trajectories, and that Fisher vectors are superior to bag-of-words encodings for video recognition tasks. In all three tasks, we show substantial improvements over the state-of-the-art results.

1 Introduction

Real-world video recognition must handle substantial variation, clutter, occlusion, and camera motion. The paper addresses these challenges by improving dense trajectories and feature encoding, then evaluates the resulting framework across action classification, localization, and complex-event recognition.

  • Realistic video datasets introduce intra-class variation, background clutter, occlusion, camera motion, moving-background clutter, and low-quality artifacts.
  • Dense trajectories track densely sampled feature points with optical flow and compute descriptors along their trajectories.
  • Explicit camera-motion estimation prunes background trajectories and corrects optical flow, improving HOF and MBH motion descriptors.
  • Fisher vectors encode local descriptors into a holistic representation, and experiments report that their advantage over bag-of-words also holds across video recognition tasks.
  • The framework is evaluated on basic-action classification across six datasets, action localization in feature-length movies, and complex-event classification.
  • State-of-the-art performance is obtained on all three tasks, while modified non-maximum suppression further improves full-length-movie localization.

2 Related work

Prior work improves video recognition through trajectory processing, richer representations, learned features, and structured models. This paper instead emphasizes camera-aware dense trajectories and Fisher-vector pooling, motivated partly by the limitations of complex structured models.

  • Earlier trajectory methods considered dominant camera motion, foreground-background separation, and related motion-compensation strategies, but few addressed camera motion during trajectory extraction.
  • Other approaches improve dense trajectories through background-feature pruning, trajectory clustering, or decomposition of visual motion into dominant and residual components.
  • Learned and multimodal representations combine scene-specific pooling, person and object features, global scene features, object detectors, speech recognition, and character information.
  • Structured models capture temporal action structure or person-object interactions, while other work uses multiple-instance learning and graphical models.
  • Complex structured models can be difficult to learn from limited data with high intra-class variability and are generally more computationally demanding for large-scale settings.
  • The paper presents Fisher vectors as robust feature pooling and provides a large, diverse, comprehensive evaluation for action recognition.

3 Improving dense trajectories

The paper improves dense trajectories by estimating and compensating camera motion, removing camera-induced trajectories, and using human detection to protect homography estimation from moving people.

  • The improved trajectory pipeline estimates camera motion, removes inconsistent matches using human detection, and extracts corrected trajectory features.
  • Camera motion estimation: SURF matches and dense optical-flow matches are combined because their complementary coverage supports more balanced homography estimation.
  • Camera motion estimation: RANSAC estimates a homography that rectifies optical flow, suppressing background camera motion and enhancing foreground moving objects.
  • Improved motion descriptors: Corrected optical flow improves HOF, bringing its performance close to MBH, while combining HOF and MBH provides further improvement.
  • Removing camera-induced trajectories: Camera-consistent trajectories are removed from the warped flow, retaining trajectories related to human actions under pan, tilt, and zoom.
  • Limitations: The approach can fail when severe motion blur makes matching and optical-flow estimation unreliable, or when humans dominate the frame and distort homography estimation.

4 Feature encoding

The paper compares Fisher vectors and bag-of-words for encoding local video descriptors, while also testing weak spatio-temporal layout representations. Fisher vectors capture first- and second-order statistics and are normalized before linear-SVM classification.

  • 4.1 Fisher vector: Fisher vectors encode first- and second-order statistics between video descriptors and a diagonal-covariance Gaussian mixture model, extending bag-of-words representations.The representation uses soft assignments to Gaussian components and gradients with respect to their means and variances.
  • 4.1 Fisher vector: Each descriptor type produces a 2DK-dimensional Fisher vector after PCA reduction, GMM encoding, power normalization, and ℓ2 normalization.A linear SVM is then used for classification.
  • 4.2 Bag-of-words baseline: Bag-of-words serves as a baseline using soft assignments to the same Gaussian vocabulary as the Fisher vectors.Both linear and RBF-χ2 SVM kernels are considered for bag-of-words classification.
  • 4.3 Classification setup: Separate normalized encodings for different descriptor types are concatenated, with class weighting and regularization selected consistently across experiments.The regularization parameter C is chosen by training-set cross-validation over C ∈ {3^-2, 3^-1, · · · , 3^7}.
  • 4.4 Spatial layout: Spatio-temporal pyramids and spatial Fisher vectors add weak location information by computing encodings over spatio-temporal cells.These representations move beyond completely orderless video pooling.

5 Non-maximum-suppression for localization

The localization pipeline scores sliding windows and suppresses overlapping detections, addressing standard NMS’s tendency to retain short segments. RS-NMS favors longer coverage, while DP-NMS optimizes a constrained subset over the video.

  • Window scoring and suppression: Candidate detections use sliding windows of 30, 60, 90, and 120 frames, advanced in 30-frame steps before non-maximum suppression.Windows overlapping higher-scoring detections by more than 20% are deleted.
  • Motivation: Standard NMS tends to retain short windows because characteristic action features may occur in short subsequences.Longer windows can cover more of an action but may include less characteristic or background content.
  • RS-NMS: RS-NMS rescales each segment score by its duration before applying suppression, biasing selection toward longer windows.The method is motivated by the preference for windows that better cover the action.
  • DP-NMS: DP-NMS selects a non-overlapping subset covering the video and maximizing the sum of selected-window scores.Boolean variables represent selected windows, while each window contributes its score and interval to the optimization.
  • DP-NMS: Dynamic programming solves DP-NMS with latent temporal-window states, pairwise constraints, and a Viterbi forward-backward pass in linear time.The pairwise potential enforces coverage and non-overlap, while unary potentials use the associated time-step scores.
  • Empirical behavior: In the Coffee and Cigarettes example, RS-NMS favors 120-frame windows, whereas NMS and DP-NMS mostly select 30-frame windows.The retained-window comparison reports that RS-NMS balances segment length and score best in the example.

6 Datasets used for experimental evaluation

The evaluation spans six action-recognition datasets, two movie-based action-localization datasets, and the large-scale TRECVID MED 2011 event dataset. Protocols and dataset properties vary substantially in scale, category complexity, and video duration.

  • 6.1 Action recognition: Action recognition uses Hollywood2, HMDB51, Olympic Sports, High Five, UCF50, and UCF101 under dataset-specific train-test protocols.These datasets include movie clips, YouTube videos, sports actions, and interpersonal interactions.
  • 6.1 Action recognition: Hollywood2 contains 1,707 videos from 69 movies across 12 action classes, evaluated using mean average precision.Training and test videos come from different movies.
  • 6.1 Action recognition: HMDB51 contains 6,766 sequences across 51 categories and reports average accuracy over three original train-test splits.The experiments use the original, rather than stabilized, videos.
  • 6.1 Action recognition: Olympic Sports, High Five, UCF50, and UCF101 cover sports, interactions, daily-life actions, and real-world YouTube videos with their respective evaluation protocols.UCF101 contains 13,320 clips across 101 action categories, while UCF50 contains 6,618 clips across 50 categories.
  • 6.2 Action localization: Action localization uses Coffee and Cigarettes and DLSBP, with average precision counting detections whose intersection-over-union with ground truth is at least 20%.The datasets contain drinking and smoking annotations, and sit-down and open-door annotations, respectively.
  • 6.3 Event recognition: TRECVID MED 2011 contains 15 complex event categories, 9,600 null videos, and a 32,000-video test set evaluated with average precision.The paper also reports results using EVENTS, DEV-T, and DEV-O subsets in a second experimental setup.
  • 6.3 Event recognition: TRECVID videos range from seconds to one hour and from 128 × 88 to 1920 × 1080 resolution, so they are rescaled and temporally subsampled for tractability.The impact of video resolution is evaluated separately.

7 Experimental results

The experimental evaluation is organized around action recognition, action localization, and event recognition. Results for these tasks are presented in Sections 7.1, 7.2, and 7.3.

  • 7 Experimental results: The paper evaluates action recognition, action localization, and event recognition in separate experimental subsections.The corresponding results appear in Sections 7.1, 7.2, and 7.3.

7.1 Action recognition

The experiments compare BOW and FV encodings, spatial-layout variants, and improved trajectory components across action-recognition datasets. Fisher vectors and camera-motion compensation consistently provide stronger representations, with descriptor combinations achieving the best results.

  • Feature encoding with BOW and FV: FV usually outperforms BOW by 6-7% with the same number of visual words on Hollywood2 and HMDB51.FV with 64 visual words even exceeds BOW with 1024 visual words.
  • Feature encoding with BOW and FV: BOW performance saturates above K = 8,192, while FV remains competitive with similar-dimensional representations and is eight times faster at comparable settings.The comparison uses K = 32,768 for BOW and K between 64 and 128 for FV; FV still leads by 2% on HMDB51.
  • Evaluation of improved trajectory features: HOF improves around 5% on Hollywood2 and around 10% on HMDB51 after optical-flow stabilization, becoming comparable to MBH.Both WarpFlow and Combined settings substantially outperform the unstabilized alternatives.
  • Evaluation of improved trajectory features: MBH improves by over 2% on both datasets after explicit camera-motion compensation.The compensation makes motion boundaries clearer despite MBH's existing robustness to camera motion.
  • Evaluation of improved trajectory features: Combining HOG, HOF, and MBH achieves the best performance because HOF and MBH provide complementary motion information.HOF captures zero-order motion, whereas MBH captures first-order derivatives.
  • Comparison to the state of the art: Human detection improves motion descriptors by typically over 1%, and the improved framework exceeds prior results across the evaluated action datasets.Reported results include over 5% improvement on Olympic Sports, 69.4% on High Five, 91.7% on UCF50, and 86.0% on the referenced benchmark.

7.2 Action localization

The localization experiments evaluate non-maximum suppression and improved trajectory features on four movie-action classes. Re-scored suppression and the improved representation produce strong gains, but results are unstable on the small datasets and human detection is not uniformly beneficial.

  • Evaluation of non-maximum suppression: RS-NMS significantly improves over standard NMS on two of four action classes, while DP-NMS is slightly inferior to RS-NMS.For RS-NMS, zero overlap is beneficial; standard NMS is relatively insensitive to zero versus 20% overlap.
  • Evaluation of improved trajectory features: Combining all descriptors usually performs better than individual descriptors, but improved trajectories are outperformed by the baseline on three of four classes in one setting.Per-class localization results are less consistent than action-recognition results.
  • Evaluation of improved trajectory features: Human detection can hurt localization performance: omitting it performs better for drinking and smoking in the Coffee & Cigarettes dataset.The paper suggests inaccurate homography estimation as a possible explanation for this result.
  • Comparison to the state of the art: The improved trajectory features substantially outperform prior localization results on all four action classes.The largest difference is for drinking, where the result is over 23% better than Gaidon et al. (2011).

7.3 Event recognition

The event-recognition experiments apply improved trajectories and Fisher vectors to TRECVID MED 2011 under practical computational constraints. The representation achieves a large state-of-the-art gain, while video quality and irrelevant temporal background limit improvements.

  • Evaluation of improved trajectory features: HOG+MBH is the best two-channel combination, while combining all three channels performs similarly to that variant.The ordering reflects complementarity between appearance information from HOG and motion information from HOF and MBH.
  • Evaluation of improved trajectory features: Removing all spatio-temporal information reduces performance from 45.9 to 43.8, demonstrating the value of weak geometric information.This result holds for the highly unstructured TRECVID MED videos.
  • Evaluation of improved trajectory features: Improved trajectories outperform baseline dense trajectories at every tested resolution, but higher resolutions than 320 provide little additional performance.Even low-resolution videos contain enough matches for reliable homography estimation.
  • Evaluation of improved trajectory features: The gain from improved trajectories is less pronounced on TRECVID MED than on action recognition because poor video quality and short informative subsequences leave substantial background clutter.These dataset characteristics may limit the beneficial effects of the improved features.
  • Computational cost: Computing improved trajectories increases runtime by around a factor of two, with the final representation running around 10× slower than real video on one core.The final setting resizes videos to 480-pixel width and uses HOG, HOF, MBH, FV, H3, and SFV without human detection.
  • Comparison to the state of the art: 31.6% AP dramatically improves over the previous state of the art on TRECVID MED 2011.The system uses concatenated Fisher vectors from HOG, HOF, and MBH with a linear SVM.

8 Conclusions

The paper combines explicit camera-motion estimation with improved dense trajectories and alternative Fisher-vector encoding. Its framework substantially improves results across action recognition, action localization in movies, and complex event recognition.

  • Explicit camera-motion estimation improves dense trajectories by removing background trajectories and warping optical flow with a robust homography.A human detector removes inconsistent matches from the camera-motion estimation process, improving robustness.
  • The framework explores Fisher vectors, spatio-temporal pyramids, and spatial Fisher vectors to encode descriptors and weak geometric layouts.
  • The evaluation covers action recognition, action localization in movies, and complex event recognition.
  • The proposed pipeline significantly outperforms the state of the art on all three tasks.Action localization also benefits from re-scoring before non-maximum suppression to reduce bias toward overly short windows.
Loading 1504.05524v1…