PaperlayerView exact revision on arXiv

Source-linked AI summary

ActionVLAD: Learning spatio-temporal aggregation for action classification

Rohit Girdhar, Deva Ramanan, Abhinav Gupta, Josef Sivic, Bryan Russell

arXiv:1704.02895v1cs.CV

TL;DR

Action recognition needs representations that capture long-term spatio-temporal structure rather than relying on local frame classifications. The paper integrates two-stream CNNs with learnable ActionVLAD aggregation, jointly pooling across space and time while keeping appearance and motion representations separate. This representation outperforms the two-stream base architecture and comparable baselines across HMDB51, UCF101, and Charades.

  • Problem

    Action recognition lacks an appropriate video-level representation that aggregates appearance and motion evidence across long-term spatio-temporal structure.

  • Method

    The paper integrates two-stream networks with an end-to-end trainable spatio-temporal NetVLAD extension that aggregates convolutional descriptors across the video.

  • Results

    13% relative improvement over the two-stream base architecture, with performance outperforming comparable baselines on HMDB51, UCF101, and Charades.

  • Takeaways & Limitations

    Joint space-time pooling is important, while appearance and motion are best aggregated into separate representations.

  • Takeaways & Limitations

    The approach assumes that ActionVLAD's strong regularization makes large models practical with limited action-classification training data.

Abstract

from arXiv · show

In this work, we introduce a new video representation for action classification that aggregates local convolutional features across the entire spatio-temporal extent of the video. We do so by integrating state-of-the-art two-stream networks with learnable spatio-temporal feature aggregation. The resulting architecture is end-to-end trainable for whole-video classification. We investigate different strategies for pooling across space and time and combining signals from the different streams. We find that: (i) it is important to pool jointly across space and time, but (ii) appearance and motion streams are best aggregated into their own separate representations. Finally, we show that our representation outperforms the two-stream base architecture by a large margin (13% relative) as well as out-performs other baselines with comparable base architectures on HMDB51, UCF101, and Charades video classification benchmarks.

1. Introduction

Action recognition needs video representations that capture long-term spatio-temporal structure, beyond independently classifying frames or short snippets. ActionVLAD addresses this by aggregating convolutional descriptors across the scene and the full temporal span while integrating appearance and motion streams.

  • Motivation: Current video datasets are relatively small, domain-specific, and sometimes noisily labeled, limiting progress in CNN-based action recognition.The passage contrasts video datasets containing tens of thousands of videos and a few hundred classes with ImageNet-scale image data.
  • Approach: ActionVLAD represents an action through appearance and motion primitives, such as players, a ball, running, jumping, and shooting for a basketball shot.These primitives are aggregated over the video rather than assigned as a single class to each frame.
  • Motivation: Two-stream models decompose videos into appearance and motion streams but largely disregard long-term temporal structure.They classify individual frames or short blocks of up to 10 frames and may average scores across segments.
  • Motivation: Temporal averaging can confuse composite actions whose sub-actions are shared across classes, such as basketball shooting, running, dribbling, jumping, and throwing.Local frames may not uniquely identify the complete action, making late fusion or score averaging suboptimal.
  • Approach: The paper proposes ActionVLAD, an end-to-end trainable representation that aggregates convolutional descriptors across image regions and the entire video timeline.The approach integrates a spatio-temporal extension of NetVLAD with two-stream networks.

2. Related Work

Prior action-recognition work includes hand-engineered trajectory descriptors, CNN-based two-stream and spatio-temporal models, and methods for explicit or implicit temporal aggregation. ActionVLAD extends trainable feature aggregation to video and combines it with two-stream architectures, outperforming comparable baselines on standard benchmarks.

  • Action recognition: Action recognition remains difficult because actions vary substantially within classes and large-scale training datasets are hard to annotate.The paper focuses on classifying a video clip into one of K action classes.
  • Prior representations: Dense trajectories aggregate hand-engineered appearance and motion descriptors into fixed-length bag-of-visual-words representations.Camera-motion compensation can further improve these shallow representations.
  • CNN approaches: CNN-based approaches include 3D spatio-temporal convolutions and two-stream architectures that separately process motion and appearance before fusing outputs.Two-stream architectures have generally outperformed spatio-temporal convolutions in the discussed prior work.
  • Temporal structure: Methods for modeling long-term temporal structure include constrained grammars and pooling approaches based on Fisher vectors or point trajectories.The paper positions its approach among methods that represent temporal structure implicitly through feature aggregation.
  • ActionVLAD: ActionVLAD introduces an end-to-end trainable spatio-temporal extension of NetVLAD integrated with two-stream networks.Unlike traditional aggregation as post-processing, the representation is trained within the video architecture.
  • Results: The final representation improves over the two-stream base architecture by 13% relative and outperforms comparable-base-architecture baselines on HMDB51, UCF101, and Charades.The contribution also evaluates pooling across space and time and strategies for combining streams.

3. Video-level two-stream architecture

The architecture converts sampled appearance and motion features into a trainable, fixed-length video representation by aggregating descriptors across space and time. It compares pooling locations and stream-fusion strategies within a two-stream VGG16-based network.

  • Architecture: Sampled RGB and flow frames provide appearance and motion features that are aggregated into a single fixed-length video vector before classification.The aggregation vocabulary and feature extractors are learned jointly.
  • Spatio-temporal aggregation: ActionVLAD assigns local descriptors to learned action-word cells and sums their residuals across the entire video.The resulting cell descriptors are normalized, stacked, and L2-normalized into the video representation.
  • Spatio-temporal aggregation: Joint space-time pooling represents descriptor distributions containing multiple sub-actions more fully than treating the video as a single max- or average-pooled descriptor.Cell-wise pooling preserves separate residual aggregates for different regions of descriptor space.
  • Aggregation placement: The aggregation layer can be placed at different network levels, and the study evaluates pooling choices built on a VGG16 two-stream architecture.The base network is pretrained at frame level and supplies features to the trainable ActionVLAD layer.
  • Stream combination: The paper explores combining appearance and motion through alternative stream-fusion strategies, including ActionVLAD over concatenated features.These alternatives are designed as jointly trainable representations.

4. Experiments

Experiments show that trainable ActionVLAD benefits from convolutional, jointly spatio-temporal aggregation and separate late fusion of appearance and motion streams. It improves over standard pooling, two-stream baselines, and comparable prior methods across action-recognition benchmarks.

  • Datasets and Evaluation: The experiments evaluate ActionVLAD on UCF101, HMDB51, and Charades, using three splits for final UCF101 and HMDB51 reporting.Charades evaluation uses mAP and weighted average precision (wAP).
  • Training ActionVLAD Representation: Training ActionVLAD jointly with preceding layers improves performance beyond a fixed k-means-initialized representation, while K = 32, 64, and 128 remain stable.On HMDB51 split 1 RGB, the reported results are 49.1%, 51.2%, and 51.1% for K = 32, 64, and 128; K = 1 gives 43.2%.
  • Where to ActionVLAD?: The best insertion point is the last convolutional layer, conv5 3, rather than conv4 3 or fully connected layers.The authors attribute this to preserved spatial variation and greater diversity across frames in conv5 3 features.
  • Baseline aggregation techniques: Average and max pooling over conv5 3 features significantly reduce performance, even relative to the two-stream architecture.These pooling methods collapse the video’s feature distribution into a single point, whereas ActionVLAD aggregates residuals across cells.
  • Combining Motion and Appearance: Late fusion performs best because appearance and motion features are separated in feature space and can retain complementary information.Concatenation uses the same number of cells to model a larger feature space, limiting modeling power.
  • Comparison against State of the Art: ActionVLAD outperforms comparable VGG-16-based methods on HMDB51 and UCF101 and all previously reported methods on Charades.With VGG-16, ActionVLAD reports 66.9% on HMDB51 and 92.7% on UCF101, exceeding the cited ResNet-50 and ResNet-152 two-stream baselines.
  • Visualization: Visualization shows learned action words focusing on recurring visual regions, while ActionVLAD tracks appearance and flow patches across frames.For a brushing-hair video, these contributions support the correct classification where the baseline predicted clap.

5. Conclusion

The paper concludes that ActionVLAD provides a successful, end-to-end trainable approach for spatio-temporal video aggregation and improves over two-stream VGG-16 methods on HMDB51 and UCF101.

  • Conclusion: ActionVLAD is an end-to-end trainable spatio-temporal aggregation method that outperforms most prior two-stream VGG-16 approaches on HMDB51 and UCF101.The authors suggest applying it as an ActionVLAD CNN layer to future video architectures and related tasks such as temporal localization.

Appendices

The appendices describe multi-crop testing and late fusion with improved dense-trajectory scores.

  • Implementation: ActionVLAD can pool features from multiple crops at test time, extending the single-center-crop evaluation used in the main paper.The appendix compares single-crop testing with a strategy using four corner crops and the center crop.
  • Implementation: For all datasets, iDT scores are combined with ActionVLAD predictions through late fusion using a weighted average for each class and video.The iDT scores were obtained from the authors of the cited datasets’ or methods’ sources.

C.1. Confusion Matrix

ActionVLAD corrects confusions that the standard two-stream model makes between visually similar actions, with gains linked to focusing on local discriminative features.

  • ActionVLAD most improves classes such as ‘climb stairs’, ‘hit’, and ‘drink’, which the two-stream model often confuses with similar actions.Examples include ‘climb stairs’ versus ‘walk’, ‘hit’ versus ‘golf’, and ‘drink’ versus ‘eat’ or ‘kiss’.
  • The model can focus on local appearance or motion features, such as a golf club, to disambiguate similar actions.
  • Table 7 presents videos that the two-stream model classified incorrectly but ActionVLAD classified correctly.The examples are sorted by ActionVLAD classifier score, with green labels for ActionVLAD predictions and red labels for two-stream predictions.
  • The accompanying visualization describes ‘action words’ learned by the method and highlights receptive-field centers assigned to specific cluster centers.

E.1. Combining Motion and Appearance

Appearance and motion features occupy mostly distinct regions of feature space, supporting late fusion of independently trained ActionVLAD representations.

  • The tSNE embedding shows clear separation between L2-normalized conv5 features from the appearance and motion streams.The embedding uses corresponding frames from 1000 randomly sampled frames, with some mixing but mostly distinct feature-space regions.
  • The separation indicates potentially complementary information between the two streams and supports late fusion of their representations.The figure passage states that experiments confirm this strategy.

F. Experimental details: Charades

For Charades, training uses trimmed action boundaries while testing processes each untrimmed video as a whole; the reported examples illustrate corrected confusions and local-feature comparisons.

  • Charades training treats each provided trimmed action boundary as a separate clip, whereas testing processes the complete untrimmed video without boundary information.
  • At test time, 25 frames are extracted from each complete video and evaluated using the provided mAP/wAP scripts.
  • Both Charades models use RGB information only, without flow, and are trained over a base BN-inception network.
  • The confusion-matrix comparison marks positive changes in red and negative changes in blue, with improvements expected on the diagonal and off-diagonal errors reduced.
  • The tSNE visualization compares appearance and motion conv5 features, whose mostly distinct regions support late fusion of independently trained representations.
Loading 1704.02895v1…