PaperlayerView exact revision on arXiv

Source-linked AI summary

Temporal Pyramid Pooling Based Convolutional Neural Networks for Action Recognition

Peng Wang, Yuanzhouhan Cao, Chunhua Shen, Lingqiao Liu, Heng Tao Shen

arXiv:1503.01224v2cs.CV

TL;DR

Video action recognition must handle clips with varying numbers of frames while retaining useful spatial-temporal information. This paper proposes an encoding and temporal pyramid pooling CNN that combines appearance and motion features, achieving strong results on Hollywood2 and HMDB51 with fewer training videos.

  • Problem

    Image-level CNNs expect single-image inputs, whereas videos contain varying numbers of frames; existing sampling and 3D-convolution approaches have important limitations.

  • Method

    An encoding layer and temporal pyramid pooling layer convert frame-level activations from an arbitrary number of frames into a fixed-length video representation, while concatenating appearance and motion features.

  • Results

    The method achieves the best performance on Hollywood2, second-best reported performance on HMDB51, and improved results from combining appearance and motion information.

  • Takeaways & Limitations

    The network supports variable-length video inputs while leveraging image-level CNN initialization and combining complementary appearance and motion information.

Abstract

from arXiv · show

Encouraged by the success of Convolutional Neural Networks (CNNs) in image classification, recently much effort is spent on applying CNNs to video based action recognition problems. One challenge is that video contains a varying number of frames which is incompatible to the standard input format of CNNs. Existing methods handle this issue either by directly sampling a fixed number of frames or bypassing this issue by introducing a 3D convolutional layer which conducts convolution in spatial-temporal domain. To solve this issue, here we propose a novel network structure which allows an arbitrary number of frames as the network input. The key of our solution is to introduce a module consisting of an encoding layer and a temporal pyramid pooling layer. The encoding layer maps the activation from previous layers to a feature vector suitable for pooling while the temporal pyramid pooling layer converts multiple frame-level activations into a fixed-length video-level representation. In addition, we adopt a feature concatenation layer which combines appearance information and motion information. Compared with the frame sampling strategy, our method avoids the risk of missing any important frames. Compared with the 3D convolutional method which requires a huge video dataset for network training, our model can be learned on a small target dataset because we can leverage the off-the-shelf image-level CNN for model parameter initialization. Experiments on two challenging datasets, Hollywood2 and HMDB51, demonstrate that our method achieves superior performance over state-of-the-art methods while requiring much fewer training data.

I. INTRODUCTION

Video action recognition must adapt image-based CNNs to clips with varying frame counts. The proposed network addresses this with encoding, temporal pyramid pooling, and appearance–motion feature concatenation.

  • I. INTRODUCTION: Image-level CNNs are naturally mismatched to videos because they require a single image while videos contain varying numbers of frames.This input-format mismatch motivates the proposed variable-length design.
  • I. INTRODUCTION: Existing CNN-based video methods either sample a fixed number of frames, risking missed information, or use 3D convolutions that cannot directly leverage image-level CNN initialization.The 3D-convolution strategy also requires very large training datasets to reduce over-fitting risk.
  • I. INTRODUCTION: The proposed network accepts an arbitrary number of video frames through an encoding layer followed by temporal pyramid pooling into a fixed-length video representation.Temporal pyramid pooling also models weak temporal structure within videos.
  • I. INTRODUCTION: The network additionally concatenates appearance and motion information to form its video representation.This combines complementary cues from CNN appearance features and motion features.

II. RELATED WORK

Prior action-recognition systems aggregate handcrafted or learned local and mid-level representations into video-level features. The proposed approach instead uses a deep CNN architecture that combines image-derived appearance features with frame-level motion features.

  • II. RELATED WORK: Traditional action-recognition methods commonly use handcrafted interest points or trajectories with bag-of-features or Fisher-vector encoding for video representations.These pipelines aggregate local descriptors into fixed-dimensional video-level features.
  • II. RELATED WORK: Other approaches mine discriminative mid-level representations, such as subvolumes, attributes, action parts, salient regions, or actons, and fuse their classification outputs or re-encode them globally.The representations are used either directly for classification or as local features for another encoding stage.
  • II. RELATED WORK: Deep action-recognition models include 3D CNNs that convolve across spatial-temporal domains to capture motion in consecutive frames.Large video datasets have been used to reduce over-fitting in spatial-temporal models.
  • II. RELATED WORK: The proposed architecture concatenates FC7 appearance activations with frame-level motion features, then applies FCa and temporal pyramid pooling before classification.The encoding and pooling stages transform features from varying numbers of frames into a fixed-length video-level representation.

B. Network architecture

The network combines frame-level appearance and motion features, encodes them, and uses temporal pyramid pooling to produce a fixed-length video representation for classification.

  • B. Network architecture: Appearance features from an off-the-shelf CNN are concatenated with frame-level motion features before encoding and temporal pooling.The appearance stream uses five convolutional and two fully connected layers, while motion features are derived from local trajectory encodings.
  • B. Network architecture: A supervised feature-merging procedure reduces the high-dimensional Fisher-vector motion representation before network processing.The procedure merges same-class features, clusters feature signatures, and groups dimensions into a target lower-dimensional representation.
  • B. Network architecture: The encoding and temporal pyramid pooling module converts features from a varying number of frames into a fixed-length video-level representation.The encoding layer maps activations to poolable feature vectors, while temporal pooling captures coarse-to-fine temporal structure.
  • B. Network architecture: Temporal pyramid pooling partitions frames at coarse and fine levels, pools each segment, and concatenates the pooled results.The coarse level pools the whole video; the fine level pools evenly divided temporal segments.
  • B. Network architecture: The classification layer applies pooling and a softmax transformation to produce class probabilities, trained against ground-truth labels with a loss function.The output is a probability distribution over video classes, and stochastic gradient descent updates the model parameters.

C. Late fusion model

The late-fusion variant uses separate appearance and motion network streams, removing the feature concatenation layer and combining their outputs after classification.

  • C. Late fusion model: Late fusion independently processes appearance and motion information in two network streams before combining their final probability distributions.Each stream is similar to the proposed network, but neither includes the feature concatenation layer.

1) Motion feature:

Motion features are computed from local trajectories around each frame, reduced in dimensionality, and supplied alongside raw frame images as network input.

  • 1) Motion feature:: Each frame’s motion feature is built from Fisher-vector encodings of trajectories within its neighboring 11 frames.The neighborhood spans frames -5 to 5 around the target frame.
  • 1) Motion feature:: The C1-FC7 parameters are initialized from a pretrained vgg-fast model and kept fixed, while FCa and FCb are learned with momentum-based stochastic gradient descent.Training uses momentum 0.9, weight decay 5 × 10^-4, and 25 epochs.
  • 1) Motion feature:: Experiments evaluate the method on Hollywood2 and HMDB51, datasets containing complex actions and substantial uncontrolled scene variation.Hollywood2 reports mean average precision, while HMDB51 reports average classification accuracy over classes and splits.

B. Performance evaluation

The method achieves the strongest reported performance on Hollywood2, with high-dimensional frame-level motion features outperforming low-dimensional alternatives.

  • 1) Hollywood2:: More than 4% improvement over improved dense trajectory makes the method the best performer on Hollywood2.The comparison identifies improved dense trajectory as the most competitive listed method.
  • 1) Hollywood2:: High-dimensional motion features perform better than low-dimensional features in the reported comparisons.
  • 1) Hollywood2:: The Hollywood2 results compare methods using low-dimensional 4096 or high-dimensional 20000 frame-level motion features.LD denotes 4096 dimensions, while HD denotes 20000 dimensions.

2) HMDB51:

On HMDB51, the method ranks second among reported approaches and outperforms a frame-sampling CNN despite using only HMDB51 for training. Combining motion representations further improves recognition.

  • 2) HMDB51:: The method achieves the second-best reported performance on HMDB51.The best-performing comparison uses stacked Fisher vector encoding, whereas the proposed method uses a single encoding module.
  • 2) HMDB51:: The method outperforms a frame-sampling CNN trained on the combined HMDB51 and UCF101 datasets, while training only on HMDB51.The authors describe this as a more principled treatment of video-length variation.
  • C. Fusion of motion net and global Fisher Vector: The fusion experiments compare motion representations using low-dimensional 4096 or high-dimensional 20000 feature encodings.The supplied table caption defines LD and HD for frame-level motion features.
  • C. Fusion of motion net and global Fisher Vector: Combining global motion encodings with frame-level motion representations boosts recognition performance by a large margin.The two representation types are reported to compensate for each other when describing video motion.

D. Motion vs. appearance

Motion features outperform appearance features on both datasets, while the proposed encoding layer particularly benefits appearance features and adds little for already-coded motion features.

  • Motion features significantly outperform appearance features on both Hollywood2 and HMDB51.
  • The proposed network outperforms ATP and AAP by 8% and 10% on Hollywood2, and ATP and APP by 1.7% and 3.8% on HMDB51, using appearance features.The comparison isolates the benefit of the encoding layer before pooling.
  • For motion features, the proposed method performs comparably to TTP because Fisher vectors are already coding vectors ready for pooling.By contrast, FC7 outputs are not well-tuned for pooling, making the encoding layer beneficial for appearance features.
  • Temporal pyramid pooling improves classification by better describing videos’ temporal structure.The cited HMDB51 failures involve actions with similar appearance and motion patterns.

E. Early fusion vs. late fusion

The study compares early feature fusion with late score fusion for appearance and motion information. Both improve over single-input networks, but early fusion performs better on both datasets.

  • Early fusion improves results over late fusion by around 3% on Hollywood2 and 2% on HMDB51.Early fusion concatenates motion and appearance features before training, whereas late fusion averages separate networks’ softmax scores.
  • Both early and late fusion outperform networks with a single input, showing that appearance and motion information are complementary.
  • On Hollywood2, early fusion performs better for every class except “Eat.”The passage attributes this pattern to late fusion training the two streams independently without adapting parameters for their combination.

F. The impact of the temporal pyramid parameter

The experiments vary the temporal pyramid parameter b to assess how temporal segmentation affects pooling. The supplied passages also report that temporal pyramid pooling improves classification by representing video structure.

  • Varying b from 0 to 7 evaluates the impact of temporal segmentation, with b = 0 representing no temporal pyramid.The two-level pyramid includes one level over the full video and another dividing it into b segments.
  • Temporal pyramid pooling improves classification because it better describes the temporal structure of videos.
Loading 1503.01224v2…