Source-linked AI summary
Rank Pooling for Action Recognition
Basura Fernando, Efstratios Gavves, Jose Oramas, Amir Ghodrati, Tinne Tuytelaars
TL;DR
The paper addresses the challenge of representing video-wide temporal evolution for action recognition. It introduces rank pooling, which learns chronological frame ordering and uses ranking-function parameters as video representations, improving performance and complementing local and static features.
Problem
Existing methods capture local changes but are not designed to model higher-level, video-wide appearance and motion evolution associated with actions.
Method
Rank pooling fits learning-to-rank models to arrange video frames chronologically and uses their parameters as a representation capturing appearance evolution and dynamics.
Results
Fusing rank pooling with local motion and appearance methods improves performance by up to 10%, while rank pooling is also complementary to static CNN-based representations.
Takeaways & Limitations
Rank pooling provides a useful and robust representation of video-wide temporal information that complements local temporal and static feature methods.
Takeaways & Limitations
The moving average alternative is unstable, requires choosing a window size T, and is undefined for the final video timestamps because of boundary effects.
Abstract
from arXiv · showhide
We propose a function-based temporal pooling method that captures the latent structure of the video sequence data - e.g. how frame-level features evolve over time in a video. We show how the parameters of a function that has been fit to the video data can serve as a robust new video representation. As a specific example, we learn a pooling function via ranking machines. By learning to rank the frame-level features of a video in chronological order, we obtain a new representation that captures the video-wide temporal dynamics of a video, suitable for action recognition. Other than ranking functions, we explore different parametric models that could also explain the temporal changes in videos. The proposed functional pooling methods, and rank pooling in particular, is easy to interpret and implement, fast to compute and effective in recognizing a wide variety of actions. We evaluate our method on various benchmarks for generic action, fine-grained action and gesture recognition. Results show that rank pooling brings an absolute improvement of 7-10 average pooling baseline. At the same time, rank pooling is compatible with and complementary to several appearance and local motion based methods and features, such as improved trajectories and deep learning features.
1 Introduction
Action recognition needs representations that capture video-wide temporal evolution despite variable and nonlinear action speeds. The paper proposes rank pooling, which learns chronological frame ordering and uses the ranking-function parameters as a video representation.
- Rapidly growing video content makes extracting information from realistic videos, including human actions and activities, a major challenge.
- Video-wide appearance evolution is difficult to model because action speed varies across videos and can change nonlinearly within one video.
- Existing temporal-evolution models have had disappointing recognition impact, while temporal pyramids remain insufficient for capturing characteristic action ordering.
- Rank pooling trains a linear ranking machine to arrange a video's frames chronologically based on frame content.
- The ranking-function parameters form a video representation that captures appearance and its evolution over time, while classifiers use it to distinguish actions.
- The broader functional-pooling framework uses parameters of fitted functions as representations and explores models beyond ranking functions.
2 Related work
Prior action-recognition methods capture local motion, temporal structure, or sequence dynamics, but often require complex models or do not represent video-wide evolution directly. Rank pooling instead learns video-specific temporal information without class labels and embeds chronological structure in function parameters.
- Many appearance- and motion-based methods capture local changes but are not designed for higher-level, video-wide evolution associated with actions.
- HMMs and CRFs model video dynamics but can require large training sets to estimate their parameters.
- Rank pooling captures video-specific dynamic information without class labels, then uses standard discriminative classifiers such as SVMs for action classes.
- Temporal pyramids, feature histograms, frequency-domain aggregation, and recurrent networks offer alternative ways to represent sequence information.
- Unlike exemplar SVMs, rank pooling learns relative frame ordering without requiring negative data.
- The ranking-function parameters provide a lower-dimensional embedding of chronological frame structure and can represent sequence dynamics as a meta-representation.
- The paper generalizes rank pooling to a framework using functional parameters as video representations and hypothesizes broader stable parametric mappings.
3 Video representations
The paper represents video-wide temporal dynamics by fitting functions to frame sequences and using their parameters as video descriptors. Rank pooling learns a ranking function that orders frames chronologically, while subspace pooling and other parametric models provide alternative functional representations.
- 3.1 Functional parameters as temporal representations: Each video is modeled as a smoothed sequence of frame vectors, with dynamics describing how the vector-valued input changes over time.Frames are represented as D-dimensional vectors and the smoothed sequence is denoted V.
- 3.1 Functional parameters as temporal representations: Functional pooling encodes a video’s appearance evolution through parameters of a function fitted to its frame sequence.The same function family is used across videos, while each video’s parameters characterize its specific dynamics.
- 3.2 Rank pooling: Rank pooling learns a linear ranking function that orders frames chronologically and uses its parameter vector as the video representation.Pair-wise ranking machines or point-wise SVR can learn the temporal encoding.
- 3.2 Rank pooling: Rank pooling focuses on relative frame ordering, making its representation insensitive to variability in action speed while retaining appearance information.Because the ranker acts on frame content, the representation captures appearance and its evolution over time.
- 3.4 Functional parameters as temporal pooling: Regularization makes rank pooling less susceptible to local observation noise, while fitting the data trend rather than aggregating frames through max or sum operators.The paper contrasts this with max and sum pooling, which may create artificial samples unrelated to the underlying temporal distribution.
- 3.4 Functional parameters as temporal pooling: Subspace pooling uses principal components of the video to represent temporal evolution, and the paper also experimentally explores HMMs and neural networks as functional models.A robust subspace projection can be obtained from the frame-time covariance structure, with dimensionality controlled by retaining leading components.
4 Frame representations
The paper compares frame representations for rank pooling, progressing from independent frames to smoothed moving averages and time-varying mean vectors, while also introducing nonlinear feature maps. These representations aim to make temporal evolution smoother and more informative for learning video dynamics.
- Independent Frame Representation: Independent frame representations can vary sharply, weakly connect features to time, and produce nonsmooth ranking scores.The predicted score s_t = u_i^T · v_t is less smooth when ranking is learned from independent frames.
- Moving Average (MA): Moving averages smooth frame signals and create local temporal dependencies, improving the relationship between features and frame time.The representation uses a window of size T and captures dependencies among surrounding frames.
- Moving Average (MA): Moving averages require choosing a window size and can leave the final video timestamps undefined because of boundary effects.Different action tempos make selecting T nontrivial, and the representation remains generally unstable.
- Time-Varying Mean Vectors: Time-varying mean vectors normalize the cumulative mean appearance up to time t, yielding a smoother relationship between the representation and time.The resulting ranking scores are described as smooth and almost monotonically increasing.
- Time-Varying Mean Vectors: The method can also process videos in reverse order to model temporal evolution from future frames toward past frames.The reverse formulation uses the same ranking objective while traversing the video backward.
- Nonlinear Rank Pooling: Nonlinear rank pooling applies nonlinear feature maps to each smoothed frame representation while retaining linear ranking machines in primal form.The paper describes a posneg map that doubles feature dimensionality and separates positive and negative components.
- Rank Pooling Pipeline: The processing pipeline extracts frame features, forms temporal representations, fits ranking functions, and uses their parameters as video representations.Figure 4 presents this sequence for rank pooling-based action recognition.
- Visualization: Rank pooling representations preserve crisp temporal changes in KTH action examples that standard mean pooling loses.The visualization includes forward and reverse rank pooling alongside standard average pooling across six action classes.
5 Overview
The action-classification pipeline converts frame-level descriptors into video-level parametric representations and then applies supervised classification. A visual KTH experiment shows that rank pooling captures motion structure more clearly than average pooling.
- Action Classification Pipeline: The pipeline processes each video frame, records its location, and extracts frame feature encodings.Typical inputs include HOG, HOF, MBH, and TRJ descriptors followed by Fisher-vector or Bag-of-Words encodings.
- Action Classification Pipeline: A parametric pooling model fits each video, and its parameters u_i become the video representation used for classification.The resulting training set pairs each video representation with its class label for supervised learning.
- Action Classification Pipeline: The final classifier uses supervised learning on the video representations, with nonlinear SVMs and χ2 feature maps applied to u_i.The feature vectors lie in R^D according to the pipeline description.
- Advantages: Rank pooling requires no supervised information to obtain video-order constraints and fits temporal evolution through a principled objective.The paper presents this as an advantage of using function parameters as video representations.
- Visualizing Dynamics: On KTH examples, forward and reverse rank pooling capture motion information, while average pooling loses temporal details.The visualization covers boxing, clapping, waving, walking, jogging, and running.
- Visualizing Dynamics: Rank pooling appears to distinguish opposite running directions and capture some motion periodicity in the KTH visualizations.This observation is reported for the running examples in Figure 5.
6 Experiments
Experiments evaluate rank pooling across generic, fine-grained, and gesture-recognition datasets, testing representations, variants, robustness, efficiency, and combinations with existing features. Rank pooling consistently captures video-wide temporal information and improves recognition, particularly when combined with local and static representations.
- Datasets: Rank pooling is evaluated on HMDB51, Hollywood2, MPII Cooking, and ChaLearn Gesture recognition datasets using consistent dataset-specific experimental settings.The benchmarks cover generic action, Hollywood action, fine-grained cooking activity, and gesture recognition.
- Frame representations and encodings: Time-varying mean vectors consistently outperform independent frames and moving averages for rank pooling across features.The paper attributes this to smoothing noise and stronger correlation between appearance and time.
- Action classification: +6.6% HMDB51, +7.1% Hollywood2, +8.6% MPII Cooking, and +9.3% ChaLearn: combining local methods with rank pooling substantially improves results.Forward-and-reverse rank pooling improves over forward-only pooling, with further gains from nonlinear rank pooling.
- Action classification: Hollywood2 improves by about 7% on average across classes, with larger gains for “answer phone” and “handshake,” but none for “drive car.”The paper links the exception to useful contextual evidence already present in the car scene.
- Robustness: Rank pooling remains stable when frames are randomly removed, including when up to 20% of frames are missing.Average pooling and temporal pyramids degrade as more frames are removed, whereas rank pooling changes little and may slightly improve.
- State-of-the-art comparison: Rank pooling outperforms local temporal methods and complements improved trajectories and CNN-based max-pooled features, improving prior state-of-the-art results by up to 10%.The method is also reported to achieve strong ChaLearn results without explicitly defining task-specific hand-posture or hand-trajectory steps.
7 Discussion and Conclusion
Rank pooling models video appearance and dynamics through learned temporal representations, with learning-to-rank models performing best among explored parametric alternatives. The work focuses on moderately long-video action classification while identifying broader potential applications.
- Rank pooling aggregates video information by fitting learning-to-rank models and using their parameters as the video representation.The method is presented as an unsupervised, learning-based temporal pooling approach.
- Learning-to-rank linear machines capture video temporal dynamics better than the other parametric models explored, including principal component analysis.
- Temporal smoothing and sequence preprocessing are important for modeling temporal evolution in video sequences.
- The current work primarily evaluates rank pooling for action classification on moderately long videos, while suggesting applications in other video tasks.Suggested applications include caption generation, action detection, retrieval, dynamic texture, and summarization.
- Rank pooling is concluded to be a novel and accurate method for capturing temporal evolution of appearances and dynamics in videos.