Source-linked AI summary
Bag of Visual Words and Fusion Methods for Action Recognition: Comprehensive Study and Good Practice
Xiaojiang Peng, Limin Wang, Xingxing Wang, Yu Qiao
TL;DR
Action recognition needs robust representations despite substantial variation in video content, while the effects of individual BoVW components and descriptor fusion had not been comprehensively established. The paper comprehensively evaluates BoVW stages and fusion methods, proposes a hybrid representation, and reports state-of-the-art results on three challenging datasets.
Problem
The effects of choices across BoVW stages and descriptor-fusion methods for action recognition had not been comprehensively investigated.
Method
The paper evaluates BoVW features, preprocessing, codebook generation, encoding, pooling and normalization, and fusion methods, then combines complementary BoVW models in a hybrid representation.
Results
61.1% on HMDB51, 92.3% on UCF50, and 87.9% on UCF101 are reported as state-of-the-art recognition results.
Takeaways & Limitations
Every BoVW step is crucial, and hybrid representation provides a simple yet effective basis for the reported action-recognition system.
Takeaways & Limitations
Super-vector encodings are promising but produce much higher-dimensional representations, making effective dimension reduction a future research direction.
Abstract
from arXiv · showhide
Video based action recognition is one of the important and challenging problems in computer vision research. Bag of Visual Words model (BoVW) with local features has become the most popular method and obtained the state-of-the-art performance on several realistic datasets, such as the HMDB51, UCF50, and UCF101. BoVW is a general pipeline to construct a global representation from a set of local features, which is mainly composed of five steps: (i) feature extraction, (ii) feature pre-processing, (iii) codebook generation, (iv) feature encoding, and (v) pooling and normalization. Many efforts have been made in each step independently in different scenarios and their effect on action recognition is still unknown. Meanwhile, video data exhibits different views of visual pattern, such as static appearance and motion dynamics. Multiple descriptors are usually extracted to represent these different views. Many feature fusion methods have been developed in other areas and their influence on action recognition has never been investigated before. This paper aims to provide a comprehensive study of all steps in BoVW and different fusion methods, and uncover some good practice to produce a state-of-the-art action recognition system. Specifically, we explore two kinds of local features, ten kinds of encoding methods, eight kinds of pooling and normalization strategies, and three kinds of fusion methods. We conclude that every step is crucial for contributing to the final recognition rate. Furthermore, based on our comprehensive study, we propose a simple yet effective representation, called hybrid representation, by exploring the complementarity of different BoVW frameworks and local descriptors. Using this representation, we obtain the state-of-the-art on the three challenging datasets: HMDB51 (61.1%), UCF50 (92.3%), and UCF101 (87.9%).
1 Introduction
The paper comprehensively evaluates BoVW components and fusion methods for action recognition, then proposes a hybrid representation that combines complementary BoVW models and descriptors. Its resulting system achieves state-of-the-art performance on HMDB51, UCF50, and UCF101.
- Video action recognition is difficult because actions vary across motion speeds, viewpoints, backgrounds, poses, objects, scenes, temporal boundaries, and video quality.
- BoVW with local features is a dominant action-recognition framework comprising feature extraction, preprocessing, codebook generation, encoding, and pooling or normalization.
- The study examines local features, ten encoding methods, eight pooling and normalization strategies, and fusion methods to identify effective BoVW practices.
- Every BoVW step contributes to recognition rate, while an improper choice in one step may counteract improvements from other steps.
- Descriptor complementarity and BoVW-model complementarity motivate the hybrid representation, which combines outputs from multiple BoVW models.
2 Framework of Bag of Visual Words
The BoVW framework converts local video features into a global representation through five processing stages before classification with a model such as a linear SVM.
- BoVW begins with feature extraction and feature preprocessing, then generates a codebook, encodes features, and applies pooling and normalization.
- The resulting global representation is fed into a classifier such as a linear SVM for action recognition.
2.1 Feature Extraction
The paper describes local spatiotemporal features as practical representations for action recognition, focusing on STIPs and improved dense trajectories with multiple descriptors.
- Low-level local features are popular because they are robust to background clutter and do not depend on human-body detection or tracking.
- Multiple descriptors represent different aspects of a local visual pattern, while sparse and dense features can behave differently across BoVW variants.
- STIPs use 3D-Harris to detect sparse regions with high motion salience and extract HOG and HOF descriptors.
- Improved Dense Trajectories track pixels with a median filter and extract five descriptor types from dense trajectories.
2.2 Feature Pre-processing
Feature preprocessing addresses the high dimensionality and correlation of local descriptors before unsupervised learning, using PCA and whitening; the evaluation finds it important for recognition.
- PCA maps original features to fewer linearly uncorrelated principal components, reducing dimensionality before subsequent BoVW processing.
- Whitening follows PCA to equalize feature variance across dimensions.
- The preprocessing transform maps f ∈ R^M to a PCA-whitened result x ∈ R^N using a PCA reduction matrix U and diagonal whitening matrix Λ.
- Although some encoding methods omit preprocessing, the evaluation finds it greatly improves recognition performance.
2.3 Codebook Generation
Codebook generation uses either feature-space partitioning with representative codewords or generative modeling of feature distributions. The paper contrasts k-means and GMM, including their assignment behavior and captured information.
- Codebook generation partitions feature space into codeword-centered regions or models the feature distribution generatively.k-means represents the first approach, while GMM represents the second.
- k-means: k-means constructs K clusters by assigning local descriptors to prototype codewords and minimizing an objective function.The assignment indicators specify descriptor membership in each cluster.
- GMM: GMM models the distribution of features using K Gaussian components with mixture weights, means, and covariance matrices.Its parameters are learned by maximum likelihood using the iterative EM algorithm.
- Comparison: k-means performs hard assignment, whereas GMM uses posterior probabilities for soft assignment and captures both codeword means and distribution shape.This gives GMM richer distributional information than k-means.
2.4 Encoding Methods
The paper organizes encoding methods into voting, reconstruction, and super-vector families, then relates their coding mechanisms and representation properties. Super-vector methods aggregate higher-order statistics into high-dimensional representations that outperform the other two groups in the reported experiments.
- The paper classifies thirteen encoding methods into voting-based, reconstruction-based, and super-vector groups.The formulations and output dimensions are summarized in Table 1.
- Voting based encoding methods: Voting methods directly assign each descriptor’s votes across codewords using strategies such as hard, soft, localized, salient, and group-salient coding.VQ votes only for the nearest codeword, SA votes for all codewords, and SA-k votes for k nearest codewords.
- Voting based encoding methods: VQ is a special case of SA-k when k equals 1, while salient methods modify hard-assignment weights using neighboring visual words.The paper describes VQ, salient coding, and group salient coding as hard-assignment strategies.
- Reconstruction based encoding methods: Reconstruction methods encode descriptors by minimizing reconstruction error with regularization, including sparse and locality-constrained approaches.OMP and SPC pursue sparsity, LCC explicitly encourages locality, and LLC provides a faster local-coordinate implementation.
- Super vector based encoding methods: Super-vector methods aggregate higher-order statistics into high-dimensional representations, including LTC, SVC, VLAD, and Fisher Vector.These representations retain richer codeword information and are reported to outperform voting- and reconstruction-based methods.
- The paper presents a unified generative perspective and summarizes practical techniques to expose relationships among encoding methods.This analysis is intended to provide insights for developing new encoding methods.
2.5 Pooling and Normalization Methods
Pooling converts descriptor codes into a global video representation, while normalization reduces dependence on the number of extracted descriptors. The paper reviews sum and max pooling together with ℓ1, ℓ2, power, and intra-normalization strategies.
- Pooling: Pooling aggregates the codes of all local descriptors into a global representation p for the video.The paper identifies sum pooling and max pooling as common strategies.
- Pooling: Max pooling selects the largest code response for each component across the extracted descriptors.For component k, the pooled value is the maximum among descriptor-specific code values.
- Pooling: Sparse features may prefer max pooling according to theoretical analysis of average and max pooling.
- Normalization: Normalization makes the pooled representation invariant to the number of extracted local descriptors.The paper discusses ℓ1-normalization, ℓ2-normalization, and power normalization, including combinations of power with ℓ1 or ℓ2 normalization.
- Normalization: Intra-normalization normalizes super-vector representations block by block, with each block corresponding to one codeword.The block norm may be ℓ1 or ℓ2.
3 Feature Fusion
Feature fusion combines complementary visual descriptors at descriptor, representation, or score level, with the appropriate level depending on descriptor dependence. The proposed hybrid representation additionally fuses outputs from multiple BoVW variants and descriptors.
- Video action recognition uses multiple descriptors because appearance, motion, and motion-boundary views provide distinct visual information.
- Fusion levels: Fusion can occur at descriptor level, representation level, or score level.Descriptor fusion concatenates descriptors before BoVW; representation fusion combines separate BoVW outputs; score fusion combines classifier scores.
- Choosing a fusion method: Highly correlated descriptors favor descriptor-level fusion, whereas independent descriptors favor score-level fusion for lower-dimensional and more stable classifier learning.
- Hybrid representation: The hybrid representation fuses outputs from multiple BoVW variants and descriptors to exploit their complementarity.The paper reports that it improves a single BoVW model and achieves state-of-the-art results on three challenging datasets.
4 Empirical Study
The empirical study shows that preprocessing, codebook size, encoding, pooling, and normalization choices materially affect BoVW action-recognition performance. Results also reveal feature-dependent trade-offs between accuracy, efficiency, and representation dimension.
- Pre-processing: PCA-Whiten substantially improves recognition across encoding methods, making preprocessing important even for methods beyond Fisher vectors.With PCA-Whiten, VQ reaches 56.1%, compared with 43.9% previously reported using the same local feature and encoding method.
- Codebook generation: Recognition generally improves with larger codebooks before reaching a plateau, but suitable sizes differ between super-vector and other encoding methods.Performance saturates around 256 GMM components for super-vector methods and 8,000 codewords for the other two encoding categories.
- Encoding methods: Super-vector encodings outperform other encoding categories because they preserve codeword affiliations and higher-order descriptor statistics.Fisher vectors typically outperform VLAD and SVC by retaining first- and second-order statistics, although high-dimensional representations may be costly.
- Feature-dependent trade-offs: Local-feature type affects codebook and encoding choices: large codebooks suit dense iDTs, whereas overly large codebooks can overfit sparse STIPs.Super-vector methods also have much higher dimensionality; for iDTs with 256 components, FV and VLAD dimensions are 102,400 and 51,200.
- Pooling and normalization: Sum pooling with power ℓ2-normalization is the strongest general combination, while intra-normalization benefits dense features but can hurt sparse ones.Power normalization is especially effective after sum pooling, where it smooths bursty histograms and reduces the influence of frequent codewords.
5 Conclusion
The paper comprehensively studies five BoVW components and proposes a hybrid representation. The resulting action recognition system achieves state-of-the-art performance on three challenging datasets.
- The study examines local features, pre-processing, encoding, pooling and normalization, and fusion methods within the BoVW pipeline.
- Every BoVW pipeline step contributes to final recognition rate, so an improper choice can counteract improvements from other steps.
- The proposed hybrid representation is simple yet effective and achieves state-of-the-art performance on three challenging datasets.