PaperlayerView exact revision on arXiv

Source-linked AI summary

Attention Clusters: Purely Attention Based Local Feature Integration for Video Classification

Xiang Long, Chuang Gan, Gerard de Melo, Jiajun Wu, Xiao Liu, Shilei Wen

arXiv:1711.09550v1cs.CVcs.LG

TL;DR

Video classification research often models temporal patterns, but the paper questions whether long-term temporal information is necessary for common tasks. It proposes attention clusters with a shifting operation to integrate local feature sets, achieving competitive results across three datasets, including 79.4% top-1 and 94.0% top-5 accuracy on Kinetics.

  • Problem

    The paper examines whether temporal information, especially longer-term patterns, is necessary for competitive video classification when local features may be identifiable and approximately unordered.

  • Method

    The paper uses purely attention-based local-feature integration with multiple attention units grouped into attention clusters and a shifting operation for more diverse signals.

  • Results

    79.4% top-1 and 94.0% top-5 accuracy are achieved on the Kinetics validation set, with competitive results across three video-classification datasets.

  • Takeaways & Limitations

    Attention clusters support effective single- or multimodal video classification while abandoning temporal cues and accelerating training.

  • Takeaways & Limitations

    The paper leaves application to low-level local features and integration into end-to-end-trained networks for future work.

Abstract

from arXiv · show

Recently, substantial research effort has focused on how to apply CNNs or RNNs to better extract temporal patterns from videos, so as to improve the accuracy of video classification. In this paper, however, we show that temporal information, especially longer-term patterns, may not be necessary to achieve competitive results on common video classification datasets. We investigate the potential of a purely attention based local feature integration. Accounting for the characteristics of such features in video classification, we propose a local feature integration framework based on attention clusters, and introduce a shifting operation to capture more diverse signals. We carefully analyze and compare the effect of different attention mechanisms, cluster sizes, and the use of the shifting operation, and also investigate the combination of attention clusters for multimodal integration. We demonstrate the effectiveness of our framework on three real-world video classification datasets. Our model achieves competitive results across all of these. In particular, on the large-scale Kinetics dataset, our framework obtains an excellent single model accuracy of 79.4% in terms of the top-1 and 94.0% in terms of the top-5 accuracy on the validation set. The attention clusters are the backbone of our winner solution at ActivityNet Kinetics Challenge 2017. Code and models will be released soon.

1. Introduction

The paper questions whether long-term temporal patterns are necessary for common video classification and instead investigates attention-based integration of local features. It motivates this approach through local-feature redundancy, identifiability, approximate unorderedness, and multiple informative components.

  • Motivation: Video classification methods commonly use CNNs or RNNs to model temporal interactions, especially longer-term patterns.The paper questions whether such temporal information is indispensable for common classification tasks.
  • Motivation: Local features often repeat across frames, so holistic aggregation can reduce redundancy without preserving every temporal change.Adjacent frames may differ only minutely, particularly during slow movement.
  • Motivation: A few highly informative frames may provide enough information for classification, making temporal-pattern analysis unnecessary in some videos.The paper illustrates this with a brushing-teeth clip recognizable from its initial frame.
  • Motivation: Local features may be approximately unordered for classification, although temporal order remains relevant to understanding a video's narrative.The paper notes that humans can categorize a pole-vault video even after its frames are permuted.
  • Proposed direction: Attention naturally aggregates repeated features, emphasizes significant local signals, and accepts unordered sets with varying numbers of features.These properties motivate abandoning temporal cues for purely attention-based local-feature integration.
  • Proposed direction: Multiple attention units form attention clusters because one unit may focus on only one video aspect and discard other informative components.The proposed shifting operation increases diversity among attention units, improving training efficiency and classification accuracy over simple concatenation.

2. Related Work

Prior video-classification research uses attention, CNNs, optical flow, and recurrent models to extract or integrate visual and temporal information. The paper positions its approach against limitations of hard or guided attention, simple pooling, and unsatisfactory RNN-based accuracy.

  • Attention-based methods: Early attention models used hard binary region selection with recurrent networks, which may be difficult to train.Soft attention replaced hard selections with weighted averages, while video models such as Soft-Attention LSTM used recurrent glimpses.
  • Attention-based methods: Soft attention models may require supplementary guidance sources that add computational cost without sufficient classification improvements.This motivates alternative attention formulations for video classification.
  • Attention-based methods: Multiple self-attention units have been used for representations, but penalty functions forcing distinct weight vectors may be too restrictive for video classification.The paper distinguishes its setting from approaches designed to enforce attention diversity through penalties.
  • CNN and motion methods: CNNs extract local spatio-temporal features for video classification, while simple pooling and fusion methods provide limited gains.Prior work also combines RGB appearance and optical-flow motion signals, with probability fusion improving accuracy.
  • Recurrent methods: RNN and LSTM architectures model long-term temporal interactions, but reported video-classification accuracy has been unsatisfactory.This may indicate that long-term temporal interactions are not crucial for these tasks.

3. Approach

The approach integrates unordered local video features into fixed-length global representations using attention units, clusters, and a shifting operation. For multimodal inputs, independent clusters produce modality-specific representations that are concatenated for classification.

  • 3. Approach: The framework comprises local feature extraction, local feature integration, and global feature classification using CNNs, attention clusters, and fully connected and softmax layers.The main contribution lies in local feature integration.
  • 3.1. Local Feature Set: Local features are represented as rows of an L × M matrix X, allowing variable numbers of features and treating them as an unordered set.Permuting rows should not affect results, while the system produces fixed-length global vectors for classification.
  • 3.2. Attention: Attention computes global features as weighted averages of local feature vectors, with the weighting function assigning one weight to each local feature.The weighting function takes X as input and outputs a weight vector a with ℓ1 norm 1.
  • 3.3. Attention Clusters: An attention cluster groups independent attention units operating on the same input to represent multiple pertinent video components.For cluster size N and feature dimension M, concatenating unit outputs yields a global feature of dimensionality NM.
  • 3.4. Shifting Operation: The shifting operation adapts each attention unit with a learnable linear transformation and separate ℓ2 normalization, encouraging units to diverge while preserving scale-invariance.The operation shifts weighted sums in feature space and facilitates optimization of the entire network.
  • 3.5. Overall Architecture for Video Classification: For multimodal video classification, separate attention clusters process different feature sets, and their modality-specific outputs are concatenated into a global representation.The design addresses differing modality distributions, dimensionalities, and scales.

4. Analysis and Visualization

The Flash–MNIST analysis evaluates attention weighting functions, cluster sizes, and shifting through accuracy and attention-map visualizations. Larger clusters improve performance and convergence, while shifting diversifies attention and raises accuracy.

  • Flash–MNIST Dataset: Flash–MNIST extends digit classification to 25-frame videos, requiring recognition of the digit set across 2^10 = 1024 categories.The dataset uses noisy backgrounds and separately generated training and test samples.
  • Effect of Weighting Function: Attention weighting functions outperform Average on Flash–MNIST, with FC1 selected as the default because FC2 adds computation without consistent benefit.FC2 performs slightly better than FC1 for small clusters, but its extra parameters do not yield overall benefits.
  • Effect of Attention Cluster Size: Increasing cluster size substantially improves classification when small, then produces nearly unchanged results after reaching a certain level.Comparisons replicate Average outputs to keep parameter counts identical apart from weighting-function parameters.
  • Effect of Attention Cluster Size: Larger clusters converge faster and can reduce total training time despite requiring more computation for attention and shifting operations.The operations remain sufficiently efficient for reasonably large clusters.
  • Attention Cluster Visualization: Eight-unit attention clusters learn different information, with separate units attending to distinct digits such as 4, 6, and 7.The visualization compares attention maps with and without shifting; larger HSV values indicate larger weights.
  • Effect of Shifting Operation: 87.1% accuracy with shifting exceeds 83.3% without shifting, while the shifted model also converges more rapidly at the same cluster size.Without shifting, some attention weights match; shifting makes the weights diverge and is associated with more diversified information.

5. Experiment on Real Video Classification

The paper evaluates attention-cluster integration on three trimmed video datasets using RGB, flow, and audio features. Shifting improves attention-cluster performance and multimodal integration achieves strong Kinetics results.

  • Datasets: The experiments evaluate attention-cluster methods on three trimmed video classification datasets: UCF101, HMDB51, and Kinetics.UCF101 and HMDB51 use three training/testing splits; Kinetics results are reported on the validation split.
  • Feature extraction: RGB, flow, and audio local features are extracted with CNNs and integrated using modality-specific attention clusters.The multimodal architecture concatenates the outputs of independent attention clusters into a global representation.
  • Single-modality experiments: With shifting, training remains stable for large attention clusters, whereas without shifting accuracy can decline as cluster size increases because of overfitting and harder optimization.For small clusters, increasing the number of attention units improves accuracy in both settings.
  • Single-modality experiments: Shifting universally improves accuracy over no shifting and leads to faster convergence, suggesting greater diversity in the attention mechanism.The comparison is reported across the tested single-modality settings.
  • Single-modality experiments: Attention clusters improve pretrained TSN results by 2.0% for RGB, 1.5% for flow, and 2.6% for audio in top-1 accuracy.The method also beats other fusion methods using the same local features.
  • Multimodal integration: The best multimodal configuration reaches 79.4% top-1 and 94.0% top-5 accuracy on the Kinetics validation set using 64 RGB and 32 flow and audio attention units.The approach also improves over three-stream fusion methods using the same local features.
  • Comparison with state of the art: On UCF101 and HMDB51, the approach improves over CNN two-stream fusion and achieves competitive results against published methods; on Kinetics, it improves over strong fusion baselines.The Kinetics comparison reports a state-of-the-art result in the paper’s evaluation.

6. Conclusion

The conclusion presents attention clusters with shifting as an architecture for integrating local video features without relying on temporal cues. Experiments on three video datasets report strong results for both single- and multimodal inputs, while future work targets low-level features and end-to-end training.

  • Conclusion: The proposed architecture integrates local feature sets using attention clusters with a shifting operation.The paper analyzes and visualizes the mechanism on Flash–MNIST.
  • Conclusion: Experiments on three well-known video classification datasets find excellent results for single-modality and multimodal integration while accelerating training.
  • Future work: Future work will apply the architecture to low-level local features and investigate relationships between features at different spatial coordinates.The authors also plan to integrate it into end-to-end-trained networks.

A. Details of the Flash–MNIST Experiments

The appendix provides implementation details for generating Flash–MNIST, extracting local features, and training the attention-cluster experiments.

  • Appendix scope: The appendix documents Flash–MNIST dataset generation, local feature extraction, and attention-cluster training procedures.

A.1. Dataset Generation

Flash–MNIST is generated from noisy frames containing randomly sampled MNIST digits, then used to train CNN-based frame feature extractors.

  • Dataset generation: Flash–MNIST contains 102,400 training samples and 10,240 test samples generated through synthetic video construction.
  • Dataset generation: Each video sample begins with 25 randomly generated 28 × 28 noise frames.Noise pixel intensities are sampled according to the MNIST training-data distribution.
  • Dataset generation: A category is sampled from 1024 possible Flash–MNIST categories, and each digit receives one or two randomly selected MNIST images.A target video may contain 0–20 MNIST digit images.
  • Dataset generation: Sampled digit images are inserted into randomly selected frames by overlaying them with the background using the maximum value at each pixel.
  • Local feature extraction: The local-feature extractor is pretrained on digit or noisy-background classification before features are extracted from Flash–MNIST frames.The pretraining task has 11 categories: digits 0–9 or a noisy background.
  • Local feature extraction: The pretrained CNN’s final fully connected layer is removed, yielding 50-dimensional local features for Flash–MNIST frames.

A.3. Attention Cluster Training Details

The model extracts local features, integrates them with attention clusters into a global representation, and classifies videos with fully connected and softmax layers.

  • Attention clusters produce the global feature representation from extracted local features before classification.
  • A fully connected layer with 1024 hidden units and a softmax layer perform classification.
  • Dropout with probability 0.5 is applied before the final fully connected layer to reduce overfitting.
  • Parameters are updated with Adam at a learning rate of 0.001 for at most 100 epochs.

B. Details of Multimodal Integration Methods

The multimodal integration comparison includes simple averaging and flattening baselines alongside methods using temporal modeling, including TS-LSTM, Temporal-Inception, and bidirectional LSTMs.

  • The comparison evaluates Average and Flatten as basic multimodal integration baselines.
  • Average: Average concatenates the global averages of each modality and feeds the resulting global feature to a fully connected classifier.
  • Flatten: Flatten flattens each modality's local features, concatenates the three modality outputs, and uses them for classification.
  • Temporal modeling: TS-LSTM and Temporal-Inception are extended from two-modality integration methods to three modalities, with TS-LSTM using 5 segments.
  • Temporal modeling: Bidirectional LSTMs process RGB, flow, and audio features with 1024/512/512 hidden units, whose averaged outputs are concatenated for classification.
Loading 1711.09550v1…