PaperlayerView exact revision on arXiv

Source-linked AI summary

A Discriminative CNN Video Representation for Event Detection

Zhongwen Xu, Yi Yang, Alexander G. Hauptmann

arXiv:1411.4006v1cs.CV

TL;DR

Large-scale event detection needs video representations that reduce heavy feature-extraction costs without sacrificing performance. This paper combines CNN descriptor encoding with latent concept descriptors, achieving over 30% relative improvement over prior state-of-the-art representations on MED datasets.

  • Problem

    Improved Dense Trajectories offer strong event-detection performance but incur heavy computation, while CNN average pooling does not surpass them.

  • Method

    The paper encodes frame-level CNN descriptors, uses latent concept descriptors as frame representations, and applies Product Quantization for faster event search.

  • Results

    More than 30% relative improvement over state-of-the-art video representations is achieved on the large-scale MED dataset.

  • Takeaways & Limitations

    The representation delivers stronger large-scale event detection while remaining feasible on a single GPU-equipped machine and supporting faster compressed-data event search.

  • Takeaways & Limitations

    High-dimensional pool5 descriptors can cause heavy computation and instability in video pooling.

Abstract

from arXiv · show

In this paper, we propose a discriminative video representation for event detection over a large scale video dataset when only limited hardware resources are available. The focus of this paper is to effectively leverage deep Convolutional Neural Networks (CNNs) to advance event detection, where only frame level static descriptors can be extracted by the existing CNN toolkit. This paper makes two contributions to the inference of CNN video representation. First, while average pooling and max pooling have long been the standard approaches to aggregating frame level static features, we show that performance can be significantly improved by taking advantage of an appropriate encoding method. Second, we propose using a set of latent concept descriptors as the frame descriptor, which enriches visual information while keeping it computationally affordable. The integration of the two contributions results in a new state-of-the-art performance in event detection over the largest video datasets. Compared to improved Dense Trajectories, which has been recognized as the best video representation for event detection, our new representation improves the Mean Average Precision (mAP) from 27.6% to 36.8% for the TRECVID MEDTest 14 dataset and from 34.0% to 44.6% for the TRECVID MEDTest 13 dataset. This work is the core part of the winning solution of our CMU-Informedia team in TRECVID MED 2014 competition.

1. Introduction and Related Work

Complex event detection is challenging because events exhibit greater intra-class variation than concepts or actions, while established video representations such as improved Dense Trajectories are computationally expensive at scale. This paper addresses CNN-based representation limitations with encoding, latent concept descriptors, and efficient search acceleration.

  • 1. Introduction and Related Work: Improved Dense Trajectories achieve strong complex event detection performance but incur heavy computation costs that restrict large-scale use.They outperform features such as STIP and Dense SIFT.
  • 1. Introduction and Related Work: CNN video representations face limited labeled data, coarse frame-level supervision, and the need for a discriminative aggregation method.MEDTest 13 and MEDTest 14 provide only 100 positive examples per event, video labels do not identify which frames contain event semantics, and average pooling is the standard baseline for frame descriptors.
  • 1. Introduction and Related Work: The paper introduces encoding techniques for CNN descriptors and latent concept descriptors formed by aggregating information from multiple deeper-network spatial locations.Frames pass through the deep CNN only once for descriptor extraction.
  • 1. Introduction and Related Work: The combined representation delivers more than 30% relative improvement over the state-of-the-art video representation on the large-scale MED dataset.The approach can run on a single GPU-equipped machine in two days.
  • 1. Introduction and Related Work: The work also applies Product Quantization to the CNN video representation to accelerate event-search execution.This is presented as an additional contribution alongside the representation improvements.

2. Preliminaries

The work uses the 16-layer VGG architecture from the ILSVRC 2014 classification winning solutions [36] and adopts established notation for CNN activations [6] [11].

  • Architecture and notation: The network has 13 convolutional layers, five followed by max-pooling, and three fully connected layers, following the 16-weight-layer VGG configuration [36].The paper denotes activations of the last pooling layer as pool5 and the fully connected layers as fc6 and fc7 [6] [11].

3. Video CNN Representation

The section builds video-level CNN representations by comparing standard pooling with Fisher-vector and VLAD encoding, finding VLAD most discriminative for CNN descriptors. It also proposes latent concept descriptors from pool5 to retain spatial information without the prohibitive dimensionality of flattened features.

  • Video pooling: The standard representation normalizes frame descriptors, averages them across frames, and re-normalizes the resulting video vector, while max pooling performs similarly and is omitted.The video representation is x_video = 1/N sum_i x_i, where x_i is a frame descriptor and N is the number of extracted frames.
  • Video pooling: Fisher vectors and VLAD encode all frame descriptors, with Fisher vectors using a GMM and VLAD using K-means centers; both apply normalization after encoding.Fisher vectors have size 2D′K after PCA, whereas VLAD has size D′K; VLAD-k with k = 5 is used by default and receives intra-normalization.
  • Video pooling: VLAD has the best discriminative ability for CNN descriptors, with positive-negative similarities nearer zero and less overlap with positive-positive similarities than average pooling or Fisher vectors.The comparison uses cosine-similarity distributions on the MEDTest 14 training set; the conclusion is consistent with Section 5.1.
  • Latent concept descriptors: Flattening pool5 preserves spatial information but produces 25,088-D VGG features versus 4,096-D fc6 and fc7 features, creating substantial computational and video-pooling stability costs.In the stated VGG configuration, pool5 has dimensions a × a × M with a = 7 and M = 512.
  • Latent concept descriptors: The proposed latent concept descriptors represent pool5 through responses of independent last-layer convolutional filters, using each filter’s response as a descriptor dimension.The approach treats convolutional filters as generalized linear classifiers corresponding to latent concepts and avoids directly using the high-dimensional flattened pool5 vector.

4. Experiment Settings

Experiments use the TRECVID MEDTest 13 and 14 event-detection benchmarks under 100Ex and 10Ex training conditions, with CNN-based features evaluated by linear SVM and mAP.

  • Datasets: Experiments use the largest labeled event-detection datasets, TRECVID MEDTest 13 and MEDTest 14, each containing 20 complex events with 10 overlapping.MEDTest 13 covers E006–E015 and E021–E030, whereas MEDTest 14 covers E021–E040.
  • Training conditions: The evaluation uses linear SVM with 100Ex and 10Ex conditions, providing 100 or 10 positive exemplars per event, respectively.For 100Ex, 5-fold cross-validation selects C; for 10Ex, C is fixed at 1 following.
  • Evaluation: Labels for MEDEval 13 and MEDEval 14 are not publicly available, limiting direct access to the evaluation annotations.
  • Feature extraction: CNN descriptors are sampled every five frames from center crops using Caffe and the best publicly available model [36], with VLAD and Fisher vectors generated by vlfeat.The preprocessing follows [6].
  • Evaluation: Performance is measured using Mean Average Precision (mAP) for binary event classification according to the NIST standard.

5. Experiment Results

The experiments show that VLAD encoding of CNN and latent concept descriptors substantially improves video event detection over average pooling and IDT, while SPP, fusion, and compression preserve or further improve performance. The resulting representation achieves strong benchmark results with practical storage and prediction speed.

  • CNN video pooling: VLAD-encoded CNN features significantly outperform IDT and CNN average pooling across MEDTest 13 and MEDTest 14 under both 100Ex and 10Ex settings.The comparison includes improved Dense Trajectories and its HOG and MBH components.
  • Latent concept descriptors: VLAD-encoded latent concept descriptors outperform average-pooled pool5 features, while adding SPP further increases performance with minimal computation cost.SPP aggregates deeper-stage features to provide multiple levels of spatial information.
  • Parameter analysis: PCA dimensions above 256-D retain similar performance, whereas 128-D damages it; increasing VLAD centers improves discrimination until K = 512 becomes overly sparse.These findings characterize the tradeoff between performance, storage, and descriptor sparsity.
  • Parameter analysis: Removing SSR power normalization reduces mAP from 33.2% to 27.0%, while removing intra-normalization reduces it from 33.2% to 30.6%.The results demonstrate that both normalization steps materially contribute to VLAD performance.
  • Compression and efficiency: PQ compression with B = 4 maintains or slightly improves performance, stores the MEDEval 14 collection in 3.1 GB, and enables prediction on 200,000 videos in 4.1 seconds per event using 20 threads.B = 8 causes a slight performance drop; the speed test uses compressed rather than original features.
  • Overall comparison: 44.6% mAP on 100Ex and 29.8% on 10Ex significantly outperform the top TRECVID MED 2013 competition systems, despite their use of more than 10 feature types and sophisticated schemes.The reported results are for MEDTest 13 and compare against systems combining extensive feature sets.

6. Conclusion

The paper addresses the high computational cost of TRECVID MED by generating video representations from CNN descriptors, introducing encoding techniques and latent concept descriptors for this purpose.

  • The work targets the substantial computation costs of feature extraction and classification in TRECVID Multimedia Event Detection.
  • It is presented as the first application of encoding techniques to generate video representations from CNN descriptors.
  • The paper proposes latent concept descriptors to generate CNN-based video representations.

Appendices · A. Non-linear Classifiers on CNN Descriptors

The appendix shows that exponential-χ2 and RBF kernel SVMs significantly improve CNN-descriptor video classification over standard linear classifiers [6] [11]. It also explains their feature-dependent distance formulations and notes that non-linearity limits scalability to very large video collections.

  • A. Non-linear Classifiers on CNN Descriptors: Fine-tuning improvements are larger for fc6 and fc7 than pool5, suggesting domain-specific non-linear classifiers account for much of the gain [11].
  • A. Non-linear Classifiers on CNN Descriptors: Non-linear classifiers significantly boost video classification over the standard linear approach when applied to CNN descriptors after average pooling [6] [11].The evaluated classifiers are exponential-χ2 kernel SVM and RBF kernel SVM.
  • A. Non-linear Classifiers on CNN Descriptors: Kernel similarity depends on a distance metric between feature points and a scale parameter σ, with A defined from average training-feature distances.
  • A. Non-linear Classifiers on CNN Descriptors: The kernels operate on deep features from CNN intermediate layers, including pool5, fc6, and fc7.
  • A. Non-linear Classifiers on CNN Descriptors: For the exponential-χ2 kernel, Dist(X_i, X_j) is χ2 distance, whereas the RBF kernel uses Euclidean distance.
  • A. Non-linear Classifiers on CNN Descriptors: Although effective after average pooling, non-linearity is difficult to apply to large-scale event detection such as MEDEval 14 with 200,000 videos.The appendix therefore focuses on improving average pooling at the scale of MEDTest 13 and MEDTest 14.

B. Experiment Results for Non-linear Classifiers

Non-linear classifiers substantially improve CNN-based event detection over linear classifiers and hand-crafted improved Dense Trajectories across MEDTest 13 and 14 settings, while preserving low-dimensional features. Exponential-χ2 SVM performs best but introduces efficiency costs that explicit feature mapping only partially addresses.

  • Overall comparison: Non-linear classifiers effectively boost performance, creating a clear advantage over improved Dense Trajectories while retaining low-dimensionality benefits.The experiments compare MEDTest 13 and MEDTest 14 across 100Ex and 10Ex settings, with improved Dense Trajectories included as a baseline.
  • Kernel comparison: Exponential-χ2 SVM outperforms both linear and RBF classifiers across all tested layers and settings, averaging 5% absolute mAP above improved Dense Trajectories.Both RBF and exponential-χ2 classifiers significantly outperform the linear classifier; the exponential-χ2 advantage holds across all layers and settings.
  • Layer comparison: Among linear models, fully connected layers with ReLU neurons significantly outperform corresponding layers without ReLU neurons except on MEDTest 14 10Ex.This pattern is consistent with the layer choices reported in previous work [11].
  • Efficiency limitation: Non-linear classifiers require kernel-matrix computations against all training exemplars for each new test exemplar, making them unsuitable for large-scale efficiency requirements.Linear classifiers instead require only dot-product operations with the learned classifier parameters.
  • Approximation: Explicit feature mapping reduces the non-linear performance gap by about 2% absolute mAP while using three times the original feature dimensionality.The drop is attributed mainly to approximating the χ2 kernel.
Loading 1411.4006v1…