PaperlayerView exact revision on arXiv

Source-linked AI summary

The ImageNet Shuffle: Reorganized Pre-training for Video Event Detection

Pascal Mettes, Dennis C. Koelma, Cees G. M. Snoek

arXiv:1602.07119v1cs.CV

TL;DR

Video event detection commonly uses deep networks pre-trained on ImageNet’s 1,000-class subset, leaving most of the full hierarchy unused. This paper reorganizes that hierarchy for pre-training, and reports improved event-detection performance, including state-of-the-art results on TRECVID benchmarks.

  • Problem

    Leading video event-detection approaches pre-train on ImageNet’s 1,000-class subset, leaving over 90% of the full hierarchy’s images unused despite its 21,814 classes and more than 14 million images.

    Source: §1 · p. 1

  • Method

    The paper reorganizes the complete ImageNet hierarchy using contrasting bottom-up and top-down approaches to address class imbalance and over-specificity during deep-network pre-training.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Reorganized pre-training outperforms standard pre-training, and the approaches achieve state-of-the-art event detection on TRECVID MED 2013 and 2015.

    Source: §1 · p. 2, §6 · p. 7, §5.4 · p. 7

  • Takeaways & Limitations

    The reorganized deep representations remain complementary to motion and audio features, whose fusion improves event-detection performance.

    Source: §5.3 · p. 6, §5.4 · p. 7

    Fusing the VLAD encoding with motion and audio did not improve performance, and VLAD required more computation and storage than averaging.

    Source: §5.3 · p. 7

Abstract

from arXiv · show

This paper strives for video event detection using a representation learned from deep convolutional neural networks. Different from the leading approaches, who all learn from the 1,000 classes defined in the ImageNet Large Scale Visual Recognition Challenge, we investigate how to leverage the complete ImageNet hierarchy for pre-training deep networks. To deal with the problems of over-specific classes and classes with few images, we introduce a bottom-up and top-down approach for reorganization of the ImageNet hierarchy based on all its 21,814 classes and more than 14 million images. Experiments on the TRECVID Multimedia Event Detection 2013 and 2015 datasets show that video representations derived from the layers of a deep neural network pre-trained with our reorganized hierarchy i) improves over standard pre-training, ii) is complementary among different reorganizations, iii) maintains the benefits of fusion with other modalities, and iv) leads to state-of-the-art event detection results. The reorganized hierarchies and their derived Caffe models are publicly available at http://tinyurl.com/imagenetshuffle.

1. INTRODUCTION

Video event detectors use deep-network representations pre-trained on ImageNet, usually on its 1,000-class subset. This paper instead reorganizes the complete hierarchy to address uneven class sizes and over-specific categories, evaluating the resulting representations on event detection.

1. INTRODUCTION

  • The target events include Renovating a home, Birthday party, and Attempting a bike trick, and leading approaches learn video representations with deep convolutional networks.Their pipeline pre-trains on ImageNet, extracts frame features, and pools them into a video representation.Source: §1 · p. 1
  • The complete ImageNet hierarchy contains over 14 million images and 21,814 classes, but standard pre-training uses only 1,000 classes and leaves over 90% of its images untouched.The authors investigate using the full dataset for event detection in web videos.Source: §1 · p. 1
  • The hierarchy is imbalanced: Yorkshire Terrier has 3,072 images, while 296 other classes have only one image each.Some categories, including Siderocyte and Gametophyte, also seem too specific for video event detection.Source: §1 · pp. 1–2
  • The proposed protocols reorganize the full hierarchy to address class imbalance and over-specificity, with state-of-the-art results on TRECVID MED 2013 and 2015.The Caffe models and video-feature extraction instructions are publicly available.Source: §1 · p. 2

2. RELATED WORK

Prior event-detection systems pre-train deep networks on ImageNet, extract frame representations, and pool them into video representations. This paper differs by reorganizing concept-class hierarchies before event training, rather than changing representations afterward or organizing event classes.

2.1 Event detection with pre-trained networks

  • Event-detection pipelines use pre-trained deep networks to represent frames and pool them into fixed-size video representations; this work changes the pre-training data by using the complete ImageNet hierarchy.Average pooling and VLAD or Fisher Vector encodings are among the described pooling approaches.Source: §2.1 · p. 2
  • The paper also evaluates fusion with motion and audio features to test how deep representations combine with information from other modalities.The cited modalities are Motion Boundary Histogram features and MFCC features, encoded using Fisher Vectors.Source: §2.1 · p. 2

2.2 (Re)organizing hierarchies for events

  • Unlike EventNet’s hierarchy of event relations [26], this work uses concept-class relations to build a general set of concepts for network pre-training.It reorganizes visual ontologies before event training rather than changing representations afterward using text or video examples.Source: §2.2 · p. 2

3. REORGANIZED PRE-TRAINING

The paper uses WordNet-linked ImageNet class relations to reorganize the hierarchy for pre-training. It proposes bottom-up operations to merge, promote, or subsample classes and a top-down breadth-first selection strategy.

3. REORGANIZED PRE-TRAINING

  • The two approaches use opposing directions through the hierarchy: bottom-up starts from its original deeper classes, while top-down retains more general classes through thresholded selection.The top-down method uses tt as the minimum image count for class selection.Source: §3 · p. 2, §3.2 · p. 4

3.1 Bottom-up reorganization

  • Roll merges classes connected by a single child-parent link, where the child may be over-specific and the parent has no other children.The example moves Black mamba and Green mamba images into Mamba.Source: §3.1 · p. 3
  • Bind combines a sparse subtree when its combined image count is below tb, grouping a parent and children into a semantically coherent class.The example combines Hammerhead shark images with those of its three sparsely represented children.Source: §3.1 · p. 3
  • Promote moves a class below threshold tp to its parent, while subsampling randomly reduces images in classes above threshold ts.Promotion retains scarce examples without preserving them as imbalanced separate classes; subsampling addresses the reverse case of over-populated classes.Source: §3.1 · p. 4
  • Bottom-up reorganization applies four operations in order: roll single-child connections, bind sparse subtrees, promote remaining classes below tp, and subsample classes above ts.The thresholds control combined subtree size for binding and image counts for promotion and subsampling.Source: §3.1 · p. 4

3.2 Top-down reorganization

  • Top-down reorganization starts at the hierarchy’s head node and proceeds breadth-first, selecting classes with at least tt images until the desired class count is reached.At each next layer, classes are ordered by image count so those with the most images are selected first if the class-count limit is reached.Source: §3.2 · p. 4

4. EXPERIMENTAL SETUP

The experiments use TRECVID MED 2013 web videos and deep convolutional networks. They extract frame features from specified network layers, average representations over each video, and train event-specific classifiers.

4.1 Dataset

  • TRECVID MED 2013 has roughly 27,000 test videos and 20 everyday events, with separate tasks providing either 10 or 100 positive training videos per event.Each event classifier also uses roughly 5,000 negative videos and is evaluated by mean Average Precision on ranked test videos.Source: §4.1 · p. 4

4.2 Implementation details

  • The evaluation focuses on 22-layer GoogLeNet and also compares with AlexNet, using Caffe with the networks’ provided layer definitions and hyperparameters.AlexNet has five convolutional and three fully connected layers.Source: §4.2 · p. 4
  • Frame features come from AlexNet’s second fully connected layer (4,096 dimensions) or GoogLeNet’s pool5 layer (1,024 dimensions); soft-max features have one dimension per hierarchy class.Source: §4.2 · p. 4
  • Video representations average frame features, apply ℓ1-normalization, and train a separate χ2-kernel SVM for each event with C set to 100.Source: §4.2 · p. 4

5. EXPERIMENTS

The experiments test bottom-up class-merging settings, compare reorganized pre-training with standard pre-training, and evaluate fusion across layers, networks, encodings, and modalities. Reorganized pre-training and the resulting fusions perform strongly on TRECVID MED 2013 and 2015, including state-of-the-art comparisons.

5.1 Bottom-up operation parameters

  • 0.446 mAP with 100 positives and 0.296 with 10 positives are the best fully connected results from 4,437 classes; soft-max peaks at 0.441 and 0.286 from 12,988 classes, respectively.The fully connected layer favors merging more classes, whereas soft-max performance improves when more classes are maintained.Source: §5.1 · p. 5

5.2 Standard versus reorganized pre-training

  • Top-down soft-max improves over standard GoogLeNet by 8.6% absolute mAP with 100 positives and 9.2% with 10; bottom-up and top-down are competitive, with layer-specific advantages.The approaches outperform standard pre-training; bottom-up is slightly better on fully connected features, while top-down is slightly better on soft-max features.Source: §5.2 · p. 5, Figure 5 · p. 6

5.3 Fusing representations and modalities

  • Fusing fully connected and soft-max layers helps both reorganizations, while bottom-up/top-down fusion reaches 0.475 mAP with 100 positives and 0.324 with 10 on MED 2013.The approaches differ by an average of 2.7% per event; the authors note that features from both layers can be extracted in one network pass.Source: §5.3 · p. 6, Table 1 · p. 6
  • For the 10-positive task, bottom-up VLAD reaches 0.339 mAP versus 0.305 for frame averaging.VLAD fusion with motion and audio did not improve results; its greater computational and storage costs led the authors to focus on averaging for that fusion.Source: §5.3 · pp. 6–7
  • Deep networks outperform motion and audio individually, while combining the modalities produces a further performance jump.The authors attribute the complementary information to CNNs focusing on spatial visual information while excluding temporal and audio information.Source: §5.3 · p. 6, Table 2 · p. 7

5.4 Comparison to the state-of-the-art

  • On TRECVID MED 2013, deep networks outperform prior state-of-the-art methods on both positive-video tasks, and fusion with motion and audio improves further; on MED 2015, fused representations also outperform the state of the art.The 2015 comparison covers pre-specified and ad-hoc tasks; deep representations alone are among the top contenders.Source: Table 3 · p. 7, §5.4 · p. 7

6. CONCLUSIONS

The paper uses the complete ImageNet dataset to pre-train deep networks for video event detection, reorganizing its hierarchy with complementary bottom-up and top-down approaches. On TRECVID MED 2013, the resulting networks outperform standard pre-training, complement one another, retain fusion benefits, and reach a state-of-the-art result.

  • On TRECVID MED 2013, networks trained on the reorganized hierarchies outperform standard pre-trained networks, complement one another, retain benefits from multimodal fusion, and reach a state-of-the-art result.Source: §6 · p. 7
  • The paper uses the complete ImageNet dataset for deep-network pre-training rather than the prescribed 1,000-class subset.Source: §6 · p. 7
  • Bottom-up reorganization merges classes upward from the hierarchy’s deepest parts, while top-down reorganization selects rich generic classes from its top.Source: §6 · p. 7
  • The reorganized hierarchies are used to pre-train deep networks for frame representation in video event detection.Source: §6 · p. 7

Related papers

Loading 1602.07119v1…