PaperlayerView exact revision on arXiv

Source-linked AI summary

EventNet: A Large Scale Structured Concept Library for Complex Event Detection in Video

Guangnan Ye, Yitong Li, Hongliang Xu, Dong Liu, Shih-Fu Chang

arXiv:1506.02328v2cs.CV

TL;DR

Complex event detection has limited access to rich semantic representations, and existing event-specific concept approaches cover only a small set of predefined events. EventNet builds a large, WikiHow-structured library of events and YouTube-derived concepts; it reports a 207% improvement over 20K ImageNet concepts in zero-shot event retrieval.

  • Problem

    Existing event-detection methods often use low-level features, and event-specific concept approaches are limited to known events, leaving novel unseen events without relevant concepts.

    Source: §1 · p. 1

  • Method

    EventNet discovers 500 events from WikiHow, mines event-specific concepts from YouTube video tags, trains concept models, and organizes events and concepts in a WikiHow hierarchy.

    Source: §1 · p. 2

  • Results

    207% improvement over 20K ImageNet concepts is reported for zero-shot event retrieval.

    Source: §1 · p. 2

  • Takeaways & Limitations

    EventNet's tree structure helps match event queries to semantically relevant concepts.

    Source: §9 · p. 10

    The ontology construction relies on subjective evaluation; the authors defer automatic discovery methods until their accuracy improves.

    Source: §4.2 · p. 4

Abstract

from arXiv · show

Event-specific concepts are the semantic concepts designed for the events of interest, which can be used as a mid-level representation of complex events in videos. Existing methods only focus on defining event-specific concepts for a small number of predefined events, but cannot handle novel unseen events. This motivates us to build a large scale event-specific concept library that covers as many real-world events and their concepts as possible. Specifically, we choose WikiHow, an online forum containing a large number of how-to articles on human daily life events. We perform a coarse-to-fine event discovery process and discover 500 events from WikiHow articles. Then we use each event name as query to search YouTube and discover event-specific concepts from the tags of returned videos. After an automatic filter process, we end up with 95,321 videos and 4,490 concepts. We train a Convolutional Neural Network (CNN) model on the 95,321 videos over the 500 events, and use the model to extract deep learning feature from video content. With the learned deep learning feature, we train 4,490 binary SVM classifiers as the event-specific concept library. The concepts and events are further organized in a hierarchical structure defined by WikiHow, and the resultant concept library is called EventNet. Finally, the EventNet concept library is used to generate concept based representation of event videos. To the best of our knowledge, EventNet represents the first video event ontology that organizes events and their concepts into a semantic structure. It offers great potential for event retrieval and browsing. Extensive experiments over the zero-shot event retrieval task when no training samples are available show that the EventNet concept library consistently and significantly outperforms the state-of-the-art (such as the 20K ImageNet concepts trained with CNN) by a large margin up to 207%.

1. INTRODUCTION

The paper addresses the difficulty of representing complex events with rich semantics, especially when few or no training videos are available, and the limited coverage of existing event-specific concepts. It introduces EventNet, a large-scale structured concept library, and reports a 207% improvement over 20K ImageNet concepts in zero-shot event retrieval.

1. INTRODUCTION

  • Existing event detection often relies on low-level features, while event-specific concept methods cover few predefined events and cannot handle novel unseen events.Complex events combine objects, actions, and scenes, and can be difficult to analyze when training videos are scarce or absent.Source: §1 · pp. 1–2
  • EventNet organizes 500 WikiHow-derived events and 4,490 event-specific concept models trained on 95,321 YouTube videos in a WikiHow-defined hierarchy.Event names query YouTube; concepts are discovered from returned-video tags, and CNN features support the concept models.Source: §1 · p. 2
  • Cascaded concept matching first finds relevant events, then matches concepts associated with those events; users can restrict matching to a high-level category.The authors note that short event names can make direct semantic-similarity matching unreliable.Source: §1 · p. 2
  • 207% improvement over 20K ImageNet concepts in zero-shot event retrieval is reported for EventNet.The paper also reports performance gains across benchmark video event datasets.Source: Figure 1 · p. 1, §1 · p. 2
  • The paper presents EventNet as an event ontology, a basis for event-specific concept mapping, and a benchmark video dataset.It also identifies semantic recounting and concept matching as uses of the ontology structure.Source: §1 · p. 2

2. RELATED WORK

Prior concept-based event detection either borrows concepts not tailored to target events or discovers event-driven concepts for a limited set of known events. EventNet seeks broader event coverage and associates concepts with events in a structured ontology.

2. RELATED WORK

  • Event-independent concepts may miss event-specific semantics, while event-driven concepts depend on prior knowledge of target events and do not cover novel unknown events.Prior approaches include concepts mined from TRECVID descriptions and concepts discovered from Flickr tags for known events.Source: §2 · p. 2
  • EventNet explores events and concepts using WikiHow and YouTube, with YouTube-based training intended to better capture concepts in user-generated event videos.The authors contrast this with Flickr-trained concept models, which they say do not generalize well to event videos because of cross-domain data variation.Source: §2 · p. 2
  • EventNet contains 500 event categories and 95K videos, alongside event-specific concepts, while the cited comparison datasets contain fewer event categories.The passage contrasts it with TRECVID MED's 48 events, CCV's 20, and another structured dataset's 239 events.Source: §2 · pp. 2–3

3. CHOOSING WIKIHOW AS EVENTNET ONTOLOGY

The paper chooses WikiHow as a source for a broad event ontology because it contains how-to articles about daily-life tasks and organizes them hierarchically. A coverage study finds relevant WikiHow matches for 93% of the tested event classes.

3. CHOOSING WIKIHOW AS EVENTNET ONTOLOGY

  • WikiHow is selected as an ontology source because its how-to articles cover many aspects of daily-life events.The source describes WikiHow as a forum for manuals on human daily-life tasks; it reports over 300K articles as of February 2015.Source: §3 · p. 3
  • 169/182 = 93% coverage is reported when exact, partial, and relevant WikiHow article matches count as successful.The study queries WikiHow with event classes from five existing video datasets and examines the top 10 returned articles for each class.Source: §3 · p. 3
  • The WikiHow hierarchy contains 2,803 categories, 19 top-level nodes, and paths up to seven levels deep.Parent categories are typically more generic than their child categories.Source: §3 · p. 3
  • The hierarchy can help users localize a search area and improve concept matching, while also supporting browsing and visualization.The authors also describe using detections at parent and child nodes to enhance event detection performance.Source: §3 · pp. 3–4

4. CONSTRUCTING EVENTNET

EventNet is constructed by selecting representative video events from WikiHow and mining event-specific concepts from YouTube video tags. Its ontology construction combines manual category pruning and event selection with tag filtering and WikiHow's hierarchy.

4.1 Discovering Events

  • Coarse-to-fine selection first filters irrelevant categories, avoiding video crawling and processing for all 300,000 WikiHow articles.The approach then selects representative events from articles in the remaining categories.Source: §4.1 · p. 4
  • WikiHow categories are manually pruned when they are not visually describable and none of their articles contain video events, with article review before removal.An abstract category such as Science can remain if it contains instructional articles corresponding to video events.Source: §4.1 · p. 4
  • After pruning, the current EventNet version selects one representative video event per category using criteria including human activity, concrete meaning, visual consistency, and genericity.The passage describes the first and fourth criteria; the criteria about concrete meaning and visual consistency are specified in p30.Source: §4.1 · p. 4

4.2 Mining Event Specific Concepts

  • EventNet mines concepts from YouTube tags for 500 events, retaining 4,490 concepts across 95,321 videos and organizing them under WikiHow categories.For each event, the authors query YouTube, filter returned videos using frequent title or tag words, and keep tags matched to existing object, scene, or action libraries.Source: §4.2 · p. 4
  • The authors note that EventNet construction depends heavily on subjective evaluation and defer automatic discovery methods while their accuracy improves.They identify computer-vision concept verification and text-based event extraction as possible alternatives.Source: §4.2 · p. 4

5. PROPERTIES OF EVENTNET

EventNet contains 500 event nodes and 4,490 concept nodes in a WikiHow-based tree, alongside 95,321 videos. Its events show category-level variation and no detected redundancy across different branches under the authors' evaluation.

EventNet Statistics

  • EventNet contains 682 WikiHow category nodes, 500 event nodes, and 4,490 concept nodes, with 95,321 videos averaging about 277 seconds each.The tree's deepest event-node depth is eight; the corpus totals 7,334 hours.Source: §5 · p. 5

Event Distribution

  • Sports and fitness, hobbies and craft, food and entertainment, and home and garden contain the most events, while work world, relationships, philosophy and religion, and youth contain the fewest.The authors describe the least-populated categories as abstract and not describable in videos.Source: §5 · p. 5

Event Redundancy

  • The evaluation found zero redundant events across different EventNet branches among the 500 query events.The authors exclude same-branch pairs because parent-child event pairs, such as “cook meat” and “cook poultry,” are treated as desired hierarchy relations.Source: §5 · p. 5

6. LEARNING CONCEPT MODELS FROM DEEP LEARNING VIDEO FEATURES

EventNet concept models use CNN-derived video features and one-vs-all linear SVM classifiers. The CNN is trained to classify 500 events, while concept classifiers use frames from concept-associated videos as positive training data and sampled videos from other events as negatives.

6. LEARNING CONCEPT MODELS FROM DEEP LEARNING VIDEO FEATURES

  • One-vs-all linear SVMs trained on CNN video features serve as the EventNet concept models.The CNN extracts features from video content, and the SVMs are trained on top of those features.Source: §6 · p. 5

6.1 Deep Feature Learning with CNN

  • The CNN takes RGB frames as input and predicts a score distribution over 500 EventNet events; its feature descriptor is a 4,096-dimensional, L2-normalized vector.Training uses 40 sampled frames per video, and the descriptor is extracted from the second-to-last CNN layer.Source: §6.1 · p. 5

6.2 Concept Model Training

  • For each concept, associated videos provide positive training data and an equal number of randomly sampled videos from other events provide negatives for a linear SVM.The authors acknowledge that this practice risks false negatives because videos without a concept label are not necessarily negative examples.Source: §6.2 · p. 5
  • Table 2 compares the top two matched events for queries without and with EventNet structure.The supplied caption identifies the comparison but does not state the outcomes.Source: Table 2 · p. 6
  • Video-level concept representations average the concept-score vectors across uniformly sampled frames.Each frame's vector contains scores from all 4,490 concept models.Source: §6.2 · p. 6

7. LEVERAGING EVENTNET STRUCTURE FOR CONCEPT MATCHING

EventNet concept matching uses event context because identical words can denote different concepts under different events. Phrase-level matching can still fail on ambiguous or short event names, so the ontology hierarchy can constrain the search.

7. LEVERAGING EVENTNET STRUCTURE FOR CONCEPT MATCHING

  • Because concepts such as “dog” are distinct under “feed a dog” and “groom a dog,” matching must account for their associated events.The paper contrasts this with approaches that rank concepts by semantic similarity alone.Source: §7 · p. 6
  • EventNet matching first finds relevant events, then selects concepts associated with those events.Event names are matched using phrase-level similarity; the resulting event context guides concept selection.Source: §7 · p. 6
  • Ambiguous word senses and limited information in event names can cause phrase-similarity matching to select irrelevant events.The paper’s examples include “wedding shower” matching “take a shower” and “landing a fish” matching “landing an airplane” or “cook fish.”Source: §7 · p. 6
  • Users can restrict event matching to chosen top-level categories, narrowing the search when phrase similarity is unreliable.The paper reports that this use of EventNet’s hierarchy produces more relevant matches than phrase similarity alone.Source: §7 · p. 6

8. EXPERIMENTS

The experiments assess EventNet representations on two benchmark datasets and across retrieval, recounting, structure-assisted matching, and event classification. EventNet improves zero-shot retrieval over the strongest reported baseline, while the experiments also examine the ontology as a recounting and benchmark resource.

8.1 Dataset and Experiment Setup

  • The evaluation uses MED and CCV test sets without their training videos and compares concept representations using mean Average Precision.The baselines include Classemes, Flickr Concept Representation, ImageNet-1K and ImageNet-20K, alongside EventNet ECR.Source: §8.1 · pp. 6, 8

8.2 Task I: Zero-Shot Event Retrieval

  • 207% relative improvement raises TRECVID MED mAP from 2.89% to 8.86% over ICR-20K; CCV mAP rises from 30.82% to 35.58% (15.4% relative improvement).EventNet ECR also performs best on most event categories in both datasets.Source: Figure 6 · p. 7, §8.2 · p. 8
  • With more concepts, ECR gains consistently on TRECVID MED, but ICR-20K can match or exceed it on CCV; ECR mAP can fall when lower-ranked concepts are added.The authors suggest CCV’s object categories may suit ImageNet concepts better, whereas MED’s complex events may suit EventNet; Table 3 compares ECR with reported state-of-the-art results on MED.Source: Figure 7 · p. 7, §8.2 · p. 9, Table 3 · p. 9

8.3 Task II: Semantic Recounting in Videos

  • Semantic recounting ranks a video’s 4,490 event-specific concept scores and selects the highest-scoring concepts as its salient cues.The paper presents example recounts from MED and CCV and describes a two-step process that predicts relevant events before applying their concepts.Source: §8.3 · p. 9, Figure 8 · p. 9

8.4 Task III: Effects of EventNet Structure for Concept Matching

  • Restricting event matching to manually chosen EventNet categories improves zero-shot retrieval compared with matching across all events.Both methods select 15 concepts per query; Table 4 compares matching without and with EventNet structure.Source: §8.4 · p. 9, Table 4 · p. 9

8.5 Task IV: Multi-Class Event Classification

  • On EventNet’s 19 high-level categories, Figure 9 reports mean top-1 accuracy of 38.91% and mean top-5 accuracy of 57.67%.The experiment treats the 95,321 videos across 500 event categories as a benchmark and provides standard partitions; Figure 10 shows top predictions for sample videos.Source: §8.5 · pp. 9–10, Figure 9 · p. 10, Figure 10 · p. 10

9. CONCLUSION

EventNet organizes 500 WikiHow-derived events and 4,490 YouTube-tag-derived concepts into a hierarchy, with classifiers trained on features from 95,321 videos. Experiments show strong zero-shot retrieval results and report that the tree structure helps match queries to relevant concepts.

9. CONCLUSION

  • EventNet contains 500 events mined from WikiHow and 4,490 event-specific concepts discovered from YouTube video tags.Large-margin classifiers are trained using deep-learning features from 95,321 YouTube videos, and the events and concepts are organized in a WikiHow-based tree.Source: §9 · p. 10
  • Experiments on two benchmark event datasets show that EventNet outperforms prior concept libraries in zero-shot event retrieval.The conclusion also reports that EventNet’s tree structure helps match event queries to semantically relevant concepts.Source: §9 · p. 10
  • The authors plan to expand EventNet using additional events and knowledge resources and to pursue tree-structured event modeling.Source: §9 · p. 10
Loading 1506.02328v2…