PaperlayerView exact revision on arXiv

Source-linked AI summary

The THUMOS Challenge on Action Recognition for Videos "in the Wild"

Haroon Idrees, Amir R. Zamir, Yu-Gang Jiang, Alex Gorban, Ivan Laptev, Rahul Sukthankar, Mubarak Shah

arXiv:1604.06182v1cs.CV

TL;DR

THUMOS addresses action analysis in Internet videos by extending evaluation beyond restricted, strongly supervised settings and documenting a benchmark and challenge. Its reported findings favor sliding-window descriptions and separate content/context modeling, while highlighting limited action-class coverage and strong supervision requirements in prior evaluations.

  • Problem

    Prior action-recognition methods were typically evaluated on few action classes with strongly supervised training and test sets, limiting large-scale analysis from Internet videos.

  • Method

    The paper documents the THUMOS dataset and challenge, including data design, evaluation protocols, participating approaches, and future dataset improvements.

  • Results

    Sliding window outperforms global description, and separate modeling of content and context improves performance in the challenge results.

  • Takeaways & Limitations

    THUMOS provides a benchmark for large-scale action analysis from Internet videos and supports evaluating action classification and detection approaches.

  • Takeaways & Limitations

    Prior evaluations typically covered only a small number of action classes and required strongly supervised training and test sets.

Abstract

from arXiv · show

Automatically recognizing and localizing wide ranges of human actions has crucial importance for video understanding. Towards this goal, the THUMOS challenge was introduced in 2013 to serve as a benchmark for action recognition. Until then, video action recognition, including THUMOS challenge, had focused primarily on the classification of pre-segmented (i.e., trimmed) videos, which is an artificial task. In THUMOS 2014, we elevated action recognition to a more practical level by introducing temporally untrimmed videos. These also include `background videos' which share similar scenes and backgrounds as action videos, but are devoid of the specific actions. The three editions of the challenge organized in 2013--2015 have made THUMOS a common benchmark for action classification and detection and the annual challenge is widely attended by teams from around the world. In this paper we describe the THUMOS benchmark in detail and give an overview of data collection and annotation procedures. We present the evaluation protocols used to quantify results in the two THUMOS tasks of action classification and temporal detection. We also present results of submissions to the THUMOS 2015 challenge and review the participating approaches. Additionally, we include a comprehensive empirical study evaluating the differences in action recognition between trimmed and untrimmed videos, and how well methods trained on trimmed videos generalize to untrimmed videos. We conclude by proposing several directions and improvements for future THUMOS challenges.

1. Introduction

The introduction motivates THUMOS as a benchmark for realistic action classification and temporal detection in untrimmed videos. It describes the challenge’s dataset innovations and studies how temporal clutter and context affect recognition.

  • Motivation: Trimmed-video testing is impractical because it assumes precise temporal segmentation and creates an unrealistically clean test distribution.It can also let methods exploit artifacts such as test-clip length.
  • Motivation: Real-world actions occur within complex dynamic scenes, making performance on temporally segmented clips difficult to predict for applications.Temporal clutter simplifies recognition but removes relationships among people and objects present in realistic videos.
  • Benchmark design: THUMOS introduced untrimmed positive and background videos, enabling realistic classification and temporal detection over 101 action classes.The challenge uses publicly available YouTube videos manually annotated with action labels and temporal spans.
  • Benchmark design: Background videos share scenes and objects with positive videos but omit target actions, reducing the usefulness of appearance and static information.This supports a multi-label binary classification task in which a video may contain multiple actions.
  • Paper contributions: The paper evaluates video-level classification and temporal detection, quantifies temporal-clutter effects, and examines context from temporal neighborhoods.It evaluates an IDTF + Fisher Vectors + SVM pipeline and reviews THUMOS’15 submissions.
  • Findings: Temporal clutter reduces recognition performance, while untrimmed videos provide contextual information that can improve recognition.The conclusion also reports that sliding-window descriptions and separate content/context modeling improve performance.

2. Related Datasets

The related-dataset discussion situates THUMOS among action and event benchmarks by comparing realism, scale, annotation, and task scope. THUMOS combines carefully annotated actions with negative background videos and temporal spans.

  • THUMOS: THUMOS uses trimmed UCF101 clips for training but adds untrimmed positive and background videos for validation and testing.The added videos support more realistic action recognition and temporal detection.
  • Dataset comparisons: Unlike Sports-1M, which is large but sports-focused and weakly annotated, THUMOS provides carefully annotated videos and negative backgrounds for each action class.Background videos make recognition more difficult because scene similarity reduces the utility of appearance features.
  • TRECVID: TRECVID evaluates broader video concepts and events, whereas THUMOS focuses on atomic actions primarily affected by actors’ motion.TRECVID includes shot-level or clip-level annotations and event-oriented evaluation tasks.
  • Dataset comparisons: THUMOS background videos support explicit study of motion and stationary or non-action context in action recognition.Because action and background scenes are similar, classifiers and detectors must model motion accurately.

3. The THUMOS Challenge Tasks

THUMOS evolved from trimmed-video classification toward classification and temporal detection in untrimmed videos. The tasks require recognizing action presence and, for detection, localizing instances in time.

  • Classification: Classification predicts the presence or absence of each of 101 action classes for every video using real-valued confidences.
  • Classification: Actions are not mutually exclusive: each may occur once, multiple times, or never in a test video.
  • Classification: Since 2014, participants train using UCF101, validation, and background sets, then report results on a withheld test set without manual annotation.
  • Temporal Detection: Temporal detection outputs intervals and confidence values for all detected instances of 20 pre-selected actions in untrimmed videos.
  • Temporal Detection: Detection is comparatively harder because correctly localized instances with incorrect class labels are treated as incorrect detections.
  • Task Evolution: THUMOS replaced spatio-temporal localization with temporal detection because temporal detection is more tractable in long videos and often more useful than spatial localization.

4. The THUMOS Dataset

THUMOS’15 extends the benchmark with realistic untrimmed action and background videos, semantic attributes, and temporal annotations. Its collection pipeline combines automated search with manual filtering, annotation, downloading, and verification.

  • Video Collection: Positive and background candidates were collected from YouTube using Freebase topics supplemented with manually chosen search keywords.
  • Background Videos: Background videos share scenes, actors, and objects with an action while omitting that action, and must avoid positive instances of other action classes.
  • Video Collection: The collection pipeline saves video and channel metadata, then applies annotation, downloading, and final verification stages.
  • Annotation: Annotators mark valid action instances as positive and label videos irrelevant when they meet criteria such as slow motion, blur, animation, compilation, or excessive length.
  • Annotation: Each positive video can also receive secondary-action labels, including hierarchical relationships such as BasketballDunk implying Basketball.
  • Attributes: The benchmark provides semantic relationships among the 101 action classes, although video-level annotations for the attributes are not provided.
  • Dataset Design: THUMOS’15 was designed to provide a realistic action-recognition scenario using untrimmed videos rather than the earlier trimmed-video setting.

5. Submission and Evaluation

THUMOS evaluates video-level action recognition with confidence scores and temporal detection with scored intervals. Both tasks use average precision, while detection additionally evaluates localization across multiple IoU thresholds.

  • Action Recognition: Classification systems output real-valued confidence scores for each of 101 actions in every test video.
  • Action Recognition: Because untrimmed videos may contain long background portions, multiple action instances, or no target action, confidence scores must reflect video-level presence.
  • Submission Rules: At most five runs are allowed per team, and the best-performing run is selected as the primary submission for ranking.
  • Action Recognition: Classification uses interpolated Average Precision per class and mean Average Precision across all action classes.
  • Temporal Detection: Temporal detection submissions specify video name, start time, end time, class label, and confidence score for each detection.
  • Temporal Detection: Detections are confidence-ranked and matched to ground truth using Intersection over Union, with matches above a threshold counted as true positives.
  • Temporal Detection: Multiple detections for one annotated action are penalized, unmatched annotations become false negatives, and ambiguous intervals are excluded from evaluation.

6. Methods

THUMOS’15 participants used deep-learning and hand-crafted video features, with varied pooling, fusion, and classification strategies for classification and temporal detection.

  • Classification: Classification methods combined deep CNN features with Improved Dense Trajectory features, while all teams primarily used SVM classifiers.Networks included VGGNet, GoogleNet, ClarifaiNet, and C3D; KRR was an alternative classifier for one team.
  • Classification: Two-stream CNNs modeled both spatial frames and temporal optical flow for video analysis.The spatial stream captured static frames, while the temporal stream used optical flows.
  • Classification: Frame-level CNN features were converted to video representations using VLAD or mean/max pooling.Most teams adopted VLAD and conventional pooling schemes to aggregate frame-level features.
  • Classification: Fusion strategies varied, with average fusion most popular because of its simplicity and good generalizability.Other strategies included weighted fusion, logistic regression fusion, and geometric mean fusion.
  • Temporal Detection: Temporal detection attracted fewer participants than classification because of higher computational requirements and its recent introduction.The challenge received five runs from one team for THUMOS’15.
  • Temporal Detection: Temporal detection pipelines extracted iDT features, encoded segments with improved Fisher Vectors, and classified 20 actions plus background.The pipeline used nine temporal windows, concatenated 9×101 scores, and applied median filtering for smoothness.

7. Results

THUMOS’15 classification achieved 61.3% mAP overall, but performance varied substantially across actions and was strongly affected by visual similarity and background context. Temporal detection benefited from motion features and multiple temporal scales, while appearance and scene context could reduce detection performance.

  • Classification Results: All top-10 classification teams used iDT features, and all teams used deep learning features.The analysis also identified LCD coding with VLAD, fine-tuned CNNs, two-stream CNNs, and audio fusion as effective approaches.
  • Classification Results: 61.3% mAP was achieved across action classes, with AP ranging from 19.8% to 96.4%.The sorted AP distribution indicated an approximately even spread of easy and difficult actions.
  • Per-action Results: Bowling and Surfing were easy classes, whereas BlowDryHair and Haircut were difficult because of visual similarity.The authors state that more advanced techniques are needed to distinguish visually similar classes.
  • Per-action Results: Top-ranked negative samples were visually similar to positive samples, while difficult actions occurred across varied scene backgrounds.These observations indicate that current algorithms may rely substantially on background scenes rather than action content alone.
  • Background Videos: 76.3% mAP without background videos was 15% higher than the official result with background videos.Background videos contained samples that were visually but not semantically similar to target classes.
  • Temporal Detection: Run 1 achieved approximately 41% mAP at 10% overlap using iDT features, outperforming runs that fused appearance and scene features.This contrasted with classification, where deep-network appearance features generally improved performance.
  • Temporal Detection: A single temporal scale caused an approximately 30% performance drop compared with using nine scales.Detection required distinguishing action segments from the rest of positive videos, where appearance features could reduce discrimination.

8. Action Recognition in Untrimmed Videos

The paper studies how action recognition representations trained and evaluated under trimmed-video assumptions behave in untrimmed videos, where actions occupy only part of longer sequences. It compares global, content, context, sliding-window, loose-crop, and separate content-context representations for classification and temporal detection.

  • Study Design: The study addresses differences between trimmed and untrimmed videos, generalization from trimmed training, and content-versus-context representation choices.Experiments used 20 action classes with manually annotated action intervals.
  • Representations: Global representation aggregates descriptors over the full video without using ground-truth action intervals.It represents direct deployment of a traditional recognizer trained on trimmed data.
  • Representations: Content Only assumes known action boundaries and extracts one descriptor per action interval, matching trimmed-video setups.Context Only instead extracts descriptors from the entire video outside annotated action intervals.
  • Representations: Content and Context Modeling concatenates representations computed separately from action intervals and surrounding temporal context.The representation is designed to test whether separately modeling content and context is beneficial.
  • Feature Extraction: The feature pipeline uses Improved Dense Trajectory descriptors, Fisher Vector encoding, and additive aggregation over 10-frame chunks.HOF and MBH capture motion, HOG captures static scene information, and L2 normalization produces interval descriptors.
  • Results: Sliding-window representations improved results because actions are usually shorter than untrimmed videos and can be captured in smaller windows.For temporal detection, IDTF achieved 0.67 mAP, and a 4-second window exceeded a 2-second window by 0.03.

9. Future Directions

Future directions expand THUMOS toward larger, richer, and more semantically structured video understanding benchmarks. Proposed extensions add classes, dense annotations, weakly supervised spatio-temporal localization, and question answering.

  • Larger and richer benchmarks: Weakly supervised spatio-temporal localization would train on untrimmed videos without frame-level annotations or bounding boxes.The test set would nevertheless contain such annotations for evaluation.
  • Larger and richer benchmarks: The proposed THUMOS extension would substantially increase action diversity and scale, targeting at least 200 instances per action.It addresses deficiencies in the number of classes and instances per class while increasing storage requirements.
  • Semantic and structured understanding: The benchmark aims to move beyond visual perception toward holistic understanding of objects, actions, attributes, interactions, and textual descriptions.The authors explicitly seek explanations of complex spatial and temporal interactions, not only detection.
  • Semantic and structured understanding: The planned dataset would combine dense annotations for objects, actions, scenes, attributes, relationships, and textual descriptions.WordNet is proposed to organize labels and transfer related attributes, while annotators verify the resulting structures.
  • Annotation and labeling: Future label design must balance consistency with diversity because reusing labels can limit the introduction of new object and action categories.WordNet relations such as hyperonymy and synonymy are proposed to help manage this trade-off.
  • Semantic and structured understanding: Region-level captions, graphical representations, and question-answer pairs would support reasoning about object interactions and complex temporal behavior.The proposed questions target details such as body-part use, action duration, instruments, and participants.

Conclusion

The paper documents the THUMOS benchmark, its collection and evaluation procedures, and analyses action recognition in untrimmed videos. The study finds that sliding-window representations and separate content-context modeling improve performance, while future challenges should add richer tasks and annotations.

  • Benchmark and challenge: The paper describes THUMOS data collection, annotation procedures, evaluation metrics, and methods and results from the THUMOS’15 competition.Its core tasks include action classification and temporal detection.
  • Empirical findings: Sliding window outperforms global description for action recognition in untrimmed videos.This conclusion comes from the study of untrimmed videos introduced in the 2014 challenge.
  • Empirical findings: Separate modeling of content and context is helpful for improving performance.The analysis explicitly examines the roles of actions as content and backgrounds as context.
  • Future directions: Future challenges are proposed to include spatio-temporal localization and weakly supervised action recognition tasks.The paper also proposes large-scale untrimmed data with dense object, action, and textual annotations.

Appendix A. List of 101 actions

The appendix provides the complete list of 101 actions, identifying which actions are used to evaluate temporal detection.

  • Action list: The appendix lists all 101 UCF actions and THUMOS actions, with boldface marking those used for temporal detection evaluation.The list defines the action categories associated with the evaluation task.
Loading 1604.06182v1…