PaperlayerView exact revision on arXiv

Source-linked AI summary

A Pursuit of Temporal Accuracy in General Activity Detection

Yuanjun Xiong, Yue Zhao, Limin Wang, Dahua Lin, Xiaoou Tang

arXiv:1703.02716v1cs.CV

TL;DR

Detecting activities in untrimmed video requires identifying both what an activity is and when it starts and ends, yet action segments can be difficult to delimit and vary greatly in duration. The paper combines bottom-up temporal proposals with cascaded relevance and completeness classification, and reports better state-of-the-art performance across THUMOS14 and ActivityNet, including across different activity temporal structures.

  • Problem

    Untrimmed-video detection must locate action boundaries, but action parts can reveal the class and action durations vary widely.

    Source: §1 · p. 2

  • Method

    The framework uses bottom-up temporal actionness grouping to propose regions and cascaded classifiers to filter background and incomplete candidates.

    Source: §1 · p. 2

  • Results

    The method outperforms the state of the art on THUMOS14 and ActivityNet across IOU levels and adapts to their different temporal structures.

    Source: §1 · p. 2, §6.6 · p. 8

  • Takeaways & Limitations

    The framework is reported to localize temporal boundaries precisely across datasets with different activity temporal structures.

    Source: §7 · p. 9

Abstract

from arXiv · show

Detecting activities in untrimmed videos is an important but challenging task. The performance of existing methods remains unsatisfactory, e.g., they often meet difficulties in locating the beginning and end of a long complex action. In this paper, we propose a generic framework that can accurately detect a wide variety of activities from untrimmed videos. Our first contribution is a novel proposal scheme that can efficiently generate candidates with accurate temporal boundaries. The other contribution is a cascaded classification pipeline that explicitly distinguishes between relevance and completeness of a candidate instance. On two challenging temporal activity detection datasets, THUMOS14 and ActivityNet, the proposed framework significantly outperforms the existing state-of-the-art methods, demonstrating superior accuracy and strong adaptivity in handling activities with various temporal structures.

1. Introduction

Temporal activity detection must identify both an action and its start and end in untrimmed video. The paper addresses inaccurate boundaries and variable action durations with a proposal-and-classification framework, reporting gains on THUMOS14 and ActivityNet.

1. Introduction

  • On ActivityNet 2016, the top detection method's mAP falls from 42.5% at 0.5 IOU to 0.06% at 0.95 IOU.The introduction uses this degradation to illustrate limitations in accurate localization.Source: §1 · p. 1
  • A short segment can identify an action while omitting much of it, so recognizing its class does not ensure accurate start and end localization.Figure 1 illustrates this distinction with a red detection that has less than 0.4 IOU yet includes a representative part of the action.Source: Figure 1 · p. 1, §1 · p. 2
  • Action durations range from seconds to minutes, making sliding-window coverage costly and dense-frame processing expensive for long actions.High localization accuracy with sliding windows would require many scales and small steps.Source: §1 · p. 2
  • The framework combines bottom-up temporal actionness grouping with cascaded filters that distinguish background from incomplete proposals, and uses sparse snippet sampling to reduce long-proposal cost.The proposal scheme avoids assumptions about activity duration and is designed to improve temporal boundaries.Source: §1 · p. 2
  • On THUMOS14 and ActivityNet, the method outperforms the state of the art across IOU levels, with especially notable gains at 0.7 and 0.9.The paper also presents its performance across the two datasets as evidence of adaptability to different temporal structures.Source: §1 · p. 2

2. Related Work

Prior action-recognition methods generally classify trimmed videos rather than locate actions in untrimmed ones. Earlier temporal detectors used sliding windows, while the paper distinguishes its approach through bottom-up proposals and separate relevance and completeness checks.

2. Related Work

  • Action-recognition methods typically assume trimmed videos and focus on classification, leaving temporal localization aside.The related-work section notes that these methods usually ignore where an action begins and ends.Source: §2 · p. 2
  • Object detection commonly uses proposal-plus-classification pipelines, with bottom-up proposals often requiring fewer candidates than sliding windows.The paper presents this paradigm as a precedent, while noting temporal detection poses additional challenges.Source: §2 · p. 2
  • Figure 2 shows the proposed sequence: TAG generates candidates, then cascaded classifiers assess relevance and completeness and retain complete Triple Jump instances.The diagram depicts background and non-complete proposals being rejected.Source: Figure 2 · p. 3
  • Earlier temporal detectors relied mainly on sliding-window candidates; later approaches added deep networks but retained limits in localization or supported duration.The cited C3D-based method handles windows spanning at most 512 frames, about 17 seconds.Source: §2 · pp. 2–3

3. Framework Overview

The framework separates temporal proposal generation from candidate classification to address the distinction between action wholes and parts and the wide variation in action duration. It uses TAG for proposals and a cascade to filter background and incomplete candidates.

3. Framework Overview

  • The framework first generates class-agnostic temporal regions that may contain actions, then classifies candidates by action presence and class.Its designs target the difficulty of distinguishing whole actions from parts and of handling widely varying durations.Source: §3 · p. 3
  • Temporal Actionness Grouping (TAG) scores snippets as action or background and groups them bottom-up into proposals of varied lengths with boundary-sensitive transitions.Unlike sliding windows, TAG makes no assumption about activity duration.Source: §3 · p. 3
  • The classifier cascade first removes background proposals, then rejects candidates that capture only an incomplete part of an action.The paper treats actionness and completeness as distinct characteristics.Source: §3 · pp. 3–4

4. Temporal Region Proposals

Temporal proposals are built by scoring video snippets for actionness and grouping high-scoring fragments. TAG tolerates some low-actionness snippets, uses multiple thresholds to produce varied proposal granularities, and aims for broad coverage with a moderate proposal set.

4. Temporal Region Proposals

  • Each snippet combines a frame with its optical flow, and a learned binary classifier estimates whether it lies within an activity instance.The classifier is trained with annotated action instances as positives and randomly sampled non-action regions as negatives.Source: §4 · p. 4
  • TAG thresholds actionness into consecutive fragments, then expands proposals by merging fragments until low-actionness snippets exceed the tolerance threshold γ.The illustrated process can absorb some background-labeled snippets before reaching γ.Source: §4 · p. 4, Figure 3 · p. 4
  • Using multiple actionness and tolerance thresholds yields diverse proposal granularities; a typical video has about 30 proposals per minute, spanning less than 10 to more than 5000 frames.The design avoids dataset-specific manual parameter tuning and prunes near-duplicates with nonmaximal suppression.Source: §4 · p. 4
  • TAG focuses proposals on action-related content and often yields more accurate boundaries, achieving high recall with a moderate proposal count that benefits classifier training.The proposal method uses action fragments that are sensitive to temporal transitions.Source: §4 · pp. 4–5

5. Detecting Action Instances

The detector first classifies proposals as activity or background, then uses class-specific completeness filters to identify incorrectly localized candidates from within-segment structure and surrounding context.

5. Detecting Action Instances

  • A two-step pipeline first rejects background proposals and classifies the rest, then filters incomplete or over-complete instances with class-specific completeness filters.Separating completeness testing from activity classification avoids entangling two goals that could severely confuse classifiers.Source: §5 · p. 5

5.1. Activity Classification

  • At test time, snippet-wise scores are aggregated into proposal-level scores; training excludes negatives with at least 5% overlap to avoid confusing classifiers with action parts.Positive training proposals must overlap a ground-truth instance with IOU above 0.7; only non-background proposals proceed to completeness filtering.Source: §5.1 · p. 5

5.2. Completeness Filtering

  • Completeness filters target both incomplete and over-complete proposals using a two-level temporal pyramid, before-and-after scores, and class-specific SVMs trained with hard negative mining.The representation captures temporal structure within a proposal and context around it.Source: §5.2 · p. 5
  • Completeness filters effectively remove incorrectly localized proposals at insignificant computational cost.Nonmaximal suppression uses IOU thresholds of 0.6 on ActivityNet and 0.2 on THUMOS14.Source: §5.2 · p. 5

6. Experimental Results

The experiments evaluate the framework on ActivityNet and THUMOS14, examine its components, and compare it with prior methods. Results cover proposal quality, detection performance, and adaptation across datasets.

6.1. Datasets

  • The framework uses ActivityNet v1.2 (9,682 videos, 100 classes), v1.3 (19,994 videos, 200 classes), and THUMOS14, training THUMOS14 detectors on validation data and evaluating on test.ActivityNet versions use training, validation, and testing splits; THUMOS14 has no separate training set with temporal annotations.Source: §6.1 · p. 6

6.4. Exploration Study

  • TAG achieves higher average recall with considerably fewer proposals than compared methods and shows no severe performance drop on unseen activity classes.The unseen-class test uses an actionness classifier trained on ActivityNet v1.2.Source: Table 1 · p. 6, Table 2 · p. 6, §6.4 · p. 7
  • With completeness filtering included, TAG proposals yield comparable mAP on THUMOS14 and much better mAP on ActivityNet v1.2 than sliding windows.The sliding-window configurations were adjusted to achieve higher average recall using many more proposals.Source: Table 3 · p. 6, §6.4 · p. 7
  • Deeper CNN architectures benefit temporal detection, while learned completeness filters perform consistently across datasets where tuned duration heuristics do not.The architecture comparison is on ActivityNet v1.2 validation data; the heuristic comparison covers two representative duration-based forms.Source: Table 4 · p. 7, Table 5 · p. 7, §6.4 · p. 7

6.5. Comparison with State of the Arts

  • On THUMOS14, the framework beats prior state-of-the-art methods by over 10% in most cases; ActivityNet mAP is 26% vs 17.8% at 0.75 and 6.66% vs 2.88% at 0.95.The authors also report outperforming other state-of-the-arts by around 10% in average mAP on ActivityNet.Source: Table 6 · p. 8, Table 7 · p. 8, Table 8 · p. 8

6.6. Discussion

  • The discussion attributes high temporal accuracy and adaptation across varying temporal structures to the two-stage cascade, and notes TAG's generalization to unseen activities.The authors characterize sparse TAG proposals as conducive to detection performance.Source: §6.6 · p. 8

7. Conclusion

The paper presents a generic temporal action detection framework combining temporal actionness grouping with cascaded proposal classifiers. It reports improved state-of-the-art performance, precise boundary localization, and effectiveness across datasets with different activity structures.

7. Conclusion

  • The framework combines temporal actionness grouping for proposals with cascaded proposal classifiers.Together, these components form the paper’s generic framework for temporal action detection.Source: §7 · pp. 8–9
  • The framework improves performance over previous state-of-the-art methods.Source: §7 · pp. 8–9
  • The method localizes temporal boundaries precisely and works on datasets with different activity structures.Source: §7 · p. 9

Related papers

Loading 1703.02716v1…