PaperlayerView exact revision on arXiv

Source-linked AI summary

What Actions are Needed for Understanding Human Actions in Videos?

Gunnar A. Sigurdsson, Olga Russakovsky, Abhinav Gupta

arXiv:1708.02696v1cs.CV

TL;DR

Video understanding still raises basic questions about activity representation and extent, and activity-detection datasets have historically been limited. The paper analyzes datasets, annotations, categories, algorithms, and potential cues, finding that boundary ambiguity does not block progress and that objects and poses hold promise for future algorithms.

  • Problem

    Video understanding still lacks settled answers about how to represent activities, define their spatial and temporal extent, and account for goals and intentions; good activity detection and recognition datasets have also been lacking.

    Source: §1 · p. 1

  • Method

    The paper analyzes video datasets, human annotators, activity categories, recognition approaches, and potential cues for understanding videos.

    Source: §1 · p. 1

  • Results

    Human disagreement and temporal ambiguity do not present significant roadblocks to activity-understanding progress, while object- and pose-level scene understanding holds promise for future algorithms.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Rich, multi-label, fine-grained activity benchmarks offer opportunities for complex joint high-level reasoning about human activities.

    Source: §6 · p. 8

    The perfect-information oracles are lower bounds for their information types, and better use of perfect information could yield better performance.

    Source: §5 · p. 7

Abstract

from arXiv · show

What is the right way to reason about human activities? What directions forward are most promising? In this work, we analyze the current state of human activity understanding in videos. The goal of this paper is to examine datasets, evaluation metrics, algorithms, and potential future directions. We look at the qualitative attributes that define activities such as pose variability, brevity, and density. The experiments consider multiple state-of-the-art algorithms and multiple datasets. The results demonstrate that while there is inherent ambiguity in the temporal extent of activities, current datasets still permit effective benchmarking. We discover that fine-grained understanding of objects and pose when combined with temporal reasoning is likely to yield substantial improvements in algorithmic accuracy. We present the many kinds of information that will be needed to achieve substantial gains in activity understanding: objects, verbs, intent, and sequential reasoning. The software and additional information will be made available to provide other researchers detailed diagnostics to understand their own algorithms.

1. Introduction

Video understanding still faces basic questions about activity categories, boundaries, goals, and intentions, despite advances in static image understanding. The paper analyzes datasets, annotations, algorithms, and possible cues to identify promising directions.

1. Introduction

  • The paper frames activity understanding as an open problem, asking how to represent activities and determine their spatial and temporal extent.It also asks what role goals and intentions play in understanding activities.Source: §1 · p. 1
  • A lack of good activity detection and recognition datasets has been a significant problem, unlike object recognition's use of generic datasets to define scope and evaluation metrics.The passage names PASCAL, ImageNet, and COCO as examples.Source: §1 · p. 1
  • Historically, datasets such as UCF101 focused on activities hardly seen in daily life, while recent large-scale datasets depict a variety of complex human activities in untrimmed videos.Examples include ActivityNet, Sports1M, Charades, and MultiTHUMOS [40].Source: §1 · p. 1
  • The analysis evaluates whether context, objects, and intentions help approaches handle increasingly complex activity categories, videos, and temporal relationships.It also considers whether verbs, activity sequences, or joint reasoning over objects and activities offer promising directions.Source: §1 · pp. 1–2
  • The paper provides an in-depth analysis of video datasets, human annotators, activity categories, recognition approaches, and potential cues for understanding.It also presents an open-sourced tool that renders algorithm diagnostics and suggests improvements.Source: §1 · pp. 1–2
  • The paper examines how to define activity categories and whether to classify activities or localize them despite uncertain temporal boundaries.It also asks whether goals and intentions help define and understand activities.Source: §1 · p. 1

2. Evolution of Activity Recognition

Activity-recognition datasets have progressed from short clips of simple motions and sports toward sequences of everyday activities. The newer datasets pose different challenges through their category range, label density, and scene diversity.

2. Evolution of Activity Recognition

  • Successive datasets moved from simple motions to more complex sports and cluttered everyday sequences, with each wave bringing new challenges and improved algorithms.Short individual activity clips, often sports-focused, preceded datasets centered on sequences of everyday activities.Source: §2 · p. 2
  • ActivityNet has many categories and a dense hierarchy but one activity label per video, while THUMOS and MultiTHUMOS [40] provide multi-label sports videos.Charades contains diverse videos with dense annotations of common daily activities across varied scenes.Source: §2 · p. 2
  • The analysis uses Charades as its main testbed and brings in MultiTHUMOS [40], THUMOS, and ActivityNet to assess generality.The datasets use normalized mAP; Charades has 9,848 videos, 157 activity categories, and labels that can co-occur. The evaluated baselines are Two-Stream, IDT, LSTM on VGG-16, ActionVLAD, and Asynchronous Temporal Fields.Source: §2 · p. 2

3. What are the right questions to ask?

The section examines activity-category granularity and temporal boundaries, finding that richer categories are more distinctive while boundary ambiguity does not prevent useful evaluation. It also relates classification quality to localization performance.

3.1. What are the right activity categories?

  • Pose similarity is more distinctive across (noun,verb) categories than verb supercategories (p<0.01), supporting the view that verbs alone provide unclear supervision.The paper frames nouns and verbs as atomic action units because some verbs are ambiguous without their objects or prepositions.Source: §3.1 · p. 3

3.2. Do Activities Have Temporal Extents?

  • Human boundary agreement is 72.5% IOU in Charades and 58.7% in MultiTHUMOS; agreement rises with activity duration (ρ=0.50), and category-level ambiguity is widespread.In Charades, starting-point error is lower than ending-point error (0.9±0.8 sec versus 1.4±1.4 sec), and category variation in IOU (13.0% standard deviation) is below average IOU (27.5%). Brief activities may need more careful annotation.Source: §3.2 · p. 3
  • Removing boundary regions raises human consensus IOU from 72.5% to 79.8%, and subsequent annotators cover 82.7% of an activity’s center third.The authors define the omitted boundary regions around each instance’s start and end using α equal to one-third of its duration.Source: §3.2 · p. 3
  • Ignoring ambiguous boundary regions raises Two-Stream accuracy from 9.6% mAP to 10.9% mAP, while baseline gains of 0.96±0.33% mAP remain below the 1.26% mAP average difference between methods.The authors conclude that boundary ambiguity does not change evaluation ordering and current datasets still support evaluating temporal extents.Source: §3.2 · p. 3
  • On Charades, a perfect video classifier achieves 56.9% mAP localization accuracy, five times current state-of-the-art performance; the authors argue classification gains could improve localization.The perfect classifier emits the same binary prediction for every frame, and the authors also report that it improves over their localization baseline on all datasets.Source: §3.2 · pp. 3–4

4. What are existing approaches learning?

This section analyzes how current activity-recognition methods respond to category similarity, training data, temporal structure, and person- and pose-related attributes. Across these analyses, fine-grained category discrimination and stronger temporal and person-based reasoning emerge as important directions.

4.1. Analyzing correct and incorrect detections

  • Accuracy falls as more categories share an object or verb; TFields confuses unrelated categories less (29.1% compared with 41.0−44.0%), but more often confuses categories sharing an object.Source: §4.1 · p. 4, Figure 4 · p. 5

4.2. Training Data

  • Removing half the data hurts small categories more (ρ=0.18, p=0.03), and limiting categories to 100 examples still leaves widely varying accuracy changes.The reported changes range from a 65.8% decrease to a 52.2% increase.Source: §4.2 · p. 5
  • Categories with more examples have slightly more pose diversity (ρ=0.09) and object-sharing categories (ρ=0.13); current methods improve mainly on small categories.The authors suggest models that generalize across categories and more expressive models for large categories, potentially by dividing them into subcategories.Source: §4.2 · p. 5

4.3. Temporal Reasoning

  • Short actions do worse (ρ=0.21−0.42), and motion affects algorithms differently; the authors suggest combining IDT trajectories with Two-Stream longer-term pooling.Source: Figure 6 · p. 6, §4.3 · p. 6
  • Temporal averaging usually helps except for LSTM, but effects vary by class; videos with 1–4 activities score 22.1% mAP versus 16.6% for videos with 14 or more.Smoothing does not help classes with much motion; TFields' high-level temporal modeling helps it use context.Source: §4.3 · p. 6, Figure 7 · p. 6

4.4. Person-based Reasoning

  • People sized 48.7–78.3 pixels yield the best accuracy across multiple datasets; removing the person hurts more than removing other image regions, while person-only retraining raises mAP from 15.6% to 17.9%.Source: Figure 8 · p. 7, §4.4 · p. 7
  • Pose variability correlates with category accuracy on Charades (ρ=0.28), while pose similarity between categories correlates with their test-time confusion (ρ=0.39).The authors conclude that pose plays a significant role in current activity recognition and that harnessing pose is likely to yield substantial advances.Source: §4.4 · p. 7

5. Where should we look next?

Oracle analyses identify promising sources of improvement while showing that their value depends on the dataset and baseline. Intent helps across datasets, temporal information helps datasets with multiple actions per video, and object understanding is more effective than temporal reasoning alone on Charades.

5. Where should we look next?

  • The oracle tests map perfect objects, verbs, per-frame temporal context, intent clusters, or pose clusters to activity predictions.The intent oracle assigns each video an activity distribution based on its cluster; the temporal oracle predicts frame-level action probabilities and max-pools them over time.Source: §5 · p. 7
  • Oracle scores are lower bounds for these information types, since better ways of using perfect information may perform better.The authors caution that the oracle implementations do not exhaust the possible gains from each information source.Source: §5 · p. 7
  • An intent oracle nearly perfects performance on datasets with fewer categories, while a temporal oracle helps Charades and MultiTHUMOS but not ActivityNet or THUMOS.The figure compares oracle combinations across datasets.Source: Figure 9 · p. 8, §5 · p. 8
  • On Charades, object understanding is more effective than temporal reasoning alone, while the oracle results indicate room for improvement across several information types.The pose oracle samples poses in isolation without modeling motion; only 30 types of intent are considered.Source: §5 · p. 8, Figure 9 · p. 8
  • Adding time helps Two-Stream most, while intent helps IDT least, motivating algorithms that combine complementary benefits of IDT, LSTM, and ActionVLAD.Two-Stream models no temporal information beyond optical flow; the authors suggest learning complementary algorithmic strengths.Source: §5 · p. 8
  • Two-Stream predicted the correct Charades intent cluster only 15.1% of the time, suggesting intent clusters should be learned jointly with discriminative features.The test compared predicted clusters against label distributions.Source: §5 · p. 8

6. Discussion and Conclusions

The paper finds that annotation ambiguity does not significantly impede progress and identifies fine-grained scene and person understanding as promising. Its analysis examines activity attributes, algorithms, and datasets in greater depth than prior surveys, and points to rich multi-label benchmarks for joint high-level reasoning.

6. Discussion and Conclusions

  • The authors describe their work as the first in-depth analysis of action recognition in videos, building on prior diagnosis studies and surveys.The paper cites prior diagnosis and analysis work and surveys, while distinguishing its depth of study.Source: §6 · p. 8
  • Human disagreement and ambiguity are inevitable in activity annotation but do not present significant roadblocks to progress in activity understanding.The authors base this conclusion on analyses of multiple activity attributes, algorithms, and datasets.Source: §6 · p. 8
  • Detailed understanding of objects and human poses holds promise for the next generation of activity-recognition algorithms.The authors also identify rich, multi-label, fine-grained benchmarks as opportunities for complex joint reasoning about activities.Source: §6 · p. 8
  • For subset analysis, normalized precision adjusts precision using the average positive and negative counts from the Charades test set.The method follows a normalization approach similar to.Source: §6 · p. 8
Loading 1708.02696v1…