PaperlayerView exact revision on arXiv

Source-linked AI summary

Human Action Recognition and Prediction: A Survey

Yu Kong, Yun Fu

arXiv:1806.11230v3cs.CV

TL;DR

The paper addresses how computer-vision systems can understand human actions and intentions from videos, including the challenge of acting before complete execution. It surveys recognition and prediction techniques, representations, deep architectures, datasets, evaluation protocols, and future directions, concluding that action predictability can serve as a prior for stronger prediction methods.

  • Problem

    Understanding human actions and intentions from videos remains difficult because actions vary in appearance and pose, prediction faces uncertain futures, and existing datasets may not represent real-world conditions.

  • Method

    The paper systematically surveys state-of-the-art action recognition and prediction methods, including representations, deep architectures, datasets, evaluation protocols, and future research directions.

  • Results

    The survey reports that action predictability can be used as a prior to inspire more powerful action prediction methods.

  • Takeaways & Limitations

    Action recognition and prediction provide video understanding for current and future states across practical applications such as surveillance, autonomous driving, entertainment, and retrieval.

  • Takeaways & Limitations

    Existing datasets and models remain difficult to generalize to real-world scenarios because datasets often omit conditions such as poor lighting and extreme weather, while large-scale annotations may be insufficient or inaccurate.

Abstract

from arXiv · show

Derived from rapid advances in computer vision and machine learning, video analysis tasks have been moving from inferring the present state to predicting the future state. Vision-based action recognition and prediction from videos are such tasks, where action recognition is to infer human actions (present state) based upon complete action executions, and action prediction to predict human actions (future state) based upon incomplete action executions. These two tasks have become particularly prevalent topics recently because of their explosively emerging real-world applications, such as visual surveillance, autonomous driving vehicle, entertainment, and video retrieval, etc. Many attempts have been devoted in the last a few decades in order to build a robust and effective framework for action recognition and prediction. In this paper, we survey the complete state-of-the-art techniques in action recognition and prediction. Existing models, popular algorithms, technical difficulties, popular action databases, evaluation protocols, and promising future directions are also provided with systematic discussions.

1 Introduction

This survey examines vision-based human action recognition from complete executions and action prediction from incomplete videos, emphasizing their applications, methods, datasets, evaluation, and future directions.

  • Applications: Recognition and prediction support applications including video retrieval, visual surveillance, entertainment, rehabilitation, human-robot interaction, and autonomous driving.These applications use computational analysis to reduce manual video monitoring or anticipate dangerous situations.
  • Task distinction: Action recognition infers human actions after complete execution, whereas action prediction reasons about actions from temporally incomplete video.Prediction supports before-the-fact understanding when systems must react before an action finishes.
  • Action settings: Action videos span individual actions, human interactions, human-object interactions, group actions, RGB-D actions, and multi-view actions.These categories illustrate the range of action settings studied in computer vision research.
  • Survey organization: The survey organizes recent advances across action representations, models, algorithms, datasets, evaluation protocols, and future research directions.Figure 1 provides a hierarchical overview with representative works for the discussed topics.
  • Scope: The survey focuses on video-based recognition and prediction involving one or more people, including RGB-D videos but excluding static-image action learning.Its output is an action label derived from a series of video frames.

2 Human Perception of Actions

Human action understanding extends beyond recognizing what happens to inferring why it happens and who performs it. The survey notes that computer vision has extensively studied action recognition and prediction, while narrative understanding remains less explored.

  • Human action understanding can be decomposed into action recognition, intention understanding, and narrative understanding.
  • Recognition and prediction primarily address what an action is and why it is performed.
  • Narrative understanding concerns the agent’s identity or social role, but few computer vision studies address it.
  • Goal-oriented actions support outcome prediction by linking observed actions to an end state under environmental constraints.

3 Action Recognition

Action recognition systems transform videos into representations and infer action labels from them. The survey describes this pipeline and notes that deep networks increasingly combine representation and classification end to end.

  • A typical action recognition pipeline contains action representation followed by action classification.
  • Action representation converts an action video into either a feature vector or a sequence of vectors.
  • Action classification infers an action label from the resulting vector representation.
  • Deep networks merge representation and classification into a unified end-to-end trainable framework that generally enhances classification performance.

3.1 Shallow Approaches

Shallow action-recognition approaches design representations and classifiers to handle motion, appearance, pose, and spatiotemporal variation. They include holistic and local features, motion descriptors, trajectories, and direct or sequential classification strategies.

  • Action representation: Action representation must efficiently characterize actions while separating action categories despite variations in speed, viewpoint, appearance, and pose.
  • Holistic Representations: Holistic representations encode the entire human subject’s space-time shape but may capture cluttered-background noise.They combine spatial pose information with dynamic body information.
  • Holistic Representations: MEI records where motion occurs, whereas MHI additionally encodes how motion occurs through recency-dependent pixel intensity.Brighter MHI values correspond to more recent motion.
  • Holistic Representations: MEI and MHI are sensitive to viewpoint changes, motivating 3D motion history volumes with location- and rotation-invariant Fourier features.
  • Motion representations: Optical flow captures apparent horizontal and vertical motion between consecutive frames under unchanged illumination assumptions.
  • Local Representations: Local representations focus on salient space-time regions and improve robustness to translation and appearance variation.
  • Local Representations: Space-time interest points detect large motion changes across spatial and temporal dimensions, after which local pixel, gradient, and flow features can be extracted.
  • Local Representations: Tracking interest points into feature trajectories captures longer-duration motion information that short-duration interest points miss.

3.2 Deep Architectures

Deep action-recognition architectures learn video representations through convolution and temporal modeling. The survey contrasts 2D, 3D, multi-stream, temporal-sampling, fusion, and recurrent designs, highlighting their capabilities and limitations.

  • Deep feature learning can generalize well, and scaled networks with massive labeled datasets have achieved high recognition performance across action datasets.
  • Space-time Networks: 2D convolution processes individual frames, whereas 3D convolution captures spatiotemporal dynamics across consecutive frames.
  • Space-time Networks: 3D ConvNets directly learn hierarchical spatiotemporal representations but require more parameters, making them harder to train and limiting ImageNet pre-training.
  • Space-time Networks: C3D extends 3D ConvNets to a deeper architecture trained on large-scale datasets and learns video embeddings that perform competitively across video-analysis benchmarks.
  • Space-time Networks: Inflating pretrained 2D image-classification filters into the temporal dimension enables deeper spatiotemporal extractors to reuse ImageNet filters.
  • Multi-Stream Networks: Two-stream networks separately model static appearance and motion, while later methods add temporal sampling, stream fusion, or recurrent layers to capture longer-range dynamics and cross-stream interactions.

3.3 Learning with Limited Data/Label

Limited annotation and training data motivate weakly supervised, self-supervised, and few-shot action-learning methods. These approaches use incomplete labels, intrinsic video signals, or minimal examples to learn action representations.

  • Large video collections are expensive to annotate and may contain noisy labels, motivating weakly supervised learning.
  • Weakly-supervised Action Learning: Weakly supervised methods learn from video- or clip-level annotations, including action localization in untrimmed videos without annotated temporal boundaries.
  • Unsupervised and Self-supervised Action Learning: Self-supervised methods pre-train networks from signals within videos, including chronological order, temporal coherence, motion consistency, and continuity.
  • Unsupervised and Self-supervised Action Learning: Chronological-order tasks train models to distinguish ordered from shuffled frames or predict the original order of frames and clips.
  • Few-shot Learning: Few-shot and zero-shot methods address recognition with minimalist data or no training samples for some categories by retrieving, generating, or transferring representations.

3.4 Summary

Although deep networks dominate action recognition, shallow methods remain useful because they are easier to train and generally perform well on small datasets.

  • Shallow methods remain useful because they train easily, perform well on small datasets, and can remain competitive on large datasets.

4 Action Localization and Detection

Action localization and detection identify where actions occur in videos, with detection additionally assigning action types. The survey reviews shallow, deep, fully supervised, weakly supervised, and multimodal approaches.

  • Action localization finds spatiotemporal regions containing actions, while action detection additionally identifies the action type in each region.
  • Evaluation of recent detection methods uses mAP@α, the mean Average Precision measured at different intersection-over-union thresholds.
  • Early Methods: Early detection methods segmented untrimmed videos with temporal windows, extracted features, and classified clips to determine action types and temporal locations.
  • Deep Methods: Deep approaches generate action proposals from long videos and use localization networks to refine classification and recognize action labels.
  • Weakly Supervised Methods: Weakly supervised detection reduces annotation requirements by learning from video- or clip-level labels, including pseudo-boundaries and uncertainty-aware pseudo-labels.
  • Multimodal Methods: Skeleton and RGB-D modalities support localization and detection through spatiotemporal attention and graph-based distillation for multimodal representation learning.

5 Action Prediction

Action prediction infers action categories before complete execution, enabling prompt responses but requiring decisions from partial observations. The survey covers early classification, anticipation, and long-term prediction, including temporal progress, predictability, and contextual uncertainty.

  • Action prediction infers labels before action execution ends, unlike recognition, which waits for the complete video.
  • Early Action Classification: Early action classification recognizes unfinished videos from their beginning portions, although conventional recognition methods were designed for complete executions.
  • Early Action Classification: Short-term prediction commonly uses partial observations defined by progress level and observation ratio after segmenting each video into K = 10 segments.
  • Early Action Classification: Action predictability varies across categories, with actions characterized as instantly, early, or late predictable.
  • Action Anticipation: Action anticipation predicts future actions from observed action histories, requiring models to detect seen actions and infer what follows.
  • Long-Term Action Prediction: Long-term prediction targets complex actions and intended goals, but uncertainty makes object interactions and scene context important sources of information.
  • Summary: The survey identifies interpretability of temporal extent, long-term temporal correlations, and multimodal data as open opportunities for stronger prediction models.

6 Motion Trajectory Prediction

Motion trajectory prediction forecasts pedestrians’ future paths and destinations for applications such as surveillance and self-driving cars. The survey emphasizes that reliable forecasting must account for physical scenes and interactions among neighboring pedestrians.

  • Motion trajectory prediction forecasts a pedestrian’s future moving path, including possible destinations and motion patterns.
  • Trajectory prediction is essential for visual surveillance and self-driving cars, where anticipating pedestrian motion is critical.
  • Forecasting is difficult because pedestrians adapt their motion to neighboring people in crowded environments.
  • Methods incorporate physical scene understanding, social interactions, human-scene interactions, and origin or destination priors.
  • RNN/LSTM approaches generate long trajectories, while social pooling models dependencies between pedestrians and adversarial losses can represent multiple socially acceptable futures.

7 Datasets

Action-recognition datasets span controlled and unconstrained settings, interaction types, modalities, and scales. The survey highlights that controlled collections support foundational research but may not train systems for real-world conditions.

  • Popular datasets differ in subjects, background noise, appearance and pose variation, camera motion, and other factors used to compare algorithms.
  • Controlled Datasets: Controlled datasets include Weizmann, KTH, INRIA XMAS, UT-Interaction, and TV-Interaction, covering individual, multiview, and human-human actions.
  • Controlled Datasets: BIT-Interaction and MultiSports introduce clutter, occlusion, multiple people, fine-grained instances, and large-scale spatiotemporal localization annotations.
  • Unconstrained Datasets: Controlled datasets may not adequately train approaches for real-world scenarios, motivating large-scale videos collected from the Internet.

20BN-SOMETHING-SOMETHING dataset

The surveyed datasets extend action understanding beyond basic RGB recognition through object-centered, large-scale, first-person, atomic, and RGB-D recordings. They provide varied action semantics, modalities, and environmental conditions.

  • The object-interaction dataset contains 108,499 video clips across 174 classes and supports learning representations of object physical properties.
  • Moments-in-Time contains over 1,000,000 labeled 3-second clips across 339 categories, including people, animals, objects, and natural phenomena.
  • EPIC-Kitchens provides 55 hours of first-person kitchen video with 125 verb classes, 300 noun classes, and 39,600 action segments.
  • HAA500 is a human-centric atomic-action dataset with 500 classes spanning sports, musical instruments, games and hobbies, and daily actions.
  • RGB-D Datasets: RGB-D datasets use depth alongside RGB, while NTU-RGB+D additionally provides 3D skeleton and infrared data across varied environments.
  • RGB-D Datasets: MSR Daily Activity, 3D Online Action, CAD-120, and UTKinect-Action provide smaller RGB-D collections for daily, cross-environment, continuous, and high-level action tasks.

8 Evaluation Protocols for Action Recognition and Prediction

Evaluation protocols differ across action recognition, action prediction, trajectory prediction, and action detection or localization, reflecting their different application purposes. Recognition commonly uses accuracy-oriented measures, while prediction emphasizes performance at progressively larger observation ratios.

  • Action recognition is evaluated with dataset- and method-dependent measures, including leave-one-out accuracy, confusion matrices, per-frame accuracy, and average precision.
  • Action prediction reports recognition accuracy at observation ratios from 10% to 100%, emphasizing accurate recognition as early as possible.
  • Trajectory prediction commonly uses Average Displacement Error, Final Displacement Error, and Average Non-linear Displacement Error.
  • Action detection and localization use recall-based measures, Average Recall across IoU values, AUC, and mAP at specified IoU thresholds.

9 Future Directions

The survey identifies future directions spanning real-world generalization, annotation-efficient learning, interpretability, multimodal and long-term reasoning, and physical understanding of actions. It emphasizes datasets and models that capture richer temporal, environmental, semantic, and physical variation.

  • Dataset: Existing datasets generalize poorly to real-world conditions because they often assume good lighting, favorable weather, or restricted scenarios.
  • Benefitting from image models: Future work should transfer knowledge from image models to video models, exploiting pretrained spatial relationships through inflation or domain adaptation.
  • Interpretability on temporal extent: Temporal interpretability should explain which primitives and frames support recognition and why some actions can be predicted early.
  • Learning from multi-modal data: Multimodal learning could use visual, audio, and textual information, including adjectives and adverbs that describe motion strength and other fine-grained properties.
  • Learning long-term temporal correlations: Long-term temporal reasoning should model sequential action order and interactions among actions, objects, scenes, and their temporal arrangements to predict extended sequences.
  • Physical aspect of actions: Physical-action understanding should capture fine-grained human-object interactions, spatial relationships, and basic physical properties beyond high-level action labels.
  • Learning actions without annotations: Large action datasets make manual labeling prohibitive, while automatic labeling and crowdsourcing still require verification or suffer from label diversity and errors.
  • Learning actions without annotations: Streaming and untrimmed videos violate assumptions about known action boundaries, motivating approaches that learn from unlabeled or untrimmed video.

10 Conclusion

The survey describes a shift from recognizing present human actions to reasoning about future actions, enabled by big data and powerful models. It synthesizes state-of-the-art methods, representations, architectures, datasets, evaluation protocols, and future directions.

  • The survey covers state-of-the-art action recognition and prediction techniques, including hand-crafted features, models, algorithms, deep architectures, datasets, and evaluation protocols.
  • It presents future research directions for video-based understanding of human movements and their practical applications.
Loading 1806.11230v3…