PaperlayerView exact revision on arXiv

Source-linked AI summary

Going Deeper into Action Recognition: A Survey

Samitha Herath, Mehrtash Harandi, Fatih Porikli

arXiv:1605.04988v2cs.CV

TL;DR

Action recognition seeks to understand meaningful human actions across diverse visual settings and applications, despite evolving definitions and earlier limitations in controlled environments. This survey reviews handcrafted representations and deep-learning techniques, synthesizing advances and fallbacks to clarify available methods and future research needs. It consolidates action as the most elementary meaningful interaction between humans and their surroundings.

  • Problem

    Action recognition must interpret human actions across visual data and broad applications, while action definitions and earlier schemes remain varied and often limited to controlled environments.

  • Method

    The survey reviews pioneering handcrafted representations, deep-learning approaches, taxonomies, and recognition results across action-recognition research.

  • Results

    The survey consolidates action as the most elementary human–surrounding interaction with a meaning.

  • Takeaways & Limitations

    The review identifies what is currently available and what needs to be addressed for semantically aware action-recognition algorithms.

  • Takeaways & Limitations

    The survey chiefly concerns human actions, although actions can also be defined more broadly without dependence on humans.

Abstract

from arXiv · show

Understanding human actions in visual data is tied to advances in complementary research areas including object recognition, human dynamics, domain adaptation and semantic segmentation. Over the last decade, human action analysis evolved from earlier schemes that are often limited to controlled environments to nowadays advanced solutions that can learn from millions of videos and apply to almost all daily activities. Given the broad range of applications from video surveillance to human-computer interaction, scientific milestones in action recognition are achieved more rapidly, eventually leading to the demise of what used to be good in a short time. This motivated us to provide a comprehensive review of the notable steps taken towards recognizing human actions. To this end, we start our discussion with the pioneering methods that use handcrafted representations, and then, navigate into the realm of deep learning based approaches. We aim to remain objective throughout this survey, touching upon encouraging improvements as well as inevitable fallbacks, in the hope of raising fresh questions and motivating new research directions for the reader.

Introduction

The survey frames action recognition as understanding meaningful human–surrounding interactions, reviews competing definitions, and organizes prior work through a dual taxonomy. It matters across applications from surveillance to robotics and human-computer interaction.

  • The survey aims to clarify existing action-recognition methods and identify what remains needed for semantically aware action algorithms.
  • Human motion ranges from simple limb movements to coordinated movements involving multiple limbs and the whole body.A football kick illustrates simple motion, whereas jumping involves legs, arms, head, and the body.
  • Prior literature distinguishes action primitives or short actions from activities composed of subsequent or complex action sequences.Examples include left leg forward as a running primitive and jumping hurdles as an activity involving starting, running, and jumping.
  • The survey defines an action as the most elementary human–surrounding interaction with a meaning.Meaning is associated with the interaction’s category, while surrounding may refer to particular objects such as those involved in brushing hair.
  • Its taxonomy combines a deep-learning section with handcrafted video representations organized by the locality at which features are constructed.The authors present this dual organization as useful for highlighting essential components of both categories.
  • Action recognition supports applications including video surveillance, human-computer interaction, robotics, medical diagnosis, elderly care, and sports analytics.

1. Where to start from?

Early action-recognition systems used explicit 3D models or holistic motion representations, while later methods increasingly favored local and deep representations. Holistic approaches preserve spatiotemporal structure but are constrained by variation and limited fine-detail modeling.

  • Accurate 3D modeling from video is difficult and expensive, motivating holistic or local alternatives.
  • Holistic representations encode global human-body structure, shape, and movement, whereas local representations extract local features.
  • Early approaches represented actions with 3D models, including WALKER’s sequences of 3D structures and Rohr’s connected-cylinder pedestrian model.
  • Holistic Representations: MEI and MHI encode motion in single images: MEI records where motion occurs, while MHI records its temporal history.MHI intensity increases for more recent movement, and its gradients can help identify moving or cluttered background regions.
  • Holistic Representations: Spatiotemporal volumes and surfaces characterize actions through silhouette evolution, boundary-reaching time, and geometric properties such as curvature.These 3D extensions and surface properties are reported as robust to viewpoint changes.
  • Although holistic representations preserve spatial and temporal structure, local and deep representations became favored because holistic methods struggle with viewpoint, appearance, occlusion, and fine-detail variation.

2. Local Representation based Approaches

Local action representations emerged from Space-Time Interest Points and follow a modular pipeline from detecting interest points through descriptor extraction to aggregation.

  • Local representations emerged from Laptev’s Space-Time Interest Points and apply a pipeline of detection, descriptor extraction, and aggregation.

2.1 Interest Point Detection

Interest-point detectors extend image-based corner or Hessian ideas into space-time by requiring both spatial structure and temporal variation. Their sparsity and sensitivity to uncontrolled-video artifacts create important recognition challenges.

  • 2.1 Interest Point Detection: 3D-Harris detects points with large spatial variations and non-constant motions by extending the Harris corner detector into space-time.Unlike 2D Harris, temporal significance is required for detection.
  • 2.1 Interest Point Detection: In the ballet example, still facial regions and a low-variation waist produce no spatiotemporal interest points despite visible spatial features.The detector therefore emphasizes regions changing both spatially and temporally.
  • 2.1 Interest Point Detection: 3D-Hessian provides an alternative detector that uses second-order derivatives instead of the gradients used by 3D-Harris.
  • 2.1 Interest Point Detection: True spatiotemporal corners can be rare in domains such as facial expressions, so overly sparse STIPs may hinder action recognition.Dollar et al. address this by separating spatial and temporal filtering, producing responses from spatially distinctive regions undergoing complex motion.
  • 2.1 Interest Point Detection: Uncontrolled videos can generate irrelevant interest points from camera shake, although nearby static background features may provide useful action context.Statistical pruning is proposed to remove irrelevant detections while retaining potentially informative static features.

2.2 Local Descriptors

Local descriptors are extracted from cuboids or trajectories around detected interest points. Earlier methods commonly used fixed cuboids, while later work questioned that shape and introduced trajectories.

  • 2.2 Local Descriptors: A cuboid is a 3D cube of pixels around a detected interest point, and earlier local-descriptor methods almost unanimously used cuboids.Separate studies in 2009 questioned fixed-shaped cuboids and introduced trajectories for action recognition.

EDGE AND MOTION DESCRIPTORS

Edge and motion descriptors represent spatiotemporal structure through gradient orientations, optical flow, or derivatives of optical flow. Motion-boundary representations emphasize moving-object boundaries, while MPEG Flow reduces motion-field computation cost.

  • EDGE AND MOTION DESCRIPTORS: HoG3D extends Histogram of Oriented Gradients into the spatiotemporal domain as a motion descriptor.
  • EDGE AND MOTION DESCRIPTORS: HoF summarizes optical-flow motion over local regions, while MBH uses spatial derivatives of optical flow to form a more robust descriptor.MBH is computed from motion-boundary fields.
  • EDGE AND MOTION DESCRIPTORS: MPEG Flow provides a motion field virtually free during video decoding, addressing the computational expense of computing optical-flow fields.
  • EDGE AND MOTION DESCRIPTORS: Motion-boundary images emphasize moving-object boundaries that spatial gradients disregard by taking gradients of optical-flow fields.The horse-riding example shows spatial, horizontal, and vertical motion-boundary representations.

PIXEL PATTERN DESCRIPTORS

Pixel-pattern descriptors encode local intensity neighborhoods or feature statistics, extending image descriptors into spatiotemporal video representations. Region covariance descriptors additionally exploit covariance structure and its robustness properties.

  • PIXEL PATTERN DESCRIPTORS: LBP encodes a pixel neighborhood by quantizing relative intensities, and its extensions include spatiotemporal Volume LBP descriptors.
  • PIXEL PATTERN DESCRIPTORS: Region Covariance Descriptors represent a region using the covariance matrix of extracted low-level or mid-level features.The feature set may include gradients, RGB intensities, SIFT, or HoG.
  • PIXEL PATTERN DESCRIPTORS: RCDs use their natural Riemannian structure to provide robustness to scale and translation variations and resilience to noise.

From Cuboids to Trajectories

Early action-recognition descriptors used fixed cuboids, but trajectories track features through time and better accommodate spatial motion changes. Trajectory-based representations also support relative-motion analysis and camera-motion compensation.

  • From Cuboids to Trajectories: Fixed cuboids extracted descriptors around detected spatiotemporal interest points, but interest points may shift spatially across a cuboid’s temporal extent.This motivated the introduction of trajectories as an alternative to fixed-shaped cuboids.
  • From Cuboids to Trajectories: Trajectories track features over time, enabling local features to be extracted along the tracked path.Messing et al. and Matikainen et al. popularized trajectory-based local features, including trajectory velocity.
  • From Cuboids to Trajectories: Relative differences in trajectory direction, magnitude, and location characterize action categories involving human-human interactions such as hand-shaking.Trajectory rectification using camera motion improves representations, while clustering can estimate dominant motion.
  • From Cuboids to Trajectories: Trajectory-based analysis resembles point-light-walker perception, where motion is recognized from tracked points attached to a person’s body.Johansson’s point-light-walker study is cited as an analogy for the role of trajectories.

Sparse or Dense?

The survey contrasts sparse interest-point sampling with dense sampling and reports a shift toward dense representations in later image and video-classification studies.

  • Sparse or Dense?: Dense sampling became preferred over sparse interest points in later image and video-classification studies.The survey summarizes this transition as “sparse is old, dense is new.”

2.3 Aggregation

Aggregation converts variable-sized sets of local video features into fixed-size descriptors suitable for recognition. The survey covers BoV, Fisher Vectors, VLAD, sparse coding, hierarchical methods, and temporal-coherence models.

  • 2.3 Aggregation: Aggregation is needed because videos contain varying numbers of local features, whereas methods such as SVMs generally require fixed-size vectors.BoV and dictionary-learning approaches are presented as natural aggregation choices.
  • 2.3.1 AGGREGATION WITH BOV: BoV represents videos with visual-word occurrence histograms, but its original form ignores temporal information.Spatiotemporal grids address this by aggregating descriptors within video subregions, while hierarchical BoV incorporates spatiotemporal neighborhoods across vocabulary levels.
  • 2.3.2 AGGREGATION VIA FISHER VECTOR: Fisher Vector encoding combines generative and discriminative classification benefits and uses first- and second-order statistics rather than only visual-word occurrences.Fisher Vectors along trajectories achieved state-of-the-art performance in several cited studies.
  • 2.3.2 AGGREGATION VIA FISHER VECTOR: VLAD removes second-order information from Fisher Vectors, reducing descriptor dimensionality to almost half that of FVs.VLAD descriptors are also used with spatiotemporal representations in several cited action-recognition studies.
  • 2.3.3 Dictionary-Based Aggregation: Sparse coding aggregates local spatiotemporal features through learned dictionaries and pooling, while class-specific dictionaries address the limited representational power of a common dictionary for new classes.Related dictionary approaches use transfer learning, saliency, detector banks, and spatial-temporal pyramids.
  • 2.3.4 Aggregation via Temporal Coherence: Temporal-coherence methods represent videos through ordered frame descriptors, temporally structured atomic actions, or sequence models such as HMMs and CRFs.Hierarchical temporal models operate across multiple temporal granularities using CNN features, linear dynamical systems, and VLAD codes.

3. Deep Architectures for Action Recognition

Deep action-recognition architectures are organized into spatiotemporal, multiple-stream, generative, and temporal-coherency networks, with different mechanisms for modeling appearance, motion, and temporal structure. The survey highlights advances alongside constraints such as rigid temporal spans, increased parameters, and violations of temporal coherence.

  • The survey groups deep action-recognition architectures into four categories: spatiotemporal, multiple stream, deep generative, and temporal coherency networks.Each category is discussed with open questions and possible improvements.
  • Spatiotemporal Networks: Spatiotemporal networks use convolutional architectures to learn video representations, while temporal fusion schemes vary from early and late fusion to slow fusion and temporal pooling.Three-dimensional filters capture temporal structure but impose a predefined temporal input span.
  • Spatiotemporal Networks: Longer temporal input durations and combining networks with different temporal awareness improve performance, while homogeneous 3 × 3 × 3 filters outperform filters with varying temporal depth in one study.The longer-duration result is attributed to Varol et al. (2016), whereas the filter comparison comes from Tran et al. (2015).
  • Multiple Stream Networks: Multiple-stream networks combine raw RGB frames with optical flow, and intermediate fusion after the last convolutional layer improves performance while halving parameters relative to the original network.The temporal stream benefits from stacked optical flow and multi-task learning, while the spatial stream benefits from image pretraining.
  • Deep Generative Models: Deep generative models learn temporal data with little or no supervision, including Dynencoders for compact spatiotemporal representation and adversarial networks for video prediction.Dynencoder reconstruction error can also be used for classification; adversarial training makes the generator and discriminator compete during learning.
  • Temporal Coherency Networks: Temporal-coherency methods exploit motion-rich frame tuples and can improve accuracy over networks trained from scratch, but abrupt scene changes can violate the temporal-coherency assumption.The survey specifically identifies advertisements inserted during sports videos as an example of disruptive scene variation.

4. A Quantitative Analysis

The survey compares handcrafted and deep-learning action-recognition methods across increasingly challenging datasets. It finds similar state-of-the-art performance, complementary benefits from dense trajectories, and persistent challenges in data, temporal structure, and practical deployment.

  • Datasets: Action datasets range from controlled clips with basic actions to videos containing camera motion, viewpoint variation, clutter, and inconsistent resolution.The latter conditions require recognition methods to compensate for substantial recording variability.
  • Datasets: Motion cues alone cannot distinguish all action classes, making object information important when actions are defined by related objects.The survey gives 23 distinct billiard categories in Sports-1M as an example.
  • Recognition Results: Deep architectures remain difficult to train on small and medium action datasets because they are data-hungry.Sports-1M was assembled to make training and tuning very deep networks more feasible.
  • Recognition Results: State-of-the-art handcrafted and deep-learning solutions perform equally well across the surveyed action-recognition results.The survey contrasts this with image classification, where comparable handcrafted and deep methods show a different performance gap.
  • Recognition Results: Dense trajectory fusion improves the recognition accuracy of most surveyed state-of-the-art deep-learning solutions.The survey interprets this as evidence that learned deep representations and handcrafted trajectory descriptors are complementary.
  • Future Directions: Practical action recognition requires joint detection and recognition, while fine-grained action tasks can constrain recognition to refined action sets.The survey identifies both directions as areas needing deeper understanding for real-life applications.

5. Conclusion

Video action recognition is challenging because systems must handle variation, noise, and motion cues while modeling complex articulated movements. The survey reviews handcrafted and deep representations and comparatively analyzes these two major research directions.

  • Conclusion: Video analysis must handle scale variation, intra-class diversity, noise, and motion cues simultaneously.These requirements make video data analysis more complicated than static image analysis.
  • Conclusion: The survey reviews handcrafted representations and deep architectures, then provides a comparative analysis of both prevailing approaches.Its motivation is the wide application range and complexity of human articulated-motion patterns.
Loading 1605.04988v2…