Source-linked AI summary
Advances in Human Action Recognition: A Survey
Guangchun Cheng, Yiwen Wan, Abdullah N. Saudagar, Kamesh Namuduri, Bill P. Buckles
TL;DR
Human action recognition addresses video-based recognition across a range of applications. This survey reviews recent methods and datasets using an earlier taxonomy, finding single-layered methods common for direct action recognition and hierarchical methods common for high-level activities.
Problem
Human action recognition must address both actions and high-level activities across applications such as surveillance, human-machine interfaces, and video retrieval.
Source: §1 · p. 1Method
Results
The reviewed literature devotes substantial effort to single-layered recognition directly from videos or images, while most surveyed high-level activity methods use hierarchical approaches.
Source: §5 · p. 21Takeaways & Limitations
The survey identifies more realistic datasets and real-world scenes as settings for further high-level action-recognition research.
Source: §5 · p. 21Most popular test datasets remain simple, constrained, and structured, while reported accuracy is low for more realistic Hollywood and YouTube videos.
Source: §5 · p. 21
Abstract
from arXiv · showhide
Human action recognition has been an important topic in computer vision due to its many applications such as video surveillance, human machine interaction and video retrieval. One core problem behind these applications is automatically recognizing low-level actions and high-level activities of interest. The former is usually the basis for the latter. This survey gives an overview of the most recent advances in human action recognition during the past several years, following a well-formed taxonomy proposed by a previous survey. From this state-of-the-art survey, researchers can view a panorama of progress in this area for future research.
1. Introduction
Human action recognition matches video observations to action labels and supports applications such as surveillance, human-machine interfaces, sports analysis, and retrieval. This survey reviews recent developments using a taxonomy from earlier work.
1. Introduction
- Human action recognition is an active computer-vision topic, motivated partly by growing video records and applications including surveillance, human-machine interfaces, sports analysis, and retrieval.The introduction singles out high-level behavior recognition as especially interesting.Source: §1 · p. 1
- An action is a sequence of body movements that can involve several body parts concurrently, and recognition matches observations such as video to learned patterns to assign an action label.The described system process detects people or body parts, tracks them, and recognizes actions from the tracking results; its tracking-based paradigm is unreliable in cluttered scenes.Source: §1 · pp. 1–2
- The survey focuses on state-of-the-art research not covered by earlier surveys and compares recent developments with previously reported methods using a similar taxonomy.It follows the single-layered and hierarchical categories of Aggarwal and Ryoo.Source: §1 · p. 2
- The paper reviews public datasets, then single-layered approaches by representation and integration, followed by hierarchical methods and a conclusion.Its taxonomy figure is identified as a hierarchical approach-based taxonomy [1].Source: §1 · pp. 2–3, Figure 1 · p. 3
2. Datasets
The survey examines datasets collected since 2009 and compares them across several aspects; the KTH dataset contains six actions performed by 25 subjects in four scenarios.
2. Datasets
- The survey focuses on newly collected datasets in use since 2009 and analyzes and compares them across several aspects.Earlier datasets are covered in.Source: §2 · p. 3
2.1. The KTH Dataset
- KTH contains 2,391 sequences of six actions performed by 25 subjects across four scenarios, recorded with a static camera at 25 fps and 160×120 resolution.The four scenarios vary outdoors, scale, clothing, and indoor setting; the sequences are split into training, validation, and test sets by subject.Source: §2.1 · p. 3
2.2. The Weizmann Dataset
The Weizmann database covers ten natural actions performed by nine subjects and includes additional walking sequences captured from varying viewpoints.
2.2. The Weizmann Dataset
- The database contains 93 sequences of ten natural actions performed by nine subjects.The actions include running, walking, skipping, jumping variations, sideways galloping, waving, and bending.Source: §2.2 · p. 3
- All sequences use a static camera at 25 fps and 180×144 resolution, with ten additional walking sequences spanning viewpoints from 0 to 81 relative to the image plane.Extracted masks and background sequences are provided.Source: §2.2 · p. 4
2.3. The IXMAS Dataset
IXMAS provides multiview daily-life action sequences with silhouettes and visual hulls, while CMU MoBo records four walking variations on a treadmill using six cameras.
2.3. The IXMAS Dataset
- IXMAS contains 2,145 sequences of 13 daily-life actions performed three times by 11 subjects, filmed with five calibrated and synchronized cameras.The dataset provides extracted silhouettes and reconstructed visual hulls.Source: §2.3 · p. 4
2.4. CMU MoBo Dataset
- CMU MoBo covers four walking conditions performed by 25 subjects on a treadmill, with more than 8,000 images per subject captured by six high-resolution color cameras.Sequences are 11 seconds long at 30 fps and 640×480 resolution; extracted silhouettes are provided.Source: §2.4 · p. 4
2.5. HOHA-1 (Hollywood Human Actions I) Dataset
HOHA-1 contains movie-derived samples of eight human actions, with separate movie sets for training and testing.
2.5. HOHA-1 (Hollywood Human Actions I) Dataset
- HOHA-1 covers eight actions drawn from 32 movies.Source: §2.5 · p. 4
- Its training data comes from 12 movies and contains 219 samples.Source: §2.5 · p. 4
- Its test data comes from 20 different movies and contains 211 manually verified samples.Source: §2.5 · p. 4
2.6. HOHA-2 (Hollywood Human Actions II) Dataset
HOHA-2 extends HOHA with additional actions and scene classes, using movie-separated training and test samples.
2.6. HOHA-2 (Hollywood Human Actions II) Dataset
- HOHA-2 extends HOHA and covers 12 actions and 10 scene classes across 69 movies.Source: §2.6 · pp. 4–5
- The dataset contains 3669 samples, with 823 training samples from 33 movies.Source: §2.6 · p. 5
- Its test set contains 884 manually verified samples from 36 movies not used for training.Source: §2.6 · p. 5
2.7. Human Eva Dataset
Human Eva provides synchronized video and 3D pose data for several actions; Human Eva II extends the walking-and-jogging sequences.
2.7. Human Eva Dataset
- Human Eva-I contains four grayscale and three color video sequences, calibrated and synchronized with 3D body poses.Source: §2.7 · p. 5
- Human Eva-I covers six actions performed by four subjects, with video captured at 640x480 resolution and 60 Hz.Source: §2.7 · p. 5
- Human Eva II extends the walking-and-jogging combination sequences using two subjects.Source: §2.7 · p. 5
2.8. CMU Mocap Dataset
The CMU Mocap Dataset organizes motion-capture actions into six categories and 23 subcategories, performed by 144 subjects.
2.8. CMU Mocap Dataset
- The CMU Mocap Dataset covers six categories, subdivided into 23 subcategories.Source: §2.8 · p. 5
- Its actions were performed by 144 subjects.Source: §2.8 · p. 5
- The actions were captured using 12 Vicon infrared MX-40 cameras.Source: §2.8 · p. 5
3. Single-layered Approaches
This section reviews recent single-layered action-recognition methods and compares them with earlier work. It organizes recognition by how videos are represented and how temporal relationships are handled.
3. Single-layered Approaches
- Space-time approaches treat time as a video-volume dimension, while sequential approaches model ordered observations; the latter generally achieve better results.The survey reviews recent progress in both branches and compares it with earlier methods.Source: §3 · p. 7
- Single-layered approaches recognize actions directly from raw video, and the survey compares recent progress with previously surveyed methods.The section covers space-time and sequential approaches.Source: §3 · p. 7
3.1. Advances in Space-Time Approaches
- Space-time approaches use motion-related information across several video representations, while raw-volume methods instead match the full volume as a feature or template.Full-volume matching can suffer from noise and meaningless background information.Source: §3.1 · p. 8, §3.1.1 · p. 8
3.1.1. Action Recognition with Space-Time Volumes
- MHI-based approaches combine motion history with foreground or HOG appearance features, while other methods use silhouettes, skeletons, accumulated motion, or learned motion descriptors.Hu et al. use SMILE-SVM classification; the section also describes covariance-based silhouettes, motion templates, and semilatent topic models.Source: §3.1.1 · pp. 8–10
3.1.2. Action Recognition with Space-Time Trajectories
- Dense-trajectory methods sample points across frames, track them using dense optical-flow displacement, and compute HOG, HOF, and MBH descriptors around interest points.Figure 3 illustrates sampling at multiple spatial scales, tracking over L frames, and the resulting descriptors.Source: §3.1.2 · p. 11, Figure 3 · p. 11
3.1.3. Action Recognition with Space-Time Local Features
- Space-time local features can be sparse or dense; reported methods include saliency-selected interest points, multiscale clouds, and alternatives to sparse 3D Harris corners.Le et al. also learn hierarchical invariant spatiotemporal features from unlabeled video using stacked convolutional ISA.Source: §3.1.3 · pp. 11–14
3.2. Sequential Approaches
- Sequential approaches represent actions as ordered observations, extracted from frames or frame sets, and include exemplar-based and state model-based methods.The survey provides a comparison of space-time approaches in Table 2.Source: §3.2 · p. 14, Table 2 · p. 15
3.2.1. exemplar-based approaches
- Exemplar-based methods compare an input video’s observation sequence with action templates or samples, using approaches such as DTW or prototype sequences matched with FastDTW.Lin et al. construct prototype sequences from shape-motion features using a hierarchical prototype tree.Source: Figure 2 · p. 10, §3.2.1 · p. 16
4. Hierarchical Approaches
Hierarchical recognition models high-level activities from simpler sub-activities, and the survey groups these approaches as statistical, syntactic, and description-based, following the taxonomy in [1].
4. Hierarchical Approaches
- Hierarchical approaches represent complex activity structure and flexibly cover individual actions, human or object interactions, and group activities.They also offer an interface for incorporating prior knowledge; single-layer methods can recognize atomic actions or be extended into hierarchical models.Source: §4 · p. 17
4.1. statistical approaches
- Statistical approaches use multi-level HMMs and DBNs, while P-nets address both concurrent and sequential sub-activities.A four-layer probabilistic latent model clusters spatiotemporal features into atomic actions and recognizes actions without specifying the number of latent states.Source: §4.1 · p. 18
- Other statistical methods use hierarchical body-motion representations and cascade CRFs, hierarchical K-means trees, or domain knowledge to address insufficient training data.Source: §4.1 · p. 18
4.2. syntactic approaches
- Syntactic methods represent atomic sub-activities as symbol strings, but this representation limits concurrent-action recognition and often requires user-provided production rules.Kitani et al. introduced a method to learn rules automatically; a proposed four-level grammar distinguishes strong, weak, and stochastic spatiotemporal relations.Source: §4.2 · p. 19
4.3. description-based approaches
- Description-based methods encode activities through temporal, spatial, and logical relations among embedded sub-activities, supporting sequential and concurrent actions.Approaches include Allen interval predicates, formal syntax, Bayesian belief networks, Petri nets, event logic, and probabilistic extensions; MLNs were also used for probabilistic inference.Source: §4.3 · pp. 19–20
- Description-based examples include temporal-logic recognition using multiple input sources and basketball event annotation combining probabilistic and logical inference.The basketball framework requires semantic descriptions of typical scenarios and uses Allen’s Interval Logic and MLNs to handle uncertainty in low-level observations.Source: §4.3 · p. 20
- Because common standard datasets are not sufficiently used, description-based approaches are compared functionally rather than statistically.Source: §4.3 · p. 20
5. Conclusion
The survey reviews 50 influential proposals from the preceding three years using a prior taxonomy and reports datasets used for validation. It finds extensive work on direct, single-layer recognition alongside emerging hierarchical approaches to high-level activities, while realistic datasets remain challenging.
5. Conclusion
- The survey identifies 50 influential proposals from the preceding three years, organizing state-of-the-art developments by category and reporting validation datasets.Its discussion uses the taxonomy of a previous survey, distinguishing recognition directly from images from recognition based on low-level sub-actions.Source: §5 · pp. 20–21
- Much research targets direct single-layer recognition, especially with space-time volumes and local features, while high-level activity methods mainly use statistical, syntactic, or description-based hierarchies.The survey notes interest in combining formal descriptors and probabilistic reasoning to interpret actions.Source: §5 · p. 21
- Popular test datasets remain simple, constrained, and structured, while realistic Hollywood and YouTube videos are challenging and have low reported accuracy.The survey calls for further high-level action-recognition research on datasets and in real-world scenes.Source: §5 · p. 21
- The survey cannot review every approach; it anticipates domain-specific techniques and says a cross-domain framework would benefit the community.Source: §5 · p. 21