PaperlayerView exact revision on arXiv

Source-linked AI summary

Space-Time Representation of People Based on 3D Skeletal Data: A Review

Fei Han, Brian Reily, William Hoff, Hao Zhang

arXiv:1601.01006v3cs.CV

TL;DR

Existing 3D skeleton-based human representations need a comprehensive synthesis of their methods, properties, and supporting resources. This paper surveys and categorizes the field across four perspectives, reviews acquisition methods and datasets, and finds that multimodal, body-part, and manifold-based approaches are often advantageous for richer or longer activities. Skeleton-only representations remain limited in texture and shape information, while handcrafted features depend heavily on domain knowledge and parameter tuning.

  • Problem

    The field lacks a dedicated comprehensive review of 3D skeleton-based human representations and their construction approaches.

  • Method

    The paper surveys 3D skeleton-based representations, categorizes them by modality, encoding, structure and transition, and feature engineering, and reviews acquisition methods and benchmark datasets.

  • Results

    The review finds that multimodal representations usually achieve better accuracy than single-modality methods, while body-part and manifold-based representations can improve recognition for complex or long-time activities.

  • Takeaways & Limitations

    Space-time skeleton representations provide robust, efficient options for human gesture, action, and activity recognition, with richer feature structures useful as activity complexity increases.

  • Takeaways & Limitations

    Skeleton data cannot encode texture information effectively, and handcrafted features require substantial domain knowledge and careful parameter tuning.

Abstract

from arXiv · show

Spatiotemporal human representation based on 3D visual perception data is a rapidly growing research area. Based on the information sources, these representations can be broadly categorized into two groups based on RGB-D information or 3D skeleton data. Recently, skeleton-based human representations have been intensively studied and kept attracting an increasing attention, due to their robustness to variations of viewpoint, human body scale and motion speed as well as the realtime, online performance. This paper presents a comprehensive survey of existing space-time representations of people based on 3D skeletal data, and provides an informative categorization and analysis of these methods from the perspectives, including information modality, representation encoding, structure and transition, and feature engineering. We also provide a brief overview of skeleton acquisition devices and construction methods, enlist a number of public benchmark datasets with skeleton data, and discuss potential future research directions.

1. Introduction

3D skeleton-based human representations have gained attention because they compactly encode human attributes and offer robustness and real-time advantages. This survey addresses the field's lack of a dedicated comprehensive review by categorizing methods, comparing their properties, and covering datasets and acquisition background.

  • Human representations encode compact features of shape, pose, and motion for human-centered reasoning systems across applications such as video analysis, robotics, and augmented reality.
  • 3D perception data offers geometric information and robustness to illumination, scale, and rotation, while affordable RGB-D cameras and increased computation accelerated its adoption.
  • 3D skeleton representations model whole-body joint relationships and can be robust to scale, illumination, viewpoint, rotation, and motion-speed changes while supporting high-frame-rate computation.
  • The survey fills a literature gap as the first review dedicated to human representations based on 3D skeleton data.
  • It reviews 171 representations, including 150 papers from the recent five years, and categorizes them by modality, coding, structure and transition, and feature engineering.
  • The review also lists benchmark datasets, discusses skeleton acquisition and construction as background, and focuses mainly on gesture, action, and activity recognition.

2. Background

This background section surveys how 3D skeleton data are acquired, constructed, and organized into public benchmark resources. It contrasts sensing technologies and computational reconstruction methods while noting the scope of the dataset review.

  • Acquisition devices: 3D skeleton data can come directly from motion-capture, time-of-flight, and structured-light devices.These devices provide different skeletal body models and acquisition characteristics.
  • Acquisition devices: Motion-capture systems track markers with optical cameras or inertial sensors but are typically expensive and restricted to controlled indoor environments.Optical systems track reflective markers, whereas inertial systems estimate body-part rotation.
  • Acquisition devices: Structured-light sensors do not require markers, are inexpensive, and can provide 3D skeleton information in real time.Kinect SDK and OpenNI can generate skeletal models from color-depth data.
  • Skeleton construction: Skeleton construction methods include body-part recognition, nearest-neighbor or ICP matching, and RGB-image methods based on visual features or known poses.Body-part recognition can classify depth-image pixels and infer joints using mean-shift estimation.
  • Skeleton construction: Construction can use single images, multiple camera views, or learned models, including monocular 3D estimation, stereo depth recovery, and deep neural networks.Multi-camera systems estimate depth before applying depth-based skeleton construction methods.
  • Benchmark datasets: Public benchmark datasets contain 3D skeleton data collected across scenarios, with early resources commonly acquired using MoCap systems.Human3.6M is described as a large MoCap dataset containing 3.6 million poses and corresponding images.

3. Information Modality

The survey classifies skeleton-based human representations by the information modality used to derive features from raw 3D skeletal data. Its modality taxonomy comprises joint displacement, orientation, raw position, and combined information.

  • Modality definition: Skeleton-based representations are constructed from features computed from raw 3D skeletal data acquired through sensing technologies.The survey defines each feature type extracted from an individual sensing technique as a modality.
  • Modality taxonomy: The information-modality taxonomy contains four categories: joint displacement, orientation, raw position, and combined information.The reviewed approaches are summarized in Tables 3–6 by category.
  • Modality taxonomy: Table 3 organizes joint-displacement representations by encoding, structure and transition, feature engineering, and temporal or invariance properties.Its notation includes concatenation-, statistics-, and Bag-of-Words-based encoding; low-level, body-part, and manifold structures; and multiple feature-learning approaches.

3.1. Displacement-Based Representations

Displacement-based representations encode spatial relationships among joints or temporal changes across frames. The survey highlights pairwise relative positions, reference-joint differences, temporal displacements, and joint movement volumes as representative constructions.

  • Overview: Joint-displacement features are widely used because their structure is simple and implementation is easy.They capture either within-frame displacement between joints or displacement of the same joint across time.
  • Spatial displacement: Spatial displacement representations calculate coordinate differences between joints within the same frame.Relative joint displacements represent 3D spatial relationships among skeletal coordinates.
  • Spatial displacement: Pairwise relative joint position is the most widely studied displacement feature and can be normalized for invariance to body position, orientation, and size.For joints p_i and p_j, the displacement is p_ij = p_i − p_j.
  • Spatial displacement: Reference-joint features compute coordinate differences between every joint and a manually selected reference joint.The reference and target joints are expressed in the world coordinate system.
  • Temporal displacement: Temporal displacement representations compare joint locations across frames and typically combine spatial and temporal information.Examples use current, previous, initial, or earlier frames to characterize motion and reduce action-recognition latency.
  • Movement volume: Joint movement volume summarizes each joint’s motion range along the x, y, and z axes as a volume feature.The ranges are L_a = max(a_j) − min(a_j), and the volume is V_j = L_xL_yL_z.

3.2. Orientation-Based Representations

Orientation-based representations encode skeletal geometry through directions between joints or temporal changes in joint orientation. They emphasize invariance to human position, body size, and camera orientation while using spatial or temporal orientation structures.

  • Orientation invariance: Orientation-based features are generally invariant to human position, body size, and camera orientation.
  • Spatial orientation of pairwise joints: Pairwise spatial-orientation methods compute displacement-vector orientations between skeletal joints acquired at the same time step.
  • Spatial orientation of pairwise joints: Joint-to-centroid and joint-to-torso orientations represent joint angles or orientation matrices relative to body-centered references.
  • Spatial orientation of pairwise joints: Relative joint orientations can connect adjacent first-order orientation vectors and quantize continuous directions into eight symbols for noise robustness.
  • Temporal joint orientation: Temporal joint-orientation methods compare orientations of the same joint across frames, including velocity-orientation histograms and phase-space trajectory models.

3.3. Representations Based on Raw Joint Positions

Raw joint positions are encoded directly, flattened across frames, statistically summarized, organized into trajectories, or supplied to deep models. These approaches range from simple sensor-derived features to trajectory and end-to-end recurrent representations.

  • Raw joint positions: Raw sensor joint positions provide a direct information source for constructing space-time 3D human representations.
  • Raw joint positions: Flattening joints within each frame produces a sequence matrix whose columns contain joint coordinates at individual time points.
  • Raw joint positions: Cov3DJ statistically summarizes flattened 3D joint vectors across a temporal sequence using their covariance.
  • Raw joint positions: Key-joint selection methods retain more descriptive joints, with evolutionary selection followed by position, scale, and rotation normalization in one approach.
  • Trajectory-based representations: Trajectory-based methods form joint trajectories from raw positions and extract temporal features such as wavelet encodings.
  • Deep-learning representations: Deep skeleton representations commonly use raw joint positions as input to end-to-end hierarchical recurrent neural networks.

3.4. Multi-Modal Representations

Multi-modal representations integrate complementary skeletal information sources to encode people in 3D space. The reviewed approaches combine spatial and temporal cues, including distances, coordinates, and temporal joint variations.

  • Multi-modal representations: Multi-modal construction combines multiple skeletal information sources to increase the descriptive power of 3D human representations.
  • Multi-modal representations: Spatiotemporal representations can integrate pairwise joint distances, spatial joint coordinates, and temporal variations of joint locations.

3.5. Summary

The survey links representation choices to invariance and task requirements. Displacements and orientations support view invariance, temporal cues support dynamic-motion encoding, and fusion is associated with improved performance, while robust representations may require sophisticated construction methods.

  • Advantages and limitations: Displacement-based representations are invariant to people’s absolute locations and camera-relative orientations, supporting view-invariant spatiotemporal representations.
  • Advantages and limitations: Orientation-based representations also provide view invariance and are usually invariant to human scale because they do not depend on displacement magnitude.
  • Advantages and limitations: Robust human representations typically require more sophisticated construction techniques, including deep learning.
  • Task suitability: Temporal information is needed to encode dynamic human motions, whereas non-temporal representations suit pose and gesture recognition.
  • Task suitability: Trajectory-based representations can be designed to be insensitive to motion speed, and fusion of multiple modalities typically improves performance.

4. Representation Encoding

The survey groups skeleton-representation encoding into concatenation-based, statistics-based, and bag-of-words methods, which integrate extracted features into classifier-ready vectors through different levels of processing.

  • Encoding methods are grouped into concatenation-based, statistics-based, and bag-of-words approaches.These methods integrate extracted features into a final feature vector for classifiers or other reasoning systems.
  • 4.1. Concatenation-Based Approach: Concatenation directly combines skeleton features, such as joint displacements and orientations, into a one-dimensional vector.One example concatenates joint angles, angle velocities, and joint velocities into a 130-dimensional vector per frame before combining frames.
  • 4.2. Statistics-Based Encoding: Statistics-based encoding summarizes feature distributions without quantization, using representations such as covariance matrices and histograms.Covariance-based vectors can have a size independent of the number of frames, while histogram methods count joints in spatial bins.
  • 4.3. Bag-of-Words Encoding: Bag-of-words encoding quantizes high-dimensional features into codewords from a learned dictionary and represents instances by normalized code frequencies.K-means and sparse coding are commonly used to construct dictionaries for this process.
  • 4.4. Summary: Concatenation is efficient for real-time applications, whereas bag-of-words generally reports superior performance but requires dictionary construction and feature quantization.Statistics-based methods are relatively noise-robust but may lack discriminative power because they do not identify representative patterns or model data structure.

5. Structure and Topological Transition

The survey organizes representations by whether they use low-level Euclidean features, body-part mid-level features, or manifold transitions, with increasing structural modeling across these categories.

  • Structure-and-transition methods comprise low-level Euclidean representations, body-part mid-level representations, and manifold-based representations.The categories distinguish direct feature extraction from explicit body-structure modeling and topological transformation.
  • 5.1. Low-Level Features in Euclidean Space: Low-level approaches extract skeletal features directly in Euclidean space, using either single-layer or multilayer constructions.EigenJoints, for example, applies PCA to pairwise joint displacements without modeling the body hierarchy.
  • 5.1. Low-Level Features in Euclidean Space: Deep learning methods create multilayer representations directly from joint positions, with intermediate and output layers encoding different feature levels.These approaches have attracted increasing attention for learning representations from skeletal joint locations.
  • 5.2. Mid-Level Features Based on Human Body Parts: Body-part models incorporate coarse kinematic structure and can capture relationships among both joints and body parts.Examples decompose the body into parts such as arms, legs, and torso, then model spatial configurations and temporal movements.
  • 5.3. Manifold-Based Representations: Manifold-based methods map skeleton trajectories from 3D Euclidean space into another topological space, treating them as curves there.A Lie-group representation models geometric joint relationships through 3D rotations and translations.
  • 5.4. Summary: The survey reports good performance across categories, while body-part and manifold representations can improve recognition for increasingly complex, especially long-duration, activities.The authors associate this advantage with the limited representation capability of low-level feature structures in such settings.

6. Feature Engineering

Feature engineering has shifted from manually designed skeleton features toward learned representations, although hand-crafted features remain dominant and deep learning introduces computational costs.

  • Feature engineering methods are categorized as hand-crafted features, dictionary learning, unsupervised feature learning, and deep learning.The survey frames these categories within a broader transition from manual engineering to automated learning and extraction.
  • 6.1. Hand-Crafted Feature Engineering: Hand-crafted features manually encode geometric, statistical, morphological, or other attributes of 3D skeletal data.Examples include joint distances, articulation angles, relative positions, selected informative joints, and key-frame features.
  • 6.2. Automated Feature Learning: Dictionary learning represents features as sparse combinations of learned basis elements and commonly uses k-means or sparse coding.It forms the foundation of bag-of-words encoding and adapts dictionaries to task-specific data.
  • 6.2. Automated Feature Learning: Unsupervised feature learning discovers low-dimensional features from higher-dimensional inputs but still relies on manually crafted initial features.PCA and feature-selection methods are examples described by the survey.
  • 6.2. Automated Feature Learning: Multimodal learning can fuse RGB-D and skeletal information using sparsity-inducing regularization to produce compact, informative features.The cited approaches use group-l1 or Frobenius-norm regularization for feature integration.
  • 6.3. Summary: Hand-crafted features remain prevalent but require domain knowledge and parameter tuning, while deep learning learns from raw skeleton data at higher computational cost.The survey notes that this cost may limit current suitability for online, real-time applications.

7. Discussion

Across benchmark datasets, no approach is consistently best; multimodal, bag-of-words, and learning-based methods generally perform favorably, while the survey identifies representation and evaluation gaps for future work.

  • 7.1. Performance Analysis of the Current State of the Art: The comparison evaluates precision and efficiency across MSR Action3D, CAD-60, MSRC-12, and HDM05, covering Kinect v1 and motion-capture sensors.The detailed comparison is reported in Table 7.
  • 7.1. Performance Analysis of the Current State of the Art: No single approach guarantees the best performance across all datasets, and methods vary in performance between benchmarks.The survey also reports generally better activity-recognition performance for multimodal methods than single-modality methods.
  • 7.1. Performance Analysis of the Current State of the Art: Bag-of-words encoding can improve performance, while learning-based feature engineering generally outperforms traditional hand-crafted methods in the reported comparisons.The latter group includes deep learning, unsupervised feature learning, and dictionary learning.
  • 7.2. Future Research Directions: Skeleton data cannot encode texture information effectively, limiting representation of human-object interaction and motivating fusion with texture and shape models.The survey identifies multisensory integration as a potential way to increase descriptive power.
  • 7.2. Future Research Directions: Different devices provide skeleton data with different kinematic models, motivating cross-training and unified formats for general representations and larger learning datasets.Unifying formats could help integrate benchmark datasets for data-driven methods such as deep learning.
  • 7.2. Future Research Directions: The field needs representation-level quantitative evaluation protocols independent of learning and application-level evaluations.The proposed direction is to design metrics that facilitate quantitative comparison of representations.
  • 7.2. Future Research Directions: Future systems should learn arbitrary poses and outdoor skeletons while preserving real-time processing for online applications.The survey points to stereo vision and LiDAR as possible outdoor sensing sources.

8. Conclusion

The survey categorizes 3D skeleton-based space-time human representations across four perspectives and compares their advantages and limitations. It finds benefits from multimodal and learning-based construction while identifying several future research opportunities.

  • The survey categorizes skeleton-based representations by information modality, representation encoding, structure and topological transition, and feature engineering.It also compares the pros and cons of methods within each perspective.
  • Multimodal representations that integrate multiple feature sources usually achieve better accuracy than representations using a single feature modality.
  • Learning-based construction approaches, including deep, unsupervised, and dictionary learning, show promising performance compared with traditional hand-crafted feature engineering.
  • The survey covers 3D skeleton acquisition and construction methods alongside existing space-time representation techniques.
Loading 1601.01006v3…