PaperlayerView exact revision on arXiv

Source-linked AI summary

VPN: Learning Video-Pose Embedding for Activities of Daily Living

Srijan Das, Saurav Sharma, Rui Dai, Francois Bremond, Monique Thonnat

arXiv:2007.03056v1cs.CV

TL;DR

ADL recognition requires distinguishing subtle or visually similar actions that challenge conventional video models. VPN addresses this by embedding RGB cues and 3D poses together and applying topology-aware, coupled spatio-temporal attention. The model outperforms state-of-the-art action-classification results across four public datasets.

  • Problem

    ADL recognition must resolve subtle spatio-temporal patterns and similar visual patterns that make actions difficult to distinguish.

  • Method

    VPN projects RGB video features and 3D poses into a common embedding space and uses a topology-aware attention network with coupled spatial-temporal weights.

  • Results

    VPN outperforms state-of-the-art action-classification results on four public datasets.

  • Takeaways & Limitations

    The model combines RGB appearance with pose geometry to produce more discriminative features for ADL action classification.

  • Takeaways & Limitations

    The temporal-alignment design assumes pose variation across a few consecutive frames is negligible, and the embedding is proposed for future use with noisy poses.

Abstract

from arXiv · show

In this paper, we focus on the spatio-temporal aspect of recognizing Activities of Daily Living (ADL). ADL have two specific properties (i) subtle spatio-temporal patterns and (ii) similar visual patterns varying with time. Therefore, ADL may look very similar and often necessitate to look at their fine-grained details to distinguish them. Because the recent spatio-temporal 3D ConvNets are too rigid to capture the subtle visual patterns across an action, we propose a novel Video-Pose Network: VPN. The 2 key components of this VPN are a spatial embedding and an attention network. The spatial embedding projects the 3D poses and RGB cues in a common semantic space. This enables the action recognition framework to learn better spatio-temporal features exploiting both modalities. In order to discriminate similar actions, the attention network provides two functionalities - (i) an end-to-end learnable pose backbone exploiting the topology of human body, and (ii) a coupler to provide joint spatio-temporal attention weights across a video. Experiments show that VPN outperforms the state-of-the-art results for action classification on a large scale human activity dataset: NTU-RGB+D 120, its subset NTU-RGB+D 60, a real-world challenging human activity dataset: Toyota Smarthome and a small scale human-object interaction dataset Northwestern UCLA.

1 Introduction

ADL recognition is difficult because actions can be subtle, visually similar, or observed from different views. VPN addresses these challenges by aligning RGB cues with 3D poses and combining spatial embedding with topology-aware spatio-temporal attention.

  • Challenges: ADL actions are difficult to distinguish because they exhibit subtle motions, similar visual patterns, and camera-view variation.Examples include pouring water versus pouring grain, rubbing hands versus clapping, and actions observed from different cameras.
  • Research gap: Few studies have addressed ADL recognition in indoor scenarios compared with recognition from internet videos.
  • Existing modalities: 3D poses provide geometric, view-adaptive, and illumination-robust cues but omit appearance information important for human-object interactions.
  • Research gap: Pose-driven attention methods improve recognition but do not explicitly align RGB cues with corresponding 3D poses.
  • Design assumption: Temporal embedding is considered unnecessary because poses are already aligned across time, and pose variation over consecutive ADL frames is assumed negligible.This assumption also covers cases where the number of poses differs from the number of image frames.
  • Proposed approach: VPN combines spatial embedding with an attention network that models human-joint topology and tightly couples spatial and temporal attention.The model is end-to-end trainable and can operate as a layer on top of 3D ConvNets.

2 Related Work

Existing action-recognition methods model appearance, temporal dynamics, or pose, but often lack the alignment and topology-aware coupling needed for challenging ADL. VPN combines RGB features and 3D poses through spatial embedding and jointly varying spatio-temporal attention.

  • RGB-based methods: Image descriptors and sequential networks model appearance evolution over time, while 3D convolutions directly model spatio-temporal patterns.
  • RGB-based methods: Holistic 3D ConvNets are trained over entire video volumes and may be too rigid for subtle ADL spatio-temporal patterns.
  • Attention methods: Attention mechanisms added to 3D ConvNets extract salient spatio-temporal patterns but can rely heavily on appearance and pixel position.
  • Pose-based methods: Pose-based methods address view variation but do not provide the appearance cues required to distinguish actions with similar motion.
  • VPN: VPN enforces RGB–pose correspondences through spatial embedding and uses human topology to activate relevant joints for attention.
  • VPN: Its spatio-temporal coupler makes spatial attention weights vary over time rather than treating spatial and temporal attention independently.

3 Proposed Action Recognition Model

VPN combines RGB video features and 3D poses through spatial embedding and pose-driven attention. Its attention network models human-body structure and coupled spatio-temporal saliency, while the embedding aligns visual and pose representations.

  • VPN takes a 3D visual feature map and corresponding 3D poses to align human joints, modulate features across space and time, and support action classification.The modulated feature map is produced by weighting the original feature map according to learned relevance.
  • The attention network combines a pose backbone with a spatio-temporal Coupler to learn coupled attention weights for the visual feature map.The pose backbone processes pose sequences, while the Coupler derives spatial, temporal, and joint spatio-temporal weights.
  • GCNs process pose graphs to learn spatial relationships among 3D human joints and produce salient pose features for attention estimation.Each pose is represented as a graph with weighted joint connectivity, and the resulting features are aggregated over time with residual processing.
  • The Coupler combines inflated spatial and temporal attention weights into AST, allowing spatial saliency to vary across time before modulating the video feature map.A residual connection preserves properties of the original visual features after attention weighting.
  • Spatial embedding projects global visual features and pose-based latent attention vectors into a common space and measures their correspondence there.Transformation matrices map both modalities into a shared De-dimensional space, while normalized representations support the embedding loss.
  • The embedding loss supplies feedback to pose-based attention vectors, improving spatial attention weights compared with an attention network without embedding.The loss also updates the visual backbone toward pose-aware characteristics while training with the classification objective.

4 Experiments

VPN is evaluated through component ablations, qualitative analyses, and comparisons with state-of-the-art methods across four activity-recognition datasets. The experiments show that its attention network and spatial embedding improve recognition, especially for fine-grained and visually similar activities.

  • Experimental setup: VPN is evaluated on NTU-60, NTU-120, Toyota-Smarthome, and Northwestern-UCLA using action-classification experiments.NTU-120 and Smarthome contain stronger fine-grained and similar-appearance challenges, motivating the ablation studies on those datasets.
  • Ablation study: The attention network improves classification by up to 8.4% on NTU-120 and 5.4% on Smarthome over the I3D baseline.Adding spatial embedding provides further gains of up to 0.9% on NTU-120 and 4.4% on Smarthome.
  • Ablation study: The GCN pose backbone exploits human-joint topology, while the spatio-temporal coupler provides fine spatial attention for important video frames and improves disambiguation of similar-appearance actions.The coupler improves classification by up to 1% on NTU-120; GCN particularly improves Snapping Finger and Apply cream on face over LSTM.
  • Ablation study: Normalized Euclidean loss outperforms bidirectional KL divergence for spatial embedding, while one-directional KL losses reduce classification accuracy.The comparison supports two-way feedback between visual features and the attention network.
  • Ablation study: Spatial embedding improves spatial attention across experiments, with large gains for fine-grained Smarthome actions such as pouring water (+77.7%) and pouring grains (+76.1%).Other reported gains include cutting bread (+50%), pouring from kettle (+42.8%), and inserting teabag (+35%).
  • Qualitative analysis: VPN activates relevant body joints and produces sharper, more discriminative feature maps than I3D for actions with similar appearance.The qualitative examples distinguish hopping from jumping through leg activations and distinguish two-handed headphone use from one-handed hair flicking through hand activations.
  • Comparison with the state-of-the-art: VPN outperforms state-of-the-art methods across NTU-60, NTU-120, Toyota-Smarthome, and Northwestern-UCLA, with average gains of 3.9% and 4.9% on NTU-120 and Smarthome.It also improves over the NTU-60 state of the art by up to 0.8% using RGB+Poses and over N-UCLA by 0.4%.

5 Conclusion

The paper proposes VPN for ADL classification, combining video-pose embedding with attention to produce more discriminative action features. The model outperforms state-of-the-art results on four public datasets.

  • VPN combines an accurate video-pose embedding with an attention network for ADL classification.The embedding and attention network together yield a more discriminative feature map.
  • The attention network leverages human-joint topology and a coupler to provide spatio-temporal attention weights across each video.
  • VPN outperforms state-of-the-art action-classification results on four public datasets.The conclusion describes this as a first step toward combining RGB and pose through explicit embedding.

Appendix overview

The appendix provides computational and experimental details for VPN, including normalization, pose-backbone baselines, divergence-loss comparisons, and additional model insights.

  • Section 1 details normalization of the Euclidean loss used for spatial embedding of RGB and pose.
  • Section 2 describes LSTM pose-backbone baselines with or without a coupler in ablation-study tables.
  • Section 3 explains divergence losses compared with normalized Euclidean loss in ablation studies.
  • Section 4 provides additional insights about VPN.

1 Details on normalization of Euclidean loss

The normalization details define projected video and pose representations on the unit hypersphere and use a small positive value to avoid division by zero.

  • The projected video and pose feature representations are normalized to the unit hypersphere.The representations are obtained by dividing each feature by its L2 norm.
  • A small positive ϵ prevents division by zero during normalization.

2 LSTM Pose backbone with or without coupler baselines

The appendix compares an LSTM pose backbone and attention configurations with or without the coupler. The LSTM baseline uses a frozen three-layer pretrained model, while the no-coupler variant applies spatial and temporal attention separately.

  • The LSTM pose backbone uses a pretrained three-layer stacked LSTM with frozen cell-gate weights.Its output feature vector concatenates all LSTM output features over time.
  • Residual connections are added between the original pose input and LSTM output tensor for comparison with the GCN pose backbone.The passage notes that these residual connections do not improve the act…
  • Without the coupler, spatial and temporal attention weights are applied separately to two RGB streams before their modulated feature maps are concatenated.

3 Baselines with KL divergence loss

The spatial embedding is evaluated with KL-divergence losses that align visual and pose embeddings, including directional and bidirectional variants.

  • KL-divergence losses compare visual and pose embeddings in the spatial-embedding experiments.
  • The evaluated directional losses are DKL(fe||Pe) and DKL(Pe||fe) for n samples.
  • The bidirectional KL-divergence loss sums DKL(fe||Pe) and DKL(Pe||fe).

4 Detailed qualitative analysis of VPN

VPN’s components improve fine-grained action recognition by aligning RGB and pose information, modeling human-joint structure, and coupling attention across space and time. Qualitative analyses show stronger discrimination of similar actions, while remaining errors arise under limited data and degraded visual or pose quality.

  • 4.1 Illustration to show the impact of VPN components: Each VPN component improves Top-5 per-class action-classification accuracy relative to the I3D baseline.The evaluated components are spatial embedding, GCN pose backbone, and spatio-temporal coupler.
  • 4.1 Illustration to show the impact of VPN components: Spatial embedding improves fine-grained recognition by aligning RGB images with 3D poses.
  • 4.1 Illustration to show the impact of VPN components: The GCN pose backbone boosts classification by incorporating human joint configuration into attention-weight computation.
  • 4.1 Illustration to show the impact of VPN components: Spatio-temporal coupling provides discriminative attention weights that better disambiguate actions with similar appearance.
  • 4.2 Comparison with state-of-the-art methods: 52.7% improvement occurs for hush, 40.7% for staple book, and 36.2% for reading over I3D on subtle-motion actions.
  • 4.3 Result visualization: Approximately 42% and 27% accuracy improvements are reported for action pairs previously confused by I3D on NTU RGB+D.VPN better discriminates staple book from cutting papers and taking something out of bag from put something into a bag.
  • 4.3 Result visualization: Toyota Smarthome errors remain for actions affected by extremely low training counts, occlusion, low resolution, or low-quality poses.Using tablet reaches 13% accuracy and remains confused with using laptop; drink from glass is not recognized.
Loading 2007.03056v1…