Source-linked AI summary
Learning Discriminative Representations for Skeleton Based Action Recognition
Huanyu Zhou, Qingjie Liu, Yunhong Wang
TL;DR
Skeleton-based action recognition can confuse actions because skeletons omit interactive objects and contextual clues. The paper introduces a multi-level Feature Refinement Head that combines spatial-temporal decoupling with contrastive feature refinement. Experiments on three benchmarks report competitive results and improved discrimination of ambiguous actions.
Problem
Skeleton representations omit interactive objects and contextual information, making similar actions difficult to distinguish.
Method
The Feature Refinement Head decouples features spatially and temporally, then uses prototypes and ambiguous samples for contrastive feature refinement across multiple GCN stages.
Results
The method obtains competitive results on NTU RGB+D, NTU RGB+D 120, and NW-UCLA and improves discrimination of confusing action categories.
Takeaways & Limitations
Feature refinement can be used as a plug-and-play auxiliary head for stronger supervision of ambiguous skeleton-based actions.
Takeaways & Limitations
Few-shot ambiguous actions with insufficient data remain unexplored, and the module introduces extra training costs.
Abstract
from arXiv · showhide
Human action recognition aims at classifying the category of human action from a segment of a video. Recently, people have dived into designing GCN-based models to extract features from skeletons for performing this task, because skeleton representations are much more efficient and robust than other modalities such as RGB frames. However, when employing the skeleton data, some important clues like related items are also discarded. It results in some ambiguous actions that are hard to be distinguished and tend to be misclassified. To alleviate this problem, we propose an auxiliary feature refinement head (FR Head), which consists of spatial-temporal decoupling and contrastive feature refinement, to obtain discriminative representations of skeletons. Ambiguous samples are dynamically discovered and calibrated in the feature space. Furthermore, FR Head could be imposed on different stages of GCNs to build a multi-level refinement for stronger supervision. Extensive experiments are conducted on NTU RGB+D, NTU RGB+D 120, and NW-UCLA datasets. Our proposed models obtain competitive results from state-of-the-art methods and can help to discriminate those ambiguous samples. Codes are available at https://github.com/zhysora/FR-Head.
1. Introduction
Skeletons provide compact, robust inputs for GCN-based action recognition, but omit interactive objects and context that distinguish similar actions. The proposed feature refinement module uses contrastive learning and multi-level supervision to improve discrimination of ambiguous actions.
- Skeletons are compact, viewpoint-robust representations that support lightweight, computationally efficient GCN-based action recognition.
- Missing interactive objects and contextual information makes actions such as Writing, Reading, and Typing on a keyboard difficult to distinguish from skeletons alone.
- The feature refinement module decouples hidden features spatially and temporally, then uses contrastive learning to increase discrimination between ambiguous actions.
- Confident samples maintain class prototypes, while ambiguous samples are calibrated through feature-space relationships with confident samples.
- The plug-and-play module can be jointly trained with other losses and discarded during inference.
- Experiments on NTU RGB+D, NTU RGB+D 120, and NW-UCLA report significant improvement over state-of-the-art models.
2. Related Work
Related work develops pose-estimation and action-recognition methods from sequences, images, and human-body graphs, with GCNs becoming the mainstream skeleton-based approach. Contrastive learning and fine-grained ambiguity modeling provide relevant foundations, but the cited work identifies no prior method targeting ambiguity in skeleton-based action recognition.
- Human Pose Estimation: Human pose estimation methods use temporal information from 2D pose sequences to address depth ambiguity in 3D poses.
- Action Recognition: Early action-recognition methods use sequence classification, image-like skeleton representations, RGB-skeleton combinations, or 3D CNNs, without explicitly exploiting human-body spatial structure.
- Skeleton-Based Action Recognition: GCNs became mainstream because human joints and bones naturally form graphs, enabling explicit exploitation of body topology for skeleton-based action recognition.
- Contrastive Learning: Contrastive learning commonly distinguishes transformed views using augmentation, large batches, momentum encoders, or dynamic queues.
- Ambiguous Samples: Prior ambiguity-focused methods mainly address fine-grained image classification through bilinear pooling, metric learning, or pairwise interaction.
- Ambiguous Samples: The cited related work reports no existing method aimed at solving ambiguity in skeleton-based action recognition.
3. Methodology
The method adds a training-only Feature Refinement Head to GCN backbones, decoupling spatial-temporal features and using contrastive refinement to improve representations of ambiguous actions. Multi-level heads refine features at several backbone stages while adding no inference-time computation.
- Backbone: The backbone processes T × V × 3 skeleton sequences with temporal CNNs and graph convolutions, then produces multi-scale feature vectors for K action categories.Strided units reduce the temporal dimension while increasing the channel dimension.
- Feature Refinement Head: The FR Head is a plug-and-play module that refines multi-level backbone features using spatial-temporal decoupling and contrastive learning with class prototypes and ambiguous samples.It can be combined with GCN-based networks and is used only during training.
- Multi-Level Feature Selection: FR Heads are placed at the 1st, 5th, 8th, and final TGN layers, with stage-specific weights combining their local contrastive losses into a multi-level objective.The 5th and 8th layers use strided operations.
- Spatial-Temporal Decoupling: Spatial-temporal decoupling uses parallel branches that pool either spatial or temporal dimensions, squeeze features with 1 × 1 convolutions, and apply contrastive losses to both representations.The branches produce unified feature vectors with channel size Ch.
- Contrastive Feature Refinement: During training, correctly classified samples update class prototypes through exponential moving averages, while false negatives and false positives are identified as ambiguous samples for feature-space calibration.The method pulls false negatives toward the relevant confident samples and penalizes false positives near that class.
- Training Objective: The full objective combines cross-entropy with the weighted multi-level contrastive loss, and the refinement head is discarded during inference.The contrastive loss gives stronger supervision to lower-confidence true-positive samples through the predicted class probability.
4. Experiments
Experiments across three skeleton-action datasets evaluate the FR Head through ablations, hyper-parameter studies, backbone integration, ambiguous-action analysis, and state-of-the-art comparisons. The method improves ambiguous-action discrimination and achieves competitive or best results across multiple benchmarks.
- Datasets and setup: The evaluation covers NTU RGB+D, NTU RGB+D 120, and NW-UCLA using their recommended cross-subject, cross-set, cross-view, or cross-camera protocols.NTU RGB+D 120 contains 113,945 sequences across 120 classes; NW-UCLA contains 1,494 clips across 10 categories.
- Ablation study: All proposed refinement submodules improve the baseline, with contrastive loss and spatial-temporal decoupling contributing most; combining them performs better still.The ablation includes CL Loss, ST Decouple, and multi-level refinement, while additional parameters affect training time but not inference.
- Ablation study: Increasing refinement weights toward later stages produces the optimal configuration, while emphasizing earlier layers more heavily can hurt performance.The selected configuration is λ1 = 0.1, λ2 = 0.2, λ3 = 0.5, λ4 = 1, wcl = 0.1.
- Combined with other backbones: Around 1.0% accuracy improvement is obtained across GCN backbones after adding the FR Head, with approximately 0.5M extra parameters ignored during inference.The module is evaluated with five widely used GCN-based backbones on NTU RGB+D 120.
- Performance on ambiguous actions: The method substantially improves Hard Level actions, which are associated with similar and easily misclassified actions.Hard Level actions have CTR-GCN accuracy below 70%; the analysis uses the X-Sub setting with joint input.
- Performance on ambiguous actions: Across 60 ambiguous groups, the method improves group-wise accuracy in most groups and produces more discriminative, compact t-SNE clusters than CTR-GCN.Each group contains an anchor action and three frequently confused actions.
- Comparison with the state of the art: The models outperform most existing methods on the three datasets, achieving the best results on NTU RGB+D 120, NTU RGB+D X-Sub, and NW-UCLA.On NTU RGB+D X-View, the method reaches state-of-the-art performance with a gap from the best result.
5. Conclusion
The proposed feature refinement module enhances multi-level GCN features across spatial and temporal dimensions, using contrastive learning to address ambiguous skeleton-based actions. Experiments on three benchmarks show improved discrimination and strong performance, while few-shot settings remain open for future work.
- The module enhances multi-level GCN features along spatial and temporal dimensions, using high-confidence and misclassified samples for contrastive calibration.FP and FN samples are incorporated to make use of misclassified actions.
- On three widely used benchmarks, the proposed method obtains satisfactory results and outperforms state-of-the-art methods.
- Ambiguous actions in few-shot settings with insufficient data remain to be explored.The authors identify this as future work despite performance on three public large-scale datasets.