Source-linked AI summary
3D Human Action Representation Learning via Cross-View Consistency Pursuit
Linguo Li, Minsi Wang, Bingbing Ni, Hang Wang, Jiancheng Yang, Wenjun Zhang
TL;DR
Unsupervised 3D skeleton action representation is limited by costly labels, single-view contrastive supervision, and insufficient use of complementary multi-view information. CrosSCLR combines SkeletonCLR with cross-view consistent knowledge mining to exchange high-confidence samples and embedding distributions, achieving remarkable action-recognition results on NTU-60 and NTU-120. The findings support multi-view knowledge as useful complementary supervision for unsupervised skeleton representation learning.
Problem
Unsupervised skeleton representation learning seeks stronger supervision without costly annotations, but single-view contrastive learning can misclassify similar samples and overlooks multi-view information.
Method
CrosSCLR integrates single-view SkeletonCLR with CVC-KM, which exchanges high-confidence positive/negative samples and embedding context across views.
Results
CrosSCLR achieves remarkable action-recognition results on NTU-60 and NTU-120 under unsupervised settings.
Takeaways & Limitations
Cross-view high-confidence knowledge helps extract more discriminating and comprehensive skeleton representations.
Takeaways & Limitations
Single-view contrastive learning provides only one positive pair and may force highly similar samples apart, motivating reliance on richer embedding distributions and multi-view information.
Abstract
from arXiv · showhide
In this work, we propose a Cross-view Contrastive Learning framework for unsupervised 3D skeleton-based action Representation (CrosSCLR), by leveraging multi-view complementary supervision signal. CrosSCLR consists of both single-view contrastive learning (SkeletonCLR) and cross-view consistent knowledge mining (CVC-KM) modules, integrated in a collaborative learning manner. It is noted that CVC-KM works in such a way that high-confidence positive/negative samples and their distributions are exchanged among views according to their embedding similarity, ensuring cross-view consistency in terms of contrastive context, i.e., similar distributions. Extensive experiments show that CrosSCLR achieves remarkable action recognition results on NTU-60 and NTU-120 datasets under unsupervised settings, with observed higher-quality action representations. Our code is available at https://github.com/LinguoLi/CrosSCLR.
1. Introduction
The paper targets limitations of unsupervised skeleton representation learning, especially contrastive methods’ treatment of similar samples and their underuse of multi-view information. It proposes CrosSCLR, combining single-view SkeletonCLR with cross-view knowledge mining, and reports strong unsupervised results on NTU datasets.
- Unsupervised skeleton learning is motivated by the expense and time required to annotate large 3D skeleton datasets.
- Cross-view contrastive learning treats samples with similar motion patterns as positive across complementary joint and motion views, expanding hidden positive pairs.
- Traditional contrastive learning uses one augmented positive pair while potentially forcing highly similar samples apart as negatives.
- CrosSCLR combines parallel single-view SkeletonCLR models with CVC-KM to mine high-confidence cross-view positive samples and exchange complementary knowledge.
- CrosSCLR achieves remarkable action-recognition results on NTU-RGB+D 60/120 under unsupervised settings.
2. Related Work
Prior work spans self-supervised pretext tasks, deep models for skeleton sequences, and unsupervised skeleton representation learning. The paper positions CrosSCLR as addressing the limited use of skeletons’ natural multi-view knowledge.
- Self-Supervised Representation Learning: Self-supervised sequence methods generate supervision through frame orders, space-time puzzles, or prediction, but depend heavily on pretext-task quality.
- Skeleton-based Action Recognition: Skeleton action-recognition research progressed from handcrafted features to RNN, CNN, and graph-convolutional approaches for temporal or structural modeling.
- Unsupervised Skeleton Representation: Unsupervised skeleton representation methods include non-deep approaches and deep encoder-decoder or GAN-based methods, often relying on reconstruction or prediction.
- Unsupervised Skeleton Representation: CrosSCLR is introduced because prior unsupervised skeleton methods do not exploit the data’s natural multi-view knowledge.
3. CrosSCLR
CrosSCLR combines single-view SkeletonCLR with cross-view consistent knowledge mining to learn unsupervised 3D skeleton action representations. It exchanges complementary embedding knowledge across views to improve contrastive context consistency and produce more regular, discriminating representations.
- CrosSCLR: CrosSCLR combines SkeletonCLR for single-view representation learning with CVC-KM for cross-view knowledge exchange and cooperative training.The framework uses two key modules: SkeletonCLR and cross-view consistent knowledge mining.
- SkeletonCLR: SkeletonCLR uses augmented skeleton sequences as positive pairs, momentum-updated encoders, projection layers, and a FIFO memory bank for negative embeddings.Its contrastive objective pulls positive embeddings together and pushes memory-bank negatives away.
- Limitations of Single-View Contrastive Learning: Multi-view skeleton data provides complementary information that can reveal hidden positive pairs and improve the regularity of learned representations.Examples include views that differ in joint information but are similar in motion.
- Cross-View Consistent Knowledge Mining: CVC-KM selects high-confidence similar samples across views and transfers their contrastive context, including positive samples and embedding distributions.This cross-view exchange encourages consistent contrastive contexts and more regular embedding spaces.
- Cross-View Consistent Knowledge Mining: CrosSCLR first trains views independently, then begins cross-view mining after the models become sufficiently stable to provide reliable knowledge.The two-stage schedule avoids introducing unreliable cross-view information too early.
- Limitations of Single-View Contrastive Learning: SkeletonCLR’s single positive pair overlooks neighboring samples, potentially forcing highly similar negatives apart and producing irregular class structure.The paper identifies this as a limitation for forming clustered embeddings useful for downstream classification.
4. Experiments
Experiments evaluate CrosSCLR on NTU-60, NTU-120, and a disjoint NTU-61-120 transfer setting using unsupervised pre-training and linear or fine-tuned evaluation. CrosSCLR improves single-view and ensemble representations, benefits from cross-view embedding context, and performs best with conservative knowledge mining.
- Experimental settings: CrosSCLR jointly pre-trains joint, motion, and bone views, then evaluates frozen encoders with a supervised linear classifier.The encoder is a reduced-channel ST-GCN, and training uses 300 epochs with InfoNCE for the first 150 epochs before Lcross.
- Effectiveness of CrosSCLR: 79.88 vs 76.44 on xview shows CrosSCLR-joint outperforming SkeletonCLR-joint, while motion rises from 50.82 to 77.59.The reported comparison also shows a much smaller joint–motion performance gap after cross-view training.
- Effectiveness of CrosSCLR: CrosSCLR improves multi-view ensemble results and produces more closely clustered, more discriminating embeddings with similar joint and motion distributions by epoch 300.The qualitative comparison describes the resulting embedding spaces as more regular than SkeletonCLR’s.
- Ablation study: K = 1 gives the best result, whereas larger K values reduce performance because less-confident information can mislead unsupervised learning.K controls the number of mined samples and therefore the depth of cross-view knowledge exchange.
- Ablation study: Removing embedding context worsens CrosSCLR, indicating that similarity or distribution among samples is important beyond the mined positive-sample setting.The comparison includes SkeletonCLR with single-view knowledge mining and CrosSCLR without embedding context.
- Comparison and transfer: CrosSCLR defeats the other unsupervised method and some supervised methods on NTU-120, transfers from NTU-61-120 to NTU-60, and exceeds supervised ST-GCN after fine-tuning.It also reports higher performance with only 1% and 10% randomly selected labels, and the fine-tuned model outperforms supervised ST-GCN on NTU-60 and NTU-120.
5. Conclusion
The work introduces CrosSCLR for unsupervised 3D skeleton-based action representation, combining single-view contrastive learning with cross-view consistent knowledge mining. Experiments report remarkable action-recognition results on NTU datasets.
- CrosSCLR exploits multi-view high-confidence knowledge as complementary supervision for unsupervised 3D skeleton-based action representation.
- The framework integrates single-view contrastive learning with cross-view consistent knowledge mining modules.These modules convey contrastive settings and embedding context among views through high-confidence sample mining.
- Experiments show remarkable action-recognition results for CrosSCLR on NTU datasets.