Source-linked AI summary
Motion-Saliency Complementary Masked Modeling for Point Cloud Video Understanding
Wei Wang, Yiding Sun, Yuyan Wang, Zhuoyue Zhang, Zhengqiao Li, Dongfu Yin, Chen Li
TL;DR
Point cloud video SSL needs to learn localized geometric motion despite expensive dense labels, varying sampling, and generally unavailable point correspondences. MoSaiC combines curriculum motion-saliency masking, explicit correspondence-free rotation supervision, and complementary-view token consistency, with experiments showing transferable gains across recognition, temporal segmentation, and semantic segmentation. The method is especially effective for dense and low-label settings, while robustness to degraded inputs remains an important future direction.
Problem
Point cloud video SSL must learn localized geometric changes without reliable point-wise correspondences, reducing dependence on expensive frame-level and point-level annotations.
Method
MoSaiC couples CMSM for curriculum motion-salient masking, NFM and TDD for motion targets, and CTCP for token-level consistency across complementary masked views.
Results
Experiments across four datasets and action recognition, temporal action segmentation, and semantic segmentation demonstrate consistent effectiveness, including label-efficient gains.
Takeaways & Limitations
MoSaiC learns transferable point cloud video representations particularly for dense spatio-temporal tasks and low-label settings, with downstream inference retaining only the backbone and no additional cost.
Takeaways & Limitations
The fixed exclusion threshold may need a weighted or learned formulation for sensors with lower point density than the studied datasets.
Abstract
from arXiv · showhide
Point cloud video representation learning is crucial for 3D dynamic scene understanding. In this paper, we propose MoSaiC, a novel Motion-Saliency Complementary masked modeling framework for self-supervised point cloud video representation learning. MoSaiC couples three components: Curriculum Motion-Saliency Masking (CMSM), which guides the masking process toward motion-salient tokens under a curriculum schedule; Normal-Flow Motion (NFM) modeling, which supervises the local rigid rotation of each token in the Lie algebra so(3) as an explicit geometric motion target; and Cross-view Token Consistency Prediction (CTCP), which enforces consistency between two complementary masked views at the token level. Together, these components allow MoSaiC to effectively capture both appearance and motion dynamics. Extensive experiments on multiple downstream tasks, including action recognition, temporal action segmentation, and point-level semantic segmentation, demonstrate the effectiveness of our approach.
1. Introduction
Point cloud video SSL must learn localized geometric change without reliable point correspondences, while existing objectives separately limit motion-aware masking, explicit motion modeling, and token-level cross-view consistency. MoSaiC couples curriculum masking, correspondence-free geometric motion targets, and complementary-view consistency to address these gaps and improve downstream performance.
- Point cloud videos support dynamic 3D perception, but dense annotations across frames, temporal boundaries, and point-level semantics are prohibitively expensive.
- Effective SSL must distinguish localized motion from static geometry despite varying sampling density and unavailable point-wise correspondences.
- Existing methods use motion-agnostic masking, coarse or implicit motion targets, and single-view or clip-level objectives that leave token representations insufficiently constrained.
- MoSaiC integrates CMSM for curriculum-based motion-salient masking, NFM and TDD for explicit and complementary motion targets, and CTCP for cross-view token consistency.
- Curriculum masking, moderate mask ratios, and motion supervision produce gains across action recognition, temporal action segmentation, and semantic segmentation, including HOI4D.
2. Related Work
Related work spans masked reconstruction, contrastive or distillation learning, and hybrid point cloud video SSL, but these approaches only partially address motion-aware target selection, explicit rotation supervision, and token-level consistency. MoSaiC jointly targets all three requirements.
- Masked reconstruction methods recover point-tube geometry or temporal statistics, while contrastive and distillation methods learn invariance across complete, partial, or augmented observations.
- Hybrid methods combine reconstruction with motion or semantic objectives, including trajectory prediction and masked-versus-visible tube contrast.
- Prior approaches generally lack motion-dependent masking, an explicit rotation target, and token-level consistency across complementary masked observations.
- MoSaiC jointly supplies curriculum motion-salient masking, correspondence-free so(3) supervision with TDD, and complementary-view token consistency.
3. Method
MoSaiC pre-trains point cloud video encoders with motion-saliency-driven complementary masking, multi-target reconstruction, and token-level cross-view consistency. Its components jointly model appearance, local rotation, temporal descriptor changes, and complementary masked views.
- Pipeline: MoSaiC tokenizes point cloud videos into spatio-temporal tokens and feeds complementary masked views into a shared asymmetric encoder-decoder.The encoder is retained for downstream tasks.
- Curriculum Motion-Saliency Masking: CMSM computes motion saliency from temporal SHOT differences, standardizes scores within each clip, and converts them into masking probabilities.Saliency probabilities are mixed with a uniform distribution to balance dynamic regions and static background.
- Curriculum Motion-Saliency Masking: A curriculum begins with uniform masking, then gradually increases motion-focused masking after a warmup period.This schedule lets the encoder acquire a generic geometric prior before emphasizing highly dynamic regions.
- Curriculum Motion-Saliency Masking: CMSM samples View A directly and samples View B from View A’s visible tokens, creating asymmetric complementary views with controlled overlap.The construction ensures tokens masked in one view tend to remain visible in the other for CTCP.
- Reconstruction Targets and Losses: Masked tokens are reconstructed using appearance targets, NFM local-rotation targets, and TDD temporal descriptor differences.Appearance uses neighborhood points and Chamfer Distance; NFM regresses correspondence-free axis-angle rotation in so(3), while TDD captures descriptor changes.
- Cross-view Token Consistency Prediction: CTCP aligns a token’s decoded feature when masked in one view with its encoded feature when visible in the other view.A predictor, stop-gradient target branch, and asymmetric sampling are used to prevent collapse.
4. Experiments
MoSaiC consistently improves action recognition, temporal action segmentation, semantic segmentation, label efficiency, linear probing, and cross-dataset transfer. Its gains are linked to complementary supervision over motion-salient tokens, local rotation geometry, and cross-view consistency, with modest additional pre-training cost.
- Action Recognition: MoSaiC achieves the best reported accuracy on SHREC’17 and NvGesture, reaching 91.9% and 89.3%, respectively.It also reaches 95.18% on MSR-Action3D, exceeding the strongest masked-modeling baseline by 0.42 points.
- Temporal Action Segmentation: On HOI4D temporal action segmentation, MoSaiC reaches 81.9% Frame-Acc, 83.6 Edit, and 86.2/83.4/76.8 on F1@{10, 25, 50}.On the P4Transformer backbone, it exceeds M2PSC by 6.0 Frame-Acc and 10.9 F1@50.
- Temporal Action Segmentation: The largest segmentation gains occur for rotation-defined classes, while translation-dominated or static classes improve by only 2.1 to 4.8 points.This pattern supports the role of NFM’s explicit orientation-change supervision in precise boundary localization.
- Semantic Segmentation: MoSaiC reaches 43.7% mIoU on P4Transformer and 44.3% on PPTr for HOI4D semantic segmentation.It exceeds the strongest backbone-matched competitors by 1.4 and 2.0 mIoU, respectively; geometric motion supervision is attributed to retaining local surface orientation.
- Representation Quality and Transfer: With only 10% of labels, MoSaiC reaches 77.8% accuracy versus 62.4% for the from-scratch baseline, while linear probing reaches 86.9% accuracy.Cross-dataset transfer also reaches 91.4% on SHREC’17 and 87.9% on NvGesture.
- Ablations and Efficiency: CMSM, NFM, and CTCP each improve the three downstream tasks, and the full model performs best, indicating complementary contributions.MoSaiC requires 30.2 hours of pre-training versus 21.5 hours for MaST-Pre, while fine-tuning and inference use the backbone’s cost.
5. Discussion
The discussion highlights MoSaiC’s narrowest advantage on MSR-Action3D, where coarse whole-body actions reduce the added value of local rotation supervision, and notes a deployment boundary under low point density.
- Qualitative Visualization: Figure 5 compares the input with complementary Mask-A and Mask-B views and their Recon-A and Recon-B outputs.Each sample is presented across the original input, two masked views, and two corresponding reconstructions.
- Deployment Boundary: A learned or weighted exclusion threshold is proposed as a next step for sensors with lower point density than the studied datasets.The discussion identifies threshold design as a deployment consideration under degraded sensing conditions.
- Why the Margin on MSR-Action3D Is Smallest: +0.42 accuracy is MoSaiC’s narrowest margin on MSR-Action3D, reaching 95.18 versus C2P’s 94.76.The benchmark contains 20 coarse whole-body actions that are largely identifiable from posture in a handful of frames.
- Why the Margin on MSR-Action3D Is Smallest: Local rotation supervision contributes less when actions are posture-identifiable and dominated by coarse, global motion.C2P’s complete-to-partial distillation aligns more closely with this benchmark’s whole-sequence completeness objective.
- Feature Visualization: Figure 6 contrasts t-SNE encoder features trained from scratch with those obtained after MoSaiC pre-training, using colors for action categories.The left panel shows training from scratch, while the right panel shows MoSaiC pre-training.
6. Conclusion
MoSaiC jointly uses motion-saliency-guided masking, explicit local geometric motion supervision, and complementary cross-view consistency for transferable point cloud video representations. Experiments across four datasets and three downstream task families show consistent effectiveness, especially for dense spatio-temporal tasks and low-label settings.
- 6. Conclusion: MoSaiC jointly exploits motion-saliency-guided masking, explicit local geometric motion supervision, and complementary cross-view consistency.These components form the paper’s self-supervised masked-modeling framework for transferable point cloud video representations.
- 6. Conclusion: Experiments across four datasets and three downstream task families demonstrate consistent effectiveness, particularly for dense spatio-temporal tasks and low-label settings.The conclusion identifies robustness under severe input degradation as an important direction for future work.
CRediT authorship contribution statement
The authors’ contributions span conceptualization, methodology, software, validation, resources, visualization, investigation, and writing and editing responsibilities.
- Authorship Contributions: Wei Wang led conceptualization, methodology, and original-draft writing.
- Authorship Contributions: Yiding Sun contributed methodology, software, validation, and original-draft writing.
- Authorship Contributions: Yuyan Wang contributed resources and visualization, while Zhuoyue Zhang and Zhengqiao Li contributed investigation.
- Authorship Contributions: Dongfu Yin and Chen Li contributed writing through review and editing.