PaperlayerView exact revision on arXiv

Source-linked AI summary

Joint-Embedding Prediction of Masked Point Tubes for Self-Supervised Learning on 4D Point Cloud Videos

Jheng-Ling Lee, Shang-Tse Chen

arXiv:2608.24093v1cs.CVcs.LG

TL;DR

4D point cloud video learning needs useful representations from unlabeled data while avoiding costly labels and excessive emphasis on low-level coordinate details. The paper uses teacher-free JEPA-style masked point-tube latent prediction with SIGReg, and reports stronger initialization across downstream recognition, limited-label, and transfer settings. The authors position the approach as a promising alternative to reconstruction-centered pretraining within the evaluated scope.

  • Problem

    4D point cloud video representation learning is difficult because annotations are costly and reconstruction-based pretraining can overemphasize low-level geometric details.

  • Method

    A teacher-free shared-encoder JEPA framework predicts masked point-tube latents from visible spatiotemporal context and uses SIGReg to discourage collapse.

  • Results

    The learned representations improve downstream fine-tuning, limited-label learning, and cross-dataset transfer across action and gesture recognition benchmarks.

  • Takeaways & Limitations

    Latent prediction with SIGReg provides a strong initialization for 4D action and gesture recognition, especially with limited labeled data.

Abstract

from arXiv · show

Self-supervised representation learning for 4D point cloud videos is challenging because annotations are costly and reconstruction-based pretraining can overemphasize low-level geometric details. We propose a JEPA-style framework that learns from unlabeled spatiotemporal point clouds through latent point-tube prediction. Instead of reconstructing raw coordinates, the model masks spatiotemporal regions and predicts their target representations from visible context representations in feature space. To stabilize latent prediction, we incorporate Sketched Isotropic Gaussian Regularization, which encourages non-collapsed embeddings without relying on explicit reconstruction targets. This formulation aims to capture both spatial structure and temporal dynamics while keeping the pretraining objective aligned with downstream semantic recognition. Experiments on action and gesture recognition benchmarks show that the learned representations improve downstream fine-tuning, limited-label learning, and cross-dataset transfer. These results suggest that JEPA-style latent prediction is a promising alternative to reconstruction-centered pretraining for 4D point cloud videos.

Introduction

4D point cloud video pretraining needs annotation-efficient objectives that capture spatiotemporal structure without being dominated by coordinate-level reconstruction. The paper proposes a teacher-free JEPA-style framework with SIGReg and evaluates it across multiple recognition settings.

  • 4D point cloud videos contain evolving unordered point sets, but action and gesture labels are costly to obtain.
  • Existing methods may bias learning toward local geometry, impose manual motion assumptions, or depend on external teacher targets.
  • The framework predicts masked point-tube representations from visible spatiotemporal context in latent space rather than reconstructing raw coordinates.
  • SIGReg regularizes the latent distribution to discourage representation collapse without requiring a separate teacher branch.
  • Experiments cover full-label fine-tuning, semi-supervised learning, few-shot recognition, transfer learning, and qualitative attention-map visualization.

4D Point Cloud Video Understanding

Prior 4D point cloud video research develops temporal, geometric, contrastive, masked, and distillation-based representations. JEPA reframes prediction around latent target representations, while SIGReg addresses the associated collapse problem.

  • 4D Point Cloud Video Understanding: Supervised 4D models extract spatial geometry and temporal motion from unordered point sets using point-based and transformer architectures.
  • Self-Supervised Learning: Self-supervised methods derive motion-sensitive or feature-space signals from unlabeled point cloud videos through temporal pretexts and contrastive objectives.
  • Self-Supervised Learning: Distillation methods train students from partial observations using complete sequences as guidance.
  • Masked Modeling: Masked video methods extend reconstruction to spatiotemporal point tubes and may add temporal cardinality or motion supervision.
  • JEPA and Collapse Prevention: JEPA predicts target representations from visible context, reducing pressure to preserve every low-level detail, while SIGReg regularizes embeddings toward an isotropic Gaussian distribution.

Method

The method tokenizes 4D point cloud videos into spatiotemporal point tubes, masks most tokens, and predicts their latent representations from visible context. Its objective combines latent prediction with SIGReg to preserve non-collapsed embeddings.

  • Tokenization: The framework converts input clips into local spatiotemporal point-tube tokens that serve as masking and prediction units.
  • Tokenization: Point-tube centers provide spatial and temporal positional information, allowing the encoder and predictor to distinguish token locations.
  • Masking: Randomly masking 75% of tokens leaves visible context from which the model must infer the masked representations.
  • Architecture: A P4D tokenizer, Transformer encoder, and lightweight predictor replace coordinate reconstruction with JEPA-style latent prediction.
  • Architecture: The shared encoder produces full-sequence target latents and visible-context features, avoiding a separate EMA teacher in the SIGReg path.
  • Predictor: The predictor receives context features and masked target centers, then retains outputs at target query slots as predicted latents.
  • Training Objective: The loss combines masked latent prediction with SIGReg, encouraging predictable spatiotemporal semantics while maintaining a non-degenerate approximately isotropic latent space.

Experiments

Experiments across action recognition, few-shot learning, transfer, attention visualization, and ablations show that the proposed latent-prediction framework provides useful representations and benefits from SIGReg and target-position conditioning.

  • MSRAction-3D: The framework improves the P4Transformer backbone and achieves the strongest result among listed P4Transformer-based methods on MSRAction-3D.The authors attribute this to useful spatiotemporal initialization from latent prediction with SIGReg.
  • NTU RGB+D: On NTU RGB+D, the model outperforms the supervised P4Transformer baseline and listed self-supervised baselines in full-label fine-tuning.This result uses 100 pretraining epochs, compared with 200 epochs reported for MaST-Pre and M2PSC.
  • Semi-Supervised Learning: Pretraining yields a substantially larger improvement with 50% labeled NTU RGB+D training videos than with full labels.The test split and evaluation protocol remain unchanged in this limited-label setting.
  • Few-Shot Learning: Pretraining improves mean accuracy over training from scratch at 1-, 3-, and 5-shot budgets on MSRAction-3D, with the largest gains at 1 and 3 shots.It also reduces standard deviation substantially at 3 and 5 shots, while the smaller 5-shot gain reflects the greater availability of labels.
  • Transfer Learning: NTU RGB+D pretraining improves P4Transformer transfer accuracy on SHREC’17 at both 30 and 50 fine-tuning epochs.The larger relative improvement with fewer epochs suggests useful initialization for faster adaptation.
  • Visualization: Attention maps concentrate on action-relevant body parts, including arms for arm wave and side-boxing and the punching hand for forward punch.The visualization is qualitative and indicates emphasis on informative spatiotemporal regions rather than only low-level geometric structure.
  • Ablation Studies: Removing SIGReg lowers accuracy to 89.90% and causes early target-latent collapse, while the full model stabilizes near unit target-latent standard deviation.The EMA target encoder underperforms the shared-encoder SIGReg formulation, and detaching target latents produces the largest downstream drop.
  • Ablation Studies: Removing target-position queries reduces accuracy from 94.08% to 93.03%, showing that visible context is useful while positional conditioning adds measurable benefit.The ablated target slots use only the shared learned mask token.

Conclusion

The framework learns masked spatiotemporal point-tube representations through latent prediction rather than coordinate reconstruction, with SIGReg regularizing the shared-encoder path. Experiments include qualitative visualizations, sensitivity ablations, and downstream evaluation, while the study remains limited in backbone and benchmark scope.

  • The framework masks spatiotemporal point-tube tokens and predicts their target representations from visible context in latent space.
  • The pretrained encoder is evaluated through attention-map visualizations and sensitivity ablations on MSRAction-3D.
  • SIGReg regularizes the learned latent distribution to reduce representation collapse during shared-encoder training.
  • The results show strong initialization for 4D action and gesture recognition, especially with limited labeled data.
  • The study is limited to P4Transformer-based backbones and a modest benchmark scope.

Overview

The supplementary document provides implementation details, experimental protocols, training settings, evaluation procedures, and additional diagnostics supporting the main paper.

  • The document specifies the finite-sample SIGReg objective, including latent tensor shape, random projections, characteristic-function statistics, and gradient paths.
  • It reports the experimental environment, data protocols, training settings, random seeds, and evaluation procedure.
  • It concludes with ablation diagnostics and per-seed few-shot results.

SIGReg Details

SIGReg operates on full-sequence encoder latents by projecting them into one-dimensional distributions and matching those distributions to a standard Gaussian through a finite-sample discrepancy. The regularizer is combined with masked prediction in the training objective, with shared-encoder gradients enabled by default.

  • SIGReg receives full-sequence encoder latents Z with shape (S, B, d), representing token positions, minibatch elements, and encoder dimensions.
  • Each forward pass samples and normalizes a projection matrix with J = 1024 non-trainable random directions.
  • The projected empirical distribution is matched to a standard Gaussian characteristic function using a truncated Epps–Pulley discrepancy.
  • The implementation approximates the characteristic-function integral on a fixed frequency grid with K = 17 knots and folded quadrature weights.
  • Pretraining minimizes L = Lpred + λSIGRegLSIGReg, with prediction targets not detached in the default shared-encoder path.

Implementation Details

The supplementary implementation uses specified hardware, datasets, optimization settings, random seeds, controlled ablations, and video-level evaluation procedures. These details define the experimental protocols and reporting choices used across the benchmarks.

  • Experiments use one CUDA device per run on a workstation with four NVIDIA RTX 6000 Ada GPUs and the listed software stack.
  • Dataset protocols report class counts, video-level split sizes, cross-subject splits, and sliding-window clip extraction.
  • The semi-supervised protocol uses 20,160 of 40,320 NTU RGB+D training videos, while few-shot MSRAction-3D samples 1, 3, or 5 videos per class across 20 classes.
  • Training uses AdamW with warmup and cosine decay, Smooth L1 masked-prediction loss during pretraining, and cross-entropy during fine-tuning.
  • Controlled ablations vary target detachment, target queries, mask ratio, SIGReg weight, or EMA targets while keeping other settings fixed.
  • Video-level top-1 accuracy sums clip probabilities within each test video, and the best fine-tuning-epoch result is reported.

Additional Results

The additional results show that latent-space diagnostics, masking, and SIGReg jointly determine downstream performance, while few-shot evaluation uses paired per-seed comparisons. The reported tables also specify the optimization and architecture settings underlying these experiments.

  • Prediction-Loss Diagnostics: Prediction loss alone is not a reliable model-selection criterion because near-zero loss can accompany collapsed target latents and lower downstream accuracy.The diagnostics average prediction loss and target standard deviation over minibatches and the final ten pretraining epochs.
  • Prediction-Loss Diagnostics: The full shared-encoder SIGReg formulation performs best by maintaining non-collapsed target latents while preserving an informative predictive objective.EMA targets retain lower variance, whereas stop-gradient targets preserve variance but make masked-target prediction harder and reduce downstream accuracy.
  • Sensitivity Ablations: Moderate masking performs best because low masking weakens prediction, whereas high masking removes too much visible context.Target variance remains stable across the sensitivity settings.
  • Sensitivity Ablations: Intermediate SIGReg strength provides the best downstream balance between maintaining target variance and limiting prediction loss.Weak regularization insufficiently maintains variance, while strong regularization increases prediction loss.
  • Few-Shot Multi-Run Results: Few-shot robustness is evaluated with paired Scratch and Pretrained initializations using identical labeled support sets for each seed.Table 12 reports per-seed all-class video-level top-1 accuracy with means and standard deviations over ten run seeds.
Loading 2608.24093v1…