PaperlayerView exact revision on arXiv

Source-linked AI summary

Align then Adapt: Rethinking Parameter-Efficient Transfer Learning in 4D Perception

Yiding Sun, Jihua Zhu, Haozhe Cheng, Chaoyi Lu, Zhichuan Yang, Lin Chen, Yaonan Wang

arXiv:2602.23069v2cs.CV

TL;DR

Because 4D point-cloud datasets are much smaller and harder to collect than 3D data, efficiently transferring 3D models to 4D perception remains challenging due to overfitting and modality gaps. PointATA addresses this with sequential embedder alignment and parameter-efficient temporal adaptation, outperforming existing methods across multiple tasks, including +8.7% segmentation accuracy on HOI4D.

  • Problem

    4D point-cloud videos are two orders of magnitude smaller than static 3D datasets, motivating transfer from 3D models while existing parameter-efficient methods face overfitting and modality-gap limitations.

  • Method

    PointATA first aligns 3D and 4D embeddings using OTDD and a trainable 4D embedder, then adds a Point Video Adapter and Spatial Context Encoder for geometric and temporal modeling.

  • Results

    PointATA outperforms existing methods across 3D and 4D perception tasks, including +8.7% segmentation accuracy on HOI4D and +0.9% mIoU on Synthia 4D.

  • Takeaways & Limitations

    PointATA preserves static-dynamic alignment and improves generalization while enabling stronger 4D perception with fewer parameters.

Abstract

from arXiv · show

Point cloud video understanding is critical for robotics as it accurately encodes motion and scene interaction. We recognize that 4D datasets are far scarcer than 3D ones, which hampers the scalability of self-supervised 4D models. A promising alternative is to transfer 3D pre-trained models to 4D perception tasks. However, rigorous empirical analysis reveals two critical limitations that impede transfer capability: overfitting and the modality gap. To overcome these challenges, we develop a novel "Align then Adapt" (PointATA) paradigm that decomposes parameter-efficient transfer learning into two sequential stages. Optimal-transport theory is employed to quantify the distributional discrepancy between 3D and 4D datasets, enabling our proposed point align embedder to be trained in Stage 1 to alleviate the underlying modality gap. To mitigate overfitting, an efficient point-video adapter and a spatial-context encoder are integrated into the frozen 3D backbone to enhance temporal modeling capacity in Stage 2. Notably, with the above engineering-oriented designs, PointATA enables a pre-trained 3D model without temporal knowledge to reason about dynamic video content at a smaller parameter cost compared to previous work. Extensive experiments show that PointATA can match or even outperform strong full fine-tuning models, whilst enjoying the advantage of parameter efficiency, e.g. 97.21 \% accuracy on 3D action recognition, $+8.7 \%$ on 4 D action segmentation, and 84.06\% on 4D semantic segmentation.

I. INTRODUCTION

PointATA addresses data scarcity and transfer barriers in 4D perception by aligning static 3D and dynamic 4D representations before efficient adaptation. It combines this two-stage strategy with parameter-efficient temporal modules and reports gains across multiple benchmarks.

  • Motivation: 4D point cloud datasets are much smaller than static 3D datasets, making reusable 3D pre-trained models attractive for cross-modal 4D perception.Maintaining separate fully trained weights for increasing numbers of 4D tasks is also described as impractical.
  • Transfer barriers: Existing 4D parameter-efficient transfer methods are limited by overfitting and an insufficiently measured modality gap between static and dynamic point clouds.The introduction identifies both issues as barriers to further progress in 4D transfer.
  • PointATA: PointATA aligns 3D and 4D embeddings with an optimal-transport-inspired point align embedder before adapting the model for 4D perception.The alignment stage minimizes distance between joint distributions of 4D and 3D embeddings while retaining the 3D backbone as a source of prior knowledge.
  • PointATA: The adaptation stage adds a point-video adapter and spatial-context encoder to exploit temporal information while keeping the large 3D model frozen.These engineering-oriented components are intended to use pre-trained 3D models more efficiently.
  • Results: +1.79% recognition accuracy on MSR-Action3D, +8.7% segmentation accuracy on HOI4D, and +0.9% mIoU on Synthia 4D are reported for PointATA.The experiments evaluate PointATA on multiple 4D-related tasks and report gains on recognition and segmentation benchmarks.
  • Results: PointATA is presented as improving accuracy and resource utilization while preserving static-dynamic alignment and improving generalization.The contribution statements characterize the method as more efficient than existing approaches and as reusing static 3D priors.

II. RELATED WORK

The paper situates PointATA within 3D pre-training, point-cloud video modeling, and efficient cross-modal transfer. Its workflow aligns dynamic features with static source features and then performs parameter-efficient task adaptation.

  • Background: Recent 3D pre-trained models provide reusable spatial representations, while point-cloud videos require models to combine spatial and temporal cues.Point-cloud videos are unordered in space but ordered in time, motivating spatio-temporal modeling.
  • 4D perception: Point-cloud video methods evolved from voxel-based 4D convolutions toward point-based models that track motion or enlarge receptive fields with transformers.The passage notes information loss and latency in voxelization as reasons for the shift toward point-based approaches.
  • Efficient transfer: Efficient tuning research commonly transfers source-model knowledge through embedding alignment or adds lightweight modules and tokens.This taxonomy is drawn from related work in NLP and vision.
  • Alignment: Static-to-dynamic transfer involves both dimensionality mismatch and unreliable cross-domain metrics, which PointATA addresses through embedder reshaping and OTDD-based feature-label comparisons.The proposed comparison remains applicable even when source and target label sets are unrelated or disjoint.
  • PointATA workflow: PointATA uses a two-stage workflow: it aligns 4D feature distributions with 3D source features, then minimizes target task loss with a frozen 3D model and learnable adapters.The workflow is described as pre-training an embedder for alignment followed by 4D parameter-efficient transfer learning.

B. Stage1: Embedder Alignment

Stage 1 aligns dynamic 4D embeddings with static 3D representations by minimizing distributional discrepancy between feature–label domains. Optimal transport supports comparison across potentially unrelated label sets, while class-weighted stochastic estimation reduces computational cost.

  • Stage 1: Embedder Alignment: PointATA replaces the dynamic embedder to match the dimensionality of the frozen 3D model before alignment.The 4D embedder follows P4DConv, while the 3D embedder follows mini-PointNet.
  • Stage 1: Embedder Alignment: Optimal transport compares feature–label pairs across static and dynamic domains, even when their label sets are unrelated or disjoint.Labels are represented as distributions over corresponding features in a shared embedding space.
  • Stage 1: Embedder Alignment: Class-weighted stochastic OTDD estimates the overall domain distance by sampling each 4D class and weighting its class-wise distance.The procedure uses class-specific loaders, uniform subsampling, and a weighted sum of class-wise OTDD values.
  • Stage 1: Embedder Alignment: The alignment objective minimizes the order-2 Wasserstein distance between 4D and 3D joint distributions in the embedding space.This measures the distributional gap across feature and label dimensions.
  • Stage 1: Embedder Alignment: Sinkhorn-based computation lowers optimal-transport complexity, allowing the 4D embedder to use a reasonable time budget while improving transfer performance.The aligned 4D features are encouraged to resemble static 3D features without forcing orthogonal semantics onto specific source classes.

C. Stage2: Efficient Adaptation

Stage 2 adds parameter-efficient temporal and spatial modeling to a pretrained 3D model. PointATA uses a Point Video Adapter to capture spatio-temporal cues while keeping the added computation and parameters small.

  • Stage 2: Efficient Adaptation: PointATA adapts a spatial-only 3D model for dynamic point clouds through an adapter architecture designed for parameter-efficient transfer.The design targets a small task-specific parameter set while retaining performance advantages associated with adapters.
  • Stage 2: Efficient Adaptation: The Point Video Adapter inserts depth-wise separable spatio-temporal convolution into a low-dimensional bottleneck.Operating on compressed features, such as 128D representations, keeps the adapter parameter- and compute-efficient.
  • Stage 2: Efficient Adaptation: Grouped convolutions learn local point features independently, then pointwise convolutions fuse cross-channel information for layerwise temporal understanding.This simultaneously enhances temporal modeling and reduces the cost of processing 4D data.

D. Overall Interaction

The overall interaction combines frozen pretrained 3D components with learnable 4D adaptation modules. The layer update adds feedforward attention, point-video adaptation, and spatial-context encoding to the 4D features.

  • Overall Interaction: Spatial Context Encoders share an architecture across layers but use distinct parameters.They are implemented as origin MLPs to provide global spatial context.
  • Overall Interaction: The frozen 3D model supplies pretrained feedforward and attention parameters, while the Point Video Adapter and Spatial Context Encoder remain learnable.The frozen weights were pretrained on static point clouds and are kept fixed during adaptation.

IV. EXPERIMENT

Experiments evaluate PointATA across recognition, segmentation, gesture, and scene-flow tasks using standardized settings and established 3D baselines. The study also includes detailed analyses of architecture and tuning choices.

  • IV. EXPERIMENT: PointATA is evaluated on 3D action recognition, 4D action segmentation, 4D semantic segmentation, gesture recognition, and 4D scene-flow prediction.Architecture design, tuning strategies, adapter locations, training logs, and feature visualizations are analyzed separately.
  • IV. EXPERIMENT: PointBERT, Point-MAE, and PointGPT-S serve as baselines to support fair comparison.The evaluation settings are standardized in Section IV-A.

A. Experiment Settings

PointATA is evaluated across multiple 4D perception tasks using frozen 3D-pretrained baselines and standardized experimental settings. Results include strong performance on action recognition while emphasizing efficient transfer from static 3D models.

  • Baselines and training: PointATA is applied to PointBERT, Point-MAE, and PointGPT-S baselines with their pre-trained weights frozen.The three baselines are pre-trained on ShapeNet for 300 epochs.
  • MSR-Action3D setup: The MSR-Action3D experiments use 567 depth videos across 20 action classes, with 270 training videos and 297 test videos.Each frame is randomly down-sampled to 2048 points during fine-tuning.
  • 3D action recognition: 96.5% accuracy is attained on the NTU-RGBD dataset for 3D action recognition.The method competes with supervised and self-supervised baselines while using basic motion extraction and plain feature encoding.

D. 4D Action Segmentation: HOI4D

On HOI4D action segmentation, PointATA improves over P4Transformer and competing 4D tuning schemes by using temporal and spatial-context modeling. Its visualizations indicate more accurate action boundaries and less over-segmentation.

  • Dataset: The HOI4D benchmark contains 2,971 training scenes and 892 test scenes, with 150 frames and 2,048 points per sequence.Evaluation reports frame-level accuracy, segmental edit distance, and segmental F1 at a 50% IoU threshold.
  • Architecture: PointATA uses PVA&SCE to harvest temporal cues and guide frame-level attention to motion.The design is presented as a favorable accuracy-efficiency trade-off relative to complex 4D and 3D pre-training pipelines.
  • Visualization: P4Transformer suffers from severe over-segmentation attributed to overfitting local appearance changes while neglecting motion continuity.PointATA aggregates richer context, locates action boundaries more accurately, and markedly reduces over-segmentation.

F. Gesture Recognition: SHREC’17

PointATA performs strongly on gesture recognition and scene flow prediction, combining frozen 3D representations with temporal modeling components. The reported results span fine-grained hand gestures and challenging motion estimation conditions.

  • Gesture recognition: SHREC’17 contains 2,800 videos covering 28 gestures, split into 1,960 training videos and 840 test videos.The model architecture matches that used for MSR-Action3D.
  • Gesture recognition: 96.5% accuracy is achieved by PointGPT-S+ATA on SHREC’17 gesture recognition.Point-BERT+ATA reaches 96.4% and Point-MAE+ATA reaches 95.5%.
  • Gesture recognition: PointATA complements frozen 3D structural hand cues with temporal dynamics and joint identities through the point video adapter.SHREC’17 is described as a noisy, fine-grained task involving finger counting and joint localization.
  • Scene flow prediction: PointATA surpasses 97% accuracy on both Acc3DS and Acc3DR for KITTI scene flow prediction.The point video adapter interleaves depth-wise convolutions with temporal-modeling blocks to capture complex motion.

H. Ablation Studies

The ablations show that PointATA’s alignment, adapter design, and training strategy improve transfer performance, generalization, and computational efficiency while limiting overfitting.

  • Computational Advantages: PointATA improves inference speed by 35.3% and reduces GPU burden by 17.6% on average versus the baseline.Most computation is spent on cross-modal alignment in Stage 1, while SCE and PVA run faster and use less memory than standard adapters.
  • Hyperparameter Sensitivity: Increasing adapter depth beyond four blocks reverses the initial gains, indicating an optimal capacity threshold rather than benefits from continual scaling.Too-small kernels leave the receptive field insufficient for point-level motion and degrade downstream accuracy by 1.3–2%.
  • Architecture Design: Without its proposed components, performance is 77.35%; adding PA and PVA improves it by 17.77%, while SCE unlocks further gains.The complete design surpasses comparable competitors and several 4D-specific pre-training models on MSR-Action3D.
  • Scaling and Generalization: PointATA scales beyond small backbones and short clips when longer clips and denser points improve recognition with ATA enabled.The evaluation includes PTM and PointM2AE as additional foundation models.
  • Training Stability: PointATA reaches 87.9% training accuracy and 87.7% test accuracy, reducing the train-test gap to 0.2% under the reported setting.Naive adapter tuning reaches 100% training accuracy but only 87.8% test accuracy, producing a 12.2% gap.
  • Tuning Strategies: ATA significantly outperforms vanilla adapters and several parameter-efficient tuning methods while keeping the pre-trained backbone frozen.PVA and SCE capture temporal information while preserving a lightweight model.
  • Alignment: After static-dynamic alignment, t-SNE features form tighter and more distinct clusters than with naive adapter tuning.The visualization reports smaller intra-class distances and wider inter-class gaps after alignment.

V. CONCLUSION

PointATA addresses transfer barriers from 3D to 4D perception through staged alignment and adaptation, while outperforming existing methods across multiple tasks. The authors acknowledge insufficient modeling capacity in certain outdoor scenarios as a limitation.

  • Method: PointATA measures the 3D–4D distribution gap with OTDD in Stage 1, then uses PVA and SCE in Stage 2 to capture geometric and dynamic information.The pipeline is presented as a two-stage static-to-dynamic point-cloud adaptation method.
  • Results: PointATA outperforms existing methods on 3D action recognition, 4D action recognition, gesture recognition, and 4D scene flow prediction.The conclusion reports effectiveness across multiple visual benchmarks.
  • Limitation: PointATA may exhibit insufficient modeling capacity in certain outdoor scenarios because of the authors’ limited resources.The authors identify this as a target for future work.
Loading 2602.23069v2…