PaperlayerView exact revision on arXiv

Source-linked AI summary

MAETrack: Unleashing the Potential of Pretrained Geometric Priors for 3D Single Object Tracking

Sifan Zhou, Qiwei Wang, Linyue Tan, Ziyu Liu, Ziyu Zhao, Xiaobo Lu

arXiv:2609.16695v1cs.CV

TL;DR

MAETrack addresses limitations in transferring pretrained representations to 3D single object tracking by selectively preserving shallow geometric information. It reports improvements over conventional fine-tuning, particularly for geometrically challenging categories, while the benefit may vary across categories and datasets.

  • Problem

    Transfer of pretrained representations to 3D single object tracking lacks an explicit mechanism for preserving and adapting useful information.

  • Method

    MAETrack is a lightweight adaptation framework that selectively inherits shallow geometric representations through Layer-Selective Initialization.

  • Results

    MAETrack significantly outperforms vanilla fine-tuning, particularly for geometrically challenging categories, with negligible overhead.

  • Takeaways & Limitations

    Effective transfer requires selectively inheriting shallow geometric representations rather than relying on conventional full-network fine-tuning.

  • Takeaways & Limitations

    The benefit of MAE transfer may vary across categories and datasets.

Abstract

from arXiv · show

Large-scale pre-training has transformed representation learning in 2D vision, yet its transferability to 3D single object tracking (SOT) remains insufficiently understood. Directly fine-tuning self-supervised 3D encoders, such as masked autoencoders (MAE), often leads to sub-optimal adaptation because the reconstruction objective is not fully aligned with the spatial-temporal matching requirements of tracking. In this paper, we observe that this difficulty can be interpreted as a layer-wise transfer mismatch: shallow layers tend to preserve transferable geometric cues, while deeper layers become increasingly specialized to the reconstruction pretext task and are less suitable for downstream tracking. Based on this observation, we propose MAETrack, a lightweight adaptation framework for transferring pre-training MAE representations to 3D SOT. MAETrack includes Layer-Selective Initialization (LSI), which initializes only the shallow stages of the tracking backbone from pre-trained weights while re-initializing deeper stages, and Geometric Residual Gating (GRG), which reinforces structurally salient regions in the search BEV features before template-search fusion through residual spatial modulation. Extensive experiments on standard 3D SOT benchmarks show that MAETrack consistently improves upon vanilla fine-tuning baselines with limited computational overhead. More broadly, our results suggest that effective transfer from 3D reconstruction pre-training to 3D tracking is not merely a matter of partial fine-tuning, but depends on a tracking-oriented transfer principle that preserves shallow geometry while adapting deeper representations to the downstream objective.

1. Introduction

3D SOT requires spatial-temporal reasoning under sparse and occluded observations, but MAE reconstruction representations can mismatch tracking needs. MAETrack selectively transfers shallow geometry and adapts search features, improving tracking over full-network fine-tuning.

  • 3D SOT must continuously localize an object under sparse, partial, and occluded observations, where target representations can degrade substantially.
  • MAE reconstruction favors dense geometric recovery, making deeper representations increasingly reconstruction-specific and less compatible with tracking-oriented spatial-temporal matching.
  • Shallow MAE layers preserve more transferable geometric and boundary cues, motivating selective transfer rather than indiscriminate full-network fine-tuning.
  • Layer-Selective Initialization inherits shallow BEV-MAE weights while re-initializing deeper layers to reduce reconstruction-specific semantic influence.
  • Geometric Residual Gating spatially modulates search BEV features before template-search fusion to reinforce structurally salient regions.
  • MAETrack consistently improves Success and Precision over conventional full-network fine-tuning baselines on standard 3D SOT benchmarks.

2. Related Work

3D SOT has progressed from Siamese matching toward motion-centric modeling, while self-supervised MAE methods have mainly been evaluated for static 3D perception. MAETrack targets the underexplored transfer of geometric pretraining into tracking.

  • Existing 3D SOT methods include Siamese matching, proposal generation, structural priors, transformers, memory banks, and motion-centric regression.
  • Appearance matching remains sensitive to limited discriminative texture in LiDAR point clouds, while tracking methods rely heavily on labeled tracking supervision.
  • MAE variants such as Point-MAE, Voxel-MAE, GeoMAE, BEV-MAE, and Multi-view MAE learn geometric representations through masked reconstruction.
  • Although MAE methods perform well in static classification and detection, their utility for cross-frame association and real-time 3D SOT remains largely unexplored.
  • MAETrack transfers shallow pretrained layers, applies residual spatial gating to search features, fuses them with template features, and predicts a 3D bounding box.

3. Methodology

MAETrack treats MAE-to-3D-SOT transfer as a layer-wise adaptation problem: shallow geometric features are preserved, while deeper reconstruction-specialized representations adapt to tracking. It combines Layer-Selective Initialization with Geometric Residual Gating to reinforce informative search regions before template-search fusion.

  • Problem Formulation: The tracking pipeline extracts shared BEV features from template and search point clouds, fuses them, and predicts target center and heading while retaining template box size.The tracking network predicts the target state frame by frame from template-search representations.
  • Motivation: MAETrack preserves transferable shallow geometry while allowing deeper representations to adapt to tracking-specific objectives.The framework addresses negative transfer from uniformly fine-tuning reconstruction-oriented features.
  • Layer-Selective Initialization: Layer-Selective Initialization inherits MAE weights for selected shallow stages, randomly initializes deeper stages, and keeps all parameters trainable.A binary transfer mask determines which stages inherit pretrained weights; deeper stages are decoupled from reconstruction-specific initialization.
  • Geometric Residual Gating: Geometric Residual Gating adaptively enhances structurally salient regions in search BEV features while preserving pretrained geometric consistency.The module targets informative regions such as boundaries and occupied areas before template-search fusion.
  • Framework Overview: MAETrack uses a unified tracking backbone with LSI followed by feature-level modulation, rather than uniformly inheriting parameters across the network.Its two complementary components partition transferable and task-specific representations along network depth.

4. Experiments

Experiments follow common 3D-SOT protocols on KITTI and nuScenes, using Success and Precision under one-pass evaluation. The setup applies category-specific crops, perturbation-based candidates, selective initialization, and trajectory-based inference.

  • Dataset and Metrics: Experiments evaluate MAETrack on KITTI and nuScenes using Success and Precision under one-pass evaluation.nuScenes provides 700 training and 150 validation sequences across 40K point-cloud frames, whereas KITTI has only 19 training and 2 validation sequences.
  • Dataset and Metrics: Each frame pair generates four candidates through reference-box perturbation, with synchronized horizontal flipping additionally used on nuScenes.Category-dependent crop ranges are used for cars, pedestrians, trucks, trailers, and buses.
  • Inference: During inference, the model regresses relative motion and applies it to the previous target box to locate the target in the current frame.Algorithm 1 presents the complete trajectory inference process.

4.1. Comparison with SOTA Method

MAETrack is evaluated against state-of-the-art 3D trackers on KITTI and nuScenes using Success and Precision, with additional analyses of speed and point-cloud sparsity. It improves tracking performance while retaining practical inference speed and showing particular benefits when geometric evidence is limited.

  • Results on nuScenes Dataset: MAETrack outperforms the strong P2P baseline across almost all nuScenes categories, including 66.05%/73.57% on Car.It improves Truck and Trailer by +4.38%/+5.16% and +3.87%/+5.55%, respectively, and Bus by +3.57%/+4.73%.
  • Running Speed: MAETrack runs at 84 FPS on an NVIDIA RTX 4090 despite GRG overhead, yielding a better accuracy-speed trade-off.Speed is measured as the average running time over all frames in the Car category.
  • Robustness to Point-cloud Sparsity: MAETrack improves P2P in most sparse point-cloud intervals on both KITTI and nuScenes, with gains concentrated in low-point regimes.On KITTI [0,10), it increases Success/Precision from 64.8/75.0 to 67.1/77.7; on nuScenes, it increases 63.3/71.4 to 64.3/71.7.
  • Robustness to Point-cloud Sparsity: The sparsity advantage narrows in dense intervals, while the [40,50) KITTI interval is an exception with lower performance and only 3 sequences and 80 frames.The authors characterize this interval as statistically less stable.

4.2. Ablation Study

Ablations show that LSI and GRG jointly improve tracking, while layer-wise diagnosis supports selectively transferring shallow geometric representations and adapting deeper ones. GRG performs best as residual modulation on the search branch before template-search fusion.

  • Overall Module Ablation: Combining LSI and GRG improves nuScenes Car from 64.61/71.98 to 66.05/73.57 and Pedestrian from 45.64/74.62 to 47.21/75.96.LSI and GRG individually improve the baseline, and their combination produces the strongest result.
  • Effect of Layer-Selective Initialization: Shallow-stage initialization transfers better than deep-stage initialization: Only Stage 1 reaches 65.39/72.90, whereas Only Stage 3 and Only Stage 4 fall to 63.65/71.23 and 63.14/69.60.Full-network initialization provides only limited gains over the baseline.
  • Effect of Layer-Selective Initialization: Cumulative shallow-stage initialization improves KITTI Car from 73.20/85.40 to 75.20/87.50, and different cumulative settings lead to complementary best results across nuScenes metrics.The default LSI setting also improves nuScenes Car and Pedestrian over full-network initialization.
  • Effect of Layer-Selective Initialization: CKA similarity declines from 0.84 at Stage 1 to 0.36 at Stage 4, while shallow stages also achieve stronger geometric probing results.These findings support the distinction between transferable shallow geometric priors and reconstruction-specialized deeper representations.
  • Effect of Geometric Residual Gating: Search-only GRG consistently performs best across KITTI and nuScenes, reaching 75.20/87.50 on KITTI Car and 66.05/73.57 on nuScenes Car.Applying GRG after fusion or to both branches is less effective, indicating that its benefit depends on search-side residual modulation before fusion.

4.3. Visualization results

Visualization results examine GRG gate maps and tracking boxes across dense and sparse nuScenes trajectories. MAETrack tracks targets more accurately than P2P across these conditions.

  • GRG Visualization: GRG gate maps range from 0.283 to 0.995, with peak values concentrated along ground-truth box boundaries.The corresponding activation changes are aligned with the target vicinity, indicating target-aware spatial modulation.
  • Tracking Visualization: MAETrack tracks targets across dense and sparse nuScenes scenarios, whereas P2P produces inaccurate box estimates.The comparison covers diverse trajectories and conditions.

5. Conclusion

MAETrack addresses a layer-wise mismatch between reconstructive 3D pre-training and downstream tracking by preserving shallow geometric knowledge while adapting deeper representations. On KITTI and nuScenes, it outperforms vanilla fine-tuning with negligible overhead, while remaining limited in pre-training sources, temporal modeling, backbone scope, and multi-object validation.

  • Conclusion: MAETrack targets a layer-wise objective mismatch in which shallow layers retain transferable geometric cues while deeper layers specialize in reconstruction and can impede tracking.The framework transfers foundational geometric weights from shallow stages and adapts deeper representations for tracking.
  • Conclusion: MAETrack combines Layer-Selective Initialization with Geometric Residual Gating to transfer shallow geometry and reinforce structurally salient regions through spatial modulation.LSI inherits only foundational geometric weights, while GRG modulates features before tracking fusion.
  • Conclusion: MAETrack significantly outperforms vanilla fine-tuning on KITTI and nuScenes, particularly for geometrically challenging categories, with negligible computational overhead.The reported benchmark scope includes KITTI and nuScenes.
  • Conclusion: The paper argues that hierarchical geometric priors, rather than only final pre-training outputs, can support tracking-oriented and efficient robotic perception systems.The proposed approach reconciles these priors with temporal matching.
  • Limitations and Future Work: Limitations include reliance on a single pre-trained source, absent explicit temporal modeling, focus on CNN backbones, and no multi-object tracking validation.Future work proposes multimodal or heterogeneous pre-training, stronger handling of fast motion or severe occlusion, transformer extensions, and multi-object tracking.

CrediT authorship contribution statement

The authorship statement assigns investigation, methodology, software, validation, writing, administration, formal analysis, visualization, funding, and related contributions across the listed authors.

  • CrediT authorship contribution statement: Sifan Zhou is credited with investigation, methodology, software, validation, writing, and project administration.
  • CrediT authorship contribution statement: Qiwei Wang contributed methodology, validation, visualization, and writing.
  • CrediT authorship contribution statement: Linyue Tan contributed software, formal analysis, validation, and writing, while Ziyu Liu contributed writing.
  • CrediT authorship contribution statement: Ziyu Zhao contributed writing and formal analysis, and Xiaobo Lu contributed methodology, writing, project administration, and funding acquisition.
Loading 2609.16695v1…