ProMo3D: Probing Motion Cues in Frozen 3D Foundation Models

1 Introduction

• Existing approaches either require substantial annotation and computation or use efficient but brittle, architecture-specific heuristics that underperform learned motion decoders.The challenge is disentangling static geometry, camera-induced motion, and independently moving objects.Source: §1 · p. 1
• PMD converts anchor-based pairwise motion-state compatibility scores into dense dynamic score maps and coarse masks, optionally refined with SAM2.The framework uses a lightweight probe trained on 160 videos.Source: Figure 1 · p. 2, §1 · p. 2
• The authors investigate whether low-capacity probes can recover motion cues from frozen 3D tokens without complex decoders or expensive fine-tuning.Prior findings suggest dynamic-scene robustness and geometric constraints in these representations may provide useful cues.Source: §1 · p. 2
• Probing across π3X, Depth Anything 3, and DUSt3R localizes motion cues in frozen backbones without architecture-specific heuristics.Rank-restricted probing and spectral analysis on π3X find a compact motion subspace, with a low-rank quadratic probe sufficient to extract dense masks.Source: §1 · p. 2
• PMD transfers zero-shot to three motion-segmentation benchmarks, outperforming training-free heuristics and remaining competitive with supervised and self-supervised methods.It is trained on 160 videos using a single 16 GB GPU, without target-benchmark fine-tuning.Source: Figure 1 · p. 2, §1 · p. 3

2.1 Feed-forward 3D Foundation Models

• Feed-forward 3D models predict geometry and camera parameters from unposed images, while newer extensions broaden the paradigm to dense geometry, Gaussian reconstruction, and ray prediction.Their training on diverse geometric data, including dynamic sources, offers a potential substrate for motion cues.Source: §2.1 · p. 3

2.2 Dynamic Scene Reconstruction and Motion Segmentation

• Dynamic-scene methods address geometry, camera motion, and object motion through scene-specific reconstruction, feed-forward 4D prediction, or motion segmentation.Motion segmentation draws on motion–appearance cues, optical flow, object-centric grouping, or appearance refinement.Source: §2.2 · p. 3
• Training-free frozen-model methods use brittle, architecture-specific heuristics, whereas learned motion decoders can require high capacity and additional optical-flow cues.The paper targets the gap between these regimes.Source: §2.2 · p. 3

2.3 Representation Probing and Latent Analysis

• Representation probing tests whether information is decodable from neural representations, and prior work applies it to physical quantities, geometric priors, and object binding.The paper applies low-capacity probes to test for a static–dynamic discriminative subspace in frozen 3D representations.Source: §2.3 · p. 3

3 Method

• Token features come from a frozen 3D foundation model at a selected block, with temporal or multi-view context and token pairs sampled within the target frame.Dynamic masks are projected onto the token grid to assign binary labels, with 1 denoting dynamic and 0 static.Source: §3 · p. 3
• Pairwise probing labels a token pair as positive when both tokens share the same motion state and negative otherwise.The probe predicts pairwise compatibility and is trained on sampled pairs while the backbone remains frozen.Source: §3 · p. 4

3.1 Pairwise Probe Functions

• The probe functions model pairwise compatibility using independent linear terms, channel-wise multiplicative interactions, or cross-channel bilinear interactions.The primary low-rank bilinear form is symmetrized, matching the symmetric pair label and enforcing order invariance; its rank is at most 2k.Source: §3.1 · p. 4

3.2 Probing-based Motion Distillation

• PMD turns pairwise compatibility into a dense estimator by querying the trained probe between each token and a fixed-index anchor.The first token is the anchor in the experiments; for padded backbones it lies in the padded region and is static by construction.Source: §3.2 · p. 4
• PMD converts anchor–token compatibility into token-level dynamic scores, then reshapes and bilinearly upsamples them to image resolution.All anchor–target scores can be evaluated in parallel, adding only lightweight computation to frozen-backbone inference.Source: §3.2 · p. 4

4.1 Experimental Setup

• Probes train on 160 YouTube-VOS videos, validate on 80, and are evaluated using J-M and J-R on DAVIS, SegTrackv2, and FBMS-59.No evaluation-benchmark data is used for training or validation; targets are annotation-defined and need not match instantaneous physical motion.Source: §4.1 · p. 5

4.2 Layer-wise Motion Cue Emergence

• Low-rank quadratic probes outperform linear and diagonal probes across three backbones, with motion cues also appearing in intra-frame blocks and prediction heads.Cue quality varies across backbones, and subsequent experiments average a fixed backbone-specific block group for every target sequence and dataset.Source: §4.2 · p. 5, Figure 2 · p. 6

4.3 Zero-shot Cross-Benchmark Motion Segmentation

• PMD generalizes zero-shot from YouTube-VOS to all three test benchmarks; π3/π3X-PMD surpass Easi3R and VGGT4D while remaining competitive with supervised methods.Qualitative examples also show cleaner score maps and more coherent coarse masks for semantic categories absent from the 160-video training subset.Source: Table 2 · p. 6, §4.3 · p. 6, Figure 3 · p. 7
• On DAVIS17, coarse π3X-PMD reaches 73.7 J-M and 91.0 J-R versus GeoMotion’s 75.4 J-M and 87.2 J-R; SAM2 refinement raises it to 81.9 J-M and 92.3 J-R.The reported J-R gains are 1.2/1.3 points for π3/π3X-PMD, compared with 5.4 for GeoMotion and 9.9–14.8 for Easi3R.Source: §4.3 · pp. 6–7, Table 3 · p. 7
• π3X-PMD adds 4.2 ms/frame (0.87%) to frozen-backbone inference on an RTX 5070 Ti, using a 0.295M-parameter probe trained on 160 videos.Its scoring and aggregation add 1.62G FLOPs per frame; these figures exclude frozen-backbone inference from the added FLOPs.Source: §4.3 · p. 7, Table 4 · p. 8

4.4 Compact Static–Dynamic Discriminative Subspace and Parsimonious Decoding

• With fewer training videos, normalized gain and downstream mask quality degrade gradually, while the rank-8 probe matches or exceeds the full-rank probe.The mask trend holds both before and after SAM2 refinement.Source: §4.4 · p. 8, Figure 5 · p. 9
• PCA shows static–dynamic token separation, while steep spectral decay and low-rank performance saturation indicate compact probe-induced motion cues.The concentration is especially pronounced in earlier layers.Source: Figure 4 · p. 8, §4.4 · p. 8
• Shuffling motion labels within frame groups reduces pair accuracy from 95.72% to 84.48% and balanced accuracy from 91.67% to 50.34%.The return toward baseline indicates that probe gains rely on correct token–label correspondence.Source: §4.4 · pp. 8–9, Table 5 · p. 9

5 Limitations

• ViT token granularity limits spatial precision, and binary static–dynamic masks do not distinguish articulation, local deformation, or object displacement.Static instances can also receive elevated dynamic scores when another instance of the same category is moving; object-centric supervision and frozen semantic features may contribute.Source: §5 · p. 9, Figure 6 · p. 10

• PMD trained on 160 YouTube-VOS videos transfers zero-shot to DAVIS, SegTrackv2, and FBMS-59 while remaining competitive with substantially heavier methods.Source: §6 · p. 10
• Lightweight low-rank quadratic probes expose static–dynamic cues in frozen tokens from π3X, DA3, and DUSt3R without high-capacity motion decoders.Source: §6 · p. 10
• For π3X, rank-restricted and spectral analyses find the probe-decoded motion cues concentrated in a compact low-dimensional discriminative subspace.Source: §6 · p. 10
• The results motivate treating motion understanding in multi-view pretrained 3D models as lightweight decoding of frozen tokens rather than re-learning motion-specific features through large-scale training.Source: §6 · p. 10

A Implementation Details

• Experiments use one 16 GB NVIDIA GeForce RTX 5070 Ti GPU, with probe training, mask extraction, and SAM2 refinement run on one GPU at a time.No distributed training is used.Source: A Implementation Details · p. 14
• Probes train for 16 epochs on 160 YouTube-VOS videos, using 200K within-frame token pairs per epoch and validation on 80 videos.DAVIS, SegTrackv2, and FBMS-59 videos or annotations are excluded from probe-parameter optimization.Source: A Implementation Details · p. 14

B PMD Inference Algorithm

• At inference, three-frame groups with stride 2 provide backbone context, while scores and masks are decoded for the target frame.Token scores are averaged across blocks, reshaped and upsampled; coarse masks use Otsu thresholding with offset 0.2.Source: A Implementation Details · p. 14, B PMD Inference Algorithm · p. 15
• Optional PMD+SAM2 refinement erodes the prompt mask twice, filters components by mean dynamic score, then uses surviving components as SAM2 prompts.The refinement operations are optional; they are not part of coarse PMD.Source: A Implementation Details · p. 14, B PMD Inference Algorithm · p. 15

A Implementation Details

• 73.0 versus 72.4 mean J-M: Otsu+0.1 exceeds the retained Otsu+0.2 default, while tested threshold-rule variation remains limited.Across-rule standard deviation of mean J-M is 0.69 for fixed thresholds 0.40–0.60 and 0.97 for Otsu offsets 0.0–0.2; extreme thresholds reduce performance.Source: Table 6 · p. 14, A Implementation Details · p. 14

C.1 Fixed Reference-Token Convention

• The fixed a = 0 reference is never adapted per image or benchmark; a stress test finds that a fully dynamic reference moderately raises background response without overturning foreground recovery.The tested non-padded backbones are DA3 and DUSt3R; for padded π3/π3X variants, the reference lies in the padded region and is static by construction.Source: §C.1 · p. 15, Figure 7 · p. 16
• Without an independent reference, all-static and uniformly moving scenes share a no-independent-foreground interpretation; no dedicated all-static camera-motion evaluation is reported.Source: §C.1 · p. 15

D Capacity Analysis of Low-Rank Quadratic Probes

• The appendix interprets PMD’s limited-video generalization through the capacity of its low-rank pairwise probe; symmetrization gives a bilinear matrix of rank at most 2k.The symmetric form matches the symmetric pair label and enforces order invariance.Source: D Capacity Analysis of Low-Rank… · p. 16, §D.1 · p. 17

C.2 Token-Level Class Imbalance.

• Static tokens dominate across the evaluation benchmarks; the reported imbalance statistics contextualize the all-static baseline and normalized-gain metric, not anchor selection or training.Source: §C.2 · p. 16, Table 8 · p. 17

D.2 Capacity Scaling

• Validation on 80 YouTube-VOS sequences selects k = 8 as the default; k = 4 recovers most attainable performance, while k = 8 gives the best tested Accuracy and Normalized Gain.The DAVIS-all rank sweep is diagnostic only and is not used to select k.Source: Table 9 · p. 17, §D.2 · p. 17, Table 10 · p. 18

D.3 Shuffled-label Selectivity Control

• Shuffled motion labels bring the rank-8 π3X probe close to the always-same baseline and chance balanced accuracy, while correctly labeled probes retain gains above 11 percentage points at both capacities.This supports dependence on correct token–motion-state correspondence, but does not distinguish physical motion from correlated semantic or objectness information.Source: §D.3 · p. 18

D.4 Sample-Complexity Intuition

• With about 3.2 × 10^6 highly correlated sampled pairs, the full-rank probe’s approximately 1.88 × 10^7 parameters exceed the sample count and may be prone to overfitting.The comparison is a capacity-based intuition under correlated, video-derived data, not a formal generalization bound.Source: §D.4 · p. 18
• At k = 8, PMD has approximately 2.95 × 10^5 parameters, and its O(kd) scaling provides a capacity-based explanation for efficient generalization from limited video diversity.Source: §D.4 · p. 18

D.5 Comparison to GeoMotion Decoder Capacity

• Even at k = 256, PMD is nearly an order of magnitude smaller than GeoMotion’s trainable motion head; for k ≤ 16, the parameter gap exceeds two orders of magnitude.The comparison is about capacity and parameter counts, not an architecture-independent sample-complexity bound.Source: §D.5 · p. 19, Table 11 · p. 19

E Computational Efficiency and Inference Cost

• Table 12 reports a 0.295M-parameter rank-8 probe adding 1.62G FLOPs per original frame, while GeoMotion uses an 80.3M-parameter motion branch.The comparison concerns trainable capacity and added inference cost, not universal full-pipeline efficiency.Source: Table 12 · p. 20, E Computational Efficiency and… · p. 20
• Coarse PMD adds 4.2 ms/frame (0.87%) to π3X inference, while optional SAM2 adds 468.0 ms/frame and dominates the additional latency.The end-to-end measurements use released pipelines on one RTX 5070 Ti, with native settings rather than equalized resolutions or module costs.Source: E Computational Efficiency and… · p. 20, Table 13 · p. 20
• PMD is not universally faster or lower-memory: GeoMotion has lower total latency in both configurations, and SegAnyMo has lower measured peak VRAM.The efficiency claim is limited to PMD’s small added readout cost and trainable capacity.Source: E Computational Efficiency and… · p. 21

F Broader Impacts

• The paper identifies potential accessibility benefits in resource-constrained settings and warns that motion tracking could be repurposed for privacy-invasive surveillance.It advocates privacy-preserving safeguards for deployment.Source: F Broader Impacts · p. 21

G.1 Additional Layer-wise Probing Results

• Across VGGT, π3, and MapAnything, low-rank quadratic probes expose stronger motion decodability in geometric blocks than in pose or camera heads.VGGT’s camera head approaches the all-static baseline; π3 retains decodable cues there, but its geometric blocks are stronger.Source: §G.1 · p. 21, Figure 8 · p. 21

G.2 Additional Backbone Benchmarking

• Across additional backbones, PMD is not specific to one architecture; π3 and π3X combine strong coarse masks with small SAM2 J-Recall gains but larger J-Mean improvements.The table reports J-Mean/J-Recall and defines ΔJ-R as the change in the fraction of frames exceeding 0.5 Jaccard, not a measure of mask extent or boundaries.Source: §G.2 · p. 22, Table 14 · p. 22

G.3 Backbone Disparity and Source-validation Decodability

• 10.59 points higher mean source-validation NG, 4.54 points lower across-block standard deviation, and 4.77 points higher mean target J-M distinguish π3X from DA3.The reported target J-M is averaged across DAVIS17, SegTrackv2, and FBMS-59; the DAVIS17 coarse-mask gap is 4.1 J-M and remains 4.2 points after SAM2.Source: §G.3 · p. 22, Table 15 · p. 23
• Source-validation pairwise NG is positively associated with coarse target J-M, including within the five multi-view backbones.These descriptive associations do not isolate a causal architectural driver because architecture, pretraining data, token flow, and prediction heads vary together.Source: §G.3 · pp. 22–23, Table 17 · p. 23

G.4 Additional Feed-forward Geometry-based Comparisons

• The feed-forward geometry-based comparison separates coarse-mask quality from SAM2’s effect, reporting J-Mean, J-Recall, and ΔJ-R before and after refinement.A small ΔJ-R for methods with high coarse J-R means few additional frames cross the 0.5 Jaccard threshold, not necessarily little improvement in mask quality.Source: §G.4 · p. 23

G.5 Additional Scene-Diversity Starvation Results

• Across π3X, DA3, and DUSt3R, normalized gain degrades gradually as training videos are reduced; π3X has the highest gain and degrades most slowly.Uncertainty remains modest overall and widens mainly in the most data-starved regimes.Source: §G.5 · pp. 23–24, Figure 9 · p. 24

H Additional Qualitative Results

• The qualitative section examines how network depth and scene-diversity starvation affect motion–semantic entanglement, alongside failure cases and limitations.Source: H Additional Qualitative Results · p. 24

G.6 Extended Benchmark Results

• The extended benchmark includes DAVIS16 and DAVIS-all and marks refinement and target-domain training explicitly; PMD uses 160 YouTube-VOS videos and transfers across three benchmarks.The table notes that non-PMD scores come from cited papers and that complete-pipeline comparisons differ in supervision and inference priors.Source: §G.6 · p. 24, Table 19 · p. 25

H.1 Visualization of the Effect of Network Depth and Scene-Diversity Starvation

• With 160 training videos, π3X-PMD highlights moving targets in shallow and intermediate layers, while deeper layers entangle motion with semantics; with 10 videos, static foregrounds receive high dynamic scores.The deeper-layer entanglement is consistent with increased effective rank required for decoding.Source: §H.1 · p. 25

H.2 Failure Cases and Limitation Analysis

• PMD produces binary static–dynamic masks rather than distinguishing motion types, magnitudes, localized deformation, in-place articulation, or explicit 3D displacement and velocity.In the Figure 11 example, the breakdancer receives the strongest scores while nearby people with subtler motion receive moderately elevated scores.Source: §H.2 · p. 26, Figure 11 · p. 26
• Static instances can receive elevated dynamic scores when another instance of the same category moves, including categories absent from training; object-centric supervision may contribute, but the sources were not isolated.Figure 10 shows that training on 10 rather than 160 videos exacerbates semantic leakage across layers.Source: §H.2 · p. 26, Figure 10 · p. 26
• Evaluation is limited to frozen feed-forward 3D reconstruction backbones; matched comparisons with video segmentation and tracking models, 3D-pretraining specificity, and training beyond 160 videos remain unresolved.The authors leave performance saturation, an optimal data budget, and the value of additional source diversity open.Source: §H.2 · p. 26

I Existing Assets, Licenses, and Terms of Use

• The paper reports that its pretrained models, tools, datasets, and benchmarks were obtained from official sources and used under their respective licenses and terms.FBMS-59 is used only for research evaluation under custom research-only terms that prohibit commercial use.Source: I Existing Assets, Licenses, and Terms… · pp. 26–28

2. Limitations

• The main benchmark tables lack confidence intervals, statistical tests, or error bars for all methods and backbones; the data-starvation uncertainty analysis covers only experiments with multiple subset seeds.Appendix G reports mean and one-standard-deviation bands for those data-starvation experiments.Source: §2 · p. 31

10. Broader impacts

• The paper identifies unauthorized surveillance as a potential misuse of motion segmentation and says no new high-risk model, image generator, or dataset requiring controlled-release safeguards is released.The paper also discusses potential positive societal impacts, including accessibility and computational efficiency for dynamic scene understanding.Source: §10 · pp. 32–33
• The supplementary release includes anonymized code and sampled split files with README instructions for reproducing experiments, but no new dataset or benchmark.The README describes directory structure, dependencies, dataset preparation, and reproduction commands.Source: §10 · p. 33
• The paper reports no crowdsourcing or human-subject research, making participant instructions, compensation details, and IRB approval inapplicable.The authors state that the work does not involve crowdsourcing or human-subject research.Source: §10 · p. 34
• LLMs were used only for writing refinement, formatting, visualization, and file-I/O assistance, not for core methods, experiments, data, analysis, or scientific claims.The authors therefore mark the LLM declaration as not applicable under the checklist guidance.Source: §10 · pp. 34–35