Source-linked AI summary
Motion- and Gravity-Aware Feedforward 3D Reconstruction
Shiyao Qian, Yuan Ren, Dongfeng Bai
TL;DR
Appearance-based feed-forward reconstruction can struggle with dynamic scenes and weak geometric cues. This paper trains transformer models with privileged residual-flow and gravity signals, using visual inputs alone at inference. It reports consistent gains in accuracy and robustness, with the preferred injection stage varying by backbone.
Problem
Appearance-driven feed-forward models lack explicit motion and orientation cues, while dynamic scenes and weak visual evidence can challenge geometric reasoning.
Source: §1 · p. 2, §2.1 · p. 4Method
The framework uses residual flow and gravity as privileged training signals under LUPI, transferring their geometric benefits to models that use standard visual inputs at inference.
Source: §1 · p. 2, §6 · p. 17Results
Experiments show consistent improvements in reconstruction accuracy and robustness, especially for dynamic scenes, ambiguous geometry, and challenging camera motion.
Source: §6 · p. 17Takeaways & Limitations
LUPI offers a practical way to introduce motion and physical priors into feed-forward reconstruction without sacrificing inference efficiency.
Source: §6 · p. 17The current formulation uses rigid optical-flow components; the authors identify streaming or temporally coherent settings with explicit sequential inputs and ego-motion estimates as a future direction.
Source: §5 · p. 16
Abstract
from arXiv · showhide
Feed-forward transformer models have recently achieved strong performance in multi-view 3D reconstruction, but they still rely primarily on appearance-based attention and often struggle in dynamic scenes with weak geometric cues. In this work, we introduce a motion- and gravity-aware framework under the Learning Using Privileged Information (LUPI) paradigm. During training, residual optical flow and gravity direction are provided as privileged signals to guide geometric representation learning; during inference, the model uses standard visual inputs only. We integrate these priors into transformer backbones with lightweight architectural changes, preserving feed-forward efficiency. Experiments on standard benchmarks show consistent gains in reconstruction accuracy and robustness, especially in scenes with dynamic objects, ambiguous structure, and challenging camera motion. Our analysis further shows that the optimal injection stage is backbone-dependent: middle-layer integration is generally preferable for VGGT, while late-stage integration is more effective for {\pi}3. These results suggest that LUPI offers a practical way to inject motion and physical priors into feed-forward 3D reconstruction without increasing test-time complexity.
1 Introduction
Feed-forward reconstruction relies heavily on appearance, leaving dynamic scenes and weak or repetitive visual evidence challenging. The paper introduces motion and gravity cues through LUPI and reports accuracy and robustness gains, particularly in challenging scenes.
1 Introduction
- Appearance-driven models can struggle to distinguish viewpoint change from scene dynamics, while missing orientation cues can leave geometry ambiguous or spatially inconsistent.These difficulties are especially relevant in dynamic scenes and when visual evidence is weak or repetitive.Source: §1 · p. 2
- During fine-tuning, the approach treats new modalities as privileged data and is evaluated on two architectures with distinct designs.The paper reports consistent reconstruction accuracy and robustness gains, especially with dynamic content, complex camera motion, and scale ambiguity.Source: §1 · pp. 2–3
- The contribution is framed as a motion- and gravity-aware augmentation for feed-forward transformer reconstruction, with experiments and ablations on standard multi-view benchmarks.The stated aim includes improved robustness in dynamic scenes.Source: §1 · p. 3
- The method adds optical flow for static–dynamic separation and gravity direction as a global orientation prior for spatial consistency.It uses these cues to augment feed-forward transformer reconstruction.Source: §1 · pp. 2–3
2 Related Work
The related work describes feed-forward reconstruction's shift toward transformer-based correspondence reasoning, while noting its limited use of explicit motion and physical priors. It also situates the paper's use of LUPI as training-only guidance for geometric interpretation.
2.1 Feed-forward 3D reconstruction
- Feed-forward systems have moved from explicit cost volumes toward transformer-based correspondence learning, with VGGT, π3, and mapping-oriented systems illustrating the range of approaches.The section also discusses single-view depth pretraining and models that unify matching with geometry prediction.Source: §2.1 · pp. 3–4
- Existing feed-forward methods generally lack explicit motion and physical priors, motivating this work's lightweight injection of optical-flow and gravity cues.The cited gap is consequential for dynamic scenes and ambiguous textures, where visual-token reasoning can be brittle.Source: §2.1 · p. 4
2.2 Learning Using Privileged Information (LUPI)
- LUPI provides supplemental teacher guidance during training that is withheld at inference, and the paper applies this paradigm to motion and orientation priors in transformer-based 3D models.The related-work discussion places LUPI as a generalized mechanism for knowledge transfer across domains.Source: §2.2 · p. 4
3 Methodology
The method estimates residual flow and gravity as privileged cues, injects their encoded features into transformer representations, and fine-tunes under LUPI so visual inputs alone are used at inference.
3.1.1 Optical flow estimation and decomposition
- Residual flow is computed by subtracting a global affine approximation of camera-induced motion from optical flow, with deviations marking motion or structure the model may need to reason about.The affine model may leave residuals even in static scenes, particularly around depth variation, discontinuities, or occlusions.Source: §3.1.1 · p. 5
3.1.2 Gravity direction estimation
- Gravity directions are estimated offline for training images, retaining only estimates whose optimization converges.GeoCalib recovers gravity from a dense perspective field, and failed estimates are excluded from supervision.Source: §3.1.2 · p. 6
3.2 Privileged Information Injection
- Encoded privileged cues are injected at different aggregator stages: residual-flow features join patch tokens, while gravity embeddings join camera tokens where available.The preferred injection position depends on the backbone; gravity features are designed to stabilize camera estimation and cross-view geometric consistency.Source: §3.2 · pp. 7–8
3.3 Finetuning
- Fine-tuning uses two stages: first align privileged encoders and downstream decoders, then unfreeze the model and use modality dropout to train it to infer missing privileged features.During dropout iterations, the visual encoder acts as the student learning motion- or gravity-related features from visual inputs.Source: §3.3 · p. 9
4 Experiments
The experiments use a 13-dataset train/benchmark split and tailor privileged-feature injection and fine-tuning schedules to the two backbones.
4.1 Datasets
- 13 datasets are split into 11 training datasets and 2 benchmarking datasets.Flow maps are computed for 6 datasets with continuous scene frames; sampling per epoch varies by dataset.Source: §4.1 · pp. 9–10
4.2 Implementation Details
- VGGT receives privileged information at aggregator layer 12, while π3 receives gravity and residual-flow features before their respective decoders.For π3, gravity is injected before the camera decoder and residual flow before the point and confidence decoders.Source: §4.2 · p. 10
- Single-modality fine-tuning uses 2 alignment epochs followed by 8 distillation epochs; dual-modality training adds gravity after flow in a sequential 5-plus-5-epoch schedule.Stage 2 uses scene-wise modality dropout of 0.5; the two privileged-feature weights are fixed at 1.0 for the first two epochs.Source: §4.2 · pp. 10–11
4.3 Dense Multi-View Evaluation
Dense multi-view evaluation finds gains from motion and gravity priors across both backbones, with the combined priors outperforming either alone. The advantage over baseline grows with input-view count.
4.3 Dense Multi-View Evaluation
- The performance gap over baseline widens as input-view count increases, consistent with stronger gains in dense settings.The authors attribute this trend to improved cross-view geometric aggregation amid motion- and orientation-related inconsistencies.Source: §4.3 · p. 11
- Improvements across π3 and VGGT indicate that the framework generalizes across feed-forward reconstruction backbones.The passage describes the backbones as requiring no specialized architectural modifications.Source: §4.3 · p. 11
- Combined residual-flow and gravity injection outperforms either single prior, while residual flow alone consistently beats baselines across key metrics on both backbones.The reported metrics include depth relative error, ray alignment, and inlier-based statistics.Source: §4.3 · p. 11, Figure 3 · p. 12, Figure 4 · p. 12
4.4 Ablation Studies
The ablations favor residual over full optical flow, show that the best injection stage depends on the backbone, and find more stable global structure with gravity injection.
4.4.1 Residual vs. Full Flow Injection
- Residual-flow injection achieves the best performance on both backbones, whereas full optical flow is comparable to baseline and slightly degrades some metrics.The authors suggest residual flow is more useful because it removes camera-induced rigid motion and isolates motion deviations associated with independently moving objects or challenging regions.Source: §4.4.1 · p. 13
4.4.2 Layer-wise Injection Analysis
- Middle-layer injection performs best on most VGGT metrics, late-layer injection is optimal for π3, and early injection performs worst across cases.The authors relate the preference to geometry-aware tokens and later refinement in VGGT, head-specific isolation in π3, and insufficient cross-view context at the input stage.Source: §4.4.2 · pp. 13–14, Table 3 · p. 14
4.4.3 Gravity Prior Analysis
- With gravity injection, reconstructions show flatter ground surfaces and more vertically aligned walls, suggesting improved global orientation rather than altered local geometry.The figure pairs show reconstructions without gravity above those with gravity.Source: §4.4.3 · p. 15, Figure 5 · p. 15
4.5 Analysis
The analysis links stronger gains at higher view density to cross-view consistency, identifies backbone-dependent injection preferences, and describes flow and gravity as complementary priors.
4.5 Analysis
- As view density grows, dynamic regions introduce stronger multi-view conflicts, which residual flow helps suppress.The analysis says the increased gains reflect improved cross-view geometric aggregation rather than compensation for sparse observations.Source: §4.5 · pp. 15–16
- Late-stage injection works best for π3, middle-layer injection is generally more effective for VGGT, and early-stage injection is weaker in both.The proposed explanation is that shallow appearance-dominant features are less suitable for privileged geometric guidance.Source: §4.5 · p. 16
- Residual flow improves local dynamic consistency while gravity anchors global orientation; together they improve geometric coherence without changing feed-forward inference.The analysis assigns the priors complementary structural scales.Source: §4.5 · p. 16
5 Limitations
The formulation includes rigid optical-flow components, but unordered image inputs without temporal information leave those components with limited benefit in the experiments. The authors propose testing rigid-motion cues in temporally coherent settings as future work.
5 Limitations
- Without temporal ordering or timestamps, rigid flow provides no consistent camera-dynamics signal across views and offers limited reconstruction benefit in the experiments.The authors note that rigid flow can encode camera velocity and angular motion, but this information is not consistent across unordered inputs.Source: §5 · p. 16
- Future work could model sequential inputs and ego-motion estimates explicitly to incorporate rigid-motion cues in temporally coherent reconstruction.The authors suggest this may further improve robustness and support more principled dynamic-scene reconstruction.Source: §5 · pp. 16–17
6 Conclusion
The framework trains feed-forward reconstruction models with residual-flow and gravity cues as privileged signals, while retaining standard visual inputs at inference. Experiments report consistent accuracy and robustness gains, and conclude that LUPI can add motion and physical priors without sacrificing inference efficiency.
6 Conclusion
- Residual flow and gravity cues guide training as privileged signals, while inference uses standard visual inputs only.The framework is presented under the Learning Using Privileged Information paradigm.Source: §6 · p. 17
- Experiments show consistent accuracy and robustness improvements in dynamic scenes, ambiguous geometry, and challenging camera motion.The optimal injection stage also varies by architecture.Source: §6 · p. 17
- The authors conclude that LUPI introduces motion and physical priors without sacrificing feed-forward inference efficiency.They identify temporally coherent and physically grounded reconstruction with richer privileged training signals as a direction for future work.Source: §6 · p. 17
A Finetuning Details
Fine-tuning uses curated, spatially overlapping image sequences and geometric supervision, with losses adapted to model architecture and dataset conditions. The appendix also specifies optimizer settings and learning-rate scheduling.
A Finetuning Details
- Input sequences retain at least 25% co-visibility with one other view, and training targets include point maps, ray directions, camera poses, and per-view confidence.Normal and depth-gradient supervision is added for synthetic datasets.Source: A Finetuning Details · p. 23
- The objective combines robust losses for point maps and z-depth, ray directions, and camera pose or relative pose, with normal and gradient losses when smooth synthetic ground truth is available.Confidence weighting and exclusion of the top 5% of errors address noisy real-world data; π3 uses pairwise relative pose losses.Source: A Finetuning Details · p. 23
- The listed loss configurations differ by model and data type, including additional normal and gradient terms for synthetic data.The appendix gives separate VGGT and π3 equations for the synthetic and real-data configurations.Source: A Finetuning Details · p. 23
- Lpt, Lz, and Lray denote robust regression losses for point maps, z-depth, and ray directions, while Lpose is VGGT's global camera pose loss.Lrel is the pairwise relative pose loss for reference-view-free π3; Wconf weights global point-map predictions, and Mtop5% masks the highest 5% of errors.Source: A Finetuning Details · pp. 23–24
- AdamW training uses differentiated peak learning rates, a 20% linear warmup, and half-cycle cosine decay to a final rate two orders of magnitude lower.Peak rates are 5×10^-6 for the DINOv2 image encoder, 5×10^-5 for gravity and flow encoders, and 1×10^-5 for other components.Source: A Finetuning Details · p. 24
B Additional Ablation Results
In π3 fine-tuning, residual-flow injection with 10% of the training data is reported to outperform increasing the dataset size tenfold, with similar results for VGGT. Additional ablations compare injection locations across both backbones, while a dataset-scale study reports improved π3 performance with more data.
B Additional Ablation Results
- Residual-flow injection with 10% of the training data yields higher benchmarking performance than increasing dataset size tenfold for π3, with similar results for VGGT.Figures 7 and 8 compare these fine-tuning settings for π3 and VGGT, respectively.Source: B Additional Ablation Results · p. 24, Figure 7 · p. 25, Figure 8 · p. 26
- π3 performance improves consistently as effective dataset size increases during fine-tuning.Figure 6 compares effective dataset scales while holding total training steps constant.Source: B Additional Ablation Results · p. 24, Figure 6 · p. 25
- For π3, residual-flow injection locations are compared with the default and with placements before or within the aggregator's middle layers; gravity placements are also compared with the default and middle-layer alternatives.The cited figures report benchmarking across varying view counts, but the supplied passages do not state which location performs best.Source: B Additional Ablation Results · pp. 24, 26, Figure 9 · p. 26, Figure 10 · p. 27
- For VGGT, residual-flow and gravity ablations compare the default injection point with placements before or after the aggregator module.Figures 11 and 12 cover residual flow and gravity, respectively.Source: Figure 11 · p. 27, Figure 12 · p. 28