VI3: Grounding Pretrained 3D Foundation Models with Inertial Cues

• 3DFMs predict camera poses and dense per-pixel geometry from unposed images in a feed-forward pass, but remain limited to unknown global scale.The paper contrasts these models’ dense geometric priors with traditional feature-based pipelines that produce sparse geometry and can be brittle in unconstrained environments.Source: §1 · p. 1
• Because monocular 3DFMs use scale-invariant supervision, converting their scale-free predictions into physically anchored metric reconstructions remains an open gap.Existing metric approaches may be overridden by data-driven priors and can depend on precise camera calibration or upstream metric poses.Source: §1 · p. 2
• IMUs provide metric readings that make real-world scale observable and can complement monocular vision for metric grounding.The passage cites visual-inertial research [6] [12] as supporting the sufficiency of inertial sensing to metrically ground vision.Source: §1 · p. 2
• VI3 initializes and preintegrates IMU readings into a gravity-aware motion reference, then injects scale through compatible input conditioning or test-time refinement.The method is presented as model-agnostic and requires no ground truth, optimization backend, map, or external finetuning corpus.Source: §1 · p. 2
• Across diverse aerial datasets and representative 3DFM backbones, VI3 metrically grounds frozen models from coarse inertial motion estimates, producing scales close to real-world scale.The authors describe this as their experimental finding and identify IMU-only physical anchoring as a contribution.Source: §1 · p. 2

• Feed-forward 3DFMs predict camera poses and dense geometry from unposed images, but their SfM-trained outputs retain unknown global scale.DUSt3R [50] and MASt3R [24] predict pointmaps from image pairs; VGGT [47] and π3 jointly estimate attributes across multiple views.Source: §2 · p. 2
• Scale-aware models inject metric information through attention, auxiliary heads, or scale tokens, but their learned priors inherit training distributions or depend on dedicated metric-input frontends.The passage names AMB3R, MapAnything [21], and other approaches, while distinguishing VI3’s use of inertial signals to condition or refine outputs.Source: §2 · p. 2
• Classical visual-inertial systems estimate metric gravity-aligned trajectories but yield sparse reconstructions and use stateful backends, while recent learned-3D systems rely on external inertial optimization.MASt3R-Fusion uses a factor-graph backend and VIGS-SLAM [57] optimizes a per-scene 3D Gaussian representation; VI3 instead uses inertial cues as a lightweight anchor.Source: §2 · p. 2
• Recent work [8] [17] [49] [55] focuses on refinement for long-sequence stability and affordability; we propose refinement to adapt 3DFM weights by injecting metric scale into short monocular video, anchoring geometry to a physical reference.Source: §2 · p. 3

• A 3DFM M takes n consecutive unposed frames and predicts geometric outputs that typically include pixel-wise depth and camera extrinsics.Source: §3 · p. 3
• The setup assumes pre-calibrated camera intrinsics, although 3DFMs may also predict them.Source: §3 · p. 3
• Per-image point clouds can be computed from depth maps and camera intrinsics.Source: §3 · p. 3

4.1. Preintegration and state propagation

• Preintegration between consecutive camera frames yields relative rotation, relative position, and velocity change from bias- and noise-corrected IMU readings.The method assumes constant biases over short visual-inertial clips; the inertial states are recoverable given the biases, gravity, and first-frame velocity.Source: §4.1 · p. 3, §4 · p. 3

4.2. Inertial state initialization

• The initialization estimates gyro bias from angular-velocity measurements and 3DFM relative rotations, and estimates scale from stacked velocity-increment constraints using total least squares.It assumes negligible accelerometer bias for short integrations.Source: §4.2 · p. 4
• The initial velocity is refined by aligning its orientation with the gravity-free first pair and taking its magnitude from the median of frame back-projections.Sanity checks reset scale to 1 for anti-aligned visual and inertial motion and revert velocity if the estimated magnitude is nonpositive.Source: §4.2 · p. 4
• Gravity initialization combines GeoCalib image cues with inertial averaging and a kinematic least-squares estimate, whose scale and gravity are coupled through fixed-point iteration.The iteration starts from an image-based or inertial gravity seed and converges within two to three iterations.Source: §4.2 · p. 4
• ∼3–7× under-scaled Base models score ≈0 in δ<1.25, while GT-free TTR restores metric depth and pulls trajectory and depth scale toward 1.Rotation metrics also improve substantially.Source: Table 1 · p. 5
• When motion is sufficient, image-based and inertial gravity seeds converge within approximately 1°; if GeoCalib is rejected as an outlier, initialization adopts the purely inertial estimate.The fallback uses its corresponding scale and initial velocity.Source: §4.2 · p. 5

5. Metrically anchoring 3DFMs

• VI3 offers input conditioning for models that accept metric inputs and test-time refinement that jointly adjusts scale and model parameters to align the metric output with the inertial trajectory.The scale is applied to translations, leaving rotation and intrinsics untouched.Source: §5 · p. 5
• Pose supervision is applied to consecutive frames and is globally invariant because shared rigid transforms of prediction and target cancel.Source: §5 · p. 5
• The refinement couples depth maps and camera poses with a self-supervised cross-view consistency loss that anchors predicted depths to the inertial-recovered pose scale.The loss compares reprojected depth with source depth at valid reprojected coordinates.Source: §5 · pp. 5–6
• Two regularizers preserve structural integrity by anchoring the first-frame position and depth to their initial predictions.The refinement adapts to the input stream's photometric distribution while keeping geometry bounded by the 3DFM structural prior.Source: §5 · p. 6

6. Experiments

• The evaluation spans synthetic TartanAirV2, EuRoC-MAV, and UZH-FPV, covering noise-free or high-parallax through highly dynamic, low-parallax motion.TartanAirV2 appears in the training corpus of all evaluated backbones except VGGT.Source: §6 · p. 6

6.1. Experimental setup

• The study evaluates five released pretrained backbones on sampled eight-frame clips, reports dataset-level medians, and measures pose, scale, and dense depth without trajectory alignment.Re-runs on fixed hardware vary by less than 0.01 m, 0.04°, and 0.03 in scale.Source: §6.1 · pp. 6–7

6.2. Results

• The reported refinement reaches scale ≈1 in all cases, propagates that scale to translation, and its cross-view loss preserves geometry without corrupting 3D awareness, while rotation improves consistently.Source: §6.2 · p. 8
• For metric-aware models, inertial conditioning or test-time refinement improves results in out-of-distribution cases, while refinement can help when learned priors override metric inputs.The paper also reports consistent rotation improvement.Source: §6.2 · p. 8

6.3. Additional analyses

• Initialization sensitivity is concentrated in initial velocity: direction and especially magnitude errors degrade scale recovery, while bias and gravity noise have less effect.Across backbones, direction error stays small while estimated velocity magnitude varies more widely.Source: §6.3 · p. 8, Figure 4 · p. 8

7. Conclusion

• VI3 uses an IMU to anchor scale-ambiguous geometric outputs from pretrained 3DFMs to metric observations.The framework offers multiple ways to inject scale, including a dedicated refinement method.Source: §7 · p. 8
• Across different backbones and scenes, learned scale priors and input conditioning are not reliable enough alternatives to physical measurement.The conclusion notes that data-driven priors can be inaccurate and prone to out-of-distribution failures.Source: §7 · p. 8
• Future work may extend physical anchoring to signals such as GNSS, wheel odometry, or depth, and to downstream mapping or novel-view synthesis.Source: §7 · p. 8

Supplementary Material

• Supplementary experiments and ablations analyze VI3’s inertial initialization and test-time refinement.Source: A. Additional results · p. 12

Ground-truth inertial initialization

• GT-free initialization maintains a small performance gap to ground-truth inertial parameters in rotation and depth, with a larger translation gap, consistently across datasets and models.Test-time refinement absorbs errors from the initial inertial estimate; the comparisons cover up-to-scale and metric models.Source: §A.1 · p. 12, Table 8 · p. 13, Table 9 · p. 13

Gravity consensus

• Image-based gravity estimation typically fails in textureless scenarios, while gravity consensus uses inertial estimates to detect large errors and switch to a visual-inertial least-squares estimate.The consensus configuration outperforms each gravity estimate alone on average.Source: §A.1 · p. 12, Table 5 · p. 12

Initial velocity-scale estimation

• On noisy EuRoC and UZH-FPV data, total least squares yields scale estimates closer to 1 than ordinary least squares; on noiseless TartanAirV2, their errors are similar.The TLS assumption that both visual and inertial counterparts are noisy does not hold for noiseless TartanAirV2.Source: §A.1 · p. 12, Table 6 · p. 12

Initial velocity magnitude

• Using all frame pairs and a median to estimate initial velocity magnitude slightly lowers translation errors and improves scale estimates on out-of-distribution scenes with moderate parallax, particularly EuRoC.This is compared with scaling only the first frame pair.Source: §A.1 · p. 12, Table 7 · p. 12

A.2. Test-time refinement

• Adding reprojection depth consistency to pose supervision improves pose through multi-view constraints but degrades scale estimates; depth regularization improves metric-depth correctness.The ablation is reported for VGGT on EuRoC with GT-free initialization.Source: §A.2 · p. 13, Table 10 · p. 14
• Strides 2 or 3 balance parallax and translation: smaller strides provide insufficient parallax, while larger ones reduce multi-view overlap and alter perspective effects.Stride 2 is used in the main results; stride 2 is optimal for initial-velocity direction and stride 3 for magnitude.Source: §A.2 · p. 13, Figure 5 · p. 14
• Eight context frames are the upper bound for fitting some evaluated models in memory.The reported configuration treats 8 frames as a trade-off between refinement capacity, time, and memory.Source: §A.2 · pp. 13–14

B. Additional qualitative results

• VI3 is shown on four TartanAirV2 environments and Vicon Room 2, and recovers from an overscaled metric-aware baseline in high-speed UZH-FPV Indoor Forward scenes.The examples include moderate motion and varied scene objects in Vicon Room 2.Source: B. Additional qualitative results · p. 14

C. Limitations

• Metric scale is recoverable only where motion provides sufficient translational parallax, an observability limit rather than an estimation failure.Model behavior in this regime varies with predicted inter-frame velocities.Source: C. Limitations · p. 14
• The approach recovers metric motion and geometry across backbones and datasets, but models differ in how gracefully they degrade when motion is insufficiently informative.The estimate operates on the models’ predicted inter-frame velocities.Source: C. Limitations · p. 14