OTT3R: Multi-View 3D Reconstruction and Fast Dataset Generation at 1% Compute

1 Introduction

• Large models’ size and training costs limit edge deployment and prevent researchers without industrial-scale compute from training or adapting them.Inference-only acceleration does not address adaptation or architecture iteration, while prior single-workstation distillation was effectively restricted to monocular reconstruction.Source: §1 · p. 2
• SfM is slow, fragile, and parameter-sensitive, and the community lacks a COLMAP-like system for neural 3D pseudo-label generation.Its failure modes include textureless regions, repeated structures, and dynamic objects; lower-resolution processing further increases failure rates.Source: §1 · pp. 2–3
• OTT3R distills π3 into a 102M-parameter student and supplies neurally generated pseudo-labels at 0.2%–1.6% of large models’ training compute.It extends single-workstation, two-GPU distillation to multi-view reconstruction using teacher SE(3) outputs for pose supervision and configurable sampling for wide-baseline pairs.Source: §1 · p. 3
• Zero-shot, the general student beats COLMAP on 7-Scenes and DTU completion but does not replace the teacher out of distribution; the domain-specialized student is the deployable artifact.On 7-Scenes, the specialized student is 4× more accurate than COLMAP, with 980× throughput.Source: §1 · p. 3
• The pipeline generated dense supervision, including camera poses, for 667K images in 3.5 hours on two commodity GPUs and produced a cache roughly 3× smaller per image than prior distillation work [31].The 102M-parameter student compresses the 959M-parameter teacher 9.4×, runs up to 7× faster, and completes general training at approximately 1.6% of VGGT’s compute.Source: §1 · p. 3

2 Related Work

• SfM combines feature extraction, matching, geometric verification, and bundle adjustment, with failure modes and runtime that grows superlinearly with image count.Learned matching, self-supervised descriptors, and differentiable bundle adjustment address parts of the gap, but the workflow remains fragile and slow at higher resolutions.Source: §2 · p. 4
• Feed-forward models predict dense per-pixel geometry, and joint-attention systems such as Fast3R, VGGT, and π3 produce scene-consistent results in one forward pass.Pairwise DUSt3R requires post-hoc pose alignment, while subsequent work explores matching heads, sequential memory, dynamic-scene handling, priors, sparse-view estimation, and stereo coupling.Source: §2 · p. 4
• Most efficiency methods reduce inference cost on trained models, whereas methods training new architectures, including Speed3R and Reliev3R, require non-commodity hardware.Inference-time token merging, sparse attention, and quantization can also be applied to OTT3R’s student.Source: §2 · p. 4
• Distill3R [31] demonstrated two-GPU, single-workstation distillation but lacked camera-pose supervision and used temporally local sampling, yielding effectively monocular reconstructions.OTT3R instead uses teacher SE(3) outputs and configurable sampling to provide wide-baseline transitions for pose learning.Source: §2 · p. 5
• Prior distillation work also targets monocular depth, multi-view depth without compression, or per-scene models [13], rather than a general compact multi-view model.DUNE, adopted as OTT3R’s backbone, uses heterogeneous co-distillation to form compact 3D-aware representations.Source: §2 · p. 5

3.1 Problem Formulation

• The student learns local per-pixel 3D point maps, confidence maps, and camera-to-world SE(3) poses from π3 teacher outputs for unposed RGB views.Global geometry is recovered by transforming local points with predicted poses rather than storing global point maps, reducing redundant supervision, storage, and training cost.Source: §3.1 · p. 5

3.2 Pseudo-Label Generation Pipeline

• The pipeline generates dense RGB-derived supervision without parameter tuning and did not fail on tested sequences, including low-resolution and low-texture sequences where SfM breaks down.It compresses the teacher’s outputs before caching and uses configurable strided, temporal, or mixed sampling; strided sampling supplies wide-baseline transitions needed for pose estimation.Source: §3.2 · pp. 5–7, Figure 2 · p. 6
• The pipeline generated supervision for 667K images in 3.5 hours on two RTX 6000 Ada GPUs, storing 667K images in 283 GB.It generates more than twice Distill3R’s per-GPU throughput and improves per-image cache cost 3×; generation and training are decoupled and wall-clock scales linearly with GPU count.Source: §3.2 · p. 7

3.3 Student Architecture

• A shared DUNE ViT-Small encoder feeds a multi-layer transformer with alternating cross-view attention and separate decoders for points, confidence, and camera poses.The paper reports comparable performance with DINOv2, indicating robustness to encoder choice within this family.Source: §3.3 · p. 7

3.4 Distillation Loss

• The point loss uses scale-aligned L1 geometry supervision, with teacher-confidence masks and depth-based weights emphasizing closer points.A camera loss supervises relative translation and rotation across all view pairs, providing dense supervision for small and large viewpoint changes.Source: §3.4 · pp. 7–8
• The hybrid confidence loss blends BCE based on student reconstruction error with L1 regression to teacher confidence, weighted per pixel by teacher confidence.High-confidence regions teach confidence from geometry quality; low-confidence regions defer to the teacher’s uncertainty. A normal loss adds surface-orientation consistency to the point loss.Source: §3.4 · p. 8

4.1 Implementation Details

• The 102M-parameter student compresses the 959M-parameter teacher 9.4× and trains at 1.62% of VGGT’s training compute.The general model trains for 192 hours on two RTX 6000 Ada GPUs; the 7-Scenes specialization trains for about 23 hours.Source: §4.1 · p. 9

4.2 Results

• On 7-Scenes, the domain-specific student is 4× more accurate than COLMAP and runs 980× faster, with near-teacher completion.The general student beats COLMAP on 7-Scenes and DTU completion zero-shot but does not reach the teacher on out-of-distribution multi-view geometry.Source: Table 1 · p. 10, §4.2 · p. 10
• On CO3D, teacher agreement falls from 61.9% RRA@5 on temporal pairs to 38.7% on strided wide-baseline pairs.The corresponding median rotation errors are 2.8° and 15.3°; student rows measure agreement with the teacher, not absolute accuracy.Source: Table 2 · p. 10, §4.2 · p. 11
• At 128 views on RTX 6000 Ada at 518×224, OTT3R completes inference in 2.2 s using 5.2 GB; its teacher needs 12.3 GB, while VGGT runs out of memory with 48 GB VRAM.Source: §4.2 · p. 12, Table 5 · p. 12
• On ARKitScenes, the general student reaches 97.0% δ < 1.25 versus the teacher’s 98.8%; on Habitat, it reaches 93.9% versus 98.3%.On 7-Scenes, the most out-of-distribution dataset, it falls to 85.7% δ < 1.25 and 13.1% Abs Rel.Source: §4.2 · p. 12

4.3 Ablations

• Confidence-loss variants trade geometric quality and calibration, while encoder ablations show that greater capacity does not guarantee better reconstruction at fixed compute.Pure-L1 and pure-BCE degrade 3–7% versus the hybrid; DINOv2-B (280M) records 9.66 cm accuracy and 20.97 cm completion, while halving the student to 54M also degrades quality.Source: §4.3 · p. 13, Table 7 · p. 14
• The student does not match teacher multi-view geometric precision, particularly on out-of-distribution scenes, and its pseudo-label pipeline inherits teacher limitations on strongly dynamic or non-rigid scenes.The authors recommend the teacher for high-fidelity general reconstruction and domain specialization for target distributions or deployment scenarios.Source: §4.3 · pp. 13–14
• Without direct SE(3) supervision, completion degrades 17×; temporal-only sampling degrades it 9×.The reported changes are 2.84 to 49.57 cm and 2.84 to 25.22 cm, respectively; the ablations associate pose supervision with alignment and sampling diversity with wide-baseline training pairs.Source: §4.3 · pp. 12–13, Table 6 · p. 14

• The 102M-parameter student trains on a single workstation at roughly 1.6% of the compute used by comparable feed-forward 3D models.Source: §5 · p. 14
• At 274 FPS, the student remains competitive with the teacher on in-distribution monocular depth.Source: §5 · p. 14
• After specialization at approximately 0.2% of the teacher’s compute, the student beats COLMAP by 4× on 7-Scenes multi-view accuracy.Source: §5 · p. 14
• The pseudo-label pipeline produces dense per-pixel supervision for hundreds of thousands of images in hours and scales linearly with GPU count.The paper presents this as a mechanism for generating large-scale 3D supervision for applications such as robotics.Source: §5 · p. 14

• The teacher is run once to cache quantized point maps, confidence, and SE(3) poses, after which the student trains without reinvoking it.Source: Figure 4 · p. 20, A Training Pipeline · p. 20
• Each batch loads pre-sampled views with cached teacher targets, and gradients flow only through the student.The student’s point and normal heads use teacher geometry, while its camera head uses teacher relative poses across view pairs.Source: A Training Pipeline · p. 20
• Precomputed supervision leaves only the student’s forward and backward passes online, so memory and runtime are governed by the 102M-parameter student rather than the teacher.Source: A Training Pipeline · p. 21

• Both temporal and strided sampling produce coherent geometry in the shown 20-view CO3D reconstructions at 518 × 224 resolution.Source: B Qualitative Multi-View Reconstructions · p. 21
• Strided sampling spans a wide baseline, and its higher relative-rotation error appears as noise rather than visible structural misalignment in these examples.Source: B Qualitative Multi-View Reconstructions · p. 21
• The figure places temporal sampling on the left and strided sampling on the right for each scene pair.Source: B Qualitative Multi-View Reconstructions · p. 21

• On 7-Scenes, zero-shot out-of-distribution pose errors prevent the general student’s views from aligning, yielding 34.84 versus 2.84 cm completion for general and specialized students.Source: C Failure Cases · p. 24
• On CO3D, wide-baseline relative-rotation errors can misalign strided reconstructions, and near-symmetric objects produce front/back confusions shared with the teacher.Source: C Failure Cases · p. 24, Figure 6 · p. 26
• The illustrated failure comparisons pair the general and specialized students on the same 7-Scenes sequence and temporal versus strided samples of the same CO3D object.Source: Figure 5 · p. 25, Figure 6 · p. 26