Source-linked AI summary
LVSPM: Long Sequence View Synthesis and Pose Estimation Model
Xi Chen, Yachi Zhang, Linghao Chen, Minghua Liu, Hao Su, Zexiang Xu, Xiaoshuai Zhang
TL;DR
LVSPM addresses the challenge of long-sequence pose-free view synthesis without dense 3D labels by jointly predicting camera poses and novel views from unposed images. It achieves state-of-the-art results across benchmarks for pose estimation and view synthesis, with graceful scaling to larger scenes.
Problem
Existing view-synthesis methods commonly assume known camera poses, while pose-free methods are largely limited to sparse views and geometry-based pose models rely on costly dense 3D supervision.
Source: §1 · pp. 1–2Method
LVSPM uses a sequence-to-sequence model with a LaCT backbone and per-view camera tokens to jointly estimate poses and synthesize novel views from RGB images with camera-pose supervision.
Source: §1 · pp. 2–3Results
LVSPM achieves state-of-the-art results across RE10k, Co3Dv2, and DL3DV for pose estimation and view synthesis, surpassing pose-dependent baselines and scaling gracefully as scene complexity increases.
Source: §1 · p. 3, §5 · p. 15Takeaways & Limitations
The authors frame LVSPM as a step toward practical, scalable 3D perception using cheaper supervision without dense 3D geometry labels.
Source: §5 · p. 15Performance may degrade under extreme lighting changes or in weakly textured scenes, and specialized geometry-focused models may perform better in some edge cases.
Source: §5 · p. 15
Abstract
from arXiv · showhide
We present LVSPM, a generalizable model that jointly estimates camera poses and synthesizes novel views from uncalibrated image collections. Trained with only RGB images and pose supervision, LVSPM avoids dense 3D ground truth and employs test-time training (TTT) layers to scale seamlessly to hundreds of input views. On RealEstate10k, Co3Dv2, and DL3DV, LVSPM surpasses VGGT in pose estimation across 16-256 views, with especially large margins at strict thresholds. For novel view synthesis under a practical protocol where more views cover larger scenes, LVSPM achieves state-of-the-art pose-free quality---surpassing even pose-dependent models in PSNR---and still maintains high quality as scene scale grows, while baselines collapse. The code is available at https://burningdust21.github.io/Projects/LVSPM .
1 Introduction
LVSPM targets long-sequence pose-free view synthesis without dense 3D labels, jointly estimating camera poses and synthesizing novel views from unposed inputs. It uses a sequence-to-sequence model and reports state-of-the-art results across three benchmarks.
1 Introduction
- Pose-free feed-forward synthesis methods are mostly limited to sparse views, while geometry-based pose estimators do not synthesize views and rely on costly dense 3D supervision.Existing NVS methods also commonly assume known camera poses, often estimated through computationally expensive and fragile SfM pipelines.Source: §1 · pp. 1–2
- The model jointly predicts camera poses and novel-view images from unposed inputs using RGB images and camera-pose supervision, without dense 3D labels.It tokenizes input views and uses a LaCT backbone with per-view camera tokens for scalable sequence modeling.Source: §1 · p. 2
- LVSPM is introduced as a pose-free view-synthesis model that scales to 256 input views through a LaCT backbone with learnable camera tokens.It predicts camera poses and novel views in a unified sequence-to-sequence framework.Source: §1 · p. 3
- Removing novel-view-synthesis supervision reduces AUC3 from 71.9 to 50.9; the proposed equi-temporal protocol isolates scene-scale effects from sampling density.The authors report that full LVSPM surpasses geometry-based methods trained with dense point-map supervision.Source: §1 · p. 3
- LVSPM achieves state-of-the-art results on RE10k, Co3Dv2, and DL3DV for long-sequence pose estimation and view synthesis, surpassing pose-dependent baselines and scaling to large scenes.Prior methods degrade more rapidly as scene complexity increases.Source: §1 · p. 3
2 Related Work
Prior work has advanced view synthesis, camera estimation, and long-sequence modeling, but typically handles these problems separately or retains pose and dense-geometry requirements. LVSPM extends minimal-bias synthesis and long-sequence modeling to jointly estimate poses and synthesize views.
Novel View Synthesis
- Many view-synthesis methods use 3D representations or architectural priors, while LVSM reduces these biases but assumes known camera poses.Optimization-based NeRF and 3D Gaussian Splatting methods can produce photo-realistic results at substantial per-scene computational cost.Source: §2 · p. 3
Pose-Free View Synthesis
- Long-sequence models such as Long-LRM and LaCT handle tens or hundreds of images but assume known poses; LVSPM extends LaCT to pose-free joint synthesis and pose estimation.Earlier pose-free feed-forward approaches often use fewer than ten inputs, and AnySplat supports tens of views while requiring dense 3D supervision.Source: §2 · pp. 3–5
Camera Pose Estimation
- Geometry-focused pose-estimation models such as DUSt3R and VGGT achieve strong feed-forward results but rely on dense 3D labels and do not directly synthesize views.LVSPM draws on VGGT’s camera-estimation mechanism within a view-synthesis framework.Source: §2 · p. 4
3 Method
LVSPM jointly estimates camera poses and synthesizes novel views from unposed image sequences, framing both tasks as sequence-to-sequence prediction. It uses LaCT blocks and learnable camera tokens, with photometric, pose, and intrinsic supervision.
3 Method
- LVSPM jointly predicts camera poses and novel-view RGB images from unposed input views as a sequence-to-sequence task.The formulation avoids explicit 3D representations and precomputed structure-from-motion pipelines.Source: §3 · p. 5, §3.1 · p. 5
3.2 Model details
- Input images become patch tokens with learnable camera tokens, while target views are encoded as Plücker-ray tokens with a learnable token.The first input view is the canonical reference frame; scene translation is normalized using the distance between the first and furthest points.Source: §3.2 · pp. 6–7
- Each of 24 LaCT blocks combines windowed self-attention, a test-time-training layer, and a feed-forward MLP to process input and target tokens.The architecture is described as processing variable-length sequences up to 1M tokens.Source: §3.1 · p. 5, §3.2 · p. 7
- Only input-view tokens update TTT weights, leaving target tokens read-only so target views can be synthesized independently and in parallel at 58.8 FPS.The shared TTT hidden state is queried by camera tokens for pose prediction and target ray tokens for synthesis; the authors report that NVS supervision improves pose accuracy without dense 3D supervision.Source: §3.2 · pp. 7–8
- Lightweight two-layer MLP heads decode target tokens into RGB images and camera tokens into quaternion, translation, and field-of-view parameters.The authors report better camera estimation than VGGT with fewer model parameters.Source: §3.2 · p. 8
3.3 Training Objective
- Training combines photometric RGB, camera-pose, and camera-intrinsic losses, with separate weights for the three objectives.The RGB loss includes squared pixel error and LPIPS; pose and intrinsic losses use L1 terms.Source: §3.3 · p. 8
4 Experiments
LVSPM is evaluated for pose estimation and view synthesis across multiple datasets and input lengths, including 256-view sequences. The experiments also examine its training and architecture choices, evaluation protocol, and inference speed.
4.1 Experiment Settings
- The experiments evaluate joint pose estimation and view synthesis on RE10K, Co3Dv2, and DL3DV-10K, with view synthesis also tested zero-shot on Tanks and Temples and MipNeRF360.Training combines synthetic and real-world datasets; evaluation reports pose AUC metrics and NVS PSNR, SSIM, and LPIPS.Source: §4 · p. 9, §4.1 · pp. 9–10
4.2 Camera Pose Estimation
- At 256 views, LVSPM reaches 90.1 AUC3 on DL3DV-10K versus 83.6 for VGGT and 83.8 for DA3, while Fast3R and Cut3R fall to 8.0 and 19.2.Across RE10K and Co3Dv2, LVSPM is best or second-best in all reported entries; pose results show especially large advantages at strict thresholds.Source: Table 2 · p. 10, Table 1 · p. 10, §4.2 · p. 11
- The equi-temporal protocol makes additional input views cover more of a video sequence, isolating scene-scale effects while keeping sampling density approximately constant.It evaluates RE10K and Co3Dv2 up to 128 views and DL3DV up to 256 views.Source: §4.2 · p. 11
4.3 View Synthesis
- At 16 views, LVSPM achieves 25.77 PSNR on DL3DV-10K, above pose-dependent DepthSplat at 24.80 and pose-free AnySplat at 23.15; its advantage over AnySplat grows to 3.54 dB at 256 views.Under the equi-temporal protocol, LVSPM’s PSNR falls by 3.68 dB across a 16× increase in scene scale, versus AnySplat’s 4.60 dB; DepthSplat fails beyond 64 views.Source: Table 3 · p. 12, §4.3 · p. 12
4.4 Ablation Study
- Synthetic pretraining improves DL3DV pose estimation and rendering; removing NVS supervision lowers AUC3 from 71.9 to 50.9, while smaller TTT chunks and fewer layers also degrade performance.The NVS-supervision ablation is conducted on ASE; the full model uses 24 TTT layers and a chunk size of 8224 tokens.Source: §4.4 · p. 14
4.5 Inference Time comparison
- 256-image pose inference takes under 2 seconds on an H100 GPU, and LVSPM scales linearly with input views while rendering at 58.8 FPS regardless of their number.Timing comparisons use 512×288 resolution for LVSPM and several baselines, and 518×280 for VGGT.Source: §4.5 · pp. 14–15
5 Conclusion and Limitations
LVSPM jointly estimates world-aligned camera poses and synthesizes novel views from long unposed image sequences, using RGB images and pose supervision instead of dense geometric supervision. It scales to 200+ views and maintains strong rendering quality as scene coverage grows, though training complexity and challenging scene conditions remain limitations.
5 Conclusion and Limitations
- 200+ views: LVSPM maintains strong rendering quality as scene coverage grows under equi-temporal evaluation, while existing baselines degrade more rapidly as scene complexity increases.The evaluation protocol pairs additional input views with larger scene coverage.Source: §5 · p. 15
- LVSPM jointly estimates world-aligned camera poses and synthesizes novel views, using RGB images and pose supervision without dense 3D geometric supervision.The framework is enabled by an efficient test-time training mechanism.Source: §5 · p. 15
- The multi-stage training pipeline adds complexity, and performance may degrade under extreme lighting changes or in weakly textured scenes.The passage relates these scene conditions to difficulty in reliable geometric reasoning.Source: §5 · p. 15
- Specialized geometry-focused models may still perform better in certain edge cases, despite LVSPM's competitive pose estimation and joint view synthesis.Source: §5 · p. 15
- Future work includes fully self-supervised training and investigating scalable 3D scene understanding with minimal geometric priors.Source: §5 · p. 15