Source-linked AI summary
FLARE: Feed-forward Geometry, Appearance and Camera Estimation from Uncalibrated Sparse Views
Shangzhan Zhang, Jianyuan Wang, Yinghao Xu, Nan Xue, Christian Rupprecht, Xiaowei Zhou, Yujun Shen, Gordon Wetzstein
TL;DR
Uncalibrated sparse-view reconstruction must recover camera poses, geometry, and appearance from as few as 2-8 images, while prior approaches face accuracy, generalization, or efficiency limitations. FLARE uses a feed-forward cascade in which pose estimates guide two-stage geometry and appearance learning. It reports state-of-the-art performance across pose estimation, geometry reconstruction, and novel-view synthesis with inference below 0.5 seconds.
Problem
Sparse-view reconstruction from uncalibrated images requires estimating camera poses, geometry, and appearance, while existing methods face limitations in accuracy, generalization, dense geometry, or efficiency.
Method
FLARE progressively estimates camera poses, camera-centric point maps, global geometry, and appearance in a cascaded feed-forward pipeline.
Results
FLARE achieves state-of-the-art results in camera pose estimation, point cloud estimation, and novel-view synthesis with inference in less than 0.5 seconds.
Takeaways & Limitations
Camera poses serve as geometric proxies that support high-quality reconstruction and photorealistic novel-view synthesis from very sparse inputs.
Abstract
from arXiv · showhide
We present FLARE, a feed-forward model designed to infer high-quality camera poses and 3D geometry from uncalibrated sparse-view images (i.e., as few as 2-8 inputs), which is a challenging yet practical setting in real-world applications. Our solution features a cascaded learning paradigm with camera pose serving as the critical bridge, recognizing its essential role in mapping 3D structures onto 2D image planes. Concretely, FLARE starts with camera pose estimation, whose results condition the subsequent learning of geometric structure and appearance, optimized through the objectives of geometry reconstruction and novel-view synthesis. Utilizing large-scale public datasets for training, our method delivers state-of-the-art performance in the tasks of pose estimation, geometry reconstruction, and novel view synthesis, while maintaining the inference efficiency (i.e., less than 0.5 seconds). The project page and code can be found at: https://zhanghe3z.github.io/FLARE/
1. Introduction
FLARE addresses sparse-view 3D reconstruction by estimating camera poses first, then using them to guide geometry and appearance learning. The feed-forward system targets uncalibrated inputs and reports high-quality reconstruction with inference below 0.5 seconds.
- Traditional SfM-MVS pipelines rely on handcrafted feature matching, are non-differentiable, and struggle with sparse or limited viewpoints.
- Existing optimization-based, regression-based, and point-map methods face initialization, generalization, accuracy, dense-geometry, or inference-time limitations.
- FLARE uses a cascaded learning paradigm that estimates camera poses first and conditions subsequent geometry and appearance learning on them.
- Its geometry pipeline predicts camera-centric point maps before a neural scene projector unifies them into a coherent global structure.
- Trained on large public datasets, FLARE targets camera pose estimation, point cloud estimation, and novel-view synthesis.
- 0.5 seconds enables photorealistic Gaussian-Splatting novel-view synthesis from as few as 2-8 input images.
- The paper presents an efficient, feed-forward, differentiable system and identifies camera poses as proxies that simplify complex 3D learning tasks.
2. Related Work
Related work spans conventional SfM and MVS pipelines, neural sparse-view reconstruction, and pose-free novel-view synthesis. These approaches increasingly use learning or differentiable optimization, but sparse-view generalization and efficient global alignment remain challenges described in the supplied passages.
- Conventional SfM estimates camera poses and sparse 3D structure through feature matching followed by incremental bundle adjustment.
- Traditional MVS reconstructs dense geometry through depth estimation, depth fusion, and surface reconstruction using photometric consistency and regularization.
- Neural representations have supported novel-view synthesis, first mainly in dense-view single-scene settings and later in sparse-view scenarios.
- Pose-free novel-view methods include joint camera-and-radiance optimization and newer 3D Gaussian-based approaches for uncalibrated images.
- Recent generalizable reconstruction methods still show degraded sparse-view performance or limited generalization to complex 3D scenes.
3. Method
FLARE uses camera pose estimation as an intermediate representation in a cascaded feed-forward pipeline for reconstructing geometry and appearance from uncalibrated sparse views. It predicts local and global geometry, initializes 3D Gaussians, and trains the system jointly with pose, geometry, and rendering objectives.
- Cascaded Learning: Estimated camera poses condition subsequent geometry and appearance learning as spatial initialization and geometric priors.The pipeline uses pose estimates to guide camera-centric geometry, global projection, and Gaussian appearance modeling.
- Neural Pose Predictor: FLARE directly estimates camera poses from uncalibrated sparse views with an end-to-end transformer, avoiding feature matching.Image patches and learnable camera latents are processed by a decoder-only neural pose predictor.
- Multi-view Geometry Estimation: The two-stage geometry module first predicts dense point maps in local camera coordinates, then projects them into global scene space with refined poses.A learnable transformer geometry projector replaces direct reprojection, making global reconstruction more robust to pose inaccuracies.
- 3D Gaussians for Appearance Modeling: Point maps serve as Gaussian centers, while a regression head predicts opacity and appearance parameters from geometry and fused image features.The model combines a DPT appearance head with pretrained VGG features and a shallow CNN decoder.
- 3D Gaussians for Appearance Modeling: Scene and Gaussian scales are normalized before differentiable rasterization produces rendered novel views.The renderer enables end-to-end optimization through reconstruction loss.
- Training Loss: Training jointly minimizes camera pose, geometry, and Gaussian splatting losses, including pose Huber terms, confidence-aware point regression, and rendering supervision.The splatting loss combines L2, VGG perceptual, and depth losses, with separate weights for the three components.
4. Experiment
FLARE is evaluated for camera pose estimation, sparse-view geometry reconstruction, and novel-view synthesis across public datasets and challenging uncalibrated inputs. The experiments also examine its implementation range and component contributions.
- FLARE is trained on a mixture of eight public datasets featuring diverse scene types.
- The model is trained from scratch with 8-view inputs, using an encoder as the only pretrained component.
- Sparse-view 3D reconstruction: Sparse-view reconstruction is evaluated using camera-pose AUC and geometry accuracy, completeness, and overall Chamfer distance.
- Camera pose estimation: FLARE generalizes from 2 to 25 input views despite being trained with 8 views.
- Sparse-view 3D reconstruction: FLARE achieves better geometry reconstruction with less noise than DUSt3R, MASt3R, and Spann3R while offering faster inference than the optimization-based baselines.
- Novel-view synthesis: For novel-view synthesis, FLARE substantially outperforms pose-free and pose-required baselines on DL3DV and RealEstate10K without requiring camera extrinsics.
- Ablation study: Ablations show that camera-pose conditioning, camera-centric geometry, joint training, and the DPT head improve geometry learning, while rendering loss raises completeness but slightly lowers accuracy.
5. Discussion
FLARE combines feed-forward inference with cascaded estimation for sparse-view, uncalibrated reconstruction. Its reported scope includes camera poses, geometry, appearance, and photorealistic novel-view synthesis.
- FLARE infers camera poses, geometry, and appearance from sparse-view uncalibrated images within 0.5 seconds.
- The cascade progressively estimates camera poses, geometry, and appearance and improves performance over previous methods.
- Training on public datasets gives FLARE reconstruction priors that generalize to very sparse real-world views and support photo-realistic novel-view synthesis.
Appendix
The supplementary material expands the paper’s implementation, experiments, and architecture descriptions.
- The supplement provides extended data-processing and training details, additional experimental results, and a comprehensive network-architecture description.
A. Implementation Details
The implementation adapts data processing and baseline evaluation to support sparse-view training and comparison across varying input-image counts.
- Data processing: DL3DV depth annotations are generated with COLMAP multi-view stereo and converted into point maps, with photometric and geometric consistency used to remove noisy depth.
- Novel-view synthesis baselines: For DL3DV baseline comparisons, two source views nearest the target are selected because the compared methods were trained on two views.
- Input images: The model can process any number of input images subject to GPU-memory capacity by duplicating the shared source token.
B. Experiments
Experiments examine how FLARE behaves as the number of input views changes and how its reconstruction compares with prior methods. Performance improves with more views but eventually plateaus, while dense-view results place FLARE between DUSt3R and MASt3R.
- Comparison with Sparse-view Methods: MVSplat is evaluated with two views because its performance degrades when additional input frames are used.The reported results therefore use its optimal two-view setting.
- Qualitative Reconstruction: DTU sparse-view reconstructions are visualized through the input image, depth map, and point cloud.
- Varying Input Views: FLARE performance improves in pose and surface accuracy as the number of input views increases, though gains gradually plateau.The study evaluates 2, 6, 10, 16, and 25 source views with two query views fixed for testing.
- Varying Input Views: Under the 2-view setting, FLARE produces a reasonable shape, but its precision remains limited.
- Dense-view Reconstruction: On DTU dense-view reconstruction, FLARE outperforms DUSt3R but underperforms MASt3R on accuracy, completeness, and overall metrics.The comparison uses the dense-view setting, although dense reconstruction is not FLARE’s primary objective.