Source-linked AI summary
Free View Synthesis
Gernot Riegler, Vladlen Koltun
TL;DR
The paper addresses novel-view synthesis from freely distributed input images, seeking free camera movement and photorealistic results in general scenes. It combines SfM/MVS-derived proxy geometry with recurrent feature mapping and blending, and produces sharp novel views without per-scene adaptation. Across challenging real-world datasets, it outperforms prior and concurrent methods, while assuming shared intrinsics for video input and acknowledging illumination variation as a metric challenge on DTU.
Problem
Novel-view synthesis should support free movement and photorealistic images from unstructured inputs in general scenes, beyond methods requiring regular camera arrangements.
Method
The method uses SfM/MVS-derived 3D proxy geometry to warp encoded source features into the target view, then recurrently blends them with an encoder-decoder network.
Results
The approach produces sharp novel views that outperform prior and concurrent methods across challenging real-world datasets, reducing LPIPS relative to EVS, LLFF, NeRF, and NPBG on reported comparisons.
Takeaways & Limitations
A trained model generalizes to previously unseen scenes without fine-tuning or per-scene optimization while supporting arbitrary numbers of source images per target view.
Takeaways & Limitations
For video input, the method assumes shared camera intrinsics, and varying illumination in DTU scenes limits the interpretability of low-level metrics such as PSNR.
Abstract
from arXiv · showhide
We present a method for novel view synthesis from input images that are freely distributed around a scene. Our method does not rely on a regular arrangement of input views, can synthesize images for free camera movement through the scene, and works for general scenes with unconstrained geometric layouts. We calibrate the input images via SfM and erect a coarse geometric scaffold via MVS. This scaffold is used to create a proxy depth map for a novel view of the scene. Based on this depth map, a recurrent encoder-decoder network processes reprojected features from nearby views and synthesizes the new view. Our network does not need to be optimized for a given scene. After training on a dataset, it works in previously unseen environments with no fine-tuning or per-scene optimization. We evaluate the presented approach on challenging real-world datasets, including Tanks and Temples, where we demonstrate successful view synthesis for the first time and substantially outperform prior and concurrent work.
1 Introduction
The paper targets photorealistic free movement through general scenes from unstructured input images. Its learned approach outperforms prior and concurrent methods across challenging real-world datasets.
- Free view synthesis must support unrestricted movement through a scene while producing photorealistic images.The motivation requires both free camera or body movement and images that are perceptually indistinguishable from reality.
- The method uses proxy geometry to reproject encoded source-image features into a novel target view and recurrently blends them.The recurrent blending network handles target views that may deviate significantly from source views and accepts varying numbers of input images.
- The approach addresses free view synthesis from unstructured input images in general scenes.
- More than 2× lower LPIPS error is achieved on all Tanks and Temples scenes relative to state-of-the-art methods including EVS and LLFF.LPIPS is also significantly reduced relative to EVS and LLFF on DTU.
- The method reduces LPIPS relative to NeRF and NPBG on Tanks and Temples and performs on par with them on DTU.Unlike the dataset-dependent strengths of NPBG and NeRF, the approach performs well across both datasets.
2 Related Work
Prior image-based rendering methods commonly impose geometric or camera-layout constraints, while newer deep-learning methods learn blending or neural scene representations. The paper is positioned for unstructured input views and avoids per-scene feature optimization used by some concurrent approaches.
- Classical image-based rendering synthesizes novel views directly from input images using different mappings of source information to target views.
- Early light-field methods either require dense, regularly spaced camera grids or restrict targets to linear combinations of source views.These constraints differ from the freely distributed input views targeted by the paper.
- Unstructured view synthesis commonly uses 3D proxy geometry to guide mapping and blending, with prior systems relying on accurate geometry, depth maps, or missing-depth compensation.
- Deep-learning approaches use neural networks for image blending, neural scene representations, or joint geometry estimation and blending.
- Several learned rendering systems use plane-sweep volumes or multi-plane image representations to predict or compose views under structured camera settings.
- Neural Point-Based Graphics associates learned feature vectors with 3D points and optimizes those features per scene, requiring extractor training for each new scene.Single-image methods discussed in related work allow only small viewpoint deviations rather than unrestricted scene travel.
3 Method
The method preprocesses unstructured input images into calibrated views and proxy geometry, then selects relevant sources and recurrently maps and blends their features into a novel target view.
- 3.1 Preprocessing: The preprocessing stage estimates image poses and 3D proxy geometry using structure-from-motion, multi-view stereo, and meshing.SfM provides camera poses and a sparse point cloud; MVS depth maps are fused into a coherent point cloud before meshing.
- 3.1 Preprocessing: The proxy geometry supplies depth maps for source and target views, while meshing addresses invalid depth values associated with point-cloud rendering.The resulting surface mesh M is used to derive depth maps for arbitrary target views.
- 3.2 Selection of Source Images: The network supports an arbitrary number of source images, but selects the K views with maximal target-view overlap based on proxy geometry.The selection score counts target pixels mapped into valid source-image domains, with a 1% depth-consistency threshold for occlusion and outlier handling.
- 3.3 Mapping and Blending: A shared convolutional encoder extracts source features, which are warped into the target view using its proxy-derived depth map.Out-of-domain warped features are zeroed and accompanied by a mask; bilinear interpolation handles non-central projected locations.
- 3.3 Mapping and Blending: A recurrent convolutional blending decoder processes the warped features and predicts per-pixel confidence and color, combined by a soft-argmax.Confidence values determine softmax weights for aggregating the per-source color predictions into the final image.
- 3.3 Mapping and Blending: Training with withheld source images uses a perceptual loss comparing the synthesized image with the held-out ground truth.The perceptual term uses features from layers of a pretrained VGG-19 network.
4 Experimental Evaluation
The evaluation tests architectural choices and source-view scaling, then compares free-view synthesis against prior and concurrent methods on Tanks and Temples, DTU, and new recordings. The method performs strongly across unstructured and constrained settings, while qualitative and quantitative results expose distinct baseline weaknesses.
- Experimental setup: The evaluation trains on 17 of 21 Tanks and Temples scenes and tests on unseen Truck, Train, M60, and Playground scenes.Testing uses withheld target images and evaluates novel-view synthesis on scenes absent from training.
- Architectural choices: The architectural ablations evaluate encoding, recurrent mapping and blending, masks, invalid-depth handling, and soft-argmax choices under a leave-one-out protocol.Fixed-input and recurrent variants are compared on Tanks and Temples.
- Architectural choices: Encoding source images before mapping and blending improves results, while recurrent units benefit performance by propagating blending information between source images.The comparison with Ours w/o GRU and Cat Global Avg. supports the recurrent-network design choice.
- Source-view scaling: Image fidelity improves with the number of source images up to 7 images and then saturates.More source images are especially important when the target view lies farther from the scene or object than any source view, because they cover more of the view frustum.
- Tanks and Temples: On Tanks and Temples, the method reduces LPIPS error by more than a factor of 2 on all scenes relative to state-of-the-art methods including EVS and LLFF.The evaluation uses withheld novel-view sequences and reports PSNR, SSIM, and LPIPS, with LPIPS better aligned to human perception than the low-level metrics.
- Cross-dataset comparison: The method also outperforms concurrent NeRF and NPBG on Tanks and Temples and performs on par with them on DTU, without test-scene adaptation or fine-tuning.NPBG performs well on Tanks and Temples but poorly on DTU, while NeRF shows the opposite pattern; the proposed method performs well across datasets.
- Tanks and Temples: On Tanks and Temples, LLFF produces strong ghosting, EVS misses fine details or image parts, NeRF is blurry or fails, and NPBG produces competitive images.The proposed method produces sharp details and is superior to the other methods in LPIPS.
- DTU: On DTU, the method produces sharp interpolation and extrapolation results, reasonable inpainting when geometry is missing, and realistic results despite view-dependent illumination.DTU uses 49 poses, with 10 targets: 6 interpolation views and 4 extrapolation views.
5 Conclusion
The method targets novel view synthesis from unstructured images acquired through natural motion. SfM/MVS preprocessing and recurrent feature mapping produce sharp views, while temporal consistency remains future work.
- The method synthesizes novel views from unstructured input images acquired by natural motion through a scene.
- Standard SfM and MVS provide camera parameters and 3D proxy geometry before recurrent mapping and blending synthesize new views.
- The recurrent architecture accepts an arbitrary number of source images per target view, reducing reliance on hand-crafted source-selection heuristics.
- The method produces sharp images for target views that depart significantly from the inputs.
- DTU view extrapolation results are reported separately from interpolation results.