Source-linked AI summary
Stereo Radiance Fields (SRF): Learning View Synthesis for Sparse Views of Novel Scenes
Julian Chibane, Aayush Bansal, Verica Lazova, Gerard Pons-Moll
TL;DR
Scene-specific neural view synthesis methods require dense views and costly retraining, motivating models that generalize from sparse inputs. SRF addresses this with an end-to-end architecture that emulates multi-view stereo through learned image-feature similarities. Trained across scenes, it generalizes to new scenes from 10 sparse views, improves with 10–15 minutes of fine-tuning, and outperforms SOTA methods in the sparse-view setting.
Problem
NeRF-like view synthesis methods are trained for a specific scene, require dense multi-view inputs, and need retraining to generalize to new scenes.
Method
SRF is an end-to-end neural view synthesis architecture that predicts radiance from projected image features and learned ensembles of pairwise similarity scores emulating stereo.
Results
SRF generalizes to new scenes from 10 sparse, spread-out views, improves after 10–15 minutes of fine-tuning, and significantly outperforms SOTA methods in the 10-view setting.
Takeaways & Limitations
SRF learns common structure across scenes and produces sharper sparse-view results while also yielding an interpretable 3D representation for colored meshing.
Takeaways & Limitations
Challenging BRDFs and reflective regions can pose problems for SRF's stereo-matching approach, although fine-tuning mitigates some issues.
Abstract
from arXiv · showhide
Recent neural view synthesis methods have achieved impressive quality and realism, surpassing classical pipelines which rely on multi-view reconstruction. State-of-the-Art methods, such as NeRF, are designed to learn a single scene with a neural network and require dense multi-view inputs. Testing on a new scene requires re-training from scratch, which takes 2-3 days. In this work, we introduce Stereo Radiance Fields (SRF), a neural view synthesis approach that is trained end-to-end, generalizes to new scenes, and requires only sparse views at test time. The core idea is a neural architecture inspired by classical multi-view stereo methods, which estimates surface points by finding similar image regions in stereo images. In SRF, we predict color and density for each 3D point given an encoding of its stereo correspondence in the input images. The encoding is implicitly learned by an ensemble of pair-wise similarities -- emulating classical stereo. Experiments show that SRF learns structure instead of overfitting on a scene. We train on multiple scenes of the DTU dataset and generalize to new ones without re-training, requiring only 10 sparse and spread-out views as input. We show that 10-15 minutes of fine-tuning further improve the results, achieving significantly sharper, more detailed results than scene-specific models. The code, model, and videos are available at https://virtualhumans.mpi-inf.mpg.de/srf/.
1. Introduction
SRF combines neural rendering with classical multi-view stereo to synthesize novel views from sparse inputs while generalizing across scenes. Experiments report sharper results, novel-scene generalization, and minutes-scale fine-tuning instead of scene-specific retraining.
- Contribution: SRF is an end-to-end neural multi-view synthesis approach that generalizes to novel scenes from sparse test-time views.Unlike NeRF-like methods, it is not trained specifically for one scene.
- Method: SRF uses image-based feature encoding and learned pairwise similarities to reason about scene geometry rather than memorizing radiance at 3D locations.Its architecture emulates correspondence computation from classical stereo without explicitly computing correspondences.
- Results: 100 or more training scenes enable SRF to generalize to novel scenes using only 10 sparse and spread-out input views.Training on a single scene also produces reasonable results on a new scene, indicating structure learning rather than scene memorization.
- Results: Fine-tuning on the 10 test views takes minutes rather than the 2–3 days required to retrain scene-specific methods.The paper reports that fine-tuning further improves results.
- Contribution: SRF combines classical multi-view reconstruction, learning from multiple scenes, and neural rendering in one model.The contribution is framed as bringing together paradigms often treated separately in novel view synthesis.
- Results: In sparse, spread-out view settings, SRF produces much sharper results than SOTA baselines such as NeRF.The sharper outputs support the claim that multi-view reconstruction structure constrains learning and improves generalization.
2. Multi-View View Synthesis
Multi-view view synthesis seeks to render a new virtual-camera view from input camera views. SRF combines neural rendering with classical correspondence reasoning, using learned pairwise similarity scores and image features to support sparse-view generalization.
- Problem: The goal is to synthesize an image for a new virtual camera given N camera views.Prior approaches differ in whether they use no geometry, correspondences, or explicit geometry.
- Classical and learned approaches: Classical multi-view stereo relies on correspondences across views, while SRF integrates this geometric insight into a learning-based architecture.SRF replaces explicit correspondence computation with learned pairwise scores.
- SRF architecture: SRF projects each candidate 3D point into reference images, extracts local features, and encodes them with a learned stereo function before decoding to a NeRF representation.The method uses an encoder for multi-view features and an MLP-based ensemble of similarity scores.
- Neural rendering: SRF predicts radiance at continuous locations and uses volume rendering, but conditions predictions on point image features and similarity functions rather than point coordinates alone.This combines contemporary neural rendering with classical computer vision end to end.
- Geometric intuition: A photo-consistent surface point projects to corresponding, similar-looking image regions across views, unlike a nonsurface point.This observation motivates using cross-view agreement to reason about scene surfaces.
3. Method
SRF synthesizes novel views by conditioning color and density predictions on learned multi-view image features rather than scene-specific point coordinates. Its stereo-inspired module aggregates pairwise and multi-view evidence, then volume-renders sampled points into the target image.
- 3. Method: Surface points are identified through photo-consistency: corresponding projections should match across views, while nonsurface points project to different regions.This observation motivates SRF's learned approximation of classical stereo correspondence finding.
- 3. Method: SRF projects each sampled 3D point into reference images and extracts multi-scale CNN features at the projected locations.The reference images have known camera parameters, and the resulting descriptors encode the point using image evidence.
- 3. Method: SRF applies learned pairwise similarity functions to projected feature descriptors, producing a Stereo Feature matrix without explicitly computing correspondences.A bank of K neurons processes all S = N^2 − N ordered view pairs, with each output representing a learned similarity or color-propagation signal.
- 3. Method: A convolutional multi-view stage aggregates pairwise features, then max pooling produces a fixed-dimensional encoding for varying numbers of input views.The pooled vector combines correspondence and color information beyond individual view pairs.
- 3. Method: An MLP decodes the stereo encoding into point color and density, and density-weighted volume rendering fuses sampled ray colors into the target pixel.Training uses multi-view images end-to-end with image supervision alone, without 3D data or stereo-module supervision.
4. Experiments
Experiments show that SRF generalizes to unseen scenes from sparse views, produces colored meshes without 3D supervision, and improves further with brief fine-tuning, while reflective regions remain challenging.
- Quantitative Results: Table 1 reports consistent SRF improvements over all baselines in PSNR, SSIM, and LPIPS, with fine-tuning yielding sharper geometry and appearance and fewer artifacts.PSNR and SSIM are higher-is-better metrics, while LPIPS is lower-is-better.
- Meshing Predictions: SRF produces colored meshes from 10 input images despite being trained for view synthesis without 3D supervision.The mesh is extracted by thresholding predicted density and applying Marching Cubes.
- Unconstrained Generalization: SRF generalizes to novel scenes from only 10 sparse, arbitrarily spread-out views and produces sharp rendered objects.The model is trained across scenes, unlike scene-specific NeRF optimization.
- Limitations: Challenging BRDFs and reflective regions can pose problems for stereo matching, although fine-tuning helps mitigate some issues.The authors identify view-dependent modeling as a possible future direction.
- Fine-Tuning: Around 15 minutes of fine-tuning offers a quality-speed trade-off, while convergence occurs around 90 minutes and optimization time falls from 2 days to minutes.Fine-tuning further sharpens results compared with the baselines.
- Natural Generalization: Training on one synthetic tractor for 30 minutes still enables generalization to a substantially different microphone in geometry and appearance.The authors attribute this behavior to classical stereo geometry built into the network.
5. Discussion and Conclusion
SRF combines classical stereo feature matching with neural rendering in an end-to-end architecture for view synthesis. It generalizes across scenes from sparse inputs, but reflections and texture-less regions remain challenging.
- Method: SRF predicts radiance from projected multi-view features processed in pairs, learning implicit correspondences through self-supervised rendering.This emulates feature matching in classical stereo without explicit correspondence computation.
- Limitations: Modeling reflections and texture-less regions remains challenging, although fine-tuning ameliorates but does not completely overcome this limitation.The limitation follows from SRF’s strong inspiration from classical stereo matching.
- Results: SRF generalizes across DTU scenes from 10 arbitrarily sparse, spread-out views and produces realistic novel views.The model learns common structure across multiple scenes rather than relying solely on scene memorization.
- Results: 10-15 minutes of fine-tuning on target views further improves SRF results, which significantly outperform SOTA methods trained on the new scene for 2 days.The comparison is specifically reported for the 10-view sparse setting.
- Results: SRF implicitly computes an interpretable 3D representation that supports colored meshing without 3D supervision.This extends the model beyond image synthesis to an interpretable geometric representation.