Source-linked AI summary
Point-Based Neural Rendering with Per-View Optimization
Georgios Kopanas, Julien Philip, Thomas Leimkühler, George Drettakis
TL;DR
Existing neural-rendering approaches either cannot recover from MVS geometry errors or incur expensive volumetric training and inference. The paper introduces an MVS-initialized, per-input-view optimized point-based renderer with differentiable splatting, probabilistic depth testing, and camera selection, achieving broad quality and application benefits while retaining stated limitations in difficult reconstruction cases.
Problem
Neural renderers based on MVS cannot recover from reconstruction errors, while volumetric neural representations suffer from expensive training and inference.
Method
The paper combines MVS initialization with per-input-view optimization of depth and reprojected features in a differentiable point-based framework using camera selection, bi-directional EWA splatting, and probabilistic depth testing.
Results
The renderer outperforms all tested previous view-synthesis methods in quality and speed in almost all presented scenes, and supports IBR, harmonization, and stylization.
Takeaways & Limitations
The pipeline combines stable MVS-derived 3D information with neural optimization in input views for novel-view synthesis, harmonization, and consistent stylization.
Takeaways & Limitations
The method may fail when large geometry is missing, when scenes contain reflections or transparency requiring multiple depths, or when MVS reconstruction completely fails.
Abstract
from arXiv · showhide
There has recently been great interest in neural rendering methods. Some approaches use 3D geometry reconstructed with Multi-View Stereo (MVS) but cannot recover from the errors of this process, while others directly learn a volumetric neural representation, but suffer from expensive training and inference. We introduce a general approach that is initialized with MVS, but allows further optimization of scene properties in the space of input views, including depth and reprojected features, resulting in improved novel-view synthesis. A key element of our approach is our new differentiable point-based pipeline, based on bi-directional Elliptical Weighted Average splatting, a probabilistic depth test and effective camera selection. We use these elements together in our neural renderer, that outperforms all previous methods both in quality and speed in almost all scenes we tested. Our pipeline can be applied to multi-view harmonization and stylization in addition to novel-view synthesis.
1. Introduction
The paper presents a differentiable point-based neural-rendering pipeline that starts from MVS reconstruction and optimizes per-input-view properties to improve novel-view synthesis. Its components support efficient, temporally coherent rendering across free-viewpoint IBR, harmonization, and stylization.
- Motivation: MVS-initialized neural rendering optimizes per-input-view depth and reprojected features, combining explicit geometry with neural optimization.The joint optimization seeks a compromise between correcting depth errors and reducing blur artifacts in novel views.
- Method: The neural renderer optimizes per-input-view attributes before producing temporally consistent renderings for a given scene.Optimizable attributes include reprojected features and depth.
- Applications: The framework supports free-viewpoint IBR, multi-view color harmonization, and multi-view consistent image stylization.These applications use the same per-view optimization framework across multiple multi-view imaging tasks.
- Method: The framework introduces camera selection, differentiable point splatting with bi-directional EWA filtering, and probabilistic per-view depth testing.Together, these components form the point-based multi-view imaging framework and neural renderer.
- Results: The neural renderer outperforms all tested previous view-synthesis methods in quality and speed in almost all presented scenes.The reported comparison uses both quantitative measures and visual quality.
2. Related Work
Related work spans traditional IBR, MVS-guided and learned neural rendering, point-based methods, and multi-view stylization and harmonization. The paper positions its approach around per-view information, efficient camera selection, and difficult wide-baseline capture settings.
- Traditional IBR: Traditional IBR reprojects and blends input images using approximate geometry, with SfM and MVS enabling automatic 3D proxies for free-viewpoint navigation.Per-view methods improve unreliable depth by representing information separately in each input view.
- Traditional IBR: Automatic rendering quality for difficult cases such as vegetation and thin structures remains a focus of specialized IBR methods.The paper reports improvement over other automatic methods without user input for these cases.
- Camera Selection: Camera selection reduces the input views used for novel-view projection because large datasets cannot fit all views within GPU memory limits.The paper builds on K-Maximum coverage to seek fast and accurate camera selection based on pixel coverage.
- Neural Rendering: MVS-guided neural methods refine geometry, learn blend weights, or project features using MVS geometry and per-view depth maps.Follow-up work also uses MVS meshes to project features more stably.
- Point-Based Rendering: Point-based neural rendering supports feature reprojection, but prior methods either assume point positions are correct or use points to compute differentiable-rendering gradients.The paper distinguishes its jointly optimized point-based pipeline from these approaches.
- Neural Rendering: Neural radiance and video-based methods offer promising rendering results but can produce artifacts in sparse wide-baseline scenes or require involved capture setups.The paper specifically reports artifacts for NeRF methods in wide-baseline scenes, especially vegetation.
- Stylization and Harmonization: The approach also relates to multi-view stylization and harmonization, where wide-baseline inputs lack the reliable optical flow often used for video stylization and view-dependent effects may be undesirable to remove.Harmonization can improve textured meshes from MVS reconstructions while preserving view-dependent effects for IBR.
3. Differentiable Multi-View Rendering with Per-View Optimization
The framework enables differentiable optimization of per-view attributes through camera selection, bi-directional point splatting, alpha blending, uncertainty modeling, and probabilistic depth testing.
- Per-view optimization adjusts projected features, depth, and other attributes within a neural-rendering framework.
- 3.1. Camera Selection: N = 4 cameras expose the coverage advantage of the proposed selection method over [RK20] in the Ponche scene.The comparison shows [RK20] missing lower-right content while the proposed selection covers the frame.
- 3.2. Bi-Directional Differentiable Point Cloud Rasterization: Bi-directional EWA splatting combines inverse lifting and forward projection Jacobians to reproject input-view samples into the novel view differentiably.The procedure supports per-view optimization while accounting for the combined stretch of both transformations.
- 3.2. Bi-Directional Differentiable Point Cloud Rasterization: Alpha blending and uncertainty-aware splats soften point boundaries, overlaps, and geometric uncertainty during differentiable rasterization.Uncertainty scales the Gaussian covariance and can itself be refined during optimization.
- 3.3. Probabilistic Depth Testing for Point-Based Rendering: Probabilistic depth testing resolves uncertain visibility after rasterization using depth distributions from separately reprojected input views.Soft decisions allow points that do not directly intersect a ray to contribute depth information, avoiding losses from hard visibility under noisy MVS geometry.
4. Temporally Consistent Neural Rendering and Optimization
The method combines differentiable point splatting with per-view optimization and weighted feature pooling to produce temporally stable neural renderings. An interactive approximation accelerates point reprojection while preserving visual quality with only marginal degradation.
- Bi-directional point splatting supports a neural renderer for multi-view rendering and per-view attribute optimization.
- Each input view is encoded separately, weighted and pooled, then decoded into the final rendering.The pooled representation uses 64 channels and supports visibility-aware weighting.
- Camera selection uses temporal filtering and weight refinement to reduce popping when views enter or leave the selected set.The method uses λ = 0.05 for temporal filtering and selects N = 9 views unless otherwise stated.
- Texture-stretch weights favor front-facing views, while probabilistic depth weights account for point visibility.The combined weights include camera-selection, texture-stretch, and probabilistic-depth terms.
- The weighted-average pooling improves temporal stability and permits changing the number of selected views without retraining.Normalization keeps the feature magnitude distribution consistent for the decoder as views are added.
- Per-view colors, depth, normals, uncertainty, and six latent features are jointly optimized to improve rendering.The latent features extend the input color channels and are initialized to 0.5.
- At runtime, an OpenGL point-splatting approximation uses 10 depth layers for interactive rendering, with only marginal quality degradation.The approximation reduces the probabilistic depth test to at most 10 alpha-weighted depth samples.
5. Applications and Results
The framework supports free-viewpoint image-based rendering, multi-view harmonization, and style transfer. It produces sharp results in difficult structures, while optimized brightness coefficients address exposure inconsistencies across views.
- Applications: The pipeline is demonstrated for image-based rendering, multi-view harmonization, and multi-view style transfer.
- Implementation: The interactive implementation integrates custom CUDA and OpenGL point-splatting components for training, accurate rendering, and display.
- Image-Based Rendering: The image-based renderer produces sharp vegetation details and recovers some artifacts caused by thin structures in reconstructed geometry.Examples include vegetation and thin structures in the evaluated scenes.
- Multi-View Harmonization: The method harmonizes multi-view imagery by optimizing a per-view brightness coefficient multiplied with each input image's color.The approach targets exposure and other camera-parameter differences that create inconsistencies between captured views.
- Multi-View Harmonization: A regularization term discourages the degenerate solution in which all brightness coefficients become zero.The regularization weight is λ = 0.2.
- Multi-View Style Transfer: Style transfer jointly optimizes input-image colors with photo-consistency, style, and content terms across views.The method demonstrates the effect on five Museum images.
6. Evaluation
Evaluation shows strong visual and quantitative performance across novel-view synthesis, while ablations clarify how depth testing, feature optimization, and geometry-related parameters address distinct artifacts. The method remains limited by overfitting, view-dependent effects, and severely erroneous or incomplete reconstructions.
- IBR and Neural Rendering: Our method achieves better overall visual quality across the evaluated scenes and methods, with sharper vegetation and improved thin-structure details.NeRF++ is comparable in some cases and performs better for thin structures in Stairs.
- Quantitative Evaluation: Our method obtains the best scores for all metrics and all scenes in the quantitative leave-one-out evaluation.The authors caution that these metrics do not always correspond well with visual perception because ULR shows visual artifacts despite ranking highly.
- Ablations: Disabling the probabilistic depth test produces incorrect transparent regions, demonstrating its role in resolving visibility during soft rasterization.The ablation is illustrated on the Museum scene.
- Ablations: Reprojected features sharpen rendering, especially for vegetation and railings, while jointly optimized depth, normals, and uncertainty handle different reconstruction artifacts.The framework does not correct depth alone; its strength is optimizing these components together for a compromise between geometry errors and sharp rendering.
- Efficiency Ablations: Reducing model size preserves high-quality results, while reducing the number of input cameras causes minimal quality degradation depending on the scene.The main constraint on camera count is inference memory, particularly in the challenging Salon scene.
- Limitations: Overfitting can cause visible temporal flickering, although using 20K rather than 40K iterations significantly reduces it.This behavior is reported for the Hugo scene.
- Limitations: Normals and depth are optimized separately, and enforcing consistency between them may improve geometry optimization.Correctly modeling view-dependent effects remains an open challenge requiring reflection flow.
- Limitations: If the initial MVS reconstruction is severely erroneous, optimization may fail similarly to other methods, including missing geometry, reflections, transparency, or complete reconstruction failure.Such cases can also produce temporal instability.
7. Conclusion
The paper presents a differentiable point-based pipeline that optimizes per-input-view properties for neural rendering, with applications in image-based rendering, harmonization, and stylization. Its combination of probabilistic depth testing and efficient camera selection supports temporally consistent rendering and improves novel-view synthesis.
- The pipeline enables per-input-view optimization through differentiable splatting, probabilistic depth testing between input views, and efficient camera selection.These components are combined to define a temporally consistent neural renderer.
- For image-based rendering, the method optimizes depth, color, and reprojected features, improving neural rendering quality in regions such as vegetation.
- Multi-view harmonization improves exposure-related inconsistencies, while multi-view stylization achieves consistency across views.
- The results combine stable 3D information from multi-view stereo with neural optimization in input views to improve novel-view synthesis.