Source-linked AI summary
Neural Point-Based Graphics
Kara-Ali Aliev, Artem Sevastopolsky, Maria Kolos, Dmitry Ulyanov, Victor Lempitsky
TL;DR
The paper addresses the lengthy and brittle pipeline required to model real scenes with explicit surfaces and meshes. It represents scenes as point clouds with learned descriptors, renders their multi-resolution projections through a neural network, and reports photorealistic views while avoiding surface estimation and meshing, including for challenging thin parts.
Problem
Real-scene modeling often requires noisy point-cloud processing, surface reconstruction, meshing, and material estimation, motivating a simpler representation and rendering pipeline.
Method
The method attaches learned descriptors encoding local geometry and appearance to raw point-cloud points and jointly learns a rendering network that transforms their rasterizations into views.
Results
The approach models complex RGB-D and RGB scenes, achieving similar quality to a mesh-based analog and surpassing it where meshing is problematic, such as thin parts.
Takeaways & Limitations
Point clouds can serve as geometric proxies for neural rendering, with deep rendering networks handling missing connectivity, geometric noise, and holes.
Takeaways & Limitations
The model cannot currently fill very big geometry holes realistically and has not been evaluated for dynamic scenes involving motion or relighting.
Abstract
from arXiv · showhide
We present a new point-based approach for modeling the appearance of real scenes. The approach uses a raw point cloud as the geometric representation of a scene, and augments each point with a learnable neural descriptor that encodes local geometry and appearance. A deep rendering network is learned in parallel with the descriptors, so that new views of the scene can be obtained by passing the rasterizations of a point cloud from new viewpoints through this network. The input rasterizations use the learned descriptors as point pseudo-colors. We show that the proposed approach can be used for modeling complex scenes and obtaining their photorealistic views, while avoiding explicit surface estimation and meshing. In particular, compelling results are obtained for scene scanned using hand-held commodity RGB-D sensors as well as standard RGB cameras even in the presence of objects that are challenging for standard mesh-based modeling.
1 Introduction
The paper targets the brittle, multi-stage pipeline for modeling real scenes by replacing explicit surface processing with a learned point-based neural rendering system. It evaluates the approach on RGB-D and RGB captures, including scenes difficult for mesh-based methods.
- Motivation: Classical scene modeling combines scanning, point-cloud cleanup, surface reconstruction, meshing, texturing, material estimation, and view synthesis.The pipeline must process noisy and incomplete point clouds before producing photorealistic views.
- Approach: The proposed system uses raw point clouds, learned neural descriptors, and a deep convolutional network to generate photorealistic novel views.Descriptors encode geometric and photometric properties and are learned jointly with the rendering network.
- Approach: The approach eliminates many classical pipeline steps, including surface estimation and meshing.It retains the point cloud as the scene geometry representation.
- Evaluation: The method models scenes captured with handheld RGB-D cameras and simple RGB streams reconstructed using structure-from-motion or similar techniques.The experiments include comparisons with ablations and competing approaches.
- Conclusion: The results suggest that 3D points can provide sufficient geometric proxies for neural rendering in many cases.This conclusion is stated in the context of modern deep networks and the reported experiments.
2 Related work
The work combines point-based graphics, image-based rendering, and neural rendering while replacing explicit surface attributes with learned point descriptors. Its key distinction from closely related neural rendering is the use of point-cloud geometry without surface estimation or meshing.
- Point-based graphics: Point-based graphics traditionally renders point clouds using oriented disks called surfels, with estimated orientations and radii combined through splatting.Earlier work also explored replacing linear splatting with deep convolutional networks.
- Point-based graphics: The proposed method keeps 3D points as primitives and encodes local geometric and photometric parameters in learned neural descriptors.It does not explicitly use surface orientation, disk radii, or color during rasterization.
- Deep image based rendering: Image-based rendering methods commonly warp existing photographs using coarse scene-geometry approximations and neural networks.These methods combine warped images or post-process warping results with learned models.
- Deferred neural rendering: Deferred neural rendering jointly learns neural textures and a convolutional rendering network, making it the approach most related to this work.Both methods learn representations alongside neural rendering networks.
- Deferred neural rendering: Unlike deferred neural rendering, this approach uses point-based geometry and avoids surface estimation and meshing.The paper reports extensive comparisons with deferred neural rendering and discusses the relative advantages and disadvantages.
3 Methods
The method rasterizes a descriptor-augmented point cloud from a target camera at multiple resolutions, then uses a neural rendering network to synthesize an RGB image. It learns both descriptors and network parameters by minimizing image mismatch against training views.
- Rendering: A point cloud P with descriptors D is projected into a target camera view, using descriptors as pseudo-colors and z-buffering to remove occluded points.The resulting raw image S(P, D, C) has M descriptor channels over a W × H pixel grid.
- Progressive rendering: Progressive rendering creates T raw images at different spatial resolutions instead of using splatting with estimated disk radii.Higher resolutions preserve more detail but suffer stronger bleeding, while lower resolutions reduce bleeding at the cost of geometric detail.
- Rendering network: The rendering network Rθ maps all multi-resolution raw images into a three-channel RGB image.The network is based on a gated-convolution U-Net that integrates raw images at their respective resolutions.
- Model creation: Model fitting compares rendered and ground-truth RGB images using a mismatch objective, implemented with a perceptual loss based on pretrained VGG activations.Training scenes provide point clouds, ground-truth images, and known camera parameters.
- Model creation: Optimization updates both rendering-network parameters and scene-specific neural descriptors through backpropagation, using ADAM.The rendering network can be pretrained on similar scenes and then fine-tuned to a new scene.
4 Experiments
Experiments evaluate the approach across RGB-D, professional-camera, and smartphone-captured scenes using holdout views and comparisons with neural, mesh-based, and ablated methods. Results generally favor learnable neural descriptors, especially where meshing is difficult, while revealing flickering under camera misregistration.
- Datasets and protocol: The evaluation covers ScanNet rooms, professionally captured people, and smartphone-captured Owl and Plant objects.Experiments use multiple scene types and compare methods on held-out views.
- Datasets and protocol: The two-stage protocol pretrains on scene families, fine-tunes on new-scene frames, and evaluates on dissimilar holdout viewpoints.Holdout views are excluded from fitting and selected to differ sufficiently from fine-tuning views.
- Baselines: 1.96M parameters and 62ms per FullHD image characterize the lightweight rendering network used across approaches.The timing is reported on a GeForce RTX 2080 Ti.
- Baselines: Comparisons include the proposed method, color-only point rasterization ablations, neural rerendering variants, textured meshes, and deferred neural rendering.The compared systems use shared rendering-network training conditions where applicable.
- Results: Learnable neural descriptors generally outperform other methods, sometimes by a large margin, across the quantitative and qualitative comparisons.The reported exception is the VGG metric on Person 1.
- Results: The proposed point-based method performs better when meshing is problematic, including thin-object scenes such as Studio and Plant, whereas DNR benefits from good meshes.When mesh artefacts are relatively rare, the proposed method can look visually similar to DNR.
- Limitations: Camera misregistrations can produce flickering during camera motion; supersampling and bilinear downsampling reduce it at the cost of barely noticeable blur.The anti-aliasing procedure does not affect the resolution at which neural rendering is performed.
5 Discussion
The approach uses 3D points with learned local descriptors and a rendering network to model complex scenes. It matches mesh-based rendering quality while remaining limited by large geometry holes and untested dynamic scenes.
- Composed scenes can be created by translating and rotating a point cloud while preserving the objects' world scale.
- Each 3D point carries a local descriptor encoding geometry and appearance for neural rendering.
- A rendering network converts point rasterizations, using learned descriptors as pseudo-colors, into realistic views.
- The method handles missing connectivity, geometric noise, and holes gracefully through the deep rendering network.
- Rendering quality is similar to mesh-based analogs and surpasses them where meshing is problematic, such as thin parts.
- Very large geometry holes cannot yet be filled realistically, and dynamic scenes with motion or relighting remain uninvestigated.