Source-linked AI summary
GRF: Learning a General Radiance Field for 3D Representation and Rendering
Alex Trevithick, Bo Yang
TL;DR
Existing 3D reconstruction methods struggle with discrete representations, 3D supervision, limited generalization, or insufficient rendering detail. GRF learns a general radiance field from sparse 2D observations by projecting local pixel features into 3D and aggregating them with attention. The method generates high-quality, realistic novel views for new objects, unseen categories, and challenging real-world scenes.
Problem
Existing methods use discrete representations, require 3D supervision, or fail to generalize and preserve rich geometric patterns across novel geometries.
Method
GRF projects local features from posed and calibrated 2D images to 3D query points, aggregates multi-view features with attention, and predicts color and volumetric density.
Results
GRF represents diverse 3D contents from 2D views with strong generalization to novel geometries and produces exceptionally realistic rendered views with fine-grained details.
Takeaways & Limitations
A single GRF model can infer high-quality novel views for new objects from both seen and unseen categories in a single forward pass.
Takeaways & Limitations
The CNN module for extracting pixel features is not optimized in this work, and depth scans could be integrated to address visual occlusions explicitly.
Abstract
from arXiv · showhide
We present a simple yet powerful neural network that implicitly represents and renders 3D objects and scenes only from 2D observations. The network models 3D geometries as a general radiance field, which takes a set of 2D images with camera poses and intrinsics as input, constructs an internal representation for each point of the 3D space, and then renders the corresponding appearance and geometry of that point viewed from an arbitrary position. The key to our approach is to learn local features for each pixel in 2D images and to then project these features to 3D points, thus yielding general and rich point representations. We additionally integrate an attention mechanism to aggregate pixel features from multiple 2D views, such that visual occlusions are implicitly taken into account. Extensive experiments demonstrate that our method can generate high-quality and realistic novel views for novel objects, unseen categories and challenging real-world scenes.
1. Introduction
GRF addresses the limited resolution, supervision demands, and poor generalization of existing 3D representations by learning a general radiance field from sparse 2D observations. It combines projected local pixel features with multi-view attention to infer detailed geometry and appearance for novel views and geometries.
- Classic SfM and SLAM pipelines reconstruct sparse, discrete point clouds that cannot capture geometric details.
- Explicit voxel, point-cloud, and mesh representations remain discrete and sparse, while their training commonly requires large-scale 3D supervision that limits generalization.
- NeRF achieves high fidelity from 2D supervision but encodes one structure in MLP weights and lacks rich geometric patterns for photo-realistic rendering across novel geometries.
- GRF takes posed and calibrated 2D images, a 3D query point, and a query viewpoint to predict that point’s RGB value and volumetric density.
- GRF extracts local features from input pixels, reprojects them to 3D query points, aggregates multi-view information with attention, and uses an MLP to infer point appearance and geometry.
- The geometry-aware attention module combines local features across views so visual occlusions are implicitly addressed for better generalization.
2. Related Work
Prior work spans classical reconstruction, explicit and implicit 3D representations, and neural rendering, but these approaches face limitations in continuity, supervision, generalization, or image sharpness. GRF instead uses multi-view feature projection and aggregation to represent continuous 3D content from 2D images.
- Classic Multi-view Geometry: SfM and SLAM extract and match hand-crafted local features before bundle adjustment, but typically reconstruct sparse, discrete point clouds rather than continuous 3D structures.
- Geometric Deep Learning: Voxel, octree, point-cloud, and mesh methods can predict realistic shapes but usually require ground-truth 3D labels and produce discrete representations.
- Neural Implicit 3D Representations: Neural implicit methods map 3D locations to occupancy, distance, or radiance fields, enabling differentiable rendering supervised by 2D images or raw 3D point clouds.
- Novel View Synthesis and Neural Rendering: Many generative novel-view methods learn the manifold of 2D images instead of exploiting underlying 3D geometry for consistent multi-view synthesis.
- GRF Pipeline: GRF projects per-pixel features from multiple input views to each 3D point, aggregates them, and feeds the result to an MLP for color and volumetric density.
- Novel View Synthesis and Neural Rendering: Neural rendering methods support reconstruction from 2D supervision for voxels, point clouds, meshes, and implicit surfaces, but often sacrifice synthesized-image sharpness.
3. GRF
GRF represents arbitrary 3D content from posed 2D images by transferring and aggregating view-specific pixel features at query points, then rendering their appearance and geometry. Its end-to-end pipeline uses CNN features, reprojection, attention aggregation, and NeRF-style volume rendering to synthesize novel views without 3D training data.
- Overview: GRF takes posed images, camera intrinsics, a query 3D point, and a viewpoint to predict that point’s RGB value and volumetric density.The network returns appearance and geometry for arbitrary locations and viewpoints in 3D space.
- Extracting General Features for 2D Pixels: The pipeline extracts robust per-pixel features with a CNN, optionally combining local high-frequency and global hierarchical information through skip connections and fully connected layers.Camera location is stacked with each pixel so features are explicitly aware of relative 3D position; camera rotation and intrinsics were not found to improve performance noticeably.
- Reprojecting 2D Features to 3D Space: For each query point, multi-view geometry reprojects features from every input image, using the nearest pixel inside an image and a zero vector when the projection falls outside.The operation treats features as representations along each light ray because paired depth scans are unavailable, and retains them in continuous 3D space.
- Obtaining General Features for 3D Points: Attention aggregation combines the reprojected features into a permutation-invariant point representation that can process an arbitrary number of input views.Shared MLPs first incorporate query-point information, producing position-aware features before aggregation.
- Obtaining General Features for 3D Points: The resulting point features preserve geometric patterns from multiple 2D observations while remaining aware of query-point location, supporting general representations for geometry and appearance.The attention-based design is intended to address visual occlusions and improve generalization.
- Rendering 3D Features via NeRF: The renderer concatenates aggregated point features with the query viewpoint, predicts color and density through NeRF-style MLPs, and integrates sampled points along rays using volume rendering.The four modules are connected and trained end-to-end, allowing novel 2D images to be synthesized using only 2D image supervision.
4. Experiments
Experiments evaluate GRF on unseen objects, categories, synthetic scenes, and real-world scenes. Results show direct generalization, faster adaptation, improved photorealism, and the importance of position-aware local features and attention.
- Generalization to Unseen Objects: GRF achieves comparable car-category performance to SRNs while inferring unseen object representations in a single forward pass.The comparison favors SRNs because SRNs is retrained on new objects, whereas GRF is not.
- Generalization to Unseen Objects: Without retraining, SRNs fails to reconstruct unseen car instances, whereas GRF generalizes directly by learning pixel-local patterns.SRNs instead generates similar cars from learned prior knowledge because its latent code is not updated for unseen objects.
- Generalization to Unseen Categories: GRF maintains similar performance on unseen ShapeNet categories, and six input views significantly improve its performance and generalization over two views.SRNs improves only marginally with six views despite retraining.
- Generalization to Unseen Scenes: GRF generalizes to novel Synthetic-NeRF scenes with complex geometries and quickly learns high-quality representations from limited new-scene iterations.Compared with NeRF trained from scratch, GRF achieves much better results given the same number of training iterations.
- Real-World Scenes: GRF surpasses NeRF on real-world scenes, especially SSIM and LPIPS, while producing fine-grained geometries with fewer artifacts.SSIM and LPIPS favor photorealism more than PSNR's average per-pixel accuracy.
- Ablation Study: Removing viewpoint information or local pixel features has the greatest ablation impact, while max-pooling is sub-optimal for addressing visual occlusions.The attention module focuses on informative visible patches, such as the windshield view where the windshield is not occluded.
5. Conclusion
The paper models 3D geometries as a general radiance field learned from sparse 2D observations. It synthesizes realistic novel views by mapping pixel features into 3D and using attention to address occlusions, while identifying better CNN features and depth scans as future directions.
- Conclusion: GRF learns robust 3D point features from sparse 2D observations by mapping pixel features into 3D and using attention to address visual occlusions.The method represents 3D geometries as a general radiance field and synthesizes realistic novel views.
- Conclusion: Future work includes more advanced CNN modules for better pixel features and depth scans for explicitly addressing visual occlusions.These are the paper's stated limitations and proposed directions.
A.1. Details of Network Architecture
The appendix specifies dataset-specific CNN and attention modules, Slot Attention settings, positional embeddings, and common training and view-selection procedures.
- CNN Module: The CNN module uses separate configurations for ShapeNetv2, Synthetic-NeRF, and the real-world dataset, with average pooling for the latter two.The cited table descriptions identify the dataset-specific modules and state that average pooling aggressively downsamples feature maps.
- Attention Module: AttSets is used for ShapeNetv2 attention experiments because its computational efficiency supports large-scale training.
- Attention Module: Slot Attention uses two slots, two iterations, and hidden size 128, producing a flattened 256-dimensional vector.
- Embedding: Inputs use positional embedding length L = 5, except rotation, which is converted to a quaternion and embedded at L = 4.
- Training: Training uses Adam for 200k-300k iterations, with learning rates varying by dataset and real-world scene.Each gradient step uses 1000 ShapeNetv2 rays with 32 coarse and 64 fine samples.
- Testing: ShapeNetv2 testing supplies the four closest views by cosine similarity to the desired novel view.
A.2. Details of Experimental Results on the Synthetic-NeRF Dataset in Section 4.3
The Synthetic-NeRF appendix reports multi-scene and novel-scene comparisons using PSNR, SSIM, and LPIPS, with GRF evaluated against established baselines.
- Multi-scene learning: Table 10 evaluates GRF trained simultaneously on four Synthetic-NeRF scenes for multi-scene learning.SRNs, NeRF, and NSVF scores are included for comparison despite being trained on single scenes.
- Novel-scene evaluation: Table 11 compares GRF and NeRF on four novel Synthetic-NeRF scenes in the Group 1&2 experiments.
A.3. Details of Experimental Results on the real-world dataset (3DScan) [6] in Section 4.3
The 3DScan experiments test GRF on four 360-degree-scanned chair scenes, using shared training followed by limited finetuning on a held-out scene.
- Experimental setup: GRF is pretrained for 100000 iterations on 100 images from each of three 3DScan chair scenes, then finetuned on a fourth scene using 50 sparse views.The scenes are identified by indices 00032, 00027, 00279, and 00169.
- Visualization: Figure 11 presents quantitative and qualitative novel-view-synthesis results for a real-world chair after finetuning.
A.4. Details of experimental results on the real-world dataset (LLFF) in Section 4.4.
The real-world and Synthetic-NeRF evaluations compare GRF with multiple radiance-field and view-synthesis baselines using standard image-quality metrics.
- Real-world evaluation: Table 12 compares GRF, SRNs, LLFF, and NeRF on the real-world dataset for single-scene learning.The listed metrics are PSNR, SSIM, and LPIPS.
- Synthetic-NeRF evaluation: Table 13 compares GRF, SRNs, NV, NeRF, and NSVF on Synthetic-NeRF using PSNR, SSIM, and LPIPS.PSNR is reported in dB.
- Synthetic-NeRF results: GRF outperforms NSVF on both PSNR and SSIM in the Synthetic-NeRF single-scene experiments.
A.5. Analysis of Attention Mechanism
The attention analysis visualizes which input view contributes most to each rendered pixel, showing that GRF selects informative patches and handles self-occlusion.
- Attention analysis: GRF computes feature vectors and attention masks for multiple input views at surface points along queried rays.Surface points are selected using predicted volume density before features and masks are computed.
- Attention analysis: For each rendered pixel, the view with the greatest summed attention score is identified as the main contributor to its RGB prediction.The resulting selections form a Max Attention Map aligned with the rendered RGB image.
- Visualization: The Max Attention Map assigns each rendered pixel to the highest-attention input view using color-coded correspondences.Purple, green, and blue indicate input images #1, #2, and #3, respectively.
- Interpretation: GRF focuses on the most informative pixel patch and handles occlusion by selecting the view where the car windshield is visible.For the windshield, attention favors input image #2 and ignores images #1 and #3, where it is self-occluded.
A.6. Generalization to Visual Occlusions and Variable Input Images
GRF is evaluated with one, two, five, and ten input views to test generalization under sparse observations and severe visual occlusions. Even with one view it recovers general unseen-object shape, while additional views improve reconstruction of originally occluded parts.
- Experimental setup: The attention modules aggregate features from an arbitrary number of input views, enabling evaluation across varying view counts.The experiments specifically target sparse views with severe visual occlusions.
- 1-view Reconstruction: 1-view reconstruction feeds one image of a novel car into a model trained with five images per object and renders nine new views.This setting leaves most of the object self-occluded.
- 2-view Reconstruction: 2-view reconstruction supplies two images of the novel car and renders the same nine novel views, leaving many parts occluded.Compared with one view, the network receives more information while occlusion remains substantial.
- 5-view / 10-view Reconstruction: 5-view and 10-view reconstruction feed five and ten views of the novel object while rendering the same set of new images.These settings provide progressively more visual input for comparison.
- Analysis: Even with one view, GRF recovers an unseen object's general 3D shape and visually occluded parts, while more views progressively improve occluded-region reconstruction.The authors attribute one-view recovery primarily to CNN-learned hierarchical high-level shape features.
- Real-world results: Figures 14 and 15 present qualitative novel-view depth and RGB estimation results on the real-world dataset.The figures evaluate both depth and RGB outputs in the real-world setting described in Section 4.4.