Source-linked AI summary
Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering
Bangbang Yang, Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng Zhang, Zhaopeng Cui
TL;DR
Existing neural rendering methods generally encode scenes without object identity, limiting realistic high-level editing. The paper learns an object-compositional neural radiance field with scene and object pathways plus scene-guided training for occluded clutter. It reports competitive static novel-view synthesis and realistic object-level editing in real-world scenes.
Problem
Existing neural rendering methods usually encode an entire scene as a whole, limiting object-level editing such as moving or adding furniture.
Method
A two-pathway architecture encodes scene geometry and appearance separately from standalone objects conditioned on learnable object activation codes, with scene-guided training for occlusions.
Results
The system achieves competitive static-scene novel-view synthesis and produces realistic rendering for object-level editing in clustered real-world scenes.
Takeaways & Limitations
The learned representation supports high-quality novel views while allowing multiple objects to be moved, rotated, or duplicated.
Takeaways & Limitations
The method relies on spatial smoothness to render unseen textures beneath objects because observations are lacking.
Abstract
from arXiv · showhide
Implicit neural rendering techniques have shown promising results for novel view synthesis. However, existing methods usually encode the entire scene as a whole, which is generally not aware of the object identity and limits the ability to the high-level editing tasks such as moving or adding furniture. In this paper, we present a novel neural scene rendering system, which learns an object-compositional neural radiance field and produces realistic rendering with editing capability for a clustered and real-world scene. Specifically, we design a novel two-pathway architecture, in which the scene branch encodes the scene geometry and appearance, and the object branch encodes each standalone object conditioned on learnable object activation codes. To survive the training in heavily cluttered scenes, we propose a scene-guided training strategy to solve the 3D space ambiguity in the occluded regions and learn sharp boundaries for each object. Extensive experiments demonstrate that our system not only achieves competitive performance for static scene novel-view synthesis, but also produces realistic rendering for object-level editing.
1. Introduction
The paper targets editable, realistic rendering of cluttered real-world scenes, where existing scene-level neural representations lack object identity and do not readily support manipulation. It introduces a unified object-compositional model trained from posed images and rough 2D instance masks.
- Motivation: Existing neural rendering methods encode scenes globally or densely without reliably supporting object-level edits such as moving or adding furniture.The motivating editing question is whether realistic rendering can be preserved after scene modification.
- Approach: The proposed system learns a unified neural rendering model that respects real-world object placement while rendering manipulated objects standalone.Objects can be moved, rotated, or duplicated and rendered from novel viewpoints.
- Contribution: The two-pathway architecture combines a scene branch for scene rendering with an object branch for object-compositional rendering and occlusion-aware learning.The scene branch also supports seamless whole-scene rendering and assists object learning.
- Challenge: Rough 2D instance masks create 3D ambiguity in occluded regions because masked-out rays may correspond to hidden object parts or empty space.This ambiguity can produce over-suppressed or fragmented object radiance fields.
- Results: Experiments report standard novel-view synthesis performance on par with or better than state-of-the-art methods while retaining high-quality editable rendering.The claimed scope includes clustered, real-world scenes and object manipulation.
2. Related Work
The related work positions neural rendering as a broad family of learned scene representations for novel-view synthesis, including voxels, point clouds, meshes, multi-plane images, and implicit functions.
- Neural Rendering: Neural rendering learns novel-view synthesis from 2D images using representations including voxels, point clouds, textured meshes, multi-plane images, and implicit functions.The paper surveys multiple representation families before focusing on implicit neural rendering.
- Implicit Representations: SRN maps spatial coordinates to feature vectors with MLPs and uses differentiable ray marching to generate images.It represents a continuous scene as an opaque surface.
- Implicit Representations: NeRF represents scenes with volume density and view-dependent color in implicit fields for photorealistic novel-view synthesis.
3. Method
The method jointly learns scene and object radiance fields through two pathways, scene-guided supervision, and a combined training objective. At rendering time, object-specific fields are transformed and composited with the background to support editing.
- 3.1. Overview: The scene branch encodes whole-scene geometry and appearance, while the object branch encodes standalone objects conditioned on learnable activation codes.The framework can learn multiple objects simultaneously without training a separate model for each object.
- 3.2. Network Architecture: The two branches use hybrid voxel and coordinate embeddings, combining interpolated voxel features with positional encodings of sampled points and ray directions.The scene pathway uses scene voxel features, spatial coordinates, and viewing directions as inputs.
- 3.3. Object-Compositional Learning: Object supervision uses 2D instance masks to encourage opacity on the target object and transparency elsewhere while matching rendered object colors.The supervision includes balanced weighting for mask signals.
- 3.3. Object-Compositional Learning: Object colors and opacities are rendered by transmittance-weighted ray integration, with alpha determined from object density and sampling distance.The rendered object opacity is the accumulated transmittance-weighted alpha along each ray.
- 3.3. Object-Compositional Learning: Scene guidance uses scene-branch transmittance to bias object-branch sampling so occluded object regions receive appropriate supervision.This addresses incomplete masks caused by foreground occlusion without simply discarding empty-mask supervision.
- 3.4. Joint Optimization: The total training loss is the sum of scene and object losses optimized jointly for the two branches.The scene loss matches predicted and true pixel colors, while the object branch uses the object-supervision loss.
- 3.5. Editable Scene Rendering: Editable rendering removes target-region samples from the background stage, transforms object-specific radiance fields, and composites all components by ray-distance ordering.This pipeline supports user-defined object movement, rotation, and duplication.
4. Experiments
Experiments evaluate full-scene rendering, individual-object rendering, editable scene manipulation, and ablations on ToyDesk and ScanNet. The method delivers competitive scene quality, sharp standalone objects, realistic edits, and improved object rendering from scene guidance and 3D guard masks.
- Experimental setup: The evaluation uses two real-world datasets and compares scene rendering against neural point-cloud, NeRF, and voxel-based methods, followed by editing and ablation studies.ToyDesk includes posed images, 2D instance segmentation, frequent toy occlusions, recovered camera poses and meshes, and manually labeled target-object meshes.
- Scene rendering: PSNR, SSIM, and LPIPS results show the method is comparable to or better than state-of-the-art methods on ToyDesk and ScanNet.On ToyDesk, NeRF-based methods perform worse with far-away backgrounds, while the proposed hybrid space embedding preserves central-object details and surrounding textures.
- Individual-object rendering: Direct 3D masks produce mottled objects or include background textures, whereas the object branch is designed to render standalone objects cleanly.Incomplete and inaccurate reconstructed meshes make strict surface sampling sparse, relaxed sampling include background, and bounding-box clipping include still more background texture.
- Scene editing: On ToyDesk, rotating, moving, and duplicating objects produces vivid standalone renderings with sharp boundaries and seamless integration into the scene.The edited objects preserve the correct spatial relationship with the surrounding scene.
- Scene editing: Compared with NPCR, the method moves objects while keeping background textures nearly unchanged and avoiding reported artifacts such as false occlusions, holes, and cloudy textures.The method independently outputs target-object radiance fields and uses volumetric rendering for realistic edits with intact textures and correct spatial relationships.
- Ablation study: Scene guidance and the 3D guard mask significantly improve target-object rendering quality and enable intact object radiance fields despite frequent partial occlusion.The ablation evaluates two ScanNet scenes with foreground-furniture occlusions using masked, tightly cropped object regions.
- Rendered segmentation: Rough 2D instance masks can yield smooth, accurate rendered segmentation while preserving fine object details after multi-view training converges.The paper attributes resistance to single-view mask noise and geometry cues from the converged 3D structure learned from images.
5. Conclusion and Future Works
The framework supports high-fidelity novel-view synthesis and editable rendering of real-world scenes using posed images and rough 2D instance masks. Future improvements target unseen textures, camera-related artifacts, and scene lighting.
- The method renders novel views of real-world scenes while supporting manipulation of multiple objects, including moving, rotating, and duplicating them.
- Unseen textures beneath objects currently rely on network spatial smoothness because observations are lacking.Scene completion methods are suggested as a possible improvement.
- Camera-pose and ray-direction optimization could reduce pose noise and rolling-shutter effects to produce clearer backgrounds.
- Integrating a scene-lighting model is identified as a future direction for more realistic scene editing.