Source-linked AI summary
Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering
Bangbang Yang, Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng Zhang, Zhaopeng Cui
TL;DR
Neural rendering raises the question of whether furniture can be moved or added while maintaining realistic rendering. This paper introduces an object-compositional scene model that achieves on-par or better standard novel-view synthesis while supporting high-quality editable rendering.
Problem
A key open question is whether neural rendering can modify scenes, such as by moving or adding furniture, while retaining realistic rendering.
Source: §1 · p. 1Method
A two-pathway model encodes scene appearance and standalone objects conditioned on learnable activation codes, with scene guidance addressing occlusion ambiguity.
Source: §1 · p. 2, §3.1 · p. 3, §3.3 · p. 4Results
The system performs on-par or better than state-of-the-art methods for standard novel-view synthesis while maintaining high-quality editable scene rendering.
Source: §1 · p. 2Takeaways & Limitations
The framework supports rendering individual objects for manipulation without training a separate model for each object.
Source: §3.1 · p. 3Because observations are lacking, the method relies on network spatial smoothness to render unseen textures beneath objects.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Implicit neural rendering techniques have shown promising results for novel view synthesis. However, existing methods usually encode the entire scene as a whole, which is generally not aware of the object identity and limits the ability to the high-level editing tasks such as moving or adding furniture. In this paper, we present a novel neural scene rendering system, which learns an object-compositional neural radiance field and produces realistic rendering with editing capability for a clustered and real-world scene. Specifically, we design a novel two-pathway architecture, in which the scene branch encodes the scene geometry and appearance, and the object branch encodes each standalone object conditioned on learnable object activation codes. To survive the training in heavily cluttered scenes, we propose a scene-guided training strategy to solve the 3D space ambiguity in the occluded regions and learn sharp boundaries for each object. Extensive experiments demonstrate that our system not only achieves competitive performance for static scene novel-view synthesis, but also produces realistic rendering for object-level editing.
1. Introduction
Neural rendering can produce realistic views from posed images, but existing scene representations do not support object-level editing well. The paper proposes a unified object-compositional rendering system that addresses occlusion ambiguity and supports realistic scene and object rendering.
Motivation and limitations
- Whole-scene neural networks such as NeRF [17] and SRN [26] are hard to scale to large scenes, while volumetric methods [12] lack object identity and thus do not support high-level editing.Virtual tours motivate the task, and neural rendering offers realistic images learned directly from posed-image collections.Source: §1 · p. 1
Editable scene rendering
- A top-down unified model learns real-world object placements from posed scene images and supports moving, rotating, or duplicating objects without separate object-capture training.This contrasts with OSF [7], which learns one model per object and requires images of each object captured beforehand; the paper says this requirement limits OSF to synthetic data for cluttered scenes.Source: §1 · pp. 1–2
Object-compositional architecture
- The two-pathway architecture renders objects standalone for novel viewpoints or new locations, while a non-editable scene branch supplies biased ray sampling and dense depth to handle occluded regions.The scene branch also renders content omitted from instance segmentation, enabling seamless whole-scene rendering; sharp object boundaries are needed to avoid background bleeding.Source: §1 · p. 2
Occlusion ambiguity
- Rough 2D masks create 3D ambiguity because occluded object locations and locations outside the object are both unlabeled, which can cause over-erasure and cloudy renderings.The scene branch identifies occluded regions where no gradient should be applied rather than supervising them as empty space.Source: §1 · p. 2
Contributions
- The system performs on-par or better than state-of-the-art methods for standard novel-view synthesis while retaining high-quality editable rendering.The authors describe it as the first editable neural scene rendering system trained from posed images and 2D instance masks.Source: §1 · p. 2
2. Related Work
Prior neural rendering methods often encode scenes as a whole, while object-decomposite approaches have limitations in real-world settings; this work supports standalone object rendering and manipulation in real-world scene scans.
Neural Rendering
- Neural rendering spans voxel, point-cloud, mesh, multi-plane-image, and implicit-function representations, but whole-scene encoding makes individual-object rendering nontrivial.SRN [26] maps space coordinates to features for differentiable ray marching, NeRF [17] uses density and view-dependent color, and NSVF [12] uses sparse voxel octrees.Source: §2 · p. 2
Object-Decomposite Rendering
- Traditional modeling and rendering pipelines supported editing and novel-view synthesis, including synthetic-object insertion and real–synthetic compositing with captured object light fields.Karsch et al. estimate environment lighting, while Cossairt et al. use a light-field interface and specific capture hardware.Source: §2 · p. 2
- Neural object-decomposite methods are limited by single-object training [7] or reliance on dynamic street scenes and class-specific canonical coordinates [19], unlike this method's multi-object manipulation in real-world scans.The proposed method avoids canonical object coordinates and learns a compact object-compositional model.Source: §2 · p. 2
3. Method
The method combines scene- and object-specific radiance fields, learning object representations from instance masks while using scene geometry to handle occlusion, then renders editable scenes by recombining background and transformed objects.
3.1. Overview
- The two-pathway architecture encodes scene geometry and appearance in a scene branch and standalone objects in an object branch conditioned on learnable activation codes.Shuffled activation codes let the framework learn multiple objects simultaneously, without separate training for each object.Source: §3.1 · pp. 2–3, Figure 2 · p. 3
3.2. Framework of Object-Compositional NeRF
- A hybrid embedding combines positional encodings of coordinates, interpolated scene voxel features, and ray directions; the object branch additionally receives shared object voxel features and an object-specific activation code.The scene branch predicts scene color and opacity, while the conditioned object branch predicts the selected object's color and opacity with other regions empty.Source: §3.2 · p. 3
3.3. Object-Compositional Learning
- Instance masks supervise object opacity and masked color, while scene-guided sampling and a 3D guard mask protect occluded regions from erroneous empty-mask gradients.The guard mask is built beyond online scene depth pushed forward by ϵ, excluding visible instance space; samples inside it are pruned, while visible regions remain supervised.Source: §3.3 · pp. 3–4, Figure 3 · p. 4
3.4. Joint Optimization
- The scene and object branches are jointly optimized by adding the scene color-reconstruction loss to the object-supervision loss.The scene loss minimizes squared error between predicted and true pixel colors.Source: §3.4 · p. 4
3.5. Editable Scene Rendering
- Editable rendering removes original objects from the scene background, transforms object-specific fields to user-defined locations, then composites them by ray distance.The background stage prunes sampling in target regions; the object stage renders the selected objects at their manipulated locations.Source: §3.5 · p. 4
4. Experiments
Experiments on ToyDesk and ScanNet evaluate scene rendering, object rendering and editing, and framework ablations.
4.1. Dataset
- ToyDesk provides two desk layouts with posed images and 2D instance labels for toys; ScanNet supplies RGB-D scans and instance annotations, with selected views split 80% for training and 20% for testing.ToyDesk meshes and camera poses are recovered using SfM, multi-view stereo, and mesh generation; ScanNet views are selected by angle and distance to a central object.Source: §4.1 · p. 5
4.2. Data Preparation and Experiment Details
- For fair ScanNet comparisons, depth loss is added for NeRF, NSVF, and our method, and shared point clouds initialize voxel representations; ToyDesk excludes NPCR and uses SfM point clouds.Our method does not require sensor depth for training; depth supervision is optional, with additional no-depth results deferred to the supplementary material.Source: §4.2 · p. 5
4.3. Comparison of Scene Rendering
- Our scene branch is comparable to or better than state-of-the-art methods on PSNR, SSIM, and LPIPS, preserving fine details and textures even where voxel support is missing.On ToyDesk, NeRF-based approaches struggle with far backgrounds; on ScanNet, NPCR and Sparse Voxel show color errors, and NPCR omits chair parts.Source: Table 1 · p. 5, §4.3 · p. 5, Figure 4 · p. 6
4.4. Comparison of Individual Object Rendering
- The object branch renders clean individual objects without background, unlike strict mesh masks that produce mottling, relaxed masks that include background, or bounding-box ray clipping.The strict and relaxed mesh-mask radii are 0.05 m and 0.1 m, respectively.Source: §4.4 · pp. 5–6
4.5. Comparison of Scene Editing
- ToyDesk edits rotate, move, and duplicate toys with sharp object boundaries and seamless spatial integration; unlike NPCR, our method keeps background textures nearly unchanged and avoids severe editing artifacts.NPCR can move background patterns and produce false occlusions, chair holes, or cloudy textures; Sparse Voxel has issues similar to bounding-box approaches.Source: §4.5 · pp. 6–7, Figure 7 · p. 7
4.6. Ablation Study
- Scene guidance and the 3D guard mask significantly improve object rendering under frequent occlusion, while object voxel features further boost quality; ϵ = 0.05 performs better, and nearby choices are similar.The ablations use object-area metrics on ten test views each from ScanNet scenes 0033 and 0038; the authors find the guard-mask choice is not sensitive to ϵ.Source: §4.6 · p. 7, Table 2 · p. 7, Table 3 · p. 7
4.7. Visualization of Rendered Segmentation
- Despite jittery input masks, multi-view supervision yields smooth, accurate rendered segmentations while preserving object details such as a chair handle.The authors attribute resistance to single-view mask noise to multi-view supervision and geometry cues from the learned 3D structure.Source: §4.7 · p. 8
5. Conclusion and Future Works
The framework supports high-fidelity novel-view synthesis and editable rendering of real-world scenes, including manipulation of multiple objects. Future work targets unseen textures, camera-related artifacts, and more realistic lighting.
Conclusion
- The framework enables high-fidelity novel-view synthesis and editable rendering of real-world scenes, including moving, rotating, or duplicating multiple objects.It is trained using posed images and rough 2D instance masks.Source: §5 · p. 8
Future Works
- Because limited observations leave under-object textures unseen, the method relies on network spatial smoothness; scene completion could improve these regions.Source: §5 · p. 8
- Optimizing camera poses and ray directions could mitigate pose noise and rolling shutter, producing a clearer background in real-world scenes.Source: §5 · p. 8
- Integrating a scene lighting model is a promising direction for achieving more realistic scene editing.Source: §5 · p. 8