Source-linked AI summary
Editable Free-viewpoint Video Using a Layered Neural Representation
Jiakai Zhang, Xinhang Liu, Xinyi Ye, Fuqiang Zhao, Yanshun Zhang, Minye Wu, Yingliang Zhang, Lan Xu, Jingyi Yu
TL;DR
Editable free-viewpoint video for large-scale dynamic scenes remains difficult because recent dynamic neural methods focus on reconstruction rather than visual manipulation. The paper introduces layered ST-NeRF representations for per-entity spatial and temporal editing from sparse cameras, and experiments demonstrate high-quality, photo-realistic editable results with wide-range free viewing. Its scope is constrained by scene parsing, human segmentation, occlusion, and bounding-box tracking limitations.
Problem
Existing dynamic free-viewpoint methods lack editing functions for changing scene perception, while prior retiming work does not explore rich 3D spatial editing.
Method
The method represents each dynamic entity and the environment as a continuous space-time ST-NeRF layer, with disentangled location, deformation, and appearance for per-entity editing.
Results
Experiments demonstrate high-quality, photo-realistic editable free-viewpoint videos with spatial and temporal effects while supporting wide-range free viewing.
Takeaways & Limitations
The layered representation enables explicit spatial and temporal manipulation of dynamic entities while maintaining high realism and free viewing across a wide range.
Takeaways & Limitations
The approach is limited by similar-appearance entities, human-segmentation dependence, severe occlusions, failed bounding-box tracking, and incomplete information outside tracked boxes.
Abstract
from arXiv · showhide
Generating free-viewpoint videos is critical for immersive VR/AR experience but recent neural advances still lack the editing ability to manipulate the visual perception for large dynamic scenes. To fill this gap, in this paper we propose the first approach for editable photo-realistic free-viewpoint video generation for large-scale dynamic scenes using only sparse 16 cameras. The core of our approach is a new layered neural representation, where each dynamic entity including the environment itself is formulated into a space-time coherent neural layered radiance representation called ST-NeRF. Such layered representation supports fully perception and realistic manipulation of the dynamic scene whilst still supporting a free viewing experience in a wide range. In our ST-NeRF, the dynamic entity/layer is represented as continuous functions, which achieves the disentanglement of location, deformation as well as the appearance of the dynamic entity in a continuous and self-supervised manner. We propose a scene parsing 4D label map tracking to disentangle the spatial information explicitly, and a continuous deform module to disentangle the temporal motion implicitly. An object-aware volume rendering scheme is further introduced for the re-assembling of all the neural layers. We adopt a novel layered loss and motion-aware ray sampling strategy to enable efficient training for a large dynamic scene with multiple performers, Our framework further enables a variety of editing functions, i.e., manipulating the scale and location, duplicating or retiming individual neural layers to create numerous visual effects while preserving high realism. Extensive experiments demonstrate the effectiveness of our approach to achieve high-quality, photo-realistic, and editable free-viewpoint video generation for dynamic scenes.
1 INTRODUCTION
Editable free-viewpoint video for large-scale dynamic scenes remains unresolved: existing dynamic methods focus on reconstruction, while prior editing work is limited in temporal effects. This paper introduces a layered neural representation enabling spatial and temporal editing from 16 cameras across up to 180 degrees.
- Research gap: Existing dynamic free-viewpoint methods remain limited by capture volume or fragile human motions and focus on reconstruction without visual editing functions.Prior editing work explored neural retiming but did not address rich 3D spatial editing for free-viewpoint video.
- Proposed capability: The paper presents the first editable photo-realistic free-viewpoint video approach for large-scale dynamic scenes using 16 cameras covering up to 180 degrees.The approach supports spatial and temporal editing while retaining wide-range free viewing.
- Layered representation: Each dynamic entity and the environment are modeled as separate space-time coherent neural layers to support per-entity manipulation.The representation disentangles location, deformation, and appearance across dynamic entities.
- Disentanglement: Layer-wise 4D label-map tracking explicitly disentangles spatial information, while a continuous deform module implicitly disentangles temporal motion.These components are part of the proposed layered neural representation for large-scale dynamic scenes.
- Training: A layered loss and motion-aware ray-sampling strategy enable efficient training with multiple performers in large dynamic scenes.The training design targets the computational demands of the layered representation.
2 RELATED WORK
Prior work spans image-based rendering, neural scene representations, dynamic reconstruction, and video editing, but these approaches retain important modeling or editing limitations. The paper combines a layered neural representation with explicit spatial and temporal manipulation for photo-realistic free-viewpoint video.
- Related work: Traditional image-based rendering interpolates textures from input images, while neural rendering learns mappings or constructs scene representations for novel-view synthesis.Neural scene representations include point clouds, voxels, and implicit representations.
- Dynamic scene reconstruction: Earlier dynamic-scene methods use assumptions such as piece-wise planar rigid motion or locally rigid deformation, limiting complicated scene geometry manipulation.These assumptions constrain their ability to support further visual-effect rendering.
- Proposed representation: The proposed pipeline models each dynamic entity and the environment as individual neural layers, using ST-NeRF to represent each layer continuously across space and time.The representation is based on scene-parsing results and supports per-entity editing.
- Neural editing: Disentangled layer inputs allow affine transformation, duplication, and performer-motion retiming, while depth-aware rendering reassembles layers for free-viewpoint results.The method targets photo-realistic spatial and temporal visual effects.
- Dynamic scene reconstruction: Dynamic reconstruction methods address illumination changes and moving objects using cues including silhouettes, stereo, segmentation, and photometric information.These approaches target reconstruction from camera arrays or single-camera input.
- Video editing: The paper focuses on human-related video elements, extending video-editing work toward free-viewpoint manipulation rather than only 2D interactions or temporal effects.Earlier video-editing methods included annotations, navigation, and direct manipulation of tracked 2D objects.
3 OVERVIEW
The approach parses dynamic scenes into tracked entity layers, models each layer with a continuous space-time ST-NeRF, and reassembles them through object-aware rendering. This layered representation supports editable, photo-realistic free-viewpoint video generation while addressing occlusion and multi-entity motion.
- Scene Parsing: The system begins with scene parsing that produces coarse 4D label maps, tracked 3D bounding boxes, and layer-wise labels for dynamic entities.Multi-view stereo supplies coarse geometry, while multi-view tracking and label refinement address occlusions and identity association.
- Layered Neural Representation: Each dynamic entity and the environment are represented as separate continuous space-time neural radiance fields called ST-NeRF.Tracked bounding boxes provide spatial anchors, while a continuous deformation module encodes motion across views and timestamps.
- Neural Editing: Disentangled layer location, deformation, and appearance enable per-entity spatial and temporal edits while preserving photo-realistic free-viewpoint rendering.Supported operations include affine transformation, duplication, retiming, transparency adjustment, insertion, and removal.
- Layered Spatio-Temporal Neural Renderer: Fusing geometry and appearance across views and timestamps in canonical space allows ST-NeRF to address missing information caused by occlusion or foreshortening.The representation is described as self-supervised and supports complete, photo-realistic novel-view synthesis despite visibility challenges.
- Layered Spatio-Temporal Neural Renderer: The renderer samples ray segments within entity bounding boxes, queries the corresponding ST-NeRFs, and reassembles densities and colors with object-aware volume rendering.A second sampling stage uses density-derived probabilities, and overlapping object regions jointly supervise the corresponding networks.
5 NEURAL SCENE EDITING
The layered representation enables controllable space-time editing of individual dynamic entities while preserving depth-aware, photo-realistic free-viewpoint rendering. Users can manipulate entities spatially and temporally, including transforming, duplicating, inserting, removing, retiming, and adjusting transparency.
- Neural Scene Editing: The layer-wise design disentangles entity location, timing, deformation, and appearance for controllable space-time editing during inference.Users directly manipulate each entity’s ST-NeRF input position and timing.
- Neural Scene Editing: The framework supports high-realism neural editing across a wide viewing range, including scale and location manipulation, duplication, and retiming.These operations are demonstrated for editable free-viewpoint video generation.
- Spatial Affine Transformation: Affine transformations of tracked 3D bounding boxes rearrange neural canonical spaces to support photo-realistic spatial edits.The bounding box acts as a spatial anchor for fusing motion and appearance across views and timestamps.
- Temporal Retiming Transformation: Temporal retiming transforms entity timestamps and recomposes the scene with object-aware volume rendering to alter performers’ motion timelines.The transformation is applied during scene composition and when querying entity densities and colors.
- Object Insertion and Removal: Object insertion and removal modify target performers’ 3D bounding boxes during scene composition while volume rendering preserves high realism.Insertion adds new boxes to scene coordinates, whereas removal deletes existing ones.
- Transparency Adjustment: Transparency editing scales an entity’s density by a scalar without extra training, enabling translucent visual effects.The new density is defined as σ′ = s·σ.
6 RESULTS
The evaluation uses eight large-scale indoor scenes captured with 16 synchronized cameras and examines editing effects, comparisons, ablations, and limitations. Results show photo-realistic free-viewpoint editing, stronger rendering and decomposition than baselines, and sensitivity to view count and scene-parsing assumptions.
- Implementation Details: The dataset contains eight indoor dynamic scenes with two or three performers, captured by 16 synchronized cameras covering up to 180 degrees.The cameras record 1920 × 1080 RGB streams at 25 frames per second.
- Comparisons: Our approach achieves the best performance in all four rendering-accuracy metrics against NeRF, NeRF-T, Neural Volumes, AGI, and HVR.The quantitative evaluation uses PSNR, SSIM, MAE, and LPIPS on captured reference views.
- Novel Space-time Editing Results: The framework supports depth-aware, photo-realistic editing of performer position, rotation, duplication, scale, retiming, and time freezing.Examples include moving a violinist, duplicating dancers, synchronizing asynchronous actions, and freezing a superhero mid-jump.
- Comparisons: Qualitatively, the method produces vivid, sharp, temporally consistent renderings without flickering and supports editing effects unavailable in previous baselines.Compared with Layered Neural Rendering, it yields depth-aware and physically correct rendering for two overlapping performers.
- Ablation Study: Removing the deformation module, time input to the radiance network, or layer-wise loss causes dynamic-handling failures, blur, or incorrect decomposition.The complete model achieves photo-realistic results with better decomposition, and also outperforms its variations across all reported metrics.
- Ablation Study: Held-out-view performance decreases as the training view count is reduced, while fewer views can slightly increase training-view PSNR through overfitting.The 16-view setting provides better held-out-view metrics and novel-view rendering than the four-view setting.
- Limitations and Discussions: The method may fail for similarly appearing entities, severely occluded entities, or light-changing regions outside tracked bounding boxes, and currently focuses on dynamic humans using 16 cameras.Unmodeled regions can produce 3D ghosting artifacts during editing, although manually correcting bounding boxes can fix that case.
7 CONCLUSION
The paper presents a layered neural pipeline for high-quality, editable free-viewpoint videos of large dynamic scenes from relatively sparse 16-camera capture. It disentangles dynamic entities for realistic spatial and temporal editing while preserving wide-range free viewing, and reports favorable comparisons to state of the art.
- The approach generates high-quality editable free-viewpoint videos of large-scale dynamic scenes from relatively sparse 16 RGB cameras.
- A qualitative view ablation shows better held-out-view rendering when more views are used.
- Its layered neural representation models spatially and temporally consistent correlations for individual dynamic entities and supports their editing.
- The deform module encodes temporal motion, while object-aware volume rendering reassembles the neural layers.
- The framework enables explicit spatial and temporal manipulations while maintaining high realism and supporting wide-range free viewing.