PaperlayerView exact revision on arXiv

Source-linked AI summary

Compressible-composable NeRF via Rank-residual Decomposition

Jiaxiang Tang, Xiaokang Chen, Jingbo Wang, Gang Zeng

arXiv:2205.14870v2cs.CV

TL;DR

NeRF’s implicit representation is difficult to manipulate, and prior composition methods can be constrained by shared renderers or large storage. This paper uses tensor rank decomposition with rank-residual learning to adjust model size and compose transformed models. It reports rendering quality comparable to state-of-the-art methods while adding compressibility and composability.

  • Problem

    NeRF’s implicit representation makes scene manipulation difficult, while prior approaches can require shared renderers or substantial storage.

    Source: §1 · p. 1

  • Method

    The method uses tensor rank decomposition and rank-residual learning so rank truncation adjusts model size and rank concatenation composes transformed models.

    Source: §1 · p. 2

  • Results

    The method achieves rendering quality comparable to state-of-the-art methods while adding compressibility and composability.

    Source: §1 · p. 2, §4.4 · p. 9

  • Takeaways & Limitations

    Models can be dynamically adjusted for different levels of detail and transformed and composed without extra optimization or training constraints.

    Source: §1 · pp. 1–2

    The method does not account for lighting conditions after composition and currently supports only bounded objects.

    Source: §4.4 · p. 9

Abstract

from arXiv · show

Neural Radiance Field (NeRF) has emerged as a compelling method to represent 3D objects and scenes for photo-realistic rendering. However, its implicit representation causes difficulty in manipulating the models like the explicit mesh representation. Several recent advances in NeRF manipulation are usually restricted by a shared renderer network, or suffer from large model size. To circumvent the hurdle, in this paper, we present an explicit neural field representation that enables efficient and convenient manipulation of models. To achieve this goal, we learn a hybrid tensor rank decomposition of the scene without neural networks. Motivated by the low-rank approximation property of the SVD algorithm, we propose a rank-residual learning strategy to encourage the preservation of primary information in lower ranks. The model size can then be dynamically adjusted by rank truncation to control the levels of detail, achieving near-optimal compression without extra optimization. Furthermore, different models can be arbitrarily transformed and composed into one scene by concatenating along the rank dimension. The growth of storage cost can also be mitigated by compressing the unimportant objects in the composed scene. We demonstrate that our method is able to achieve comparable rendering quality to state-of-the-art methods, while enabling extra capability of compression and composition. Code will be made available at https://github.com/ashawkey/CCNeRF.

1 Introduction

NeRF’s implicit representation is difficult to manipulate, while prior compositional approaches can require shared renderers and explicit representations can require substantial storage. This work uses tensor rank decomposition to support adjustable detail and model composition without extra optimization.

1 Introduction

  • NeRF manipulation remains challenging, while shared-renderer approaches constrain new objects and explicit no-network representations can require large storage.Shared renderers require new objects to use a fixed renderer; large storage hinders composing scenes with many objects.Source: §1 · p. 1
  • Rank-residual learning encourages lower ranks to preserve important scene information, enabling low-rank truncations to adjust detail and model size without retraining.The representation uses vector- or matrix-based rank components and weights for each feature channel.Source: §1 · p. 2
  • Different models can be transformed and composed by concatenating rank components, with object transformations controlling position and orientation.The operations require no extra optimization or shared renderer.Source: §1 · p. 2, Figure 1 · p. 2
  • Compressing less important objects can mitigate storage growth when composing complex scenes.The paper connects compression and composition through their shared use of rank.Source: §1 · p. 2
  • The method combines comparable rendering quality to state-of-the-art methods with additional compressibility and composability.Its model size can be adjusted after training, and different models can be transformed and composed without training constraints.Source: §1 · p. 2

2 Related Work

NeRF representations range from neural networks to explicit and hybrid structures, with different trade-offs in speed and storage. This work builds on tensor decomposition and prior compositional representations while focusing on compression and cross-scene composition.

2.1 Scene Representation with NeRF

  • Neural-network NeRFs use MLPs, hybrid representations store explicit features to reduce MLP computation, and no-network methods model scenes without neural networks.The passage identifies sparse voxels, octrees, hashmaps, and tensor factorization as examples of hybrid structures.Source: §2.1 · pp. 2–3

2.2 Tensor Decomposition and Low-rank Approximation

  • Tensor decomposition generalizes matrix SVD; the paper’s approach focuses on preserving low-rank approximation for compressing learned neural radiance fields.TensoRF also uses CP and Vertex-Matrix decompositions, but its MLP design disrupts the tensor decomposition property relevant to compression and composition.Source: §2.2 · p. 3

2.3 Manipulation and Composition of NeRF

  • Prior compositional NeRFs can require shared MLPs or retraining, while Plenoxels supports direct composition but has a large dense index matrix.The model structure uses rank weights and vector- or matrix-based components; their ratio trades model size against performance.Source: §2.3 · p. 3, Figure 2 · p. 4

3 Methodology

The method represents NeRF scenes with tensor decompositions learned from 2D image supervision, then uses rank-residual learning to support rank truncation and composition of transformed models. Its design combines vector- and plane-based factors, enabling model-size and detail adjustment without retraining.

3.1 Preliminaries on Neural Radiance Fields; 3.2 Preliminaries on Tensor Decomposition

  • NeRF estimates pixel color by volume rendering samples along a ray, and differentiable rendering permits optimization from 2D images by minimizing pixelwise L2 error.Tensor decompositions represent the scene as a high-order tensor with a core and factor matrices; CP is a special case with a superdiagonal core.Source: §3.1 · p. 4, §3.2 · p. 4

3.3 Decompose NeRF without MLP

  • The hybrid feature-volume representation shares factorized matrices across correlated feature channels while using channel-specific rank weights, combining compact vector factors with more computation-friendly plane factors.The hybrid variant independently mixes vector- and matrix-based ranks, trading model size against performance.Source: §3.3 · p. 5

3.4 Rank-residual Learning for Compressibility

  • Rank-residual learning simultaneously supervises outputs from accumulated rank truncations so lower ranks learn more of the scene approximation and later groups learn residual error.With M much smaller than the total rank count, supervision is applied at group boundaries; empirical sorting and truncation selects remaining ranks between those boundaries.Source: §3.4 · pp. 5–6
  • After training, rank truncation adjusts model size and rendering quality without extra optimization, using a simple slice to meet a storage bound or performance target.The method aims for near-optimal compression at targeted ranks; Figure 3 illustrates the HY-S model on LEGO, with dashed lines marking rank-residual supervision.Source: §3.4 · p. 6, Figure 3 · p. 6

3.5 Composability without Constraints

  • Models with different resolutions and decomposition forms can be composed by concatenating rank components, while per-object transformation matrices support affine transformations before querying each object.The rendering combines object densities for opacity and density-weighted colors; model size and rendering time grow linearly with total rank.Source: §3.5 · pp. 6–7
  • In a composed scene, compressing less important objects trades rendering quality for lower storage and improved efficiency.The method applies this compression capability to scenes containing many objects.Source: Figure 5 · p. 7, §3.5 · p. 7

4 Experiments

The experiments show that rank truncation preserves rendering quality near rank-specific retrained results, while the representation also supports transformed scene composition. Comparisons report quality comparable to recent methods, with compression and composition as additional capabilities.

4.1 Implementation Details

  • Experiments use NeRF-synthetic and Tanks and Temples, with implementation on one NVIDIA V100 GPU.The setup uses degree-3 spherical harmonics and occupancy-mask pruning.Source: §4.1 · p. 7

4.2 Compression Results

  • Proposed compression remains comparable to rank-specific optimal results at all targeted ranks, while the baseline degrades significantly.Table 1 reports PSNR for compression strategies; Figure 6 visualizes the quality difference.Source: §4.2 · p. 8, Table 1 · p. 8, Figure 6 · p. 8
  • Rank-residual learning concentrates rank importance in lower ranks, supporting truncation; comparisons use retrained optimal, original-loss baseline, and proposed models.The optimal setting retrains at the target rank, while the baseline and proposed settings truncate trained models.Source: §4.2 · pp. 8–9

4.3 Composition Results

  • Different models can be affinely transformed and composed without extra optimization, with occlusion and collision handled in the resulting scene.Less important objects in complex compositions can be compressed to trade model size against rendering quality.Source: §4.3 · p. 9

4.4 Comparisons and Discussion

  • Rendering quality is comparable to recent methods rather than the best reported, while the design adds compression and composition capabilities.The method omits an MLP renderer, which can worsen specular detail but otherwise complicates sharing across scenes or composition.Source: §4.4 · p. 9, Table 2 · p. 9
  • The method does not model lighting for consistent relighting after composition and currently supports only bounded objects.The authors suggest reflectance models and a background model as possible directions for these boundaries.Source: §4.4 · p. 9

5 Conclusion

The conclusion presents a compressible and composable neural radiance field for manipulating multi-object scenes. Rank-residual learning supports model-size adjustment across levels of detail, while models can be transformed and composed together.

5 Conclusion

  • The representation combines compressibility and composability to support manipulation of complex scenes with multiple objects.The authors state that this could further facilitate NeRF-based scene representation in real-world applications.Source: §5 · pp. 9–10
  • Rank-residual learning enables near-optimal low-rank approximation and dynamic model-size adjustment for different levels of detail.The model supports different levels of detail without requiring extra optimization, as described in the conclusion.Source: §5 · p. 9
  • Models can be transformed and composed together as in a 3D editor.The conclusion connects these capabilities to efficient and convenient manipulation of multi-object scenes.Source: §5 · p. 10

A Additional Implementation Details

The appendix describes model settings for density and color components, with a focus on compressing color. It compares CP, HY, and HY-S configurations and illustrates rank-truncation compression.

A.1 Model Settings

  • The settings are denoted by rank-component counts for density and color, with compression focused on color because density takes less storage and underlies correct color.The notation begins as Rdensity and continues with the color component count in the supplied passage.Source: §A.1 · p. 13
  • Figure 9 compares compression strategies and shows that rank-residual compression reaches near-optimal quality without retraining, unlike sort-and-truncate.The example uses the HY-S model on LEGO and retrains an optimal model for comparison.Source: Figure 9 · p. 14
  • CP uses vector components only, while HY combines vector and matrix components to reduce the ranks needed for good rendering.CP uses less storage but requires more ranks, increasing GPU memory use and slowing querying; matrix components in HY take more storage but reduce the needed ranks.Source: §A.1 · p. 14
  • HY-S combines vector and matrix components and adds color matrix-component groups to allow model size adjustment over a larger range.It uses five rank-residual groups, beginning with a group containing 96 vector components.Source: §A.1 · p. 14

A.2 Training Details

The appendix reports training settings and contrasts the proposed decomposition with TensoRF-SH. It also provides qualitative comparisons and identifies a training-time and PSNR comparison table.

A.2 Training Details

  • Training runs for 30,000 iterations with Adam, occupancy-grid pruning, bounding-box shrinking, and density-factor L1 regularization.The occupancy grid is calculated at steps 2,000 and 4,000; density regularization encourages parameter sparsity.Source: §A.2 · p. 14

A.3 Difference from TensoRF-SH

  • HY permits arbitrary ratios of vector- and matrix-based rank components, whereas TensoRF’s VM decomposition fixes the ratio at 1:1.The passage states that this flexibility lets HY control model size more flexibly.Source: §A.3 · p. 14
  • The authors report rendering quality comparable to state-of-the-art methods, with differences hard to discern in most cases.Figure 10 presents qualitative comparisons with recent works.Source: Figure 10 · p. 15
  • Table 3 compares training time and PSNR between methods.The supplied table caption gives the comparison scope but no numerical results.Source: Table 3 · p. 15

B Additional Experimental Results

The additional experiments illustrate different levels of detail, rank-based compression, and qualitative rendering comparisons. The examples show how detail allocation and truncation are used, while the supplied table captions identify further comparisons.

B.1 Visualization of Different Levels of Detail

  • A primary bulldozer uses the full model while a smaller secondary bulldozer uses a truncated model, saving storage without harming overall rendering quality.The objects are transformed and composed into a barn scene.Source: §B.1 · p. 15

B.2 Visualization of the Near-optimal Compression

  • The HY-S LEGO example compares rank-residual compression across five groups with truncation from a full-rank model and a retrained optimal model.The authors say this demonstrates the near-optimal low-rank approximation property of rank-residual learning.Source: §B.2 · p. 15

B.3 Qualitative Comparisons with SOTA

  • Rendering quality is reported as comparable to state-of-the-art methods, with differences hard to discern in most cases.Figure 10 provides the qualitative comparisons; the authors consider the quality sufficient for compression and composition.Source: §B.3 · p. 15
  • Tables 4 and 5 compare training time and PSNR across SH degrees and parallel versus sequential rank-residual training strategies.The supplied captions identify the comparison topics but do not provide numerical results.Source: Table 4 · p. 16, Table 5 · p. 16

B.4 Time Analysis

The method is more efficient to train than vanilla NeRF, but slower than some recent methods and TensoRF. Rank-residual learning increases training time relative to TensoRF.

B.4 Time Analysis

  • Training is more efficient than vanilla NeRF, but does not match the speed of recent methods focused on training speed.The comparison is reported for average training time over nerf-synthetic.Source: §B.4 · p. 16
  • Rank-residual learning increases training time compared with TensoRF.Source: §B.4 · p. 16
  • The reported timing measure is average training time over nerf-synthetic.Source: §B.4 · p. 16

C Additional Ablation Study

On the materials dataset, which contains rich view-dependent effects, SH degree 3 achieves the best PSNR. Degrees 1 and 4 make convergence harder and increase training time.

C.1 Influence of the SH degrees

  • SH degree 3 achieves the best PSNR on the materials dataset with rich view-dependent effects.Source: §C.1 · p. 16
  • Using too few or too many SH degrees, such as 1 or 4, makes the model hard to converge.Source: §C.1 · p. 16
  • SH degrees 1 or 4 also take more time to train.Source: §C.1 · p. 16

C.2 Parallel rank-residual training

The study compares sequential staged training with two parallel rank-residual strategies. Parallel training without detaching is reported to ease early-stage training by letting later stages model complex details.

C.2 Parallel rank-residual training

  • Parallel training without detaching lets later stages model complex details, which the authors think eases training of earlier stages so they can focus on fundamentals.Source: §C.2 · p. 16
  • Sequentially training each stage to convergence is significantly slower.Source: §C.2 · p. 16
  • Parallel training with detached outputs performs worse because later stages are optimized before earlier stages have fully converged.Source: §C.2 · p. 16

Related papers

Loading 2205.14870v2…