Source-linked AI summary
TensoRF: Tensorial Radiance Fields
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, Hao Su
TL;DR
TensoRF addresses the long reconstruction time and inefficiency of radiance-field representations by modeling scenes as factorized low-rank tensors rather than purely MLP-based fields. Its CP and VM decompositions enable compact, fast reconstruction with high rendering quality, while bounded-scene support remains a limitation.
Problem
NeRF achieves photorealistic rendering but purely MLP-based models can require hours or days to train, while voxel-grid methods have cubic memory growth and may require lengthy preprocessing.
Method
TensoRF represents radiance fields as 4D tensors corresponding to voxel-grid spatial axes and feature channels, then factorizes them into compact low-rank components using CP or vector-matrix decomposition.
Results
Less than 30 min per scene and less than 75MB memory are achieved while providing better rendering quality than NeRF trained for 20+ hours; VM decomposition further improves reconstruction quality and optimization speed.
Takeaways & Limitations
Tensorized low-rank components provide a compact radiance-field representation that supports efficient reconstruction and high-quality novel-view rendering.
Takeaways & Limitations
The method currently supports only bounded scenes with a single bounding box and cannot handle unbounded scenes containing both foreground and background content.
Abstract
from arXiv · showhide
We present TensoRF, a novel approach to model and reconstruct radiance fields. Unlike NeRF that purely uses MLPs, we model the radiance field of a scene as a 4D tensor, which represents a 3D voxel grid with per-voxel multi-channel features. Our central idea is to factorize the 4D scene tensor into multiple compact low-rank tensor components. We demonstrate that applying traditional CP decomposition -- that factorizes tensors into rank-one components with compact vectors -- in our framework leads to improvements over vanilla NeRF. To further boost performance, we introduce a novel vector-matrix (VM) decomposition that relaxes the low-rank constraints for two modes of a tensor and factorizes tensors into compact vector and matrix factors. Beyond superior rendering quality, our models with CP and VM decompositions lead to a significantly lower memory footprint in comparison to previous and concurrent works that directly optimize per-voxel features. Experimentally, we demonstrate that TensoRF with CP decomposition achieves fast reconstruction (<30 min) with better rendering quality and even a smaller model size (<4 MB) compared to NeRF. Moreover, TensoRF with VM decomposition further boosts rendering quality and outperforms previous state-of-the-art methods, while reducing the reconstruction time (<10 min) and retaining a compact model size (<75 MB).
1 Introduction
TensoRF replaces slow-to-train NeRF MLPs with a compact factorized tensorial radiance-field representation. CP and VM decompositions target efficient reconstruction while maintaining or improving rendering quality.
- Hours or days of NeRF training motivate a radiance-field representation that is faster to reconstruct while retaining high-quality rendering.
- TensoRF represents radiance fields as an explicit voxel grid of features and factorizes its 4D tensor into low-rank components.The grid’s three spatial modes correspond to XYZ, while the fourth mode represents feature channels.
- CP decomposition achieves photorealistic rendering and a more compact model than purely MLP-based NeRF, but complex scenes may require more components and longer training.
- VM decomposition uses vector-matrix outer products to express more complex tensor data with fewer components, leading to faster reconstruction and better rendering.It relaxes the ranks of two modes by jointly modeling them in a matrix factor.
- TensoRF supports volumetric rendering by decoding density and view-dependent color from compact features, with neural or spherical-harmonics decoding options.Tensor features also support efficient trilinear interpolation for continuous-field modeling.
- TensoRF models achieve realistic novel-view synthesis with lower memory complexity and high computation efficiency, including reconstruction within 30 min.CP reduces space complexity to O(n), while VM reduces it to O(n^2).
2 Related Work
Related work spans tensor decompositions, radiance-field representations, voxel-grid methods, and generalizable scene models. TensoRF applies CP and introduces VM factorization to compactly model radiance fields through per-scene optimization.
- Tensor decomposition: CP and Tucker are widely used tensor decompositions, while block term decomposition combines their principles in variants applied to vision and learning.
- Tensor decomposition: TensoRF directly applies CP decomposition and introduces VM decomposition, which can be viewed as a special case of block term decomposition.
- Scene representations and radiance fields: NeRF introduced radiance fields for photorealistic novel-view synthesis, alongside broader scene representations including meshes, point clouds, volumes, and implicit functions.
- Scene representations and radiance fields: Unlike methods trained across scenes, TensoRF focuses on per-scene optimization and leaves generalizable extensions for future work.
- Concurrent work: DVGO and Plenoxels optimize voxel features directly, whereas TensoRF factorizes feature grids into compact components to reduce memory requirements.
- Tensor decomposition: Figure 2 contrasts CP’s sum of vector outer products with VM’s sum of vector-matrix outer products.
3 CP and VM Decomposition
CP represents tensors as sums of compact rank-one vector components, while VM replaces paired vector modes with matrices. This trade-off increases per-component capacity and reduces the number of components needed for complex data.
- CP decomposition: CP decomposition factorizes a 3D tensor into a sum of outer products of vectors, producing compact rank-one components.Each tensor element is computed as a sum of products of corresponding vector entries.
- CP decomposition: CP can produce high-quality radiance-field results, but its strong compactness may require many components and increase reconstruction costs for complex scenes.
- Vector-Matrix decomposition: VM decomposition factorizes a tensor into multiple vector-matrix outer products rather than CP’s outer products of pure vectors.
- Vector-Matrix decomposition: VM relaxes two mode ranks per component while restricting the third mode to rank one, allowing higher ranks through matrix factors.
- Vector-Matrix decomposition: By combining two modes into matrices, VM components express more complex high-dimensional data and require fewer components than separate vector factors.
- Vector-Matrix decomposition: VM reduces memory complexity from O(N 3) for dense grids to O(N 2), despite each component containing more parameters than a CP component.
- Tensor for scene modeling: For scene modeling, tensor modes correspond to the XYZ axes, and the three VM component counts are usually set equally for scenes with similarly complex axes.
4 Tensorial Radiance Field Representation
TensoRF represents radiance fields with factorized tensor grids that separately model density and appearance, enabling compact storage and efficient continuous evaluation for rendering.
- Radiance field representation: TensoRF models a radiance field with separate geometry and appearance grids, where density is direct and appearance is decoded from features and viewing direction.The geometry grid is single-channel, while the appearance grid contains P feature channels.
- Vector-matrix decomposition: VM decomposition factorizes geometry into vector-matrix components and appearance into spatial vector-matrix factors plus feature-mode vectors.The feature-mode vectors are kept separate because that mode is often lower-dimensional and lower-rank.
- Rendering: Given a 3D location and viewing direction, the factorized field produces continuous volume density and view-dependent color for radiance-field reconstruction and rendering.The formulation applies to TensoRF with both CP and VM decomposition.
- Factor interpretation: The factorized components encode spatial distributions along XYZ axes, while stacked appearance vectors form a global dictionary of scene-wide appearance commonalities.Using fewer components than the grid resolutions yields a compact representation of high-resolution grids.
- Efficient feature evaluation: A voxel density is evaluated by indexing and multiplying factor values, while appearance evaluation forms a full feature vector before applying the shared appearance matrix.The appearance computation uses concatenated component values and computes all P channels required by the shading function.
- Continuous field evaluation: Trilinear interpolation is performed by linear or bilinear interpolation of vector and matrix factors, avoiding recovery of eight tensor values.This directly recovers interpolated density and appearance values with lower runtime computation and memory costs.
5 Implementation details
TensoRF is implemented in PyTorch with either an MLP or spherical-harmonics decoder and optimized through staged, single-GPU tensor-factor training.
- Software and decoder: TensoRF is implemented in PyTorch without customized CUDA kernels, using either an MLP or spherical-harmonics function for feature decoding.Both decoders use P = 27 appearance features; the MLP has two fully connected layers with 128-channel hidden layers.
- Optimization: Training uses Adam with separate initial learning rates for tensor factors and the neural decoder, processing batches of 4096 pixel rays on one Tesla V100 GPU.The tensor-factor learning rate is 0.02, while the MLP decoder uses 0.001 when neural features are used.
- Coarse-to-fine training: The feature grid is initialized at N0 = 128 voxels per dimension and progressively upsampled through scheduled linear and bilinear interpolation steps.Upsampling occurs at steps 2000, 3000, 4000, 5500, and 7000.
- Configuration analysis: The implementation analyzes the effects of total training steps, grid resolution, and the total number of decomposition components.These parameters are varied as part of the implementation analysis.
6 Experiments
Experiments show that TensoRF achieves strong rendering quality while reducing reconstruction time and model size across synthetic, 360° object, and forward-facing scenes.
- Analysis of different TensoRF models: 31.56 PSNR is achieved by the best TensoRF-CP model with 384 components and 500^3 voxels, outperforming vanilla NeRF while requiring less than 4MB.The CP model is the best among the evaluated CP variants.
- Analysis of different TensoRF models: 31.81 PSNR is achieved by the smallest TensoRF-VM model with 48 components and 200^3 voxels, using only 8.6MB and outperforming the best CP model.VM models generally need fewer components, while 192 components are usually adequate.
- Optimization efficiency: Less than 30 min is required for reconstruction by nearly all TensoRF models, substantially faster than NeRF and many previous methods.The largest VM model is the stated exception.
- Comparisons on 360° scenes: 15k steps are sufficient for many VM models to achieve PSNRs that are already state-of-the-art.The comparison covers 360° scenes and multiple datasets.
- Comparisons on 360° scenes: All TensoRF CP and VM models outperform NeRF on three datasets while using substantially less optimization time and fewer steps; VM-192 achieves the best PSNRs and SSIMs on all datasets.The comparison reports PSNR, SSIM, optimization steps, time, and model size.
- Comparisons on 360° scenes: More appearance and geometry details and fewer outliers are visible in TensoRF renderings than in the comparison methods.This qualitative comparison uses the VM-192-30k model on two Synthetic NeRF scenes.
- Comparisons on 360° scenes: More than 70x speed up is obtained relative to NeRF’s approximately 1.5-day optimization, while TensoRF models remain below 75MB.The reported reconstruction-time comparison is for the evaluated models.
- Forward-facing scenes: Forward-facing evaluations show better quality and substantially smaller model sizes than Plenoxels, with comparable or faster reconstruction speed.TensoRF also outperforms NeRF while taking significantly less reconstruction time.
7 Conclusion.
The conclusion presents TensoRF as a compact tensor-factorized radiance-field representation that supports fast, high-quality scene reconstruction. Its VM decomposition improves reconstruction quality and optimization speed while maintaining a small memory footprint.
- Representation: TensoRF models radiance fields compactly as factorized low-rank tensor components using tensor decomposition techniques.The representation is proposed for high-quality scene reconstruction and rendering.
- Decomposition: VM decomposition provides better reconstruction quality and faster optimization than classical CP decomposition within the TensoRF framework.The conclusion contrasts the two decomposition techniques at the method level.
- Results: Less than 30 min per scene and less than 75MB are reported for efficient reconstruction and compact modeling, with better rendering quality than NeRF.NeRF is reported to require substantially longer training time, exceeding 20 hours.
A TensoRF Representation Details.
TensoRF represents density and appearance feature grids as tensors and factorizes them into compact vector or vector-matrix components. These factors encode spatial features while supporting continuous radiance-field evaluation.
- Number of components: TensoRF-CP uses Rσ + Rc components, while TensoRF-VM uses 3(Rσ + Rc) components because VM has three component types.The CP component count is therefore matched by using a larger R than VM.
- Number of components: For VM models, the evaluated component settings range from 48 to 384, with Rσ and Rc allocated separately for density and appearance.The reported settings include 48, 96, 192, and 384 total components.
- Forward-facing settings: Forward-facing scenes use more components for the X−Y plane because those spatial modes contain more visible appearance information in NDC space.These components correspond to AZ = vZ ◦ MX,Y and can be viewed as compressed neural MPIs.
- Number of parameters: For a 300 × 300 × 300 grid with P = 27 appearance channels, dense storage requires 756M parameters, whereas CP and VM use factorized parameterizations.The passage introduces the parameter-count comparison between dense and factorized representations.
- Feature grids: Each voxel stores one density channel and P appearance channels, with appearance features used to compute view-dependent colors.The grid is split into density and appearance feature grids.
- Tensor factorization: CP factorizes the density and appearance tensors into vectors, whereas VM factorizes them into vector and matrix factors using outer products.Each voxel relates to one value from each XYZ-mode factor.
B More Implementation Details.
TensoRF optimizes tensor factors with rendering loss, regularization, and implementation strategies tailored to scene conditions. The regularizers promote sparsity or smoothness, while occupancy masking and learning-rate decay improve reconstruction efficiency and stability.
- Loss functions: The training objective combines squared rendering error with a weighted regularization term.The loss uses the ground-truth color and a regularization weight ω.
- Loss functions: L1 regularization is applied directly to density-factor parameters to encourage sparsity and improve extrapolated views and final renderings.The paper reports reduced floaters and outliers, and applies the loss only to density parameters.
- Loss functions: The L1 sparsity loss is normalized by the total number of tensor-factor parameters, with ω = 0.0004 for Synthetic-NeRF and Synthetic-NSVF.The normalization uses the component count and factor dimensions.
- Loss functions: For few-view or imperfectly captured real datasets, TensoRF uses total variation loss instead of L1 sparsity loss.This setting includes LLFF and Tanks&Temples, and downweights appearance parameters by 0.1 in the TV loss.
- Implementation: Binary occupancy masks computed at steps 2000 and 4000 avoid empty-space computation during reconstruction.The masks are generated from intermediate volume-density predictions.
- Implementation: Exponential learning-rate decay reduces the initial rates by a factor of 0.1 by the end of optimization, stabilizing late reconstruction.The decay is applied at every training step.
C More Evaluation.
The evaluation examines regularization, supervision, and the low-rank scene representation. TensoRF remains competitive without L1 regularization, while its tensor formulation addresses the over-parameterization of dense feature grids and captures shared scene structure.
- Ablation: Without L1 regularization, TensoRF still exceeds NeRF fidelity, reaching 31.01 on average on Synthetic-NeRF.The remaining gap with L1 is mainly attributed to floaters in empty space.
- Discussion: Dense feature grids are under-determined: a 300^3 grid with 27 channels has over 700M parameters, versus 64M supervised pixels from one hundred 800 × 800 images.The paper frames pruning, coarse-to-fine reconstruction, and additional losses as responses to this imbalance.
- Discussion: TensoRF models a 5D radiance-field function expressing both scene geometry and appearance, motivating a generally low-rank 4D tensor representation.The motivation is that scenes contain repeated geometry structures and material properties.
- Discussion: The tensor model combines low-rank constraints with neural networks through global basis components shared across spatial and feature dimensions.These components represent commonalities in scene geometry and appearance.
E Limitations and Future Work.
TensoRF reconstructs bounded 360° objects and forward-facing scenes, but its current formulation does not support unbounded scenes containing both foreground and background content.
- Limitations: TensoRF currently supports only bounded scenes represented by a single bounding box, excluding unbounded foreground-and-background scenes.The paper suggests combining separate foreground and spherical-background fields as a possible extension.
G Per-scene Breakdown.
The paper reports per-scene quantitative breakdowns and rendering examples across Synthetic-NeRF, Synthetic-NSVF, Tanks&Temples, and forward-facing datasets. The figures list the scenes included in each qualitative evaluation.
- Quantitative breakdown: Per-scene quantitative results are provided for Synthetic-NeRF, Synthetic-NSVF, Tanks&Temples, and forward-facing datasets.The breakdown covers the datasets listed in the appendix evaluation tables.
- Rendering examples: Synthetic-NeRF rendering examples are shown for Ship, Hotdog, Lego, Mic, Chair, Drums, Materials, and Ficus.Figure 6 orders these scenes from top to bottom.
- Rendering examples: Synthetic-NSVF rendering examples are shown for Spaceship, Robot, Toad, Lifestyle, Palace, Wineholder, and Steamtrain.Figure 7 orders these scenes from top to bottom.
- Rendering examples: Tanks&Temples rendering examples are shown for Family, Ignatius, Truck, Caterpillar, and Barn.Figure 8 orders these scenes from top to bottom.
- Rendering examples: Forward-facing rendering examples are shown for Flower, Fern, Fortress, Horn, Leaves, Orchids, T-Rex, and Room.Figure 9 orders these scenes from top to bottom.