Source-linked AI summary
Mixture of Volumetric Primitives for Efficient Neural Rendering
Stephen Lombardi, Tomas Simon, Gabriel Schwartz, Michael Zollhoefer, Yaser Sheikh, Jason Saragih
TL;DR
Existing neural-rendering representations struggle to combine thin-structure completeness, high resolution, memory efficiency, and real-time speed. MVP addresses this with movable volumetric primitives, shared convolutional computation, and tracking-linked parameterization, achieving high-quality dynamic rendering with reported real-time performance. The method still requires coarse tracked initialization and high-end hardware for real-time operation.
Problem
Existing meshes, volumetric grids, and implicit fields do not jointly provide thin-structure completeness, high resolution, memory efficiency, and real-time rendering.
Method
MVP combines movable volumetric primitives with shared convolutional computation and conditional ray marching to represent occupied space efficiently.
Results
MVP produces high-fidelity dynamic rendering with efficiency closer to mesh-based approaches and can render 896 × 832 binocular stereo views at 40Hz.
Takeaways & Limitations
MVP unifies volumetric and primitive-based representations while supporting dynamic scenes, drivability, high resolution, and efficient rendering.
Takeaways & Limitations
MVP requires a coarse tracked mesh for primitive initialization and a high-end computer and graphics card for real-time performance.
Abstract
from arXiv · showhide
Real-time rendering and animation of humans is a core function in games, movies, and telepresence applications. Existing methods have a number of drawbacks we aim to address with our work. Triangle meshes have difficulty modeling thin structures like hair, volumetric representations like Neural Volumes are too low-resolution given a reasonable memory budget, and high-resolution implicit representations like Neural Radiance Fields are too slow for use in real-time applications. We present Mixture of Volumetric Primitives (MVP), a representation for rendering dynamic 3D content that combines the completeness of volumetric representations with the efficiency of primitive-based rendering, e.g., point-based or mesh-based methods. Our approach achieves this by leveraging spatially shared computation with a deconvolutional architecture and by minimizing computation in empty regions of space with volumetric primitives that can move to cover only occupied regions. Our parameterization supports the integration of correspondence and tracking constraints, while being robust to areas where classical tracking fails, such as around thin or translucent structures and areas with large topological variability. MVP is a hybrid that generalizes both volumetric and primitive-based representations. Through a series of extensive experiments we demonstrate that it inherits the strengths of each, while avoiding many of their limitations. We also compare our approach to several state-of-the-art methods and demonstrate that MVP produces superior results in terms of quality and runtime performance.
1 INTRODUCTION
Existing volumetric methods trade memory, resolution, and rendering speed, while the ideal representation must also support driving and high-quality rendering. MVP combines volumetric completeness with primitive-based efficiency through movable primitives, shared computation, and empty-space-aware rendering.
- Motivation: Volumetric methods trade a large memory footprint against computationally expensive rendering, limiting resolution, detail, and real-time applications.The paper identifies memory efficiency, fast rendering, drivability, and high rendering quality as the desired combination.
- Limitations of Existing Methods: Neural Volumes uses an O(n^3) uniform voxel grid, which prevents high resolutions despite a warp field that improves occupancy.Its warp-field effectiveness is limited by warp resolution and unsupervised inverse-warp learning.
- Limitations of Existing Methods: Neural Radiance Fields address resolution with a compact representation but require billions of MLP evaluations, producing render times around thirty seconds per frame.NeRF also only handles static scenes, while coarse-to-fine acceleration can miss small structures.
- MVP: MVP uses jointly generated, overlapping volumetric primitives that are selectively ray-marched to reduce computation in empty space.Convolutional networks share computation when generating primitives for nearby regions.
- MVP: MVP combines volumetric and primitive-based advantages through a representation with correspondence and tracking integration, motion interpolation, and adjustable granularity.The representation is described as generalizing volumetric and primitive-based approaches.
2 RELATED WORK
Prior neural-rendering representations include point, mesh, multilayer, grid, and MLP-based approaches, each balancing topology, geometry, memory, view range, or rendering efficiency. MVP is presented as a hybrid trade-off that combines volumetric completeness with primitive-based efficiency.
- Point-based Representations: Point-based representations handle topological changes because they impose no connectivity between points.
- Mesh-based Representations: Mesh-based methods can render high-resolution models efficiently when accurate surface geometry is available but struggle with hair, eyes, and mouth interiors.
- Multi-Layer Representations: Multi-Plane Images synthesize views from depth-aligned textured planes but are normally limited to a small restricted view-box.MatryODShka is cited as enabling real-time 6DoF video view synthesis for VR by converting omnidirectional stereo images to MSIs.
- Grid-based Representations: Grid-based representations are easy to optimize and need no fixed template mesh, but their cubic memory footprint wastes capacity on empty space and limits resolution.
- MLP-based Representations: MLP-based representations model shape and appearance with implicit functions, including Neural Radiance Fields for volumetric scene representation.
- Summary: MVP is summarized as a hybrid that trades off volumetric and primitive-based representations while retaining fine detail, fast rendering, drivability, and reduced memory constraints.
3 METHOD
MVP represents dynamic scenes with a volumetric representation that combines volumetric and primitive-based advantages. It is trained end-to-end from 2D multi-view image observations using an encoder-decoder framework.
- 3 METHOD: MVP uses a novel volumetric representation for dynamic scenes to achieve high-performance decoding and efficient rendering.
- 3 METHOD: The representation is trained end-to-end from 2D multi-view image observations.
3.1 Neural Scene Representation
MVP models occupied regions with minimally overlapping, moving volumetric primitives linked softly to a guide mesh, enabling efficient sampling and rendering. An encoder-decoder predicts geometry, motion, color, opacity, and background components from a latent code, while windowing encourages primitives to move rather than retain opacity in empty regions.
- 3.1 Neural Scene Representation: Each primitive models a local region with a uniform voxel grid, enabling fast sampling and conditional ray marching that skips empty or fully occluded space.
- 3.1 Neural Scene Representation: Primitive motion, color, and opacity are parameterized by a convolutional network that shares computation across primitives for efficient decoding.
- Guide Mesh: A coarse tracked or reconstructed mesh initializes and regularizes primitive placement, while primitives may deviate from it when reconstruction improves.Static scenes may use COLMAP reconstruction; dynamic scenes use multi-view non-rigid tracking.
- Primitive Parameterization: Each primitive contains position, orientation, scale, and a payload describing its local volumetric content.
- Primitive Motion: Allowing primitives to move addresses initialization deficiencies, low guide-mesh quality, and incomplete object coverage, although gradients arise only where primitives currently cover space.
- Opacity Fade Factor: The opacity window decays toward primitive edges, downscaling edge gradients so training favors explaining content through motion rather than payload opacity.The window does not prevent edge opacity from becoming large; it makes those values learn more slowly.
- Network Architecture: A VAE-based encoder-decoder uses a 256-dimensional latent code to parameterize the guide mesh and volumetric primitives.The encoder encourages a structured latent space during training and can be discarded after training.
- Network Architecture: The decoder predicts guide-mesh geometry, primitive motion deviations, voxel-grid opacity, and view-dependent RGB color using fully connected, convolutional, and transposed-convolutional modules.
3.2 Efficient and Differentiable Image Formation
MVP focuses computation on occupied 3D regions and forms images by efficiently ray-marching primitives before differentiable volumetric aggregation. Its rendering design skips empty space, locates irregular primitives efficiently, and supports end-to-end image formation.
- 3.2 Efficient and Differentiable Image Formation: MVP concentrates encoder-decoder capacity on occupied regions, enabling high-resolution representation and efficient decoding.
- 3.2.1 Efficient Raymarching: Efficient ray marching is designed to skip samples in empty space and use efficient payload sampling.
- 3.2.1 Efficient Raymarching: A data-parallel BVH identifies which irregular, moving primitives contain each ray-marching evaluation point.
- 3.2.1 Efficient Raymarching: Given occupied-space samples, MVP uses accumulative volume rendering to integrate color and opacity for differentiable image formation.
- 3.2.1 Efficient Raymarching: The rendering integral is solved numerically along rays defined by an origin, direction, and sampled interval.
- 3.2.2 Differentiable Volumetric Aggregation: Early stopping terminates ray marching once accumulated opacity exceeds 1.0 − 𝜖early, avoiding samples with insignificant impact on pixel color.
- 3.2.2 Differentiable Volumetric Aggregation: When samples lie in multiple primitives, their values are combined in BVH order using the accumulation scheme.
- 3.2.2 Differentiable Volumetric Aggregation: The additive integration formulation is ordering-independent up to saturation and reduces backward-pass memory by avoiding the full O(n^2) operation graph.
3.3 End-to-end Training
MVP is trained end-to-end from multi-view video using jointly optimized rendering, geometry, volume, deformation, and latent-space regularization terms. The training combines photometric supervision with mesh-based motion constraints while allowing primitives to deviate when image reconstruction improves.
- End-to-end optimization: Trainable parameters Θ are optimized to explain multi-view video comprising Nframes · Ncams training images.
- End-to-end optimization: ADAM minimizes the objective with stochastic mini-batch optimization, uniformly sampling rays from each image and using learning rate lr = 0.0001.
- Training objective: The training objective combines photometric reconstruction, coarse geometry reconstruction, volume minimization, delta magnitude, and KL-divergence losses.
- Training objective: Photometric reconstruction compares synthesized pixels with ground-truth pixels to enforce photo-realistic image matching.
- Training objective: Mesh reconstruction uses tracked mesh vertices to pull weakly linked volumetric primitives toward approximately correct positions.
- Training objective: The volume minimization prior discourages excessive primitive overlap and prevents large primitives from covering empty space and reducing resolution.
4 RESULTS
MVP is evaluated through dataset experiments, ablations, and comparisons showing how primitive count, priors, voxel resolution, step size, and architecture affect quality and runtime. The optimized model improves the quality–efficiency trade-off over Neural Volumes and NeRF while supporting detailed dynamic rendering.
- Experimental Setup: The evaluation uses approximately 100 synchronized cameras, held-out views, qualitative and quantitative metrics, and extensive ablations across challenging sequences.Scenes are trained for 500,000 iterations, taking roughly five days on a single NVIDIA Tesla V100.
- Comparisons: MVP outperforms state-of-the-art methods in reconstruction quality and runtime, rendering binocular 896 × 832 views at 40Hz and improving over NeRF runtime by three orders of magnitude.Compared with Neural Volumes, MVP produces sharper, more detailed reconstructions while rendering much faster; it also captures both geometric and texture details against NeRF.
- Number of Primitives: 256 primitives with 8 million voxels form the best model configuration, while hundreds of primitives provide the best quality–performance balance in the primitive-count study.The study keeps the base comparison at approximately 2 million voxels and evaluates held-out views.
- Primitive Volume Prior: Stronger primitive-volume priors reduce overlap and speed raymarching, but excessive strength causes holes and increases reconstruction error.The trade-off arises because over-shrinking leaves image evidence insufficient to expand the primitives.
- Voxel Count and Step Size: Smaller raymarching steps and more voxels recover finer detail and reduce reconstruction error, but increase runtime because decoding and raymarching scale linearly with their respective workloads.Recovered details include hair and wrinkles.
- Architecture: The convolutional architecture outperforms a simple latent-to-motion linear layer for almost all primitive counts and produces configurations that follow scene surfaces more closely.The linear model instead creates more zero-opacity primitives that fly away and waste resolution.
5 LIMITATIONS
MVP has several practical limitations: it depends on coarse tracked mesh initialization, high-end hardware for real-time performance, and scene-specific primitive-count selection.
- MVP requires a coarse tracked mesh to initialize volumetric primitive positions, rotations, and scales.The authors aim to replace this requirement with image-based self-organization.
- Real-time performance requires a high-end computer and graphics card, partly because overlapping primitives require multiple trilinear interpolations per sample point.The authors suggest reducing primitive overlap through regularization.
- The number of volumetric primitives is predefined and must be empirically determined for each scene type.Future work could optimize this selection automatically.
6 CONCLUSION
The paper presents MVP as a dynamic-scene representation that combines volumetric and primitive-based paradigms. It reports high-resolution, fast, drivable rendering with higher quality than the current state of the art.
- MVP handles dynamic scenes, renders quickly, supports driving, and represents 3D space at high resolution.
- An encoder-decoder network regresses a mixture of volumetric primitives as the core scene representation.
- MVP uses combined 2D and 3D supervision during training.
- The unified representation generalizes volumetric and primitive-based paradigms, combining their advantages for high-performance decoding and efficient dynamic-scene rendering.
- The comparisons report higher-quality results than the current state of the art.