Source-linked AI summary
Nuvo: Neural UV Mapping for Unruly 3D Representations
Pratul P. Srinivasan, Stephan J. Garbin, Dor Verbin, Jonathan T. Barron, Ben Mildenhall
TL;DR
Existing UV methods are designed for well-behaved meshes, while reconstruction and generation can produce geometry that yields fragmented atlases. Nuvo optimizes neural-field UV mappings over visible points, and the paper reports competitive results on simple meshes and better results than baselines on challenging NeRF-derived geometry.
Problem
Existing UV mapping methods target well-behaved meshes, whereas reconstruction and generation produce NeRF fields and extracted meshes for which these methods can yield unusably fragmented atlases.
Source: §1 · p. 2Method
Nuvo uses neural fields and visible-point sampling to optimize a well-behaved UV mapping without requiring a mesh or strong assumptions about manifoldness, connectivity, or smoothness.
Source: §1 · p. 2Results
Nuvo is competitive with OptCuts on simpler meshes and significantly better than all baselines on challenging NeRF-extracted geometry.
Source: §4.4 · p. 8Takeaways & Limitations
Nuvo produces editable UV atlases without severe fragmentation and distortion for challenging geometry from 3D reconstruction and generation.
Source: §5 · p. 8Point-sample optimization cannot absolutely guarantee bijectivity or globally minimal distortion, and Nuvo lacks interactive controls for cut locations or preferred low-distortion regions.
Source: §4.4 · p. 8
Abstract
from arXiv · showhide
Existing UV mapping algorithms are designed to operate on well-behaved meshes, instead of the geometry representations produced by state-of-the-art 3D reconstruction and generation techniques. As such, applying these methods to the volume densities recovered by neural radiance fields and related techniques (or meshes triangulated from such fields) results in texture atlases that are too fragmented to be useful for tasks such as view synthesis or appearance editing. We present a UV mapping method designed to operate on geometry produced by 3D reconstruction and generation techniques. Instead of computing a mapping defined on a mesh's vertices, our method Nuvo uses a neural field to represent a continuous UV mapping, and optimizes it to be a valid and well-behaved mapping for just the set of visible points, i.e. only points that affect the scene's appearance. We show that our model is robust to the challenges posed by ill-behaved geometry, and that it produces editable UV mappings that can represent detailed appearance.
1. Introduction
UV mapping flattens 3D surfaces to support detailed-appearance representation and editing, but geometry from neural reconstruction and generation can make existing methods produce fragmented atlases. Nuvo uses neural fields and visible-point sampling to optimize mappings, addressing this setting alongside prior mesh-parameterization approaches.
1. Introduction
- NeRF-derived geometry is often non-manifold, non-smooth, and highly disconnected, leaving existing UV methods unable to operate or producing fragmented atlases that complicate editing and differentiable rendering.Triangulated density fields can also contain holes into hidden geometry and many small, bumpy triangles.Source: §1 · p. 2
- Nuvo optimizes a neural-field UV mapping from visible surface samples, requiring neither a mesh-specific representation nor strong manifoldness, connectivity, or smoothness assumptions.It is optimized from scratch for each scene using losses that encourage a well-behaved mapping for observed points.Source: §1 · p. 2
- Nuvo produces high-quality, low-fragmentation atlases across well-behaved meshes, NeRF density fields, meshes extracted from NeRF, and text-to-3D meshes.The paper reports that these atlases can represent and edit detailed surface appearance.Source: §1 · p. 2
2. Related Work
- Only a few prior mesh-parameterization methods jointly optimize cuts and UV mappings; many require a user-defined cut or a manifold mesh and rely on mesh connectivity.Nuvo instead samples surface points rather than explicitly building and solving mesh-dependent systems.Source: §2 · p. 2
- Prior neural mapping work NeuTex and follow-ups [39] cannot represent multi-chart atlases, while OptCuts cannot run on high-complexity meshes extracted from neural fields even after manual cleanup.Methods that improve an existing UV mapping also require an initial parameterization.Source: §2 · p. 3
3. Method
Nuvo represents each chart with neural mappings between visible 3D surface points and 2D texture coordinates, and learns chart assignments alongside those mappings. Its losses jointly encourage compact charts, low distortion, usable texture coordinates, and mappings that cover and invert the scene and UV domain.
3. Method
- Nuvo uses one chart-assignment MLP, n 3D-to-UV texture-coordinate MLPs, and n inverse UV-to-3D surface-coordinate MLPs.The mapping is optimized from samples of the scene geometry and texture space.Source: §3 · p. 3, §3.1 · p. 4
3.1. Losses
- Two cycle-consistency losses regularize mappings in both directions: 3D-to-UV-to-3D and UV-to-3D-to-UV.The first encourages injectivity but not full UV coverage; the reverse cycle encourages coverage but does not prevent degenerate mappings over parts of the scene.Source: §3.1 · p. 4
- Chart-assignment entropy encourages points mapped from chart i's UV space to be assigned confidently to chart i.Together with the cycle losses, it encourages a partition into bijective charts.Source: §3.1 · p. 4, Figure 3 · p. 4
- A symmetric Chamfer loss encourages UV samples mapped into 3D to lie on the input geometry and cover its sampled points.Without this constraint, UV regions could map to points off the surface while still satisfying the reverse cycle loss.Source: §3.1 · p. 4
- The clustering loss penalizes distances from surface points to chart centroids in proportion to their chart-assignment probabilities, discouraging fragmented atlases.Each point's distance to a chart centroid is weighted by the probability that the point belongs to that chart.Source: §3.1 · p. 5
- Distortion losses encourage angle-preserving mappings and uniform stretch across the scene.The conformal term regularizes transformed orthogonal tangent vectors, while the stretch term penalizes deviations from an optimized average stretch.Source: §3.1 · p. 5
- A jointly optimized pixel grid represents surface normals in UV space, and matching them to true normals helps encourage injectivity.The normal-matching penalty is optimized alongside the mapping MLPs.Source: §3.1 · p. 5
4. Experiments
Nuvo is evaluated on conventional meshes and challenging NeRF-derived geometry for view synthesis and editable mapping. Across these experiments, it produces competitive results on well-behaved meshes and stronger mappings on challenging NeRF geometry, with a limitation in guaranteeing global mapping properties and interactive control.
4.4. Discussion
- Nuvo is competitive with OptCuts on well-behaved meshes and significantly outperforms all baselines on NeRF-derived meshes, where OptCuts cannot compute mappings.The comparisons show Nuvo’s mapping on amigurumi, while OptCuts recovers a high-quality mapping on the lion mesh but not challenging NeRF-derived meshes.Source: Figure 5 · p. 6, Figure 6 · p. 7, §4.4 · p. 8
4. Experiments
- The view-synthesis evaluation uses Zip-NeRF reconstructions of gardenvase and amigurumi, while the broader tests include bunny and lion meshes and NeRF-extracted meshes.The NeRF-derived meshes have non-smooth geometry, thousands of connected components, and non-manifold edges.Source: §4 · p. 5, §4.1 · p. 6
4.1. Nuvo Represents Detailed Appearance
- For view synthesis, Nuvo performs similarly or better than BakedSDF while using less memory by allocating none to triangles that do not affect view synthesis.The evaluation uses the same view-dependent appearance model, optimized in UV atlas space for Nuvo and on mesh vertices for BakedSDF.Source: §4.1 · p. 6, Table 1 · p. 6
- Baking Nuvo’s continuous UV mappings onto mesh vertices incurs only a minimal decrease in view-synthesis performance.The baked coordinates are stored as vertex attributes, providing a piecewise-linear approximation of the MLP-predicted mappings.Source: §4.1 · p. 6
4.2. Nuvo Produces Editable Mappings
- Boundary measures the fraction of rendered pixels away from chart boundaries, while Editability measures how well arbitrary textures can be wrapped on the geometry.Editability averages angular and area distortion over triangles, excluding pixels on chart boundaries.Source: §4.2 · pp. 7–8
4.3. Ablation Studies
- The full model achieves the best tradeoff across fragmentation, distortion, and UV-space use; texture optimization also avoids the two-sided ear-coordinate degeneracy seen without it.Without texture optimization, both sides of the bunny’s left ear map to the same UV coordinates.Source: §4.3 · p. 8, Figure 7 · p. 8
4.4. Discussion
- Point-sample optimization makes bijectivity and globally minimal distortion difficult to guarantee, and Nuvo lacks interactive controls for cut locations or regions where distortion should be minimized.These are stated as limitations and directions for future work.Source: §4.4 · p. 8
5. Conclusion
Nuvo produces editable UV atlases without severe fragmentation or distortion on challenging geometry from 3D reconstruction and generation. It focuses on visible surfaces and uses neural fields rather than mesh vertices, and the authors identify creative editing of reconstructed content as a potential application.
5. Conclusion
- Nuvo produces editable UV atlases without severe fragmentation or distortion, including for challenging geometry from 3D reconstruction and generation.The method focuses on visible surfaces and parameterizes mappings with neural fields rather than directly on mesh vertices.Source: §5 · p. 8
6. Video
The section directs readers to the supplemental video for additional results.
6. Video
- The authors encourage readers to watch the supplemental video for additional results.No additional video content is described here.Source: §6 · p. 11
7. Experimental Details
The experimental details specify the MLP architecture, point-sampling procedures for different geometry types, optimization settings, and runtimes.
MLP Architecture
- The chart-assignment, texture-coordinate, and surface-coordinate MLPs each use eight fully connected layers with 256 channels and ReLU activations.The chart-assignment positional-encoding degree is 1; the other two use degree 4.Source: §7 · p. 11
Point Sampling
- Nuvo samples surface points from mesh triangles for well-behaved meshes, from ray-mesh intersections for NeRF-extracted meshes, and by volume-rendering-weight resampling for NeRF density fields.The rays come from training images for Zip-NeRF and from random circular-view images for DreamFusion.Source: §7 · p. 11
Runtimes
- Nuvo optimization takes about 20 minutes on meshes and 40 minutes directly on NeRF density fields; OptCuts takes over two hours on bunny and over 35 hours on lion.xatlas takes about two minutes on bunny and 40 minutes on lion.Source: §7 · p. 11
8. Metrics
The section defines four rendered-image metrics for atlas fragmentation, area and angular distortion, and overall editability.
8. Metrics
- Boundary measures the fraction of rendered pixels that do not lie on chart boundaries, with boundary pixels identified by UV discontinuity rather than depth discontinuity.A pixel is a boundary pixel when its UV coordinate is more than 0.1 from an adjacent pixel’s coordinate and it is not on a depth discontinuity.Source: §8 · p. 11
- Stretch measures triangle area distortion as UV-space area divided by mesh area, scoring uniform stretch as 1.The score is 1 minus the median difference between each triangle’s stretch and the median triangle stretch.Source: §8 · p. 11
- Conformal measures angular distortion using tangent and bitangent directions, with mappings without angular distortion scoring 1.The metric is defined as 1 minus the median cosine between the tangent vectors.Source: §8 · p. 11
- Editability is the mean of Stretch and Conformal, each weighted by Boundary; editable mappings therefore need low fragmentation and distortion.Boundary, Stretch, and Conformal should each be close to 1.Source: §8 · p. 11
9. Synthetic Data Sources
The supplied materials describe scanned and generated meshes used in examples, alongside visual comparisons of UV mappings and loss ablations.
9. Synthetic Data Sources
- The lion and Stanford Bunny are scan-derived meshes, while the Nefertiti bust and Hydria vase are additional scanned examples.The lion scan depicts the Bayon Lion statue; the Nefertiti bust was scanned at Berlin’s Neues Museum, and the vase was digitized by the Regional Digitalisation Lab of the Malopolska Institute of Culture.Source: §9 · pp. 11–12
- On the bunny, removing the distortion loss causes non-uniform warping, while removing chart clustering causes fragmented, irregular charts.Both changes make appearance editing more cumbersome.Source: Figure 8 · p. 12
- The bunny comparison shows Nuvo with a simple boundary and low distortion, while xatlas and Blender produce fragmented atlases and NeuTex produces a distorted mapping.OptCuts maps the well-behaved bunny successfully but cannot map the more challenging NeRF-extracted meshes.Source: Figure 9 · p. 12
- The tile texture used in the main paper’s Figure 5 comes from ambientCG.The passage identifies the texture as Tiles101.Source: §9 · p. 12
10. Ablation Visualization
The ablation visualization shows the effects of removing the distortion and chart-clustering losses, with quantitative ablation results reported in Table 3.
10. Ablation Visualization
- Figure 8 visualizes how removing the distortion or chart-clustering loss affects the bunny’s UV mapping.Table 3 provides quantitative results for all loss ablations.Source: §10 · p. 12
- Table 3 contains quantitative results for all loss ablations.The supplied passage does not report the table’s values.Source: §10 · p. 12
11. Additional Qualitative Results
The additional qualitative comparisons report that Nuvo’s mappings are better suited to editing; the expanded comparison documents metrics used to calculate editability.
11. Additional Qualitative Results
- Figure 9 qualitatively compares Nuvo with existing methods and reports that Nuvo’s mappings are better suited to editing.The supplied passage does not enumerate the comparison outcomes.Source: §11 · p. 12
12. Expanded Editability Comparison
- In the expanded editability comparison, Table 4 reports the Boundary, Area, and Angle metrics, each higher-is-better, and states that these metrics are used to compute Table 2’s Editability metric.Source: Table 4 · p. 13