Source-linked AI summary
ExMesh++: From Multi-View Images to Relightable UV-PBR Mesh Assets via Topology-Adaptive Reconstruction and Decomposition
Chuanjin Fan, Lifan Wu, Wenjie Chang, Hanzhi Chang, Wenfei Yang, Tianzhu Zhang
TL;DR
Existing reconstruction methods often do not directly yield editable mesh-UV-PBR assets, and joint geometry-material-light optimization can make decomposition less reliable. ExMesh++ stages topology-adaptive explicit carrier reconstruction before fixed-carrier UV-PBR decomposition, with one-bounce diffuse indirect illumination. It achieves competitive geometry, material-recovery, and relighting performance while exporting assets usable in downstream workflows.
Problem
Practical relightable reconstruction requires editable mesh-UV-PBR assets, but existing representations and coupled optimization do not reliably provide a stable final carrier and decomposition.
Method
ExMesh++ adaptively reconstructs mesh geometry, topology, and UVs, then freezes the carrier to optimize shared UV-space PBR maps, lighting, and one-bounce diffuse interreflection.
Results
Experiments demonstrate competitive geometry, material-recovery, and relighting performance, with exported assets directly usable for editing, relighting, and composition.
Takeaways & Limitations
The framework produces explicit UV-PBR mesh assets that support relighting and downstream DCC workflows.
Takeaways & Limitations
The light-transport model captures only one-bounce diffuse interreflection and excludes multi-bounce transport and indirect specular reflection.
Abstract
from arXiv · showhide
Multi-view reconstruction extends beyond surface recovery to editable and relightable mesh assets. Such assets require well-formed topology, valid UV parameterization, and explicit PBR material maps. Existing surface reconstruction approaches optimize implicit fields, Gaussian primitives, or other intermediate representations. Converting them into such assets often requires surface extraction and texture baking. Inverse-rendering methods estimate materials and illumination, yet these components often remain tied to neural fields or point-based primitives rather than the final mesh. Joint optimization of geometry, materials, and lighting may also allow these variables to compensate for one another, leading to ambiguous decomposition. To address these limitations, we present ExMesh++, a staged framework for reconstructing relightable UV-PBR mesh assets from multi-view images. The first stage refines explicit mesh geometry and topology through adaptive vertex splitting and merging, while maintaining UV consistency as the topology changes. The second stage fixes the resulting mesh-UV carrier and optimizes UV-space PBR maps together with environment lighting. Building on this stable carrier, ExMesh++ models one-bounce diffuse indirect illumination through secondary-ray tracing with shared UV-PBR materials. Experiments demonstrate competitive geometry accuracy, strong relighting performance, and direct usability of the exported assets in standard DCC workflows.
1 Introduction
ExMesh++ targets editable, relightable mesh-UV-PBR assets rather than surface representations alone. Its staged design separates topology-adaptive carrier reconstruction from material-light decomposition and supports downstream asset workflows.
- Multi-view reconstruction for practical graphics requires well-formed topology, valid UVs, and explicit PBR maps in addition to geometric accuracy.
- Existing neural, Gaussian, and inverse-rendering approaches often do not directly produce final editable mesh-UV-PBR assets.
- ExMesh++ reconstructs an explicit mesh-UV carrier through adaptive vertex splitting and merging while maintaining UV consistency during topology changes.
- The second stage freezes the carrier and optimizes UV-space PBR maps with environment lighting, reducing geometry-appearance compensation during decomposition.
- One-bounce diffuse indirect illumination uses shared UV-PBR materials, and experiments evaluate geometry, relighting, novel-view synthesis, and asset usability.
2 Related Work
Related work spans implicit surfaces, Gaussian primitives, explicit mesh optimization, UV parameterization, and inverse rendering. ExMesh++ is motivated by preserving a stable mesh-UV carrier while separating material-light decomposition and modeling lightweight indirect illumination on the mesh.
- Neural implicit surface reconstruction: Neural implicit methods improve multiview surface reconstruction but retain optimized representations that are not directly final mesh assets.
- Gaussian-based surface reconstruction: Gaussian-based methods improve efficiency and surface awareness, yet their primitives remain unstructured or require mesh extraction for explicit surfaces.
- Texture representations and UV parameterization: Topology-changing optimization must update face-corner UV indices, seams, and islands without disrupting texture-surface correspondence.
- Inverse rendering: Inverse-rendering methods decompose geometry, materials, and illumination for relighting across neural fields, Gaussian primitives, and mesh-based formulations.
- Motivation: Joint optimization can let geometry and appearance compensate for one another, motivating staged carrier reconstruction followed by material-light decomposition.
3 Overview
ExMesh++ uses two stages: topology-adaptive reconstruction first establishes a compact mesh-UV carrier, then fixed-carrier inverse rendering estimates PBR materials and lighting. Its topology operations split selected edges or collapse edges to simplify local structure.
- Stage I alternates differentiable geometry and texture optimization with vertex splitting and merging while updating UVs for topology consistency.
- Stage II freezes the mesh and UV coordinates, then decomposes appearance into UV-space PBR maps and environment illumination.
- Vertex splitting inserts a vertex on a selected edge and replaces adjacent faces, while vertex merging collapses an edge to simplify local topology.
4 Topology-Adaptive Mesh-UV Reconstruction
ExMesh++ adaptively refines explicit mesh topology while preserving a valid UV parameterization, then optimizes appearance through differentiable rendering. Vertex splitting targets complex or under-resolved regions, while merging removes redundant or degenerate structures and maintains a compact mesh-UV asset.
- Asset representation: Stage I represents the asset with explicit vertices and triangular faces, UV coordinates, a face-corner UV index map, and a fixed-resolution RGB texture.This decouples appearance resolution from geometric density during reconstruction.
- Vertex splitting: Vertex splitting prioritizes faces using temporal position-gradient scores and local curvature, combining optimization signals with geometric variation.Candidates are restricted to relatively large faces to avoid ineffective refinement in already dense regions.
- Vertex splitting: The splitting edge maximizes length divided by endpoint degree, favoring long edges while reducing unstable changes near high-valence vertices.Interior-edge splits create four faces, while boundary-edge splits create two; invalid splits are discarded when the new vertex is too close to an endpoint.
- Vertex merging: Merging candidates are selected from faces that are never rendered or geometrically degenerate, with small-area filtering to avoid oversimplifying large structures.The degeneracy measure uses face area and the squared maximum edge length.
- Vertex merging: Vertex merging collapses a boundary edge when present or otherwise the shortest edge, removes the lower-degree endpoint, updates adjacent faces, and skips conflicting operations.The operation eliminates degenerate faces and restores clean local topology.
- UV maintenance and optimization: UV updates remain consistent through splitting and merging by preserving seam-separated coordinates, compacting unreferenced UV entries, and maintaining valid face-corner indices.Stage I renders RGB, alpha, and depth outputs for image, depth, silhouette, smoothing, and deformation losses, producing a mesh, UV parameterization, and RGB texture.
5 UV-PBR Asset Decomposition
Stage II fixes the mesh-UV carrier and decomposes appearance into UV-space PBR maps and environment lighting, with direct and one-bounce diffuse illumination rendered through shared materials.
- Stage-II representation: The second stage fixes vertex positions, topology, and UV coordinates while optimizing PBR maps and environment lighting in shared UV space.Optimized maps include base color, roughness, normal, optional metallic, and a learnable RGB lat-long environment map.
- Stage-II representation: PBR attributes are queried from UV maps at visible surface intersections using face indices, barycentric coordinates, and differentiable bilinear sampling.The same attribute-query procedure is used for primary and secondary surface points.
- Direct illumination: Direct illumination integrates environment radiance, visibility, the metallic-roughness BRDF, and the shading-normal cosine over the upper hemisphere.Monte Carlo sampling uses an importance distribution based on environment brightness and spherical-area weights, with ray tracing for visibility tests.
- Direct illumination: Metallic-roughness decomposition activates diffuse reflection for non-metallic components and shifts toward base-color-controlled specular reflection as metallic approaches one.Roughness controls the microfacet distribution and the concentration of specular highlights.
- Indirect illumination: One-bounce diffuse indirect illumination samples secondary directions, traces rays on the fixed mesh, and evaluates shared UV-PBR attributes at valid secondary hits.Indirect illumination excludes indirect specular reflection and is introduced gradually by increasing its loss weight from zero to the target value.
- Stage-II optimization: Stage-II training combines image reconstruction with material and environment regularization to reduce decomposition ambiguity and suppress spatial or illumination noise.Material regularization encourages smooth visible attributes, while environment regularization suppresses high-frequency noise; an early chroma term discourages global color casts.
6 Experiments
ExMesh++ achieves competitive geometry reconstruction and strong relighting while producing compact, editable mesh-UV-PBR assets. Ablations show that topology adaptation, UV maintenance, staged decomposition, and indirect lighting each contribute to the final results.
- Geometry Reconstruction: ExMesh++ achieves an average Chamfer Distance of 0.58 on DTU with 13 minutes of training and a compact mesh representation.PGSR and GeoSVR achieve lower Chamfer Distance but require longer training and larger geometric representations.
- Geometry Reconstruction: ExMesh++ recovers detailed, structurally complete DTU surfaces with fewer floating artifacts, redundant faces, and isolated fragments than compared methods.The meshes maintain smooth and intact surfaces while recovering local details.
- Relighting and Material Recovery: ExMesh++ obtains the highest relighting PSNR on Synthetic4Relight while remaining competitive in novel-view synthesis, albedo, and roughness estimation.The method directly produces explicit mesh-UV-PBR assets.
- Relighting and Material Recovery: On Stanford-ORB, ExMesh++ outperforms compared methods on all novel-scene relighting metrics and achieves the lowest depth, normal, and Chamfer Distance errors.Visual comparisons also show fewer illumination-color, smoothing, and high-frequency-noise artifacts than competing approaches.
- Indirect Lighting: One-bounce indirect lighting produces more continuous environment-dependent illumination changes around the can lid and occluded regions than competing indirect-light components.The full model retains local color transfer without the colorful high-frequency artifacts or dark-region concentration observed in alternatives.
- Topology Ablation: Adaptive splitting and merging preserve a compact mesh and more continuous surface, whereas random operations destabilize local triangulation quality.Using only merging reduces vertices but cannot increase resolution in complex regions, while using only splitting adds local detail at the cost of more vertices.
- UV Reconstruction: Periodic UV reconstruction redistributes texture space after topology changes, producing clearer local details and a more regular UV layout.Without reconstruction, repeated splitting and merging can cause atlas distortion, fragmented islands, and uneven texel allocation.
- PBR Ablation: The two-stage design is necessary because Stage-I RGB textures mix intrinsic color with shadows and highlights, while Stage-II-only reconstruction leaves rough geometry and local appearance errors.Gray base-color initialization recovers main colors but produces noisier albedo maps.
7 Limitations
ExMesh++ is constrained by simplified appearance and light-transport models, computationally expensive UV regeneration, and an object-level evaluation scope.
- The appearance model uses an opaque, isotropic metallic-roughness BRDF and excludes anisotropic reflection, transmission, and subsurface scattering.Metallic optimization is disabled in benchmarks, and optional metallic-map recovery is demonstrated through editing rather than quantitatively evaluated.
- One-bounce diffuse interreflection captures local color bleeding and occluded-region illumination but omits multi-bounce transport and indirect specular reflection.Richer light transport could improve physical realism while increasing optimization and rendering costs.
- CPU-based xatlas UV regeneration becomes a bottleneck for high-resolution meshes.Scaling beyond object-level assets requires more efficient UV parameterization and memory management for larger scenes, texture atlases, and complex meshes.
8 Conclusion
ExMesh++ reconstructs editable, relightable UV-PBR mesh assets through staged geometry-topology refinement followed by material and lighting decomposition. Experiments show competitive performance, and the exported assets support editing and relighting in standard DCC workflows.
- ExMesh++ uses Stage I to refine explicit mesh geometry and topology through vertex splitting and merging while maintaining valid UV parameterization.
- Stage II freezes the mesh-UV carrier and decomposes appearance into UV-space PBR material maps and environment lighting.
- The shared UV-PBR representation supports one-bounce diffuse indirect illumination without an additional learned residual appearance field.
- Experiments on synthetic and real-captured datasets demonstrate competitive geometry, material-recovery, and relighting performance.
- Exported assets can be directly edited, relit, and composed with artist-created content in standard DCC workflows.