Source-linked AI summary
Deep Mesh Reconstruction from Single RGB Images via Topology Modification Networks
Junyi Pan, Xiaoguang Han, Weikai Chen, Jiapeng Tang, Kui Jia
TL;DR
Single-view mesh reconstruction must handle objects whose topology differs from a deformable template, while multiple patches create stitching and meshing difficulties. The paper proposes an end-to-end framework that alternates deformation with face-pruning topology modification and boundary refinement. Experiments report better quantitative and qualitative performance than existing methods, especially for complex topologies, although face pruning can produce non-closed meshes.
Problem
Template-deformation methods cannot represent differing topologies well, while multiple patches create disconnected surfaces and stitching or meshing difficulties.
Method
The framework progressively alternates mesh deformation and topology modification from a single genus-0 template, then refines open boundaries.
Results
The method significantly outperforms existing methods quantitatively and qualitatively, particularly for shapes with complex topologies.
Takeaways & Limitations
Topology can be evolved during single-view mesh reconstruction rather than fixed by the initial genus-0 template.
Takeaways & Limitations
Face-pruning operations inherently produce non-closed meshes, requiring post-processing from densely sampled point clouds to reconstruct closed surfaces.
Abstract
from arXiv · showhide
Reconstructing the 3D mesh of a general object from a single image is now possible thanks to the latest advances of deep learning technologies. However, due to the nontrivial difficulty of generating a feasible mesh structure, the state-of-the-art approaches often simplify the problem by learning the displacements of a template mesh that deforms it to the target surface. Though reconstructing a 3D shape with complex topology can be achieved by deforming multiple mesh patches, it remains difficult to stitch the results to ensure a high meshing quality. In this paper, we present an end-to-end single-view mesh reconstruction framework that is able to generate high-quality meshes with complex topologies from a single genus-0 template mesh. The key to our approach is a novel progressive shaping framework that alternates between mesh deformation and topology modification. While a deformation network predicts the per-vertex translations that reduce the gap between the reconstructed mesh and the ground truth, a novel topology modification network is employed to prune the error-prone faces, enabling the evolution of topology. By iterating over the two procedures, one can progressively modify the mesh topology while achieving higher reconstruction accuracy. Moreover, a boundary refinement network is designed to refine the boundary conditions to further improve the visual quality of the reconstructed mesh. Extensive experiments demonstrate that our approach outperforms the current state-of-the-art methods both qualitatively and quantitatively, especially for the shapes with complex topologies.
1. Introduction
Single-view mesh reconstruction benefits from learned shape priors but remains constrained by template topology. The paper introduces progressive topology modification from a single genus-0 template to improve reconstruction of complex shapes and mesh quality.
- Motivation: Single-view reconstruction is difficult because one image lacks correspondence and contains large occlusions, despite deep networks learning useful 3D shape priors.Mesh representations are emphasized because they preserve fine shape details and support geometry regularizers.
- Limitations of prior methods: Template-deformation methods remain tied to the template topology, while multiple patches introduce disconnected surfaces, intersections, atlas difficulties, and uncertain patch counts.These issues make complex-topology reconstruction difficult to use in practice.
- Proposed approach: The framework progressively alternates mesh deformation with topology modification, pruning high-error faces to evolve a mesh from a single genus-0 template.An error-prediction network estimates reconstructed-face error, and faces with large error are removed.
- Contributions: The paper contributes an end-to-end framework, a topology modification network, and demonstrations of improved reconstruction accuracy and meshing quality over state-of-the-art methods.The topology modification network is designed to integrate with other mesh learning frameworks.
2. Related Works
Prior 3D reconstruction methods use multiview geometry, volumetric or mesh representations, and rendering-based learning, but mesh methods generally retain fixed template topology. This work addresses that limitation with topology modification from a genus-0 model.
- Prior representations: Multiview stereo can produce high-quality reconstructions but is restricted by available views and appearance assumptions, motivating learned single-view shape generation.Large-scale 3D shape collections support deep learning approaches to 3D representation and reconstruction.
- Mesh representations: Voxel representations are simple and convolution-compatible, whereas meshes better preserve fine shape details and support geometry regularizers for applications.Mesh learning approaches commonly deform template meshes because changing topology is difficult.
- Topology limitation: Differentiable-renderer methods can train mesh generators without 3D supervision, but existing approaches do not dynamically modify template topology.The paper positions its topology modification network as the first such approach in this line of work.
- Paper response: The proposed method generates complex-topology meshes from a genus-0 3D model by introducing a topology modification network.This directly targets the fixed-topology limitation of prior mesh-learning methods.
3. Topology-adaptive Mesh Reconstruction
The framework progressively reconstructs a mesh by alternating vertex deformation with face-based topology modification, then refines open boundaries for smoother results. Its training combines reconstruction, error-estimation, boundary, and geometry losses.
- Progressive Reconstruction: The pipeline progressively updates a predefined mesh by deforming vertex coordinates while separately modifying vertex connectivity.The deformation module maps template vertices toward the target surface while maintaining connectivity; the topology module updates connection relationships.
- Progressive Reconstruction: The network uses two successive deformation-and-topology stages followed by a boundary refinement subnet, starting from a 2562-vertex sphere mesh.An encoder extracts a 1024-dimensional image feature with ResNet-18 before decoding the mesh progressively.
- Topology Modification: The topology modification network estimates reconstruction errors at sampled face points and prunes faces whose estimated errors exceed a threshold.The threshold trades off reconstruction error against excessive triangle removal that can damage the generated surface.
- Boundary Refinement: Boundary refinement moves each boundary vertex within its incident-edge plane and regularizes boundary curves to suppress zigzags.The boundary regularizer enforces smooth and consistent open boundaries after triangle pruning.
- Training Objectives: Training combines Chamfer distance, quadratic error regression, boundary regularization, and geometry constraints for surface smoothness.Chamfer distance measures discrepancy between sampled generated-mesh and ground-truth points, while the error network regresses reconstruction errors.
- Evaluation: Qualitative comparisons evaluate input images, N3MR, Pixel2Mesh, AtlasNet-25, the baseline, the proposed method, and ground truth.The comparison includes methods with deformation-based reconstruction and the authors' framework without topology modification and boundary refinement.
4. Experiments
Experiments evaluate reconstruction accuracy, mesh quality, robustness, and ablations across synthetic and real-image settings. The method consistently outperforms baselines, while progressive shaping and boundary refinement improve reconstruction quality.
- Experimental setup: Experiments use five ShapeNet categories, rendered 224 × 224 RGB images, and 10,000 uniformly sampled surface points as ground truth.Each model has 24 rendered views, and the setup follows prior work for fair comparison.
- Comparisons: The method is compared with N3MR, Pixel2Mesh, AtlasNet-25, and a baseline without topology modification and boundary refinement.Quantitative evaluation uses Chamfer Distance and Earth Mover’s Distance after ICP alignment.
- Comparisons: Qualitatively, the method reconstructs complex topologies while avoiding the severe self-intersections and overlaps observed with AtlasNet-25.N3MR misses fine geometry, deformation-based methods struggle with complex topology, and AtlasNet-25 fails on some complex instances.
- Comparisons: The approach consistently outperforms state-of-the-art methods in CD and EMD across all five categories, especially chairs and tables with complex topologies.Both metrics are computed from 10,000 points sampled from generated meshes.
- Ablation studies: Progressive shaping enables more accurate face pruning, whereas removing it can destroy the generated surface geometry.The threshold study finds τ ∈ [0.05, 0.2] balances the two directional reconstruction distances; experiments use τ = 0.1.
- Ablation studies: Boundary refinement produces cleaner meshes with higher visual quality, and the method achieves similar performance from sphere and unit-square initial meshes.Additional experiments evaluate real images, Poisson surface reconstruction, and 3D shape autoencoding against AtlasNet-25.
5. Conclusion
The paper concludes that its end-to-end framework reconstructs meshes with varied topologies from single-view images and improves results quantitatively and qualitatively. It identifies non-closed meshes as a limitation addressed by post-processing and proposes differentiable mesh stitching as future work.
- Conclusion: The framework progressively combines mesh deformation, topology modification, and boundary refinement to reconstruct meshes with various topologies from single-view images.Topology modification operates through face pruning, while boundary refinement improves boundary conditions.
- Conclusion: Extensive experiments show significant quantitative and qualitative improvements over existing methods.
- Conclusion: Face pruning can produce non-closed meshes, but Poisson surface reconstruction can create closed surfaces from densely sampled point clouds.
- Conclusion: Future work includes designing a differentiable mesh-stitching operation for the open boundaries introduced by face pruning.