Source-linked AI summary
Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images
Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, Yu-Gang Jiang
TL;DR
Single-image reconstruction methods often produce volumes or point clouds that lose surface detail and are difficult to convert into meshes. Pixel2Mesh instead uses graph convolutions and perceptual image features to progressively deform an ellipsoid in a coarse-to-fine process, and reports significantly better results than prior volume- and point-cloud-based methods.
Problem
Volume and point-cloud outputs lose important surface details and are non-trivial to reconstruct as lightweight, detail-capable mesh models.
Method
An end-to-end graph-convolutional network progressively deforms an ellipsoid using perceptual image features, graph unpooling, and mesh-specific losses.
Results
The authors report significantly better results than previous state-of-the-art methods using 3D volumes or point clouds.
Takeaways & Limitations
The paper presents mesh representation as a promising direction for direct 3D reconstruction from single images and identifies components for follow-up work.
Takeaways & Limitations
The method only produces meshes with the same topology as the initial mesh.
Abstract
from arXiv · showhide
We propose an end-to-end deep learning architecture that produces a 3D shape in triangular mesh from a single color image. Limited by the nature of deep neural network, previous methods usually represent a 3D shape in volume or point cloud, and it is non-trivial to convert them to the more ready-to-use mesh model. Unlike the existing methods, our network represents 3D mesh in a graph-based convolutional neural network and produces correct geometry by progressively deforming an ellipsoid, leveraging perceptual features extracted from the input image. We adopt a coarse-to-fine strategy to make the whole deformation procedure stable, and define various of mesh related losses to capture properties of different levels to guarantee visually appealing and physically accurate 3D geometry. Extensive experiments show that our method not only qualitatively produces mesh model with better details, but also achieves higher 3D shape estimation accuracy compared to the state-of-the-art.
1 Introduction
Single-image 3D reconstruction is difficult, and volume or point-cloud outputs lose surface detail while making mesh recovery non-trivial. Pixel2Mesh addresses this by progressively deforming a mesh with graph-based convolutions, image features, and mesh-specific regularization.
- Motivation: Single-image 3D reconstruction is challenging, while volume and point-cloud representations lose surface details and are difficult to convert into usable meshes.Meshes are lightweight, model shape details, and are easy to deform for animation.
- Approach: The model deforms a mesh from a mean shape toward target geometry rather than directly synthesizing a structured graph.Progressive residual deformations allow gradual refinement and can incorporate prior knowledge through the initial mesh.
- Approach: A graph-based fully convolutional network represents mesh vertices and edges while integrating 3D geometry with perceptual features from the input image.This addresses the challenge of combining irregular 3D graphs with regular-grid image features.
- Approach: Graph unpooling starts with fewer vertices and increases resolution during propagation, helping the network place representative vertices before adding local details.Shortcut-enhanced GCN layers provide larger receptive fields and more movement steps.
- Approach: Surface normal, edge, and Laplacian losses regularize smoothness, vertex distribution, and face-intersection behavior in the generated mesh.These higher-order losses rely on known graph connectivity.
- Contributions: The paper contributes an end-to-end RGB-image-to-mesh architecture with perceptual feature projection and coarse-to-fine 3D geometry prediction.The stated design is intended to make mesh prediction more reliable and easier to learn.
2 Related Work
Prior reconstruction methods use multiple views, shape retrieval, category-specific deformable models, voxels, point clouds, or geometry images. These approaches face restrictions involving view coverage, category generality, resolution, surface connectivity, or repository requirements.
- Multi-view and learning-based methods: Multi-view reconstruction methods are limited by the coverage available from multiple views and by the object's appearance.The passage contrasts these restrictions with learning-based single- or few-image approaches.
- Learning-based methods: Learning-based methods rely on shape priors from data, with some approaches retrieving and deforming components or category-specific deformable models.Shape retrieval is described as ill-posed, while category-specific reconstruction can lack details and popular-category coverage.
- Voxel and point-cloud methods: Voxel methods are constrained by GPU memory and typically produce low-resolution outputs, limiting their usefulness as shape representations in game and movie industries.Octrees improve resolution under limited memory, but remain voxel-based.
- Voxel and point-cloud methods: Point clouds avoid voxel drawbacks but lack local point connectivity, leaving point positions unconstrained and making direct mesh recovery difficult.The passage states that generated point clouds are usually not close to a surface.
- Graph-based methods: Graph and surface-manifold networks are natural representations for meshed objects, but the paper states they had not been adopted for single-image 3D reconstruction.The authors identify their base network as a graph neural network.
3 Method
The method represents a mesh as an irregular graph whose vertices carry features, then applies graph convolutions that exchange information across neighboring vertices. Its cascaded deformation network increases resolution while progressively refining vertex locations.
- Graph representation: A mesh is represented as M = (V, E, F), with vertices, edges, and faces defining the 3D object's shape.Feature vectors are attached to mesh vertices.
- Graph convolution: Graph convolution operates on an irregular graph by updating each vertex from its own features and those of neighboring vertices.Shared edge parameters allow the operation to handle vertices with different degrees.
- Graph convolution: Vertex features concatenate 3D coordinates, learned 3D shape encoding, and image features when available.Convolutional updates change the features and are equivalent to applying a deformation.
- Cascaded deformation: The cascaded network uses three mesh deformation blocks, each increasing mesh resolution and estimating vertex locations for subsequent image-feature extraction.The blocks form a coarse-to-fine deformation process.
3.2 System Overview
The system combines a 2D image-feature network with a cascaded graph-based mesh-deformation network. It progressively deforms an ellipsoid and adds vertices so mesh detail increases from coarse structure to finer geometry.
- System architecture: A 2D CNN extracts perceptual features from the input image for the cascaded graph-based mesh-deformation network.The framework is end-to-end and outputs a mesh in camera coordinates.
- System architecture: Three deformation blocks and two graph-unpooling layers progressively update vertex locations while increasing vertex count and preserving triangular mesh topology.Starting from fewer vertices supports coarse-to-fine detail handling.
- Initialization: The model begins from an average-size ellipsoid placed at a common camera-coordinate location and deforms it toward the desired 3D model.The initial ellipsoid contains 156 vertices and does not require prior 3D-shape knowledge.
3.4 Mesh deformation block
The mesh deformation block projects current 3D vertices into the image to pool perceptual features, concatenates them with vertex features, and uses G-ResNet to predict updated coordinates and shape features.
- 3D vertices are projected onto the image plane, where perceptual features are pooled from nearby pixels using bilinear interpolation.
- The pooled image features are concatenated with each vertex’s 3D shape feature before graph-based residual processing.
- G-ResNet outputs new coordinates and 3D shape features for every vertex in the deformation block.
- The image feature network uses VGG-16 features from conv3_3, conv4_3, and conv5_3, producing a 1280-dimensional perceptual feature.
G-ResNet
G-ResNet addresses limited graph receptive fields with deep residual graph convolutions, while edge-based unpooling increases mesh resolution without causing imbalanced vertex degrees.
- Each graph convolution exchanges information only between neighboring vertices, creating a limited receptive field for mesh deformation.
- G-ResNet uses 14 graph residual convolutional layers with 128 channels to improve information exchange across the mesh.
- Edge-based unpooling adds vertices while preserving regular vertex degrees, unlike face-based unpooling.
- The graph network predicts updated vertex coordinates through an additional graph convolutional branch.
3.6 Losses
The method combines multiple mesh losses to constrain vertex locations and geometric properties, because vertex regression alone does not guarantee visually appealing meshes.
- The model uses Chamfer, normal, Laplacian, and edge-length losses to constrain different properties of the predicted mesh and deformation.
- Chamfer loss constrains the locations of predicted mesh vertices relative to the ground-truth shape.
- Regressing vertices near their correct positions is insufficient by itself to produce a visually appealing 3D mesh.
Normal loss
The normal loss uses ground-truth surface normals to constrain local edges and promote consistency between the predicted mesh’s local tangent plane and the observed surface.
- The normal loss penalizes the inner product between each local edge and the ground-truth normal at the closest corresponding vertex.
- Its geometric effect is to encourage neighboring edges to be perpendicular to the observed surface normal.
- Although the loss is generally nonzero away from planar surfaces, it is equivalent to matching the normal of a locally fitted tangent plane.
Regularization
The method uses multiple regularizers to prevent unstable mesh deformation, preserve local detail, and discourage geometrically implausible edges. Their weighted combination balances these objectives during training.
- Large deformations can create flying vertices by favoring local consistency when the estimate is far from ground truth.This instability can trap optimization in a local minimum even with Chamfer and Normal losses.
- Laplacian regularization limits vertex movement and potentially avoids mesh self-intersection.It also encourages neighboring vertices to undergo similar movements.
- The Laplacian term acts as surface smoothing for the initial ellipsoid and later restricts deformation so subsequent blocks add fine-grained details.
- Edge length regularization penalizes flying vertices because they usually produce long edges.
- The overall loss combines Chamfer, Normal, Laplacian, and edge-length terms with fixed weights λ1 = 1.6e −4, λ2 = 0.3, and λ3 = 0.1.
4 Experiment
The experiments evaluate reconstruction accuracy, component contributions, efficiency, and real-world generalization. Pixel2Mesh outperforms competing methods broadly, while qualitative analyses show why mesh-specific design choices matter.
- The ShapeNet dataset contains 50k rendered models across 13 categories, using Choy et al.’s training/testing split for fair comparison.
- The evaluation reports F-score, Chamfer Distance, and Earth Mover’s Distance using sampled points; higher F-score is better, while lower CD and EMD are better.
- Comparison to state of the art: Pixel2Mesh outperforms other methods in every category except watercraft on F-score, with at least 10% improvement under smaller thresholds.
- Comparison to state of the art: Pixel2Mesh achieves the best mean CD and EMD in most categories, while volume and point-cloud conversions lose details or make mesh recovery difficult.
- Ablation study: Ablations show that removing shortcut connections causes a large performance gap, while removing normal, Laplacian, or edge-length losses damages mesh quality differently.
- Number of deformation blocks: Three deformation blocks provide the chosen balance between accuracy and efficiency: additional blocks help but saturate, and four blocks slow the approach substantially.
5 Conclusion
The approach directly reconstructs triangular meshes from single images by progressively refining them with graph-based processing and image guidance. It reports better results than prior volume- and point-cloud-based methods, while remaining limited to the initial mesh topology.
- Conclusion: The network progressively refines triangular meshes using perceptual image features and cascaded graph convolutions.Meshes are trained end-to-end with chamfer and normal losses.
- Conclusion: Figure 8 compares qualitative outputs from volume, point-cloud, N3MR, and the proposed method against ground truth.
- Conclusion: The method achieves significantly better results than previous state-of-the-art approaches using volume or point-cloud representations.
- Future work: The method produces meshes only with the same topology as the initial mesh.Future work targets more general cases, including scene-level and multi-view reconstruction.