Source-linked AI summary
Pixel2Mesh++: Multi-View 3D Mesh Generation via Deformation
Chao Wen, Yinda Zhang, Zhuwen Li, Yanwei Fu
TL;DR
The paper studies 3D mesh generation from a few color images with known camera poses, addressing limited single-view evidence and weak cross-view shape accuracy. It introduces a graph-convolutional deformation network that iteratively refines a coarse mesh by reasoning over nearby vertex hypotheses using pooled multi-view features. Experiments report accurate geometry, alignment beyond input viewpoints, and generalization across categories, view counts, and mesh initializations.
Problem
Single-view shape generators have limited evidence for occluded geometry, while the paper seeks better 3D mesh generation from a few posed color images.
Method
MDN uses a GCN to iteratively deform a coarse Pixel2Mesh mesh, selecting nearby vertex hypotheses from perceptual feature statistics pooled across multiple images.
Results
The model produces accurate 3D shapes aligned to input and arbitrary viewpoints and generalizes across semantic categories, input-view counts, and coarse-mesh quality.
Takeaways & Limitations
Cross-view feature reasoning through deformation improves mesh surface detail and geometry beyond naive multi-view extensions while preserving broad generalization.
Abstract
from arXiv · showhide
We study the problem of shape generation in 3D mesh representation from a few color images with known camera poses. While many previous works learn to hallucinate the shape directly from priors, we resort to further improving the shape quality by leveraging cross-view information with a graph convolutional network. Instead of building a direct mapping function from images to 3D shape, our model learns to predict series of deformations to improve a coarse shape iteratively. Inspired by traditional multiple view geometry methods, our network samples nearby area around the initial mesh's vertex locations and reasons an optimal deformation using perceptual feature statistics built from multiple input images. Extensive experiments show that our model produces accurate 3D shape that are not only visually plausible from the input perspectives, but also well aligned to arbitrary viewpoints. With the help of physically driven architecture, our model also exhibits generalization capability across different semantic categories, number of input images, and quality of mesh initialization.
1. Introduction
The paper addresses the limits of single-view shape generation by using a few views to improve 3D mesh geometry through learned cross-view deformation. Its Multi-View Deformation Network combines graph convolution with multi-view feature reasoning and supports broad generalization.
- Single-view methods often produce rough occluded geometry and generalize poorly across semantic categories because one image provides limited visual evidence.
- A few additional images provide more information about the object’s 3D shape, while traditional multi-view geometry offers analytically defined cross-view correspondences.Traditional methods can nevertheless struggle with large baselines, poorly textured regions, and very limited input views.
- MDN extends Pixel2Mesh’s deformation strategy by using a GCN to refine a coarse mesh with features pooled from multiple images.It proposes vertex-wise hypothesis deformations and relocates vertices toward positions that best explain cross-view features.
- Rather than hallucinating shape solely from priors, MDN reasons from correlations across views by imitating correspondence search.This physically driven design is intended to improve accuracy and generalization.
- The framework is trained end-to-end with differentiable 3D soft argmax and handles varying numbers of input views in one forward pass.The soft argmax replaces the non-differentiable selection required when searching deformation hypotheses.
- The proposed framework produces accurate mesh shapes and generalizes across semantic categories, input-view counts, and coarse-mesh quality.
2. Related Work
Prior work spans volumetric, point-cloud, and mesh-based shape representations, alongside single-view hallucination and multi-view reconstruction methods. Pixel2Mesh++ instead deforms a mesh using features pooled from all input images simultaneously.
- 3D shape generation has used volumes, point clouds, meshes, geometry images, and depth images, with meshes offering compactness and favorable surface properties.
- Single-view methods infer visible structure and often hallucinate invisible parts from learned data priors across several 3D representations.Some related approaches generate shape by deforming an initialization, which is closer to this paper’s strategy.
- Traditional multi-view stereo relies on photo-consistency correspondences but is vulnerable to large baselines, occlusions, and texture-less regions.
- The proposed system uses a 2D CNN and GCN to iteratively refine a Pixel2Mesh coarse shape through multi-view feature pooling.
- Unlike methods that sequentially feed images, this approach deforms the mesh using features pooled from all input images at once.
3. Method
The Multi-View Deformation Network refines a coarse mesh by generating local deformation hypotheses and reasoning over features pooled from multiple posed images. It uses differentiable weighted deformation updates and a re-sampled Chamfer loss to improve mesh quality.
- Coarse-to-fine framework: The framework uses Pixel2Mesh to generate a coarse 2466-vertex shape before MDN adds geometric detail.The initialization can use Pixel2Mesh trained from single or multiple views.
- Deformation hypotheses: MDN generates local deformation hypotheses for each vertex and uses a graph convolutional network to reason an optimal deformation.Each vertex samples 42 nearby positions from a level-1 icosahedron, forming a 43-node local graph with 162 edges.
- Cross-view feature pooling: Known camera poses project vertices and hypotheses into all input images, where VGG perceptual features are bilinearly pooled and summarized with mean, max, and standard deviation.The pooled statistics keep feature dimensionality invariant to the number of input views and encourage learning cross-view correlations.
- Deformation reasoning: A scoring network applies soft argmax to produce the updated vertex location as a weighted sum of all deformation hypotheses.Hypothesis scores are normalized with softmax, and the deformation reasoning unit uses shared weights across local graphs.
- Training objective: The model is trained with Pixel2Mesh losses plus a re-sampled Chamfer distance computed from 4000 uniformly sampled mesh points.Sampling points per triangle proportional to area helps remove artifacts when comparing the predicted mesh with ground truth.
- Optimization: The full model is trained end-to-end after separate initialization training, including fine-tuning VGG perceptual features during the final training stage.The schedule uses 30 epochs for multi-view Pixel2Mesh initialization followed by 20 epochs with the whole model trainable.
4. Experiments
Experiments compare MDN with prior multi-view methods and test its generalization, initialization robustness, ablations, and iterative refinement. MDN improves coarse meshes using multi-view information and maintains performance across categories, input-view counts, and initializations.
- Comparison to Multi-view Shape Generation: MDN significantly improves the MVP2M coarse-shape baseline, while direct multi-view feature integration provides little or no improvement.The comparison uses F-score; MDN further improves the multi-view Pixel2Mesh initialization.
- Comparison to Multi-view Shape Generation: Our model produces better surface details and more accurate geometry than volume-based methods and Pixel2Mesh, including from arbitrary viewpoints.Qualitative comparisons include 3DR2N2, LSM, multi-view Pixel2Mesh, MDN, and ground truth.
- Cross-Category Generalization: MDN generalizes across semantic categories, with only slightly lower performance when the test category is excluded from training and good transfer from one category to others.Categories with more complex geometry transfer better, while very simple categories provide limited improvement.
- Number of Views: MDN performs consistently better with more input views, despite training with three views, and a three-view model performs close to models trained with four or five views.The results indicate that additional view information can be exploited at test time and that training with fewer views remains effective.
- Robustness to Initialization: MDN is robust to noisy, shifted, and externally sourced initialization meshes, removing noise, aligning inputs, and adding geometry details.The tested alternatives include translated or randomly perturbed MVP2M meshes and meshes converted from 3DR2N2.
- Qualitative and Quantitative Ablation: Feature statistics outperform ordinary feature concatenation, while disabling re-sampled Chamfer loss consistently lowers all metrics.Statistics preserve input-order invariance and encode cross-view correlations; re-sampling helps sparse vertices and irregular faces.
- Number of Iteration: Performance improves with more deformation iterations and roughly saturates at three, which is the chosen inference setting.Qualitative results show thin structures and surface details being recovered throughout the iterations.
5. Conclusion
The paper presents a graph-convolutional framework that iteratively deforms coarse meshes using cross-view information from multiple images. It reports accurate surface geometry, state-of-the-art performance, and generalization across categories, input counts, and initialization quality.
- 5. Conclusion: MDN iteratively searches nearby locations around mesh vertices to deform coarse meshes using cross-view information.The framework targets mesh generation from multiple images and improves direct-prediction outputs such as Pixel2Mesh.
- 5. Conclusion: The model achieves state-of-the-art performance with accurate surface details and good generalization across semantic categories, input counts, and coarse-mesh quality.The conclusion contrasts accurate geometry with shapes that are merely plausible from input views.
A. Network Architecture
The architecture samples deformation hypotheses around mesh vertices, pools multi-view perceptual features through camera projection, and uses graph convolutions to select updated coordinates.
- A. Network Architecture: A level-K icosahedron is recursively constructed by sampling edge midpoints from a level-(K-1) icosahedron.
- A. Network Architecture: Level-1 icosahedron vertices and edges define deformation hypotheses and the local GCN graph topology, with coordinates scaled to a predefined radius.
- A. Network Architecture: Six residual graph-convolution layers process current vertices, hypothesis features, and hypothesis coordinates, while softmax weights produce new coordinates by weighted sums.
- A. Network Architecture: Perceptual feature pooling projects each 3D vertex onto image feature maps to retrieve features at the corresponding 2D coordinates.
- A. Network Architecture: Known camera extrinsics transform each world-coordinate vertex into every input camera coordinate before multi-view feature pooling.
B. Baselines Methods
The baselines extend Pixel2Mesh to combine multiple views either by merging independently generated shapes or by pooling multi-view features within one forward pass.
- B. Baselines Methods: P2M-M independently generates a shape from each input image, transforms the shapes into world coordinates, averages their SDFs, and extracts triangular meshes.
- B. Baselines Methods: The second baseline modifies Pixel2Mesh's perceptual pooling layers so one network forward pass accesses features from all input images.
- B. Baselines Methods: The baseline comparison is presented as examples of results from both methods.
C. More Experiment Results
Additional evaluation reports overall best Chamfer distance against prior methods and baselines, with the best category-level performance except for a few simple categories.
- C. More Experiment Results: The method achieves the overall lowest Chamfer distance compared with all previous methods and baselines.
- C. More Experiment Results: Category-level performance is best for most categories, except for a few categories where geometry and texture are too simple to learn cross-view information.
C.2.1 Effect of Re-sample Loss
Further experiments show that re-sampled loss reduces geometric artifacts, iterative deformation recovers detail, and MDN improves varied coarse initializations while generalizing to non-ellipsoid meshes.
- C.2.1 Effect of Re-sample Loss: Re-sampled Chamfer loss penalizes large flying triangles caused by a few flying vertices, leaving the full model free from these artifacts.
- C.2.1 Effect of Re-sample Loss: Thin structures and surface details are recovered progressively throughout later deformation iterations.
- C.2.1 Effect of Re-sample Loss: MDN consistently improves both P2M-M and MVP2M coarse shape initializations, and is slightly better with P2M-M initialization.
- C.2.1 Effect of Re-sample Loss: Table 5 reports Chamfer distance for each semantic category and marks methods that do not require camera extrinsics.
- C.2.1 Effect of Re-sample Loss: MDN generalizes directly to 3DR2N2 chair meshes without finetuning when trained on the chair class.
C.3. More Qualitative Results
Qualitative comparisons across multiple examples show that Pixel2Mesh++ produces accurate shapes aligned with input views while preserving surface details, with fewer self-intersections observed than Pixel2Mesh.
- Across qualitative examples, Pixel2Mesh++ outputs accurate shapes that align well with input views and maintain good surface details.The comparisons include 3D-R2N2, LSM, Pixel2Mesh, Pixel2Mesh++, and ground-truth results.
- Pixel2Mesh++ shows fewer self-intersections than Pixel2Mesh, despite neither method explicitly handling self-intersection.The authors conjecture that cross-view geometric reasoning makes shape generation more stable and robust.
- Figures 14–16 present additional qualitative comparisons using two camera views, four reconstruction methods, and ground truth for each example.