Source-linked AI summary
MeshLRM: Large Reconstruction Model for High-Quality Meshes
Xinyue Wei, Kai Zhang, Sai Bi, Hao Tan, Fujun Luan, Valentin Deschaintre, Kalyan Sunkavalli, Hao Su, Zexiang Xu
TL;DR
MeshLRM addresses sparse-view 3D mesh reconstruction with a feed-forward model that directly integrates differentiable mesh extraction and rendering into a NeRF-based LRM. It simplifies the architecture and uses staged training to improve efficiency, while achieving reported quality and speed improvements and supporting text-to-3D and image-to-3D generation.
Problem
Sparse-view mesh reconstruction needs efficient, accurate feed-forward methods because NeRF-based LRMs do not directly produce high-quality meshes.
Method
MeshLRM fine-tunes a pretrained NeRF LRM with differentiable Marching Cubes and rasterization, using simplified decoders and staged low-to-high-resolution training.
Results
MeshLRM provides quality and speed improvements over previous work and reconstructs high-quality meshes in under one second.
Takeaways & Limitations
The method directly supports high-quality mesh reconstruction and downstream text-to-3D and image-to-3D generation.
Takeaways & Limitations
MeshLRM is not sufficiently robust for complex materials and requires input camera poses.
Abstract
from arXiv · showhide
We propose MeshLRM, a novel LRM-based approach that can reconstruct a high-quality mesh from merely four input images in less than one second. Different from previous large reconstruction models (LRMs) that focus on NeRF-based reconstruction, MeshLRM incorporates differentiable mesh extraction and rendering within the LRM framework. This allows for end-to-end mesh reconstruction by fine-tuning a pre-trained NeRF LRM with mesh rendering. Moreover, we improve the LRM architecture by simplifying several complex designs in previous LRMs. MeshLRM's NeRF initialization is sequentially trained with low- and high-resolution images; this new LRM training strategy enables significantly faster convergence and thereby leads to better quality with less compute. Our approach achieves state-of-the-art mesh reconstruction from sparse-view inputs and also allows for many downstream applications, including text-to-3D and single-image-to-3D generation. Project page: https://sarahweiii.github.io/meshlrm/
1 INTRODUCTION
MeshLRM targets fast, accurate sparse-view mesh reconstruction by directly integrating differentiable mesh extraction and rendering into a NeRF-based large reconstruction model. Its training and architectural changes stabilize mesh optimization while improving reconstruction quality and speed.
- Motivation: MeshLRM directly reconstructs high-fidelity meshes from sparse-view images using a transformer-based large reconstruction model.The model is designed for efficient feed-forward inference without post-optimization.
- Approach: Differentiable Marching Cubes and rasterization enable end-to-end mesh training from the NeRF density field.Mesh rendering losses optimize density fields to produce surfaces better suited to mesh extraction.
- Training: Ray opacity loss stabilizes DiffMC fine-tuning by suppressing density in empty space and reducing floater artifacts.Sparse DiffMC gradients otherwise affect only surface voxels, creating poor local minima.
- Architecture: The simplified LRM removes complex components such as pretrained DINO image encoding and uses small density and color decoders.These changes support efficient training and mesh reconstruction.
- Evaluation: MeshLRM benchmarks sparse-view reconstruction and generation, producing high-quality outputs in under one second.The evaluation covers synthetic and real reconstruction datasets and generation with other multiview methods.
2 RELATED WORK
Prior mesh reconstruction relies on complex, dense-image pipelines, while neural and generalizable methods trade representation or view-coverage constraints for efficiency. MeshLRM instead combines LRM-style sparse-view processing with differentiable mesh generation.
- Mesh Reconstruction: Classical photogrammetry combines structure from motion, multiview stereo, and surface extraction but requires dense images and long processing times.It can also face calibration and appearance challenges.
- Neural Reconstruction: Neural reconstruction commonly uses volumetric radiance fields with per-scene optimization, which can later be converted into meshes.This motivates methods that produce meshes more directly.
- Neural Reconstruction: Generalizable neural methods enable fast few-shot reconstruction but often require dense overlapping views and handle only small baselines.These constraints limit complete-object reconstruction from sparse views.
- Large Reconstruction Models: The architecture processes patchified images with triplane positional tokens, then decodes triplane outputs into density and color.The same representation supports volumetric rendering and DiffMC fine-tuning.
- 3D Generation: MeshLRM incorporates differentiable marching cubes and rasterization into an LRM rather than sequentially predicting triangles.This re-targets a NeRF LRM for direct mesh generation.
3 METHOD
MeshLRM uses a simplified transformer to predict triplane NeRF features, initializes them with staged volumetric training, and then fine-tunes mesh extraction and rendering. Ray opacity regularization addresses sparse surface gradients and floaters during mesh training.
- Training Framework: The two-stage method first trains a sparse-view NeRF with volume rendering, then refines it using differentiable marching cubes and surface-rendering loss.This initialization-to-mesh pipeline enables feed-forward mesh reconstruction.
- Model Architecture: The transformer concatenates posed-image tokens with learnable triplane tokens and regresses triplane features for density and color decoding.Self-attention enables information exchange across image and triplane tokens.
- Model Architecture: The image tokenizer converts RGB pixels and Plücker ray coordinates into 9-channel patches without pretrained DINO encoding.The simpler tokenizer connects raw pixel information directly to 3D processing.
- Model Architecture: Tiny density and color MLPs reduce rendering cost, yielding a 50% training speed-up without compromising quality.Separate decoders also improve DiffMC fine-tuning stability.
- Stage 1: Efficient Training for Volume Rendering: Low-resolution pretraining followed by high-resolution fine-tuning improves volume-rendering training efficiency through shorter sequences and fewer ray samples.The schedule uses 256×256 images before 512×512 fine-tuning.
- Stage 2: Stable Training for Surface Rendering: Mesh extraction decodes a 256^3 density grid with DiffMC and renders the resulting surface using a differentiable rasterizer.The rasterizer provides surface locations for neural color rendering from triplane features.
- Stage 2: Stable Training for Surface Rendering: Mesh-rendering loss alone causes floaters because its density gradients are sparse and restricted to surface points.Unlike volume rendering, it provides little supervision in empty scene space.
- Stage 2: Stable Training for Surface Rendering: Ray opacity loss encourages near-zero density in empty space along each pixel ray to stabilize DiffMC fine-tuning.It weights density supervision according to distance from sampled points to the surface.
4 EXPERIMENTS
MeshLRM’s experiments show that staged training and differentiable surface fine-tuning improve mesh quality while retaining fast reconstruction. Across ablations and feed-forward comparisons, simplified architecture choices provide comparable or better quality with less computation.
- Volume rendering (Stage 1) training strategies: 2.6dB higher PSNR results from low-to-high-resolution training than from 512-resolution training from scratch under the same compute budget.Low-resolution pretraining accelerates convergence and improves the subsequent high-resolution stage, which also benefits mesh reconstruction.
- Effectiveness of surface fine-tuning (Stage 2): 2.5dB higher PSNR and 0.58 lower CD show that DiffMC-based surface fine-tuning substantially improves directly extracted meshes.The resulting mesh quality is comparable to volume rendering, with slightly lower PSNR but improved SSIM and LPIPS.
- Surface fine-tuning losses: The ray opacity loss prevents severe floaters, while the normal loss improves geometry and robustness to inconsistent input views.Normal-loss improvements are especially useful for generated images and for meshes used in physically based rendering.
- Tiny MLPs: Tiny MLP decoders match large-MLP performance while reducing training time from 3.6s to 2.7s per step, a 25% speedup.The heavy shared MLP also fails to converge well during Stage 2 DiffMC training, unlike the tiny separate MLPs.
- Comparisons with feed-forward methods: 4.2dB PSNR and 0.05 SSIM improvements, with CD lowered by 0.72, show MeshLRM’s final meshes outperform In3D-LRM-based mesh reconstruction.MeshLRM also improves PSNR by 1.4dB and SSIM by 0.03 over In3D-LRM volume rendering, while rendering substantially faster and using less than half the training compute.
- Comparisons with per-scene optimization methods: 0.8 seconds enables MeshLRM to achieve better or comparable performance to per-scene optimization methods that require minutes or hours per scene.The comparison includes FreeNeRF and ZeroRF for sparse-view NeRF reconstruction.
5 APPLICATIONS
MeshLRM extends sparse-view mesh reconstruction to single-image and text-to-3D applications by first generating or converting multi-view images. It outperforms several baselines and produces better mesh quality with fewer artifacts in text-driven generation.
- Image-to-3D Generation: The image-to-3D pipeline converts a single input image into multi-view images before reconstruction with MeshLRM.The conversion uses Zero123++ for InstantMesh, MeshFormer, and MeshLRM, while other baselines directly process the single image.
- Image-to-3D Generation: MeshLRM outperforms TripoSR, LGM, and InstantMesh across all four metrics on both evaluated datasets for image-to-3D generation.Compared with MeshFormer, it has slightly lower PSNR but better SSIM, LPIPS, and CD.
- Image-to-3D Generation: Generated meshes exhibit sharper textures and more accurate geometric details in additional visualizations.These visual results complement the quantitative image-to-3D comparison.
- Text-to-3D Generation: MeshLRM produces better mesh quality and fewer rendering artifacts than the original Instant3D pipeline for text-to-3D generation.Instant3D’s multi-view diffusion model supplies four views from text before MeshLRM reconstructs the mesh.
6 CONCLUSION
MeshLRM combines differentiable mesh extraction and rasterization with architectural and training improvements to produce high-quality meshes efficiently. The approach also supports downstream text-to-3D and image-to-3D generation.
- MeshLRM integrates DiffMC and differentiable rasterization to fine-tune a pre-trained NeRF-based LRM directly for mesh reconstruction.The mesh-rendering pipeline enables end-to-end optimization with mesh rendering loss.
- A simple architecture with tiny shared MLPs and simplified image tokenization supports efficient NeRF and mesh-targeted training.
- Low-to-high-resolution training significantly accelerates NeRF-based model training while supporting quality and speed improvements.
- MeshLRM directly produces high-quality meshes and can be applied to text-to-3D and image-to-3D generation.
B COMPARISON WITH PER-SCENE OPTIMIZATION METHODS
Against per-scene optimization methods, MeshLRM provides feed-forward sparse-view reconstruction with substantially faster execution while maintaining or improving reconstruction quality. It also supports image-to-3D and text-to-3D pipelines.
- MeshLRM is evaluated with four input views and compared against FreeNeRF and ZeroRF on NeRF-Synthetic and OpenIllumination.
- On OpenIllumination, MeshLRM outperforms FreeNeRF and ZeroRF by a large margin on all three reported metrics.The real captured-image dataset tests generalization beyond the rendered training images.
- 1000× to 10000× speed up over optimization-based NeRF methods is achieved for reconstruction and rendering.
- The feed-forward model can be paired with Zero123++ or Instant3D for single-image-to-3D and text-to-3D generation.
D INFERENCE ON VARIOUS NUMBERS OF VIEWS
MeshLRM remains effective across varying numbers of input views and changes in input resolution. Its downstream text-to-3D results show more accurate, smoother geometry with sharp textures.
- Text-to-3D results produce significantly more accurate and smoother geometry with sharp textures.
- Performance improves consistently as input views increase from 2 to 8, while remaining strong with only two views.
- At 1024×1024 input resolution, MeshLRM produces reasonable outputs while In3D-LRM shows significant geometry distortion.Both models were trained at 512×512 resolution.
F EFFECTIVENESS OF SURFACE FINE-TUNING (STAGE 2).
Surface fine-tuning improves mesh quality over directly applying Marching Cubes to the first-stage NeRF output. The unfine-tuned baseline exhibits severe surface artifacts.
- The MeshLRM final model avoids the severe artifacts, non-smooth surfaces, and holes produced by direct Marching Cubes on the first-stage NeRF.
G ABO AND OPENILLUMINATION DATASETS
On ABO and OpenIllumination, MeshLRM is evaluated against In3D-LRM under challenging appearance and dataset settings, with strong mesh-rendering results.
- Naive marching-cubes extraction from the volume-rendering model substantially reduces reconstructed mesh quality without mesh-refinement fine-tuning.
- 3.92dB in PSNR and 0.48 in SSIM separate MeshLRM mesh rendering from In3D-LRM + MC on ABO.ABO contains complex glossy materials, challenging both methods' Lambertian appearance assumption.
- MeshLRM's NeRF and final mesh models both outperform the corresponding In3D-LRM variants on ABO.
- The OpenIllumination comparison evaluates all models on 100 objects using object masks and tightly bounded square crops.This setting differs from the main-paper evaluation of eight objects with combined masks.
H ADDITIONAL IMPLEMENTATION DETAILS
The implementation uses a transformer-large architecture, AdamW optimization, visible-surface evaluation sampling, and a controlled multiview rendering setup.
- The transformer uses 24 layers, width 1024, 16 attention heads, head dimension 64, and MLP dimension 4096.
- AdamW training uses β2 = 0.95, weight decay 0.05 excluding biases and normalization layers, cosine decay, and warm-up.The stage-1 learning rate is 4e−4, while stage 2 uses 1e−5.
- Chamfer-distance evaluation samples 100,000 points from visible surfaces because scanned mesh interiors are unreliable.
- Testing renders 24 camera poses across eight azimuths and three elevations at 320×320 resolution within a normalized [−1, 1]^3 bounding box.
J LIMITATIONS
MeshLRM is less robust to complex materials and requires input camera poses, creating practical boundaries for material-rich scenes and real captures.
- Complex materials can cause baked shadows and white colors in specular regions because the model assumes Lambertian appearance without inverse rendering.
- Real sparse-view reconstruction remains challenging because MeshLRM requires camera poses, whose calibration can be difficult for real captures.The authors suggest combining the approach with pose-estimation techniques in future work.