Source-linked AI summary
LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation
Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, Ziwei Liu
TL;DR
Feed-forward 3D models are fast but constrained by low-resolution training, limiting detailed content creation. LGM fuses multi-view Gaussian features with an asymmetric U-Net using views generated from text or single images, achieving high-resolution generation in approximately 5 seconds while remaining dependent on input-view quality.
Problem
Feed-forward 3D models generate objects quickly but their low-resolution training limits detailed textures and complicated geometry.
Method
LGM predicts and fuses 3D Gaussians from multi-view images generated from text prompts or single-view images, using an asymmetric U-Net backbone.
Results
Approximately 5 seconds and 512 training resolution are achieved for high-resolution 3D Gaussian generation in image-to-3D and text-to-3D tasks.
Takeaways & Limitations
The framework combines high-resolution 3D generation with feed-forward efficiency and supports optional conversion of generated Gaussians into meshes.
Takeaways & Limitations
Generation quality depends strongly on four input views, while current multi-view diffusion models can introduce 3D inconsistencies and are restricted to 256 × 256 resolution.
Abstract
from arXiv · showhide
3D content creation has achieved significant progress in terms of both quality and speed. Although current feed-forward models can produce 3D objects in seconds, their resolution is constrained by the intensive computation required during training. In this paper, we introduce Large Multi-View Gaussian Model (LGM), a novel framework designed to generate high-resolution 3D models from text prompts or single-view images. Our key insights are two-fold: 1) 3D Representation: We propose multi-view Gaussian features as an efficient yet powerful representation, which can then be fused together for differentiable rendering. 2) 3D Backbone: We present an asymmetric U-Net as a high-throughput backbone operating on multi-view images, which can be produced from text or single-view image input by leveraging multi-view diffusion models. Extensive experiments demonstrate the high fidelity and efficiency of our approach. Notably, we maintain the fast speed to generate 3D objects within 5 seconds while boosting the training resolution to 512, thereby achieving high-resolution 3D content generation.
1 Introduction
LGM targets the speed–resolution gap in feed-forward 3D generation by combining multi-view Gaussian splatting with an asymmetric U-Net. It generates detailed 3D assets from text or single-view inputs in approximately 5 seconds at 512 training resolution.
- Motivation: Existing optimization-based methods can be detailed but slow, while feed-forward reconstruction models remain limited by low-resolution training and produce blurry textures or complicated geometry.The paper identifies inefficient 3D representations and low-resolution training as key bottlenecks.
- Method: LGM predicts Gaussian features from multi-view images and fuses them into 3D Gaussians for differentiable rendering.Its representation is designed to be expressive and efficient for high-resolution generation.
- Method: An asymmetric U-Net provides a high-throughput backbone for efficient end-to-end training at higher image resolutions.The architecture avoids relying on triplane-based volume rendering or transformers.
- Applications: LGM supports image-to-3D and text-to-3D generation by using multi-view diffusion models to produce the reconstruction inputs.The diffusion-generated views are processed by the Gaussian fusion network.
2 Related Work
Prior work separates into optimization-based methods that seek detail but are slow and feed-forward methods that are fast but struggle with high-resolution detail. Gaussian splatting offers an alternative representation for efficient, expressive 3D generation.
- Efficient 3D Generation: Feed-forward 3D-native methods generate assets within seconds after training, but high-resolution NeRF rendering has substantial memory demands.Gaussian splatting is presented as expressive and efficient to render, although optimization-based use requires careful initialization and densification.
3 Large Multi-View Gaussian Model
LGM reconstructs 3D Gaussians from four camera-conditioned views using an asymmetric U-Net, then renders them for image-level supervision. The framework also includes robustness training and a Gaussian-to-mesh conversion pipeline.
- Pipeline: At inference, text- or image-conditioned multi-view diffusion models generate four sparse views that serve as inputs to Gaussian reconstruction.MVDream is used for text input, while ImageDream supports image and optionally text input.
- Asymmetric U-Net: The model takes four images with camera ray embeddings and predicts four feature maps that are interpreted and fused into final 3D Gaussians.Cross-view self-attention shares information across the input views.
- Asymmetric U-Net: The input feature for each pixel combines its RGB value, ray origin, and ray direction: f_i = {c_i, o_i × d_i, d_i}.The concatenated representation forms a 9-channel input feature map.
- Asymmetric U-Net: An asymmetric output resolution limits the number of Gaussians while allowing higher-resolution input images.Each output feature-map pixel represents a 3D Gaussian, whose 14 channels encode its attributes.
- Robust Training: Data augmentation addresses the domain gap between rendered training views and diffusion-synthesized inference views.The goal is more robust training against 3D inconsistency and inaccurate camera poses.
- Training Objective: Differentiable rendering supervises RGB and alpha images from input and novel views using MSE, LPIPS, and alpha MSE losses.Eight views are rendered at each training step: four input views and four novel views.
- Mesh Extraction: The mesh pipeline trains an efficient NeRF from Gaussian renderings, extracts geometry with Marching Cubes, refines it, and bakes appearance textures.The conversion from Gaussians to NeRF to mesh takes about 1 minute with optimized implementation.
4 Experiments
Experiments evaluate LGM’s implementation, qualitative quality, diversity, ablations, and limitations across image-to-3D and text-to-3D settings. The results show benefits from high-resolution training, data augmentation, and multi-view inputs, while quality remains dependent on synthesized views.
- Implementation Details: The inference pipeline uses two multi-view diffusion models and about 10 GB of GPU memory.Camera elevation is fixed at 0 degrees, with azimuths of 0, 90, 180, and 270 degrees for four generated views.
- Qualitative Comparisons: LGM preserves input content and produces higher-quality image-to-3D Gaussians, with improved detail in unseen views over LRM.The multi-view setting mitigates blurry back views and flat geometry; generated Gaussians can usually be converted into smooth textured meshes with minimal quality loss.
- Qualitative Comparisons: LGM generates more realistic text-to-3D objects with enhanced quality and efficiency, and avoids multi-face problems.The paper attributes freedom from multi-face problems to the multi-view diffusion models.
- Qualitative Comparisons: Different random seeds yield diverse feasible 3D models from the same ambiguous text prompt or single-view image.The paper attributes this diversity to the multi-view diffusion model.
- Ablation Study: Data augmentation improves geometric robustness: without it, inference-domain gaps produce more floaters and worse geometry.The augmented model better corrects 3D inconsistency and inaccurate camera poses in generated multi-view images.
- Ablation Study: Training at 512×512 captures better details and higher-resolution Gaussians than the reduced 256×256 setting.The reduced model uses 16,384 Gaussians and still reconstructs successfully, but its details are worse than those of the large-resolution model.
- Limitations: The method’s quality depends strongly on four input views, whose 3D inconsistency, limited 256×256 resolution, and elevation-angle failures constrain generation.The paper expects better multi-view diffusion models to mitigate these limitations.
5 Conclusion
The paper presents a large multi-view Gaussian model for high-resolution 3D content generation, combining Gaussian splatting, a U-Net, data augmentation, and mesh extraction to target high resolution and efficiency.
- The model uses Gaussian splatting and a U-Net to address high memory requirements and low-resolution training in prior methods.
- The framework explores data augmentation for robustness and introduces a mesh extraction algorithm for generated 3D Gaussians.
- The approach achieves high-resolution and high-efficiency 3D object generation across various contexts.
A More Implementation Details
The implementation uses filtered Objaverse data and a fixed set of camera views arranged along a spiral path on a sphere.
- The Objaverse dataset is filtered using a specified list of words to remove unwanted content categories.
- The 100 camera views follow a spiral path on the sphere surface, with the camera radius fixed at 1.5.
B More Results
Additional results compare the proposed meshing algorithm with DreamGaussian and visualize failure cases linked to imperfect multi-view inputs, limited resolution, and challenging structures or viewpoints.
- Different Meshing Method: The proposed meshing algorithm generates smoother surfaces than DreamGaussian, benefiting subsequent tasks such as relighting.
- Different Meshing Method: The meshing method operates independently of the underlying 3D Gaussians because it relies solely on rendered images.
- Limitations: Failure cases primarily stem from flawed multi-view images produced during the initial generation step.
- Limitations: Multi-view image resolution is limited to 256 × 256, which can reduce input-image quality and cause inaccuracies in slender structures such as chairs.
- Limitations: ImageDream can struggle with images at significant elevation angles, sometimes producing dark appearances.