Source-linked AI summary
Sparc3D: Sparse Representation and Construction for High-Resolution 3D Shapes Modeling
Zhihao Li, Yufei Wang, Heliang Zheng, Yihao Luo, Bihan Wen
TL;DR
High-fidelity 3D synthesis is limited by unstructured meshes, dense-grid scaling, representation losses, and VAE modality mismatches. Sparc3D combines high-resolution sparse remeshing through Sparcubes with modality-consistent sparse-convolutional encoding through Sparconv-VAE. It reports state-of-the-art reconstruction fidelity, lower training cost, preserved detail, and integration with latent diffusion for higher-resolution generation.
Problem
Existing 3D generation pipelines face limited resolution, detail loss, topology problems, and modality mismatch when using VAE compression followed by latent diffusion.
Method
Sparc3D combines Sparcubes for high-resolution watertight remeshing with a sparse-convolutional, self-pruning Sparconv-VAE for modality-consistent reconstruction.
Results
Sparc3D achieves state-of-the-art reconstruction fidelity, reduces training cost, preserves fine details and components, and integrates with latent diffusion for higher-resolution 3D generation.
Takeaways & Limitations
The framework provides a unified route to scalable high-resolution 3D generation while retaining detailed geometry and supporting challenging mesh inputs.
Takeaways & Limitations
Sparcubes does not retain original textures and discards hidden internal elements in fully closed meshes with internal structures.
Abstract
from arXiv · showhide
High-fidelity 3D object synthesis remains significantly more challenging than 2D image generation due to the unstructured nature of mesh data and the cubic complexity of dense volumetric grids. Existing two-stage pipelines-compressing meshes with a VAE (using either 2D or 3D supervision), followed by latent diffusion sampling-often suffer from severe detail loss caused by inefficient representations and modality mismatches introduced in VAE. We introduce Sparc3D, a unified framework that combines a sparse deformable marching cubes representation Sparcubes with a novel encoder Sparconv-VAE. Sparcubes converts raw meshes into high-resolution ($1024^3$) surfaces with arbitrary topology by scattering signed distance and deformation fields onto a sparse cube, allowing differentiable optimization. Sparconv-VAE is the first modality-consistent variational autoencoder built entirely upon sparse convolutional networks, enabling efficient and near-lossless 3D reconstruction suitable for high-resolution generative modeling through latent diffusion. Sparc3D achieves state-of-the-art reconstruction fidelity on challenging inputs, including open surfaces, disconnected components, and intricate geometry. It preserves fine-grained shape details, reduces training and inference cost, and integrates naturally with latent diffusion models for scalable, high-resolution 3D generation.
1 Introduction
Sparc3D addresses resolution loss, missing geometry, and modality mismatch in existing 3D generation pipelines with Sparcubes and Sparconv-VAE. The framework preserves high-resolution geometry while supporting efficient latent-diffusion generation.
- High-fidelity 3D synthesis is difficult because mesh data is unstructured and dense volumetric representations scale cubically.
- Sparcubes converts raw meshes into watertight 1024^3 surfaces while retaining fine details and small components.It uses sparse activated voxels, flood-filled signed labels, and optimized grid-vertex deformations; conversion takes under 30 seconds with a threefold speedup over prior methods.
- Sparconv-VAE uses sparse convolutional encoding and self-pruning decoding to achieve modality-consistent, near-lossless reconstruction without global attention.
- Sparc3D achieves state-of-the-art reconstruction fidelity, reduces training cost, and integrates with latent diffusion to enhance generated-object resolution.
2 Related Work
Related work represents 3D shapes through meshes, point clouds, implicit isosurfaces, global latent vectors, or sparse voxels. These approaches trade irregular-structure handling, topological fidelity, resolution, and computational efficiency in different ways.
- Triangle meshes preserve surface detail and arbitrary topology but have irregular neighborhoods, varying vertex counts, and no canonical ordering.Autoregressive methods address these issues by generating geometry and connectivity jointly, but face limited context length and slow generation.
- Isosurface: Isosurface methods polygonize signed-distance fields, while Dual Marching Cubes improves watertightness and FlexiCubes adapts voxel sizes to local geometry.
- 2D-supervised methods may produce open surfaces or incorrect interiors because they lack volumetric constraints, whereas 3D supervision provides explicit SDFs.
- VecSet-based VAEs: VecSet-based VAEs encode local surface features into global latent-vector sets and extend them with transformers, larger models, or mixture-of-experts modules.
- Sparse voxel–based VAEs: Sparse voxel-based VAEs preserve spatial structure through voxel-aligned features and support joint modeling of geometry and texture.XCube, TRELLIS, and TripoSF progressively improve detail preservation, multimodal representation, and reconstruction resolution.
3 Method
Sparcubes reconstructs watertight surfaces from arbitrary meshes using sparse distance fields, flood-fill signs, and deformable refinement. Sparconv-VAE then compresses and reconstructs the same sparse representation with modality-consistent sparse convolutions.
- Preliminaries: Distance fields represent surface boundaries implicitly, with UDF magnitude and SDF sign distinguishing interior from exterior.For non-watertight or non-manifold meshes, determining the sign is non-trivial.
- Sparcubes: Sparcubes samples only a sparse narrow band around the input surface, reducing memory usage while retaining cube-based surface extraction.Its sparse cube grid stores sampled vertices, active eight-vertex cubes, and field values.
- Sparcubes: Sparcubes assigns coarse signs by flood fill, then optimizes sparse cube-vertex positions instead of globally refining inaccurate SDF values.Gradient-based deformation addresses alignment and ambiguous connectivity, including thin shells and open surfaces.
- Sparcubes: Differentiable rendering optionally refines the reconstructed mesh using observed depth, silhouettes, or normals when multi-view signals are available.Rendering is restricted to visible voxels to reduce computational cost.
- Sparconv-VAE: Sparconv-VAE directly compresses Sparcubes signed distances and deformations into sparse latents and decodes them without modality conversion.Sparse residual convolutions, self-pruning upsampling, sign-and-magnitude field prediction, deformation regression, and KL regularization form the training objective.
- Sparconv-VAE: The decoder can repair small occupancy-induced holes by detecting boundary loops and triangulating them through iterative ear filling.Each iteration selects the sharpest convex ear, adds a triangle, and updates the boundary until the holes close.
4 Experiments
Experiments evaluate Sparcubes remeshing, Sparconv-VAE reconstruction, and latent-diffusion generation on challenging datasets, alongside cost and supervision ablations. The framework consistently improves fidelity while reducing conversion and training costs.
- Experiment Settings: Sparcubes is evaluated on ABO, Objaverse, and Wild datasets using CD, ANC, and F1 for watertight remeshing.The comparison includes prior pipelines reported as Dora-wt.
- Watertight Remeshing Results: Sparcubes consistently outperforms prior watertight remeshing pipelines across datasets and metrics, with wt-512 exceeding prior wt-1024 quality.Qualitative comparisons show preserved components and finer geometric details.
- VAE Reconstruction Results: Sparconv-VAE outperforms prior VAEs across most datasets and metrics while reconstructing complex shapes, watertight open surfaces, and hidden internal structures.Baselines include TRELLIS, Craftsman, Dora, and XCubes.
- Generation Results: Under matched diffusion architecture and model size, Sparconv-VAE produces more detailed single-image-to-3D reconstructions than TRELLIS.Generated examples retain watertight surfaces and fine details such as pavilion ridges, facial features, and robot structures.
- Efficiency: At 512 resolution, Sparcubes conversion takes around 15 s, while 1024 resolution takes around 30 s versus around 90 s for prior methods.The modality-consistent VAE also avoids earlier pipelines’ additional SDF resampling step.
- Efficiency and Ablations: Sparconv-VAE converges in less than two days, approximately four times faster than prior methods requiring roughly seven days.Adding 2D rendering supervision yields negligible improvement for Sparconv-VAE.
5 Conclusion
Sparc3D combines topology-preserving remeshing with modality-consistent latent encoding for high-resolution 3D generation. Its scope is bounded by loss of textures and discarded hidden elements in fully closed meshes with internal structures.
- Framework: Sparc3D combines Sparcubes remeshing and Sparconv-VAE encoding to address topology preservation and modality consistency in 3D generation.Sparcubes retains fine details and small components, while Sparconv-VAE uses sparse convolutions and self-pruning without heavyweight attention.
- Implications: The framework provides a scalable foundation for high-fidelity 3D generation across AR/VR, robotics simulation, and 3D printing domains.The conclusion describes these as virtual and physical application settings.
- Limitations: Sparcubes does not retain original texture information and discards hidden elements when remeshing fully closed meshes with internal structures.These drawbacks are shared with several prior methods.