Source-linked AI summary
SparseFlex: High-Resolution and Arbitrary-Topology 3D Shape Modeling
Xianglong He, Zi-Xin Zou, Chia-Hao Chen, Yuan-Chen Guo, Ding Liang, Chun Yuan, Wanli Ouyang, Yan-Pei Cao, Yangguang Li
TL;DR
High-fidelity 3D modeling with arbitrary topology is limited by watertight preprocessing and the memory demands of dense high-resolution representations. SparseFlex uses sparse differentiable isosurfaces and frustum-aware voxel training to reconstruct and generate detailed shapes with rendering supervision. The paper reports state-of-the-art reconstruction accuracy and arbitrary-topology generation, including open surfaces and interiors.
Problem
High-fidelity 3D modeling with arbitrary topology is difficult, while rendering supervision with dense implicit fields requires excessive memory at high resolutions.
Method
SparseFlex combines a sparse Flexicubes-based isosurface representation with frustum-aware sectional voxel training that activates relevant voxels during rendering.
Results
The paper reports state-of-the-art reconstruction accuracy and high-resolution, detailed 3D shape generation with arbitrary topology, including open surfaces and interiors.
Takeaways & Limitations
SparseFlex supports high-resolution differentiable mesh reconstruction and generation using rendering supervision while representing open surfaces and reconstructing interiors.
Abstract
from arXiv · showhide
Creating high-fidelity 3D meshes with arbitrary topology, including open surfaces and complex interiors, remains a significant challenge. Existing implicit field methods often require costly and detail-degrading watertight conversion, while other approaches struggle with high resolutions. This paper introduces SparseFlex, a novel sparse-structured isosurface representation that enables differentiable mesh reconstruction at resolutions up to $1024^3$ directly from rendering losses. SparseFlex combines the accuracy of Flexicubes with a sparse voxel structure, focusing computation on surface-adjacent regions and efficiently handling open surfaces. Crucially, we introduce a frustum-aware sectional voxel training strategy that activates only relevant voxels during rendering, dramatically reducing memory consumption and enabling high-resolution training. This also allows, for the first time, the reconstruction of mesh interiors using only rendering supervision. Building upon this, we demonstrate a complete shape modeling pipeline by training a variational autoencoder (VAE) and a rectified flow transformer for high-quality 3D shape generation. Our experiments show state-of-the-art reconstruction accuracy, with a ~82% reduction in Chamfer Distance and a ~88% increase in F-score compared to previous methods, and demonstrate the generation of high-resolution, detailed 3D shapes with arbitrary topology. By enabling high-resolution, differentiable mesh reconstruction and generation with rendering losses, SparseFlex significantly advances the state-of-the-art in 3D shape representation and modeling.
1. Introduction
SparseFlex addresses the difficulty of high-fidelity 3D modeling with arbitrary topology by combining differentiable Flexicubes-based isosurfaces with sparse voxels and frustum-aware training. The resulting pipeline supports high-resolution reconstruction, open surfaces, interiors, and rendering-supervised shape generation.
- High-fidelity 3D generation remains difficult because 3D geometry is complex and arbitrary topology includes open surfaces and complex interiors.
- Rendering supervision preserves fine details and avoids watertight preprocessing, but dense implicit fields consume excessive memory at high resolutions.
- SparseFlex combines differentiable Flexicubes-based isosurface extraction with sparse voxels, reducing memory and pruning voxels near open boundaries.
- Frustum-aware sectional voxel training activates only voxels within the camera frustum and adaptively controls frustum parameters to reduce training overhead.
- The method enables mesh-interior reconstruction from rendering supervision and supports a complete shape-modeling pipeline with a VAE and rectified-flow transformer.
- The paper reports state-of-the-art reconstruction accuracy and high-resolution detailed shape generation with arbitrary topology.
2. Related Work
Prior 3D modeling approaches use point clouds, meshes, implicit fields, and open-surface representations, each with distinct resolution or topology constraints. SparseFlex-related modeling uses sparse isosurfaces and a VAE pipeline to preserve detailed arbitrary-topology shapes.
- Point clouds are flexible and easy to acquire but require an additional surface-reconstruction step because they do not represent solid surfaces.
- The SparseFlex VAE pipeline voxelizes mesh-sampled point clouds, compresses features with a sparse transformer, upsamples through self-pruning, and decodes SparseFlex.
- Triangle-mesh generation methods can resemble artist-created meshes but struggle with high-quality meshes containing many faces.
- Implicit fields produce high-quality meshes, while isosurface techniques are typically used to extract their surfaces.
- Open-surface methods include UDF-based approaches, but binary occupancy extraction can introduce discontinuities and artifacts.
- Dense-grid representations for non-watertight meshes limit high-resolution processing of complex shapes.
- VAE-based approaches encode 3D shapes into latent spaces, with geometry-focused methods often using point clouds sampled from mesh surfaces.
3. Method
SparseFlex combines differentiable Flexicubes extraction with sparse voxels and frustum-aware training to model high-resolution shapes, including open surfaces and interiors, from rendering supervision.
- SparseFlex Representation: SparseFlex uses a sparse set of surface-adjacent voxels instead of a dense grid, reducing memory while representing open surfaces by omitting empty regions.Only sparse voxels receive Dual Marching Cubes extraction.
- SparseFlex Representation: SparseFlex inherits Flexicubes’ differentiability and supports end-to-end optimization with rendering losses without watertight mesh preprocessing.The continuous, deformable SDF helps preserve fine details and represent open-surface meshes.
- SparseFlex VAE: The VAE encodes point-cloud geometry into a compact latent space and decodes SparseFlex parameters, including SDF values, deformations, and interpolation weights.Sparse voxel features are processed with a sparse transformer, while the decoder predicts the parameters of a SparseFlex instance.
- SparseFlex VAE: Self-pruning upsampling progressively increases voxel resolution while removing redundant voxels, benefiting high-resolution reconstruction and open-surface representation.Each module subdivides voxels by a factor of 4 and prunes using predicted occupancy.
- Frustum-aware Training: Frustum-aware sectional training activates only voxels inside the camera frustum, reducing high-resolution rendering cost by processing a portion of 3D space per iteration.The active set is selected using the camera parameters and the Model-View-Projection matrix.
- Frustum-aware Training: Adaptive frustum positioning enables interior reconstruction with rendering supervision by placing the camera inside the object or intersecting the mesh with the near plane.Zoomed-in viewpoints also provide more detailed surface supervision for higher-resolution training.
- Image-to-3D Generation: The trained SparseFlex VAE supports an image-conditioned 3D generation pipeline built from structure-flow and structured-latent-flow models.The pipeline is designed for high-quality shape generation from images.
4. Experiments
Experiments evaluate SparseFlex reconstruction, generation, and efficiency across diverse datasets and baselines. Results show stronger reconstruction for complex and open-surface shapes, improved detail at higher resolutions, and reduced training costs through frustum-aware sectional voxel training.
- VAE Reconstruction Evaluation: ∼82% reduction in CD and ∼88% increase in F-score are achieved as SparseFlex resolution increases.Ours256 already outperforms other baselines, while higher resolutions further improve reconstruction; watertight-trained methods perform poorly on non-watertight meshes.
- VAE Reconstruction Evaluation: SparseFlex achieves the best reconstruction performance among methods designed for open surfaces on Deepfashion3D.The comparison specifically evaluates open-surface reconstruction quality.
- Image-to-3D Generation: Generated shapes preserve sharp edges and fine details while matching input images, including images collected from real-world settings.The generation pipeline is evaluated quantitatively on Toys4k and qualitatively with in-the-wild images.
- Self-Pruning Upsampling: The self-pruning upsampling module improves open-surface reconstruction by pruning voxels near open boundaries.This allows reconstructed boundaries to align more closely with the input.
- SparseFlex and Frustum-aware Sectional Voxel Training: Frustum-aware sectional voxel training reduces training memory requirements by avoiding full surface extraction during rendering.SparseFlex reduces feed-forward runtime and GPU memory, while the sectional strategy addresses scaling limitations at higher resolutions.
- Sparse Voxel Resolutions: Higher sparse-voxel resolution reveals more details in complex structures and provides better geometry reconstruction than TRELLIS at the same resolution.The qualitative comparison highlights details such as tank tracks as resolution increases.
5. Conclusion
SparseFlex combines a sparse-structured isosurface representation with frustum-aware sectional voxel training for high-resolution differentiable reconstruction, including open surfaces and interiors. The paper extends this foundation to VAE-based image-to-3D generation while identifying remaining resolution, boundary-artifact, and interior-control limitations.
- SparseFlex enables high-resolution differentiable mesh reconstruction from rendering supervision, including open surfaces.
- Frustum-aware sectional voxel training with adaptive frustum control reduces memory consumption during high-resolution SparseFlex VAE training.
- The training strategy enables reconstruction of mesh interiors using only rendering loss.
- The complete pipeline develops image-to-3D generation following TELLIS and demonstrates high-quality generation with open surfaces.
- Open-surface boundaries may show minor artifacts at lower resolutions, high-resolution generation remains computationally demanding, and interior-structure control needs improvement.