PaperlayerView exact revision on arXiv

Source-linked AI summary

SparseFlex: High-Resolution and Arbitrary-Topology 3D Shape Modeling

Xianglong He, Zi-Xin Zou, Chia-Hao Chen, Yuan-Chen Guo, Ding Liang, Chun Yuan, Wanli Ouyang, Yan-Pei Cao, Yangguang Li

arXiv:2503.21732v1cs.CV

TL;DR

High-fidelity 3D modeling with arbitrary topology is limited by watertight preprocessing and the memory demands of dense high-resolution representations. SparseFlex uses sparse differentiable isosurfaces and frustum-aware voxel training to reconstruct and generate detailed shapes with rendering supervision. The paper reports state-of-the-art reconstruction accuracy and arbitrary-topology generation, including open surfaces and interiors.

  • Problem

    High-fidelity 3D modeling with arbitrary topology is difficult, while rendering supervision with dense implicit fields requires excessive memory at high resolutions.

    Source: §1 · p. 2

  • Method

    SparseFlex combines a sparse Flexicubes-based isosurface representation with frustum-aware sectional voxel training that activates relevant voxels during rendering.

    Source: §1 · p. 2, §3.1 · p. 4, §3.3 · p. 5

  • Results

    The paper reports state-of-the-art reconstruction accuracy and high-resolution, detailed 3D shape generation with arbitrary topology, including open surfaces and interiors.

    Source: §1 · p. 2, §3.3 · p. 5, §5 · p. 9

  • Takeaways & Limitations

    SparseFlex supports high-resolution differentiable mesh reconstruction and generation using rendering supervision while representing open surfaces and reconstructing interiors.

    Source: §1 · p. 2, §3.3 · p. 5, §5 · p. 9

Abstract

from arXiv · show

Creating high-fidelity 3D meshes with arbitrary topology, including open surfaces and complex interiors, remains a significant challenge. Existing implicit field methods often require costly and detail-degrading watertight conversion, while other approaches struggle with high resolutions. This paper introduces SparseFlex, a novel sparse-structured isosurface representation that enables differentiable mesh reconstruction at resolutions up to $1024^3$ directly from rendering losses. SparseFlex combines the accuracy of Flexicubes with a sparse voxel structure, focusing computation on surface-adjacent regions and efficiently handling open surfaces. Crucially, we introduce a frustum-aware sectional voxel training strategy that activates only relevant voxels during rendering, dramatically reducing memory consumption and enabling high-resolution training. This also allows, for the first time, the reconstruction of mesh interiors using only rendering supervision. Building upon this, we demonstrate a complete shape modeling pipeline by training a variational autoencoder (VAE) and a rectified flow transformer for high-quality 3D shape generation. Our experiments show state-of-the-art reconstruction accuracy, with a ~82% reduction in Chamfer Distance and a ~88% increase in F-score compared to previous methods, and demonstrate the generation of high-resolution, detailed 3D shapes with arbitrary topology. By enabling high-resolution, differentiable mesh reconstruction and generation with rendering losses, SparseFlex significantly advances the state-of-the-art in 3D shape representation and modeling.

1. Introduction

SparseFlex addresses the difficulty of high-fidelity 3D modeling with arbitrary topology by combining differentiable Flexicubes-based isosurfaces with sparse voxels and frustum-aware training. The resulting pipeline supports high-resolution reconstruction, open surfaces, interiors, and rendering-supervised shape generation.

  • High-fidelity 3D generation remains difficult because 3D geometry is complex and arbitrary topology includes open surfaces and complex interiors.Source: §1 · p. 2
  • Rendering supervision preserves fine details and avoids watertight preprocessing, but dense implicit fields consume excessive memory at high resolutions.Source: §1 · p. 2
  • SparseFlex combines differentiable Flexicubes-based isosurface extraction with sparse voxels, reducing memory and pruning voxels near open boundaries.Source: §1 · p. 2
  • Frustum-aware sectional voxel training activates only voxels within the camera frustum and adaptively controls frustum parameters to reduce training overhead.Source: §1 · p. 2
  • The method enables mesh-interior reconstruction from rendering supervision and supports a complete shape-modeling pipeline with a VAE and rectified-flow transformer.Source: §1 · p. 2
  • The paper reports state-of-the-art reconstruction accuracy and high-resolution detailed shape generation with arbitrary topology.Source: §1 · p. 2

2. Related Work

Prior 3D modeling approaches use point clouds, meshes, implicit fields, and open-surface representations, each with distinct resolution or topology constraints. SparseFlex-related modeling uses sparse isosurfaces and a VAE pipeline to preserve detailed arbitrary-topology shapes.

  • Point clouds are flexible and easy to acquire but require an additional surface-reconstruction step because they do not represent solid surfaces.Source: §2.1 · p. 2
  • The SparseFlex VAE pipeline voxelizes mesh-sampled point clouds, compresses features with a sparse transformer, upsamples through self-pruning, and decodes SparseFlex.Source: Figure 2 · p. 3
  • Triangle-mesh generation methods can resemble artist-created meshes but struggle with high-quality meshes containing many faces.Source: §2.1 · p. 3
  • Implicit fields produce high-quality meshes, while isosurface techniques are typically used to extract their surfaces.Source: §2.1 · p. 3
  • Open-surface methods include UDF-based approaches, but binary occupancy extraction can introduce discontinuities and artifacts.Source: §2.1 · p. 3
  • Dense-grid representations for non-watertight meshes limit high-resolution processing of complex shapes.Source: §2.1 · p. 3
  • VAE-based approaches encode 3D shapes into latent spaces, with geometry-focused methods often using point clouds sampled from mesh surfaces.Source: §2.2 · pp. 3–4

3. Method

SparseFlex combines differentiable Flexicubes extraction with sparse voxels and frustum-aware training to model high-resolution shapes, including open surfaces and interiors, from rendering supervision.

SparseFlex Representation

  • SparseFlex uses a sparse set of surface-adjacent voxels instead of a dense grid, reducing memory while representing open surfaces by omitting empty regions.Only sparse voxels receive Dual Marching Cubes extraction.Source: §3.1 · p. 4
  • SparseFlex inherits Flexicubes’ differentiability and supports end-to-end optimization with rendering losses without watertight mesh preprocessing.The continuous, deformable SDF helps preserve fine details and represent open-surface meshes.Source: §3.1 · p. 4

SparseFlex VAE

  • The VAE encodes point-cloud geometry into a compact latent space and decodes SparseFlex parameters, including SDF values, deformations, and interpolation weights.Sparse voxel features are processed with a sparse transformer, while the decoder predicts the parameters of a SparseFlex instance.Source: §3.2 · pp. 4–5
  • Self-pruning upsampling progressively increases voxel resolution while removing redundant voxels, benefiting high-resolution reconstruction and open-surface representation.Each module subdivides voxels by a factor of 4 and prunes using predicted occupancy.Source: §3.2 · p. 5

Frustum-aware Training

  • Frustum-aware sectional training activates only voxels inside the camera frustum, reducing high-resolution rendering cost by processing a portion of 3D space per iteration.The active set is selected using the camera parameters and the Model-View-Projection matrix.Source: §3.3 · p. 5
  • Adaptive frustum positioning enables interior reconstruction with rendering supervision by placing the camera inside the object or intersecting the mesh with the near plane.Zoomed-in viewpoints also provide more detailed surface supervision for higher-resolution training.Source: §3.3 · p. 5

Image-to-3D Generation

  • The trained SparseFlex VAE supports an image-conditioned 3D generation pipeline built from structure-flow and structured-latent-flow models.The pipeline is designed for high-quality shape generation from images.Source: §3.4 · p. 6

4. Experiments

Experiments evaluate SparseFlex reconstruction, generation, and efficiency across diverse datasets and baselines. Results show stronger reconstruction for complex and open-surface shapes, improved detail at higher resolutions, and reduced training costs through frustum-aware sectional voxel training.

VAE Reconstruction Evaluation

  • ∼82% reduction in CD and ∼88% increase in F-score are achieved as SparseFlex resolution increases.Ours256 already outperforms other baselines, while higher resolutions further improve reconstruction; watertight-trained methods perform poorly on non-watertight meshes.Source: §4.3 · p. 7
  • SparseFlex achieves the best reconstruction performance among methods designed for open surfaces on Deepfashion3D.The comparison specifically evaluates open-surface reconstruction quality.Source: Table 2 · p. 6, §4.3 · p. 8

Image-to-3D Generation

  • Generated shapes preserve sharp edges and fine details while matching input images, including images collected from real-world settings.The generation pipeline is evaluated quantitatively on Toys4k and qualitatively with in-the-wild images.Source: §4.4 · p. 8, Figure 6 · p. 8, Table 4 · p. 8

Self-Pruning Upsampling

  • The self-pruning upsampling module improves open-surface reconstruction by pruning voxels near open boundaries.This allows reconstructed boundaries to align more closely with the input.Source: §4.5 · p. 8

SparseFlex and Frustum-aware Sectional Voxel Training

  • Frustum-aware sectional voxel training reduces training memory requirements by avoiding full surface extraction during rendering.SparseFlex reduces feed-forward runtime and GPU memory, while the sectional strategy addresses scaling limitations at higher resolutions.Source: §4.5 · p. 8, Table 3 · p. 8

Sparse Voxel Resolutions

  • Higher sparse-voxel resolution reveals more details in complex structures and provides better geometry reconstruction than TRELLIS at the same resolution.The qualitative comparison highlights details such as tank tracks as resolution increases.Source: §4.5 · p. 8, Figure 5 · p. 8

5. Conclusion

SparseFlex combines a sparse-structured isosurface representation with frustum-aware sectional voxel training for high-resolution differentiable reconstruction, including open surfaces and interiors. The paper extends this foundation to VAE-based image-to-3D generation while identifying remaining resolution, boundary-artifact, and interior-control limitations.

  • SparseFlex enables high-resolution differentiable mesh reconstruction from rendering supervision, including open surfaces.Source: §5 · p. 9
  • Frustum-aware sectional voxel training with adaptive frustum control reduces memory consumption during high-resolution SparseFlex VAE training.Source: §5 · p. 9
  • The training strategy enables reconstruction of mesh interiors using only rendering loss.Source: §5 · p. 9
  • The complete pipeline develops image-to-3D generation following TELLIS and demonstrates high-quality generation with open surfaces.Source: §5 · p. 9
  • Open-surface boundaries may show minor artifacts at lower resolutions, high-resolution generation remains computationally demanding, and interior-structure control needs improvement.Source: §5 · p. 9

Related papers

Loading 2503.21732v1…