PaperlayerView exact revision on arXiv

Source-linked AI summary

DiffGI: Differentiable Geometry Images for High-Fidelity Thin-Shell 3D Generation

Eungjune Shim, Hansol Lee, Eunjung Ju

arXiv:2607.13365v1cs.CV

TL;DR

Existing 3D generative models struggle with thin-shell and non-manifold surfaces, while geometry-image methods suffer resolution-dependent boundary artifacts and non-differentiable reconstruction. DiffGI uses continuous TSDF geometry images and differentiable surface extraction to enable compact latent modeling, achieving stronger reconstruction and generation quality with less computation.

  • Problem

    Existing approaches struggle to represent thin-shell and open-boundary structures, while binary geometry-image boundaries lose detail and develop staircase artifacts when downsampled.

  • Method

    DiffGI combines continuous 2D TSDF geometry images with Differentiable Marching Squares, a geometry-aware VAE, and latent diffusion for differentiable compact 3D generation.

  • Results

    DiffGI consistently outperforms prior geometry-image methods on garment and furniture benchmarks while achieving competitive or superior generation quality with significantly less compute.

  • Takeaways & Limitations

    The framework enables efficient generation of thin non-manifold meshes in an ultra-compact 32 × 32 latent space with sharper boundaries and better thin-shell preservation.

  • Takeaways & Limitations

    Independent UV-chart reconstruction can produce visible seams between neighboring patches, creating challenges for downstream physical simulation.

Abstract

from arXiv · show

Existing 3D generative models predominantly rely on implicit volumetric representations, which enforce watertight topology and struggle to represent thin-shell and non-manifold geometries such as garments. Geometry image-based approaches offer a surface-centric alternative, but existing methods rely on discrete binary occupancy maps whose resolution-dependent boundary encoding causes staircase artifacts and information loss upon downsampling, while surface reconstruction remains a non-differentiable post-processing step disconnected from the learning pipeline. To address this, we propose Differentiable Geometry Image (DiffGI), an end-to-end 3D-to-2D mapping framework that seamlessly integrates surface representation and geometric optimization. DiffGI replaces binary maps with a continuous 2D Truncated Signed Distance Function (TSDF), which encodes boundary position at subpixel precision within a fixed grid resolution, eliminating resolution-dependent staircase artifacts even under aggressive downsampling. Building on this continuous field, we introduce a differentiable Marching Squares algorithm based on analytical linear interpolation, allowing gradients from 3D surface losses to propagate back to the 2D latent space. Leveraging this differentiable pipeline, we train a DiffGI-VAE augmented with a geometry-aware normal rendering loss to compress complex 3D surfaces into an ultra-compact 32X32 latent space, and instantiate a transformer-based latent diffusion model with a flow-matching objective on top of this space for conditional 3D generation. Extensive experiments on garment and object datasets demonstrate that our method achieves superior reconstruction fidelity and boundary precision compared to prior geometry-image and voxel-based approaches, while requiring significantly fewer computational resources.

1 Introduction · 2 Related Work

DiffGI addresses the limitations of implicit and occupancy-based 3D representations for thin-shell, open-boundary, and UV-dependent surfaces by combining a continuous 2D TSDF with differentiable mesh reconstruction. The resulting framework supports end-to-end optimization and compact 3D generation in a 32 × 32 latent space.

  • 1 Introduction: Implicit field methods generally assume watertight surfaces, limiting their ability to represent thin-shell and open-boundary structures such as garments and furniture frames.They can produce artificial thickness or front-back blending, while Marching Cubes meshes lack UV coordinates for some industrial pipelines.
  • 2.3 Differentiable Iso-Surface Extraction: Existing differentiable iso-surface methods make surface extraction part of optimization but typically operate on 3D grids with cubic memory scaling and without natural UV parameterization.DiffGI targets lighter computation and UV-friendly meshes using structured 2D representations.
  • 2.3 Differentiable Iso-Surface Extraction: Differentiable Marching Squares reconstructs meshes from the 2D TSDF while allowing gradients from 3D surface losses to propagate back to the 2D latent space.It brings differentiable iso-surface extraction to UV-space geometry images rather than 3D voxel or tetrahedral grids.
  • 1 Introduction: DiffGI combines TSDF geometry images, a geometry-aware normal rendering loss, and a VAE to compress complex non-manifold surfaces into a 32×32×4 latent space.The framework also uses a transformer-based latent diffusion model for efficient generation of thin non-manifold meshes.
  • 1 Introduction: DiffGI replaces binary occupancy with a continuous 2D TSDF that encodes boundary position at subpixel precision within a fixed grid resolution.This removes the resolution dependency and staircase artifacts associated with occupancy maps.
  • 2.1 Implicit and Explicit Representations for 3D Generation: The method retains structured 2D surface representations while enabling differentiable, boundary-aware surface recovery for end-to-end modeling.This addresses the optimization difficulty posed by irregular discrete topology in explicit surface representations.
  • 2.2 Geometry-Image-Based Surface Representations: Geometry images map 3D surfaces onto regular UV grids, enabling image-like surface tensors and compatibility with 2D generative models.Prior systems include Omages, GIMDiffusion, and GarmageNet, but their binary occupancy or mask channels make boundary localization resolution-dependent.

3 Method

DiffGI represents surfaces with continuous 2D TSDF geometry images and reconstructs them through differentiable Marching Squares, enabling subpixel boundary fidelity and gradient-based geometric optimization. A VAE compresses the representation to a 32 × 32 × 4 latent space using geometry-aware normal supervision, while a DiT flow-matching model supports conditional 3D generation.

  • Continuous TSDF Representation: DiffGI replaces binary occupancy maps with continuous 2D TSDFs that preserve subpixel boundary information under low-resolution, aggressive compression.Binary transitions lose fine details during downsampling and produce staircase artifacts, whereas TSDFs encode distance to the nearest boundary.
  • Differentiable Surface Reconstruction: Differentiable Marching Squares computes subpixel boundary vertices as continuous functions of adjacent TSDF values, allowing 3D geometric losses to backpropagate into the 2D tensors.Opposite-signed neighboring values guarantee a zero-crossing, and analytical interpolation replaces the discrete topology-dependent gradient barrier of traditional Marching Squares.
  • Differentiable Surface Reconstruction: The interpolation uses a regularization constant ϵ, such as 10^-5, to prevent gradient explosion when neighboring TSDF values are nearly equal.The sign term is treated as locally constant during backpropagation, so gradients flow through the continuous value ratio.
  • DiffGI-VAE: DiffGI-VAE compresses 256 × 256 × 4 geometry-image tensors into a 32 × 32 × 4 latent space and adds a differentiable normal-rendering loss to preserve curvature, orientation, wrinkles, and sharp edges.The normal loss compares rendered normals of reconstructed and ground-truth meshes, complementing pixel-level position and TSDF losses.
  • Conditional 3D Generation: A transformer-based latent diffusion model uses DiT global attention and flow matching for conditional generation on the compact latent space.The framework supports label-conditioned shape generation and image-conditioned completion of unseen back surfaces through DINOv2-Large embeddings and cross-attention.

4 Experiments

Experiments on furniture and garment benchmarks show that DiffGI-VAE improves reconstruction fidelity and boundary precision under aggressive compression, while ablations, efficiency tests, and conditional-generation studies validate its design and scalability.

  • Reconstruction Fidelity: DiffGI-VAE consistently outperforms Omages and GarmageNet in CD, EMD, and JSD on ABO and GarmageSet despite compressing 256×256 geometry images into 32×32×4 latents.On ABO, NC is slightly lower than Omages, while GarmageSet results favor DiffGI-VAE.
  • Ablation Study: Switching from occupancy to TSDF without LNormal reduces CD from 1.503×10−3 to 0.595×10−3 and JSD from 4.539 × 10−3 to 2.169 × 10−3.This supports TSDF’s stronger encoding of geometric information near boundaries.
  • Ablation Study: Adding LNormal to TSDF increases NC from 0.921 to 0.961 while further improving CD and JSD; combining TSDF with LNormal also mitigates boundary noise and aliasing.The Occ. + LNormal variant reaches NC 0.947, exceeding TSDF without LNormal at 0.921.
  • Inference Cost: DiffGI generates image-conditioned 3D outputs in ∼1.2 s on an RTX 4070 using 3.22 GB VRAM and ∼8.5 s CPU-only on a MacBook M4.These measurements demonstrate operation across server and edge-device settings.
  • Conditional Generation: Label-conditioned generation improves P-FID across ABO categories, with largest reductions of 50% for Chair and 49% for Lamp; Lamp P-KID is 0.07 versus Omages 0.26.DiffGI also produces natural garment draping and wrinkles from sewing-pattern silhouettes, with local 2D edits propagating to corresponding 3D geometry.
  • Garment Generation: DiffGI achieves CD 1.35 × 10−2, F1 0.48, dH 8.42×10−2, and BCD 2.91×10−2 with only 23K vertices on average.The results indicate stable preservation of thin fabric boundaries and compact meshes.

5 Conclusion, Limitations, and Future Work

DiffGI combines continuous TSDF geometry images with differentiable Marching Squares to enable subpixel-precise, end-to-end surface reconstruction and compact 3D representation. Its latent diffusion model delivers strong generation quality with lower compute, while limitations motivate future work on sharp edges, appearance, and garment controllability.

  • Conclusion: DiffGI replaces binary occupancy maps with continuous 2D TSDFs and integrates mesh reconstruction into learning through Differentiable Marching Squares.This enables subpixel-precise boundary reconstruction and backpropagation of 3D surface losses into the 2D latent space.
  • Conclusion: DiffGI-VAE compresses complex non-manifold surfaces into a 32×32×4 latent space, while latent diffusion outperforms prior geometry-image methods on garment and furniture benchmarks.The transformer-based diffusion model achieves competitive or superior generation quality with significantly less compute and runs in real time on consumer-grade hardware.
  • Limitations: TSDF interpolation can round extreme sharp edges, the framework does not generate RGB textures or PBR attributes, and independently reconstructed UV charts can produce visible seams.The seams arise because adjacent patches are treated as separate regions, including between the upper and lower dress parts.
  • Future Work: Future work will add feature-preserving iso-surfacing, jointly generate high-resolution textures and material maps, and separate pattern layout from 3D shape reconstruction.The proposed two-stage pipeline is intended to improve controllability over complex garment designs.

DiffGI: Differentiable Geometry Images for

This section presents “DiffGI: Differentiable Geometry Images for High-Fidelity Thin-Shell 3D Generation” and includes supplementary material.

  • The work is titled “DiffGI: Differentiable Geometry Images for High-Fidelity Thin-Shell 3D Generation.”
  • The section includes supplementary material.

S1 Resolution-Dependent Analysis: TSDF vs. Occupancy

Across resolutions without VAE compression, TSDF geometry images better preserve thin structures and open boundaries than occupancy at low resolutions, with both representations converging beyond 256. Its continuous, distributed boundary encoding is especially beneficial for complex garment geometry and robust to later VAE compression.

  • Resolution comparison: TSDF achieves lower Hausdorff Distance than Occupancy at low resolutions, especially 64 and 128, where Occupancy exhibits staircase artifacts on thin structures and open boundaries.The comparison uses resolutions 64, 128, 256, 512, and 1024 without VAE compression.
  • Resolution comparison: Beyond resolution 256, TSDF and Occupancy converge to near-identical error, establishing 256 as sufficient reconstruction fidelity for the pipeline.This conclusion comes from the resolution sweep without VAE compression.
  • Dataset-dependent behavior: TSDF provides a larger advantage on GarmageSet than ABO because garments contain dense wrinkles, curved hemlines, and irregular open boundaries.ABO furniture is dominated by flat planes and straight edges, indicating that continuous boundary encoding benefits more complex boundaries.
  • VAE compression: TSDF is more robust to VAE compression because real-valued distances distribute boundary information across neighboring pixels, unlike occupancy’s single 0/1 transition pixel.The passage states that compression amplifies the modest high-resolution raw difference into a substantial performance gap.

S2 VAE Initialization: SD1.5 Pretraining vs. Training from Scratch

Training DiffGI-VAE from scratch achieves reconstruction fidelity comparable to SD1.5 initialization, showing that final reconstruction quality primarily comes from the TSDF representation. SD1.5 initialization mainly accelerates convergence rather than improving converged fidelity.

  • Experimental setup: The comparison trains DiffGI-VAE from random initialization while keeping the architecture and all other settings identical, evaluating both models on GarmageSet with DMS extraction.This isolates the effect of initialization.
  • Reconstruction fidelity: CD is 0.47 from scratch versus 0.46 with SD1.5 initialization, while NC is identical at 0.96.The reported metrics show comparable reconstruction fidelity.
  • Interpretation: The comparable final fidelity indicates that reconstruction gains stem from the TSDF representation rather than pretrained initialization.This conclusion follows from the matched architecture and nearly coincident final metrics.
  • Convergence: SD1.5 initialization reaches low loss in far fewer steps, whereas the from-scratch model starts from noise and progressively recovers comparable reconstructions.After convergence, both models attain essentially the same fidelity.

S3 Impact of Geometric Data Augmentation and Model Scaling

Geometric augmentation expands the ABO training set from 3,800 to 500K semantically consistent samples, enabling stable training of larger DiT models and improving generation quality. Without augmentation, only DiT-Tiny converges stably, while DiT-Base is unstable and shrinking capacity sacrifices final quality.

  • Training stability: With only 3,800 original training samples, DiT-Tiny converges stably, whereas DiT-Base exhibits unstable validation loss.This comparison is reported on the ABO dataset.
  • Model scaling: 3,800 →500K samples through geometric augmentation enables DiT-Base to converge reliably and outperform DiT-Tiny in generation quality.The augmented training set supports larger model capacity rather than relying on model shrinking for stability.
  • Model scaling: Shrinking the model to achieve stability sacrifices final quality, while augmentation is essential for training larger-capacity models on limited data.The conclusion follows the observed contrast between DiT-Tiny without augmentation and augmented DiT-Base.
  • Geometric augmentation: UV-aligned geometric augmentation generates semantically consistent variations that improve stability and generalisation without changing 3D semantics.Unlike conventional image augmentations, it leverages structural constraints of geometry images.

S4 Extended Discussion on Saddle-Point Convention in DMS

The method deterministically separates the two patches in Marching Squares’ ambiguous saddle-point Cases 6 and 9. UV padding makes shared-cell ambiguities unlikely at 256×256, while boundary vertices remain continuous functions of TSDF values, preserving backpropagation.

  • Saddle-Point Convention: Marching Squares Cases 6 and 9 are treated deterministically as two separate independent regions in ambiguous saddle-point configurations.This convention is introduced for the topologically ambiguous cases.
  • Saddle-Point Convention: Inter-patch padding during UV packing spatially separates distinct surface charts, making ambiguous diagonal configurations unlikely at 256×256 resolution.Narrow bridge-like structures within one cell could still be split, but rarely arise at this operating resolution with sufficient padding.
  • Differentiability: Although the topological choice is discrete, each boundary vertex has coordinates that remain continuous functions of the TSDF values.This continuity follows the interpolation relationship defined in Eq. (1) of the main paper.
  • Differentiability: The backpropagation path is therefore fully preserved even in ambiguous saddle-point cases.Continuous boundary coordinates maintain differentiability despite the discrete topological convention.

S5 Additional Generation Results · S6 Clarifications on Evaluation Protocol

Additional results show that DiffGI generates thin-shell garments with cleaner boundaries, finer details, open boundaries, and diverse geometry under fixed conditions. Evaluation clarifications explain native mesh extraction protocols and why BCD is inapplicable to watertight TRELLIS meshes.

  • S5 Additional Generation Results: Additional qualitative results evaluate conditional generation models on GarmageSet and zero-shot image-to-3D generation on WARDROBE.The reported cases include samples not shown in the main paper and real-world clothing images excluded from training.
  • S5 Additional Generation Results: DiffGI reconstructs cleaner boundaries and finer garment details than TRELLIS and GarmageNet, while using significantly fewer vertices.TRELLIS produces overly thick meshes from its watertight assumption, whereas GarmageNet exhibits binary-occupancy boundary aliasing.
  • S5 Additional Generation Results: ∼23K vs. 109–526K vertices, with DiffGI producing cleaner boundaries and thinner shells than the compared methods.These additional image-conditioned GarmageSet results are shown in Fig. S5.
  • S5 Additional Generation Results: On unseen WARDROBE categories, DiffGI faithfully recovers thin-shell structures and open boundaries, transferring geometric priors learned by DiffGI-VAE.The WARDROBE clothing images were not used during training.
  • S5 Additional Generation Results: Given the same input image, DiffGI generates garments with varied UV chart layouts, draping, and wrinkles while remaining consistent with the condition.The results indicate diversity over both chart layouts and surface geometry rather than conditional mode collapse.
  • S6 Clarifications on Evaluation Protocol: DiffGI uses Differentiable Marching Squares for mesh extraction, while Omages uses native tessellation and GarmageNet is additionally evaluated with its official extraction.GarmageNet’s native multi-stage non-learning extraction takes about 5.1 s per mesh; DiffGI remains superior under both extraction settings.
  • S6 Clarifications on Evaluation Protocol: BCD is N/A for TRELLIS because its inherently watertight meshes have no open boundaries, making Boundary Chamfer Distance inapplicable by definition.The clarification concerns the BCD reporting in Table 5 of the main paper.
  • S6 Clarifications on Evaluation Protocol: On zero-shot WARDROBE images, DiffGI better matches input silhouettes and preserves open boundaries compared to TRELLIS.The dataset was not used during training.

S7 Role of Differentiable Surface Extraction in Enabling LNormal · S8 Additional Implementation Details

S7 shows that differentiable surface extraction is required for LNormal to supervise the VAE end to end, while S8 specifies TSDF computation, evaluation scope, augmentation references, and diffusion-model architecture documentation.

  • S7 Role of Differentiable Surface Extraction in Enabling LNormal: Differentiable surface extraction allows LNormal gradients to reach the VAE encoder and decoder instead of being blocked by post-processing.The TSDF pipeline uses DMS, while occupancy ablations use differentiable Omages-style tessellation.
  • S7 Role of Differentiable Surface Extraction in Enabling LNormal: 1.503×10−3 →0.595×10−3 CD results from switching Occ. to TSDF without LNormal, establishing the continuous TSDF gain.The comparison is reported in Table 2 of the main paper.
  • S7 Role of Differentiable Surface Extraction in Enabling LNormal: 0.921 to 0.961 NC and 0.461×10−3 CD result when LNormal is added to TSDF, complementing the TSDF benefit.This additional improvement depends on end-to-end differentiable surface extraction.
  • S8 Additional Implementation Details: The TSDF map is computed entirely in the 2D UV plane rather than from the 3-channel position map.Packed UV charts are rasterized into a binary mask at 1024×1024 before distance-transform processing.
  • S8 Additional Implementation Details: Each TSDF pixel receives a signed 2D Euclidean distance in pixels to the nearest chart contour, positive inside and negative outside charts.The distance transform operates on the rasterized packed-UV-chart mask.
  • S8 Additional Implementation Details: WARDROBE is used only for qualitative zero-shot evaluation in Fig. S6, not for quantitative benchmarks on ABO or GarmageSet.This separates the dataset’s qualitative role from the paper’s quantitative evaluation datasets.
  • S8 Additional Implementation Details: Geometric data augmentation is analyzed in detail in Sec. S3.The additional implementation-details section points readers to that separate analysis.
  • S8 Additional Implementation Details: Specifications for the three conditional generation models are summarized in Tab. S2.The architectures correspond to the conditional generation models introduced in Sec. 3.4 of the main paper.
Loading 2607.13365v1…