PaperlayerView exact revision on arXiv

Source-linked AI summary

DiffGI: Differentiable Geometry Images for High-Fidelity Thin-Shell 3D Generation

Eungjune Shim, Hansol Lee, Eunjung Ju

arXiv:2607.13365v1cs.CV

TL;DR

Binary occupancy makes geometry-image boundaries resolution-dependent, while prior differentiable surface extraction on 3D grids lacks natural UV parameterization. DiffGI combines continuous TSDF geometry images and differentiable Marching Squares with VAE compression and latent diffusion, outperforming prior geometry-image methods on garment and furniture benchmarks and achieving competitive or superior generation quality with significantly less compute.

  • Problem

    Geometry-image methods encode boundaries with binary occupancy, making fine boundary localization resolution-dependent and vulnerable to information loss under downsampling.

    Source: §3.1 · p. 5

  • Method

    DiffGI combines continuous 2D TSDF geometry images and differentiable Marching Squares with a geometry-aware VAE and transformer-based latent diffusion for 3D generation.

    Source: §1 · p. 2

  • Results

    DiffGI consistently outperforms prior geometry-image methods on garment and furniture benchmarks, while its latent diffusion achieves competitive or superior generation quality with significantly less compute.

    Source: §5 · p. 15

  • Takeaways & Limitations

    DiffGI demonstrates efficient generation of thin non-manifold meshes in an ultra-compact 32 × 32 latent space.

    Source: §1 · p. 2

    Independent UV-chart reconstruction can leave visible seams that pose challenges for downstream physical simulation.

    Source: §5 · p. 15

Abstract

from arXiv · show

Existing 3D generative models predominantly rely on implicit volumetric representations, which enforce watertight topology and struggle to represent thin-shell and non-manifold geometries such as garments. Geometry image-based approaches offer a surface-centric alternative, but existing methods rely on discrete binary occupancy maps whose resolution-dependent boundary encoding causes staircase artifacts and information loss upon downsampling, while surface reconstruction remains a non-differentiable post-processing step disconnected from the learning pipeline. To address this, we propose Differentiable Geometry Image (DiffGI), an end-to-end 3D-to-2D mapping framework that seamlessly integrates surface representation and geometric optimization. DiffGI replaces binary maps with a continuous 2D Truncated Signed Distance Function (TSDF), which encodes boundary position at subpixel precision within a fixed grid resolution, eliminating resolution-dependent staircase artifacts even under aggressive downsampling. Building on this continuous field, we introduce a differentiable Marching Squares algorithm based on analytical linear interpolation, allowing gradients from 3D surface losses to propagate back to the 2D latent space. Leveraging this differentiable pipeline, we train a DiffGI-VAE augmented with a geometry-aware normal rendering loss to compress complex 3D surfaces into an ultra-compact 32X32 latent space, and instantiate a transformer-based latent diffusion model with a flow-matching objective on top of this space for conditional 3D generation. Extensive experiments on garment and object datasets demonstrate that our method achieves superior reconstruction fidelity and boundary precision compared to prior geometry-image and voxel-based approaches, while requiring significantly fewer computational resources.

1 Introduction · 2 Related Work

DiffGI combines a continuous 2D TSDF with differentiable mesh reconstruction, connecting surface-level optimization to UV-aligned geometry images. Its VAE and latent diffusion model generate thin, non-manifold surfaces in a compact latent space.

1 Introduction

  • Implicit-field methods often struggle with thin-shell and open-boundary structures, producing artificial thickness or front-back blending, while Marching Cubes meshes lack UV coordinates.These limitations affect practical cases such as garments and furniture frames and hinder pipelines requiring simulation or material editing.Source: §1 · p. 2
  • DiffGI replaces binary occupancy with a continuous 2D TSDF and introduces Differentiable Marching Squares, enabling surface-loss gradients to reach the 2D latent space.The framework pairs this representation with a geometry-aware normal rendering loss and a transformer-based latent diffusion model.Source: §1 · pp. 2–3, §2.3 · p. 5
  • Differentiable Marching Squares makes mesh reconstruction differentiable, allowing 3D surface-loss gradients to propagate back to the 2D latent space.It brings differentiable surface recovery to a 2D UV representation.Source: §1 · pp. 2–3, §2.3 · p. 5
  • A DiffGI-VAE with geometry-aware normal rendering loss compresses complex non-manifold surfaces into a 32×32×4 latent space for generation.A transformer-based latent diffusion model with a flow-matching objective is built on this compact space; the introduction describes real-time generation on consumer-grade hardware.Source: §1 · pp. 2–3
  • DiffGI reconstruction shows sharper boundaries and better preservation of thin-shell structures than occupancy-based reconstruction.Figure 1 contrasts TSDF with normal rendering loss against an occupancy-based geometry image without normal loss.Source: §1 · p. 2, Figure 1 · p. 3
  • The 2D TSDF encodes boundary positions at subpixel precision on a fixed grid, avoiding the resolution dependence and staircase artifacts of binary occupancy maps.This continuous boundary representation is introduced in the multi-chart geometry-image setting.Source: §1 · pp. 2–3

2.1 Implicit and Explicit Representations for 3D Generation

  • UDF methods relax SDFs’ closed-surface bias but still require mesh extraction and post-processing, while explicit meshes preserve UV structure but challenge end-to-end modeling.The paper positions its structured 2D surface representation as retaining mesh structure while enabling differentiable, boundary-aware recovery.Source: §2.1 · pp. 3–4

2.2 Geometry-Image-Based Surface Representations; 2.3 Differentiable Iso-Surface Extraction

  • Geometry-image methods such as GIMDiffusion, Omages, and GarmageNet use binary masks, while differentiable extraction methods such as DMTet and FlexiCubes operate on cubic 3D grids; DiffGI uses a 2D UV TSDF.Prior geometry-image methods can require high-resolution grids or boundary heuristics, and 3D-grid extraction has cubic memory scaling without directly providing UV-aligned surfaces.Source: §2.2 · p. 4, §2.3 · pp. 4–5

3 Method

DiffGI uses a continuous TSDF geometry image and differentiable surface reconstruction to support geometric optimization, then compresses surfaces into a compact latent space for conditional 3D generation.

3.1 Continuous TSDF Geometry Image Representation

  • The continuous TSDF replaces binary occupancy with boundary distances, preserving subpixel boundary information at low resolution for high-fidelity reconstruction from compressed grids.Mesh-to-DiffGI preprocessing packs UV charts, samples a 1024×1024 position map, computes a truncated TSDF, and downsamples to 256×256×4.Source: §3.1 · p. 5

3.2 Differentiable Marching Squares (DMS)

  • Differentiable Marching Squares locates TSDF zero-crossings by linear interpolation, making reconstructed 3D vertex positions continuous functions of the TSDF and enabling geometric-loss gradients to flow to the 2D tensors.For ambiguous saddle configurations, the method treats patches as separate regions; vertex coordinates remain continuous, and vectorized extraction has O(N^2) complexity.Source: §3.2 · p. 6

3.3 Geometry-Aware Latent Compression (DiffGI-VAE)

  • DiffGI-VAE compresses 256×256×4 geometry images into a 32×32×4 latent space and uses differentiable normal rendering loss to preserve high-frequency geometric features.The reconstructed mesh is rendered with nvdiffrast and its normal map is compared with the ground-truth mesh, targeting details such as fabric wrinkles and sharp edges.Source: §3.3 · pp. 6–7

3.4 Latent Diffusion on DiffGI Latent Space

  • Conditional generation uses flow-matching latent diffusion, with DiT global attention for label- and image-conditioned tasks and a lightweight UNet-Tiny for occupancy-conditioned garment generation.The image-conditioned model injects DINOv2-Large semantic embeddings through cross-attention; garment generation conditions on sewing patterns or silhouettes.Source: §3.4 · pp. 7–8

4 Experiments

Experiments evaluate DiffGI on reconstruction, ablations, computational efficiency, and conditional 3D generation. Results report strong boundary and thin-shell performance alongside compact inference and generation capabilities.

4.2 Reconstruction Fidelity (DiffGI-VAE)

  • DiffGI-VAE outperforms baselines in CD, EMD, and JSD on ABO and GarmageSet despite compressing 256×256 geometry images into 32×32×4 latents; its NC is slightly lower on ABO but highest on GarmageSet.The reported ABO NC shortfall is attributed to information loss from 8× spatial compression, especially on flat furniture; Figure 3 shows sharper boundaries and better thin-shell detail.Source: §4.2 · pp. 9–10, Figure 3 · p. 9

4.3 Ablation Study: Differentiable Mesh Reconstruction and Normal Loss

  • Switching occupancy to TSDF without LNormal lowers CD from 1.503×10−3 to 0.595×10−3 and JSD from 4.539 × 10−3 to 2.169 × 10−3; adding LNormal to TSDF raises NC from 0.921 to 0.961.LNormal also improves TSDF CD and JSD; Occ. + LNormal reaches NC 0.947, above TSDF without LNormal, while TSDF without LNormal still outperforms it in CD, EMD, and JSD. The final TSDF + LNormal configuration mitigates noise and aliasing and sharpens boundaries.Source: Table 2 · p. 10, Figure 4 · p. 10, §4.3 · p. 11
  • Randomly initialized and SD1.5-initialized DiffGI-VAEs reach nearly identical GarmageSet fidelity: CD 0.47 vs. 0.46 and NC 0.96 for both.This supports attributing reconstruction gains to the DiffGI representation and differentiable reconstruction pipeline rather than SD1.5 VAE initialization.Source: §4.3 · p. 11

4.4 Computational Efficiency and Inference Performance

  • DiffGI image-conditioned generation takes ∼1.2 s on an RTX 4070 with 3.22 GB VRAM and ∼8.5 s on a MacBook M4 CPU, versus 52 seconds per sample for Omages.The reported comparison uses each method’s official recommended settings.Source: §4.4 · pp. 11–12, Table 3 · p. 12

4.5 Downstream Generative Tasks: Open-surface 3D Generation

  • On ABO label-conditioned generation, DiffGI reduces P-FID by 50% for Chair and 49% for Lamp; for Lamp, P-KID is 0.07 versus Omages’ 0.26.The passage notes elevated absolute Lamp P-FID due to limited training samples; qualitative results show smoother continuous structures than Omages’ broken thin frames.Source: §4.5 · p. 12, Table 4 · p. 12, Figure 5 · p. 13

Image-Conditioned Generation: Comparison with Foundation Models

  • For image-to-3D on GarmageSet, DiffGI achieves best CD (1.35 × 10−2), F1 (0.48), dH (8.42×10−2), and BCD (2.91×10−2), averaging 23K vertices.The comparison highlights practical advantages for thin-shell garment geometry, not universal superiority over general-purpose foundation models.Source: Image-Conditioned Generation:… · p. 12, Table 5 · p. 13
  • DiffGI generalizes from rendered normal-map training to real RGB front-view photographs in zero-shot WARDROBE results; pattern-silhouette conditioning produces natural draping and wrinkles, and local sleeve edits propagate to corresponding 3D geometry.Occupancy-conditioned generation uses a lightweight UNet-Tiny decoder, and local pattern edits preserve the remaining patterns.Source: Image-Conditioned Generation:… · pp. 13–14

5 Conclusion, Limitations, and Future Work

DiffGI integrates continuous geometry-image representations and differentiable mesh reconstruction into a learning pipeline, supporting compact 3D surface modeling and latent diffusion generation. The conclusion reports benchmark advantages alongside limitations in sharp-edge reconstruction, material generation, and cross-chart continuity.

Conclusion

  • DiffGI replaces binary occupancy maps with continuous 2D TSDF boundaries and differentiable Marching Squares, enabling subpixel-precise reconstruction and gradients from 3D surface losses to the 2D latent space.DiffGI-VAE compresses complex non-manifold surfaces into a 32×32×4 latent space with a geometry-aware normal rendering loss.Source: §5 · p. 14
  • DiffGI-VAE consistently outperforms prior geometry-image methods on garment and furniture benchmarks, while latent diffusion delivers competitive or superior generation quality with significantly less compute, including real-time consumer-hardware operation.Source: §5 · p. 15

Limitations

  • Linear interpolation can round extreme sharp edges; absent RGB/PBR generation and independently reconstructed UV charts can produce seams that challenge physical simulation despite not affecting reported metrics.Enforcing cross-chart boundary consistency is left for future work.Source: §5 · p. 15

Future Work

  • Future work targets feature-preserving iso-surfacing, joint high-resolution texture and material generation, and a two-stage pattern-layout and 3D-reconstruction pipeline for greater control of complex garments.Source: §5 · p. 15

— Supplementary Material — · S1 Resolution-Dependent Analysis: TSDF vs. Occupancy

The supplementary analysis isolates TSDF’s representation-level effects without VAE compression and finds its low-resolution advantage is strongest for complex boundaries. It also explains why continuous boundary encoding is more robust to VAE compression.

S1 Resolution-Dependent Analysis: TSDF vs. Occupancy

  • Without VAE compression, TSDF and Occupancy geometry images are compared at resolutions 64, 128, 256, 512, and 1024 to isolate representation effects.Source: S1 Resolution-Dependent Analysis: TSDF… · p. 19
  • TSDF achieves lower Hausdorff Distance than Occupancy at low resolutions, especially 64 and 128; both converge to near-identical error beyond 256.At 64 and 128, Occupancy exhibits staircase artifacts along thin structures and open boundaries; 256 is identified as sufficient reconstruction fidelity for the pipeline.Source: S1 Resolution-Dependent Analysis: TSDF… · p. 19
  • TSDF’s advantage is more pronounced on GarmageSet’s wrinkled garments and irregular open boundaries than on ABO’s mostly planar furniture.The analysis attributes the difference to greater boundary complexity.Source: S1 Resolution-Dependent Analysis: TSDF… · p. 19
  • TSDF distributes boundary positions as real-valued distances across neighbouring pixels, while binary Occupancy concentrates boundary information in one 0/1 transition pixel.Source: S1 Resolution-Dependent Analysis: TSDF… · p. 19
  • TSDF’s distributed boundary encoding makes it more robust to VAE compression, amplifying a modest high-resolution raw difference into a substantial compressed-performance gap.The passage relates this effect to the main paper’s Table 2 ablation.Source: S1 Resolution-Dependent Analysis: TSDF… · p. 19

S2 VAE Initialization: SD1.5 Pretraining vs. Training from Scratch

Training the DiffGI-VAE from scratch yields reconstruction fidelity comparable to SD1.5 initialization on GarmageSet, while SD1.5 mainly accelerates convergence. The comparison uses identical architecture and settings and DMS extraction.

S2 VAE Initialization: SD1.5 Pretraining vs. Training from Scratch

  • The comparable final metrics indicate that reconstruction gains stem from the TSDF representation rather than pretrained initialization.Both models attain essentially the same fidelity after training to convergence.Source: S2 VAE Initialization: SD1.5… · pp. 19–20
  • SD1.5 initialization reaches low loss in fewer steps, whereas the from-scratch model starts from noise and progressively recovers comparable reconstructions.The comparison trains the from-scratch VAE with identical architecture and all other settings, evaluating on GarmageSet under DMS extraction.Source: S2 VAE Initialization: SD1.5… · pp. 19–20
  • CD is 0.47 from scratch vs. 0.46 with SD1.5 initialization, while NC is 0.96 for both on GarmageSet under DMS extraction.Table S1 compares the two initialization settings using the same architecture.Source: S2 VAE Initialization: SD1.5… · p. 19, Table S1 · p. 20

S3 Impact of Geometric Data Augmentation and Model Scaling

On ABO, geometric data augmentation enables stable training of larger-capacity models, avoiding the quality sacrifice of shrinking the model.

S3 Impact of Geometric Data Augmentation and Model Scaling

  • After augmentation expands ABO training from 3,800 to 500K samples, DiT-Base converges reliably and generates better than DiT-Tiny; without augmentation, only Tiny is stable.With the original 3,800 samples, DiT-Base has unstable validation loss.Source: S3 Impact of Geometric Data… · pp. 20–21
  • UV-aligned geometry-image augmentation creates semantically consistent variations that improve stability and generalization without changing 3D semantics.The augmentation uses structural constraints of UV-aligned geometry images, including UV placement perturbation.Source: S3 Impact of Geometric Data… · pp. 20–21

S4 Extended Discussion on Saddle-Point Convention in DMS

DiffGI resolves ambiguous Marching Squares saddle cases by treating the two patches as separate regions, while preserving continuous boundary coordinates for backpropagation. UV padding makes problematic configurations unlikely at the operating resolution, though narrow bridges within a cell could be split.

S4 Extended Discussion on Saddle-Point Convention in DMS

  • For Marching Squares Cases 6 and 9, DiffGI deterministically treats the two saddle-point patches as separate independent regions.These cases correspond to topologically ambiguous saddle-point configurations.Source: S4 Extended Discussion on Saddle-Point… · p. 21
  • Inter-patch UV padding makes independent patches unlikely to share an ambiguous diagonal cell, although narrow within-cell bridges could be split; such cases are rare at 256×256 resolution with sufficient padding.Source: S4 Extended Discussion on Saddle-Point… · p. 21
  • The discrete saddle-case choice does not interrupt backpropagation because boundary-vertex coordinates remain continuous functions of TSDF values.This holds even in ambiguous cases.Source: S4 Extended Discussion on Saddle-Point… · pp. 21–22

S5 Additional Generation Results · S6 Clarifications on Evaluation Protocol

Additional results show DiffGI’s image-conditioned garment generation, transfer to unseen clothing images, and diversity across samples. Evaluation clarifications explain the mesh extraction protocols and why TRELLIS has no Boundary Chamfer Distance score.

S5 Additional Generation Results

  • On GarmageSet, TRELLIS produces overly thick meshes due to its watertight assumption, while GarmageNet shows boundary aliasing from binary occupancy.DiffGI instead reconstructs cleaner boundaries and finer fabric details.Source: S5 Additional Generation Results · p. 22
  • DiffGI produces ∼23K-vertex meshes versus 109–526K for the compared methods, with cleaner boundaries and thinner shells in additional GarmageSet results.The results also show finer fabric details, including collars, hems, and sleeve openings.Source: S5 Additional Generation Results · p. 22, Figure S5 · p. 23
  • On unseen WARDROBE images, DiffGI recovers thin shells and open boundaries and better matches input silhouettes than TRELLIS.WARDROBE was not used during training; these results indicate transfer of geometric priors learned by DiffGI-VAE.Source: S5 Additional Generation Results · p. 23, Figure S6 · p. 24
  • With a fixed input image, generated garments vary in UV chart layout, draping, and wrinkles while remaining consistent with the condition.The samples indicate diversity across both chart layouts and surface geometry rather than collapse to one mode.Source: S5 Additional Generation Results · p. 23

S6 Clarifications on Evaluation Protocol

  • DiffGI remains superior under both Omages-style tessellation and its native Differentiable Marching Squares extractor; GarmageNet’s native extraction takes about 5.1 s per mesh.The evaluation uses each method’s native mesh extractor except that GarmageNet is also reported with the Omages-style tessellation because its native extraction is a slow, multi-stage non-learning procedure.Source: S6 Clarifications on Evaluation Protocol · p. 23
  • TRELLIS has BCD marked N/A because its inherently watertight meshes have no open boundaries, making BCD inapplicable by definition.This explains the N/A entry in Table 5 of the main paper.Source: S6 Clarifications on Evaluation Protocol · p. 23

S7 Role of Differentiable Surface Extraction in Enabling LNormal · S8 Additional Implementation Details

The ablation shows complementary benefits from continuous TSDF representation and LNormal, with differentiable surface extraction enabling 3D supervision to reach the VAE. Additional implementation notes specify UV-plane TSDF construction and clarify evaluation scope.

S7 Role of Differentiable Surface Extraction in Enabling LNormal

  • LNormal can supervise the VAE encoder and decoder only when surface extraction is differentiable, using DMS for TSDF or differentiable Omages-style tessellation for occupancy variants.Non-differentiable mesh reconstruction blocks 3D-space supervision from reaching network weights.Source: S7 Role of Differentiable Surface… · p. 24
  • Without LNormal, switching from Occ. to TSDF reduces CD by more than half; adding LNormal raises NC from 0.921 to 0.961 and lowers CD to 0.461×10−3.The reported CD change without LNormal is 1.503×10−3 → 0.595×10−3; the complementary LNormal benefit depends on end-to-end differentiable extraction.Source: S7 Role of Differentiable Surface… · p. 24

S8 Additional Implementation Details

  • TSDF maps are computed entirely in the UV plane: packed charts are rasterized into a 1024×1024 binary mask, then a 2D Euclidean distance transform assigns signed pixel distances to chart contours.Distances are positive inside charts and negative outside; the computation does not use the 3-channel position map.Source: S8 Additional Implementation Details · p. 25
  • Pixels in empty background outside all charts receive well-defined negative TSDF values despite having no associated 3D position.The signed distance is measured within the UV plane rather than from a 3D point to the surface.Source: S8 Additional Implementation Details · p. 25
  • Before downsampling, the signed distance is clamped at 15 pixels.This follows the Mesh-to-DiffGI conversion procedure in Sec. 3.1, Step 4.Source: S8 Additional Implementation Details · p. 25
  • WARDROBE is used only for qualitative zero-shot evaluation, not for quantitative benchmarks on ABO or GarmageSet.The qualitative evaluation is shown in Fig. S6.Source: S8 Additional Implementation Details · p. 25

Related papers

Loading 2607.13365v1…