Source-linked AI summary
ShaDDR: Interactive Example-Based Geometry and Texture Generation via 3D Shape Detailization and Differentiable Rendering
Qimin Chen, Zhiqin Chen, Hang Zhou, Hao Zhang
TL;DR
Existing 3D generators face limits in fine-grained control, speed, geometric detail, or texture generation. ShaDDR detailizes coarse voxel shapes and generates textures from exemplars; the paper reports improved geometric details and clean textures compared with prior works, alongside interactive modeling.
Problem
Existing 3D shape generators can lack fine-grained control, be slow, produce low-resolution geometry without details, or omit textures.
Source: §1 · p. 1Method
ShaDDR uses exemplar-conditioned multi-resolution voxel upsampling for geometry detailization and differentiable rendering for texture generation.
Source: §1 · pp. 1–2Results
ShaDDR produces high-resolution textured shapes with improved geometric details and clean textures compared with prior works.
Source: §5 · p. 7Takeaways & Limitations
Users can edit coarse voxels to control overall structure and interactively visualize detailed textured models.
Source: §1 · p. 2Local-patch plausibility enforcement can leave generated textures without global structure awareness and geometry with global inconsistencies.
Source: §5 · p. 7
Abstract
from arXiv · showhide
We present ShaDDR, an example-based deep generative neural network which produces a high-resolution textured 3D shape through geometry detailization and conditional texture generation applied to an input coarse voxel shape. Trained on a small set of detailed and textured exemplar shapes, our method learns to detailize the geometry via multi-resolution voxel upsampling and generate textures on voxel surfaces via differentiable rendering against exemplar texture images from a few views. The generation is interactive, taking less than 1 second to produce a 3D model with voxel resolutions up to 512^3. The generated shape preserves the overall structure of the input coarse voxel model, while the style of the generated geometric details and textures can be manipulated through learned latent codes. In the experiments, we show that our method can generate higher-resolution shapes with plausible and improved geometric details and clean textures compared to prior works. Furthermore, we showcase the ability of our method to learn geometric details and textures from shapes reconstructed from real-world photos. In addition, we have developed an interactive modeling application to demonstrate the generalizability of our method to various user inputs and the controllability it offers, allowing users to interactively sculpt a coarse voxel shape to define the overall structure of the detailized 3D shape. Code and data are available at https://github.com/qiminchen/ShaDDR.
1 INTRODUCTION
ShaDDR addresses limits in fine-grained control, generation speed, geometric detail, and texture coverage by detailizing coarse voxel shapes and generating textures from exemplars. Its multi-resolution geometry generation and differentiable-rendering texture generation support interactive, controllable modeling, including from real-world photographs.
1 INTRODUCTION
- Prior generators face limited fine-grained control, slow generation, low-resolution geometry, or omitted textures.DECOR-GAN enables coarse-voxel editing but uses single-resolution upsampling and generates geometry without textures.Source: §1 · p. 1
- ShaDDR generates high-resolution textured 3D shapes from coarse voxels by transferring geometric details and textures from an exemplar shape.The exemplar provides the geometry and texture styles for conditional generation.Source: §1 · p. 1
- ShaDDR generates a 256^3-voxel building model in less than 1 second while retaining fine-grained structural control through editing coarse voxels.The interactive modeling application lets users sculpt coarse voxels to define the overall structure of the detailed model.Source: §1 · p. 2
- Two-resolution voxel upsampling adds supervision at intermediate resolutions and improves finer-scale geometric details over DECOR-GAN.The texture generator synthesizes volumetric textures from a few exemplar views using differentiable rendering.Source: §1 · p. 2
- The texture exemplars can come from photographs of real-world objects, without requiring a predefined 3D model.The paper also presents detailed 3D shape generation with textures provided from real-world photographs.Source: §1 · p. 2
2 RELATED WORK
Related work spans 3D generative models, few-shot generation, and texture synthesis. ShaDDR adapts hierarchical voxel detailization and adds texture generation, while aiming to retain coarse-shape control and fast inference with few exemplars.
3D Generative Models
- 3D generative models span representations including point clouds, voxels, meshes, structural graphs, and neural implicit functions.The related work also notes 3D-aware image synthesis methods.Source: §2 · p. 3
- Other 3D generators trade off geometry, texture, shape control, topology flexibility, speed, or training-data requirements.Textured-shape generation and shape control are not jointly provided by several of the compared approaches; test-time optimization methods can take many minutes to hours.Source: §2 · p. 3
Few-shot Generative Models
- ShaDDR uses 8–16 detailed textured shapes for supervision and upsamples coarse voxels in less than 1 second at test time.The authors connect this interactive performance to high controllability for artists and casual users.Source: §2 · p. 3
- Few-shot 3D methods use local patch copying or learned patch distributions, while several exemplar-based generators produce shapes without control over their content.The cited approaches include classic patch synthesis, voxel-patch learning, and NeRF-based scene generation.Source: §2 · p. 3
3D Generative Models
- ShaDDR builds on DECOR-GAN [Chen et al. 2021] with hierarchical network structures for higher-resolution geometric details and adds detailed textures to output shapes.The related-work discussion positions this combination as making the shapes more readily usable in real applications.Source: §2 · p. 3
Texture Synthesis
- 3D texture methods represent surface appearance with UV-mapped images, colored voxels, face convolutions, projected views, or neural fields.ShaDDR uses a colored-voxel grid supervised through differentiable rendering, where only surface voxels receive gradients, and a texture latent code controls generated textures.Source: §2 · p. 3
3 METHOD
ShaDDR turns coarse voxel shapes into detailed, textured shapes by conditioning geometry and texture generation on learned style codes. Its method uses multi-resolution geometry generation and differentiable rendering to train texture synthesis from rendered views.
3 METHOD
- ShaDDR represents output geometry as an occupancy voxel grid and texture as a 3D RGB grid, generating style-conditioned shapes that preserve the coarse input structure.The method uses a multi-resolution 3D voxel GAN for geometry and GANs on rendered 2D images for texture generation.Source: §3 · pp. 3–4
3.1 Multi-resolution geometry detailization
- To mitigate degradation from large upsampling factors, ShaDDR trains intermediate- and final-resolution voxel outputs with adversarial supervision and reconstruction losses at both resolutions.The generator produces outputs at (K/2)^3 and K^3, and the losses combine GAN and reconstruction terms at each scale.Source: §3.1 · p. 4
3.2 Texture generation via differentiable rendering
- Texture generation renders surface-voxel colors into images and uses 2D GANs, avoiding the ill-defined ambient-space colors a volumetric 3D GAN would process.The texture generator predicts a K^3 RGB grid; rendered views are also supervised with reconstruction losses.Source: §3.2 · pp. 4–5
3.3 Implementation details
- 8× geometry upsampling takes less than 1 second per content–style pair at inference; geometry and texture training take 12–24 and 18–36 hours, respectively, on one RTX 3090 Ti.Separate models are trained for each category; geometry training precedes texture training, with geometry weights fixed during texture training.Source: §3.3 · p. 5
3.4 Data preparation
- Training shapes begin as 512^3 occupancy voxels; geometry is smoothed with a Gaussian filter (σ=1.0), and non-surface colors are nearest-neighbor inpainted for the volumetric-texture ablation.Rendered view colors are projected back onto surface voxels to create colored training shapes.Source: §3.4 · p. 5
4 RESULTS AND EVALUATION
ShaDDR produces detailed geometry and textures from coarse voxel shapes, with evaluations spanning multiple object categories and comparisons of geometry and texture methods. The results also demonstrate style transfer from real-photo reconstructions and interactive editing of user-designed shapes.
4 RESULTS AND EVALUATION
- The evaluation covers cars, airplanes, chairs, buildings, and plants, using geometry metrics and rendered-image texture metrics to assess generated results.The content and style sets vary by category, and the method is also tested on plants.Source: §4 · p. 5
4.1 Geometry detailization and texture generation
- Across cars, airplanes, buildings, chairs, and plants, ShaDDR generates high-resolution geometry with fine details and textures such as car stripes and building windows.Source: §4.1 · p. 5
4.2 Ablation study
- On chair geometry, multi-resolution ShaDDR outperforms DECOR-GAN and DECOR-GAN-up on most metrics, while refining global structure and local details across intermediate and final resolutions.DECOR-GAN-up can produce locally distinctive but globally implausible geometry; the final output refines features such as sharp edges and smooth surfaces.Source: §4.2 · p. 6, Figure 4 · p. 6
- Rendering-based texture generation produces cleaner, sharper local regions and outperforms both 3D texture baselines on all metrics except FID-all for chairs.The comparison uses the same generator architecture, replacing the texture discriminator’s 2D convolutions with 3D convolutions for the baselines.Source: §4.2 · p. 6, Table 2 · p. 6, Figure 5 · p. 6
4.3 Application
- Detailed shapes reconstructed from real-world car photos serve as style shapes for generating geometry and textures on new coarse content shapes.The car meshes are reconstructed with NeuS and post-processed by manually removing the ground and closing holes.Source: §4.3 · pp. 6–7, Figure 7 · p. 7
- The method handles user-designed coarse shapes without further training or fine-tuning, and its sub-second forward pass supports real-time interactive editing.The interface lets users edit coarse voxels, choose a style, and view the detailed textured shape.Source: §4.3 · p. 7
5 CONCLUSIONS
ShaDDR detailizes coarse voxel shapes and generates textures, producing detailed textured models while allowing users to edit the coarse structure. The conclusion also identifies limitations in global consistency and geometry-texture alignment, and directions for future work.
5 CONCLUSIONS
- ShaDDR upsamples coarse voxel models into textured, geometrically detailed shapes with improved details and clean textures compared with prior work, including results trained on synthetic and real data.Its interface lets users edit the coarse content model and view the detailed result in real time.Source: §5 · p. 7
- Local-patch plausibility can miss global texture structure and produce globally inconsistent geometry; the method also lacks explicit geometry-texture alignment enforcement.The authors say misalignment was rare in their experiments, potentially because texture discriminators ensure multi-view plausibility.Source: §5 · p. 7
- Future work includes learning geometry and texture without 3D supervision and adding local control to assign styles to selected coarse-shape regions.Source: §5 · p. 7
Supplementary Material
The supplementary material provides implementation and data-preparation details, along with additional qualitative results.
Supplementary Material
- The supplement covers network architectures, loss functions, implementation settings, data preparation, and additional qualitative results.Source: Supplementary Material · p. 12
6 NETWORK ARCHITECTURE
The supplement specifies separate geometry and texture generators and discriminators, including their layers, style inputs, and category-dependent receptive fields.
6 NETWORK ARCHITECTURE
- The geometry generator uses a five-layer 3D-convolution backbone and three resolution-doubling upsampling layers, with an 8-dimensional trainable geometry-style module concatenated at specified layers.The second upsampling layer also has an intermediate-resolution output layer.Source: §6 · p. 12
- The texture generator uses a similar five-layer 3D-convolution backbone and three resolution-doubling upsampling layers, with geometry-style and trainable texture-style modules concatenated at specified layers.The geometry-style module is pretrained; the texture-style module is trainable and 8-dimensional.Source: §6 · p. 12
- The geometry discriminator has five 3D-convolution/leaky-ReLU layers followed by a 3D-convolution/Sigmoid layer, with receptive fields of 36^3 for cars and planes and 18^3 for chairs, buildings, and plants.Source: §6 · p. 12
- The texture discriminator is described with five 2D-convolution/leaky-ReLU layers followed by a 3D-convolution/Sigmoid layer, using 36 × 36 receptive fields for cars and planes and 18 × 18 for chairs, buildings, and plants.Source: §6 · p. 12
7 LOSS FUNCTION
The loss-function supplement describes multi-resolution, multi-branch discriminator losses, generator outputs and reconstruction losses, and masks used to focus generation and discrimination on relevant regions.
7 LOSS FUNCTION
- Content masks constrain geometry generation to valid regions and require occupied coarse-shape voxels to produce output in corresponding areas; style-shape masks are also defined for discriminator use.The content generator mask is computed by dilating and upsampling the coarse input; discriminator masks are computed from upsampled content or style shapes.Source: §7 · p. 12
- The geometry and texture generators’ outputs are used respectively as discriminator inputs and as geometry and texture inputs for rendering.The supplement reports that using the geometry-detailization values for γ1 and γ2 also suffices for texture generation.Source: §7 · pp. 12–13
- The discriminator loss uses global and style branches across K^3 and (K/2)^3 resolutions; the detailed setup splits each discriminator into N style branches plus one global branch to prevent model collapse.The global geometry branch evaluates patches at resolution K^3, while a style branch is associated with each style.Source: §7 · p. 12
- The geometry generator outputs an upsampled shape at resolution K^3, and its overall loss includes the reconstruction loss described in the main paper.Source: §7 · pp. 12–13
- The texture generator outputs a volumetric color grid; its loss uses rendered views from geometry and color inputs and includes the reconstruction loss.For a view, rendering derives a mask and depth from the geometry voxels, then gathers colors from the texture voxels.Source: §7 · pp. 12–13
8 IMPLEMENTATION DETAILS
Implementation details cover volumetric texture preparation, training settings, and cropping used to reduce memory footprint and training time.
8 IMPLEMENTATION DETAILS
- Surface-voxel colors are assigned from rendered views according to the voxel normal’s angle to horizontal or vertical planes.The method records surface voxels and selects a corresponding view’s pixel value based on the normal direction.Source: §8 · p. 13
- Training uses category-specific models and α and β settings, with γ=0.5, batch size 1, and learning rate 0.0001.Both geometry and texture models train for 20 epochs on one RTX 3090 Ti; category-dependent training takes 12–24 hours for geometry and 18–36 hours for texture.Source: §8 · p. 13
- Shapes are cropped to dilated bounding boxes, discarding unoccupied voxels to reduce memory use and speed training.The style-shape geometry and texture voxels are cropped as well.Source: §8 · p. 13
9 EVALUATION METRICS
The section defines geometry and texture evaluation metrics and describes the evaluation samples and quantitative comparison tables.
9 EVALUATION METRICS
- Strict-IoU measures agreement between downsampled output and input coarse voxels; Loose-IoU excludes voxels in the input’s dilated region.The downsampled output ideally matches the coarse input, while the dilated generator mask motivates the Loose-IoU variant.Source: §9 · pp. 13, 15
- The tables report quantitative comparisons of multi-resolution geometry upsampling with two baselines for car, plane, and building categories.Geometry metrics use 20 test content shapes and 16 styles per content shape; texture metrics use 20 content shapes.Source: Table 3 · p. 14, Table 4 · p. 14, Table 5 · p. 14
- LP-IoU and LP-F-score measure the share of output patches similar to at least one style-shape patch, using a 0.95 threshold; Div-IoU and Div-F-score measure style diversity.The diversity measures average patch matches across styles to remove style bias.Source: §9 · p. 15
- Cls-score is the mean accuracy of a ResNet trained to distinguish real from generated shapes using rendered image patches.The evaluation uses 24 rendered views per sample, with 64^2 random crops.Source: §9 · p. 15
- FID-all compares generated textures with all training textures, while FID-style compares them with the texture providing the style; LPIPS-style measures local patch similarity to that style.FID-all and FID-style have a noticeable scale difference.Source: §9 · p. 15
10 APPLICATION
The application lets users edit coarse voxel templates, select a style, and view detailed textured outputs in real time. The accompanying examples show results across object categories and include car shapes reconstructed from real-world images.
10 APPLICATION
- Users add or remove coarse voxels, choose a style, and visualize the detailed textured result in real time.The workflow passes the edited template to a pretrained model, extracts a surface with Marching Cubes, assigns texture colors to vertices, and displays the result.Source: §10 · p. 15
- The GUI provides mouse controls for voxel editing and camera movement, brush-size shortcuts, eight style choices, and editing/viewing mode switching.The controls also include camera zoom and repositioning.Source: §10 · p. 15
- Car examples use 64^3 coarse inputs and 512^3 outputs; airplane examples use the same resolutions.The figure captions show style shapes and coarse inputs for these category results.Source: Figure 15 · p. 16, Figure 16 · p. 17, Figure 19 · p. 20
- Car detailization comparisons show DECOR-GAN-up and multi-resolution outputs, while separate car examples condition generation on cars reconstructed from real-world images.The comparison figures also cover airplane, chair, and building geometry detailization.Source: Figure 17 · p. 18, Figure 18 · p. 19, Figure 21 · p. 22
- Chair and building examples use 32^3 coarse inputs and 256^3 outputs.The captions show style shapes and coarse inputs for both categories.Source: Figure 22 · p. 23, Figure 23 · p. 24, Figure 25 · p. 26