3DILG: Irregular Latent Grids for 3D Generative Modeling

1 Introduction

• Figure 1 covers point-cloud reconstruction and probabilistic generation conditioned on high- or low-resolution images, point clouds, or object categories.Applications 2–5 are described as especially suited to autoregressive modeling.Source: Figure 1 · p. 1, §1 · p. 2
• Earlier global latents give promising autoencoding and reconstruction results but often miss shape details that are hard to recover.Local grids capture details through latents that mainly influence nearby surface regions; multiscale grids provide some performance improvement over basic grids.Source: §1 · pp. 1–2
• Irregular grids encode shapes as fixed-length sequences of position–latent tuples, allowing latent positions to adapt to important locations on the shape.The representation is designed for transformer architectures and scales better than grid-based, especially pyramid-based, representations as the number of latents grows.Source: §1 · p. 2
• Regular-grid 3D sequences are challenging for autoregressive transformers because their quadratic complexity makes even a 16 × 16 × 16 grid difficult to train on 4–8 GPUs.The paper contrasts artifacts from low-resolution regular-grid generative models with the clean surfaces produced by its representation.Source: §1 · p. 2
• The authors report improved point-cloud reconstruction and 3D generation over grid-based state-of-the-art methods, including image-, category-, and point-cloud-conditioned generation.The probabilistic model can generate multiple shapes for the same conditioning information.Source: §1 · p. 2

2.1 Neural shape representations

• Point-cloud methods range from MLPs and hierarchical networks to transformer backbones; this paper uses a standard transformer for shape autoencoding rather than developing a general point-cloud backbone.PCT and PT add modules beyond standard transformers.Source: §2.1 · p. 3
• Global latents are simple but poorly suited to surface details or complex topology; local grids assign latents to regions, while IF-Net concatenates grids at multiple resolutions.Other cited work learns latents at 3D positions.Source: §2.1 · p. 3

2.2 Generative models

• Autoregressive transformers have also been applied to 3D point clouds and polygonal meshes, as well as floor plans and indoor scenes.The cited examples include PointGrow and PolyGen.Source: §2.2 · p. 3
• Vector-quantized autoregressive models train an autoencoder to quantize bottleneck latents into discrete indices, then train an autoregressive model to predict those indices.VQGAN builds the autoregressive model with a transformer; the paper studies VQ for neural fields of 3D shapes.Source: §2.2 · p. 3
• Prior 3D VQVAE work includes point-cloud generation and voxel generation; low-resolution bottlenecks can miss details, while higher resolution complicates autoregressive transformer training.ShapeFormer sparsifies voxel grids but requires voxelized point clouds and dataset-dependent maximum sequence lengths; this paper processes point clouds directly with fixed-size latents and outputs neural fields.Source: §2.2 · p. 3

3 Shape Reconstruction

• The reconstruction pipeline turns a sampled surface point cloud into a predicted indicator function for the volume it represents.It comprises patch construction, patch information processing, and reconstruction.Source: §3 · pp. 3–4

3.1 Patch Construction

• Farthest Point Sampling selects points from the input cloud, and K-nearest neighbors form a size-K patch around each selected point.Each patch is projected to an embedding vector.Source: §3.1 · p. 4

3.2 Transformer

• A transformer maps patch and positional-embedding pairs to local latents, with an optional vector-quantization step replacing an intermediate output by its nearest dictionary vector.The patch coordinates supply positional embeddings; the quantized variant uses a dictionary.Source: §3.2 · pp. 4–5

3.3 Reconstruction

• For an arbitrary query point, the method interpolates nearby latents with a Nadaraya-Watson estimator and predicts the indicator using an MLP with sigmoid activation.The interpolation smoothness parameter β can be fixed or learned; training uses binary cross-entropy and a commitment-loss term, with λ = 0 indicating training without vector quantization.Source: §3.3 · p. 5

4 Autoregressive Generative Modeling

• Vector quantization compresses intermediate latents to dictionary indices, while coordinates are quantized to 8-bit integers to form a discrete shape representation.Each shape is represented by coordinate-latent tuples.Source: §4 · pp. 5–6

4.1 Unidirectional Transformer

• The unidirectional transformer sorts tuples by their three coordinates and predicts each coordinate and latent component autoregressively, continuing to use transformer blocks for component decoding.It generates quadruplets one at a time rather than flattening the full sequence.Source: §4.1 · p. 6

4.2 Bidirectional Transformer

• The bidirectional transformer predicts masked latents conditioned on all tuple coordinates, using visible latents as input and iteratively sampling multiple tokens during inference.Coordinates are converted to positional embeddings, and masked positions use a special token during training and inference.Source: §4.2 · p. 6

Results

• The evaluation uses ShapeNet-v2 and reports volumetric IoU, Chamfer-L1 distance, and F-Score, with results averaged across categories as well as shown for seven large-training-set categories.The selected categories are table, car, chair, airplane, sofa, rifle, and lamp; the reported reconstruction setup uses N = 2048 and K = 32.Source: §5 · p. 6, Table 2 · p. 7
• At M = 512, the method leads in most categories against IF-Net, while even M = 64 outperforms ConvONet; increasing M from 64 to 512 boosts performance.IF-Net has better F-Score in some categories, a metric the authors describe as saturated; vector quantization slightly harms performance.Source: §5 · p. 7, Table 2 · p. 7

Qualitative Comparison

• As M increases, reconstruction quality improves particularly for complex topologies; M = 64 outperforms ConvONet, and M = 128 is comparable with IF-Net.OccNet often misses complex structures, while ConvONet and IF-Net progressively improve on it through localized and multiscale latents.Source: §5 · p. 7

6 Generative Experiments

• The generative experiments cover image-conditioned reconstruction, category-conditioned generation across 55 ShapeNet categories, and point-cloud-conditioned generation using a bidirectional transformer.The point-cloud task differs from the first two by using the bidirectional model.Source: §6 · p. 7

6.1 Probabilistic Shape Reconstruction from a Single Image

• For 224 × 224 image conditioning, probabilistic reconstruction produces detailed meshes, whereas OccNet tends to produce blurred meshes.The unidirectional transformer is trained using 40 rendered views per shape.Source: §6.1 · p. 7, Figure 7 · p. 9
• With 16 × 16 inputs, the model can generate multiple plausible interpretations, including shapes with different topologies.The figure shows eight samples per input image.Source: §6.1 · p. 8, Figure 8 · p. 9

6.2 Category-Conditioned Generation

• For category conditioning, the Grid-83 baseline fails to generate high-quality shapes, which the authors attribute to insufficient representation of surface details.The comparison uses the same sequence length for Grid-83 and the proposed model.Source: §6.2 · p. 8

6.3 Point Cloud Conditioned Generation

• For point-cloud-conditioned generation, a bidirectional transformer takes the irregular-grid coordinates as input; the results are shown in Figure 11.Source: §6.3 · p. 9

7 Conclusion

• Irregular grids place latents at arbitrary positions rather than on regular grids, and their sparsity and adaptation to object shape help the representation scale to larger models.Source: §7 · p. 9
• The authors report improvements over alternative grid-based methods in point-cloud reconstruction and generation conditioned on images, categories, or point clouds.Source: §7 · p. 9
• Future work suggested by the authors includes shape completion, textured 3D shapes, and 3D scenes.Source: §7 · p. 9

• The representation is presented for image-, point-cloud-, and class-conditioned shape generation, as well as 3D shape reconstruction.Source: Broader impact · pp. 9–10
• Category-conditioned generation is compared with Grid-83 using FID for the seven largest ShapeNet categories.Source: Figure 9 · p. 9, Table 3 · p. 10
• The category-conditioned figure shows 100 samples each for bookshelf, bench, and chair.Source: Figure 10 · p. 10
• The point-cloud-conditioned generation figure shows eight decoding steps.Source: Figure 11 · p. 10
• The authors cite general societal impacts of generative modeling, but no important negative impact specific to this work that warrants immediate concern.Source: Broader impact · p. 10