AutoDecoding Latent 3D Diffusion Models

Motivation

• 2D images and monocular videos dominate available data, while suitable 3D representations and 3D supervision remain scarce, restricting existing generative methods.Articulated objects further require representations that support deformation.Source: §1 · pp. 1–2

Scope and evaluation

• The framework handles rigid and articulated objects, supports supervised, estimated, or learned camera poses, and scales from single-category synthetic data to hundreds of thousands of real objects.Experiments include multi-view synthetic objects, MVImgNet [85], and approximately 44K CelebV-Text [83] human-motion sequences.Source: §1 · p. 2

Motivation

• Prior two-stage diffusion methods require rich latent autoencoders, but 3D autoencoders typically need scarce 3D data and therefore rely on restricted synthetic domains.The paper contrasts this limitation with ShapeNet-based approaches such as DiffRF [45] and SDFusion [10].Source: §1 · p. 2

Contribution

• The proposed volumetric autodecoder assigns each training object a 1D embedding and learns 3D representations from 2D observations using rendering consistency, without 3D supervision.Following UVA [63], the representation also supports articulated parts for non-rigid objects.Source: §1 · p. 2
• Robust normalization and denormalization enable diffusion at intermediate autodecoder resolutions, with compact low-resolution features regularizing geometry while deeper layers mainly upsample.This avoids exhaustive layer-by-layer searches and supports sparse observations.Source: §1 · p. 2

2.1 Neural Rendering for 3D Generation

• Neural-rendering generators often have limited multi-view consistency, narrow category coverage, or difficulty generalizing to multi-category image datasets.These limitations are reported for prior GAN-based 3D methods [61] [47] [50] [48] [81].Source: §2.1 · p. 2
• pi-GAN requires an accurate camera-pose distribution, while EG3D and EpiGRAF require keypoint-based pose estimates, limiting viewpoint range.The cited methods are pi-GAN (Chan et al., 2021 [5]), EG3D (Chan et al., 2022 [6]), and EpiGRAF (Skorokhodov et al. [66]).Source: §2.1 · p. 3
• Existing approaches primarily target single categories and struggle with deformable articulated objects; 3DGP [67] broadens categories but remains limited by monocular depth and front-facing scenes.The paper claims its method covers rigid and deformable objects across wider viewpoints.Source: §2.1 · p. 3

2.2 Denoising Diffusion Modeling

• Standard diffusion methods generally assume fully observable target-domain samples, whereas real-world 3D content is mostly observed through partial 2D images.DDPMs [68] [24] learn denoising from progressively corrupted data.Source: §2.2 · p. 3
• DiffRF [45] reconstructs per-object NeRF volumes but is limited by reconstruction cost and low diffusion resolution, while 3D autoencoder methods require ground-truth 3D data and focus mainly on shapes.The paper contrasts its latent-space approach with expensive 2D-diffusion-guided optimization methods [56] [39] [9].Source: §2.2 · p. 3

3 Methodology

• The two-stage pipeline learns an autodecoder from 2D images, freezes it, and trains diffusion in the compact latent space produced by G1 before decoding generated volumes with G2.At inference, diffusion samples replace the object-specific G1 embedding.Source: §3 · p. 3, Figure 1 · p. 4

3.1 Autodecoder architecture

• The canonical representation uses density and RGB voxel grids in a canonical pose, with trilinear interpolation and volumetric rendering separating object structure from camera pose.Direct voxel decoding avoids per-point MLP evaluation required by feature volumes or tri-planes.Source: §3.1 · p. 4
• To support large and diverse datasets, the decoder increases embedding length from 64 to 1024, uses four residual blocks per resolution, and adds self-attention at resolutions 8^3 and 16^3.These changes address reconstruction capacity for approximately 300K objects.Source: §3.1 · p. 4

3.2 Autodecoder Training

• Autodecoder training minimizes rendered-image reconstruction error and adds occupancy-mask supervision, while rendering four views per object reduces batch variance without significant overhead.For articulated objects, learned keypoints, differentiable PnP, and volumetric linear blend skinning align and deform rigid components.Source: §3.2 · pp. 4–5

3.3 Latent 3D Diffusion

• The latent diffusion model extends 2D convolutions and attention into 3D, with cross-attention for text conditioning and diffusion performed at selected decoder resolutions.The paper evaluates resolutions from 4^3 to 16^3 and reports that feature processing is central to using this latent space.Source: §3 · p. 3, §3.3 · p. 5
• Because latent features are bell-shaped but heavy-tailed, robust normalization uses the median and a normalized interquartile range before diffusion, then reverses the transformation during inference.This supplies a common feature range across datasets and autodecoders without imposing a prior.Source: §3.3 · pp. 5–6

4.1 Datasets and Data Processing

• The evaluation covers synthetic renderings, approximately 300K Objaverse objects, 206,990 usable MVImgNet objects, and approximately 44.4K CelebV-Text videos.The datasets include multi-view images, real-world object videos, and in-the-wild human videos.Source: §4.1 · p. 6

4.2 Unconditional Image Generation

• Our method significantly outperforms state-of-the-art methods on ABO Tables and achieves better or comparable results on PhotoShape Chairs using FID and KID.The comparison includes both GAN-based and diffusion-based approaches.Source: §4.2 · p. 7, Table 1 · p. 7
• On MVImgNet, CelebV-Text, and Objaverse, diffusion produces substantially higher-fidelity geometry and texture than direct latent sampling, with lower FID and KID that generally improve with more steps.Directly sampled identities show less sharp normals and depth, indicating spurious density in sampled volumes.Source: §4.2 · p. 7, Table 3 · p. 7

4.3 Autodecoder Ablation

• The final autodecoder significantly improves PSNR and LPIPS over the base architecture, while larger embeddings and multi-frame training provide important quality gains.Removing self-attention slightly improves PSNR and LPIPS on this smaller dataset, whereas reduced depth or embedding size degrades quality.Source: Table 2 · p. 7, §4.3 · p. 8

4.4 Diffusion Ablation

• Diffusion at resolution 83 gives the best quality–efficiency balance: 64 steps take approximately 8 seconds, while 32 steps give reasonable results in approximately 4 seconds.Lower-resolution layers can overfit, whereas higher-resolution diffusion requires substantial resources; sampling-step effects are smaller and more variable.Source: §4.4 · p. 8, Figure 3 · p. 8

4.5 Conditional Image Generation

• Text-conditioned models generate objects with reasonable geometry that generally follows prompts on MVImgNet and Objaverse, although some details may be missing.The authors attribute missing or inconsistent details partly to hallucinations in the captioning system and suggest better captioning systems as a future improvement.Source: §4.5 · p. 8

• Generated samples include text-conditioned multi-view objects with normals and depth from MVImgNet monocular videos and Objaverse rendered images.Ground-truth captions are generated by MiniGPT-4 [87].Source: Figure 4 · p. 9, Figure 5 · p. 9
• The approach learns diverse, view-consistent 3D geometry and appearance from 2D supervision for conditional and unconditional generation across synthetic, real-world, static, and articulated datasets.It supports both multi-view images and monocular videos and handles rigid as well as moving objects.Source: §5 · pp. 8–9
• Training is limited to datasets containing multi-view or video sequences for each object, so single-image datasets are unsupported.The demonstrated setting also focuses on one key person or object rather than complex multi-object scenes.Source: §5 · p. 9
• The method shares generative-model risks involving misleading content, dataset bias, privacy, and training energy, despite reducing reliance on large-scale 3D-data curation.These concerns are especially relevant when generating human content or recovering training-subject identities.Source: §5 · p. 10

• COV measures generated-set diversity by the fraction of reference point clouds covered by nearest generated neighbors.The metric is computed over generated and reference point-cloud sets.Source: §A.1 · p. 15
• MMD measures overall sample quality through the average distance from each generated point cloud to its closest reference neighbor.Chamfer Distance is one possible distance metric used for point-cloud comparison.Source: §A.1 · p. 15
• Our method achieves comparable or superior geometry to DiffRF while substantially improving perceptual quality and outperforming GAN-based approaches across perceptual and geometric metrics.The comparison uses PhotoShape Chairs and ABO Tables; Table 4 reports the quantitative results.Source: §A.1 · p. 15
• Qualitative comparisons show more coherent and complete shapes, realistic detailed textures, and more consistent image synthesis than DiffRF.Examples include avoiding missing features such as legs.Source: §A.1 · p. 15

• The added foreground loss measures how well depicted objects are separated from the background during rendering.It supplements the primary autodecoder reconstruction loss for difficult datasets.Source: §A.2 · p. 15
• Without foreground supervision, CelebV-Text reconstructions lose target-appearance fidelity and recover insufficient geometry, as indicated by opacity and surface normals.The comparison is performed after the autodecoder sees approximately 9 million training frames.Source: §A.2 · p. 15
• CelebV-Text contains diverse foreground shapes and textures against a common background color, making foreground-mask supervision important for learning object shape.Both displayed examples are trained for approximately 9 million frames.Source: Figure 6 · p. 16

• Supplementary videos show consistent novel-view synthesis for rigid multi-category objects and animated articulated objects generated by the method.The supplementary results include both conditional and unconditional generation.Source: §A.3 · p. 16

B.1 Volumetric Autodecoder

• Volumetric rendering integrates sampled density and color along camera rays to produce target images and occupancy supervision.The renderer uses camera parameters and decoded radiance-field volumes, sampling 128 points per ray during training and inference.Source: §B.1 · p. 16
• Articulated objects are decomposed into locally rigid regions whose poses are estimated from shared 3D keypoints and per-frame 2D projections.This provides canonical alignment and supports pose variation across frames and motion transfer between subjects.Source: §B.1 · p. 18
• Rigid-object pose handling uses known or estimated cameras because learned keypoint-based estimation is less reliable for sparse, randomly posed multi-view data.For continuous human videos, poses are learned during training; for rigid datasets, the method uses ground-truth or estimated camera poses.Source: §B.1 · p. 18
• The non-rigid decoder additionally produces an LBS-weight volume assigning each canonical location to the articulated components.Volumetric linear blend skinning maps canonical points to deformed positions using learned spatial correspondences.Source: §B.1 · p. 18
• The autodecoder progressively upsamples a 1D embedding into volumetric features, applying attention at intermediate resolutions before producing density, color, and optional part outputs.The diffusion input is the 8^3 latent volume with 256 features per cell, while the standard final representation is 64^3.Source: §B.1 · pp. 18–19
• Robust statistics normalize extreme-valued latent features before diffusion and de-normalize outputs before the decoder’s upsampling layers.The feature distribution is normalized to [−1, 1] because classic mean-and-standard-deviation normalization causes diffusion convergence failure.Source: Figure 9 · p. 19

B.2 Latent 3D Diffusion

• The diffusion model adapts ADM with preconditioning, 3D convolutions, 3D normalization, and 3D attention for latent volumetric features.Cross-attention is likewise extended to the 3D latent space for conditioning experiments.Source: §B.2 · p. 19
• Robust normalization centers features by the median and scales them using 0.7413 × IQR, preventing diffusion-training convergence failure caused by extreme values.The normalized features are mapped to [−1, 1] before diffusion and de-normalized before decoding.Source: §B.2 · p. 19
• The 8^3 latent-resolution model provides the best quality–speed trade-off among tested resolutions and is suited to large-scale training.The 4^3 model often fails to converge, whereas 16^3 models can exhibit spurious density values when under-trained.Source: §B.2 · p. 20, Figure 10 · p. 20

B.3 Hash Embedding

• Hash embeddings replace a separate stored embedding for every object with concatenated codebook vectors selected deterministically from each object index.The construction yields exponentially more possible embeddings than the number of stored vectors.Source: §B.3 · p. 20
• For Objaverse, hashing saves approximately 800MB of GPU memory with negligible impact on training and synthesis speed or quality.The scheme reduces model storage requirements by approximately 75% for this dataset.Source: §B.3 · p. 21
• Scaling beyond the evaluated datasets may require additional handling because per-object embeddings become intractable as dataset size grows.The hash scheme reduces storage requirements, but the authors identify larger datasets as a remaining scaling boundary.Source: §B.3 · pp. 20–21

C.1 Dataset Filtering

• CelebV-Text videos are filtered by continuity, foreground coverage, remaining frame count, file size, and availability before training.The procedure removes fade-in/out segments, low-foreground frames, very short videos, and low-quality files.Source: §C.1 · p. 21
• MVImgNet samples with failed detection, inconsistent segmentation, or unusually high autodecoder fitting loss are removed.Grounded Segment Anything supplies masks, while Grounding DINO detects bounding boxes before fitting-based filtering.Source: §C.1 · p. 21
• Objaverse models are filtered for texture, material, appearance, and polygon quality, and foreground supervision is omitted for its simple monochrome-background renders.Foreground supervision remains essential for more complex real-image datasets with motion and backgrounds.Source: §C.1 · p. 21

C.2 Additional Details

• The implementation uses PyTorch, PyTorch Lightning, Torch Fidelity, Adam optimization, and up to eight NVIDIA A100 40GB GPUs per node.Larger experiments use up to eight nodes in parallel, while FID and KID are computed with Torch Fidelity.Source: §C.2 · pp. 21–22
• CelebV-Text outputs contain 14 channels per voxel cell: three color, one density, and ten part-assignment channels.The architecture table distinguishes these channels from the outputs used for other datasets.Source: Table 5 · p. 22
• Text-conditioned generation uses MiniGPT4 captions for MVImgNet and Objaverse, followed by 1024-dimensional T5 text embeddings with fixed length 32.Captions are prompted to describe each object’s shape and color in one concise sentence.Source: §C.2 · p. 22