Source-linked AI summary
Dora: Sampling and Benchmarking for 3D Shape Variational Auto-Encoders
Rui Chen, Jianfeng Zhang, Yixun Liang, Guan Luo, Weiyu Li, Jiarui Liu, Xiu Li, Xiaoxiao Long, Jiashi Feng, Ping Tan
TL;DR
Uniform sampling can miss fine geometric details in 3D VAE training, limiting reconstruction. Dora-VAE combines sharp-edge sampling with dual cross-attention and introduces Dora-bench; experiments report improved reconstruction across shape-complexity levels.
Problem
Uniform point sampling can fail to capture geometrically salient details, limiting 3D VAE reconstruction and downstream generation quality.
Source: §1 · pp. 1–2Method
Dora-VAE combines Sharp Edge Sampling with dual cross-attention, and Dora-bench evaluates shapes by geometric complexity using Sharp Normal Error.
Source: §1 · p. 2Results
Dora-VAE significantly outperforms existing methods across varying levels of shape complexity.
Source: §6 · p. 8Takeaways & Limitations
The authors report that improved reconstruction enhances the quality of generated 3D shapes in downstream diffusion models.
Source: §1 · p. 3The stated detail-capture limitation applies to uniformly sampled Vecset methods, not specifically to Dora-VAE.
Source: §3.1 · p. 4
Abstract
from arXiv · showhide
Recent 3D content generation pipelines commonly employ Variational Autoencoders (VAEs) to encode shapes into compact latent representations for diffusion-based generation. However, the widely adopted uniform point sampling strategy in Shape VAE training often leads to a significant loss of geometric details, limiting the quality of shape reconstruction and downstream generation tasks. We present Dora-VAE, a novel approach that enhances VAE reconstruction through our proposed sharp edge sampling strategy and a dual cross-attention mechanism. By identifying and prioritizing regions with high geometric complexity during training, our method significantly improves the preservation of fine-grained shape features. Such sampling strategy and the dual attention mechanism enable the VAE to focus on crucial geometric details that are typically missed by uniform sampling approaches. To systematically evaluate VAE reconstruction quality, we additionally propose Dora-bench, a benchmark that quantifies shape complexity through the density of sharp edges, introducing a new metric focused on reconstruction accuracy at these salient geometric features. Extensive experiments on the Dora-bench demonstrate that Dora-VAE achieves comparable reconstruction quality to the state-of-the-art dense XCube-VAE while requiring a latent space at least 8$\times$ smaller (1,280 vs. > 10,000 codes).
1. Introduction
Uniform surface sampling can miss fine shape features when point budgets are limited, motivating Dora-VAE’s sharp-edge sampling and dual cross-attention. The introduction also proposes Dora-bench to evaluate reconstruction by shape complexity and salient geometric features.
1. Introduction
- Volume-based VAEs capture precise shape information but commonly use more than 10,000 latent tokens, while compact vector-set VAEs sample fewer points and can lose detail.The paper seeks better reconstruction while retaining compact representations.Source: §1 · p. 1
- Across sampling rates, SES-trained reconstruction consistently improves F-Score over uniform sampling while better preserving salient keyboard details.Figure 1 compares the sampling strategies and their reconstruction accuracy.Source: Figure 1 · p. 2, §1 · p. 2
- Uniform sampling can miss sharp features such as keyboard buttons even at higher sampling rates, limiting fine-detail capture and VAE reconstruction.The introduction connects this information loss to the quality of learned diffusion models.Source: §1 · pp. 1–2
- Sharp Edge Sampling prioritizes mesh edges with significant dihedral angles while retaining uniformly sampled points; Dora-VAE pairs SES with dual cross-attention for encoding salient and uniform regions.The paper motivates mesh-surface sampling as distinct from importance sampling methods that down-sample existing point clouds.Source: §1 · p. 2
- Dora-bench groups test shapes by geometric complexity and uses Sharp Normal Error to assess reconstruction of salient geometric features.It is presented as a more rigorous alternative to randomly selected test sets and general reconstruction metrics.Source: §1 · p. 2
- The introduction reports that Dora-VAE improves generated 3D shape quality when integrated into downstream diffusion models.The authors attribute this result to preserving geometric details with compact latent spaces.Source: §1 · pp. 2–3
2. Related work
Related work contrasts compact vector-set VAEs with detail-preserving but larger volume-based methods, and situates Dora among mesh sampling and 3D generation approaches.
2. Related work
- Volume-based methods such as XCube [45] preserve geometric detail but require > 10,000 latent tokens, whereas vector-set approaches [31] [58] [63] [65] [67] are compact but struggle with complex features.The paper identifies uniform sampling under point-budget constraints as a source of detail loss in vector-set methods.Source: §2 · p. 3
- Existing importance-sampling methods such as APES [57] sample point clouds, while Dora targets points sampled directly from mesh surfaces for shape VAE training.The related-work discussion frames mesh sampling as important for preserving complete geometric information.Source: §2 · p. 3
- 3D generation includes optimization-based methods [10, 30, 33, 34, 37, 43, 44, 48, 50, 54], large reconstruction models such as LRM [22], and native generative models [31] [58] [63] [65] [67].The paper notes that native models use a VAE followed by conditional latent diffusion, and that VAE reconstruction quality limits generated-shape quality.Source: §2 · p. 3
3. Method
The method builds on 3DShape2VecSet and augments its sampling and encoding to preserve salient geometry. It introduces Dora-bench, which stratifies test shapes by complexity and evaluates detail preservation with Sharp Normal Error.
3.1. Preliminary: 3DShape2VecSet
- 3DShape2VecSet uniformly samples Nd surface points, downsamples them with FPS, encodes sparse-to-dense cross-attention features into latent codes, then decodes occupancy values.Its encoder applies self-attention after cross-attention, and its decoder uses spatial query points.Source: §3.1 · p. 3
3.2. Dora-VAE
- Dora-VAE augments uniform mesh samples with salient points, computes separate attention features for each set, sums them, and predicts the occupancy field.Training follows 3DShape2VecSet and uses an occupancy-field loss.Source: §3.2 · p. 4
3.2.1. Sharp Edge Sampling (SES)
- SES identifies mesh edges whose dihedral angles exceed a threshold, samples salient points from those regions, and combines them with uniform points for surface coverage.It uses FPS when salient vertices are plentiful and interpolated edge samples when they are insufficient.Source: §3.2.1 · p. 4
3.2.2. Dual Cross Attention
- Dual cross-attention separately extracts features from downsampled uniform and salient points, then combines the results before occupancy prediction.The combined feature is processed by self-attention blocks.Source: §3.2.2 · pp. 4–5
3.3.1. Geometric Complexity-based Evaluation
- Dora-bench classifies test shapes into four geometric-complexity levels using salient-edge counts, drawing examples from GSO, ABO, Meta, and Objaverse.The levels span 0 < NΓ ≤ 5000 through NΓ > 50000.Source: §3.3.1 · p. 5
3.3.2. Sharp Normal Error (SNE)
- Sharp Normal Error measures mean squared error between reconstructed and ground-truth normal maps within masks of geometrically salient regions.The method identifies salient regions using Canny edge detection and dilates them to create the masks.Source: §3.3.2 · p. 5, Figure 4 · p. 5
4. Experiments
The experiments evaluate Dora-VAE across shape-complexity levels, comparing reconstruction quality and latent code length with existing methods. They also test the contributions of sharp edge sampling and dual cross-attention.
4.1. Implementation Details
- The VAE is trained on approximately 400,000 filtered Objaverse meshes using 32 A100 GPUs for two days, with batch size 2048 and learning rate 5e-5.Implementation also specifies Ndesired = 16384 and τ = 30 for sharp edge sampling.Source: §4.1 · pp. 5, 7
4.4. Quantitative Comparison
- With 256 latent codes, Dora-VAE’s CD is 3.356 versus 4.015 for XCube-VAE†; with 1,280 codes, its CD is 2.097, a 47.77% improvement over XCube-VAE†.The comparison spans Dora-bench complexity levels; the reported margins are larger on L3 and L4 shapes.Source: Table 1 · p. 7, §4.4 · p. 7
4.2. Evaluation Setting
- Reconstruction is evaluated with F-score, Chamfer Distance, and Sharp Normal Error, alongside Latent Code Length, across Dora-bench complexity levels.The evaluation compares Dora-VAE with XCube-VAE, a fine-tuned XCube-VAE†, and Craftsman-VAE.Source: §4.2 · p. 7, Table 1 · p. 7
4.3. Qualitative Comparison
- Dora-VAE is visually comparable to XCube-VAE on complex shapes while using 1,280 latent dimensions versus more than 10,000; Craftsman-VAE misses fine details.Figure 5 compares reconstructions across Dora-bench complexity levels.Source: Figure 5 · p. 6, §4.3 · p. 7
4.4. Quantitative Comparison
- On L4 shapes, Dora-VAE achieves SNE 1.579 versus 1.639 for XCube-VAE†, a 3.7% improvement in preserving geometric details.The qualitative comparison also reports that all methods are comparable at L1 and L2, while differences are clearer at L3 and L4.Source: Figure 5 · p. 6, §4.4 · p. 7, §4.3 · p. 7
4.5. Ablation Studies
- The full model outperforms variants without SES and DCA, and without DCA alone, supporting the contribution of both components.The variants are compared under the same training conditions; the first uses uniform Poisson-disk sampling with equal Nd, while the second retains SES and uses a single cross-attention.Source: §4.5 · pp. 7–8, Figure 6 · p. 8
5. Application: Single Image to 3D
The paper applies Dora-VAE to single-image 3D generation using a latent diffusion model. Under matched training conditions, generations from Dora-VAE preserve more geometric detail than those from Craftsman-VAE†.
5. Application: Single Image to 3D
- The study applies Dora-VAE to single-image 3D generation with a DiT-based latent diffusion model following CLAY.The model conditions on image features extracted from single-view images.Source: §5 · p. 8
- Dora-VAE produces generated shapes with more geometric detail than Craftsman-VAE† under the same experimental environment.Both diffusion models use the same 0.39B-parameter architecture, dataset, 32 A100 GPUs, and three days of training; XCube-VAE is excluded because its 10,000-dimensional codes are impractical for diffusion training.Source: Figure 7 · p. 8, §5 · p. 8
- The authors note that more training data and computational resources could further improve results, while this work focuses on validating the VAE’s capabilities.Source: §5 · p. 8
6. Conclusion
The conclusion presents Dora-VAE as a compact 3D shape compression and reconstruction model combining sharp edge sampling with dual cross-attention. It also reports improved reconstruction across shape-complexity levels and better geometric detail in single-image 3D generation.
6. Conclusion
- Dora-VAE combines sharp edge sampling and dual cross-attention to encode detail-rich point clouds for high-quality 3D shape compression and reconstruction.Source: §6 · p. 8
- Dora-bench categorizes shapes by geometric complexity and introduces Sharp Normal Error to assess preservation of fine geometric details.Source: §6 · p. 8
- The conclusion reports improved results across shape-complexity levels and better geometric detail in downstream single-image 3D generation.The supplementary material includes additional implementation details and VAE and generation comparisons, with limitations and future work discussed in Appendix D.Source: §6 · pp. 8–9
A. More implementation details
The supplementary methods describe Dora-bench construction, evaluation metrics, and implementation of the VAE and image-to-3D diffusion model. Training data and benchmark examples are drawn from multiple mesh sources, with four levels of geometric detail.
A. More implementation details
- Dora-bench combines ABO, GSO, Meta, and Objaverse test meshes into four detail levels, with approximately 800 samples per level.Level 4 draws predominantly from Objaverse because highly detailed models are scarce in the other listed datasets.Source: A. More implementation details · p. 9
- Evaluation uses Sharp Normal Error from normal maps rendered at 22 viewpoints, plus F-score for coverage and completeness and Chamfer Distance for point-cloud similarity.Source: A. More implementation details · p. 9
- The VAE uses 8 encoder and 16 decoder self-attention layers, with latent code length randomly selected from 256 to 1,280 during training.The image-to-3D diffusion model is DiT-based, has 0.39 billion parameters, and is trained for three days on 32 A100 GPUs.Source: A. More implementation details · p. 9
B. More comparison of VAE
On Dora-bench, Dora-VAE outperforms the other baselines, while 3DShape2VecSet underperforms across detail levels. XCube has rich visual detail, but its extracted geometry can deviate from ground truth.
B. More comparison of VAE
- Dora-VAE outperforms all other baselines in the qualitative comparisons, while XCube can show rich detail but deviate geometrically from ground truth.The authors attribute XCube's lower Chamfer distance and SNE performance despite visually appealing results to NKSR mesh-extraction quantization errors.Source: B. More comparison of VAE · p. 9
- 3DShape2VecSet [63] underperforms across every detail level, which the authors primarily attribute to its limited ShapeNet training data and weaker generalization.It was trained on ShapeNet [6], rather than the larger Objaverse [16] dataset used by the authors.Source: B. More comparison of VAE · p. 9
C. Image-to-3D Generation Comparison
In single-image 3D generation, Dora-VAE-based diffusion results show stronger geometric detail and fidelity than LRM-based approaches, and comparable geometric quality to Tripo v2.0. The reported training used three days on 32 A100 GPUs and about 400,000 samples.
C. Image-to-3D Generation Comparison
- The comparison excludes CLAY [65] and Rodin Gen-1 because of implementation unavailability and usage limitations at submission time.CRM [55] results use its official code and model; MeshFormer [36] and Tripo v2.0 results come from their demo and product website.Source: C. Image-to-3D Generation Comparison · pp. 9–10
- Dora-VAE-based generation achieves superior geometric detail and fidelity to LRM-based approaches MeshFormer [36] and CRM [55].The authors attribute those methods' limitations to their lack of explicit geometric constraints, which they say leads to unstable or lower-quality reconstructions.Source: C. Image-to-3D Generation Comparison · pp. 9–10
- Dora-VAE-based generation achieves geometric quality comparable to Tripo v2.0 with three days of training on 32 A100 GPUs and approximately 400,000 samples.The authors describe these as constrained resources and training data compared with commercial solutions.Source: C. Image-to-3D Generation Comparison · pp. 9–10
D. Limitations and Future Directions
The main stated limitation is maintaining reconstruction quality as the latent-token count is reduced further. The authors identify greater compression efficiency and more powerful image-to-3D diffusion models as future directions.
D. Limitations and Future Directions
- Maintaining high-quality reconstructions becomes challenging when reducing the latent-token count below the current setting.The authors identify this as the primary limitation of their approach.Source: D. Limitations and Future Directions · p. 10
- The authors plan to explore techniques that increase 3D VAE compression while maintaining reconstruction quality.They frame this as a direction that could potentially narrow the compression-efficiency gap between 2D and 3D methods.Source: D. Limitations and Future Directions · p. 10
- The authors envision developing more powerful image-to-3D diffusion models, expecting improved reconstruction to raise generation quality under the same training conditions.This is presented as a future direction, not a reported result.Source: D. Limitations and Future Directions · p. 10