Memorization in 3D Shape Generation: An Empirical Study

1. Introduction

• The study examines whether high-fidelity 3D models rely on memorizing training shapes, a concern arising as AI-powered 3D synthesis scales.Feed-forward diffusion models and compact latent representations have enabled fast, scalable synthesis of high-fidelity assets.Source: §1 · p. 1
• Memorization in 3D generation remains understudied despite risks of training-data leakage and failure to generalize to unseen geometries.Prior empirical work largely focused on image generation, while 3D introduces distinct geometric and modeling factors.Source: §1 · p. 1
• The framework assesses memorization in 3D generative models, addressing the lack of a standard novelty metric beyond fidelity and distributional similarity.The paper uses retrieval metrics benchmarked against human judgments and pairs its memorization measure with Fréchet Distance to separate memorization from generation quality.Source: §1 · pp. 1–2
• Smaller-dataset models show clear memorization with near-exact replicas, while recent large-scale conditional models show limited memorization and stronger generalization.The paper then studies data and modeling factors with controlled Vecset experiments.Source: §1 · p. 2
• Rendered images are more likely to be memorized than 3D shapes; greater semantic diversity and finer-grained conditioning are associated with stronger memorization.Memorization peaks at a moderate classifier-free guidance scale, while longer Vecsets and simple rotation augmentation reduce it.Source: §1 · p. 2

Shape generation in 3D vision

• Vecset provides a compact latent vector-set space that makes diffusion models easier to train and scale, and many recent state-of-the-art 3D models use it.3D shape methods also span diverse representations, structures, and conditioning modalities.Source: §2 · p. 2

Memorization in generative models

• Prior memorization studies examine correlations with dataset scale, model capacity, and training dynamics, while more mechanistic diffusion work links novelty to inductive biases and selective score fitting.The literature is motivated in part by copyright infringement concerns.Source: §2 · p. 2

3D Object Retrieval (3DOR)

• 3D retrieval uses geometric or rendering-based distances, point-cloud encoders, and multi-view encoders, with applications across AR/VR, gaming, and medicine.Multi-view encoders are often trained on relatively small rendered-image datasets.Source: §2 · p. 2

3. Methodology

• The methodology combines nearest-training-shape retrieval, a Mann-Whitney z-score, and a quality measure to distinguish memorization from low-quality generation.The retrieval benchmark uses shapes from chair, car, airplane, and table categories and excludes cases where no metric retrieves a visually similar shape.Source: §3 · p. 2, §3.1 · p. 2

3.1. Distance Metrics

• LFD achieves the highest top-1 retrieval accuracy among seven metrics on 133 generated shapes from four ShapeNet categories.For each generated shape, metrics retrieve a nearest training shape; manual inspection determines whether it is visually near-identical.Source: §3.1 · pp. 2–3, Table 1 · p. 3

3.2. Memorization Metric

• ZU measures how generated and held-out samples rank by distance to their nearest training shapes; ZU < 0 indicates generated samples are closer to training data on average.ZU is comparable across models when test and generated set sizes are fixed, but ZU ≥ 0 can reflect either generalization or low-quality samples far from training shapes.Source: §3.2 · p. 3

3.3. Evaluation Framework

• Fréchet Distance on Uni3D features serves as a generation-quality indicator, with training FD and test FD computed against their respective reference sets.Training FD can be lowered by reproducing training data; the framework therefore compares ZU among models with similar test FD to focus on memorization rather than sample quality.Source: §3.3 · p. 3

4. Evaluating Existing Models

• The evaluated models span single-category ShapeNet models, full-ShapeNet models, and Trellis trained on 500K samples, with |Ptest| = |Q| = 100.The authors assume these models already generate high-quality shapes and evaluate memorization only.Source: §4 · p. 3
• On ShapeNet chairs, LAS-Diffusion and Wavelet Generation produce training replicas, while 3DShape2VecSet and Michelangelo generate geometrically novel shapes; stronger qualitative memorization also aligns with lower ZU.The table reports ZU against ShapeNet’s chair training and test sets and shows generated samples at the 60th percentile by LFD distance to the nearest training shape.Source: §4 · pp. 3–4, Table 2 · p. 4
• Recent 3D generative models have ZU values clustered around zero; Michelangelo and Trellis are slightly negative and demonstrate reasonable generalization.The evaluation uses each model’s full reported training and test datasets; Trellis’s test set is randomly sampled from its training dataset.Source: Table 3 · p. 4, §4 · p. 4
• Overall, earlier models exhibit strong memorization, whereas recent models demonstrate effective generalization; the shift may reflect evolving data and modeling choices.The authors name increasing data diversity and dataset size, parameter scale, and 3D representations as possible factors.Source: §4 · p. 4

5. Impact of Data on Memorization

• The experiments use the evaluation framework to examine how data properties, including modality, diversity, and conditioning granularity, affect memorization.Source: §5 · p. 4

5.1. Experimental Setup

• The controlled experiments use a curated Objaverse-XL subset and a pre-trained Vecset autoencoder with Hunyuan3D 2.1 as the diffusion backbone.The curated dataset contains about 140K captioned and labeled objects, split into 120K training and 20K test examples.Source: §5.1 · pp. 4–5

5.2. Baseline Model

• After test FD plateaus around 200K steps, further training increases memorization; training FD’s strong correlation with ZU makes it an unreliable generation-quality indicator.ZU and training FD decrease together as training progresses, while test FD initially decreases before plateauing.Source: §5.2 · p. 5, Figure 3 · p. 5

5.3. Data Modality

• For a controlled comparison, image and shape models use the same text prompts and Hunyuan3D backbone; shapes use unordered Vecsets without positional embeddings, while images use a positional 2D latent grid.Source: §5.3 · p. 5

U and ZSSCD

• Images have lower ZU than 3D shapes, indicating stronger memorization; qualitative examples likewise show image replications and more novel 3D features.The authors suggest the image set’s single-view projections contain less information and that diffusion models may memorize simpler data more readily.Source: U and ZSSCD · p. 6

5.4. Data Diversity

• With a fixed data budget, higher semantic diversity increases memorization in 3D diffusion models.The increase occurs without reduced quality: test FD is nearly flat across the 32-, 64-, and 100-class settings.Source: §5.4 · p. 6, Figure 5 · p. 6

5.5. Conditioning Granularity

• Memorization consistently increases from coarse to fine-grained conditioning, while finer conditions offer greater controllability and coarser ones reduce memorization.The authors suggest mixing prompt granularities as a possible balance.Source: §5.5 · pp. 6–7

• Moderate guidance at w = 3 yields the strongest memorization; unconditional generation and generation with no guidance have low memorization but poor quality, while w > 3 reduces memorization.Source: Figure 6 · p. 7

6.1. Guidance Scale

• Moderate guidance (w = 3) produces the strongest memorization; higher scales reduce copying as samples emphasize prompt subphrases, while unguided generation has poor quality.The table’s lamp example shows w = 3 reproducing the training shape, whereas larger scales emphasize the base or shade.Source: Table 6 · p. 7, §6.1 · pp. 7–8

6.2. Latent Space

• Longer Vecsets improve generation quality and prompt alignment while reducing memorization, with longer-sequence samples no longer fully reproducing specific training shapes.Figure 7 reports the trend; Figure 8 shows high-quality, prompt-aligned examples with novel features for lengths 1024 and 1280.Source: Figure 7 · p. 8, §6.2 · p. 8, Figure 8 · p. 8

6.3. Rotation Augmentation

• At similar test FD, yaw augmentation brings ZU closer to zero (-1.66 at 350K versus -3.12 at 200K), but delays test-FD convergence (350K versus 200K).The augmented model takes longer to converge, yet has lower memorization once generation quality is similar.Source: §6.3 · pp. 8–9

7. Limitations

• Controlled-experiment findings may be affected by inaccuracies in the LFD retrieval metric.The limitation applies to experiments relying on LFD.Source: §7 · p. 9
• Training prompts may bias ZU downward even for generalizing models; test prompts could yield more accurate estimates for high-performance generators.Source: §7 · p. 9
• The study uses Vecset diffusion and a relatively small-scale model; some findings are model-specific, and its generation quality is not fully comparable with cutting-edge models.Source: §7 · p. 9

8. Conclusion

• The paper introduces a framework to assess memorization and uses it to evaluate existing 3D generators and controlled design experiments.Source: §8 · p. 9
• The experiments find more memorization for image than 3D data, and stronger memorization with greater data diversity, finer conditioning, or moderate guidance.Source: §8 · p. 9
• Longer latent Vecsets and simple rotations during training are empirical recommendations for reducing memorization without degrading performance.Source: §8 · p. 9

A.1. Distance Metrics

• Chamfer distance measures average squared nearest-neighbor distance between point clouds; the supplement samples 4096 points for its computation.Point-cloud encoders produce embeddings, and the supplement also defines an L2-normalized embedding distance for two point clouds.Source: §A.1 · p. 12
• Light Field Distance compares shapes by summing L1 distances between silhouette descriptors across 10 canonical viewpoints.Each silhouette is rendered at 256 × 256 resolution and represented by Zernike moments and Fourier descriptors.Source: §A.1 · p. 12
• For image-encoder distances, the mesh embedding averages and L2-normalizes [CLS] representations from 12 rendered views.Source: §A.1 · p. 12

A.2. Memorization Metrics

• The global Mann-Whitney U test compares generated-sample and held-out-test distances to the training set; under generalization, those distance distributions should match.The null hypothesis states that a generated distance exceeds a test distance with probability 1/2.Source: §A.2 · p. 12
• The supplement describes the U statistic as an unbiased estimator of mn Δ_T(P_test, Q), and normalizes it as a z-score under the null when m and n are at least about 20.Source: §A.2 · p. 12
• ZU much less than zero indicates data-copying, while ZU near zero indicates generalization.Source: §A.2 · p. 12

A.3. Evaluation Framework

• ZU (-3.74 vs. -0.04) separates toy memorization from generalization, while training FD is nearly identical (0.11 vs. 0.10).Generated samples near training points represent memorization; samples from the same distribution but not near training points represent generalization.Source: §A.3 · p. 13, Figure 10 · p. 13
• Training and validation losses converge early and then oscillate around 0.6, failing to reflect the baseline model’s progression from low- to high-quality generation.The losses converge at approximately 10K steps, so the authors do not treat loss as a reliable quality metric.Source: §A.3 · p. 13, Figure 11 · p. 13

B. Existing Models’ Retrieval Visualization

• Table 7 summarizes evaluated models’ conditioning types, training data, and data splits; a dash marks an unreleased official split.The section visualizes retrievals for six chair-category models and five models on their respective full training sets.Source: Table 7 · p. 13, B. Existing Models’ Retrieval… · p. 13
• NFD, LAS-Diffusion, and Wavelet Generation show strong chair-shape memorization, while conditional LAS-Diffusion, 3DShape2VecSet, and Michelangelo show novel features.The retrieval visualizations cover chair-category samples and models evaluated on their full training sets.Source: B. Existing Models’ Retrieval… · p. 13

C.1. Dataset and Model Setup

• The dataset pipeline filters and balances Objaverse-XL caption-label pairs, while the VecSetX autoencoder and Hunyuan3D 2.1 backbone form the controlled-study model setup.The selected 100 frequent classes use existing Cap3D captions; alignment filtering and further label cleanup reduce the pairs before splitting the dataset.Source: §C.1 · pp. 13–14

C.2. Baseline and Sanity Check

• Larger datasets reduce memorization, whereas larger models strengthen it; the small model’s high FD on both splits accompanies low-quality generation.The model-size comparison trains small, baseline, and large models on 50K shapes from a 16-class subset for 200K steps.Source: Figure 15 · p. 15, §C.2 · p. 15, Figure 16 · p. 15

C.3. Data Modality

• The image model retrieves source training data more often than the 3D model when both are conditioned on the same training captions.The source data retrieval rate measures how often a generated sample’s nearest training sample is the ground-truth source for its prompt.Source: §C.3 · pp. 15–16, Table 9 · p. 15

C.4. Data Diversity

• The diversity and conditioning experiments use class subsets and coarse labels alongside phrase-, sentence-, and paragraph-level captions.The class-conditional model replaces the text encoder with trainable class embeddings injected through cross-attention.Source: §C.4 · p. 16, §C.5 · p. 16

C.6. Guidance Scale

• Moderate guidance scales produce the strongest memorization, while stronger guidance reduces it; training FD can move opposite to ZU as generated diversity falls.The supplementary case studies also note that large guidance scales can reduce alignment with the complete prompt.Source: §C.6 · pp. 16–17, Figure 21 · p. 17

C.7. Latent Space

• Vecsets of length 1024 and 1280 retain high fidelity while showing reduced memorization, and changing Vecset length does not significantly affect autoencoder reconstruction quality.The additional chair and assault-rifle examples describe prompt-aligned shapes with novel features at longer lengths.Source: §C.7 · p. 18, Table 12 · p. 18, Figure 22 · p. 18

C.8. Rotation Augmentation

• LFD is rotation-sensitive across tested axes, whereas Uni3D gives near-zero distances for yaw-rotated objects; experiments therefore restrict augmentation to four discrete yaw angles.The four angles are 0°, 90°, 180°, and 270° around the y-axis.Source: §C.8 · p. 18, Figure 24 · p. 18

D. Rendering Details

• Qualitative retrievals show strong copying by NFD, unconditional LAS-Diffusion, and Wavelet Generation, while large-dataset models generally show novel retrieved shapes at moderate percentiles.The full-training-set comparison cautions that LFD is category-sensitive and notes that 3DTopia-XL often degenerates into low-quality shapes.Source: Figure 25 · p. 19, Figure 26 · p. 19