Source-linked AI summary
PQ-NET: A Generative Part Seq2Seq Network for 3D Shapes
Rundi Wu, Yixin Zhuang, Kai Xu, Hao Zhang, Baoquan Chen
TL;DR
3D generative models often produce unstructured shapes, while structural understanding requires parts and their relations. PQ-NET addresses this gap with a part-wise Seq2Seq autoencoder that reconstructs shapes sequentially and supports several generative tasks. Experiments report performance and output quality comparable or superior to several state-of-the-art generative models, with limitations in learned part relations and topology-altering interpolation.
Problem
Most existing 3D generative models produce unstructured shapes, despite the importance of representing parts, their compositions, and inter-part relations.
Method
PQ-NET encodes segmented parts with a part autoencoder, maps their feature sequence to a latent vector using Seq2Seq, and decodes the shape one part at a time.
Results
PQ-NET achieves performance and output quality comparable or superior to state-of-the-art generative models across shape generation, structured single-view reconstruction, and shape completion.
Takeaways & Limitations
The model provides a fully generative 3D shape representation as sequential part assembly, with generated shapes composed of meaningful parts.
Takeaways & Limitations
PQ-NET does not learn relations such as symmetry and does not produce topology-altering interpolation, especially between shapes with different numbers of parts.
Abstract
from arXiv · showhide
We introduce PQ-NET, a deep neural network which represents and generates 3D shapes via sequential part assembly. The input to our network is a 3D shape segmented into parts, where each part is first encoded into a feature representation using a part autoencoder. The core component of PQ-NET is a sequence-to-sequence or Seq2Seq autoencoder which encodes a sequence of part features into a latent vector of fixed size, and the decoder reconstructs the 3D shape, one part at a time, resulting in a sequential assembly. The latent space formed by the Seq2Seq encoder encodes both part structure and fine part geometry. The decoder can be adapted to perform several generative tasks including shape autoencoding, interpolation, novel shape generation, and single-view 3D reconstruction, where the generated shapes are all composed of meaningful parts.
1. Introduction
PQ-NET represents and generates 3D shapes as sequential assemblies of meaningful parts, addressing the unstructured outputs common in existing 3D generative models. Its part-wise Seq2Seq architecture supports multiple generative tasks while jointly encoding part structure and geometry.
- Most existing deep 3D generative models produce unstructured shapes, despite object perception involving parts, compositions, and inter-part relations.
- PQ-NET represents and generates 3D shapes through a linear sequence of assembled parts.The paper describes the assembly sequence as a sentence organizing the parts of a shape.
- Each segmented input part is encoded into a feature representation before sequential shape processing.The part autoencoder provides the feature representation used by the network.
- A Seq2Seq autoencoder maps part-feature sequences to a fixed-size latent vector and decodes the shape one part at a time.The encoder is bidirectional and recurrent, while the decoder reconstructs a sequential part assembly.
- The latent representation jointly captures part structure and fine geometry, enabling autoencoding, interpolation, novel generation, and single-view reconstruction.The reconstruction task can use RGB or depth images, and generated shapes remain composed of meaningful parts.
2. Related work
Related work spans holistic 3D generative models, structural shape analysis, part-based synthesis, Seq2Seq representation learning, and part-sequence assembly. PQ-NET is positioned against methods using templates, hierarchies, graphs, fixed part sets, or box primitives.
- Structural analysis of 3D shapes: Earlier structural-analysis methods model shape variability with parametric templates or probabilistic models, often requiring part correspondence.
- “Holistic” generative models of 3D shapes: Holistic generative models use voxels, point clouds, meshes, images, or implicit functions but model shape variability without explicit structure.The passage attributes this pattern partly to the scarcity of part-based representations suitable for deep learning.
- Part-based generative models: Part-based models include template-based generation, correspondence-dependent ShapeVAE, recursive hierarchical models such as GRASS and StructureNet, and joint geometry-structure synthesis.
- Part-based generative models: Other approaches generate and assemble parts using fixed part counts, factorized embeddings, or pretrained per-part generators.
- Seq2Seq: Seq2Seq is an RNN encoder-decoder framework used for sequence transformation and applications including captioning, conversation, summarization, and some 3D representation learning.
- 3D-PRNN: part sequence assembly: 3D-PRNN reconstructs 3D shapes as sequences of box primitives from depth images, whereas PQ-NET jointly models part structure and geometry in a latent space.3D-PRNN also relies on sampled primitive parameters for novel shape generation, while PQ-NET uses latent GANs.
- Single view 3D reconstruction (SVR): Single-view reconstruction research commonly maps 2D images to voxel or point-cloud shapes, with other methods fusing predicted depth maps or inferring cuboid abstractions.
3. Method
PQ-NET represents a segmented 3D shape as a sequence of part geometry and structural features, encodes that sequence into a fixed-size latent vector, and decodes it as a sequential assembly. Its training separates part geometry encoding from joint geometry-structure sequence modeling, while the latent space supports multiple downstream generation and reconstruction tasks.
- Part representation: Each part is encoded by a CNN-based geometry autoencoder whose MLP decoder represents the part as a continuous signed distance field.Parts are normalized to 64×64×64 volumes; the decoder can produce smooth geometry at arbitrary sampling resolutions.
- Seq2Seq autoencoder: A bidirectional stacked RNN encodes the part sequence and its reverse into a fixed-size latent vector, which the stacked RNN decoder converts into parts sequentially.The latent representation is initialized as the decoder state, enabling iterative part assembly.
- Part representation: Each part feature combines a geometry vector, a 6 DoF bounding box for position and size, and a one-hot part-number vector.The complete sequence element is symbolized as S_i = [F_i; t_i].
- Seq2Seq autoencoder: The decoder separately predicts geometry and structure features at each step and produces a stop signal to determine when the assembly ends.The stop iteration condition is s′_i > 0.5, and disentangling geometry from bounding-box structure improves results without a deeper network.
- Training and losses: Training uses two stages: first learning the part geometry autoencoder, then jointly modeling geometry and structure sequences with the Seq2Seq model.The second stage uses reconstruction and stop losses, with the weighted factor α empirically set to 0.01.
- Applications: The learned latent space is used for shape autoencoding, latent-GAN shape generation, interpolation, and single-view reconstruction from RGB or depth images.The sequential decoder generates shapes with geometry and segmentation, while image encoders map RGB or depth inputs into the pretrained shape latent space.
4. Results, Evaluation, and Applications
PQ-NET is evaluated on shape auto-encoding, generation, interpolation, single-view reconstruction, and additional part-assembly applications using PartNet chairs, tables, and lamps. Results show strong structure-geometry preservation and competitive comparisons, alongside failure cases involving duplicated or missing parts.
- Experimental Setup: The evaluation uses PartNet’s three largest categories—chair, table, and lamp—with shapes containing at most 10 parts.The resulting dataset contains 6305 chairs, 7357 tables, and 1188 lamps, split using official training, validation, and test partitions.
- Shape Auto-encoding: PQ-NET generally outperforms IM-NET qualitatively and quantitatively on shape auto-encoding, while jointly representing structure and geometry.The comparison evaluates IoU, symmetric Chamfer Distance, and Light Field Distance at 64^3 and 256^3 resolutions.
- Shape Auto-encoding: Cross-category training beats per-category training, indicating that PQ-NET handles different part arrangements across categories.The authors attribute this benefit partly to the simplicity of part geometry.
- Shape Generation: PQ-NET produces smooth geometry while preserving whole-shape structure, although sequential generation can sometimes duplicate or omit parts.The decomposition strategy is reported as helpful for thin structures and complex topology, while requiring a balance between geometry generation and structure recovery.
- Interpolation: Latent-space interpolation yields smooth transitions in both geometry and structure.This result is presented as evidence of continuity in the learned shape latent space.
- Single-view Reconstruction: For single-depth-image reconstruction, PQ-NET produces more accurate structures than 3D-PRNN, with the best result under PartNet’s natural part order.Performance declines slightly under the top-down order used by 3D-PRNN but remains better than 3D-PRNN.
5. Conclusion, limitation, and future work
PQ-NET represents and generates 3D shapes as assembly sequences, supporting novel generation and image-conditioned reconstruction. The paper also identifies limitations in learned part relations, interpolation, latent-space behavior, and part-order dependence.
- Conclusion: PQ-NET generates shapes as sequential assemblies, including novel shapes and reconstructions conditioned on depth scans or RGB images.
- Applications: Part-order denoising unscrambles random input orders into consistent output orders to facilitate part correspondence.The figure notes that displayed color correspondence is illustrative and not network output.
- Limitations: PQ-NET does not learn relations such as symmetry; it outputs only spatial arrangements of parts.The paper contrasts this with symmetry hierarchies and graphs, which can encode such relations.
- Future Work: The learned latent space appears to entangle part structure and geometry unpredictably, while part order affects network learning.The authors suggest studying better assembly orders and losses for quantifying them.
Supplementary Materials
The supplementary material documents implementation, data preparation, evaluation metrics, comparisons with 3D-PRNN, and additional visual results.
- Supplementary Contents: Section B describes detailed implementation of the PQ-NET architecture.
- Supplementary Contents: The supplement covers data preparation and metrics used to evaluate shape generation.
- Supplementary Contents: Additional sections compare generation with 3D-PRNN and provide visual results for shape completion and random generation.
B. Implementation Details
PQ-NET is implemented as separate part-geometry and Seq2Seq autoencoders, with documented architectures, regularization, progressive training, and fixed optimization settings.
- Architecture: The implementation separates the PQ-NET architecture and training configuration into detailed part-geometry and Seq2Seq components.Tables 4 and 5 list their specific architectural parameters.
- Architecture: The part geometry autoencoder follows IM-NET with skip connections, while the Seq2Seq autoencoder uses dropout within the GRU.The stated dropout rate is 0.2 and is intended to reduce overfitting.
- Training: Training proceeds progressively for the part geometry autoencoder at resolutions 16^3, 32^3, and 64^3 before training Seq2Seq at 64^3.The part autoencoder is fixed before Seq2Seq training.
- Training: The experiments use batch size 40 and learning rate 5e-4 for geometry training, then batch size 64 and learning rate 1e-3 for Seq2Seq training.The implementation uses PyTorch.
C. Data Preparation Details
The data preparation uses PartNet categories and hierarchical part labels, voxelizes shapes and parts at 64^3 resolution, and derives geometry and placement inputs for PQ-NET.
- Dataset: The experiments mainly use the three largest PartNet categories: chair, table, and lamp.Second-layer nodes in PartNet’s hierarchy provide the part geometry, with semantic labels taken from the dataset files.
- Dataset Statistics: Table 6 reports average, minimum, and maximum part counts for each dataset category.
- Voxelization: Original meshes are voxelized at 64^3 resolution, interiors are flood-filled, and each part is scaled to 64^3 within its bounding box.
- Sampling: Part volumes are downsampled to 32^3 and 16^3, with surface-near points sampled alongside signed-distance values.The sampling counts are 4096 at 16^3, 8192 at 32^3, and 32768 at 64^3.
- Placement Representation: Bounding-box parameters for Seq2Seq training encode each part’s deviation and translation from its local frame to the shape coordinate system.
D. Metrics
PQ-NET’s generation evaluation uses Coverage, Minimum Matching Distance, and Jensen-Shannon Divergence to assess diversity, fidelity, and distributional similarity.
- The comparisons use Chamfer distance for IM-NET and StructureNet, and IoU for 3D-PRNN.
- Coverage measures the fraction of ground-truth shapes matched by generated shapes, roughly representing generation diversity.A higher COV indicates that more ground-truth shapes can be roughly represented by generated shapes.
- Minimum Matching Distance averages each ground-truth shape’s distance to its nearest generated shape, roughly representing generation fidelity.
- Jensen-Shannon Divergence compares generated and ground-truth point distributions in a predefined voxelized Euclidean 3D space.
- The authors use code from optas/latent_3d_points for metric calculation.
E. Comparison to 3D-PRNN on Shape Generation Task
The paper compares PQ-NET with 3D-PRNN for random shape generation using matched experimental settings and reports stronger quantitative performance for PQ-NET.
- For a fair comparison, the authors sample 3D-PRNN’s first RNN input feature from training data, while PQ-NET generates shapes from random noise.3D-PRNN samples new structure within a constraint region, whereas PQ-NET uses random noise in its generation procedure.
- Additional visual comparisons of randomly generated 3D primitives are provided between PQ-NET and 3D-PRNN.
- PQ-NET outperforms 3D-PRNN on all reported COV and MMD measurements for chairs, tables, and lamps.The evaluation samples 2000 generated shapes for chairs and tables and 800 for lamps, using 1 − IoU as the distance measure.
- The reported comparison indicates that PQ-NET’s generated shapes are more diverse and plausible than those of 3D-PRNN.
F. More Results
Additional qualitative results show PQ-NET’s partial shape completion, generated shapes, and latent-space interpolations at high reconstruction resolution.
- Figure 12 presents visual results for partial shape completion.
- Figure 13 shows additional generated shapes in rows 1–6 and two latent-space interpolations in rows 7–8.
- The shapes in Figure 13 are sampled at resolution 256^3 and reconstructed using Marching Cubes.