PaperlayerView exact revision on arXiv

Source-linked AI summary

SCORES: Shape Composition with Recursive Substructure Priors

Chenyang Zhu, Kai Xu, Siddhartha Chaudhuri, Renjiao Yi, Hao Zhang

arXiv:1809.05398v1cs.GRcs.CVcs.LG

TL;DR

Combining parts from different shapes is challenging when they are misaligned, topologically incompatible, or globally unlike training exemplars. SCORES uses a recursive network and learned substructure prior to iteratively adjust assemblies, including adding or removing parts. The reported results show composition from inputs with gross misalignments and missing or redundant parts; the method is not designed to be fully generative.

  • Problem

    Composing parts with topological mismatches and positional misalignments into plausible shapes remains challenging, especially when novel composites do not globally match training exemplars.

    Source: §1 · p. 2

  • Method

    SCORES uses a recursive neural network and learned substructure prior to iteratively infer hierarchies and adjust part geometry and structure, including adding or removing parts.

    Source: §1 · p. 2

  • Results

    SCORES composes inputs from multiple sources despite gross misalignments and missing or redundant parts, with performance evaluated on ComplementMe using plausibility measurement and user studies.

    Source: §1 · pp. 2–3, §6 · p. 13

  • Takeaways & Limitations

    Shape composition extends beyond insertion, exchange, or connection of parts to composing incomplete sets with incompatible parts into plausible structures.

    Source: §6 · pp. 12–13

    SCORES is not designed to be fully generative and synthesizes novel structures only when part composition alone incurs too large a plausibility loss.

    Source: §6 · p. 13

Abstract

from arXiv · show

We introduce SCORES, a recursive neural network for shape composition. Our network takes as input sets of parts from two or more source 3D shapes and a rough initial placement of the parts. It outputs an optimized part structure for the composed shape, leading to high-quality geometry construction. A unique feature of our composition network is that it is not merely learning how to connect parts. Our goal is to produce a coherent and plausible 3D shape, despite large incompatibilities among the input parts. The network may significantly alter the geometry and structure of the input parts and synthesize a novel shape structure based on the inputs, while adding or removing parts to minimize a structure plausibility loss. We design SCORES as a recursive autoencoder network. During encoding, the input parts are recursively grouped to generate a root code. During synthesis, the root code is decoded, recursively, to produce a new, coherent part assembly. Assembled shape structures may be novel, with little global resemblance to training exemplars, yet have plausible substructures. SCORES therefore learns a hierarchical substructure shape prior based on per-node losses. It is trained on structured shapes from ShapeNet, and is applied iteratively to reduce the plausibility loss.We showresults of shape composition from multiple sources over different categories of man-made shapes and compare with state-of-the-art alternatives, demonstrating that our network can significantly expand the range of composable shapes for assembly-based modeling.

1 INTRODUCTION

SCORES addresses the difficulty of composing incompatible parts into plausible shapes, including composites with novel global structures. It uses a learned recursive substructure prior to adjust assemblies and may add or remove parts; its evaluation uses ComplementMe and user studies.

The composition challenge

  • Topological mismatches and positional misalignments make it difficult to combine selected parts into plausible shapes, while designers may seek novel outcomes.Earlier mix-n-match work focused mainly on finding compatible parts, with less effort on rearranging and deforming parts across large discrepancies.Source: §1 · p. 2

A structure-prior approach

  • SCORES treats composition as structure synthesis: it may substantially alter input geometry and structure, or add and remove parts, to produce a coherent, plausible shape.The objective is plausibility under a learned structure prior, not simply connecting the input parts.Source: §1 · p. 2
  • The method uses a recursive neural network to learn a structure prior and iteratively performs hierarchy inference and substructure adjustment.The contribution list describes the approach as resolving geometric and topological incompatibilities, including through part insertion or removal.Source: §1 · pp. 2–3
  • Because a composite may be globally unlike any training exemplar, SCORES models recurrent substructures at multiple abstraction levels rather than relying on a monolithic global prior.The paper motivates this with part groups that recur across shapes and categories despite different global structures.Source: §1 · p. 2
  • SCORES takes roughly aligned parts from two or more shapes, then iteratively infers hierarchies and adjusts substructures to reduce plausibility loss, including by adding or removing parts.The network can address missing or redundant input parts when geometric adjustment alone cannot sufficiently reduce the loss.Source: §1 · p. 2

Evaluation and contributions

  • The paper benchmarks performance on ComplementMe and evaluates it with a projective-comparison plausibility measure and user studies.Its listed contributions include a benchmark for shape-composition tasks.Source: §1 · pp. 2–3

2 RELATED WORK

Prior assembly-based modeling includes part connection, substitution, and placement, often with constraints such as close alignment, correspondences, or annotations. SCORES instead optimizes whole-assembly geometry and topology under a data-driven prior, without requiring part labels or correspondences.

Part connection

  • Part-connection methods deform and seal adjacent boundaries or optimize likely connections, generally assuming parts are already selected and closely aligned.SCORES instead targets overall geometry and topology, optionally adding or removing parts to improve plausibility.Source: §2 · p. 3

Part substitution and crossover

  • Part substitution and crossover methods exchange or retrieve parts, but require known correspondences; SCORES is correspondence-free and can modify structure to improve plausibility.Some prior systems use manually defined substructure templates while preserving the target’s global structure; SCORES learns nested substructures from data without human intervention.Source: §2 · p. 3

Part placement

  • Part-placement methods may rely on semantic relation annotations, whereas SCORES does not require part labels or relation annotations.ComplementMe’s placement network predicts where a newly suggested part should be placed, but is designed to place a single part and does not generalize to shapes unlike training exemplars.Source: §2 · p. 3

Generative shape priors

  • Unlike GRASS, which synthesizes globally exemplar-like shapes within a category, SCORES handles inter-shape, cross-category composition with greater structural diversity and part discrepancies.SCORES uses iterative hierarchical substructure rectification and models valid substructures; the paper reports it significantly outperforms GRASS and variants on composition tasks.Source: §2 · pp. 3–4
  • SCORES is an input-conditioned, iterative structure-enhancing autoencoder whose plausibility prior is based on nested substructures, allowing globally unfamiliar shapes with plausible local structure.The paper distinguishes this from fully generative models of complete shapes.Source: §2 · p. 4

3.1 Problem statement and method design

SCORES turns a noisy collection of parts into a plausible synthesized shape by optimizing its hierarchical structure and geometry. Its design interleaves hierarchy inference with local, hierarchy-guided adjustment rather than relying on a one-pass global correction.

Problem statement and method design

  • SCORES aims to turn parts from multiple shapes into a plausible synthesized shape, correcting gross misalignment and missing or redundant components without part labels or other annotation.The output may differ substantially from the inputs by adding, removing, and adjusting parts.Source: §3.1 · p. 4
  • Inputs are parts represented as oriented bounding boxes, roughly aligned to a common coordinate system; outputs are coherent parts whose geometry follows the adjusted boxes.When a new box is generated, its underlying part geometry is retrieved from a shape database.Source: §3.1 · p. 4
  • Structure inference and geometry adjustment are coupled, while large incompatibilities make single-level global adjustment likely to fail.The method therefore does not assume that a training-set structure completely explains the input parts.Source: §3.1 · p. 4
  • SCORES follows local adjustment, hierarchy-guided top-down cascading adjustment, and iterative inference of structural organization and part geometry.Subsets of the input may resemble substructures found in exemplars even when the full assembly does not.Source: §3.1 · p. 4
  • Each iteration first organizes parts into a common hierarchy, then adjusts positions and orientations to improve the plausibility of the hierarchy’s constituent substructures.A bottom-up recursive encoder represents each subtree; a top-down decoder adjusts codes using a learned manifold of valid substructures.Source: §3.1 · pp. 4–5

3.2 Iterative optimization

SCORES iteratively alternates between organizing part boxes into a hierarchy and adjusting their substructures. The adjustment uses a recursive denoising autoencoder and a learned substructure prior, and may regenerate substructures by adding or removing boxes.

3.2 Iterative optimization

  • The objective organizes part boxes into a hierarchy whose subtrees are well explained by a learned substructure model.The adjustment energy measures deviation from valid counterparts and is minimized through local substructure adjustment.Source: §3.2 · p. 5
  • Each iteration infers a plausible parse tree for the current boxes, then adjusts box geometry with a recursive denoising autoencoder trained on a learned structure prior.The decoding phase cascades adjustments through the hierarchy and may regenerate substructures, adding or removing boxes when necessary.Source: §3.2 · p. 5
  • The optimization alternates hierarchy inference with substructure adjustment, using the learned prior and trained networks for iterative optimization.The training and testing procedure first learns valid substructures, then trains encoder-decoder networks for substructure adjustment.Source: §3.2 · p. 5

3.3 Training: Substructure prior and denoising network

SCORES learns a discrete prior over valid substructures and fine-tunes a recursive autoencoder to denoise noisy assemblies. Its bottom-up encoding and top-down decoding use hierarchy context to adjust substructures, with resynthesis available when local adjustment is insufficient.

3.3 Training: Substructure prior and denoising network

  • Training first learns a substructure codebook from clean shapes, then freezes it while fine-tuning the encoder-decoder to repair synthetically corrupted assemblies.Random subtrees from training hierarchies are corrupted to mimic creative assembly; their original versions are denoising targets.Source: §3.3 · p. 5, §3.3.2 · pp. 6–7

3.3.1 Discrete latent space of substructures.

  • Valid substructures are modeled as a discrete latent space, while complete shape hierarchies provide nested examples and contextual dependencies between substructures.The prior is learned from substructures sampled from hierarchies of training shapes using a VQ-VAE codebook.Source: §3.3.1 · pp. 5–6
  • A recursive autoencoder aggregates structural information bottom-up into inner codes, then decodes top-down into outer codes that carry global layout context.Child inner codes form a parent code; outer-code decoding uses the parent’s outer code and sibling inner codes.Source: §3.3.1 · pp. 5–6

3.3.2 Fine-tuning the autoencoder for denoising.

  • During denoising, the network snaps internal-node outer codes to nearby codebook entries and continues top-down decoding to produce adjusted part boxes.The decoder is fine-tuned so noisy substructures map near clean codebook entries, with snapping handling residual denoising.Source: §3.3.2 · p. 7
  • When local adjustment cannot repair a substructure, such as when parts must be added or removed, SCORES can fully resynthesize it.The synthesizing decoder is trained jointly for this fallback.Source: §3.3.2 · p. 7

3.4 Testing: Iteratively optimizing a noisy part assembly

SCORES alternates between inferring a hierarchy for the current part boxes and adjusting substructures against a learned VQ-VAE prior. It repeats these steps until convergence, with resynthesis available for substructures that cannot be repaired by local adjustment.

3.4 Testing: Iteratively optimizing a noisy part assembly

  • The procedure alternates hierarchy inference and substructure adjustment until convergence, choosing a hierarchy that explains the current part boxes under the VQ-VAE prior.Hierarchy search is NP-hard, so SCORES uses top-down importance sampling to select subtrees for resampling; high-error nodes are more likely to benefit.Source: §3.4 · p. 7, §3.4.1 · p. 7

3.4.1 Alternating step 1: Hierarchy inference.

  • The initial hierarchy comes from source-shape reference trees when available; otherwise, SCORES builds one from the input boxes using GRASS.Subtrees from source shapes are linked under a common parent.Source: §3.4.1 · p. 7

3.4.2 Alternating step 2: Local adjustment of substructures.

  • Local adjustment encodes part boxes through the inferred hierarchy, then decodes top-down while using the substructure prior to reduce VQ error.The iterative optimization decreases overall VQ error.Source: §3.4.2 · p. 7
  • With hierarchy resampling, adjustment converges better; without it, the method converges slower and to worse output.Figure 8 compares representation-error change for a randomly perturbed bicycle, with and without hierarchy resampling.Source: §3.4.1 · p. 7, Figure 8 · p. 8
  • Small-error substructures are denoised without changing their part composition; otherwise, they are resynthesized, with an interactive threshold and node depth limiting large-substructure resynthesis.The tradeoff allows geometry adjustment that preserves input part counts or synthesis that may add or remove parts.Source: §3.4.2 · p. 8
  • After assembling adjusted boxes, SCORES aligns input parts to them and uses sparse docking-point optimization to improve connections; only about 10% of shapes required this post-adjustment.Translation adjustments were at most 2.2% of the whole-shape OBB diagonal, and scaling adjustments at most 3.5%.Source: §3.4.3 · p. 8

4 RESULTS AND EVALUATION

SCORES is evaluated on structural plausibility, heterogeneous merging, and flexible composition. It outperforms Fit-and-Diverse on reported quantitative and human plausibility measures, while also supporting part completion and removal.

Dataset and benchmark

  • Evaluation uses segmented ComplementMe shapes without part labels, while StructMerge tests randomly partitioned held-out shapes.Each test shape contributes at most 12 partitions, with each group containing between 1 and N −1 parts.Source: §4 · pp. 8–9
  • Because the OBB merge-error metric tests groups from the same source shape, cross-source merging is evaluated with visual comparisons and user studies.The authors state that no suitable automatic test is available for merging parts from different source shapes.Source: §4 · pp. 9–10

Exact composition (no synthesis)

  • SCORES achieves the fastest convergence and lowest merge error on Chair and Bicycle; removing inner leaf codes, the VQ-VAE prior, or hierarchy resampling causes the largest drops.The Inner-Outer Concatenation variant has similar curves but takes longer to converge during training.Source: §4 · p. 9

Flexible composition (with structure synthesis)

  • SCORES can complete missing parts and remove redundant or overlapping parts, with ηT controlling the trade-off between exact composition and structure synthesis.The paper reports that the default ηT of 0.7 corrects most noticeable missing or redundant parts.Source: Figure 14 · p. 10, §4 · p. 11

User study

  • SCORES fares better in both plausibility and merge quality across all six categories in the heterogeneous-merge user study.The study also reports satisfactory quality within and across fine-grained chair subcategories.Source: §4 · p. 10, Table 3 · p. 10, Table 2

Comparisons

  • 34% higher average plausibility score than Fit-and-Diverse across 10 input sets from chair, airplane, and bicycle categories.The comparison uses a learned multi-view structural plausibility measure.Source: §4 · p. 11
  • 81% of SCORES crossover outputs were judged plausible by human raters, compared with 70% for Fit-and-Diverse.Fifty raters assessed whether evolved shapes were plausible.Source: §4 · p. 11

5 APPLICATIONS

SCORES is applied to interactive assembly, shape crossover, and scan reconstruction. These applications demonstrate compositions across categories and show both the method’s uses and a scan-geometry limitation.

5 APPLICATIONS

  • SCORES supports interactive assembly-based modeling and prototype applications for shape evolution and data-driven scan reconstruction.For evolution, it is integrated into Fit-and-Diverse; for scan completion, it rectifies scan-derived OBBs using learned priors.Source: §5 · pp. 11–12
  • Figure 16 shows challenging compositions across six categories, including cross-category chair–table merging guided by a chair substructure model.The figure also shows redundant parts and newly synthesized parts in the merged results.Source: Figure 16 · p. 12
  • Fit-and-Diverse crossover with SCORES produces a collection of 36 chairs from 9 input chairs.Fuzzy correspondence identifies parts to swap before SCORES composes the resulting shapes.Source: §5 · p. 12
  • For scan completion, SCORES rectifies OBBs and synthesizes parts, but database-retrieved geometry may not precisely match the scan.The pipeline segments the scan, fits OBBs to point clusters, and retrieves and transforms database parts to fit the output boxes.Source: §5 · p. 12

6 DISCUSSION, LIMITATION, AND FUTURE WORK

SCORES broadens shape composition beyond connecting or exchanging parts, using learned substructure priors to handle defective inputs and produce novel structures. Its scope remains bounded by its training data, modeling assumptions, and the input parts’ compatibility with a plausible result.

6 DISCUSSION, LIMITATION, AND FUTURE WORK

  • Shape composition extends beyond part insertion, exchange or crossover, and connection.The authors identify this broader view as a central take-home message.Source: §6 · p. 12
  • SCORES crossover produced a chair collection from an input set of 9 chairs.The figure depicts the evolved collection alongside its input chairs.Source: Figure 17 · p. 13
  • When trained on ShapeNet structured shapes, SCORES composes parts from multiple sources despite gross misalignment and missing or redundant parts.The authors describe these results as supporting composition from corrupted inputs.Source: §6 · p. 13
  • Compositions can have structures unseen in training when their input substructures can be rectified using the training data.The authors present this as a potentially transferable concept for generative models.Source: §6 · p. 13
  • SCORES operates on unlabeled OBBs, while whether this is sufficient for scenes remains open because scene-object relations are looser than shape-part relations.Source: §6 · p. 13
  • SCORES is not fully generative, synthesizes structures only when composition has excessive VQ error, and assumes complete input parts.Future work could make it conditional on sparse inputs or extend inputs to sketches and images.Source: §6 · pp. 13–14
  • Functionality, aesthetics, style, and creativity are not explicitly modeled, and SCORES does not verify that the input parts can form the desired plausible shape.Source: §6 · p. 13
  • Rare or absent substructures may be changed into a different known structure, and a category-specific model may place cross-category parts implausibly.The authors attribute these failure cases to reliance on learned discrete substructure spaces.Source: §6 · pp. 13–14, Figure 19 · p. 14

Related papers

Loading 1809.05398v1…