PaperlayerView exact revision on arXiv

Source-linked AI summary

DualSDF: Semantic Shape Manipulation using a Two-Level Representation

Zekun Hao, Hadar Averbuch-Elor, Noah Snavely, Serge Belongie

arXiv:2004.02869v1cs.CVcs.LG

TL;DR

High-fidelity 3D representations and interpretable primitive decompositions are difficult to combine in one representation. DualSDF couples fine detail and a coarse proxy through a shared variational latent space, enabling semantically meaningful proxy edits to propagate to the detailed shape with minimal user input.

  • Problem

    High-fidelity detail and interpretable, editable primitive decompositions are difficult to achieve together in a single 3D shape representation.

    Source: §1 · p. 1

  • Method

    DualSDF decodes fine-grained detail and a coarse primitive-based proxy from a shared variational latent code, allowing proxy edits to propagate to the detailed shape.

    Source: §1 · pp. 1–2

  • Results

    The two-level approach offers simplicity and interpretability without compromising fidelity, and supports semantically meaningful manipulation with minimal user input.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Users can edit individual proxy primitives and observe corresponding changes in the high-resolution shape and other primitives at an interactive rate.

    Source: §1 · p. 2

    Successful manipulation requires each primitive to remain approximately in the same region of the shape across the class of shapes.

    Source: §3.4 · p. 6

Abstract

from arXiv · show

We are seeing a Cambrian explosion of 3D shape representations for use in machine learning. Some representations seek high expressive power in capturing high-resolution detail. Other approaches seek to represent shapes as compositions of simple parts, which are intuitive for people to understand and easy to edit and manipulate. However, it is difficult to achieve both fidelity and interpretability in the same representation. We propose DualSDF, a representation expressing shapes at two levels of granularity, one capturing fine details and the other representing an abstracted proxy shape using simple and semantically consistent shape primitives. To achieve a tight coupling between the two representations, we use a variational objective over a shared latent space. Our two-level model gives rise to a new shape manipulation technique in which a user can interactively manipulate the coarse proxy shape and see the changes instantly mirrored in the high-resolution shape. Moreover, our model actively augments and guides the manipulation towards producing semantically meaningful shapes, making complex manipulations possible with minimal user input.

1. Introduction

DualSDF addresses the difficulty of combining high-fidelity detail with interpretable, editable shape parts by representing each shape at two granularities in a shared latent space. Its proxy-based manipulation propagates edits to the detailed shape, while the variational objective and two new metrics support and assess the coupled representations.

1. Introduction

  • The representation aims to combine intuitive, editable primitive decompositions with high fidelity, a combination that is difficult to achieve in one representation.The introduction contrasts expressive high-resolution representations with more parsimonious, interpretable shape compositions.Source: §1 · p. 1
  • DualSDF uses a shared latent code to represent each shape both as fine-grained detail and as a coarse structural decomposition.A single code decodes to both representations of the same underlying shape.Source: §1 · p. 1
  • A variational autodecoder learns the shared latent space, imposing a Gaussian prior that enables sampling and encourages compactness for interpolation and optimization-based manipulation.The paper reports tighter coupling between representations than an auto-encoder or auto-decoder, including for novel shapes.Source: §1 · p. 2
  • Signed distance fields provide a common substrate: unions of simple primitives model the coarse level, while deep networks represent high-resolution detail.Using implicit representations for both levels also allows simpler implementation and evaluation.Source: §1 · p. 2
  • Edits to the coarse proxy propagate through the shared code to the high-resolution shape in a semantically meaningful way, with minimal user input.Sparse edits to some primitives can propagate to the rest while maintaining the shape on the manifold of likely shapes.Source: §1 · p. 2
  • The paper introduces cross-representation consistency and primitive-based semantic consistency to evaluate agreement between representations and consistency of primitives across shapes.The two metrics target distinct aspects of manipulation performance.Source: §1 · p. 2

2. Related Work

Related work spans expressive 3D representations and primitive-based shape decompositions, as well as hierarchical and hybrid methods. The paper positions semantic manipulation as distinct from much prior multi-level work and contrasts its learned coupling with hand-designed primitive-to-shape transformations.

2. Related Work

  • Prior mesh-based modeling methods include structured deformable meshes, but some require shared connectivity and others require part annotations for training.The structured-mesh method by Gao et al. [18] overcomes the same-connectivity constraint but needs part annotations.Source: §2 · p. 2
  • Parametric surfaces, point clouds, and implicit functions offer alternative shape representations, with point clouds described as relatively coarse and less suitable for interaction than primitives.Implicit methods model shapes as function isosurfaces and have been applied to generation, completion, and single-view reconstruction.Source: §2 · pp. 2–3
  • Primitive-based modeling includes cuboids, superquadrics, Gaussian balls, and convex polytopes, with prior work also learning which primitives approximate a shape collection.Applications include reverse engineering and reconstructing scanned or incomplete point clouds.Source: §2 · p. 3
  • Some hierarchical image-generation methods condition fine outputs on coarse ones, whereas this work focuses on semantic shape manipulation.Earlier shared-latent hybrid representations combine voxel, image, and point representations for discriminative tasks, not shape generation and manipulation.Source: §2 · p. 3
  • Tulsiani et al. fit primitives for abstract shape modeling and demonstrate manipulation, but their hand-designed transformations do not guide edits toward semantically meaningful shapes.The paper relates its coarse proxy to a skeleton that can simplify manipulation of a high-resolution model.Source: §2 · p. 3

3. Method

DualSDF represents each shape with a primitive-based coarse proxy and a high-resolution SDF, coupled through a shared latent space. The method uses this coupling to let users manipulate primitives while the high-resolution shape follows, with unconstrained attributes able to adapt.

3.1. Coarse Primitive-based Shape Representation

  • The coarse representation approximates a shape’s signed distance field with a union of simple primitives, using spheres in the presented implementation.The framework is directly applicable to other primitive types.Source: §3.1 · pp. 3–4

3.2. High Resolution Shape Representation

  • The fine-scale representation uses a neural network to learn an SDF without restricting shapes to those constructible from simple primitives.Its zero iso-surface defines the shape surface and can be retrieved with ray marching.Source: §3.2 · p. 4

3.3. Learning a Tightly Coupled Latent Space

  • A variational auto-decoder learns coarse and fine representations for each shape from a shared latent code, encouraging a compact space and tight coupling.The training objective maximizes an ELBO over the dataset; the model uses a mixture model so both representations assign high probability to a shape given its code.Source: §3.3 · pp. 4–5

3.4. Interactive Shape Manipulation

  • Users specify manipulations as objectives over primitive attributes, and gradient descent updates the latent code while unconstrained attributes can adjust freely.Sparse edits can therefore prompt adaptation of other primitives; latent-code regularization favors codes likely under the Gaussian prior, which empirically leads to more plausible shapes.Source: §3.4 · pp. 5–6

4. Experiments

DualSDF is evaluated for reconstruction quality, cross-representation and semantic consistency, and applications that use its coarse primitives as a proxy. Results describe reconstruction as comparable to state of the art, while VAD improves consistency on novel generated shapes.

4. Experiments

  • Reconstruction on unknown shapes is comparable to state-of-the-art techniques, with the VAD objective trading reconstruction performance for a smoother latent space.The comparison includes two AtlasNet variants and DeepSDF; Table 1 reports results for Airplane and Chair collections on unknown and known shapes.Source: §4 · p. 6, Table 1 · p. 6
  • VAD significantly boosts cross-representation consistency on novel generated shapes, although both VAD and auto-decoder models perform similarly on training shapes.The comparison samples shapes from the latent space and measures average volumetric IoU between coarse and fine representations.Source: §4 · p. 7, Table 2 · p. 7
  • Primitive semantic consistency is high across shapes, making primitives intuitive proxies for shape parts, though some must adapt to chair variation such as arm presence.The evaluation assigns labels using closest labeled points and measures agreement with the most frequent labels across the collection; Figure 5 colors primitives by collection-wide consistency.Source: §4 · pp. 7–8, Table 3 · p. 7, Figure 5 · p. 7
  • The framework accommodates other SDF-representable primitive types, including capsules, beyond the spherical primitives used for the illustrated results.The supplementary material also presents box primitives.Source: §4 · p. 8, Figure 7 · p. 8

4.1. Applications

  • Controllable interpolation optimizes coarse primitive attributes between two or more shapes, enabling selective changes such as interpolating only a chair’s height.This differs from shape manipulation, which operates on one shape; the method also supports ordinary interpolation through latent codes.Source: §4.1 · p. 8
  • The coarse proxy supports correspondence by mapping high-resolution surface points to their closest primitives, whose corresponding regions can be propagated across structurally different airplanes.Figure 6 illustrates consistently mapped primitive regions in Airplane reconstructions.Source: Figure 6 · p. 7, §4.1 · p. 8
  • Real-time rendering of primitives enables smooth exploration of the generative shape space before switching to slower high-resolution rendering for a selected shape.Source: §4.1 · p. 8

5. Conclusions

DualSDF represents shapes at fine and coarse granularities, coupling expressive detail with a semantic primitive decomposition. Its shared latent space supports faithful propagation of coarse edits to fine shapes.

5. Conclusions

  • DualSDF represents shapes at two levels of granularity, with an expressive fine-scale representation and a coarse primitive-based semantic decomposition.Source: §5 · p. 8
  • The two representations are tightly coupled, allowing modifications to the coarse representation to be faithfully propagated to the fine-scale shape.Source: §5 · p. 8
  • A variational autodecoder regularizes the shared latent space to support generation, manipulation, and coupling.Source: §5 · p. 8

Supplementary Material for DualSDF: Semantic Shape Manipulation using a Two-Level Representation

The supplementary material demonstrates coarse reconstructions learned with box primitives.

Supplementary Material for DualSDF: Semantic Shape Manipulation using a Two-Level Representation

  • Box primitives can be used to learn coarse shape reconstructions in the framework.Source: Figure 8 · p. 12

A. DualSDF with Box Primitives

The framework supports box primitives and can incorporate any primitive representable with a signed distance function.

A. DualSDF with Box Primitives

B. Effect of Latent Code Regularization on Shape Manipulation

Latent-code regularization favors plausible manipulation results by keeping codes near the prior and guiding edits toward semantically meaningful configurations.

B. Effect of Latent Code Regularization on Shape Manipulation

  • High-prior-likelihood latent codes usually decode to more plausible shapes.The regularization term keeps the code from deviating too far from the prior during manipulation.Source: B. Effect of Latent Code… · p. 12
  • Regularization guides user input toward meaningful shapes by moving unconstrained primitives into appropriate positions.It also guards against unreasonable configurations.Source: B. Effect of Latent Code… · p. 12

C. Analysis of Running Time

Primitive attributes and latent-code updates run quickly enough for real-time feedback in the stated single-user GPU setting. Fine-scale rendering is slower, so resolution and ray-marching steps can be traded for speed during interaction.

C. Analysis of Running Time

  • 0.66 ms obtains primitive attributes from the latent code, and 2.74 ms performs one gradient-descent update after user input.The benchmark uses a single GTX 1080ti GPU and batch size one; the authors report real-time feedback despite a less-than-optimal implementation.Source: C. Analysis of Running Time · p. 12
  • For high-resolution SDF rendering with ray marching, image resolution and ray-marching-step count are the two parameters determining render quality and speed, and Figure 11 presents their effects on both.Main-paper renderings use 64 iterations at 480 × 480 resolution.Source: C. Analysis of Running Time · p. 12
  • Fine-scale rendering is slower than primitive-based rendering at full resolution and highest quality.During interaction, reduced-resolution rendering can accompany real-time primitive rendering, with full resolution rendered when needed.Source: C. Analysis of Running Time · p. 13

D. Detailed Experimental Setting

The experimental setup uses a shared 128-dimensional latent code to decode fine SDF values and coarse sphere attributes. Training uses truncated SDF losses for both representations, with distinct truncation treatment inside and outside the shapes.

D. Detailed Experimental Setting

  • A 128-dimensional latent code feeds an 8-layer MLP for fine SDF prediction and a same-architecture network that outputs attributes for 256 spheres.The fine network concatenates the code with a 3D coordinate; the primitive network uses the code alone, with dropout only in the fine network’s hidden-layer outputs.Source: D. Detailed Experimental Setting · p. 13
  • Training uses Adam with β1 = 0.9 and β2 = 0.999, batches of 64 shapes, and 2800 epochs.Per shape, 2048 SDF values are sampled for the high-resolution representation and 1024 for the primitive representation; learning rates are 5e-9 for θ, φ and 1e-8 for µj,σj.Source: D. Detailed Experimental Setting · p. 13

E. SDF Losses

  • Both representations use truncated SDF loss during training and encoding, but the primitive representation truncates only inside the shape.The exterior SDF for primitives is retained because it is guaranteed to be a valid metric.Source: E. SDF Losses · pp. 13–14
  • Higher rendering resolution and more ray-marching iterations improve image quality at the cost of longer rendering time.The model can use lower quality during interactive manipulation and render higher quality afterward.Source: Figure 11 · p. 14

F. Additional Results

Additional results show primitive edits on chairs and airplanes, alongside standard latent interpolation and selective primitive-space interpolation. Selective interpolation can match chosen chair attributes while preserving fine details and other features.

F. Additional Results

  • Single-primitive edits on chair and airplane shapes include dragging a primitive along one axis, while similar methods can handle more complex multi-primitive operations.Figure 12 presents additional manipulation results; full sequences are in the accompanying video.Source: F. Additional Results · p. 14, Figure 12 · p. 15
  • The results contrast linear interpolation of latent codes with selective interpolation that optimizes primitive parameters.The selective method can interpolate chosen characteristics, such as a shape’s outline; Figure 13 shows latent-code interpolation and Figure 14 shows selective primitive-space interpolation.Source: F. Additional Results · p. 14, Figure 13 · p. 16, Figure 14 · p. 16
  • Selective primitive-space interpolation matches chosen chair characteristics while preserving fine details and other key features.Matching outlines preserves the left chair’s fine detail; matching primitive heights allows other attributes to change freely, and within-class primitive correspondences are generally straightforward.Source: F. Additional Results · p. 14, Figure 14 · p. 16

Related papers

Loading 2004.02869v1…