PaperlayerView exact revision on arXiv

Source-linked AI summary

Deformed Implicit Field: Modeling 3D Shapes with Learned Dense Correspondence

Yu Deng, Jiaolong Yang, Xin Tong

arXiv:2011.13650v3cs.CV

TL;DR

Dense correspondence learning is difficult for generic 3D object classes because their shapes can vary substantially in geometry and structure. The paper introduces DIF and DIF-Net, which use a shared template with instance-specific deformation and correction fields to learn shape representations and unsupervised correspondences. The resulting system produces high-fidelity shapes, high-quality dense correspondences, and compelling texture-transfer and shape-editing results.

  • Problem

    Existing methods either focus on consistently topologized categories or learn 3D shape latent spaces without explicitly modeling dense correspondences, while generic classes such as cars and chairs have substantial shape and structure variation.

  • Method

    DIF represents each shape with a shared category template implicit field plus instance-specific deformation and correction fields, and DIF-Net learns these fields and a shape latent space without correspondence or part labels.

  • Results

    DIF-Net produces high-fidelity 3D shapes and high-quality dense correspondences, with compelling texture-transfer and shape-editing results that previous methods cannot achieve.

  • Takeaways & Limitations

    The learned deformation fields provide dense correspondences for 3D shape analysis and manipulation across shapes with structure variation.

  • Takeaways & Limitations

    The representation assumes object instances are mostly composed of a few common patterns or semantic structures.

Abstract

from arXiv · show

We propose a novel Deformed Implicit Field (DIF) representation for modeling 3D shapes of a category and generating dense correspondences among shapes. With DIF, a 3D shape is represented by a template implicit field shared across the category, together with a 3D deformation field and a correction field dedicated for each shape instance. Shape correspondences can be easily established using their deformation fields. Our neural network, dubbed DIF-Net, jointly learns a shape latent space and these fields for 3D objects belonging to a category without using any correspondence or part label. The learned DIF-Net can also provides reliable correspondence uncertainty measurement reflecting shape structure discrepancy. Experiments show that DIF-Net not only produces high-fidelity 3D shapes but also builds high-quality dense correspondences across different shapes. We also demonstrate several applications such as texture transfer and shape editing, where our method achieves compelling results that cannot be achieved by previous methods.

1. Introduction

The paper targets dense correspondence learning for generic 3D object classes with substantial shape and structure variation. It introduces DIF and DIF-Net to jointly model category shapes and unsupervised dense correspondences.

  • Motivation: Existing 3D shape models either rely on consistent topology for correspondence construction or learn latent spaces without explicitly modeling dense correspondences.These limitations motivate a method for more complex object categories.
  • Motivation: Cars and chairs exhibit large shape variations and structure changes that make correspondence construction extremely challenging, even for human annotators.The paper therefore focuses on generic object classes rather than consistently topologized categories.
  • Method: DIF represents each shape with a category-shared template implicit field, a shape-specific 3D deformation field, and a shape-specific scalar correction field.Deformation aligns shapes in a shared 3D space, while correction handles structural variation.
  • Method: DIF-Net jointly learns the fields and shape latent space without correspondence or part labels, using normal consistency, deformation smoothness, and minimal correction constraints.The normal-based constraint supports unsupervised correspondence learning.
  • Results: The method achieves high-quality dense correspondences and compelling texture-transfer and shape-editing results that previous methods cannot achieve.The authors present these results as evidence for applications in 3D shape analysis and manipulation.

2. Related Work

Prior work includes morphable models, neural latent spaces, implicit representations, structured representations, and mesh deformation. The paper distinguishes DIF by combining implicit shape modeling with dense correspondence for shapes whose structures vary.

  • 3D Shape Models with Correspondence: Classical morphable models build correspondences through registration, but typically target categories with consistent topologies such as faces, bodies, and animals.They do not address complex object classes containing structure variations.
  • Learning Shape Latent Space: Deep neural methods learn latent spaces for 3D shapes but generally do not explicitly model dense correspondence among different shapes.The paper positions its method as learning both a latent space and correspondence.
  • Implicit Shape Representation: Implicit-function methods are effective for representing complicated 3D geometry, motivating implicit fields as the paper’s shape-representation basis.Prior work reports advantages over voxel- and mesh-based representations.
  • Structured Shape Representation: Structured representations decompose shapes into primitives and may provide part-level correspondences, but they do not generally model dense correspondences.The paper instead avoids dividing shapes into parts.
  • Shape Deformation: Mesh deformation methods fit shapes while preserving local details, whereas DIF is designed for implicit reconstruction and correspondence reasoning and can handle structure mismatch.Structure mismatch is identified as problematic for previous mesh deformation methods.

3. Approach

DIF represents category shapes with a shared template implicit field plus instance-specific deformation and correction fields. DIF-Net jointly learns shape codes and these fields, using deformation for dense correspondence and uncertainty estimation.

  • Shape latent space: Each object is encoded by a latent shape code, and an auto-decoder jointly learns object codes with the neural shape model.After training, new shapes can be generated by sampling the latent space, while observed shapes can be embedded through inverse optimization.
  • Implicit field representation: Signed distance fields provide a continuous representation whose zero iso-surface defines the shape surface and supports high-fidelity geometry.The field assigns each 3D point a signed scalar indicating distance to the closest surface and inside/outside status.
  • Learning and uncertainty: DIF-Net uses Hyper-Net to generate instance-dependent DIF-Net weights, while correspondence learning is trained with SDF, spatial, correction, and normal-consistency objectives.Normal consistency links corresponding surface points across shapes, and the learned template-space distance provides correspondence uncertainty that reflects structural differences.
  • Deformed implicit field: DIF decomposes each category shape into a shared template implicit field and instance-specific deformation and scalar correction fields.The template captures common category structures, while deformation and correction adapt it to individual shapes.
  • Deformed implicit field: Unlike a conventional object SDF, the learned template field can encode category structure variations through multiple iso-surfaces rather than representing one valid shape.This volumetric representation is intended to support correspondence when shapes have structural discrepancies.
  • Dense correspondence: Deformation maps object points into template space, inducing dense correspondence; correction fields add or remove structures without requiring overly complex deformation.For two shapes, correspondences can be obtained by deforming their surfaces into aligned template space and searching nearest neighbors.

4. Experiments

DIF-Net is evaluated for unseen-shape reconstruction and dense correspondence on ShapeNet categories, with additional visual, ablation, texture-transfer, and shape-editing studies. It reconstructs unseen shapes well, produces superior label transfer, and benefits substantially from its training losses and correction field.

  • Experimental Setup: DIF-Net is trained on car, airplane, chair, and table categories from ShapeNet-V2, with training taking about 4 hours on 8 NVIDIA V100 GPUs per category.All parameters are trained end-to-end using Adam with batchsize 256.
  • Ability of Shape Representation: DIF-Net performs well in representing unseen shapes and is slightly better in numerical reconstruction error than OccNet and DeepSDF.Reconstruction is evaluated on 100 test shapes using chamfer distance and earth mover distance.
  • Ability of Shape Representation: The correspondence-aware DIF-Net has slightly worse reconstruction results than a direct-SDF variant, indicating a moderate representation-capability decrease while enabling dense correspondences.The direct-SDF variant does not model dense correspondence.
  • Learned Dense Correspondence: DIF-Net produces convincing correspondences across structurally different shapes, preserves color patterns, and identifies uncertain regions that reflect shape differences.Figure 8 visualizes semantic-part alignment in template space, while Figure 10 demonstrates texture transfer using the generated correspondences.
  • Learned Dense Correspondence: Label-transfer IoU shows that DIF-Net outperforms AtlasNet, AtlasNet-v2, SIF, ShapeFlow, DualSDF, and closest-point matching by a wide margin.The evaluation uses five labeled source shapes per category and transfers labels to other shapes.
  • Ablation Study: Removing normal or smoothness losses degrades correspondence quality, while removing the correction field harms both shape representation and correspondence quality.Without the normal loss, label-transfer IoU drops; without deformation smoothness, transferred textures become highly distorted.

5. Applications

DIF-Net supports texture transfer and shape editing through learned dense correspondences and latent shape codes. These applications preserve semantic alignment across varied 3D shapes and allow point-based edits, including added structures.

  • 5.1. Texture Transfer: Rich texture patterns transfer to correct semantic areas across various ShapeNet shapes.The reported transfers preserve texture patterns between ground-truth shapes and embedded 3D shapes.
  • 5.2. Shape Editing: Shape editing moves a selected point to a desired position by solving for a new shape code.The optimization preserves correspondence, enforces the edited point to lie on the new surface, and keeps code changes small.
  • 5.2. Shape Editing: The editing formulation can add new structures by selecting a template-space point and a free point in shape space.An example is reported in Figure 11, with further details in the supplementary material.

6. Conclusion

The paper presents DIF and DIF-Net for implicit 3D shape modeling with unsupervised dense correspondences. Experiments and applications demonstrate high-quality generated shapes and correspondences, with future work targeting more generic objects and scenes.

  • 6. Conclusion: DIF represents a class of 3D shapes while providing dense correspondences, and DIF-Net learns them unsupervised through proposed loss functions.The representation and neural model jointly support shape modeling and correspondence learning.
  • 6. Conclusion: Experiments and applications collectively demonstrate high-quality shapes and correspondences generated by the method.The conclusion summarizes the method's performance across its reported evaluations and applications.
  • 6. Conclusion: Future work plans to extend DIF to more generic 3D objects and scenes.

A. More Implementation Details

The implementation trains category-specific DIF-Net models on ShapeNet using sampled surface and free-space points, MLP-based networks, joint losses, and latent-code inference optimization.

  • Data Preparation: Training uses 500K surface points with normals and 500K free-space points with SDF values for each mesh.Surface points are obtained by rendering 100 virtual views; free-space points are sampled within [−1, 1]^3 and signed using virtual-camera depth checks.
  • Network Architecture: Hyper-Net, Deform-Net, and Template Field are implemented as MLPs with category-shared template-field weights.Hyper-Net predicts weights for individual Deform-Net layers; ReLU is used in Hyper-Net and sine activations in Deform-Net and Template Field.
  • Training Details: Models are trained separately for car, airplane, chair, and table using 3K, 3.5K, 4K, and 4K training instances, respectively.Each category model is trained for 60 epochs with 4K surface and 4K free-space points sampled per shape per iteration.
  • Training Details: The model jointly learns latent codes, Hyper-Net Ψ, and Template Field T using the losses in Eq. (15).Latent codes are initialized from N(0, 0.012), while network weights use the initialization schemes specified for Hyper-Net and Template Field.
  • Inference Details: Inference obtains a test shape's latent code by optimizing Eq. (17) for 30 epochs with Adam at a learning rate of 1e−4.Inference uses the same loss-term weights and point-sampling strategy as training.

B. Different Latent Code Regularization

An alternative regularization scheme constrains latent-code posteriors toward a Gaussian distribution rather than only constraining latent-code norms. Training samples perturbed latent codes and adds the corresponding regularization loss.

  • B. Different Latent Code Regularization: The alternative regularizer constrains each latent-code posterior q(αi|Oi) toward a Gaussian distribution using KL divergence.The posterior is represented as a Gaussian whose mean equals αi, with a learnable standard-deviation code σ.
  • B. Different Latent Code Regularization: During training, the Hyper-Net receives eα sampled from N(α, σ^2I) instead of the original latent code α.This stochastic latent-code input is used while training the learnable parameters with the modified loss.
  • B. Different Latent Code Regularization: The modified objective replaces the norm regularizer with Lreg′ while retaining the other losses from Eq. (15).In practice, the Gaussian distribution is set to N(0, 0.012) and w5 is set to 1e2.
  • B. Different Latent Code Regularization: Different structures within a category are recorded in different iso-surfaces of the learned template field.Figure II presents learned template fields for four categories.

D. Learned Shape Latent Space

DIF-Net's learned latent space supports interpolation, retrieval, and sampling of 3D shapes while maintaining meaningful shape similarity and consistent correspondences.

  • Latent Space Interpolation: Interpolated shapes are reasonable, with consistent correspondences between different shapes.Even-position shapes are interpolated from their two neighbors and include color-coded correspondences.
  • Learned Shape Latent Space: Latent-space interpolation, retrieval, and sampling provide complementary ways to explore learned 3D shape structure.The supplied passages describe interpolation consistency, latent similarity, and distribution-based generation.
  • Latent Space Retrieval: Nearest-neighbor retrieval uses Euclidean distance between shape latent codes, placing similar shapes close together.Retrieved shapes are sorted in ascending latent-code distance.
  • Latent Space Sampling: The model captures the 3D-shape distribution and generates new shapes.Sampling results are shown for the model trained with Eq. (II).

E. More Details of Label Transfer

The label-transfer evaluation compares DIF-Net with closest-point, AtlasNet, AtlasNet-v2, SIF, ShapeFlow, and DualSDF using category-specific transfer procedures and qualitative visual results.

  • Source Shapes: Five manually selected source shapes per category characterize varied structures in the shape distribution for label-transfer evaluation.The source shapes are shown in Figure III.
  • Closest Point: The closest-point baseline assigns labels by voting among each point's 10 nearest labeled points in the original shape space.This method performs direct label voting without a learned template.
  • AtlasNet: AtlasNet transfers labels through fitted sphere or 25-square templates, using nearest-neighbor voting on template vertices.The two settings are Atlas-sph. and Atlas-25.
  • AtlasNet-v2: AtlasNet-v2 uses Patch Deformation with MLP adjustment and follows AtlasNet's template-based label-transfer procedure.A separate model is trained for each category.
  • SIF: SIF transfers labels in a 300-dimensional template-coordinate system built from 100 implicit kernels.Target points receive labels through voting among their 10 nearest labeled template-coordinate points.
  • ShapeFlow: ShapeFlow deforms source and target surface points into template space before 10-nearest-neighbor label voting.The evaluation uses the authors' provided chair-category model.
  • DualSDF: DualSDF labels primitives from five source shapes and assigns each target point the label of its closest primitive.The supplied models cover plane and chair categories.
  • Qualitative Results: Figure IV presents the qualitative visual results of the label-transfer task.The caption identifies Figure IV as a quantitative comparison on label transfer.

F. More Details of Shape Editing

Shape editing optimizes a new latent code to move selected points or add structures while preserving original-shape features, with a stated optimization procedure and runtime.

  • Shape Editing: Shape editing initializes the new latent code at the original code and optimizes three weighted terms with weights 1, 1, and 5.Adam uses a 1e −4 learning rate for 1,000 iterations.
  • Shape Editing: The editing optimization takes about 10 seconds.This runtime is reported for the 1,000-iteration Adam optimization.
  • Adding New Structures: Adding a new structure selects p′1 in template space and applies the same optimization process to learn the new shape code.The procedure is illustrated in Figure V.
  • More Editing Results: More editing results show selected points moved to desired positions or new structures added while original-shape features are preserved.Additional examples appear in Figure IX and the main Figure VI-related results.
  • Additional Results: The broader visual results include latent interpolation, shape retrieval, and sampled shapes alongside editing examples.Figures VI–VIII cover interpolation, retrieval, and sampling, respectively.
Loading 2011.13650v3…