PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepDeform: Learning Non-rigid RGB-D Reconstruction with Semi-supervised Data

Aljaž Božič, Michael Zollhöfer, Christian Theobalt, Matthias Nießner

arXiv:1912.04302v2cs.CVcs.AIcs.GR

TL;DR

Non-rigid reconstruction lacks large-scale training data, and self-supervised correspondence generation struggles with highly non-rigid and fast motion. The paper builds a semi-supervised RGB-D dataset and learns correspondence matching for an optimization-based reconstruction pipeline, reporting improvements over methods without learned data terms and self-supervised learning approaches.

  • Problem

    Learning-based non-rigid reconstruction lacks a large-scale training corpus, while tracker-based self-supervision fails on challenging cases such as highly non-rigid deformation and fast motion.

    Source: §1 · p. 2

  • Method

    The paper combines self-supervision with sparse annotations to create dense correspondences, then integrates learned RGB-D matching into an optimization-based reconstruction pipeline.

    Source: §1 · p. 2

  • Results

    The learned data term significantly improves reconstruction quality, and the approach is reported to outperform reconstruction methods without learned data terms and learning-based approaches using only self-supervision.

    Source: §6.2 · p. 6, §7 · p. 8

  • Takeaways & Limitations

    The dataset provides a test set and metrics for evaluating non-rigid matching and reconstruction, supporting quantitative and objective comparison.

    Source: §2 · p. 3, §7 · p. 8

    When an object is close to the background, clutter may be fused into the reconstruction, potentially causing wrong deformation-graph connectivity and tracking failures.

    Source: §6.5 · p. 8

Abstract

from arXiv · show

Applying data-driven approaches to non-rigid 3D reconstruction has been difficult, which we believe can be attributed to the lack of a large-scale training corpus. Unfortunately, this method fails for important cases such as highly non-rigid deformations. We first address this problem of lack of data by introducing a novel semi-supervised strategy to obtain dense inter-frame correspondences from a sparse set of annotations. This way, we obtain a large dataset of 400 scenes, over 390,000 RGB-D frames, and 5,533 densely aligned frame pairs; in addition, we provide a test set along with several metrics for evaluation. Based on this corpus, we introduce a data-driven non-rigid feature matching approach, which we integrate into an optimization-based reconstruction pipeline. Here, we propose a new neural network that operates on RGB-D frames, while maintaining robustness under large non-rigid deformations and producing accurate predictions. Our approach significantly outperforms existing non-rigid reconstruction methods that do not use learned data terms, as well as learning-based approaches that only use self-supervision.

1. Introduction

Non-rigid reconstruction remains challenging, and learning-based methods lack large-scale training data for general real-world scenes. The paper introduces a semi-supervised dataset and learned matching and reconstruction methods aimed at difficult deformations and motion.

1. Introduction

  • The dataset combines self-supervision with sparse user annotations to produce 400 scenes, over 390,000 RGB-D frames, and 5,533 densely aligned frame pairs.Sparse correspondences guide reconstruction to handle challenging motions that defeat tracking-based self-supervision.Source: §1 · p. 2
  • Learning-based data terms for general real-world scenes remain scarce, in part because large-scale training corpora are lacking.Existing self-supervision relies on tracker-generated correspondences and fails in challenging cases such as highly non-rigid deformation and fast motion.Source: §1 · pp. 1–2
  • The authors introduce a data-driven non-rigid correspondence matcher designed to remain robust under difficult real-world deformations, especially extreme deformations.The approach predicts a probability heatmap for a source keypoint’s location in the target frame.Source: §1 · p. 2
  • Their reconstruction approach combines learned and geometric data terms and handles significantly faster and more complex motions than the state of the art.Source: §1 · p. 2
  • The dataset includes varied dynamic sequences with segmentation masks and point correspondences between RGB-D frames.Source: Figure 2 · p. 2

2. Related Work

Prior work covers static RGB-D reconstruction, non-rigid tracking, and learned correspondence matching, but datasets and methods for general non-rigid RGB-D scenes remain limited. This work introduces a large-scale semi-supervised dataset and benchmark for non-rigid matching.

2. Related Work

  • Static RGB-D reconstruction commonly uses TSDF voxel grids or point-based representations, with camera tracking via point-to-plane ICP.Source: §2 · p. 2
  • Earlier template-less non-rigid methods were slow and limited to simple, slow motions, while later work demonstrated real-time dynamic reconstruction.Source: §2 · pp. 2–3
  • Single-camera non-rigid reconstruction methods differ in tracking and deformation formulations; multi-camera methods can improve reconstruction quality but require more complex hardware.Source: §2 · p. 3
  • Learned descriptors are well studied for rigid matching, but existing non-rigid methods are class-specific or lack the corpus needed to handle arbitrary deforming objects.The paper proposes a large-scale corpus and end-to-end learning for non-rigid matching between depth images.Source: §2 · p. 3
  • Existing non-rigid RGB-D datasets are small or scene-specific, and real-world depth recordings often lack ground-truth reconstructions for objective benchmarking.The paper introduces a large-scale semi-supervised dataset and benchmark for objective comparison.Source: §2 · p. 3

3. Data-driven Non-Rigid Matching

The paper learns RGB-D correspondences between source and target frames with a Siamese network. Its outputs include target-location heatmaps, matched-point depth, and visibility, with training losses applied to these predictions.

3. Data-driven Non-Rigid Matching

  • A Siamese network takes two 224 × 224 RGB-D patches, with the source patch centered on a feature point, to match source and target frames.The patches contain three color channels and 3D points in camera coordinates.Source: §3 · p. 3, Figure 3 · p. 4
  • The decoder predicts two target-frame probability heatmaps encoding the likelihood of the source point’s location.The sigmoid heatmap treats pixels as independent binary classifications; the softmax heatmap is a probability distribution that sums to one.Source: §3 · p. 3, Figure 3 · p. 4
  • The network also predicts a visibility score indicating whether the source point is visible or occluded in the target frame.Source: Figure 3 · p. 4, §3 · p. 4
  • The network predicts matched-point depth densely, assigning the same depth prediction to every output pixel and weighting its loss around the ground-truth match.The depth prediction is in the target camera’s coordinate system; an exponential ensures positive depth predictions.Source: §3 · p. 4
  • Training uses a weighted combination of the presented loss functions, including heatmap, depth, and visibility losses.The heatmap loss combines binary cross-entropy and negative log-likelihood after multiplying the sigmoid and softmax heatmaps elementwise.Source: §3 · p. 4

4. Non-Rigid Reconstruction Pipeline

The pipeline integrates learned non-rigid matching into GPU-based tracking and volumetric fusion, representing motion with a coarse deformation graph. Its tracking energy combines geometric, rigidity, and learned correspondence constraints, which are optimized with a data-parallel solver.

4. Non-Rigid Reconstruction Pipeline

  • The pipeline tracks dense, space-time coherent non-rigid motion while reconstructing a canonical TSDF model and fusing new observations with an exponential moving average.The tracking and volumetric fusion backend operate in parallel.Source: §4 · p. 4

4.1. Deformation Model

  • A coarse graph of K deformation nodes represents motion through convex combinations of local transformations, with 2K free variables independent of scene complexity.Each node has a rotation parameter in Lie algebra space and a translation vector.Source: §4.1 · p. 5

4.2. Optimization Terms

  • Tracking combines dense point-to-point and point-to-plane constraints, ARAP regularization, and learned heatmap correspondences filtered by visibility and depth.Correspondences are rejected below visibility 0.5 or when depth differs by more than 0.15 meters.Source: §4.2 · p. 5

4.3. Energy Optimization

  • A data-parallel Gauss-Newton solver minimizes the total energy, using preconditioned conjugate gradient to solve its sequence of linear problems.The optimized quantity is the deformation graph that best explains the data.Source: §4.3 · p. 5

5. Semi-supervised Data Acquisition

The data acquisition process builds training and evaluation data for non-rigid matching and reconstruction by combining RGB-D capture, sparse annotations, and dense frame-pair alignment.

5.1. Data Acquisition

  • The dataset contains 400 scenes and over 390,000 RGB-D frames captured with an iPad-mounted Structure Sensor.Depth was recorded at 640 × 480 and 30 frames per second; calibrated RGB was captured at 1296 × 968 pixels.Source: §5.1 · p. 5

5.2. Data Annotation

  • Annotators labeled object masks in up to 10 frames per sequence and about 20 uniformly distributed point matches per selected frame pair.Up to 10 frame pairs were selected among annotated frames; occlusion points were also labeled for visibility detection.Source: §5.2 · p. 5

5.3. Dense Data Alignment

  • Dense alignment uses a sparse deformation graph defined on the source depth map and restricted to the dynamic object by its mask.The optimized alignment energy includes color-gradient matching, target-mask silhouette penalties, and constraints enforcing annotated sparse matches.Source: §5.3 · pp. 5–6
  • Forward and backward alignment with motion interpolation addresses simple apparent topology changes, followed by a manual review that removes misaligned mesh parts when needed.The review takes about 30 seconds per frame.Source: §5.3 · p. 6

6. Experiments

The experiments assess learned non-rigid matching and reconstruction against existing methods, then examine network components, training data, and limitations. The learned approach improves matching and reconstruction, while dense semi-supervised data supports performance on more extreme deformations.

6. Experiments

  • The dataset split contains 340 training, 30 validation, and 30 test sequences, with no captured-environment overlap between training and validation or test scenes.Matching is evaluated using average 2D pixel and 3D point errors and accuracy within 20 pixels or 0.05 meters.Source: §6 · p. 6, §6.1 · p. 6

6.1. Non-Rigid Matching Evaluation

  • Our non-rigid matcher consistently outperforms the hand-crafted and learned baselines across the reported matching evaluation.The evaluation reports average 2D pixel and 3D point errors, plus accuracy within 20 pixels or 0.05 meters.Source: Table 1 · p. 6, §6.1 · p. 6

6.2. Non-Rigid Reconstruction Results

  • 21.05 cm deformation error and 14.87 cm geometry error for MonoFVV on two sequences compare with 3.63 cm and 0.48 cm, respectively, for our approach.Table 2 compares state-of-the-art reconstruction approaches; the authors report that their approach outperforms the state of the art by a large margin.Source: §6.2 · p. 6, Table 2 · p. 7

6.3. Ablation Study

  • Visibility-based correspondence pruning improves robustness to occlusion, and depth-based filtering further improves reconstruction when depth measurements or heatmaps are noisy.Figure 6 describes rejecting correspondences with visibility scores below 0.50.Source: §6.3 · p. 7, Figure 6 · p. 8

6.4. Data Generation Evaluation

  • Rigid-only training data does not generalize to non-rigid sequences; sparse matches help but are insufficient, while self-supervision improves on sparse-only training yet degrades on extreme deformations.Dense semi-supervised alignment supplies matches for more challenging deformations; Figure 7 compares prediction from self-supervised data with prediction from dense data.Source: §6.4 · p. 7, Figure 7 · p. 8
  • Dense aligned pairs spanning distant frames and oversampling more deformable samples improve prediction, while using 12.5%, 25.0%, or 50.0% of training data considerably degrades matching accuracy.The sampling strategy weights matches by how much their motion differs from the estimated average rigid motion and improves performance over non-weighted sampling.Source: §6.4 · pp. 7–8

6.5. Limitations

  • Background clutter near an object can be fused into its reconstruction, causing shape growth, incorrect deformation-graph connectivity, and tracking failures.The authors suggest background subtraction as a possible future direction.Source: §6.5 · p. 8

7. Conclusion

The paper presents RGB-D correspondence learning and a semi-supervised dataset of 400 scenes, over 390,000 frames, and 5,533 densely aligned pairs. It also provides a test set and evaluation metrics for non-rigid matching and reconstruction.

7. Conclusion

  • The dataset combines 400 scenes, over 390,000 RGB-D frames, and 5,533 densely aligned frame pairs, generated by combining self-supervision with sparse annotations.The authors also provide a test set and metrics for evaluating non-rigid matching and reconstruction.Source: §7 · p. 8
  • The proposed RGB-D network learns non-rigid correspondences, and its descriptors outperform existing hand-crafted descriptors.Source: §7 · p. 8

Appendix

  • The appendix covers the online benchmark, network and training details, GPU solver, dataset statistics, dense alignment, and additional qualitative results.Source: Appendix · p. 12

A. Online Benchmark

The online benchmark evaluates optical flow estimation and non-rigid reconstruction using the paper’s evaluation metrics. The following network-details section describes the architecture and training process.

A. Online Benchmark

  • The online benchmark evaluates optical flow estimation and non-rigid reconstruction using the metrics described in the paper.Source: A. Online Benchmark · p. 12
  • Optical-flow evaluation reports average pixel error and pixel accuracy, defined as the fraction of pixels with less than 20-pixel error.Source: A. Online Benchmark · p. 12
  • Non-rigid reconstruction is evaluated using average deformation and geometry error in centimeters.Source: A. Online Benchmark · p. 12

B. Network Details

B.1. Architecture Details

The matching network processes RGB-D patches through shared encoders and decodes features into correspondence, depth, and visibility predictions. Training proceeds in two stages, while reconstruction optimization uses a data-parallel Gauss–Newton and PCG solver.

B.1. Architecture Details

  • The Siamese network takes 224 × 224 RGB-D patches and uses shared encoders, a bottleneck, and skip-connected decoding to predict a correspondence heatmap, depth, and visibility.The encoder produces 7 × 7 × 96 features; the decoder predicts 224 × 224 heatmap and depth outputs.Source: §B.1 · p. 12

B.2. Training Details

  • Training first optimizes heatmap and depth prediction for 100,000 iterations, then trains visibility layers for 100,000 more with encoder and bottleneck weights fixed.Training uses on-the-fly rotation, translation, and horizontal-flip augmentation.Source: §B.2 · p. 12

C. Least Squares GPU Solver

  • The method performs 10 non-linear Gauss-Newton optimization steps, first linearizing vector field F with a first order Taylor expansion, then solving the resulting normal equations using a data-parallel preconditioned conjugate gradient (PCG) solver.Source: C. Least Squares GPU Solver · p. 13

B.1. Architecture Details

C. Least Squares GPU Solver

  • Each PCG step materializes the Jacobian in GPU memory and computes per-residual contributions to the system matrix product, summing them for each unknown with atomic operations.Source: C. Least Squares GPU Solver · p. 14

D. Dataset Statistics

The dataset separates training, validation, and test sequences across distinct captured environments and includes varied dynamic objects, deformations, and camera motion. Dense alignment combines geometric, sparse-match, silhouette, and color constraints to generate frame correspondences.

D. Dataset Statistics

  • The dataset split contains 340 training, 30 validation, and 30 test sequences, with no captured-environment overlap between training and validation or test scenes.Source: D. Dataset Statistics · p. 14

E. Dense Alignment Details

  • Dense alignment extracts a masked source mesh, samples a deformation graph, and estimates node transformations to align the source mesh to the target.Source: E. Dense Alignment Details · p. 14
  • Dense ICP constraints alone can yield poor frame-to-frame alignment, while adding sparse annotated-match constraints gives a substantial improvement.Source: E. Dense Alignment Details · p. 14
  • A silhouette constraint encourages projected source vertices to lie inside the target mask, using a pixel distance map that is zero inside the mask and increases outside it.Source: E. Dense Alignment Details · p. 15, Figure 17 · p. 15
  • Dense color constraints can correct alignment when useful texture is present, using color gradients to remain invariant to constant intensity changes.Source: E. Dense Alignment Details · p. 15

F. Qualitative Results

The qualitative results show heatmap predictions under highly non-rigid motion, a reconstruction comparison with KillingFusion, and reconstructions in both current and canonical frames.

F. Qualitative Results

  • Figure 18 shows that the approach works well on heatmap predictions for highly non-rigid motions.Source: Figure 18 · p. 16
  • Figure 19 qualitatively shows more robust, higher-quality reconstructions than KillingFusion [38].The comparison is qualitative.Source: Figure 19 · p. 17
  • Figure 20 presents high-quality reconstructions of both the model warped into the current frame and the canonical shape in the reference frame.Source: Figure 20 · p. 18

Related papers

Loading 1912.04302v2…