Source-linked AI summary
VolumeDeform: Real-time Volumetric Non-rigid Reconstruction
Matthias Innmann, Michael Zollhöfer, Matthias Nießner, Christian Theobalt, Marc Stamminger
TL;DR
The paper addresses real-time reconstruction and tracking of deformable scenes without a pre-defined template. It uses a unified volumetric model with sparse color and dense depth constraints, achieving robust tracking at real-time rates, including fast motion and scenes lacking geometric features.
Problem
Existing dynamic reconstruction methods either do not handle non-rigid scene elements or require a static template, while template-free approaches had not achieved real-time performance.
Method
The method jointly represents geometry and deformation volumetrically, combines sparse SIFT correspondences with dense depth constraints, and solves deformation with hierarchical data-parallel optimization.
Results
The approach demonstrates robust real-time non-rigid reconstruction, including fast motion and scenes that lack geometric features.
Takeaways & Limitations
Template-free volumetric reconstruction can track and build models of arbitrary non-rigidly deforming scenes at real-time rates.
Takeaways & Limitations
Drift is not completely eliminated, high deformation can cause problems, and the uniform grid models a relatively small spatial extent.
Abstract
from arXiv · showhide
We present a novel approach for the reconstruction of dynamic geometric shapes using a single hand-held consumer-grade RGB-D sensor at real-time rates. Our method does not require a pre-defined shape template to start with and builds up the scene model from scratch during the scanning process. Geometry and motion are parameterized in a unified manner by a volumetric representation that encodes a distance field of the surface geometry as well as the non-rigid space deformation. Motion tracking is based on a set of extracted sparse color features in combination with a dense depth-based constraint formulation. This enables accurate tracking and drastically reduces drift inherent to standard model-to-depth alignment. We cast finding the optimal deformation of space as a non-linear regularized variational optimization problem by enforcing local smoothness and proximity to the input constraints. The problem is tackled in real-time at the camera's capture rate using a data-parallel flip-flop optimization strategy. Our results demonstrate robust tracking even for fast motion and scenes that lack geometric features.
1 Introduction
The paper targets real-time reconstruction and tracking of deformable scenes without requiring a pre-scanned template. It combines a unified volumetric representation, sparse RGB features, and data-parallel optimization to improve non-rigid tracking.
- Static reconstruction methods do not handle dynamically non-rigid scene elements.
- Template-based deformable tracking requires a complete static shape template, limiting applicability in general scenes and making preparation difficult.
- Template-free methods jointly build shape models and track non-rigid deformations from partial scans, but earlier approaches did not achieve real-time performance.
- The method addresses drift and tangential-motion limitations by adding sparse RGB feature matching to depth-based tracking and using an alternative deformation representation.
- The main contributions are a unified volumetric representation for geometry and motion, global sparse SIFT correspondences, and data-parallel optimization for non-rigid alignment.
2 Related Work
Prior work spans static RGB-D reconstruction, non-rigid deformation tracking, and template-based or monocular approaches. Real-time deformable tracking existed, but template-free real-time reconstruction remained limited.
- Commodity RGB-D reconstruction methods include point-based, mesh, and implicit surface representations, with KinectFusion enabling real-time dense static reconstruction.
- Non-rigid tracking has used multi-camera systems, templates, monocular RGB video, and posture compensation for partial scans.
- Template-based real-time deformable tracking of simple motions requires a KinectFusion reconstruction of a static template beforehand.
3 Method Overview
The method processes a 30Hz RGB-D stream by matching the current observation to a deformed model, optimizing its deformation, and integrating the new data.
- 30Hz input consists of aligned color and depth maps recorded at 640×480 pixels.
- A current deformed mesh is extracted from the signed distance field using Marching Cubes and rendered to produce dense depth correspondences.
- Sparse SIFT matches between the current frame and previous frames complement dense depth correspondences for model alignment.
- The deformation field is optimized so the model explains the current depth and color observations, after which the current RGB-D data is integrated.
4 Scene Representation
Geometry and deformation are represented jointly on a shared regular volumetric grid. The scene is stored in an undeformed canonical frame while a deformation field maps it to the current pose.
- The representation combines a TSDF for the initial undeformed shape with a space deformation field on a shared regular volumetric grid.
- Each grid point stores signed distance, color, confidence, deformed position, and local rotation attributes.
- The TSDF zero level set defines the canonical pose, while new depth data is fused into that undeformed frame using confidence-weighted updates.
- The finer grid-based deformation representation uses trilinear interpolation and extracts the surface with data-parallel Marching Cubes before applying deformation.
5 Correspondence Association
The method combines dense depth correspondences with sparse SIFT color matches to track non-rigid surfaces while improving stability and reducing drift. GPU-parallel feature processing supports real-time correspondence association.
- Depth Correspondence Association: Dense projective lookup produces depth correspondences between the current deformed surface and each new depth frame.The approach rasterizes a mesh-based isosurface representation before projecting sample points into the current depth map.
- Color Correspondence Association: Sparse SIFT color correspondences complement depth constraints when depth features are insufficient for robust tracking.Feature points are lifted to 3D and stored in the canonical pose for subsequent matching.
- Correspondence Filtering: Correspondences are pruned using distance, normal, and view-direction criteria to improve optimization stability.Silhouette-adjacent points receive zero confidence when they fail the stated thresholds.
- Tracking Robustness: Combining dense and sparse correspondences improves stability and reduces drift.The sparse color constraints are designed to strengthen tracking beyond depth-only alignment.
- GPU Feature Processing: GPU-parallel SIFT processing detects features, extracts 128-dimensional descriptors, and matches features across previous frames.Descriptor extraction takes about 6ms at 640 × 480 resolution, with at most 150 keypoints per image and 64 retained correspondences per image pair.
6 Deformation Energy
Deformation is estimated by minimizing a regularized energy that combines sparse feature alignment, dense point-to-plane depth alignment, and an ARAP prior. The formulation favors smooth, locally rigid deformations while remaining close to current observations.
- Deformation Variables: The deformation variables stack 3|G| position coordinates and 3|G| rotation angles into a single optimization vector.The unknown local deformation parameters are represented together for the registration problem.
- Registration Objective: The total objective combines sparse feature alignment, dense depth alignment, and a regularizing prior with fixed weights.The data terms enforce proximity to current input, while the prior favors smooth and locally rigid deformations.
- Dense Depth Alignment: Dense point-to-plane alignment matches the current surface to captured depth and permits sliding under translational object or camera motion.The metric is described as a first-order approximation of the real surface geometry.
- Sparse Feature Alignment: Sparse feature alignment constrains canonical SIFT feature positions toward their current world-space positions and improves robustness against temporal drift.The paper states that this term also allows tracking fast motions.
- ARAP Regularization: The ARAP prior measures residual non-rigid deformation over one-ring grid neighborhoods and minimizes that component.The optimization set M contains the isosurface and its one-ring neighborhood.
7 Parallel Energy Optimization
The paper solves the high-dimensional non-linear tracking objective with a hierarchical, data-parallel flip-flop optimizer. Alternating position and rotation updates make the subproblems efficient, while multi-resolution processing addresses slow fine-grid propagation.
- Problem Formulation: The optimization contains 6N unknowns, with three position and three rotation parameters for each grid point in the isosurface neighborhood.Only values in the one-ring neighborhood M are optimized.
- Flip-Flop Optimization: The flip-flop strategy alternates position optimization with rotation optimization until convergence.Rotations are fixed while positions are updated, then positions are fixed while rotations are updated.
- Rotation Update: Rotation updates use independent closed-form Procrustes solutions that can be computed in parallel for all grid points.The rotation subproblem is equivalent to shape matching.
- Position Update: Position updates solve a quadratic optimization problem by setting derivatives to zero and solving the resulting linear system.The system uses Laplacian and constraint matrices and is solved with a GPU data-parallel preconditioned conjugate-gradient solver.
- Hierarchical Solver: Hierarchical optimization alleviates slow propagation of low-frequency updates on fine-resolution grids.Coarse grids propagate updates globally in few steps, whereas finer grids require a multi-resolution strategy.
8 Fusion
Each RGB-D frame is incrementally fused into a canonical TSDF using non-rigid fusion. The method restricts integration to optimized voxels around the current isosurface, and demonstrates reconstructions of varied non-rigid scenes at real-time rates.
- TSDF Fusion: Depth data from each RGB-D frame is incrementally fused into the canonical TSDF using a non-rigid fusion technique.The approach generalizes projective truncated signed distance function integration.
- Selective Integration: Integration is restricted to voxels in the isosurface one-ring that were included in optimization for the current frame.This differs from defining the warp field through the entire canonical frame.
- Reconstruction Results: Figure 3 shows a variety of non-rigid scenes reconstructed with the approach at real-time rates.The listed examples include upper body, sunflower, and hoodie scenes.
9 Results
The method reconstructs deformable objects in real time from scratch, with sparse color correspondences improving tracking where depth-only alignment drifts or fails. It achieves stable tracking, comparable or higher quality than template-based and DynamicFusion methods, and remains stable to approximately 3× motion speed.
- All reconstructions run in real time on a commodity desktop PC without pre-computation or a pre-scanned template.The scene model is built from scratch during capture.
- Sparse color features successfully track and reconstruct tangential motion and scenes lacking geometric features, where depth-only alignment drifts or fails.Color features provide global anchor points for robust tracking.
- The reconstruction has 1mm average error in non-occluded regions against Li et al.'s static template; occluded regions cannot be reconstructed.The comparison uses geometric distance from the final canonical reconstruction to the first-frame template mesh.
- Compared with Zollhöfer et al.'s template-tracking method, the on-the-fly reconstruction achieves similar or even higher quality.The comparison concerns a real-time template-based approach.
- Compared with DynamicFusion, the method achieves at least comparable or higher quality, with a higher-quality canonical pose attributed to higher warp-field resolution and sparse feature alignment.The higher-resolution field preserves fine-scale deformation detail, while sparse features mitigate drift and improve tracking robustness.
- Tracking remains stable up to approximately 3× speed; higher speedups cause tracking failures and reconstruction errors.The evaluation subsamples input frames to simulate 2×–6× motion.
10 Limitations
The method remains stable up to a 3× frame-skipping speedup, but tracking degrades beyond that and is sensitive to deformation-grid resolution and tracking cues. The authors also identify residual drift, difficulty with high deformation, and limited spatial extent as limitations.
- Method limitations: Drift is not completely eliminated, and high deformations such as fully bending a human arm may cause problems.The smooth regularizer distributes deformations over the grid, while a global bundle-adjustment solution exceeds the real-time budget.
- Scope limitations: A uniform grid models only a relatively small spatial extent, motivating combination with sparse surface reconstruction as a future direction.The authors also propose adaptive strategies for locally adjusting rigidity to address high deformation.
- Temporal coherence: Tracking remains stable up to a 3× speedup, but tracking quality degrades beyond this point.The SUNFLOWER sequence skips every nth frame to test temporal coherence.
- Resolution and tracking: Low-resolution warp fields cannot capture fine-scale deformations, causing drift and blur, whereas high-resolution grids mitigate drift and preserve sharp textures.The comparison also shows that depth-only tracking produces drift and blur, while the combined approach performs better.
- Resolution and tracking: Depth-only tracking can drift or fail when geometric features are absent, while sparse color features improve tracking in such cases.The combined tracker is most effective with a high-resolution deformation grid.
11 Conclusion
The paper presents a unified volumetric approach that jointly reconstructs geometry and motion for arbitrary non-rigid scenes at real-time rates. It combines sparse color and dense depth constraints with GPU-based optimization, and demonstrates reconstruction even when geometric features are absent.
- Conclusion: The method jointly reconstructs the geometric shape and motion of an arbitrary non-rigidly deforming scene at real-time rates.Its unified volumetric representation encodes both geometry and motion.
- Conclusion: Motion tracking combines sparse color and dense depth constraints in a fast GPU-based variational optimization strategy.This supports the joint reconstruction and tracking formulation.
- Conclusion: The results demonstrate non-rigid reconstruction even for scenes that lack geometric features.The authors identify applications in VR and AR involving arbitrary non-rigidly deforming objects.