Source-linked AI summary
4DComplete: Non-Rigid Motion Estimation Beyond the Observable Surface
Yang Li, Hikari Takehara, Takafumi Taketomi, Bo Zheng, Matthias Nießner
TL;DR
Range sensors miss occluded geometry, leaving existing non-rigid tracking limited to visible surfaces. 4DComplete jointly infers hidden shape and motion with a sparse fully-convolutional network trained on DeformingThings4D, and reports mutually beneficial shape–motion estimation with cross-domain generalization. The authors position the method and dataset as a new direction for generic non-rigid 4D reconstruction.
Problem
Occlusions and sensor limitations leave non-rigid motion tracking incomplete because existing methods handle only observable surfaces.
Method
4DComplete jointly recovers missing geometry and motion from partial observations using a sparse fully-convolutional network and the DeformingThings4D dataset.
Results
Shape completion and motion estimation are mutually complementary, with joint learning benefiting both tasks and the method generalizing across domains.
Takeaways & Limitations
The method and dataset open a research avenue for generic non-rigid 4D reconstruction from partial depth observations.
Takeaways & Limitations
The approach does not estimate uncertainty in hidden motion, predict surface colors, or cover loose clothing and plants in its largely articulated dataset.
Abstract
from arXiv · showhide
Tracking non-rigidly deforming scenes using range sensors has numerous applications including computer vision, AR/VR, and robotics. However, due to occlusions and physical limitations of range sensors, existing methods only handle the visible surface, thus causing discontinuities and incompleteness in the motion field. To this end, we introduce 4DComplete, a novel data-driven approach that estimates the non-rigid motion for the unobserved geometry. 4DComplete takes as input a partial shape and motion observation, extracts 4D time-space embedding, and jointly infers the missing geometry and motion field using a sparse fully-convolutional network. For network training, we constructed a large-scale synthetic dataset called DeformingThings4D, which consists of 1972 animation sequences spanning 31 different animals or humanoid categories with dense 4D annotation. Experiments show that 4DComplete 1) reconstructs high-resolution volumetric shape and motion field from a partial observation, 2) learns an entangled 4D feature representation that benefits both shape and motion estimation, 3) yields more accurate and natural deformation than classic non-rigid priors such as As-Rigid-As-Possible (ARAP) deformation, and 4) generalizes well to unseen objects in real-world sequences.
1. Introduction
4DComplete addresses incomplete non-rigid motion observations by jointly completing hidden geometry and estimating motion from partial data. Its data-driven design and DeformingThings4D dataset support shared shape–motion representations and improved performance over existing baselines.
- Occlusions restrict single-view range sensors and existing tracking methods to visible surfaces, while complete motion is needed for high-level tasks.The paper gives nursing-robot care as an example requiring whole-body shape and motion.
- Hand-crafted deformation priors require heavy tuning, may not reflect natural deformations, and struggle with larger hidden regions.They often assume a continuous surface and are mainly used as local-deformation regularizers.
- 4DComplete jointly recovers missing geometry and motion for seen and unseen regions using a sparse, fully-convolutional neural network.The method is motivated by the entanglement between object shape and its temporal evolution.
- Joint shape and motion estimation produces 4D feature representations that outperform existing baselines by a significant margin.
- The method introduces DeformingThings4D, a large-scale non-rigid 4D dataset with 1,972 animation sequences and 122,365 frames.It provides holistic ground truth including color, optical or scene flow, depth, signed distance representations, and volumetric motion fields.
2. Related Work
Prior work combines iterative non-rigid tracking, deformation regularization, and static-scene completion, but dynamic 4D completion remains under-supported by suitable datasets. 4DComplete connects these directions through joint shape–motion estimation and a synthetic training dataset.
- Non-Rigid Tracking: Non-rigid tracking methods commonly minimize point correspondences iteratively and use ARAP or embedded-deformation regularizers to control ambiguities.
- Shape and Scene Completion: Existing shape-completion methods address partial scans, depth images, rooms, buildings, or implicit scene representations, primarily in static settings.Some implicit approaches also struggle to generalize across object categories with high geometric variety.
- Non-Rigid 4D Datasets: Large-scale deforming-object 4D datasets are difficult to collect because dense motion ground truth requires costly per-point correspondence annotations.
- 4DComplete: 4DComplete uses a shared 4D encoder and parallel shape and motion decoders to predict complete TSDF and VMF outputs from partial inputs.The architecture passes predicted geometry from the shape decoder to corresponding motion and shape layers.
3. Method: 4DComplete
4DComplete represents partial shape and visible motion volumetrically, then jointly completes geometry and motion through a sparse convolutional network. Its shape and motion branches exchange predictions across hierarchy levels and are trained with complementary geometric and motion losses.
- 4DComplete recovers hidden geometry and its motion field from a single-view depth map and inter-frame scene flow.
- The input concatenates a sparse TSDF from the depth map with a volumetric motion field for visible occupied voxels.The volumetric motion field represents 3D motion vectors in R3.
- Scene flow and volumetric motion fields are converted using inverse-distance interpolation and tri-linear interpolation between point and voxel representations.The inverse-distance conversion uses K = 3 neighbors, while the reverse conversion uses the eight neighboring cube corners.
- A shared 4D encoder with sparse convolutions supports high-resolution volumetric shape and motion decoding while processing only surface regions.
- At each hierarchy level, the shape decoder supplies predicted geometry to the motion decoder, while geometry-filtered motion features return to the shape decoder.Occupancy-filtered voxels provide the geometry passed to the next level.
- Shape supervision combines log-transformed TSDF L1 loss with occupancy and TSDF proxy losses, while motion supervision uses L2 and cosine-similarity losses.The log transformation emphasizes accuracy near surface points, and cosine similarity encourages motion directions to match ground truth.
4. DeformingThings4D Dataset
DeformingThings4D is a synthetic dataset of animated humanoids and animals with dense shape and motion supervision. Its pipeline renders multiview RGB-D data and derives complete signed-distance and volumetric-motion fields from animated meshes.
- DeformingThings4D contains animated humanoids and animals with skin meshes, textures, skeletons, and dense 4D supervision.Humanoid motion comes from motion-capture data, while animal skin and motion are designed by computer-graphics experts.
- The dataset includes 147 characters across 31 categories, 1,972 animations, and 122,365 frames.
- Each animated mesh is used to generate per-frame RGB-D maps, inter-frame scene flow, signed-distance fields, and volumetric motion fields.Data generation is performed with Blender scripts.
- 4.1. Data Generation: The generation pipeline samples virtual cameras on a sphere, selects an input view, and computes projective inputs alongside complete multiview ground truth.Complete TSDFs integrate depth images from all virtual cameras, while motion fields blend vertex motion into nearby occupied voxels.
- 4.1. Data Generation: Depth maps use 42 camera viewpoints with Azure Kinect intrinsics, and scene flow is derived from vertex displacements between adjacent 25-fps frames.
- 4.1. Data Generation: Ground-truth signed-distance and motion volumes are generated at four voxel resolutions, from 1.0cm3 through 8.0cm3.Motion ground truth is computed near the mesh surface using K-nearest-neighbor vertices and Dual Quaternion Blending with K = 3.
5. Results
Experiments evaluate motion estimation, shape completion, cross-task synergy, real-world generalization, and limitations. 4DComplete improves hidden-surface deformation quality over ARAP and completes shapes more effectively across synthetic and real-world settings.
- Evaluation Metrics: 3D EPE measures average Euclidean motion-vector error, while ACC measures the fraction of vectors below specified error thresholds.The evaluation reports two ACC metrics using different thresholds.
- Hidden-Surface Motion: The hidden-surface experiment compares rigid fitting, ARAP deformation, Motion Complete, and Motion Complete with ARAP-based post-processing.The input includes the complete shape, visible vertices, and visible scene flow; the goal is predicting hidden-vertex motion.
- Hidden-Surface Motion: Motion Complete achieves lower end-point error than ARAP on most sequences, while post-processing further improves the results despite training only on humanoid motions.The test set contains one humanoid sequence and six animal sequences.
- Hidden-Surface Motion: For deer and dairy-cow sequences, ARAP produces severe neck, head, or stomach distortions, whereas the proposed method yields more natural occluded-surface deformations.The ground-truth deformation is unavailable for registration and is shown only for comparison.
- Shape Completion and Generalization: The method generalizes from synthetic training to real-world RGB-D input and produces more complete, sharper shape completions than IF-Net, including global and local detail in large scenes.The real-world inputs include VolumeDeform and Li et al. sequences; fixed-volume IF-Net loses detail on large scenes.
- Task Synergy: Joint geometry and motion learning improves both tasks: geometry-completion supervision improves visible-surface motion, and motion supervision improves geometry completion.The ablation evaluates each task after removing the other task’s prediction head.
6. Conclusion
The paper presents joint estimation of invisible shape and deformation from partial depth observations. It reports complementary benefits between shape completion and motion estimation, alongside cross-domain generalization enabled by a new animation dataset.
- 4DComplete jointly estimates invisible shape and deformation from partial depth frame observations.
- Shape completion and motion estimation are mutually complementary tasks, with joint learning benefiting each.
- The new animation dataset supports cross-domain generalization for both motion and shape.
- The method and dataset open a research avenue for generic non-rigid 4D reconstruction.