Source-linked AI summary
Capturing Hands in Action using Discriminative Salient Points and Physics Simulation
Dimitrios Tzionas, Luca Ballan, Abhilash Srikantha, Pablo Aponte, Marc Pollefeys, Juergen Gall
TL;DR
Hand motion capture has largely focused on isolated hands, whereas this paper targets hand-hand and hand-object interactions. It combines generative tracking with discriminative salient points, collision detection, and physics simulation in a unified objective for RGB-D and multi-view RGB data. Evaluations on 29 sequences report higher hand-pose accuracy than particle-swarm optimization and successful tracking with highly articulated objects.
Problem
Recent hand-capture approaches mainly address a single isolated hand, despite hands commonly interacting with other hands or objects.
Method
The framework combines a generative model, jointly associated discriminative salient points, collision detection, and physics simulation in one objective optimized with standard techniques.
Results
The approach achieved higher hand-pose accuracy than particle-swarm optimization across RGB and RGB-D camera systems and successfully tracked hands interacting with highly articulated objects.
Takeaways & Limitations
The framework captures hand and manipulated-object motion during intense interactions from monocular RGB-D and multiple synchronized RGB videos.
Takeaways & Limitations
The approach fails on very fast motion, assumes an available hand model, and can produce inaccurate poses when depth data or fingertip detections are missing.
Abstract
from arXiv · showhide
Hand motion capture is a popular research field, recently gaining more attention due to the ubiquity of RGB-D sensors. However, even most recent approaches focus on the case of a single isolated hand. In this work, we focus on hands that interact with other hands or objects and present a framework that successfully captures motion in such interaction scenarios for both rigid and articulated objects. Our framework combines a generative model with discriminatively trained salient points to achieve a low tracking error and with collision detection and physics simulation to achieve physically plausible estimates even in case of occlusions and missing visual data. Since all components are unified in a single objective function which is almost everywhere differentiable, it can be optimized with standard optimization techniques. Our approach works for monocular RGB-D sequences as well as setups with multiple synchronized RGB cameras. For a qualitative and quantitative evaluation, we captured 29 sequences with a large variety of interactions and up to 150 degrees of freedom.
1 Introduction
The paper addresses hand motion capture during hand-hand and hand-object interactions, extending beyond the field’s predominant focus on isolated hands. It combines generative tracking, discriminative salient points, collision handling, and physics simulation in one optimizable framework.
- Hand tracking remains difficult because hands have similar parts, challenging palm and forearm models, high-dimensional skeletons, and frequent self-occlusions.
- Prior research mainly focused on a single isolated hand, although human hands primarily interact with environments and manipulate objects.
- The proposed objective combines data alignment terms with collision and physical models, replacing sampling-based optimization with standard optimization techniques.
- Discriminatively learned salient points improve local optimization, while their associations are solved jointly with hand pose estimation.
- The unified framework supports monocular RGB-D and multiple synchronized RGB cameras, and evaluates interactions across 29 sequences, including two hands interacting with non-rigid objects.
2 Related Work
Related work spans generative and discriminative hand-tracking methods, with different optimization strategies and image cues. The paper combines these traditions by using a generative model with discriminative salient points and modality-specific observations.
- Markerless hand-motion capture is pursued because gloves, markers, and wearable sensors simplify tracking but require intrusive instrumentation.
- Generative methods explicitly generate pose hypotheses and evaluate their discrepancies against observed data, typically initializing each iteration from the previous pose.
- Discriminative methods learn direct mappings from image features to pose parameters, often operating on single frames but requiring costly realistic training data.
- This work combines generative tracking with discriminative salient points, using edges and optical flow for RGB sequences and depth for RGB-D sequences.
3 Pose Estimation
The framework models hands and manipulated objects consistently across monocular RGB-D and multi-view RGB sequences. Its main modality-specific change is the data term: depth-based alignment for RGB-D versus image-based alignment for RGB.
- The same hand and object models are used for RGB-D and multi-view RGB sequences, while the data term depends on the available modality.
- For RGB-D sequences, the objective is introduced using depth data, whereas RGB sequences use edges and optical flow instead.
3.1 Hand and Object Models
Hands and objects are represented as skinned meshes driven by kinematic skeletons, with rigid and articulated motion parameterized through global transformations and revolute joints. The models cover interactions reaching 150 degrees of freedom.
- The hand and object templates use triangular meshes, kinematic skeletons, and skinning weights to deform geometry according to pose parameters.
- The tracked models include rigid balls and cubes, a 7-DoF pipe, and a 76-DoF rope with 70 revolute joints.
- Each hand has 20 skinning bones; the pipe has 2 and the rope has 36, defining mesh segmentation through skinning weights.
- Rigid objects use a global 6-DoF transformation, while articulated objects add revolute-joint parameters to their pose vectors.
- Two interacting hands require 74 DoF, while adding the rope increases the modeled interaction to 150 DoF.
3.2 Objective Function
The objective function aligns template meshes with preprocessed depth data while incorporating salient points, collision and physics terms, anatomical limits, and temporal regularization.
- The objective function contains seven terms covering model-data alignment, salient-point consistency, collision penalties, physics, anatomy, and temporal regularization.The pose parameters are θ, and D denotes the current preprocessed depth image.
- Depth alignment uses complementary model-to-data and data-to-model terms to fit the posed mesh and explain the observed depth data.The model-to-data term fits visible mesh vertices to the depth point cloud, while the data-to-model term prevents degenerate self-occluded solutions.
- Salient-point consistency helps the generative model recover from tracking errors, while collision and physics terms enforce physically plausible interaction poses.The collision term penalizes finger intersections, and the physics term enhances grasping realism.
- Anatomical joint limits constrain poses, and temporal regularization favors the previous frame when observations are insufficient.
- The model-to-data alignment can use either point-to-point or point-to-plane distances, both evaluated in the experiments.Point correspondences are filtered using normal-angle and distance thresholds before computing the alignment error.
3.2.3 Fitting the data to the model - LOd2m:
The data-to-model term explains observed depth points with the posed model, while collision modeling prevents intersections and supports physically plausible interaction estimates.
- Fitting the data to the model - LOd2m: The data-to-model term prevents degenerate poses by matching observed depth data to the posed model, complementing model-to-data fitting.This matching is more expensive because correspondences or data structures may need updating after each pose iteration.
- Fitting the data to the model - LOd2m: Depth-discontinuity matching increases runtime by 40% or more with a kd-tree without reducing the pose estimation error.
- Collision detection - C: Collision detection finds colliding triangles with bounding volume hierarchies and penalizes penetration using local 3D distance fields.Local fields avoid recomputing a signed distance field for the whole mesh at every optimization iteration.
- Collision detection - C: Local distance-field penalties increase with intrusion and add across multiple collisions, discouraging unrealistic mesh intersections.The field is defined around a receiving triangle using a cone based on its circumcenter, normal, and circumcircle radius.
- Collision detection - C: The collision penalty assigns each penetrating vertex a repulsion correspondence directed opposite the receiving triangle’s normal.Its intensity is determined by the distance-field function Ψ.
3.2.5 Salient point detection - S
Discriminatively detected fingertips supplement the generative model by providing sparse, robust pose constraints that are associated with model fingertips through integer programming.
- A discriminatively trained salient-point detector compensates for the generative model’s slow recovery from ambiguities and tracking errors.
- The fingertip detector is trained on raw depth data from 56 annotated sequences, using 32 sequences for training and 24 for testing.The approximately 2000 annotated frames support a Hough forest trained with positive and negative patches.
- High-confidence detections are retained because salient points provide additional robustness and usually need to be sparse.The experiments use a confidence threshold of cthr = 3.0.
- Integer programming assigns detections to model fingertips while allowing false positives and unassigned fingertips with weighted costs.Binary assignment variables satisfy one-per-row and one-per-column constraints, as summarized in Table 1.
- Assignment costs use 3D distances between detected-region centroids and visible model-fingertip centroids, with optional detection-confidence weighting.When localization is sufficiently close, correspondences are skipped; otherwise closest points or centroids are used depending on visibility.
- The physics component produces more realistic hand-object poses than estimates without physics, including during occluded interactions.
3.2.6 Physics Simulation - P
The physics component tests whether estimated hand-object configurations are stable and adjusts nearby finger contacts to produce physically plausible poses.
- Occlusions or missing visual data can produce physically unrealistic hand-object poses, such as a cube that would fall under gravity.The example involves a cube grasped and moved by two fingers, with one contacting finger occluded.
- The method models gravity, friction, restitution, and mass, representing bodies and object parts as convex hulls to accelerate simulation.
- A current solution is stable when 35 simulation iterations with 0.1-second steps move the object centroid by less than 3mm.
- Finger parts within 10mm of the object are candidate contacts, and combinations of two to four parts are evaluated.
- The simulation selects the contact combination causing the smallest centroid displacement, then minimizes finger-part distance to the object.
3.2.7 Anatomical limits
Anatomically inspired soft constraints penalize joint angles outside independently specified revolute-joint ranges.
- Anatomically inspired joint-angle limits are enforced as soft constraints in the objective function.
- Each revolute joint has an independently defined allowed range, illustrated using a ±20.0-degree deviation penalty.
- The physics simulation evaluates combinations of nearby finger parts to identify supporting contacts that stabilize manipulated objects.
3.2.8 Regularization
Regularization addresses ill-posed pose estimation during occlusions or missing depth by favoring continuity with the previous frame, within an iterative optimization procedure.
- 3.2.8 Regularization: When occlusions or missing depth make the objective ill-posed, a regularization term penalizes deviations from previous estimated joint angles.
- 3.2.8 Regularization: Pose estimation initializes from the previous-frame pose and iterates until convergence or a maximum iteration count.
- 3.2.9 Optimization: The algorithm alternates correspondence computation with objective optimization using Gauss-Newton for the resulting nonlinear least-squares problem.
3.3 Multicamera RGB
The framework extends its objective to multiple synchronized RGB cameras by replacing depth-based 3D terms with image-based 2D measurements and multi-view salient points.
- 3.3 Multicamera RGB: For synchronized RGB videos, the objective changes mainly in its data terms because observations are 2D rather than 3D.
- 3.3 Multicamera RGB: Known camera projection functions map 3D model points into each image plane, where correspondences are established with 2D points.
- 3.3 Multicamera RGB: Salient-point detection is applied to all camera views, using finger nails instead of finger tips because multiple high-resolution views provide distinctive image features.
- 3.3 Multicamera RGB: The data-to-model term becomes an edge term matching edge pixels in all views to projected model edges, with orientation and thresholding used to reject mismatches.
4 Experimental Evaluation
Experiments evaluate the framework across interacting hands, objects, camera setups, and component configurations. The results show accurate tracking, benefits from salient points and collision handling, physically plausible poses under occlusion, and limitations for very fast motion and missing depth.
- Component Evaluation: The point-to-plane metric provides adequate pose estimation with only 10 iterations, whereas point-to-point yields lower error when iteration count is unrestricted.Point-to-plane offers a significant speed gain compared with point-to-point.
- Salient Point Detection: Salient points reduce pose-estimation error more when detection accuracy is higher; raw depth produces weak detection and only slight improvement.Detection is much more accurate on RGB data and very accurate on the Dexter dataset.
- Component Evaluation: The complete objective function achieves the best overall system performance, while simplified data-fitting variants can collapse because of unconstrained optimization.The evaluated components are local optimization, collision detection, salient points, and physics simulation.
- Interaction and Camera Evaluation: The tracker captures interacting hands and objects accurately under close interaction, severe occlusions, and multiple synchronized RGB cameras.Hand-object sequences include rigid and articulated objects, while multicamera RGB experiments remain effective with multiple occlusions.
- Dataset and State-of-the-Art Evaluation: On the Dexter dataset, average error is around 1cm for most sequences, but the tracker fails on the very fast “random” sequence.Compared with the cited PSO method, the system achieves higher tracking accuracy, though the PSO GPU implementation runs in real time while this implementation is single-threaded CPU.
- Physics Simulation: With physics simulation, poses become more physically plausible under missing or ambiguous visual data, while pose-estimation error is reduced only slightly.The effect is especially visible for sequences with occluded manipulating fingers.
- Physics Simulation: Collision detection prevents physically implausible hand-object intersections, although it slightly increases pose-estimation error.Physics simulation slightly reduces error when combined with local optimization and salient points.
5 Conclusion
The paper presents a framework for capturing articulated hand and manipulated-object motion during intense interactions from monocular RGB-D or synchronized multicamera RGB videos. It combines generative fitting, discriminative salient points, collision detection, and physics simulation, achieving higher hand-pose accuracy than a PSO comparison while producing realistic motion under occlusion and tracking highly articulated objects.
- The framework captures articulated hand and manipulated-object motion from monocular RGB-D videos and multiple synchronized RGB videos.
- A single objective function combines a generative model, discriminatively trained salient points, collision detection, and physics simulation.
- Collision and physics terms slightly reduce pose-estimation error but increase motion realism under occlusions and missing visual data.
- The model achieves higher hand-pose estimation accuracy than a particle-swarm-optimization approach across both camera systems.
- The experiments include 8 multicamera RGB sequences and 21 single-camera RGB-D sequences, including successful tracking with highly articulated objects.