Source-linked AI summary
3D Human Pose Estimation in RGBD Images for Robotic Task Learning
Christian Zimmermann, Tim Welschehold, Christian Dornhege, Wolfram Burgard, Thomas Brox
TL;DR
The paper addresses the need for real-world 3D human pose estimation to support robot learning from human demonstrations. It combines robust color-image keypoint detection with depth-based lifting and applies the resulting pose system to marker-free teaching, enabling a PR2 robot to reproduce demonstrated manipulation actions in real-world experiments.
Problem
Robotic task learning needs full 3D human pose in real-world units, whereas existing approaches commonly rely on monocular color or depth-only estimation.
Method
A CNN jointly uses color and depth to estimate 3D human pose and hand normals, then adapts demonstrations to robot kinematics and grasping capabilities.
Results
The approach outperforms existing baseline methods and enables a PR2 robot to reproduce human-demonstrated tasks without markers in real-world experiments.
Takeaways & Limitations
RGBD pose estimation supports non-expert, marker-free teaching of service-robot tasks through human demonstration.
Abstract
from arXiv · showhide
We propose an approach to estimate 3D human pose in real world units from a single RGBD image and show that it exceeds performance of monocular 3D pose estimation approaches from color as well as pose estimation exclusively from depth. Our approach builds on robust human keypoint detectors for color images and incorporates depth for lifting into 3D. We combine the system with our learning from demonstration framework to instruct a service robot without the need of markers. Experiments in real world settings demonstrate that our approach enables a PR2 robot to imitate manipulation actions observed from a human teacher.
I. INTRODUCTION
The paper targets real-world 3D human pose estimation for intuitive robot teaching, where following human motion requires handling articulation, occlusion, and hand interaction. It lifts robust 2D color predictions into 3D using depth and demonstrates marker-free task teaching on a service robot.
- Motivation: Intuitive learning from demonstration requires service robots to follow human motion, with hands central because they interact directly with the environment.The intended users should not need expert programming knowledge.
- Motivation: Robotic applications require full 3D human pose in real-world units, despite challenging appearance variation, articulation, and occlusions.Depth can also be misleading when objects occlude parts of the person.
- Contribution: The system predicts full 3D human pose and hand normals from RGBD input and outperforms existing baseline methods.The paper also demonstrates feasibility of teaching robot tasks by demonstration.
- Approach: The approach predicts 2D pose from color, then derives full 3D pose from the 2D pose and depth map.It additionally infers hand palm normals from cropped color images.
- Contribution: The system detects 3D keypoints and, when visible, hand normal vectors from color and depth inputs for robot task teaching.These predictions support learning from demonstration.
- Contribution: The approach enables marker-free action learning from human demonstrations and reproduces demonstrated actions on a robot in real-world experiments.The demonstration uses a service-robot task-learning framework.
II. RELATED WORK
Prior work largely separates 2D color pose estimation from real-world 3D estimation using depth, while monocular methods commonly produce scale- and translation-normalized poses. This work combines color and depth to lift strong 2D detections into real-world 3D coordinates.
- Color-based pose estimation: Most human pose estimation publications infer 2D keypoints from color images, enabled by large annotated datasets and robust keypoint-localization networks.The paper integrates these 2D detectors into its approach.
- Monocular 3D pose estimation: Monocular 3D methods commonly predict scale- and translation-normalized poses, limiting their practicality for many robotic applications.RGB-only approaches use assumptions to resolve depth ambiguity.
- Depth-based pose estimation: Existing real-world-unit pose approaches rely on active depth-sensing equipment, including Kinect-based methods using forests, tree walks, viewpoint-invariant representations, or volumetric convolutions.The related-work discussion identifies Kinect v1 as prominent equipment.
- RGBD integration: The architecture predicts color-image keypoints, tiles their score maps along the z-dimension, and combines them with a person-centered depth occupancy voxel grid.VoxelPoseNet then predicts 3D keypoints, while HandNormalNet processes cropped hand images for normals.
- RGBD integration: The proposed approach combines color and depth so color keypoint detectors disambiguate body sides while depth supports lifting to real-world 3D coordinates.The method is deep-learning based and jointly uses both modalities.
- Learning from demonstration: Marker-based and marker-less learning-from-demonstration methods either track attached markers, use articulated body models, or assume demonstrations already suit robot execution.The paper instead addresses marker-free teaching with robot adaptation.
III. APPROACH
The approach estimates human body keypoints and hand normals in the Kinect color-sensor frame from calibrated RGBD input. It aligns depth with color, detects 2D keypoints, and constructs a voxel representation centered using the predicted neck location.
- Problem formulation: The system estimates J human body keypoints in R^3 relative to the Kinect sensor and predicts two hand normal vectors from calibrated color and depth input.The coordinate system is defined to match the color-sensor frame.
- Problem formulation: The model predicts hand normal vectors for both hands in addition to the human body keypoints.These normals are defined in the color-sensor coordinate frame.
- Input alignment: Depth is transformed into the color frame using camera calibration, producing a sparse warped depth map because of occlusions, resolution differences, and noise.Linear interpolation is used only for visualization.
- Color keypoint detection: A color-image keypoint detector produces score maps whose maxima give predicted 2D keypoint locations.The implementation uses the Open Pose Library with fixed weights.
- Voxel representation: The warped depth map is converted into a K = 64 voxel occupancy grid centered using the back-projected predicted 2D neck keypoint and its neighborhood median depth.The grid is built from a point cloud derived from the warped depth map.
2) VoxelPoseNet:
VoxelPoseNet combines warped depth occupancy with tiled 2D keypoint scores in a 3D convolutional network, then combines voxel predictions with projected coordinates using 2D confidence.
- VoxelPoseNet: VoxelPoseNet processes a voxel occupancy grid and tiled 2D score maps with 3D convolutions to estimate 3D keypoint likelihood volumes.The voxel grid uses approximately 3 cm resolution, while tiled score maps approximate orthographic projection.
- VoxelPoseNet: The final prediction combines VoxelPoseNet coordinates with coordinates projected from 2D keypoints and the predicted depth component.The choice between the two coordinate sets is based on the 2D detector confidence.
- VoxelPoseNet: The network uses an encoder-decoder architecture inspired by U-net, with dense blocks in the encoder and intermediate losses during decoding.
B. Hand Normal Estimation
HandNormalNet estimates hand normal vectors from cropped hand images by reusing a previously trained hand-pose network without retraining.
- B. Hand Normal Estimation: HandNormalNet crops the input image around predicted hand keypoints before estimating hand normal vectors.
- B. Hand Normal Estimation: The reused hand-pose network estimates the relative transformation between the hand pose and a canonical frame, providing the normal vector.
D. Action learning
The action-learning framework transfers human motion trajectories to the robot while adapting them to the robot’s kinematics, grasping capabilities, and environmental constraints.
- D. Action learning: Human motion trajectories are transferred to the robot because its kinematics and grasping capabilities differ from the teacher’s.
- D. Action learning: Graph optimization adapts teacher hand, torso, and manipulated-object trajectories toward robot-feasible execution.The optimization accounts for geometry, grasping skills, occlusions, collisions, fixed grasps, and smooth trajectories.
IV. DATASETS
The authors create and use datasets to address the lack of high-quality Kinect v2 skeleton annotations, combining a multi-view Kinect dataset with a marker-less motion-capture dataset.
- IV. DATASETS: The absence of high-quality Kinect v2 skeleton datasets motivates recording a new dataset for training.Existing Kinect v1 datasets are technically mismatched, while newer datasets use Kinect SDK predictions as pseudo ground truth.
- IV. DATASETS: The new dataset includes 5 actors, 3 indoor locations, up to 4 viewpoints, and sequences recorded by calibrated Kinect v2 devices at 10 Hz.It contains upright, sitting, and walking poses, with training and evaluation sets separated by actors and location.
- IV. DATASETS: A second dataset uses Captury’s 12-camera marker-less motion-capture system at 120 Hz, calibrated with a Kinect v2 device.The system tracks 23 keypoints, of which 13 are used for comparison, and supports more object interaction than the MKV setup.
A. Datasets for training
The evaluation compares training-set generalization and 3D pose estimation against Kinect SDK, Naive Lifting, and monocular baselines. MKV-t provides the strongest cross-dataset training basis, while the proposed multimodal model handles challenging poses and occlusions better than the baselines.
- Training-set generalization: MKV-t provides sufficient variation to generalize to CAP-e, whereas CAP-t does not generalize to MKV-e.Joint training on CAP-t and MKV-t provides little additional improvement.
- Baseline comparison: The Kinect SDK performs poorly on challenging non-frontal and object-interaction scenes and cannot predict keypoints beyond a limited camera distance.Its failures include confusion caused by objects and people facing away from the camera.
- Baseline comparison: Naive Lifting is vulnerable to noisy depth values and produces errors under keypoint occlusion.It selects depth from neighboring pixels without learning a pose-based correction.
- Baseline comparison: Tome et al. [20] provides competitive results but lacks depth information and misses predictions in 8.7% of CAP-e frames.Kinect SDK misses 12.4% of frames, while Naive Lifting and the proposed approach miss 0%.
- Proposed approach: VoxelPoseNet outperforms its baselines by combining color for left-right disambiguation, depth for 3D localization, and learned body-configuration priors for occluded keypoints.The learned prior enables inference of 3D locations even for completely occluded keypoints.
C. HandNormalNet
The hand-normal component evaluates an adopted network on annotated samples and supports the downstream action-learning experiments. Its average angular error is reported as sufficient for the task-learning application.
- Evaluation: HandNormalNet is evaluated on 129 annotated samples from MKV using the adopted hand-normal estimation network.The hand-normal annotations were derived from detected and lifted hand keypoints.
- Evaluation: 60.3 degree average angular error is reported for hand-normal estimation.The authors state that this accuracy is sufficient for the task-learning application.
- Task-learning setting: The broader demonstration evaluation uses four manipulation tasks recorded from side or back viewpoints, making pose estimation challenging.The tasks include opening a room door and three small furniture pieces.
A. Adapting Human Demonstrations to Robot Requirements
The authors adapt human demonstrations to robot kinematics and grasping capabilities before learning action models. The adapted demonstrations support repeated PR2 execution, with failures tied to grasp localization and combined motion effects.
- Adaptation evaluation: Demonstration adaptation is compared between marker-based tracking and the proposed marker-less pose estimation across four trained tasks.The optimization results are summarized in Table III, including recorded-pose counts and gripper-to-grasp distances.
- Robot execution: The learned action models reproduce the demonstrated actions on a PR2 robot in real-world settings.The models combine gripper and base trajectories in Cartesian space and are learned from adapted demonstrations.
- Robot execution: All drawer and room-door grasping and manipulation trials succeed, while swivel-door execution has one localization failure.The sliding door is grasped successfully, but combined gripper and base motion can release its small knob.
- Failure cases: Typical baseline failures involve SDK confusion from objects or reversed body orientation and Naive Lifting errors under occlusion.The proposed approach is shown in the final row of the failure-case comparison.
- Grasp adaptation: The adaptation shifts recorded gripper poses toward drawer and door handles, unlike direct human-hand imitation, which would fail to grasp them.This adjustment accounts for the robot's grasping capabilities.
VII. CONCLUSIONS
The proposed CNN system combines color and depth to estimate 3D human pose in real-world units and supports marker-free robot task learning. Real-world experiments show a PR2 robot reproducing human-demonstrated tasks.
- The CNN system jointly uses color and depth to predict 3D human pose in real-world units.
- The approach exceeds the performance of existing methods for 3D human pose estimation.
- The system enables non-expert users to teach tasks to service robots through human demonstration.
- Real-world experiments demonstrate that a PR2 robot reproduces human-demonstrated tasks without markers on the teacher.