Source-linked AI summary
TAIHRI: Task-Aware 3D Human Keypoints Localization for Close-Range Human-Robot Interaction
Ao Li, Yonggen Ling, Yiyang Lin, Yuji Wang, Yong Deng, Yansong Tang
TL;DR
Close-range egocentric HRI demands precise task-relevant 3D keypoint localization, whereas conventional methods emphasize root-centric whole-body reconstruction. TAIHRI uses a task-aware VLM with discretized interaction-space coordinates and 2D keypoint reasoning, achieving superior accuracy on egocentric benchmarks and supporting downstream HRI tasks.
Problem
Conventional 3D pose methods focus on root-centric whole-body reconstruction, while HRI requires metric-scale localization of task-relevant body parts in the camera coordinate system.
Method
TAIHRI quantizes task-relevant 3D keypoints in an interaction space and uses sequential 2D keypoint and depth reasoning through next-token prediction.
Results
TAIHRI achieves superior 3D keypoint localization accuracy on egocentric close-range benchmarks and substantially outperforms prior methods on task-critical body parts.
Takeaways & Limitations
TAIHRI is adaptable to downstream tasks including natural language control and global-space human mesh recovery, supporting close-range HRI applications.
Takeaways & Limitations
Inference assumes that camera intrinsics, including focal length and principal point, are known for the HRI scenario.
Abstract
from arXiv · showhide
Accurate 3D human keypoints localization is a critical technology enabling robots to achieve natural and safe physical interaction with users. Conventional 3D human keypoints estimation methods primarily focus on the whole-body reconstruction quality relative to the root joint. However, in practical human-robot interaction (HRI) scenarios, robots are more concerned with the precise metric-scale spatial localization of task-relevant body parts under the egocentric camera 3D coordinate. We propose TAIHRI, the first Vision-Language Model (VLM) tailored for close-range HRI perception, capable of understanding users' motion commands and directing the robot's attention to the most task-relevant keypoints. By quantizing 3D keypoints into a finite interaction space, TAIHRI precisely localize the 3D spatial coordinates of critical body parts by 2D keypoint reasoning via next token prediction, and seamlessly adapt to downstream tasks such as natural language control or global space human mesh recovery. Experiments on egocentric interaction benchmarks demonstrate that TAIHRI achieves superior estimation accuracy for task-critical body parts. We believe TAIHRI opens new research avenues in the field of embodied human-robot interaction. Code is available at: https://github.com/Tencent/TAIHRI.
1 Introduction
Close-range egocentric HRI requires precise localization of task-relevant body parts despite occlusion, truncation, and changing viewpoints. TAIHRI addresses this need with a task-aware VLM and reports superior benchmark performance.
- Close-range egocentric cameras create severe occlusion, truncation, and viewpoint changes that challenge accurate 3D keypoint localization.
- Existing 3D pose methods mainly optimize root-centric whole-body reconstruction, which does not fully meet HRI’s need for precise task-relevant body-part localization.
- TAIHRI is a VLM for close-range HRI that quantizes task-relevant body locations in a discretized interaction space and predicts 3D coordinates through 2D keypoint reasoning.
- Interaction-centric prompts direct TAIHRI’s attention to body parts relevant to the specific HRI context, while a diverse egocentric dataset supports robustness across camera intrinsics and viewpoints.
- TAIHRI outperforms existing methods in 3D keypoint localization on egocentric close-range benchmarks and shows effectiveness in human-guided HRI tasks.
2 Related Work
Prior work advanced monocular 3D human pose and shape estimation through parametric body models, 2D keypoint supervision, and robustness to complex interactions. In parallel, vision-language models progressed toward stronger visual understanding and reasoning through large-scale vision-language pretraining.
- 3D Human Pose and Shape Estimation: SMPL-based parametric models represent human shape and pose for recovering full 3D human meshes from images.The represented variables are human shape β and pose θ.
- 3D Human Pose and Shape Estimation: 2D keypoint detection was subsequently incorporated as an intermediate supervision signal to improve 3D estimation accuracy.
- 3D Human Pose and Shape Estimation: Recent methods target robustness in complex settings including multi-person interactions, human-object interactions, and occlusions.
- Vision-Language Models: Vision-language models progressed from early image captioning and visual question answering systems toward multimodal models using large-scale vision-language pretraining.The passage cites Flamingo and GPT-4V as early examples, followed by GPT-5.2, Qwen3-VL, and Gemini-2.5 Pro.
3 Method
TAIHRI formulates close-range HRI keypoint localization in the robot camera frame by combining task-aware keypoint selection, discretized 3D prediction, and 2D-guided depth reasoning. Its training pipeline uses pose-aware rewards with group-relative policy optimization.
- Task formulation: TAIHRI localizes task-relevant 3D human keypoints in the robot’s camera coordinate system from an egocentric RGB image and human interaction prompt.The target keypoints correspond to body parts critical for the intended interaction task, with camera intrinsics assumed known during inference.
- Dataset: CloseHRI provides over 1 million close-range egocentric images covering diverse subjects, poses, and interaction scenarios.The dataset is systematically filtered to provide supervision for robust pose and interaction representations under close-range perspective views.
- Discretized interaction space: TAIHRI quantizes each keypoint’s coordinates within a predefined interaction cuboid into voxel token indices from 0 to 999 along each axis.The discretized indices are obtained by scaling coordinates by the interaction volume dimensions W, H, and D and taking the floor.
- 2D keypoint reasoning: The model decomposes monocular 3D localization into sequential 2D keypoint prediction followed by reasoning about corresponding depth values.This design uses more tractable image-plane localization and spatial relationships to support final 3D coordinate inference.
- Inference procedure: During inference, TAIHRI encodes the resized image and instruction, autoregressively predicts keypoint names and 2D and voxel-level 3D locations, then decodes metric camera-space coordinates.The predicted 3D keypoints can support HRI perception, global-space human mesh recovery, or robot control.
- Reinforcement finetuning: Reinforcement finetuning starts from an SFT-initialized model and applies GRPO using pose-aware rewards computed on visible joints.The reward combines robust Huber aggregation with a PCK-style success term, while GRPO uses group-relative advantages and a clipped KL-regularized objective.
4 Experiment
TAIHRI is evaluated on egocentric interaction datasets using metric-scale camera-coordinate localization, with comparisons against pose-estimation methods, VLMs, and ablations. It also supports task-driven keypoint selection and global human-mesh repositioning.
- Datasets and Metrics: TAIHRI is evaluated on Harmony4D-Test and EgoBody, including 6,389 and 62,155 egocentric interaction frames, respectively.Harmony4D-Test uses multi-view triangulation for accurate 3D keypoint annotations.
- Datasets and Metrics: G-MPJPE measures average Euclidean joint-position error in millimeters directly in the camera coordinate frame without root alignment.The metric penalizes both relative structural errors and absolute translation and depth errors.
- Comparative Study: TAIHRI outperforms CameraHMR, PromptHMR, and SAM 3D Body across all body-part configurations on both Harmony4D-Egocentric and EgoBody.Table 1 reports G-MPJPE for different body-part configurations, including separate SAM 3D Body backbones.
- Comparative Study: TAIHRI directly predicts camera-coordinate 3D keypoints, improving localization of task-relevant joints such as wrists and ankles that are difficult for prior methods.Prior errors arise from both pose and translation estimation, especially for joints far from the root.
- Ablation Study: Removing camera-intrinsics input causes a large performance drop, while replacing it with learnable ray embeddings also degrades performance.The ablation identifies explicit camera-parameter encoding as essential for accurate egocentric 3D localization.
- Applications: Interaction-centric prompts let users specify task-relevant body regions, while predicted 3D keypoints can anchor transformation of a normalized mesh into global robot-camera coordinates.The mesh transformation is expressed as M_global = T(M).
5 Conclusion
TAIHRI addresses the mismatch between root-centric whole-body pose estimation and close-range HRI’s need for egocentric, task-relevant keypoint localization. Its task-aware VLM design achieves superior benchmark accuracy and supports downstream interaction applications.
- TAIHRI bridges root-centric whole-body pose estimation and egocentric localization of task-relevant body parts for close-range HRI.
- The method discretizes the interaction space, uses 2D keypoint reasoning through next-token prediction, and applies task-aware prompts.
- TAIHRI achieves superior accuracy on egocentric interaction benchmarks and substantially outperforms prior methods on task-critical body parts.
- The model remains adaptable to natural language control and global-space human mesh recovery, supporting task-aware perception for embodied HRI.