PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Human-Object Interaction for 3D Human Pose Estimation from LiDAR Point Clouds

Daniel Sungho Jung, Dohee Cho, Kyoung Mu Lee

arXiv:2603.16343v1cs.CV

TL;DR

LiDAR-based 3D human pose estimation is challenged by spatial ambiguity between human and object points and sparse observations of interacting body parts. HOIL addresses these issues with HOICL and CPPool, achieving state-of-the-art performance across diverse real-world LiDAR datasets. The framework’s scope is limited by its lack of paired RGB information and reduced point density for distant people.

  • Problem

    Human–object interactions create spatial ambiguity between human and object points and severe point-count imbalance in interacting body parts, challenging robust LiDAR-based pose estimation.

  • Method

    HOIL learns interaction-aware point representations using HOICL to distinguish human and object points and CPPool to preserve information from underrepresented contacting body parts.

  • Results

    HOIL achieves state-of-the-art performance across diverse real-world LiDAR datasets for 3D human pose estimation.

  • Takeaways & Limitations

    HOIL effectively leverages human–object interaction to address spatial ambiguity and class imbalance in interaction regions.

  • Takeaways & Limitations

    HOIL does not use paired RGB information, and LiDAR point density decreases with distance, leaving distant people with few observed points per body part.

Abstract

from arXiv · show

Understanding humans from LiDAR point clouds is one of the most critical tasks in autonomous driving due to its close relationships with pedestrian safety, yet it remains challenging in the presence of diverse human-object interactions and cluttered backgrounds. Nevertheless, existing methods largely overlook the potential of leveraging human-object interactions to build robust 3D human pose estimation frameworks. There are two major challenges that motivate the incorporation of human-object interaction. First, human-object interactions introduce spatial ambiguity between human and object points, which often leads to erroneous 3D human keypoint predictions in interaction regions. Second, there exists severe class imbalance in the number of points between interacting and non-interacting body parts, with the interaction-frequent regions such as hand and foot being sparsely observed in LiDAR data. To address these challenges, we propose a Human-Object Interaction Learning (HOIL) framework for robust 3D human pose estimation from LiDAR point clouds. To mitigate the spatial ambiguity issue, we present human-object interaction-aware contrastive learning (HOICL) that effectively enhances feature discrimination between human and object points, particularly in interaction regions. To alleviate the class imbalance issue, we introduce contact-aware part-guided pooling (CPPool) that adaptively reallocates representational capacity by compressing overrepresented points while preserving informative points from interacting body parts. In addition, we present an optional contact-based temporal refinement that refines erroneous per-frame keypoint estimates using contact cues over time. As a result, our HOIL effectively leverages human-object interaction to resolve spatial ambiguity and class imbalance in interaction regions. Codes will be released.

1 Introduction

LiDAR-based 3D human pose estimation remains difficult during human–object interactions because human and object points are spatially ambiguous and interacting body parts are sparsely observed. HOIL addresses these challenges with interaction-aware representation learning and achieves strong performance across diverse real-world LiDAR datasets.

  • Motivation: Human–object interactions complicate LiDAR perception and accurate pose estimation, creating challenges for pedestrian safety in autonomous driving.Existing methods struggle particularly with keypoints in interacting body regions.
  • Challenges: Spatial ambiguity makes it difficult to distinguish human points from object points, while hands and feet contain very few LiDAR points compared with other regions.These two challenges can produce erroneous predictions and underrepresent interaction-frequent body parts.
  • Approach: HOIL learns interaction-aware point representations for robust 3D human pose estimation from LiDAR point clouds.The framework expands pre-training to diverse human–object interactions and learns an interaction prior beyond a body-pose prior.
  • Results: HOIL achieves state-of-the-art performance across diverse real-world LiDAR datasets for 3D human pose estimation.The reported outcome uses both human–object interaction-aware contrastive learning and contact-aware part-guided pooling.
  • Approach: HOICL uses contrastive learning to discriminate human and object point features, emphasizing interaction and contact regions where points are mixed.The method operates on the part-segmentation feature space during pre-training.
  • Approach: CPPool pools overrepresented non-contacting body parts aggressively while preserving information from underrepresented contacting body parts.This reallocates representational capacity toward interaction-frequent regions.

3 Method

HOIL builds on a hierarchical Point Transformer encoder-decoder and adds interaction-aware learning and pooling to address ambiguity and sparse interacting-body-part points. The architecture produces point-level segmentation and contact predictions alongside keypoint estimates.

  • 3.1 Preliminary: PTv3 serializes points, applies multi-stage grid pooling and Transformer encoding, then restores resolution through mapped unpooling and skip connections.The decoder reuses stored pooling mappings rather than recomputing spatial correspondences.
  • 3.2 Model architecture: HOIL replaces PTv3 max pooling with CPPool while retaining the remaining hierarchical encoder-decoder components.The model uses decoder features with 256 channels and learnable queries for human keypoints.
  • 3.2 Model architecture: The model iteratively updates keypoint queries through cross-attention with decoder point features, then predicts segmentation, contact, and 3D keypoint outputs.Point features produce human-object part segmentation and point-level contact, while updated queries drive keypoint prediction.
  • 3.3 Contact-aware part-guided pooling: CPPool predicts pooling weights that preserve information from interaction-critical regions during downsampling.It addresses class imbalance by replacing uniform grid-cell selection with contact- and part-guided aggregation.
  • 3.4 Human-object interaction-aware contrastive learning: HOICL uses contrastive learning to discriminate human and object point representations, emphasizing interaction and contact regions.Its objectives include human-object separation and contact-region supervision, enforcing discriminative representations where points are spatially ambiguous.
  • 3.6 Final outputs and loss functions: During real-world LiDAR fine-tuning, HOIL predicts final 3D keypoints with axis-wise one-dimensional heatmaps.The pre-training objective combines segmentation, contact, coordinate, keypoint-contact, HOICL, and CPPool losses.

4 Implementation details

The implementation uses PyTorch with a PTv3-based architecture, AdamW optimization, cosine annealing, and voxelized point clouds. Synthetic LiDAR data are generated by ray casting onto SMPL and 3D object models.

  • The implementation uses PyTorch and a PTv3-based architecture.The architecture follows PTv3’s basic configuration.
  • AdamW uses learning rates of 3 × 10−4 during pre-training and 5 × 10−4 during fine-tuning, with mini-batches of 64.A cosine annealing learning-rate schedule supports convergence.
  • Point clouds are voxelized with a grid size of 0.01, while synthetic pre-training data are generated by ray casting onto SMPL and 3D object models.

5 Experiments

Experiments pre-train HOIL on diverse human-object interaction datasets and evaluate it using pose accuracy metrics, ablations, and comparisons with state-of-the-art methods. The results show consistent gains from interaction-aware contrastive learning, contact-aware pooling, and temporal refinement, with strong performance across real-world LiDAR datasets.

  • Experimental setup: HOIL is pre-trained on five human-object interaction datasets, with sampling ratios of 1, 40, 30, 1, and 1, and objects randomly removed at ratio 0.5.The datasets are BEHAVE, CHAIRS, HODome, OMOMO, and InterCap.
  • Evaluation metrics: Evaluation reports MPJPE, PCK-3, and PCK-5 for 3D human pose estimation.PCK-3 and PCK-5 measure keypoints within 30% and 50% of torso length, respectively.
  • Ablation studies: HOICL consistently improves Waymo performance, with Contact CL producing considerable gains over other contrastive-learning techniques.The results indicate that contact regions are particularly affected by spatial ambiguity.
  • Ablation studies: 2.32% is the largest MPJPE improvement when contact is added to CPPool, while incorporating part and contact yields PCK-5 above 99%.These results support CPPool’s treatment of class imbalance in interacting body regions.
  • Ablation studies: Temporal and contact cues both improve HOIL’s 3D pose estimation performance in the InterCap temporal-refinement ablation.The per-frame HOIL setting without temporal and contact cues is the comparison baseline.
  • Comparison with state-of-the-art methods: HOIL achieves meaningful improvements across all Waymo metrics and outperforms prior methods in interaction and novel-pose scenarios.Qualitative comparisons show improved predictions for umbrella, wall, bicycle, and sitting interactions.

6 Conclusion

The conclusion presents HOIL as a framework for LiDAR-based 3D human pose estimation that targets spatial ambiguity and class imbalance in human-object interaction regions. It reports robust and accurate estimation in such scenarios, outperforming prior methods by a significant margin.

  • HOIL addresses spatial ambiguity and class imbalance in human-object interaction regions using HOICL and CPPool.
  • The framework achieves robust and accurate pose estimation in human-object interaction scenarios.
  • HOIL outperforms prior methods by a significant margin.

Supplementary Material for “Learning Human-Object Interaction for 3D Human Pose Estimation from LiDAR Point Clouds"

The supplementary material contains additional experiments, discussions, and results omitted from the main manuscript because of page limitations. It covers implementation-related definitions, analyses, temporal refinement, computational requirements, qualitative results, and limitations and societal impacts.

  • Supplementary contents: The supplement includes more qualitative results together with limitations and societal-impact discussions.
  • Supplementary contents: The supplement provides details on loss functions, keypoint and part definitions, query embeddings, and contact labels.
  • Supplementary contents: Additional analyses address spatial ambiguity, class imbalance, CTRefine quantitative results, and computational requirements.

S1 Details of loss functions

The framework combines auxiliary part and contact supervision with importance-guided pooling, alongside limb and multi-task losses. These objectives supervise segmentation, contact, keypoint coordinates, contrastive learning, and geometric consistency.

  • CPPool loss: CPPool uses auxiliary part-segmentation, contact, and importance predictors inside its pooling module.The importance predictor contributes logits to weighted feature aggregation, while part and contact predictors receive explicit supervision.
  • CPPool loss: CPPool loss sums part-segmentation and contact losses.Both terms use cross-entropy supervision from ground-truth labels.
  • Limb loss: The limb loss combines cosine-based bone-direction and SmoothL1 bone-length penalties.Both loss weights are set to 1.0.
  • Loss weights: Pre-training jointly optimizes segmentation, contact, coordinate, keypoint-contact, HOICL, and CPPool losses.Their corresponding weights are 1.0, 1.0, 0.5, 0.02, 1.0, and 1.0, respectively.

S2 Details of keypoint and part definition

The framework uses SMPL-based joints when meshes are available and dataset-defined keypoints otherwise. Its segmentation labels distinguish 24 human body parts from objects and background.

  • Keypoint definition: Keypoints are obtained from SMPL joints for mesh-based datasets and from dataset-provided definitions for datasets such as Waymo.This maintains consistency across datasets while respecting direct annotations where available.
  • Part definition: The segmentation task contains 24 human body-part classes, one object class, and one background class.Ground or wall points are assigned to the background label, while object faces share one object label.

S3 Details of keypoint query embeddings

Learnable keypoint queries represent target joints, with one embedding token assigned to each keypoint. During pre-training, the setup uses 16 queries for 15 SMPL joints and one object keypoint.

  • Keypoint query embeddings: Each learnable query embedding corresponds to one target keypoint.The number of query embeddings matches the number of keypoints in the training setup.
  • Keypoint query embeddings: Pre-training uses 16 queries representing 15 SMPL body joints and one object keypoint.

S4 Details of contact labels

Contact supervision is derived from paired human and object meshes. Point-level labels use dense mesh contact annotations, while keypoint-level labels aggregate contact across vertices associated with each body part.

  • Point-level contact: Paired human and object meshes provide the basis for dense human-object contact labels.Point-level contact is extracted on the mesh using distance-based thresholding.
  • Keypoint-level contact: A body keypoint is labeled as contact when at least one associated body-part vertex is labeled as contact.Keypoint-level labels are obtained by aggregating vertex-level contact annotations after SMPL joint regression.

S5 Analysis on spatial ambiguity issue

Segmentation quality is meaningfully associated with pose accuracy, supporting spatial ambiguity as a key challenge. The largest errors occur at frequently interacting joints such as wrists and ankles.

  • R = −0.59 indicates a meaningful negative correlation between segmentation accuracy and human pose estimation error.The analysis supports accurate differentiation of human body-part points as important for pose estimation.
  • Wrists and ankles, which frequently interact with objects, are highlighted as error-prone body joints for the state-of-the-art method.
  • Frequently interacting body-part joints also suffer from the class imbalance issue identified in the state-of-the-art analysis.

S7 Quantitative results of CTRefine

CTRefine improves temporal pose refinement beyond standard filtering approaches, while HOIL also shows qualitative gains in challenging interaction scenes. The supplementary evaluations additionally report stable inference performance and important dataset-scope qualifications.

  • Quantitative CTRefine results: Approximately 5.6% lower MPJPE than the baseline without refinement is achieved by CTRefine on InterCap, while PCK metrics also improve.CTRefine consistently outperforms Gaussian, Savitzky–Golay, and One-Euro filtering approaches.
  • Quantitative CTRefine results: CTRefine produces consistent pose-estimation improvements on SLOPER4D, although gains are modest under a strong baseline and heuristic contact supervision.
  • Evaluation scope: The SLOPER4D supplementary ablation evaluates video-based samples, whereas the main paper evaluates frame-based samples.
  • Computational requirements: HOIL contains 53.02M parameters, runs at 155.52 Hz, and processes each point cloud in approximately 6.43 ms at inference.
  • Qualitative comparisons: HOIL predicts more accurate hand and foot keypoints than DAPT in qualitative Waymo and SLOPER4D interaction examples.Examples include umbrella walking, motorcycle riding, backpack carrying, and soccer with a ball near the foot.
Loading 2603.16343v1…