PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive Interaction Network for Human Motion Prediction During Human-Robot Collaboration

Mengyuan Liu, Yangting Lin, Qiongjie Cui

arXiv:2610.16624v1cs.CV

TL;DR

Robot-aware human motion prediction matters for safe, efficient human–robot collaboration, but existing approaches often do not model robots as dynamic partners or address human–robot heterogeneity. AINet combines separate human and robot prediction branches with multi-scale interaction modeling, adaptive fusion, and coarse-to-fine pose prediction. It reports state-of-the-art performance across three HRC/HRI datasets, while noting higher errors on some abrupt collision-related actions.

  • Problem

    Most existing human motion prediction methods are not designed for robot-aware settings, which require modeling human–robot heterogeneity, asymmetric coupling, and robot-driven context changes.

    Source: §I · p. 1

  • Method

    AINet jointly trains separate human-motion and auxiliary robot-motion branches, coupled through local-global interaction modeling and context-dependent feature fusion.

    Source: §I · pp. 1–2

  • Results

    AINet achieves state-of-the-art performance across the CoMaD, CHICO, and HARPER datasets.

    Source: §I · p. 2

  • Takeaways & Limitations

    The reported results support robot-aware interaction modeling for human motion prediction across collaboration and interaction scenarios.

    Source: §V · p. 11

    The method shows higher errors on some collision-related actions involving abrupt transitions, indicating difficulty with irregular motion patterns.

    Source: §V · p. 11

Abstract

from arXiv · show

Human motion prediction during human-robot collaboration is critical for achieving safe and efficient interactions in shared environments. Unlike previous studies that have primarily focused on humans and objects, we focus on the robot-aware human motion prediction task, which explicitly models the influence of robots on human motion. This task presents new challenges, such as capturing human-robot heterogeneity and modeling complex spatial interactions. To address these issues, we develop an adaptive interaction network (AINet) model that consists of two branches: a primary branch that predicts future human motion and an auxiliary branch that estimates robot trajectories. The two branches are jointly optimized and coupled via a Local-Global Spatial Interaction (LGSI) Module, which effectively captures fine-grained and global contextual dependencies between human and robot motion sequences. Then, we introduce an Adaptive Weighted Aggregation (AWA) Module to dynamically fuse motion features using context-dependent weights, thereby increasing adaptability across diverse scenarios. Furthermore, we adopt a coarse-to-fine prediction strategy, in which a coarse pseudo-trajectory is first predicted, followed by the refinement of detailed human poses based on that trajectory. Extensive experiments on three challenging datasets reveal that our proposed method achieves superior performance, validating its effectiveness in human-robot collaboration scenarios. Our code is available at: https://github.com/LyTingHub/AINet.

I. INTRODUCTION

AINet targets robot-aware human motion prediction, where human–robot heterogeneity, asymmetric interaction, and robot-driven context changes challenge existing methods. It combines separate human and robot motion prediction with multi-scale interaction modeling, adaptive feature fusion, and coarse-to-fine pose prediction.

I. INTRODUCTION

  • Existing motion-prediction methods are not designed for robot-aware settings because they miss agent coupling, human–robot heterogeneity, or robots’ dynamic influence on people.Single-person methods neglect inter-agent coupling; multi-person methods assume homogeneous agents; HOI and scene-aware methods treat context as passive or static.Source: §I · p. 1
  • AINet uses separate human and robot motion branches, making human prediction the primary task and robot trajectory prediction an auxiliary task.The branches jointly model asymmetric, dynamic interactions in shared environments.Source: §I · p. 1
  • LGSI integrates fine-grained local details and coarse global representations to model complex human–robot spatial interactions.AWA adaptively fuses multi-grained motion features with context-based weights.Source: §I · pp. 1–2
  • The Traj-Pose Module predicts a learned multi-joint pseudo-trajectory, then refines future human poses in a coarse-to-fine strategy.This separates global motion tendency from local pose refinement without relying on a fixed hip-joint proxy.Source: §I · p. 2
  • AINet achieves state-of-the-art performance across CoMaD, CHICO, and HARPER, spanning human-robot collaboration and interaction settings.CoMaD focuses on close-proximity domestic-kitchen collaboration, CHICO on factory assembly, and HARPER on human-quadruped interactions at varied distances.Source: §I · p. 2

II. RELATED WORK

Prior human motion prediction work addresses isolated people, interactions among people, objects, or static scenes, but those settings do not fully model autonomous robots as heterogeneous, reactive partners. For HRC, AINet explicitly couples human and robot pose prediction and evaluates across structured collaboration and broader interaction scenarios.

A. Human Motion Prediction

  • Single-person methods omit external context, while multi-person methods commonly assume homogeneous agents and symmetric interactions, limiting their fit to human–robot settings.AINet instead uses dedicated human and robot branches before cross-agent interaction reasoning.Source: A. Human Motion Prediction · p. 2
  • Human-object and scene-aware methods represent passive objects or static geometry, not an autonomous robot whose motion changes human behavior over time.These representations therefore miss the bidirectional, asymmetric adaptation relevant to HRC.Source: A. Human Motion Prediction · pp. 2–3

B. Human Motion Prediction for Human-Robot Collaboration

  • Existing HRC predictors provide limited modeling of robot-driven coupling or heterogeneous roles; AINet uses separate motion branches and LGSI to combine local proximity cues with global interaction states.The paper evaluates on CoMaD and CHICO for HRC and HARPER for broader HRI behaviors.Source: B. Human Motion Prediction for… · p. 3

III. ADAPTIVE INTERACTION NETWORK FOR ROBOT-AWARE HUMAN MOTION PREDICTION

AINet predicts human and robot motion in separate, connected branches, using multiscale spatial interactions to represent their coordination. Its pipeline encodes motion in the frequency domain, refines poses through a trajectory-to-pose stage, and decodes future motion.

III. ADAPTIVE INTERACTION NETWORK FOR ROBOT-AWARE HUMAN MOTION PREDICTION

  • AINet uses connected human and robot branches, with robot motion prediction as an auxiliary task to the primary human-motion prediction task.Each branch includes motion encoding, cross-agent interaction, adaptive aggregation, trajectory-to-pose prediction, and motion decoding.Source: §III · p. 3

A. Motion Encoder and Motion Decoder

  • The Motion Encoder applies a discrete cosine transform and fully connected layers; the Motion Decoder uses fully connected layers and an inverse transform to reconstruct motion.The network predicts residuals relative to the last observed pose, and represents 3D poses in world coordinates without additional normalization or coordinate transformation.Source: A. Motion Encoder and Motion Decoder · p. 3

B. Local-Global Spatial Interaction

  • LGSI combines local, contact-sensitive geometry with global collaboration context to represent human-robot interactions at complementary spatial scales.Local cues concern nearby body parts and workspace; global cues represent pair configurations and trends such as approaching, yielding, or handover.Source: B. Local-Global Spatial Interaction · p. 4
  • LGSI aligns human and robot skeleton features with MLPs, models cross-agent correlations through attention, then maps interaction features back to each agent’s skeleton space.The alignment addresses differences between human and robot structures; skip connections preserve each agent’s motion information during interaction and reconstruction.Source: B. Local-Global Spatial Interaction · p. 4

C. Adaptive Weighted Aggregator

The Adaptive Weighted Aggregator addresses changing relative importance of local and global interaction cues by weighting them according to context. Its context comes from aligned GSI features, and a regularizer constrains the learned weights.

C. Adaptive Weighted Aggregator

  • Local cues may matter more at close range, while global interaction state may guide motion when agents are farther apart or loosely coordinated.Fixed fusion coefficients would impose the same interaction bias across these differing contexts.Source: C. Adaptive Weighted Aggregator · pp. 4–5
  • AWA combines local and global interaction features with context-dependent weights, deriving its context from aligned GSI features and normalizing that context before weight generation.The weights are constrained to sum to one.Source: C. Adaptive Weighted Aggregator · p. 5
  • A regularization term constrains the human- and robot-branch weight coefficients to [0, 1], which the authors say helps stability and generalization.Source: C. Adaptive Weighted Aggregator · p. 5

D. Traj-Pose Module

The Traj-Pose Module separates global motion tendency from local pose refinement. It predicts an adaptive pseudo-trajectory from full-body motion features, then refines poses conditioned on that trajectory.

D. Traj-Pose Module

  • Single-stage prediction must infer long-term movement tendency and fine-grained pose articulation simultaneously, which can make learning unstable.Source: D. Traj-Pose Module · p. 5
  • The two-stage strategy first generates a pseudo-trajectory and then refines future poses, establishing global dynamics before local kinematic detail.The architecture uses an ST Encoder, a Traj Module, and a Pose Module.Source: Figure 4 · p. 5, D. Traj-Pose Module · p. 5
  • A fixed hip-joint trajectory is not a universal global-motion proxy: HARPER shows large hip displacements, while CHICO and CoMaD concentrate motion in the upper limbs.The module therefore learns a pseudo-trajectory from contextualized full-body spatio-temporal features.Source: D. Traj-Pose Module · p. 5, Figure 3 · p. 5

E. Training Objective

The training objective combines human and robot prediction losses with regularization for the AWA-generated weights. Each prediction loss accounts for both position and temporal velocity errors.

E. Training Objective

  • The objective combines human-motion and robot-motion prediction losses with an AWA regularization term.The regularizer constrains generated weights to a meaningful range.Source: E. Training Objective · p. 6
  • Each human or robot prediction loss sums squared position error and squared temporal-velocity error against the ground truth.Source: E. Training Objective · p. 6
  • The velocity sequences are computed by temporal differentiation, with each frame’s velocity defined as the next joint position minus the current one.Source: E. Training Objective · p. 6

IV. EXPERIMENTS

AINet is evaluated on CoMaD, CHICO, and HARPER using MPJPE, the average Euclidean distance between predicted and ground-truth joint positions. The benchmarks span task-oriented collaboration and broader human-robot interaction.

A. Datasets

  • CoMaD and CHICO represent task-oriented collaboration, while HARPER tests generalization to a broader human-robot interaction setting.Source: A. Datasets · p. 6
  • CoMaD covers five long-horizon kitchen tasks with a robotic arm and provides 3D skeleton annotations for 11 human upper-body joints and 10 robot joints.Source: A. Datasets · p. 6
  • CHICO contains 20 operators collaborating with robotic arms in industrial assembly lines; HARPER covers 15 human–quadruped action categories, 10 involving physical contact.Source: A. Datasets · p. 6

IV. EXPERIMENTS

B. Implementation Details

The experiments compare AINet with human-motion and multi-person prediction baselines under different input settings. Training uses 60 input frames to predict the following 30.

B. Implementation Details

  • Training used one RTX 4090 GPU, batch size 256, about 6 GB of memory, and approximately two hours.HARPER training ran for 31,000 iterations, with the learning rate reduced after 30,000.Source: B. Implementation Details · p. 7

C. Baseline

  • Baselines include HRC-specific, single-person, and multi-person methods; methods requiring equal joint counts use duplicated root joints for agents with fewer joints.Source: C. Baseline · pp. 7–8
  • The evaluation compares methods using human-only input and human-plus-robot input; AINet is also tested without LGSI and AWA using only human input.Source: C. Baseline · p. 7
  • On HARPER, Ours-HR demonstrates a marginal advantage over the baselines.Source: Table IV · p. 8

D. Quantitative results

AINet improves prediction across the evaluated collaboration datasets, with the greatest relative enhancement on HARPER. Results also show that adding robot-motion input alone does not guarantee better predictions; computational efficiency is assessed for real-time use.

Comparison of the CoMaD and CHICO Dataset Results

  • On CoMaD and CHICO, adding human-robot motion input does not consistently improve MPJPE and can degrade accuracy without effective interaction modeling.The authors attribute this to interference from additional global information when interaction dynamics are not effectively modeled.Source: D. Quantitative results · p. 8

Comparison of the HARPER Dataset Results

  • AINet outperforms baselines at every HARPER time step in both Ours-HR and Ours-H configurations.The authors report the greatest improvement over baselines on HARPER, which they describe as more challenging than CoMaD and CHICO.Source: D. Quantitative results · p. 8

Action-wise results

  • Across six exemplary HARPER actions, AINet outperforms baselines; abrupt collision, contact-transition, and proximity-constrained actions have higher error variability.The authors relate that variability to motion complexity and note that such actions are more difficult for deterministic predictors.Source: D. Quantitative results · p. 8

Cross-dataset comparison

  • Across CoMaD, CHICO, and HARPER, differences in scenarios, robot types, and action characteristics support the method’s applicability across diverse settings.The authors report the greatest enhancement on HARPER and suggest its richer skeletal and interaction-category coverage may contribute.Source: D. Quantitative results · p. 8

Comparison of Computational Efficiency

  • 100.10 mm MPJPE at 1-second prediction is reported for Ours-HR on HARPER, alongside computational efficiency described as meeting real-time requirements.The comparison covers an RTX 3060 GPU and a Jetson Nano edge device.Source: D. Quantitative results · p. 8

E. Qualitative results

Qualitative examples illustrate AINet’s predictions in close-proximity kitchen collaboration, factory object lifting, and human–robot collision avoidance. Ablations report contributions from spatial interaction, adaptive weighting, learned pseudo-trajectories, and loss regularization.

E. Qualitative results

  • In CoMaD’s bottle-grasping scene, AINet predicts the human’s grasping intention more accurately than baselines, which have higher MPJPE.The figure reports MPJPE above predictions and uses dashed circles to highlight notable differences.Source: E. Qualitative results · p. 8, Figure 8 · p. 9
  • In CHICO’s relay-style object-lifting scene, AINet produces more plausible motion with accurate object placement and coordinated limb movements.The authors identify the robot arm’s motion as a cue about target locations as humans lift objects.Source: E. Qualitative results · p. 9, Figure 9 · p. 10
  • For HARPER’s “Walk+Avoid right” sequence, Ours-HR has lower MPJPE than all baselines during human collision avoidance.The example involves approaching the robot before the human stops and moves rightward to avoid collision.Source: Figure 10 · p. 10, E. Qualitative results · p. 10

F. Ablation studies

  • Removing LSI, GSI, LGSI, or AWA increases average MPJPE by 0.40 mm, 0.97 mm, 0.44 mm, or 1.57 mm, respectively.The ablations support complementary contributions from fine-grained and holistic spatial modeling and adaptive weighting.Source: Table VII · p. 10, F. Ablation studies · p. 10
  • Removing context-processing LN raises MPJPE@1000 ms from 100.11 mm to 103.58 mm, while removing context-based weights increases average MPJPE by 0.92 mm.Removing truncation or sum normalization also elevates MPJPE, supporting the tested weight constraints.Source: Table VIII · p. 10, F. Ablation studies · p. 10
  • Removing Traj or Pose raises MPJPE by 0.51 mm or 4.20 mm; the learned pseudo-trajectory also achieves the lowest average MPJPE against handcrafted baselines.The handcrafted alternatives use the hip joint or the mean of all joint features as a global motion proxy.Source: F. Ablation studies · pp. 10–11, Table IX · p. 11
  • The best loss performance uses Lhuman + Lrobot + λLreg with an appropriately chosen λ, combining task-specific supervision with AWA fusion regularization.Lreg supervises AWA-generated coefficients to support fusion of local and global features.Source: Table X · p. 11, F. Ablation studies · p. 11

V. CONCLUSION

The conclusion attributes more accurate, contextually reasonable collaborative-task predictions to modeling dynamic human–robot interaction. It also identifies higher errors on abrupt collision-related actions and describes multi-agent modeling as a possible extension.

V. CONCLUSION

  • The model uses human–robot interaction information to improve human-motion prediction during collaboration, with local-global spatial modeling and adaptive weighting contributing to contextual understanding.The conclusion identifies both spatial scales and context-based aggregation as components of its approach.Source: §V · p. 11
  • Higher errors on some collision-related actions with abrupt transitions remain a limitation, motivating possible transition-aware modeling and targeted data augmentation.The authors present these as potential future directions for improving robustness to irregular motion patterns.Source: §V · p. 11
  • The current framework focuses on single human–robot pairs; the authors describe an N-branch architecture as a possible extension to multi-agent scenarios.In that proposed design, each branch predicts one agent’s motion while incorporating the other agents as contextual inputs.Source: §V · p. 11
Loading 2610.16624v1…