PaperlayerView exact revision on arXiv

Source-linked AI summary

Collision-Aware Humanoid Whole-Body Control under Imperfect Tracking Targets

Mohitvishnu S. Gadde, Ashish Malik, Pranay Dugar, Aayam Kumar Shrestha, Fuxin Li, Alan Fern

arXiv:2609.16405v2cs.RO

TL;DR

Imperfect target motions can cause collisions when humanoid whole-body controllers are blind to scene geometry. RECAL wraps a blind WBC with geometry-aware command adaptation, improving collision-free execution while preserving tracking across evaluated settings and transferring to a real Digit V3.

  • Problem

    Blind whole-body controllers may collide when imperfect targets under-specify collision-free motion for the robot body, manipulated objects, and nearby obstacles.

    Source: §I · p. 1

  • Method

    RECAL adapts a blind WBC's commands using cross-attention between robot and held-object query points and an egocentric environment point cloud.

    Source: §I · p. 1, §VII · p. 7

  • Results

    Across evaluated scenarios, RECAL improves collision-free success over Blind while preserving free-space tracking and approaching Teacher; at medium difficulty, it reaches 0.93 CF versus 0.38 for Blind and 0.97 for Teacher.

    Source: C. Locomotion and Object Carrying… · p. 6, §VII · p. 7

  • Takeaways & Limitations

    RECAL resolves local geometric conflicts at the control level, so upstream interfaces need not provide complete collision-free whole-body references.

    Source: §I · p. 1

    RECAL assumes static, flat-ground environments and egocentric depth observations with limited angular coverage, leaving some overhead and rear obstacles unobserved.

    Source: §VII · p. 7

Abstract

from arXiv · show

Humanoid robots often execute motion commands through whole-body controllers (WBCs) that track targets while maintaining balance and stability. However, most WBCs are blind to scene geometry, which can lead to collisions from imperfect target motions that are geometrically unsafe due to perception, planning, or teleoperation errors. We propose RECAL, a Robot--Environment Cross-Attention Layer that wraps a blind WBC to trade off target tracking against collision avoidance using external scene geometry. RECAL supports collision-aware tracking of floating-base and end-effector commands, including collision avoidance for held objects. It represents the robot, held objects, and environment as point clouds, using cross-attention between robot/object points and the environment to produce geometry-aware control features. In simulation, RECAL improves collision avoidance while preserving target-tracking performance across frozen-arm and adaptive-arm locomotion, object-carrying, and standing-manipulation scenarios relative to alternative geometry-aware WBC architectures. We further demonstrate the controller on a real Digit V3 humanoid robot.

I. INTRODUCTION

RECAL addresses collisions caused by imperfect targets given to geometry-blind whole-body controllers by adapting commands using scene geometry. The paper reports improved collision avoidance with tracking preserved in simulation and demonstrates transfer to a Digit V3.

I. INTRODUCTION

  • Blind WBCs can collide when imperfect targets leave whole-body motion under-specified, including because of perception, planning, or teleoperation errors.The risk concerns the robot body, manipulated objects, and nearby obstacles.Source: §I · p. 1
  • RECAL treats targets as guidance and resolves local geometric conflicts at the control level, so upstream interfaces need not provide complete collision-free whole-body references.The overview describes privileged locomotion and reaching teachers distilled into shared student commands, with deployment using onboard observations.Source: §I · p. 1, Figure 1 · p. 2
  • RECAL improves collision-free execution while preserving target tracking, outperforms alternative scene-geometry encoders, and transfers to a real Digit V3 humanoid.The evaluations cover cluttered environments and varying degrees of tracking imperfections.Source: §I · p. 1
  • The framework tracks floating-base and end-effector targets while avoiding collisions involving the robot and held objects.It modifies commands to an underlying blind WBC using robot and object query points attending to an environment point cloud.Source: §I · p. 1

II. RELATED WORK

Prior approaches address portions of perceptive collision-aware humanoid control, but the combined setting of floating-base motion, end-effector tracking, and held-object avoidance remains distinct. The formulation treats targets as under-specified guidance and balances tracking with avoiding unintended contact.

II. RELATED WORK

  • Classical collision-aware methods use structured control and optimization, but accurate models, hand-designed constraints, and high-dimensional online optimization are difficult to scale to cluttered, contact-rich scenes.Examples include potential fields, operational-space and task-priority control, and collision-constrained locomotion planning,,,.Source: §II · p. 2
  • Existing approaches typically focus on locomotion with limited upper-body objectives or stationary reaching, rather than jointly handling floating-base motion, end-effector tracking, and held-object collision avoidance.RECAL targets this combined setting while preserving the underlying WBC command-tracking interface.Source: §II · p. 2
  • Global point-cloud and voxel encodings do not directly expose robot–obstacle relations, motivating query-conditioned geometry features for robot and held-object points.The related work contrasts global encodings with robot-aware approaches and identifies RECAL's experiments against PointNet and Voxel encoders.Source: §II · p. 2

III. PROBLEM FORMULATION

  • The control objective is to track under-specified target motions as closely as possible while avoiding unintended environmental contact across manipulation, locomotion, and loco-manipulation.Targets may combine floating-base and end-effector motions or postures, leaving whole-body degrees of freedom unspecified.Source: §III · p. 2
  • The observation includes egocentric environment geometry and end-effector boxes that represent either held objects or default end-effector collision geometry.Points for intended contact surfaces may be removed from the environment point cloud before it is input to the WBC.Source: §III · p. 3
  • A binary end-effector-freezing flag determines whether collision avoidance should preserve commanded hand positions by adjusting the base or may instead adjust hand positions.The distinction supports carrying an object with a fixed grasp as well as behaviors where the arms can move freely.Source: §III · p. 3

IV. COLLISION-AWARE WBC ARCHITECTURE

RECAL layers collision-aware command adaptation over a pre-trained blind WBC. The adapter trades off target tracking against local collision avoidance while retaining the underlying controller's balance and tracking behavior.

IV. COLLISION-AWARE WBC ARCHITECTURE

  • RECAL uses sparse robot and held-object query points to attend to the environment point cloud and produce a modified command for the blind WBC.The base WBC then maps that command to joint-level PD targets.Source: §IV · p. 3
  • The modified command trades off target tracking against local collision avoidance.The adapter can preserve intended tracking when safe and deviate from precise tracking when needed to avoid collisions.Source: §IV · p. 3
  • Layering the adapter over the existing blind controller avoids training a new WBC from scratch and leverages its stability, tracking behavior, and sim-to-real robustness.The underlying WBC provides robust balance, locomotion, and command tracking but is blind to scene geometry.Source: §IV · p. 3

A. Base Whole-Body Controller.

The architecture builds on a pre-trained WBC and adds a query-conditioned geometry encoder with a recurrent command adapter. Robot and held-object points obtain local scene features that inform modified WBC commands.

A. Base Whole-Body Controller.

  • The pre-trained Masked Humanoid Controller (MHC) maps kinematic target commands to joint-level PD position targets for the Digit V3.This work uses its loco-manipulation mode for floating-base commands and arm joint targets.Source: A. Base Whole-Body Controller · p. 3

B. RECAL Architecture.

  • RECAL encodes scene geometry at sparse robot and held-object points, then combines those features with proprioception and command information in a recurrent command adapter.The adapter outputs the modified command passed to the base WBC.Source: B. RECAL Architecture · p. 4
  • The geometry encoder uses 37 queries: 13 body keypoints, 8 corners of a carried-object proxy, and 16 hand-object queries.Cuboid proxies represent the occupied volume of held objects; inactive object queries are masked.Source: B. RECAL Architecture · p. 4
  • Cross-attention produces a local scene feature for each query, and a two-layer LSTM with 256 hidden units outputs the modified command.The recurrent state smooths corrections over time and helps compensate for partial or intermittent point-cloud observations.Source: B. RECAL Architecture · p. 4

V. RECAL TRAINING

RECAL is trained through teacher–student distillation: privileged teachers generate corrected commands, and a student learns to imitate them using deployable observations.

V. RECAL TRAINING

  • Privileged teachers provide corrected WBC-compatible target commands for the student to imitate.The teachers use geometric information available in simulation; the student uses observational inputs.Source: §V · p. 4
  • A single RECAL student imitates both teachers using deployable observation inputs.The two teacher regimes cover cluttered locomotion, optionally with carried objects, and stationary reaching near shelves.Source: §V · p. 4
  • The training setup uses complementary teachers for locomotion with clutter and stationary reaching.The reaching regime holds the base fixed while hands follow potentially imperfect targets near shelf-like structures.Source: §V · p. 4

A. Training Environments.

The training environments expose the controller to randomized clutter and potentially unsafe target commands. Privileged teachers then provide collision-avoiding command corrections for locomotion, carrying, and reaching.

A. Training Environments.

  • Randomized obstacle fields and target sequences create locomotion scenarios where commands may pass through obstacles.The setup varies robot locations, object-carrying state, and end-effector freeze mode; reaching scenes randomize cabinet and shelf geometry and object extents.Source: A. Training Environments · p. 4

B. Privileged Teachers.

  • For reaching, a privileged recurrent PPO teacher observes furniture geometry and contact-related signals, then outputs root-height and arm-command residuals.Its observations also include hand-object extents, local proximity rays, and active-arm state.Source: B. Privileged Teachers · p. 4
  • For locomotion and object carrying, an analytic CBF-style teacher projects base-velocity commands using obstacle geometry and enlarges safety regions for held objects.When end-effectors are unfrozen, it also displaces hand targets from nearby obstacles and converts them to arm commands with damped least-squares IK.Source: B. Privileged Teachers · p. 4

C. Student Distillation.

A single recurrent RECAL student learns corrected commands from both privileged teachers using deployable observations, with extra loss weight on rare corrective actions.

C. Student Distillation.

  • A single student predicts RECAL commands from deployable observations and is supervised with teacher-produced target commands.It imitates both privileged teachers.Source: C. Student Distillation · p. 5
  • The training loss upweights base-command components when the locomotion teacher deflects the base command and arm components when either teacher moves a hand away from an obstacle.The recurrent student uses truncated backpropagation through time with chunks of 64 and a cosine learning-rate schedule from 10^-4 to 10^-5.Source: C. Student Distillation · p. 5
  • A fixed query layout with masking lets one student cover locomotion, object carrying, and reaching.The masking indicates which robot or held-object geometry is active.Source: C. Student Distillation · p. 5

VI. EXPERIMENTAL RESULTS

Across locomotion and carrying tasks, RECAL improves collision-free success over the blind WBC while approaching teacher performance, with comparable tracking and stronger collision avoidance than alternative geometry encoders.

A. Baselines.

  • The baselines include the blind WBC, privileged teachers, and student policies using PointNet or voxel geometry encoders.The learned student baselines share the recurrent policy, teacher labels, and distillation setup; only the geometry encoder and observation representation change.Source: A. Baselines · p. 5

B. Evaluation Protocol and Metrics.

  • Evaluation covers locomotion, frozen-arm locomotion, object carrying, stationary reaching, and reaching with held objects across Easy, Medium, and Hard geometry.Difficulty increases through reduced trajectory-clearance margins for locomotion and reduced shelf-clearance gaps for reaching.Source: B. Evaluation Protocol and Metrics · p. 5
  • CF measures episodes completed without unintended contact, while Dev measures command-tracking error only on safe trajectory states.Locomotion Dev is planar velocity error; reaching Dev is Euclidean hand-to-target distance.Source: B. Evaluation Protocol and Metrics · p. 5

C. Locomotion and Object Carrying Performance.

  • As hands approach obstacles, their attention shifts toward nearby relevant environment geometry.The figure visualizes attention scores for left- and right-hand query points during locomotion.Source: Figure 3 · p. 6
  • At medium difficulty, RECAL reaches 0.93 CF versus 0.38 for Blind and 0.97 for Teacher, while improving CF over Blind across difficulty levels.The Dev increase is modest in harder settings.Source: C. Locomotion and Object Carrying… · pp. 5–6
  • In hard adaptive-arm locomotion, RECAL reaches 0.91 CF versus 0.65 for PointNet, while maintaining comparable Dev to alternative geometry encoders.PointNet is the strongest baseline and is competitive on easy tasks, but its gap widens as difficulty increases.Source: C. Locomotion and Object Carrying… · p. 6
  • Voxel performs substantially worse than RECAL in CF across difficulty levels, despite being trained for over 5× as many steps.The authors suggest global scene embeddings may learn robot–environment relationships less directly than RECAL’s cross-attention.Source: C. Locomotion and Object Carrying… · p. 6

D. Stationary Reaching Performance.

RECAL substantially improves reaching performance over Blind and remains closer to Teacher, with the largest gains in geometrically ambiguous settings. Held objects and tight shelf clearances make hard reaching particularly challenging.

D. Stationary Reaching Performance.

  • Teacher reaches 0.81 CF on hard reaching tasks with held objects, where tight clearances and precision demands make the tasks more challenging than locomotion.Collision avoidance requires the arms, torso, and held object to navigate shelf geometry not fully specified by the end-effector target.Source: D. Stationary Reaching Performance · p. 6
  • Blind has higher Dev than Teacher because some reaching commands, especially large height changes, exceed the WBC’s effective command range.The RL Teacher compensates by adjusting targets.Source: D. Stationary Reaching Performance · p. 6
  • RECAL’s local obstacle-aware features support collision reasoning across hands, body parts, and held-object volumes, unlike global PointNet summaries or fixed-resolution voxel encodings.The passage attributes RECAL’s advantage over alternative encoders to fine-grained robot-scene reasoning in reaching.Source: D. Stationary Reaching Performance · p. 6
  • Across reaching-like tasks, RECAL improves over Blind and remains closer to Teacher, with the largest gains under geometric ambiguity.The summary identifies frozen arms, held objects, and tight reaching spaces as settings with the largest gains.Source: D. Stationary Reaching Performance · p. 6

E. Attention Visualization.

RECAL attends to nearby and motion-relevant obstacles, and its remaining contacts are less severe than those of the blind controller. Real-robot demonstrations show command adaptation across five locomotion and reaching tasks.

E. Attention Visualization.

  • As the robot approaches obstacles, attention shifts from the global sink token to relevant points; a wall along the motion direction can attract attention while still distant.Held-object corner queries can attend to obstacles far from the hands when the carried object’s swept volume may intersect them.Source: E. Attention Visualization · pp. 6–7

F. Force Analysis.

  • RECAL nearly eliminates torso collisions, with remaining contacts primarily on the arms at magnitudes far below Blind and comparable to PointNet.It also substantially outperforms Blind and PointNet in collision-free success.Source: F. Force Analysis · p. 7, Figure 4 · p. 7
  • Tracking deviation should be interpreted alongside collision-free success: some baselines continue tracking through contact, while RECAL deviates near obstacles to reduce collision severity.RECAL preserves tracking when avoidance is unnecessary.Source: F. Force Analysis · p. 7

G. Physical Robot Demonstration

  • On Digit V3, RECAL adapts unsafe commands near obstacles, then returns to intended tracking when motion becomes feasible; all five demonstrated tasks include locomotion and reaching variants.A few scenarios include light contact with the environment.Source: G. Physical Robot Demonstration · p. 7

VII. SUMMARY AND LIMITATIONS

RECAL wraps a frozen blind WBC with geometry-aware command adaptation and improves collision-free success while preserving free-space tracking. Its current scope is constrained by perception, environment, object, and infeasibility limitations.

VII. SUMMARY AND LIMITATIONS

  • RECAL substantially improves collision-free success over the blind base controller while preserving free-space tracking and approaching teacher performance across difficulty levels.The design also outperforms PointNet and Voxel encoders and transfers to a real Digit V3 humanoid.Source: §VII · p. 7
  • RECAL assumes static, flat-ground environments and limited-coverage egocentric depth, leaving some overhead and rear obstacles unobserved.Its geometric avoidance treats perceived geometry as forbidden contact and cannot distinguish intentional or functional contact, surface semantics, or movable obstacles.Source: §VII · p. 7
  • Evaluations cover rigid cuboid-held objects and non-dexterous end-effectors, and the policy does not emit an infeasibility signal when collision-free execution is impossible.The learned behavior is also bounded by privileged teachers that do not cover the full space of collision-aware strategies.Source: §VII · pp. 7–8
Loading 2609.16405v2…