PaperlayerView exact revision on arXiv

Source-linked AI summary

A System for Fast, Resilient, and Adaptable Loco-Manipulation Behaviors on Humanoid Robots

Duncan Calvert, Luigi Penco, Dexton Anderson, Tomasz Bialek, Arghya Chatterjee, Beomyeong Park, Robert Griffin

arXiv:2609.01518v1cs.RO

TL;DR

Humanoid loco-manipulation requires coordinated locomotion, whole-body motion, perception, contact, supervision, and task adaptation. The paper presents a robot-local, runtime-editable architecture combining affordance templates, tree logic, behavior-time perception, synchronized operator control, and whole-body action execution. Demonstrations span multiple humanoid platforms and report fast, disturbance-tolerant behavior plus behavior creation and adaptation in hours.

  • Problem

    Humanoid robots must coordinate locomotion, whole-body motion, perception, contact, operator supervision, and adaptation to perform useful work in human-scale environments.

  • Method

    The system combines object-centric Affordance Templates, tree-based behavior logic, runtime-editable perception scenes, synchronized operator interaction, and whole-body action primitives.

  • Results

    The system demonstrates six task variants on Unitree H1-2 and Alex, including a 45.2-second reactive ball-sorting run under human disturbance and behavior creation or adaptation in hours.

  • Takeaways & Limitations

    The evaluated model-based behavior stack supports loco-manipulation behaviors that are fast, resilient, and adaptable across the reported tasks and robot platforms.

Abstract

from arXiv · show

There is tremendous value in humanoid robots taking on physically demanding, hazardous, and repetitive work in spaces built for humans. However, a useful robot for these spaces must coordinate locomotion, whole-body motion, perception, contact, and operator supervision. We present a robot-local, runtime-editable behavior authoring and runtime system that addresses these challenges. We argue that behavior architecture can be a primary enabler of capability, speed, and reliability, and that runtime editability enables fast behavior creation, adaptation, extension, and combination. Our behavior architecture combines object-centric Affordance Templates, a tree structure that provides organization and logic, and runtime-editable perception through a behavior scene and primitive scene actions. Our operator interface remains continuously synchronized to the robot for runtime authoring, monitoring, and repair. Action primitives execute through a whole-body controller that supports concurrent body motions and walking. Demonstrations of our system cover six task variants on Unitree H1-2 and Alex. We execute a push door traversal in 34 seconds and sort six balls by color in 45 seconds under human disturbance. Timed authoring sessions show scratch creation of new loco-manipulation behaviors and adaptation of existing ones in hours. Comparison against the literature finds our approach to be competitive with recent learned systems.

I. INTRODUCTION

The paper presents a robot-local, runtime-editable behavior architecture for humanoid loco-manipulation that coordinates perception, whole-body motion, locomotion, contact, and operator supervision. Real-robot evaluations and literature comparisons examine speed, resilience, adaptability, and multi-platform applicability.

  • The system targets humanoid work requiring coordinated locomotion, whole-body motion, perception, contact, operator supervision, and adaptation to new tasks.
  • The evaluation covers six task variants on Unitree H1-2 and Alex, with Alex as the primary evaluation robot and demonstrations also run across multiple humanoid platforms.
  • The paper evaluates the architecture through speed, resilience, and adaptability, including traversal performance, disturbance tolerance, and measured authoring or adaptation time.
  • The architecture unifies runtime-editable task logic, synchronized operator-interface state, behavior-time perception, and reusable behavior structures for robot-local execution.
  • The paper reports competitive door-traversal speed and combined evaluation of speed, repeated-trial reliability, authoring time, and adaptation time against literature results.
  • An expert operator created a novel humanoid door behavior from an empty sequence to first fully autonomous success in under two hours of measured active authoring time.

B. Door Traversals as a Benchmark Task

Door traversal is used as a compact benchmark for coordinating precise approach, complex opening manipulation, and collision-free traversal walking. The broader architecture draws on reusable affordance-based actions and behavior organization while extending to other loco-manipulation tasks.

  • Door Traversals as a Benchmark Task: Door traversal exposes coordinated approach, opening, and traversal-walk requirements in a compact and repeatable benchmark.
  • Door Traversals as a Benchmark Task: The approach requires precise footsteps that preserve arm reachability while avoiding collisions between the robot and door.
  • Door Traversals as a Benchmark Task: Spring-loaded doors require complex manipulation to open fully, while traversal walking must maintain balance and avoid collisions.
  • Broader Loco-Manipulation Scope: The architecture is intended beyond doors, including a two-table demonstration in which the robot walks between stations and sorts colored balls into containers.
  • Related Foundations: Behavior Trees organize and orchestrate behavior, while affordance primitives represent reusable constrained interaction motions such as valve turning or drawer closing.
  • Related Foundations: Affordance Templates parameterize and reuse loco-manipulation behaviors with respect to environmental affordances such as ground for walking or handles for turning.

2) Behavior Trees:

Behavior Trees organize and reactively coordinate low-level robot actions, while the broader system combines this logic with human-centered authoring, perception, and whole-body control.

  • Behavior Trees: Behavior Trees coordinate low-level actions through structured, reactive task logic using control nodes and action or condition leaves.They repeatedly evaluate execution from the root through a ticking mechanism.
  • Coactive Design: Coactive Design iteratively identifies interdependencies, selects and implements changes, and evaluates them through human feedback and performance analysis.The method motivates interfaces that make human-robot systems observable, predictable, and directable.
  • System integration: The system’s behavior architecture is intended for expert operators to develop, debug, and adapt behaviors on real humanoid hardware.Its authoring and supervision interfaces are directly motivated by Coactive Design.
  • Related systems: Prior door-traversal research includes classical decomposition, planner-based sequencing, behavior-centric systems, and learned policies across wheeled, legged, and humanoid platforms.The literature spans approach, opening, traversal, and closing tasks with differing sensing, planning, and learning assumptions.
  • Related systems: This work combines robot-local execution, runtime-editable structure, behavior-time perception, and real-world humanoid demonstrations in one system.Humanoid traversal additionally requires coordinated foot placement, balance, reachability, collision avoidance, and bimanual contacts.
  • System integration: The behavior system integrates reactive coordination with onboard semantic perception and a whole-body controller that accepts concurrent walking and body-part commands.Perception uses semantic object detection, while control combines asynchronous requests into balanced whole-body motions.

B. Process Structure

The runtime separates robot execution from the operator interface while synchronizing editable behavior state, enabling onboard autonomy alongside remote authoring and supervision.

  • B. Process Structure: Two robot-side processes run behavior and perception alongside control, while a dislocated operator process supports authoring, supervision, and visualization.This arrangement keeps high-rate sensor and scene data local while allowing the interface to inspect and edit execution.
  • B. Process Structure: If the operator UI crashes or communication degrades, runtime state supports continuing the task or stopping according to authored logic.The operator can reconnect, inspect the same execution, and resume without reconstructing behavior state.
  • Node decomposition: Each behavior-tree node contains a robot executor, operator UI, synchronized runtime state, and persistent definition.The definition stores authored content, while runtime state adds identity, status, and recent log information.
  • Synchronization: The process split requires a complex synchronization mechanism, implemented with CRDT-based tree and scene state synchronized at 30 Hz.ROS 2 DDS transports updates, and latest-timestamp arbitration resolves concurrent edits.
  • Synchronization: Synchronization serializes trees depth-first with compact partial payloads, separate topology metadata, and queued topology application.Full payloads are sent when definitions change, retransmission is requested, or fresh status is reported.
  • Synchronization: Robot-owned execution and visualization fields are mirrored one way, while bidirectional control fields support concurrent modification and conflict resolution.Bidirectional fields include execution gates, step selection, manual stepping, concurrency, preview, and failure reset controls.

E. Persistence Storage of Behaviors

Behaviors persist as structured JSON definitions, while object-centric frames make authored actions reusable across varying robot starts and accumulated execution errors.

  • E. Persistence Storage of Behaviors: JSON persistence stores behavior names, notes, parameters, child hierarchy, and node-specific type information.Each node is serialized with a type field and children array that mirrors the behavior tree.
  • E. Persistence Storage of Behaviors: The JSON files are human-readable but are intended for saving, loading, copying, and versioning rather than direct editing.The interface remains the recommended way to understand and work with behaviors.
  • H. Object-Centric Action Definition: Object-centric actions define hand poses, footsteps, and other actions relative to perceived scene objects or robot parts instead of only robot-local frames.Door approaches and grasps therefore reference door or object frames.
  • H. Object-Centric Action Definition: Object-centric action definition supports reusable behaviors by accommodating varied starting conditions and reducing compounding errors.Re-perceived object frames let later actions compensate for earlier stance deviations.
  • H. Object-Centric Action Definition: Sequential object-anchored actions can be interpreted as practical composition in which successive task stages preserve stability despite state and control errors.The paper describes approach and handle grasp as successive Lyapunov-funnel-like stages.
  • H. Object-Centric Action Definition: A reference-frame tree rooted at the world frame enables transformations among the robot and all perceived scene objects during authoring.The operator interface provides a 3D pose gizmo for defining object-relative actions.

I. Behavior Tree Structure

The system organizes reusable robot behaviors as a runtime-editable tree that combines sequencing, fallback logic, conditions, actions, concurrency, and manual rerouting. Unlike standard Behavior Trees, execution maintains a mutable next-node index and exposes richer runtime state.

  • The behavior tree organizes high-level tasks near the root and low-level primitive actions at the leaves.
  • The implementation adopts sequence, fallback, condition, and action concepts but is not a standard Behavior Tree.
  • A mutable next execution index lets the operator, sequence executor, goto node, or fallback node select the next node instead of restarting at the root.
  • Concurrency is defined through execute-after dependencies, while runtime nodes expose execution states beyond fixed success, failure, and running returns.
  • Fallback nodes pair a try branch with a catch sequence, which runs only when the try fails; fallback evaluation can also use concurrency.
  • The node library includes physical actions, scene actions, conditions, goto nodes, checkpoints, door traversal, and organizational sequence and fallback nodes.

K. Shape-Contains Condition Node

The shape-contains condition node provides fast, authorable perceptual reactivity from depth and color data. It supports branching or halting behavior execution based on configurable spatial and color criteria.

  • The condition node uses a CUDA kernel to count points and average color inside behavior-frame-relative virtual shapes such as spheres or capsules.
  • Authorable per-node thresholds define acceptable point counts and optional HSV color ranges for condition evaluation.
  • Condition results can halt a sequence on failure or select a fallback branch.
  • The check typically runs in under 10 ms and was used to detect opened door panels, doorway obstacles, and colored tennis balls.

L. Behavior Scene

The behavior scene gives behaviors exclusive control over tracked and privileged perception objects, while scene actions configure those objects and perception models at runtime. Derived scene objects convert detections and depth data into task-relevant geometric frames and door parameters.

  • The behavior scene contains actively updated tracked detections and a behavior-maintained privileged object list that behaviors alone can reference.
  • Scene actions can create, freeze, or delete scene objects and configure perception models such as YOLO and FoundationPose.
  • Derived scene object types compute new frames from persistent detections, existing scene objects, or depth data for novel tasks.
  • M. Door Panel Object: A door panel object uses stable detections of the opening mechanism and panel to define a frame centered on the mechanism and oriented toward the panel interior.
  • A door frame object estimates frame pose, push-or-pull type, and opening angle from the panel state and live depth data.

O. Approach Table Object

The approach-table object detects a table edge from point-cloud geometry and produces a ground-level approach frame for collision-free, reachable positioning. Composite frames extend this mechanism by combining existing scene frames into task-specific stances.

  • O. Approach Table Object: The approach-table algorithm uses only the point cloud and robot state to detect a table edge and project a squared-up reference frame at the robot’s feet.
  • O. Approach Table Object: The resulting frame helps the robot approach tables closely enough for arm reachability without colliding with the table.
  • Two vertical search capsules sweep forward from either side of the pelvis until they contain enough depth points or reach the search limit.
  • The algorithm supports tables of different heights through a single parameterization, with editable point thresholds and search limits.
  • Composite frames derive perception-less geometric objects from two existing scene frames and can be referenced by walking, arm, spine, and other physical actions.
  • Approach frames orient toward a target while stopping short of collision, whereas hybrid frames combine one source frame’s position with another’s orientation.
  • Composite frames can be layered to construct distant handle approach stances before selecting a door-specific behavior subtree.

IV. EXPERIMENTAL EVALUATION

Real-robot evaluations span door traversal, disturbance recovery, ball sorting, and behavior authoring on Unitree H1-2 and Alex. The demonstrations measure both runtime performance and the effort required to create autonomous behaviors.

  • Evaluation scope: Six task variants were evaluated on Alex, including door traversals, bottle carry-through, and single- and two-table ball sorting.Most evaluations used onboard stereo color vision and YOLO-based centroid estimation.
  • Speed: 45 seconds measured Alex’s right pull lever-handle door traversal, including 12 seconds approaching, 4 seconds unlatching, 15 seconds opening, and 14 seconds traversing.Limited spine yaw required more intricate opening motions than on Nadia.
  • Reliability: 11/11 push-door and 12/12 pull-door approach-and-opening trials succeeded before testing stopped.Full repeated traversal was not tested because walking control was not reliable enough and could have risked hardware damage.
  • Disturbance recovery: 65 seconds was required for a left pull traversal that recovered from five human disturbances using fallback logic and occupancy checks.The robot detected door-opening failure, waited for a blocked doorway to clear, and completed the run.
  • Manipulation: 45.2 seconds was required to sort six remaining balls correctly during continuous human disturbance, including recovery from a failed pick at 17.8 seconds.The behavior aborted placement and returned to search when the object was no longer successfully held.
  • Authoring: 31 minutes 43 seconds brought a Unitree H1-2 standing door-opening behavior from an empty tree to first autonomous looping execution.The session included diagnosis, editing, retesting, and a 32/32 repeated opening test.
  • Authoring: 11 hours 10 minutes 17 seconds were needed for Alex’s first full pull-door traversal, while a subsequent push-door behavior reached autonomous success in 1 hour 59 minutes 48 seconds.The first Alex behavior spanned five sessions over five days; the later session fixed an arm task-space failure in about nine minutes.

E. Fast Adaptation, Extension, and Combination of Behaviors

The system supports composing, mirroring, and extending existing behaviors rather than rebuilding them from scratch. Measured sessions reached autonomous success for new combined or adapted tasks within roughly two hours.

  • Combination: 1 hour 13 minutes 49 seconds produced the first autonomous bottle-carry-through-door run by combining existing pickup and door behaviors.The resulting autonomous run took 64 seconds from bottle pickup to passage through the door.
  • Adaptation: 1 hour 54 minutes 3 seconds adapted a right pull-door behavior to a left pull door despite stronger spring forces and degraded walking quality.The session used mirroring, then retuned opening motions and footsteps, adding spine yaw to prevent relatching.
  • Extension: 1 hour 50 minutes 41 seconds extended single-table ball sorting to two tables with locomotion between stations.The total included 15 minutes 30 seconds of off-robot pre-authoring before real-robot follow-up.
  • Implication: Measured sessions support combining, adapting, and extending existing behaviors in the hours regime through reusable primitives, subtrees, and scene actions.Runtime editing allowed operators to modify and retest parts of existing behaviors instead of rebuilding empty trees.

F. Comparative Analysis

The system reports fast, onboard, robot-local behaviors that compare favorably with classical literature and competitively with recent learned systems. The authors qualify these comparisons because reliability testing did not include repeated full traversals and literature results were not experimentally reproduced.

  • Speed comparison: Our door traversals are dramatically faster than classical results exceeding one minute and remain competitive with recent learned systems reporting 10 s and 15.4 s traversals.The comparison spans this work, selected literature references, and an earlier 14-second Nadia push-bar traversal.
  • System setting: The evaluated system uses two onboard color cameras and onboard autonomous computation, unlike cited systems using external motion capture, fiducials, or off-board neural inference.These differing sensing and compute configurations limit direct speed comparisons.
  • Reliability scope: Repeated reliability testing covered approach-and-opening and loco-manipulation behaviors, not repeated full doorway traversals.Traversal walking was not reliable enough during the test window without risking hardware damage.
  • Authoring comparison: Measured behavior-authoring duration is reported for both scratch creation and adaptation, unlike documented comparable loco-manipulation results found by the authors.An illustrative estimate contrasts six steps in two hours with a DoorMan-style workflow involving simulation setup, tuning, and overnight retraining.
  • Conclusion: The architecture combines Coactive Design principles, affordance templates, tree-based organization and logic, and a behavior-managed perception scene.The conclusion presents this model-based stack as effective for the evaluated loco-manipulation tasks and compatible with learning-based low-level routines.

A. Strong Points

The system’s strongest design choices are robot-local execution, runtime editability, expressive whole-body command scheduling, and behavior-time perception. Its current scope is constrained by limited object-orientation estimation, missing subtree subroutine support, incomplete force-based primitives, manual grasp planning, and reliance on human-authored scenarios.

  • Robot-locality: Robot-local computation decouples behavior execution from communications and remote-server compute failures.The authors identify robot-locality as one of the system’s most impactful design elements.
  • Runtime editability: Runtime editing enables rapid behavior bringup, modification, composition, and extension without redeployment or retraining workflows.The system supports online editing and re-testing directly on the robot.
  • Whole-body control: Asynchronous footstep, arm, leg, spine, and neck commands support expressive concurrent whole-body behaviors and integration with planners.The interface schedules these command types asynchronously and supports classical planners and AI-assisted behavior composition.
  • Perception: Behavior-time perception lets an expert operator exploit situation-dependent quality levels in depth, semantic confidence, latency, and frequency.These perception qualities can vary with lighting, situation, and hardware differences.
  • Current limitations: Object-on-table demonstrations remain centered on bottles and spherical balls because reliable object-orientation estimation is unavailable.Spherical balls require only position for grasping, while FoundationPose orientation estimates are not yet reliable enough for repeatable tests.
  • Current limitations: The system lacks returning sequence subtrees, so identical subtrees may be duplicated and edits to separate JSON instances can diverge or overwrite one another.Subroutine-like subtree execution would allow one instantiated subtree to be called from multiple places.
Loading 2609.01518v1…