PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboAssist: Interactive Human-Humanoid Planning for Long-Horizon Surgical Assistance

Jingwei Jia, Keyu Zhou, Jiewei Wang, Peisen Xu, Xingyuan Zhou, Liang Wang, Jiming Chen, Gaofeng Li, Jin Wang, Shunlei Li

arXiv:2609.39384v1cs.RO

TL;DR

Long-horizon surgical assistance must coordinate humanoid robots with evolving human activities while maintaining safety across planning and execution. RoboAssist links partially observed human-process estimates to executable robot tasks, combining affected-suffix replanning with layered safety; experiments report multi-stage completion, workflow adaptation, and improvements over full replanning in the request-order-change benchmark.

  • Problem

    Humanoid assistants must coordinate tasks with evolving human workflows, while autonomous execution alone does not ensure workflow alignment or consistent safety across planning and control.

    Source: §I · pp. 1–2

  • Method

    RoboAssist separates partially observed human-process states from executable robot tasks, updates their dependencies online, replans the affected suffix, and combines navigation, handover, and runtime safety layers.

    Source: §I · p. 2

  • Results

    In the request-order-change benchmark, RoboAssist achieves 81.25% success versus FullReplan's 68.75% and reduces mean replanning latency from 3.62 to 1.20 s (66.9%).

    Source: C. Results and failure analysis · p. 7

  • Takeaways & Limitations

    In simulated multi-stage surgical assistance, RoboAssist demonstrates task completion and workflow-request adaptation, with separate safety tests showing complementary protection under visual blind zones.

    Source: §V · p. 8

Abstract

from arXiv · show

Long-horizon surgical assistance requires humanoid robots to coordinate with evolving human activities while maintaining safety across planning and execution. We present RoboAssist, an agent-based framework for interactive human-humanoid planning that integrates workflow reasoning, task coordination, and cross-layer safety. At its core is an asymmetric dual-track representation that separates partially observed human process states from executable robot task sequences. By updating human-process estimates, scene context, and task dependencies online, RoboAssist revalidates the remaining task sequence and replans only the affected suffix when workflow requests change. A cross-layer safety architecture combines preventive navigation regulation, reactive regulation during close-range handover, and independent whole-body runtime supervision. This design couples online task coordination with safety constraints throughout execution. We demonstrate the framework on a Unitree G1 humanoid robot in long-horizon, multi-stage simulated surgical assistance scenarios encompassing multimodal interaction, instrument handling, medical material transport, navigation, and safe human-robot handover. Experiments show multi-stage task completion and adaptation to workflow-request changes. A targeted full-replanning ablation shows that residual replanning reduces plan-update latency and post-update token usage. Separate safety experiments demonstrate complementary protection across navigation, handover, and runtime supervision. Additional results and demonstrations are available online at https://roboassist.github.io.

I. INTRODUCTION

RoboAssist targets long-horizon surgical assistance, where evolving human activity and scene conditions can invalidate plans, while autonomous execution alone does not ensure workflow alignment or consistent safety. It combines separate human-process and robot-task tracks, affected-suffix replanning, and safety constraints across execution layers.

I. INTRODUCTION

  • Long-horizon assistance must coordinate instrument delivery, material transport, and handovers with evolving workflows, because changes can invalidate task dependencies and prior plans.The challenge includes reasoning about human activity, assistance needs, and timing while maintaining safety across planning and execution.Source: §I · pp. 1–2
  • Autonomous execution does not by itself ensure alignment with changing human workflows or consistent safety across planning and control.Predefined states, transitions, predicates, and recovery logic also become difficult to specify as human activities and task dependencies vary.Source: §I · pp. 1–2
  • RoboAssist separates partially observed human-process states from executable robot tasks, links them through evidence gates, and replans only the affected task suffix when dependencies change.Observation-updated task memory supports dependency revalidation as workflow and scene context evolve.Source: §I · p. 2
  • Its cross-layer safety architecture combines surgeon-aware navigation regulation, close-range handover regulation, and independent whole-body runtime supervision.The framework is evaluated on a Unitree G1 in long-horizon, multi-stage simulated surgical assistance.Source: §I · p. 2

A. Humanoid-Robot Collaboration and Surgical Assistance

Prior work addresses surgical instrument delivery, workflow anticipation, foundation-model planning, and safety in shared workspaces. RoboAssist focuses on autonomous long-horizon coordination across surgical-assistance activities, with explicit workflow- and scene-grounded task dependencies and layered safety.

A. Humanoid-Robot Collaboration and Surgical Assistance

  • Prior surgical-assistance work covers instrument delivery, video-based need anticipation, affordance prediction [5], and coordinated handover, while a humanoid surgical-workspace demonstration was teleoperated [8].Other work demonstrates whole-body loco-manipulation and scene interaction on Unitree G1 [9].Source: A. Humanoid-Robot Collaboration and… · p. 2

B. Foundation Models and Agent-Based Robotic Planning

  • Foundation-model approaches span language-conditioned skill planning –, generalist robot policies –[21], and surgical voice, vision, and VLA control for grasping and handover.RoboAssist complements these approaches by separating human-process estimates from robot tasks and updating only the affected task suffix when dependencies change.Source: B. Foundation Models and Agent-Based… · p. 2

C. Humanoid Safety-Aware Planning and Control

  • Shared-workspace safety research spans navigation, close-range interaction, and runtime execution; RoboAssist combines protected-region navigation, handover regulation, and runtime monitoring of force and joint-motion anomalies.Detected runtime violations trigger an emergency stop.Source: C. Humanoid Safety-Aware Planning and… · pp. 2–3

III. METHODOLOGY

RoboAssist addresses assistance tasks in which evolving instructions and scene conditions can invalidate remaining plans, while keeping low-level behaviors within parameterized skills. Its system combines observation-updated memory and an Agent with skill execution and a safety layer, and aligns human-process estimates with robot tasks.

1) Long-Horizon Surgical Assistance:

  • The framework targets preparing, transporting, and handing over objects under evolving instructions and scene conditions, with later skills depending on earlier physical effects and human readiness.The scope is task-level planning; low-level behaviors are encapsulated as parameterized skills in K.Source: 1) Long-Horizon Surgical Assistance · p. 3

2) System Overview:

  • RoboAssist comprises perception and memory, an LLM-driven task-level Agent, a skill executor, and a safety layer, with observations updating procedural-state estimates and task memory.The Agent maintains human-process and robot-task tracks; the executor checks preconditions and records outcomes.Source: 2) System Overview · p. 3

B. Interactive planning and online process alignment

  • RoboAssist separates human progress from executable robot tasks and links them through explicit gating relations, supporting dependency updates as the workflow evolves.The system state includes robot and scene state, linked task tracks, a skill library, observation-updated memory, and instruction and execution history.Source: B. Interactive planning and online… · p. 3, 2) System Overview · p. 3

1) Asymmetric Dual-Track Representation:

The asymmetric representation keeps uncertain human-process estimates distinct from executable robot tasks and links them with gates. This allows task readiness, physical preconditions, and preparatory actions to be handled separately as dependencies evolve.

1) Asymmetric Dual-Track Representation:

  • The two tracks remain separate because human progress inferred from voice or visual observations can be uncertain, while robot tasks are directly executable.Explicit gates connect the tracks without requiring strict temporal synchronization.Source: 1) Asymmetric Dual-Track Representation · p. 3
  • Human-process nodes represent observed or confirmed human states and predicted next actions, with estimates carrying their evidence source and a verifiable confirmation criterion.Predictions can trigger task generation or preparation, but cannot satisfy gates; missing evidence leaves a condition unconfirmed, and contradictory evidence invalidates confirmation and dependent tasks.Source: 1) Asymmetric Dual-Track Representation · p. 3
  • Robot-task nodes form a sequence of skill invocations with parameters, execution status, preconditions, and expected effects, generated from confirmed requirements and anticipated assistance needs.Physical preconditions include object availability or possession, robot configuration, and resource availability; human-process readiness is evaluated separately.Source: 1) Asymmetric Dual-Track Representation · p. 3
  • The gating map records which human-process nodes each robot task depends on, and one process node can gate multiple tasks.Tasks without human-process gates can be scheduled as preparation when their physical preconditions and safety constraints are satisfied.Source: 1) Asymmetric Dual-Track Representation · p. 3
  • The dependency map provides traceable human–robot links for consistency checking and residual replanning.Source: 1) Asymmetric Dual-Track Representation · p. 3

2) Cross-Track Consistency and Residual Replanning:

RoboAssist updates human-process and scene estimates, checks unfinished tasks against current requirements and dependencies, and regenerates only the affected task suffix when evidence invalidates the plan.

2) Cross-Track Consistency and Residual Replanning:

  • New voice and visual observations update human-process and scene state before the Agent revises the plan and checks unfinished tasks.Gate satisfaction is evaluated for each unfinished task.Source: 2) Cross-Track Consistency and… · p. 3
  • An unsatisfied human-process gate makes a task wait; residual replanning starts when new evidence invalidates a requirement or dependency, or a required precondition can no longer be established.Physical preconditions are checked at dispatch, and a future unmet precondition does not invalidate a plan if a preceding task can establish it.Source: 2) Cross-Track Consistency and… · p. 3
  • The earliest unfinished task with an invalidated requirement or dependency, or an unestablishable precondition, marks the divergence point for suffix replanning.The retained prefix includes completed records and still-valid pending tasks.Source: 2) Cross-Track Consistency and… · p. 3
  • Before reusing the retained prefix, the system revalidates mutable facts such as object possession, robot location, and resource occupancy.If an executing task becomes invalid, its skill is interrupted or canceled and the resulting physical state is returned before the affected suffix is regenerated.Source: 2) Cross-Track Consistency and… · pp. 3–4
  • The framework combines workflow knowledge, task memory, scene observations, and human requests to align tasks online, dispatch skills, and enforce runtime safety.The replanning loop includes evidence updates, state validation, and execution subject to human-process gates, physical preconditions, and safety constraints.Source: Figure 2 · p. 4, 2) Cross-Track Consistency and… · p. 4

C. Cross-Layer Safety Architecture

RoboAssist applies three safety layers with increasing intervention authority: navigation regulation, reactive handover regulation, and independent whole-body runtime supervision. The highest-authority supervisor can stop task motion until external recovery.

C. Cross-Layer Safety Architecture

  • Three safety layers increase intervention authority from navigation regulation to handover regulation and fail-safe whole-body runtime supervision.The latched runtime stop overrides navigation and handover regulation and prevents further task-motion execution until explicit external recovery.Source: C. Cross-Layer Safety Architecture · pp. 4–5
  • Navigation velocity is scaled according to clearance from the surgeon’s protected operating region and set to zero at boundary contact or entry.The clearance-dependent scaling approaches unity sufficiently far from the protected region and decreases as the robot approaches its boundary.Source: C. Cross-Layer Safety Architecture · p. 4
  • The execution loop runs a pending task only when its human-process gate and physical precondition are satisfied and runtime supervision permits motion.The loop preserves the prefix and regenerates the affected suffix when a task is invalidated.Source: C. Cross-Layer Safety Architecture · p. 4
  • Navigation regulation applies whenever locomotion is enabled, including object transportation, while close-range handover motion receives additional interaction regulation.Source: C. Cross-Layer Safety Architecture · p. 4
  • During handover, a VLM-based monitor evaluates human–robot configuration and interrupts nominal motion when proximity or human movement signals potential risk.Avoidance increases separation; nominal behavior resumes only when the interaction is again classified as safe.Source: C. Cross-Layer Safety Architecture · pp. 4–5
  • An independent runtime supervisor monitors proprioceptive feedback for abnormal force, excessive joint velocity, and persistent joint oscillation.When activated, it terminates the current skill, inhibits subsequent motion, and keeps the robot stopped until explicit external recovery.Source: C. Cross-Layer Safety Architecture · p. 5

A. Experiment setup

The simulated workspace separates surgery, object transfer, and material storage across three tables, with the robot moving retrieved objects through the handover area. Experiments use a Unitree G1 platform with specified sensing, onboard computation, interaction, planning, and manipulation components.

A. Experiment setup

  • The workspace has an operating table for surgery, a handover table for object transfer, and a sterile table for material storage.The robot retrieves objects from the sterile table, transfers them via the handover table, and assists at the operating table.Source: A. Experiment setup · p. 5
  • The platform is a Unitree G1 humanoid with Inspire FTP dexterous hands, ZED 2i and onboard head and wrist cameras, and NVIDIA Jetson Thor computation.Spoken interaction, visual inventory perception, and task decisions use Qwen3.5-Omni-Plus-Realtime, Qwen3.7-Plus, and DeepSeek V4 Pro, respectively; manipulation uses a shared GR00T N1.7 backbone and task-specific experts.Source: A. Experiment setup · p. 5
  • The safety architecture combines surgeon-aware navigation regulation, reactive handover regulation, and fail-safe whole-body runtime supervision.Source: Figure 3 · p. 5
  • The evaluated object set spans bottles, blue-capped cups, surgical instruments, syringes, gauze, surgical drapes, and aluminum boxes.Source: A. Experiment setup · p. 5

B. Autonomous agent-based interactive planning

The section compares RoboAssist with scripted and reactive planning baselines, then evaluates long-horizon task execution, workflow-order adaptation, and planning overhead.

B. Autonomous agent-based interactive planning

  • The Reactive-agent comparison tests the combined dual-track representation and evidence gating, while FullReplan versus RoboAssist isolates residual suffix replanning.The baselines share observations, memory, workflow knowledge, skills, and safety mechanisms; FullReplan retains RoboAssist’s other components but regenerates all unfinished tasks after an update.Source: B. Autonomous agent-based interactive… · p. 5

1) Long-horizon surgical assistance:

  • Task evaluation covers single-object handovers and multi-object deliveries that combine navigation, alignment, grasping, transportation, and ordered completion.There are 15 single-object trials and seven multi-object trials per method; single-object success requires handing over the correct object.Source: 1) Long-horizon surgical assistance · p. 5
  • Fig. 5(b) tracks the fraction of seven trials completing every ordered checkpoint through each of four subtasks; its final value equals multi-object task success.Representative execution stages are shown in Fig. 6.Source: 1) Long-horizon surgical assistance · p. 6

2) Human-humanoid coordination and adaptation:

  • Request-order adaptation uses 16 matched trials per method, each with one delivery-order change and success requiring all deliveries in order within the task budget.Latency runs from update acceptance to revised-queue commitment; post-update token totals exclude initial planning and perception calls, and overhead statistics use 10 logged trials per LLM-based method.Source: 2) Human-humanoid coordination and… · p. 6, Table I · p. 6
  • The independent multi-object efficiency experiment reports completion times for five successful runs per method, with markers identifying individual runs.The caption specifies successful workflows, so these completion times do not represent all attempted runs.Source: Figure 7 · p. 6, 2) Human-humanoid coordination and… · p. 6

3) Cross-layer safety evaluation:

The safety evaluation tests navigation regulation and close-range interaction and runtime protections, including an ablation of runtime supervision.

3) Cross-layer safety evaluation:

  • Navigation regulation is assessed through deceleration and stopping near the protected region, while interaction and runtime layers each face five trials.The latter trials cover close-range and visual-blind-zone conditions.Source: 3) Cross-layer safety evaluation · p. 6
  • Fig. 8 reports navigation profiles, protective responses, and mean response times for the VLM and force-feedback pathways.Negative distance in the navigation profile indicates boundary crossing; L3 OFF disables runtime supervision.Source: Figure 8 · p. 7
  • Response time measures the interval from hazard trigger to protective response, and the L3-OFF ablation tests runtime supervision under visual-blind-zone conditions.Recorded timestamps are used for the VLM and force-feedback pathways.Source: 3) Cross-layer safety evaluation · p. 7

C. Results and failure analysis

RoboAssist completes multi-stage simulated assistance tasks, adapts to workflow-request changes with lower planning overhead, and demonstrates complementary safety protection. Completion drops at the final checkpoint, though the observed cause is unresolved.

C. Results and failure analysis

  • 93.3% single-object success exceeds the 80.0% rates of both the scripted FSM and reactive agent; multi-object success is 85.7% for RoboAssist, versus 71.4% and 57.1%.The compared multi-object methods are RoboAssist, reactive agent, and scripted FSM, respectively.Source: C. Results and failure analysis · p. 7
  • 100% cumulative completion holds through the first three subtask checkpoints, then falls to 85.7% at the final checkpoint.The observed performance drop is localized to the last stage, but does not identify whether planning or physical execution caused it.Source: C. Results and failure analysis · p. 7
  • 81.25% RoboAssist end-to-end workflow completion success exceeds FullReplan's 68.75%, while mean replanning latency falls from 3.62 to 1.20 s (66.9%).The reported success rates measure end-to-end workflow completion, including physical execution.Source: C. Results and failure analysis · p. 7
  • 20.02k to 11.41k tokens (43.0%) is the decrease in mean post-update usage under the specified token accounting.The paper attributes lower planning overhead to residual replanning and notes that the reactive-agent comparison reflects the combined planning formulation.Source: C. Results and failure analysis · p. 7
  • 100% versus 40% protective-response success is reported with runtime supervision in visual-blind-zone tests; force feedback responds in 20.11 ms and the VLM pathway in 4.833 s.The pathways use different hazard triggers, so these are response times rather than a strictly matched end-to-end latency comparison.Source: C. Results and failure analysis · p. 7
  • The final-checkpoint drop leaves improving robustness at the workflow's last subtask important, while its cause remains unresolved.The reported results do not establish whether the drop stems from planning or physical execution.Source: C. Results and failure analysis · p. 7

V. CONCLUSION

  • RoboAssist links human-process estimates to executable tasks for online dependency alignment and residual replanning; evaluation on a Unitree G1 remains in simulated assistance workflows.Future work targets longer and more diverse workflows, broader interaction and environmental variations, and greater VLA skill robustness through broader training coverage.Source: §V · pp. 7–8

Related papers

Loading 2609.39384v1…