Source-linked AI summary
Physical Agentic AI: An Architecture for Orchestrating a Robot Crew with LLMs
Xinyuan Liu, Eren Sadikoglu, Riana Chatterjee, Ransalu Senanayake
TL;DR
Physical robot crews need a verified interface between open-ended language planning and state-dependent execution. The paper introduces Physical Agentic AI, combining typed robot skills, workflow contracts, a non-actuating planner, and deterministic per-dispatch authorization. Retrieval raises grounding from 51% to 96%, but enforcement reduces false dispatch to 0% and refuses all eight injected faults before motion.
Problem
Physical robot crews require action verification against robot capabilities, system state, and workflow constraints because richer planner information does not eliminate infeasible or unsafe actions.
Method
Physical Agentic AI uses typed executable skill libraries and workflow contracts with a non-actuating Mission Planner and deterministic Robot Orchestrator authorizing each dispatch.
Results
Retrieval raises skill grounding from 51% to 96%, while per-dispatch enforcement reduces false dispatch to 0% and blocks all eight injected fault scenarios with no false blocks.
Takeaways & Limitations
Live execution shows that without enforcement all eight injected faults crossed the orchestration boundary and six produced robot motion, whereas enforcement refused all eight before motion.
Abstract
from arXiv · showhide
Agentic AI frameworks interpret open-ended task goals and decompose them into multi-step plans. Richer information about embodiment-specific capabilities, physical preconditions, and cross-robot coordination improves grounding, but does not eliminate infeasible, mistimed, or unsafe physical actions. Physical robot crews therefore require an explicit architectural interface between semantic planning and execution, where every planned action is verified against robot capabilities, system state, and workflow constraints before actuation. This paper introduces Physical Agentic AI, a framework for skill-grounded robot agent orchestration, in which each robot exposes a typed library of executable skills while a foundation model planner decomposes a task into phases and assigns each phase to a robot-skill pair. A Robot Orchestration layer exposes the skill library, robot state, named locations, and workflow contracts to a non-actuating Mission Planner, while a deterministic Robot Orchestrator validates and authorizes one skill at a time. We evaluate on a drone-UGV search-and-dispatch mission, where every mission in every condition is executed live in Gazebo, and on a humanoid-quadruped transportation task using hardware-equivalent skill interfaces plus two physical trials on a Unitree G1 and Go2. Varying planner knowledge and runtime enforcement independently, we find that retrieval raises skill grounding from 51% to 96% yet leaves informed planners dispatching 23-29% of faulted steps. Per-dispatch enforcement reduces false dispatch to 0% with no false blocks, and a held-plan ablation confirms that the gate, not plan variation, is responsible. Live execution makes the difference physical: without enforcement all eight injected faults crossed the orchestration boundary and six produced robot motion; with enforcement all eight were refused before motion.
I. INTRODUCTION
Physical robot crews need an explicit interface between language-level planning and physically grounded execution. Physical Agentic AI separates planner knowledge from runtime safety through verified skills, workflow contracts, and deterministic enforcement.
- Agentic AI abstractions organize software workflows but are insufficient for physical robot crews whose actions are state-dependent, time-consuming, and potentially irreversible.
- Foundation-model robotics introduces a vulnerability when planner knowledge is conflated with runtime safety.
- Physical Agentic AI grounds language-level plans in embodiment-specific skill libraries and workflow contracts for heterogeneous robot crews.
- The foundation model plans over retrieved, pre-verified skills, while the deterministic Robot Orchestrator exclusively decides what may touch the world.
- Runtime state binding is necessary because execution-critical facts may become available only after another robot completes perception.
II. BACKGROUND
Prior work spans robot architectures, software agent orchestration, foundation-model planning, and multi-robot coordination. Physical Agentic AI combines these lines while making capability resolution and execution authorization explicit.
- Robot Architectures: Classical robot architectures and task executives separate deliberative planning from execution-time checking, dispatch, recovery, and state handling.
- Foundation Models for Robot Planning: Foundation models can generate syntactically valid plans that name no executable capability, even when given a strict capability vocabulary.
- Robot Architectures: Capability membership is checked before state, resource, and precondition checks are applied to proposed actions.
- Software Agent Orchestration: Software agentic frameworks provide mechanisms for role assignment, tool registration, and shared context, while robot systems add embodiment and execution constraints.
- Multi-Robot Coordination: Multi-robot coordination work addresses capability grounding, task allocation, feedback, communication, temporal coordination, and collision avoidance.
- Execution Safety: Safe execution requires a boundary where workflow transitions are checked at runtime rather than admitted solely because retrieval improved the planner’s proposals.
D. Multi-Robot Coordination and Physical Grounding
Physical multi-robot missions involve constraints that span robots and change during execution. The architecture therefore separates semantic proposal from deterministic authorization using runtime state and workflow contracts.
- Multi-robot systems require task allocation, communication, temporal coordination, and collision avoidance across heterogeneous agents.
- General agent-orchestration frameworks interpret underspecified human instructions, complementing structured planning and formal methods that encode feasibility or safety constraints.
- A non-actuating Mission Planner proposes workflows, while a deterministic Robot Orchestrator decides whether each action may cross the execution boundary.
- Some execution-critical information appears only after another robot acts, so it must be resolved from runtime state rather than inferred by the planner.
- Retrieval supplies grounded knowledge to the planner, whereas enforcement determines whether a proposed transition is authorized at execution time.
A. Design Principles
The architecture separates semantic reasoning from physical execution through skill-grounded planning, executable workflow contracts, and execution-time verification. A non-actuating planner proposes robot-skill workflows, while a deterministic orchestrator authorizes each dispatch.
- Design Principles: Foundation models interpret natural language and decompose tasks, but a separate execution component retains physical authority.
- Skill-Grounded Planning: The planner reasons only over bounded libraries of implemented skills with typed parameters, preconditions, and expected effects.
- Contract-Mediated Coordination: Workflow contracts encode ordering constraints, synchronization points, resource dependencies, and execution invariants independently of language-model reasoning.
- Execution-Time Verification: Every selected skill is validated against the current robot state and workflow contract immediately before actuation.
- Mission Planner: The Mission Planner decomposes requests into phases, selects robot-skill pairs, and produces workflows without directly communicating with robot controllers.
- Robot Orchestrator: The Robot Orchestrator is a deterministic runtime that maintains execution state and controls access to robot-specific interfaces.
- Robot Orchestrator: Before dispatch, the orchestrator checks skill existence, argument validity, workflow order, preconditions, and runtime-bound values.
C. Execution Protocol
Execution proceeds through retrieved context, structured workflow generation, global validation, and a limited replan before rejection or execution.
- The system retrieves current skill libraries, named locations, and workflow contracts for planning.
- The Mission Planner converts the request into an instantiated contract and ordered robot–skill–argument steps.
- The Robot Orchestrator validates skill and location grounding plus contract ordering across the complete workflow.
- A failed validation permits one replan before the mission is rejected.
D. Implementation
The implementation separates non-actuating language planning from deterministic orchestration and evaluates the architecture in simulation and hardware-equivalent physical testbeds.
- CrewAI is modified so retrieved skill, location, and contract specifications guide planning while matching runtime-enforced declarations.
- The Robot Orchestrator is implemented as procedural code with sole actuation authority.
- The simulation testbed uses an Iris quadcopter and TurtleBot3 for scalable quantitative comparisons and controlled fault injection.
- The hardware testbed uses a Unitree G1 humanoid and Go2 quadruped to provide physical validation on heterogeneous embodiments.
A. Air–Ground Orchestration in Simulation
The air–ground simulation tests orchestration with an Iris drone locating a victim and a TurtleBot3 rover dispatching to the runtime-published position across three scenarios and four conditions.
- The Iris drone localizes a victim while the TurtleBot3 rover drives to the published position in Gazebo under ROS2, PX4 SITL, and MAVROS.
- The victim position is unavailable during planning, so the rover’s goal must be resolved from runtime state at execution.
- Typed skill libraries define the available drone and rover actions, with coordinates bounded to x, y ∈[−6, 10] m.
- The Robot Orchestrator refuses symbolic dispatch when no finite victim fix is published, using the no_target_fix condition.
- SEARCH-AND-DISPATCH contracts order takeoff, search, localization, and dispatch, while their declarative specifications are shared by planner and gate.
- The evaluation separates planner knowledge from runtime enforcement through LLM-only, Skill-list, RO-prompt, and RO conditions.
- The study evaluates 20 scenarios comprising 12 nominal trials and 8 fault injections involving unavailable robots, invalid destinations, and perception faults.
3) Results:
Retrieval substantially improves skill grounding, but only dispatch-time enforcement prevents faulted steps from receiving physical actuation authority.
- 51% to 96%: retrieval raises skill grounding, while informed non-enforcing planners still dispatch 23% of grounding- or state-faulted steps.
- 0% false dispatch: RO eliminates unauthorized faulted actions and blocks all eight injected fault scenarios with 100% recall and precision.
- The workflow gate holds the rover idle until the drone publishes a detection, then releases navigation in the full-system execution timeline.
- The held prompt is identical for RO-prompt and RO, isolating enforcement as the difference in physical actuation authority.
- Runtime state validation blocks corrupted or absent coordinates with a hard no_target_fix refusal instead of dispatching a plausible replacement.
4) Gazebo Execution Validation:
Live Gazebo evaluation tests the orchestration architecture under nominal and injected-fault conditions, measuring both execution outcomes and safety-gate behavior. The results show that runtime validation prevents malformed or unsafe dispatches before robot motion.
- Evaluation setup: 20 scenarios were executed live in Gazebo across 12 nominal and 8 fault-injection missions.All conditions shared the model, schema, parser, and dispatch interface.
- Nominal execution: 0.465 m mean rover goal error was comparable to 0.494 m for a scripted non-LLM run.The reported live-execution range was 0.44–0.49 m.
- Fault handling: The rover remained at its spawn position when NaN-transmitted coordinates failed dispatch-time validation.This demonstrates rejection before the malformed goal reached motion execution.
- Fault handling: Without enforcement, all eight injected violating steps were dispatched and six produced robot motion.Two others produced no motion only because the underlying navigation service rejected malformed goals.
- Evaluation measures: The evaluation separates language-level interpretation from physical executability and safety checking across skill, contract, and dispatch metrics.The humanoid–quadruped setup uses hardware-equivalent skills and workflow contracts such as HANDOVER, CARRY, NAVIGATION-ONLY, and MANIPULATION-ONLY.
2) Results:
The results show that retrieval improves planner grounding but does not ensure safe execution, while deterministic orchestration blocks unsafe dispatches and identifies plausible substitution failures. Hardware-equivalent evaluation and physical demonstrations extend this dissociation across heterogeneous robot crews.
- Result 1: Dissociation: Retrieval repairs skill grounding from 9% to 91%, but non-enforcing arms still dispatch 26–29% of faulted steps.With identical sampled plans, deterministic gating reduced false dispatch from 25.7% (9/35 steps) to 0%.
- Result 1: Dissociation: 0% false dispatch and 100% recall were achieved with Robot Orchestrator enforcement.The reported significance for recall was p=1.6×10−4.
- Result 2: Retrieval-induced substitution: Retrieval-induced substitution changes detectable hallucinations into plausible but incorrect destinations that can bypass syntactic grounding checks.For an invalid rooftop helipad request, the fully informed planner substituted a valid but incorrect dropoff location.
- Result 2: Retrieval-induced substitution: The Robot Orchestrator detects and rejects retrieval-induced substitution through a strict request-containment rule.This constrains actions to the user-requested destination rather than merely checking whether the destination is syntactically valid.
- Physical demonstrations: Hardware demonstrations select HANDOVER for a nearby requester and CARRY for a distant requester.The G1 hands the cube directly when near; when far, it loads the Go2, which transports the cube across the room.
3) Hardware Validation:
Hardware demonstrations transfer the planning–enforcement interface to heterogeneous robots and adapt workflow structure to physical context. The demonstrations also show why enforcement matters for cross-robot sequencing and why hardware conclusions remain qualitative.
- Hardware Validation: The same request yields HANDOVER with direct G1 delivery nearby and CARRY with G1 loading followed by Go2 transport farther away.The physical context changes the selected workflow while preserving the shared planning–enforcement interface.
- Hardware Validation: The CARRY workflow authorizes Go2 departure only after the G1 completes loading, enforcing a cross-robot load-before-depart constraint.Every physical action passes through the same dispatch gate evaluated in the mocked study.
- Hardware Validation: Only two hardware trials were performed, so statistical claims are reserved for the mocked evaluation.The hardware demonstrations provide qualitative evidence of transfer rather than a statistically meaningful timing distribution.
- Hardware Validation: The hardware trials show that the same abstraction can drive a physical humanoid–quadruped crew.The experiments divide labor between foundation-model workflow selection, Robot Orchestration constraints, and validated tool execution.
- Hardware Validation: Per-dispatch enforcement removed false dispatches entirely and achieved 100% fault recall on both testbeds.In live Gazebo execution, enforcement refused all eight injected faults before the violating action executed.