Source-linked AI summary
Agent as Policy for Robotic Manipulation
Mengzhao Jia, Yang Lin, Xixin Zhang, Zhihan Zhang, Xiaobai Liu, Meng Jiang
TL;DR
Physical manipulation requires agents to adapt both evidence interpretation and action generation as task outcomes unfold. AGP makes a general-purpose agent the robot policy, allowing it to write programs, issue motions, and revise actions through a robot interface. Across diverse real-robot tasks, AGP shows strong zero-shot success, while reusable experience improves efficiency and execution cost remains a deployment barrier.
Problem
Physical tasks require flexible adaptation of evidence interpretation and action generation after outcomes such as failed insertions.
Method
AGP uses a general-purpose agent as the robot policy, with runtime programming, robot-interface actions, and feedback-driven revision.
Results
AGP achieves strong zero-shot manipulation performance across diverse real-robot tasks, while accumulated or transferred experience improves execution efficiency and weaker-agent performance.
Takeaways & Limitations
The results support general-purpose agents as viable robot policies through reasoning, programming, and physical interaction.
Takeaways & Limitations
Execution time and inference cost remain substantial barriers to practical deployment.
Abstract
from arXiv · showhide
We demonstrate that a general-purpose agent can directly drive a physical robot throughout task execution without any task-specific or environment-specific training. We introduce Agent as Policy (AGP), which places task planning and execution under the agent's control. Given a task and a robot interface, the agent interprets visual evidence, writes executable programs, issues motion commands, and revises its actions in response to physical outcomes. This brings the agent's reasoning and programming capabilities into continuous interaction with the physical world. We study AGP across multiple real-world manipulation tasks spanning precision manipulation, dynamic motions, and deformable objects. These include assembly from human videos, block construction from goal images, dice flipping, targeted throwing, and bimanual towel folding. Across assembly, block construction, and dice flipping, AGP succeeds in at least eight of ten trials for each evaluated task configuration. We further study efficiency through task experience accumulation and find that reusing saved procedures and programs shortens execution time across repeated trials. These findings support a path for general-purpose agents to act as robot policies, extending their autonomy to physical manipulation through runtime reasoning, programming, and interaction.
1 INTRODUCTION
AGP makes a general-purpose agent the robot policy, giving it runtime control over perception, programming, motion, and recovery across diverse manipulation tasks. Real-robot evaluations show strong zero-shot performance, while execution time and inference cost remain practical barriers.
- AGP lets the agent decide what to observe, how to interpret evidence, which motion to request, and how to revise execution after physical outcomes.The agent can author and revise local programs, using each result to decide whether to gather more evidence or change strategy.
- The system separates one-time task preparation from runtime execution, creating reusable task definitions and selecting tools or programs for each task.Preparation records the goal, constraints, criteria, and interface rules before the execution agent acts.
- AGP evaluates video-guided assembly, image-guided construction, dice flipping, targeted throwing, and bimanual towel folding on real robots.The task set spans precision manipulation, dynamic motions, long-horizon reasoning, and deformable objects.
- AGP’s broader contribution is to extend agent reasoning and programming into continuous physical interaction through a robot interface.The agent interprets observations, computes motion targets, requests actions, and incorporates motion feedback and new observations.
- AGP succeeds in 8 of 10 assembly trials, 29 of 30 block-construction trials, and 10 of 10 dice-flipping trials across the reported configurations.
2 RELATED WORK
Related work places agentic robotics at the intersection of program synthesis, visuomotor control, tool-using language agents, and multiagent workflows. AGP builds on these directions by retaining runtime agent control and feedback-driven adaptation for robot manipulation.
- Agentic robot systems commonly use language models for reasoning, tool use, and adaptation through execution feedback.
- Program-synthesis approaches generate observation-to-action programs and may refine or retain reusable solutions using execution evidence.
- Robot-control research includes geometric planning, learned visuomotor policies, vision-language-action models, and video or world models linking actions with future visual states.
- Language and computer-use agents combine reasoning, tools, reflection, and adaptation, while multiagent frameworks distribute workflows among communicating specialized roles.
3 A GENERAL PURPOSE AGENT AS THE ROBOT POLICY
AGP uses a general-purpose agent as the robot policy, operating through a documented interface, local workspace, runtime programs, and physical feedback. The agent receives observations, computes or selects actions, and adapts until completion or the task budget ends.
- A separate preparation agent creates a reusable task definition, while the execution agent carries out subsequent task instances at runtime.The definition includes goals, reference materials, constraints, allowed variations, completion criteria, budgets, and reporting requirements.
- At runtime, the agent maintains a workspace of scripts, notes, and observations while selecting tool calls from task specifications, conversation history, and workspace contents.The model parameters remain fixed throughout execution.
- The robot interface exchanges camera observations and proprioceptive state with agent requests for observations and actions.This two-way exchange lets the agent observe the scene, direct physical actions, and assess resulting states.
- Runtime programs interpret observations, compute action targets, and submit robot commands, including estimated object positions and target gripper poses.
- After each action, the agent uses motion feedback and new observations to revise estimates, grasp poses, action sequences, observation timing, and recovery.Execution continues until reported completion or the task budget ends, with success assessed against task criteria and recorded evidence.
- Execution-overhead instructions can combine consecutive operations and wait longer for completed action results to reduce tool overhead.
4 EXPERIMENTS
AGP is evaluated as a runtime robot policy across diverse real-world manipulation tasks, with success, resource use, model configurations, and experience reuse examined. The experiments show strong zero-shot success alongside substantial execution overhead, while accumulated or transferred experience improves efficiency and sometimes success.
- 4.2.1 MAIN RESULTS: Deformable manipulation remains difficult: sequential towel folding succeeds in 5/5 trials, while simultaneous folding succeeds in 3/5.Sequential folding averages 50.8 minutes and USD 24.14, among the highest expenditures in the suite.
- 4.2.1 MAIN RESULTS: Execution time and inference cost generally increase with task complexity, and verification can comprise a substantial share of reported runtime.In potato throwing, the throw takes 13.9 minutes on average, while trajectory analysis and verification account for approximately 40% of 22.8 minutes.
- 4.2.2 AGENT AND THINKING EFFORT COMPARISON: AGP supports successful assembly across Codex and Claude Code, but model choice affects outcomes and efficiency.Astra shows consistent success across effort levels, whereas Sol has lower inference cost but longer execution time than Astra; Fable has lower success than Opus with greater time and cost.
- 4.3.1 ACCUMULATING REUSABLE TASK EXPERIENCE: Repeated execution with accumulated experience reduces task time, including a 29.3% decrease from the first to fifth two-pair assembly execution.Response latency decreases by 47.0% while tool execution increases by 9.3%, consistent with reusing programs, geometric estimates, validated parameters, and processing scripts.
- 4.3.2 STRONG-TO-WEAK EXPERIENCE TRANSFER: Transferring Astra’s experience to Terra raises success from 1/5 to 4/5 and reduces successful-trial completion time and token usage by 34.3% and 11.4%.Mean inference cost also decreases when Astra’s experience-collection cost is excluded.
5 CONCLUSION
AGP uses a general-purpose agent as the robot policy, combining runtime programming, observation, motion generation, and physical-feedback revision. Experiments support diverse manipulation capabilities, while reusable experience improves efficiency.
- AGP uses a general-purpose agent as the robot policy through runtime programming, robot-interface interaction, and physical-feedback-based action revision.
- Recorded task experience in reusable files improves execution efficiency as it accumulates.The broader conclusion also identifies execution time, inference cost, and reliability in coordinated deformable-object manipulation as challenges.
AI USE STATEMENT
The authors disclose using generative AI tools to assist with manuscript drafting and language editing. They retain responsibility for reviewing the text and for the manuscript’s content.
- Generative AI tools assisted with drafting and language editing, including sentence structure, clarity, and readability.
- The authors reviewed and revised the resulting text and retain responsibility for the methodology, results, interpretations, conclusions, and overall content.
REPRODUCIBILITY STATEMENT
The paper identifies where experimental setup, task definitions, success criteria, trial protocols, metrics, and supplementary coding-agent details are documented. Assembly parts are adapted from the AutoMate dataset.
- Section 3 covers the agent formulation, robot interface, and runtime execution procedure, while Section 4.1 summarizes the experimental setup.
- Appendices A and B document hardware, control parameters, agent configurations, tasks, reference materials, success criteria, trial protocols, and metric definitions.
- The assembly parts are adapted from the AutoMate dataset, and Appendix C describes the supplementary coding agent.
A IMPLEMENTATION DETAILS
The implementation uses a dual-arm YAM platform with calibrated cameras, a robot-control interface, bounded motion execution, and recorded trial conditions. The system supports visual grounding, command execution, and reproducible comparisons across agent configurations.
- A.1 ROBOT PLATFORM AND CONTROL INTERFACE: Camera observations include calibration, timestamps, and robot state; wrist depth and overhead ray-plane intersections ground image measurements in the arm base frame.
- A.1 ROBOT PLATFORM AND CONTROL INTERFACE: Figure 4 documents two I2RT YAM arms, parallel grippers, wrist RGB-depth cameras, and a fixed overhead RGB camera supporting bimanual towel folding.
- A.1 ROBOT PLATFORM AND CONTROL INTERFACE: The interface provides state, status, help, frames, end-effector motion, joint motion, and gripper commands with returned poses, errors, or status information.
- A.1 ROBOT PLATFORM AND CONTROL INTERFACE: Execution software constrains velocities, accelerations, gripper travel, target envelopes, grasp heights, and maximum Cartesian steps; targeted throwing uses a task-specific timed-motion runtime.
- A.1 ROBOT PLATFORM AND CONTROL INTERFACE: Standard trials begin with empty task experience, while reuse conditions provide saved files and record model, software, hardware, budget, and inference settings.
- A.1 ROBOT PLATFORM AND CONTROL INTERFACE: The Astra sweep holds model, environment, inputs, and interface fixed, while Codex and Claude Code comparisons evaluate complete agent systems under shared scenes, references, and budgets.
B.1 TASK DEFINITIONS AND SUCCESS CRITERIA
The evaluation defines manipulation tasks by capability, instruction modality, and physical success criteria, then measures success and resource use under explicit accounting rules.
- Task definitions: Assembly uses a human video to guide eight parts into four pairs, with submillimeter nominal radial clearances demanding precise alignment.The parts and mating clearances are defined from the AutoMate assets.
- Task definitions: Block construction varies initial cube poses while keeping each configuration’s goal photograph, block set, and placement tolerances fixed.The three configurations are the pyramid, two towers, and six-block tower.
- Task definitions: Dice flipping tests grasp selection and rotation, while targeted throwing tests a joint-driven swing and timed release under fixed geometry and swing conditions.Dice success depends on the requested orientation; throwing is measured relative to the target center.
- Task definitions: Success requires the specified assembly, support, orientation, and fold relationships to remain stable after gripper release and arm withdrawal.The criteria are fixed before evaluation and assessed from the final state and execution video.
- Task definitions: Bimanual towel folding uses overhead demonstrations and evaluates the final folds after release and withdrawal, while trials classify physical failures, budget exhaustion, intervention, and interruptions.Trials begin from an approximately demonstrated initial arrangement, and geometric queries are logged separately from budgeted requests.
- Metric definitions: Completion time includes program construction, observation, computation, motion, and recovery, while token usage sums model input and output across retries.Resource means are reported over successful trials, and configurations without successes have undefined successful-trial resource means.
D.1 TASK EXECUTION TIME ANALYSIS
Execution-time analysis separates visual reasoning, model responses, local tools, and robot action service, showing that observation and decision-making consume substantial time and that experience reuse changes the runtime profile.
- D.1 Task execution time analysis: The time breakdown assigns visual capture, geometry, image processing, and model response to the visual loop, while action service includes planning, movement, settling, and rejected requests.The categories are observable activity measures, with visual reasoning and programming combined.
- D.1 Task execution time analysis: Observing the scene and deciding how to act consume substantial execution time, with the balance varying by task.Throwing spends more time in other model responses, whereas assembly and towel folding devote a large share to the visual loop.
- D.1 Task execution time analysis: Simultaneous towel folding is shorter than sequential folding in both visual-loop and robot-execution time, but timings alone do not isolate the benefit of parallel arm motion.Task strategy and trial outcomes also differ between the configurations.
- D.2 Repeated execution with persistent experience: Repeated-task protocols start with empty experience and then provide fresh contexts containing saved measurements, programs, and lessons from earlier attempts.The agent updates object positions from current images in each later execution.
- D.2 Repeated execution with persistent experience: Experience mainly reduces reasoning and programming time: two-pair assembly improves there while tool execution increases slightly, and the ring task improves most after its first cycle.The first ring cycle establishes an alignment procedure and recovers from a failed insertion.
- D.3 Transferring frozen experience to Terra: Frozen-experience transfer succeeds in four Terra trials, while the unsuccessful trial leaves the circular pair complete but the hexagonal collar separate after the narrow post tips.The empty-experience comparison uses one successful trial versus four with experience, so the observed reductions rely on small, unequal samples.
E RUNTIME PROGRAM ANALYSIS
Runtime program analysis shows the agent constructing reusable visual and geometric tools, linking their outputs directly to robot actions while adapting execution across diverse manipulation tasks.
- Program inventory: The agent retained 37 Python working files across 18 sessions, alongside 63 inherited file copies and seven copies exported for reuse.The working-file inventory excludes inherited and exported copies.
- Geometric estimation: A surface-fitting program estimates a held pin’s pose from wrist depth, computes a corrected target pose, and supplies the resulting position and quaternion to the next motion request.The reported horizontal target is approximately (0.51325, −0.15820) m, rounded in the subsequent request to (0.5133, −0.1582) m.
- Geometric estimation: Triangulation converts selected pixels from wrist and overhead cameras into calibrated rays and computes closest points on those rays to estimate position.The program uses ray origins t1 and t2 and unit directions d1 and d2 in the robot base frame.
- Runtime programs: Runtime programs perform geometric estimation, visual processing, and robot-call composition, including surface fitting, ray triangulation, segmentation, circle detection, and concurrent bimanual control.The agent also creates demonstration montages and logs full responses while printing compact results.
- Task execution: Execution traces show recovery and adaptation: assembly corrects a failed insertion, block construction follows support order, and dice flipping combines a 60° tilt with low release and regrasping.The final states shown for the four-pair assembly and dice tasks satisfy their respective target arrangements.
- Task execution: Targeted throwing uses motion probes and adjusts swing and gripper timing, while occlusion leaves part of the flight trajectory unobserved and requires records and video for assessment.Joint 4 drives the swing while other joint targets remain fixed.
F.5 BIMANUAL TOWEL FOLDING
Bimanual towel folding tests whether AGP can coordinate and recover multi-step deformable-object manipulation from demonstrations, with both sequential and simultaneous executions preserving the fold arrangement despite residual errors.
- Sequential bimanual towel folding: Sequential folding recovers from a slipped corner, corrects a turned edge, performs the three demonstrated folds, and flattens a raised corner.The final result preserves the demonstrated fold order but retains a buckle and edge offsets, making it a partial visual match.
- Simultaneous bimanual towel folding: Simultaneous folding coordinates inward transport and staged flap placement, using further edge correction to maintain clearance between the two arms.One arm holds its corner while the other recovers a slipped edge before the long-edge fold and inward transport.
- Simultaneous bimanual towel folding: Both inward folds are completed and counted as successful, although the final towel retains residual skew, a curled edge, and a gap near the central seam.Success is judged under the fixed criteria requiring the final state after release and withdrawal.
- Cross-embodiment experiment: The cross-embodiment bottle experiment uses a seven-joint P7 arm, a five-finger RealHand L6, overhead and wrist cameras, and 640 × 480 images at 15 Hz.The agent reorients the wrist, closes the fingers in stages, and commands a 2 cm lift before the bottle slips during holding.
- Failure and scope: A six-block tower can be assembled upright before collapsing during withdrawal, illustrating that final stability—not merely placement completion—is part of success.The exact contact or instability causing the collapse remains unresolved, and a displaced cube can also move beyond the interface’s 0.65 m radial grasp limit.
- Implications and limitations: AGP may reduce task-specific engineering and produce reusable programs or trajectories, but practical deployment remains constrained by execution time, inference cost, safety risks, and the need for additional evaluation.Deployment around people or unfamiliar environments requires supervision and independent protective mechanisms.