Source-linked AI summary
Agent as Policy for Robotic Manipulation
Mengzhao Jia, Yang Lin, Xixin Zhang, Zhihan Zhang, Xiaobai Liu, Meng Jiang
TL;DR
Physical manipulation requires flexible interpretation of evidence and adaptation of actions to physical outcomes. Agent as Policy (AGP) gives a general-purpose agent control of robot planning and execution through runtime programming and feedback. It achieves at least 80% success in seven of eight evaluated task configurations.
Problem
Physical manipulation requires flexible evidence interpretation and action generation, including joint reasoning about the scene, attempted motion, and outcome.
Source: §1 · p. 1Method
Results
At least 80% success is achieved in seven of eight task configurations, with 100% observed success in five.
Source: §4.2.1 · pp. 5–6Takeaways & Limitations
The findings support general-purpose agents as viable robot policies, with reusable task experience improving execution efficiency.
Source: §5 · p. 9Substantial execution time and inference cost remain barriers to practical deployment.
Source: §1 · p. 2
Abstract
from arXiv · showhide
We demonstrate that a general-purpose agent can directly drive a physical robot throughout task execution without any task-specific or environment-specific training. We introduce Agent as Policy (AGP), which places task planning and execution under the agent's control. Given a task and a robot interface, the agent interprets visual evidence, writes executable programs, issues motion commands, and revises its actions in response to physical outcomes. This brings the agent's reasoning and programming capabilities into continuous interaction with the physical world. We study AGP across multiple real-world manipulation tasks spanning precision manipulation, dynamic motions, and deformable objects. These include assembly from human videos, block construction from goal images, dice flipping, targeted throwing, and bimanual towel folding. Across assembly, block construction, and dice flipping, AGP succeeds in at least eight of ten trials for each evaluated task configuration. We further study efficiency through task experience accumulation and find that reusing saved procedures and programs shortens execution time across repeated trials. These findings support a path for general-purpose agents to act as robot policies, extending their autonomy to physical manipulation through runtime reasoning, programming, and interaction.
1 INTRODUCTION
Agent as Policy (AGP) gives a general-purpose coding agent runtime control over robot perception, motion generation, and recovery, with model parameters fixed. Real-robot trials demonstrate successful manipulation, while reusable execution records improve efficiency and substantial execution time and inference cost remain deployment barriers.
1 INTRODUCTION
- Earlier approaches constrain runtime decisions to programmed rules or restrict motion generation to existing learned policies.Program generation requires explicit runtime rules (Liang et al., 2022; Chen et al., 2026). Policy orchestration selects and sequences learned policies but remains limited by their capabilities (Li et al., 2026a; Zhang et al., 2026; Galanti et al., 2026).Source: §1 · p. 1
- AGP lets the agent choose observations, interpret evidence, request motions, and revise local programs throughout execution, adapting perception and action to physical outcomes.Recovery requires reasoning jointly about the current scene, attempted motion, and outcome; the agent can gather more evidence or change its execution strategy.Source: §1 · p. 1
- A calibrated robot bridge supplies observations, geometric queries, and validated motion commands, while a persistent coding workspace supports geometry estimation, motion evaluation, and feedback-driven program revision.The bridge returns measured state and execution reports and supports coordinated joint and gripper motion for timing-dependent actions. Motion execution and monitoring run independently of agent inference.Source: §1 · p. 2
- 8 of 10 assembly trials, 29 of 30 block construction trials, and 10 of 10 dice flipping trials succeeded with AGP.Block construction spans three configurations; evaluation also includes targeted throwing and bimanual towel folding. Main trials start in fresh sessions with fixed weights, no prior task execution experience, and no simulation rehearsal.Source: §1 · p. 2
- Reusing recorded procedures, measurements, and corrections improves efficiency; transferring stronger-agent records improves weaker-agent success and reduces successful-trial mean execution time and inference cost.Experience is retained in files for reuse in subsequent executions.Source: §1 · p. 2
- Substantial execution time and inference cost remain barriers to practical deployment despite evidence that foundation MLLMs can directly control robot arms through an interface.Visual perception and reasoning support goal interpretation, motion generation, and adaptation to physical feedback during zero-shot manipulation.Source: §1 · p. 2
2 RELATED WORK
AGP positions a general-purpose multimodal language model directly as the robot policy, keeping an agent in the execution loop rather than relying on a separately trained action policy. It brings agentic reasoning, programming, tool use, and feedback into physical task execution.
2.1 AGENTIC ROBOT SYSTEMS
- AGP uses the runtime agent itself as the policy, rather than separating skill selection from motion generation within a learned policy or primitive.Prior systems select and sequence learned skills using goals, affordances, and feedback (Ahn et al., 2022; Huang et al., 2022). Recent approaches orchestrate VLA policies with progress monitoring and recovery (Li et al., 2026a; Zhang et al., 2026; Galanti et al., 2026), whereas AGP requires no separately trained action policy for zero-shot execution in previously unseen environments.Source: §2.1 · p. 3
- AGP keeps program creation and workflow revision under runtime agent control, responding to observations and execution feedback during deployment.Program-synthesis systems generate observation-to-action programs through perception and control interfaces, with validation supporting simulated and physical execution (Singh et al., 2022; Liang et al., 2022; Chen et al., 2024; Mu et al., 2024). Execution-driven refinement also revises programs or computation graphs and retains reusable solutions (Lu et al., 2026).Source: §2.1 · p. 3
2.2 ROBOT MANIPULATION
- AGP offers an alternative to action-generation-specific training by using a general-purpose foundation multimodal large language model directly as the robot policy.The manipulation literature spans geometric modeling, task and motion planning, and feedback control (Garrett et al., 2021), alongside trajectory-trained diffusion policies (Chi et al., 2023). VLA models condition on images and language (Brohan et al., 2023; Kim et al., 2024; Black et al., 2024), while video and world action models couple actions with future visual states (Li et al., 2025; Ye et al., 2026).Source: §2.2 · p. 3
2.3 AGENTIC WORKFLOWS BEYOND ROBOTICS
- AGP studies how agentic tool use, memory, feedback, and coordination shape robot policies through perception, geometry, planning, and control interfaces.Language agents combine reasoning, tools, and reflection (Yao et al., 2022; Schick et al., 2023; Shinn et al., 2023), supporting software editing and testing (Yang et al., 2024; Wang et al., 2024) and adaptive browser or desktop interaction (Zhou et al., 2023; Xie et al., 2024). Multiagent frameworks distribute workflows among communicating specialized roles that share intermediate results (Wu et al., 2023; Hong et al., 2023).Source: §2.3 · p. 3
3 A GENERAL PURPOSE AGENT AS THE ROBOT POLICY
Agent as Policy uses a general-purpose agent with fixed model parameters to control robotic manipulation through a documented interface. The agent writes programs during execution, interprets observations, and revises actions using physical feedback.
3.1 PROBLEM FORMULATION
- A separate coding agent prepares reusable task definitions, while an execution agent handles task instances under specified goals, constraints, budgets, and completion criteria.Tasks use video, image, language, or combined instructions to specify a desired persistent-scene state within elapsed-time and observation/action-request budgets. Subsequent instances launch directly with the saved definition and current instance details.Source: §3.1 · p. 3
- A general-purpose agent serves as the robot policy, selecting tools using task specifications, interface documentation, conversation history, and workspace files while model parameters remain fixed.Tool calls acquire information or execute programs that submit action targets; the workspace retains scripts, notes, and observations.Source: §3.1 · p. 4
3.2 AGENT-ROBOT BRIDGE
- The agent-robot bridge exchanges observations and action requests, supplying calibrated camera views, proprioceptive state, and post-motion target errors for assessing physical outcomes.Overhead images show the scene, while wrist images with aligned depth support spatial measurement in a shared robot coordinate frame. Motor feedback supplies joint positions and forward-kinematics end effector poses; observations are saved for inspection and processing.Source: §3.2 · p. 4
- The agent requests joint-angle or end effector targets and gripper openings, while a feedback controller executes trajectories subject to workspace and motion limits.End effector targets use Mink for inverse kinematics. The controller maintains posture between requests, executes commands sequentially within each arm, and permits concurrent execution across arms.Source: §3.2 · p. 4
3.3 RUNTIME PROGRAMMING AND EXECUTION
- The agent writes runtime programs to interpret observations and compute robot commands, then uses feedback to revise estimates, grasps, action sequences, and recovery strategies.Execution continues until reported completion or termination within budget; physical success is assessed from the resulting scene and recorded evidence. Depending on experimental condition, longer waits and combined operations reduce tool overhead.Source: §3.3 · p. 4
4 EXPERIMENTS
AGP achieves at least 80% success in seven of eight manipulation configurations, although execution time, inference cost, and deformable-object coordination remain challenges. Reusing accumulated experience improves execution efficiency, and transferring Astra’s experience to Terra improves success on two-pair assembly.
4.1.1 ROBOT PLATFORM AND AGENT CONFIGURATION
- AGP uses six-joint I2RT YAM arms with parallel grippers and fixed-weight GPT-6 Astra at high thinking effort by default.Bimanual towel folding uses two arms sharing a coordinate frame, with coordination selected by the agent. Targeted throwing uses an interface supporting timed motion programs.Source: §4.1.1 · p. 5
4.1.2 TASKS AND EVALUATION
- Video, image, and language instructions test long-horizon reasoning, action diversity, and object diversity through assembly, block construction, dice flipping, throwing, and towel folding.Trials use predefined scenes and fixed time/request budgets, permit recovery within those budgets, and end upon reset or human intervention; model and experience comparisons use matched scenes and counterbalanced order. Success follows predefined physical-outcome criteria, while time, total tokens including reasoning and retries, and inference cost summarize successful trials, with block and towel configurations reported separately.Source: §4.1.2 · p. 5
4.2.1 MAIN RESULTS
- At least 80% success is achieved in seven of eight task configurations, with 100% observed success in five.Execution time and inference cost generally increase with task complexity. Potato throwing occurs after 13.9 minutes on average, while subsequent trajectory analysis and verification account for approximately 40% of the reported 22.8 minutes.Source: §4.2.1 · pp. 5–6, Table 1 · p. 6
- 3/5 success for simultaneous towel folding is the lowest observed rate, whereas sequential folding succeeds in all five trials.Sequential folding nevertheless requires 50.8 minutes and USD 24.14 on average, among the suite’s highest expenditures. Simultaneous folding requires bringing both short ends inward together, highlighting difficulty coordinating dynamic deformable-object manipulation.Source: Table 1 · p. 6, §4.2.1 · p. 6
4.2.2 AGENT AND THINKING EFFORT COMPARISON
- AGP supports assembly with both Codex and Claude Code, while Astra maintains consistent success across thinking-effort levels and model choice remains important.On two-pair assembly with shared task references, robot functions, and trial budgets, Sol costs less but takes longer than Astra. Within Claude Code, Fable achieves lower success than Opus and uses fewer tokens on successful trials, yet incurs greater time and cost.Source: §4.2.2 · pp. 6–7, Table 2 · p. 7
4.3 IMPROVING EFFICIENCY THROUGH EXPERIENCE
- Experience studies address substantial execution latency and inference cost through repeated-task reuse and stronger-to-weaker procedure transfer on two-pair assembly.Both studies use fitting a hexagonal ring onto a narrow post and a circular collar onto a stepped cylinder, selected from the four-pair assembly task to limit evaluation costs.Source: §4.3 · p. 7
4.3.1 ACCUMULATING REUSABLE TASK EXPERIENCE
- 29.3% lower two-pair assembly task time from the first to fifth execution accompanies experience reuse; ring disassembly and reassembly also becomes faster after its first cycle.Fresh contexts reuse saved geometric estimates, validated grasp and insertion parameters, corrections, and scripts while updating current object poses; experience-writing time remains relatively stable. Assembly response latency decreases by 47.0%, while tool execution increases by 9.3%; these dependent trials support an association rather than an isolated causal effect.Source: §4.3.1 · pp. 7–8, Figure 2 · p. 8
4.3.2 STRONG-TO-WEAK EXPERIENCE TRANSFER
- 1/5 success with empty experience rises to 4/5 when Terra receives Astra’s saved experience on two-pair assembly.Among successful trials, mean completion time decreases by 34.3% and token usage by 11.4%; mean inference cost also decreases, excluding Astra experience collection. Each condition uses five fresh-context Terra trials with fixed weights and high thinking effort, comparing empty experience against frozen files collected over five Astra executions.Source: Figure 3 · p. 8, §4.3.2 · p. 8
5 CONCLUSION
AGP uses a general-purpose agent as a robot policy, supporting diverse manipulation through runtime programming and physical feedback. Accumulated experience improves efficiency, while transferring records from stronger to weaker agents improves task success and reduces execution time and inference cost on successful trials.
5 CONCLUSION
- AGP supports diverse robotic manipulation by using a general-purpose agent to interpret observations, generate motion requests, and revise actions through physical feedback.Real-robot experiments evaluate long-horizon reasoning, action diversity, and object diversity under video, image, and language instructions.Source: §5 · p. 8
- Transferring experience records from a stronger agent improves a weaker agent’s task success rate and reduces mean execution time and inference cost on successful trials.The execution-time and inference-cost improvements are reported specifically for successful trials.Source: §5 · p. 9
- Task experience recorded in reusable files improves execution efficiency as it accumulates.Source: §5 · pp. 8–9
- Execution time, inference cost, and reliability in coordinated deformable-object manipulation remain key challenges.Source: §5 · p. 9
AI USE STATEMENT
The authors used generative AI for drafting and language editing, while retaining responsibility for reviewing the text and for all manuscript content.
- The authors take full responsibility for the manuscript’s methodology, experimental results, interpretations, and conclusions.Source: AI USE STATEMENT · p. 9
- Generative AI tools assisted with drafting and language editing, including sentence structure, clarity, and readability.Source: AI USE STATEMENT · p. 9
- The authors are responsible for reviewing and revising AI-assisted text to ensure it accurately represents the research.Source: AI USE STATEMENT · p. 9
REPRODUCIBILITY STATEMENT … B TASKS AND EVALUATION PROTOCOL
The reproducibility materials specify AGP’s robot platform, observation and action interfaces, motion constraints, and trial configurations. Standard trials begin without task experience, while experience-reuse conditions supply saved files.
REPRODUCIBILITY STATEMENT
- The reproducibility statement maps the agent formulation, implementation, evaluation protocols, supplementary comparisons, timing studies, and visual references to their respective sections and appendices.Appendix B documents tasks, references, success criteria, trial protocols, and metrics; assembly parts are adapted from AutoMate (Tang et al., 2024). Appendices C, D, and F cover the pyramid coding-agent comparison and costs, execution timing and experience checkpoints, and visual references and execution sequences, respectively.Source: REPRODUCIBILITY STATEMENT · p. 9
A.1 ROBOT PLATFORM AND CONTROL INTERFACE
- Wrist depth maps pixels into the arm base frame, while overhead localization intersects camera rays with a plane at an estimated height.Captures include calibration, timestamps, and robot state; the agent can request additional views to refine estimates. Camera stream rates are independent of the agent’s decision rate.Source: §A.1 · p. 13
- The platform combines two I2RT YAM arms, parallel grippers, wrist-mounted RGB-depth cameras, and a fixed overhead RGB camera; both arms support towel folding.Figure 4 identifies the linear_4310 grippers, Intel RealSense D405 wrist cameras, and Logitech BRIO overhead camera.Source: Figure 4 · p. 13
- The robot interface exposes seven commands for inspecting state and status, retrieving help and observations, moving the end effector or joints, and controlling the gripper.Motion commands return achieved states or failure information, while observation commands return saved images and calibration data. The agent computes geometry in local programs using images, depth, and calibration.Source: §A.1 · p. 13
- Ordinary motion is constrained by joint and Cartesian speed limits, gripper travel rate, target envelopes, and a maximum Cartesian step.Joint velocity and acceleration limits are 20°/s and 40°/s^2; Cartesian translation and orientation limits are 0.03 m/s and 10°/s. Gripper targets change at 0.25 normalized travel per second; radial targets span 0.12–0.65 m, grasp-point heights span −0.050–0.60 m, and Cartesian steps cannot exceed 0.25 m.Source: §A.1 · p. 14
- The recorded throwing case uses a task-specific runtime with timed joint offsets and gripper targets to coordinate the swing and release.The executor interpolates commands at 50 Hz and records measured joint position, velocity, effort, and dispatch timing. These programs permit joint 4 velocity and acceleration limits of 180°/s and 360°/s^2.Source: §A.1 · p. 14
A.2 AGENT CONFIGURATION AND EXECUTION CONDITIONS
- Standard trials start with empty task experience, whereas reuse conditions supply specified saved files; each trial records software, hardware, budgets, and inference settings.Execution-efficiency instructions remain fixed within each condition. Traces record waiting intervals, intermediate status checks, and operations grouped within each tool call.Source: §A.2 · p. 14
A.3 MODEL AND REASONING EFFORT CONFIGURATIONS
- The Astra effort sweep fixes the model version, environment, inputs, and interface; Codex–Claude Code comparisons evaluate complete systems under shared scenes, references, and budgets.Claude Code uses native model and effort controls (Anthropic, b). Each trial records requested and resolved model identifiers and the effective effort setting.Source: §A.3 · p. 14
B.1 TASK DEFINITIONS AND SUCCESS CRITERIA · B.2 TRIAL PROTOCOL · B.3 METRIC DEFINITIONS AND RESOURCE ACCOUNTING
The evaluation fixes task-specific physical success criteria and trial inclusion rules, distinguishing task success from controller arrival tolerances. Resource accounting separates successful-trial statistics, agent tool decisions, and individual observation and action requests.
B.1 TASK DEFINITIONS AND SUCCESS CRITERIA
- Assembly uses AutoMate parts (Tang et al., 2024), with every method receiving the same human video, extracted frames, and timestamped textual guide.Aspect-ratio-preserving printed meshes have submillimeter nominal radial clearances, with c = (Dh - Ds)/2 calculated from effective bore and shaft diameters. Physical insertion tolerance also depends on printing accuracy and angular alignment.Source: §B.1 · p. 14, Table 4 · p. 16
- Block construction varies initial cube poses against fixed goal photographs, while towel folding starts approximately from demonstrated arrangements.Block sets and placement tolerances are fixed before evaluation. Towel configurations provide overhead demonstration videos and initial/final images; placements may be staged for arm clearance, and final folds are assessed overhead after release and withdrawal.Source: §B.1 · pp. 14–15
- Dice flipping balances distinct initial and target faces across predefined conditions, while throwing fixes geometry, initial arrangement, and permitted swing conditions before evaluation.Video annotations capture dice grasp selection, rotation, and regrasping. Throwing measures landing error relative to the target center, with execution records and video establishing joint motion, gripper opening, and object separation.Source: §B.1 · p. 14
- Task success requires predefined physical outcomes that remain stable after gripper release and arm withdrawal, distinct from controller arrival tolerances.Table 3 defines each task’s target capability, instruction type, and success criteria. Success is assessed from final states and execution video.Source: §B.1 · p. 14, Table 3 · p. 15
B.2 TRIAL PROTOCOL
- Trial records distinguish physical failure, budget exhaustion, human intervention, and infrastructure interruption under a predefined inclusion policy.Geometric queries and local computation are logged separately from budgeted observation and action requests. Equation 1’s k counts agent tool decisions, whereas the interaction budget counts individual observation and action requests, including requests grouped within one tool call.Source: §B.2 · p. 15, §B.3 · p. 16
B.3 METRIC DEFINITIONS AND RESOURCE ACCOUNTING
- Success rate is 100s/n, and completion-time statistics use successful trials; configurations without successes have undefined successful-trial resource means.Here s counts completely successful trials among n evaluated trials. Completion time spans task delivery through final physical verification, including programming, observation, computation, motion, and recovery; Appendix D.1 additionally includes unsuccessful trials and terminal reporting.Source: §B.3 · pp. 15–16
- Token and cost summaries use successful trials; total tokens aggregate input and output across every model request, including retries.Repeated context counts on every request, cached input counts once within that request’s input total, and normalized provider records count reasoning once as output. Tables report mean total tokens per successful trial in millions.Source: §B.3 · pp. 15–16
C SUPPLEMENTARY CODING AGENT COMPARISON ON PYRAMID CONSTRUCTION · D EFFICIENCY AND EXPERIENCE REUSE ANALYSIS
On pyramid construction, Sol uses more tokens per successful session but has lower estimated inference cost than Astra. Task timing averages include failed trials and final reporting.
C SUPPLEMENTARY CODING AGENT COMPARISON ON PYRAMID CONSTRUCTION
- Two supervisor interventions with incomplete structures and one robot bridge disconnection before physical execution account for Sol’s unsuccessful sessions.Interrupted sessions count as unsuccessful.Source: C SUPPLEMENTARY CODING AGENT… · p. 16, Table 5 · p. 17
- The supplementary pyramid comparison evaluates GPT-5.6 Sol and GPT-6 Astra within Codex, both at high thinking effort.The task follows Table 3.Source: C SUPPLEMENTARY CODING AGENT… · p. 16
- Standard API prices normalize the cost comparison, including Astra’s Fast service configuration, although executions use Pro subscriptions.Prices were retrieved September 9, 2026; per-request usage is checked against cumulative counters using Appendix B.3’s token convention.Source: C SUPPLEMENTARY CODING AGENT… · p. 16, Table 5 · p. 17
- Sol uses more tokens per successful session but has lower estimated inference cost than Astra on pyramid construction.The supplied table caption does not provide numerical results.Source: Table 5 · p. 17, C SUPPLEMENTARY CODING AGENT… · p. 17
- Mean tokens and cost cover successful sessions only, whereas total cost covers all ten sessions per model.Successes are completed pyramids recorded in session reports.Source: Table 5 · p. 17
D EFFICIENCY AND EXPERIENCE REUSE ANALYSIS
- Time allocation reports mean minutes across all evaluated trials, including failures, with final reporting included in total time.Throwing also includes trajectory analysis and verification; timing components sum to the total before rounding.Source: Table 6 · p. 17
D.1 TASK EXECUTION TIME ANALYSIS · D.2 REPEATED EXECUTION WITH PERSISTENT EXPERIENCE · D.3 TRANSFERRING FROZEN EXPERIENCE TO TERRA
Observation and decision-making consume substantial execution time, while saved experience mainly reduces reasoning and programming time across repeated executions. Terra achieves four successful assemblies using frozen Astra experience, although resource comparisons rely on small, unequal samples.
D.1 TASK EXECUTION TIME ANALYSIS
- Observation and action selection consume substantial execution time, with throwing emphasizing other model responses and assembly and towel folding emphasizing the visual loop.Timing includes failures and runs from task delivery through completion or timeout, including final reporting. The visual loop combines image capture, geometric queries, image processing, and subsequent model responses, measuring visual reasoning and programming together.Source: §D.1 · p. 17
- Simultaneous folding takes less total, visual-loop, and robot-execution time than sequential folding, but differing strategies and outcomes prevent isolating parallel-arm benefits.Overlapping activities, including simultaneous arm movements, count only once in the timing analysis.Source: §D.1 · p. 17
D.2 REPEATED EXECUTION WITH PERSISTENT EXPERIENCE
- Reasoning and programming account for the main time savings across repeated executions with persistent experience.Two-pair tool execution takes slightly longer; the ring task improves most after the first cycle establishes alignment and recovers from a failed insertion. Reasoning and programming time is a combined estimate including planning, code generation, and response or tool-start delays.Source: Table 7 · p. 18, §D.2 · p. 18
- Five repetitions per task yield successful ring cycles and reported stable two-pair assemblies throughout.GPT-6 Astra at high thinking effort starts with empty experience; later executions use fresh contexts with saved measurements, programs, and lessons while updating object positions from images. Ring cycles remove and reassemble the ring; two-pair assembly combines a hexagonal ring with a narrow post and a circular collar with a stepped cylinder.Source: §D.2 · p. 18
- Later agents reuse measurement and motion programs, update script coordinates, and retain corrected grasp and insertion procedures, linking reuse with shorter response times.Trials form an ordered sequence with changing object positions; comparisons starting every trial with empty experience are needed to isolate the effect of reuse.Source: §D.2 · p. 18
D.3 TRANSFERRING FROZEN EXPERIENCE TO TERRA
- 4 Terra trials achieve successful assembly using identical frozen Astra experience files.In the unsuccessful trial, the circular pair is completed, but the hexagonal collar remains separate and the narrow post tips during recovery.Source: §D.3 · p. 18
- 1 successful Terra trial without experience is compared with 4 using experience, making observed resource reductions dependent on small, unequal samples.Costs cover Terra inference at Standard rates. Completion follows the final result update after motion execution, or recorded task completion when no subsequent update exists.Source: §D.3 · p. 19
E RUNTIME PROGRAM ANALYSIS … F.4 TARGETED THROWING
Runtime-generated programs perform geometric estimation, visual processing, and coordinated robot execution. Qualitative examples show recovery during assembly, stable block construction, successful dice orientation, and targeted throwing into a bowl.
E RUNTIME PROGRAM ANALYSIS
- 37 Python working files across 18 sessions and 135 inline-Python shell events across 48 sessions document runtime programming.The working-file inventory excludes 63 inherited Python file copies and seven copies exported for reuse. All 18 saved-file sessions contain inline-Python events; events and files are counted separately because events can combine operations or create programs.Source: E RUNTIME PROGRAM ANALYSIS · p. 19
- Runtime geometry calculations directly inform assembly motion commands, while program extensions and position adjustments continue the feedback loop after an unsuccessful motion.The agent extends fit_head.py from depth-based plane fitting to pin-tip offset estimation and corrected target orientation and position. The next motion uses the computed target and quaternion; after a target arrival error, the agent adjusts position and continues insertion.Source: E RUNTIME PROGRAM ANALYSIS · p. 19
- Runtime programs also triangulate targets, process visual observations, and coordinate concurrent bimanual operations, although their performance contributions require controlled comparisons.Reusable triangulation returns the midpoint and separation of closest points on calibrated wrist and overhead rays; accuracy depends on correspondences and calibration. Other helpers perform color segmentation, circle detection, and frame montages, or dispatch concurrent arm operations followed by observations while logging full responses.Source: E RUNTIME PROGRAM ANALYSIS · pp. 19–20
F.1 FOUR PAIR ASSEMBLY
- Four pairs are assembled after release and withdrawal, despite a failed insertion that displaces a nut onto the table.The agent adjusts its grasp position and gripper orientation, seats the nut, and completes the remaining assemblies while preserving earlier progress.Source: §F.1 · p. 20, Figure 5 · p. 21
F.2 BLOCK CONSTRUCTION
- Three structures—the pyramid, two towers, and six-block tower—remain standing after release and withdrawal, with placement order following their support structures.The pyramid grows tier by tier; both tower bases precede their middle and top cubes. The six-block tower retains its base while successive placements maintain alignment.Source: Figure 6 · p. 22, §F.2 · p. 22
F.3 DICE FLIPPING
- Six dice finish with six pips upward after withdrawal, using grasp adjustments, a 60-degree tilt, low release, and observation-guided regrasping.The grasp adjustments accommodate the arm’s reachable poses, and subsequent observations verify orientation.Source: Figure 7 · p. 23, §F.3 · p. 23
F.4 TARGETED THROWING
- Two retained-object motion probes precede swing and release adjustments that leave the thrown object inside the bowl.Joint 4 drives the swing while other joint targets remain fixed. Occlusion hides part of the rising trajectory and apex, so assessment also uses execution records and video.Source: §F.4 · p. 24, Figure 8 · p. 25
F.5 BIMANUAL TOWEL FOLDING … I BROADER IMPACT
AGP recovers towel-folding errors and coordinates a different arm and dexterous hand, but residual misalignment, bottle slippage, and a tower collapse reveal execution limitations. Potential applications include reusable skills and trajectory collection, subject to safety, data-quality, cost, and latency constraints.
F.5 BIMANUAL TOWEL FOLDING
- Sequential towel folding preserves the demonstrated fold order after withdrawal but achieves only a partial visual match, with a remaining buckle and edge offsets.The agent recovers a slipped corner, corrects a turned-back edge, and brings the doubled towel closer to the arm bases. One arm holds while the other folds an end inward, followed by grasps that flatten a raised corner.Source: §F.5 · p. 26, Figure 9 · p. 27
- Simultaneous towel folding succeeds under the stated criteria despite residual skew, a curled edge, and a gap near the central seam.After recovering a slipped edge and completing the long-edge fold, both arms transport the short ends inward and place the flaps sequentially for clearance. A further correction anchors an outer edge while drawing a raised inner hem toward the center.Source: §F.5 · p. 28, Figure 10 · p. 29
G EXECUTION ACROSS ARM AND END EFFECTOR EMBODIMENTS
- AGP grasps and lifts a bottle using a different arm and dexterous hand, providing qualitative coordination evidence, although the bottle subsequently slips during holding.The platform combines a seven-joint P7 arm, a five-finger RealHand L6 hand, and overhead and wrist cameras. Wrist reorientation, incremental approach, and staged finger closure precede a commanded 2 cm upward wrist motion.Source: G EXECUTION ACROSS ARM AND END… · p. 30, Figure 12 · p. 31
H.1 SIX BLOCK TOWER COLLAPSE DURING WITHDRAWAL
- Six cubes stand after release, but the tower collapses during arm withdrawal; the exact contact or instability causing the collapse remains unresolved.The agent first corrects the fourth cube’s alignment and transfers the top two cubes together to overcome limited reach. The case demonstrates why success requires stability during withdrawal.Source: §H.1 · p. 32
- 0.70 m is the approximate x-coordinate of a scattered cube beyond the 0.65 m radial grasp limit, preventing rebuilding despite remaining time and command budgets.Overhead geometry and wrist depth agree on the cube’s location. Unlike the earlier alignment error, the resulting out-of-reach state ends the attempt unsuccessfully.Source: §H.1 · p. 32
I BROADER IMPACT
- AGP may reduce task-specific engineering for manipulation from video, image, and language instructions, while saved programs and execution records could support reusable robot skills.Source: I BROADER IMPACT · p. 33
- Repeated AGP execution could collect trajectories for policy learning, with practical value depending on trajectory quality, collection costs, and trained-policy performance.Synchronized visual observations, actions, and measured robot states could train vision language action policies. Failed attempts and recovery segments could support robustness research.Source: I BROADER IMPACT · p. 33
- Perception errors, faulty programs, and inappropriate motions can damage equipment or injure people; deployment requires safeguards beyond the implementation’s runtime workspace and motion limits.The implementation separates agent reasoning from motion execution; deployment around people or in unfamiliar environments requires additional safety evaluation, appropriate supervision, and independent protective mechanisms. Inference costs and execution latency also constrain practicality and accessibility.Source: I BROADER IMPACT · p. 33
Related papers
This paper cites
- OpenVLA: An Open-Source Vision-Language-Action Model2024
- $π_0$: A Vision-Language-Action Flow Model for General Robot Control2024
- SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering2024
- Diffusion Policy: Visuomotor Policy Learning via Action Diffusion2023
- RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control2023
- OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments2024
Cited by
- Pretraining Knows What, Simulation Teaches How: Compositional Generalization in a Sim-Trained Robot Agent2026
- Toward Recursive Self-Improvement in Physical AI2026
- HexaShell: A Thin Harness for Exploring the Limits of VLM Agents as Robot Coding Agents2026
- Has GPT-6-Astra Achieved Embodied Intelligence?2026
- Towards Agentic Robots: Model, Data, Environment, and Harness2026
- ChunkAct: Action Chunking for Agentic Robotics2026