Source-linked AI summary
In-Context Robot Learning with VLM Agents
Dongzhou Cheng, Taoran Yi, Ye Fang, Xingwu Zhang, Fan Feng, Yixuan Li, Gengxiong Zhuang, Rongze Wang, Shuai Yang, Wei Song, Weizhi Xue, Minyan Wu, Jie Gui, Jiaqi Wang, Tong Wu
TL;DR
Robots need deployment-time in-context learning because finite training datasets cannot cover unfamiliar tasks and situations. GPT-Policy connects a general-purpose VLM to contextual demonstrations and interaction history through a compiled, closed-loop context-to-action interface. The framework finds that human videos improve real-robot task completion without robot action labels, aligned action references further help contact-sensitive tasks, and reliable physical execution remains a limitation.
Problem
Finite training datasets cannot cover every situation robots encounter, motivating in-context adaptation from demonstrations, examples, and interaction feedback without task-specific parameter updates.
Method
GPT-Policy combines a context compiler, a general-purpose VLM that proposes robot-tool actions, and a constrained execution layer that verifies actions, executes them, and returns outcomes for replanning.
Results
Human videos improve real-robot task completion without robot action labels, while aligned action references provide further gains on contact-sensitive tasks.
Takeaways & Limitations
GPT-Policy provides an empirical framework for studying when general-purpose VLMs use context to adapt robot behavior across tasks and execution conditions.
Takeaways & Limitations
The evidence covers small task series under selected platform, model, and context conditions, and existing safeguards are insufficient by themselves for safe autonomous deployment.
Abstract
from arXiv · showhide
Enabling robots to adapt to unfamiliar environments as readily as humans remains a moonshot goal of embodied AI. No finite collection of demonstrations can cover every task and situation a robot will encounter, making the ability to learn from context at deployment essential for generalization. Such in-context learning (ICL), however, remains largely beyond the reach of existing robotic policies. The broad agentic capabilities of commercial vision-language models (VLMs), such as GPT-6 Astra, raise a compelling question: can these models learn from demonstrations, examples, and interaction feedback, then translate that information into executable and verifiable robot behavior from a new initial state without gradient updates or persistent changes to task-specific parameters? We introduce GPT-Policy, a general-agent framework for in-context robot learning. GPT-Policy integrates a context compiler that preserves task-relevant visual transitions, a VLM that proposes robot-tool actions, and a constrained controller that verifies and executes each action and reports its outcome. We evaluate its reliability and limitations through task success and efficiency metrics, matched comparisons across models, and controlled context ablations. In real-robot trials, human video demonstrations improve task completion even without robot action labels, while aligned action references yield further gains on contact-sensitive tasks. These findings position GPT-Policy as a step toward robot adaptation through in-context learning, providing an empirical foundation for translating the general-purpose capabilities of VLMs into physical behavior and clarifying the challenges that must be overcome for reliable deployment.
1 Introduction
Robotic in-context learning asks whether general-purpose VLMs can use deployment-time context to adapt robot behavior in unfamiliar situations without task-specific updates. GPT-Policy studies this through a shared closed-loop interface, finding that relevant context can improve success and efficiency while execution reliability remains unresolved.
- 1 Introduction: Robotic ICL adapts behavior from demonstrations, examples, or interaction experience at test time without gradient updates or persistent task-specific parameter changes.The robot must extract relevant information from context and apply it when the current situation differs from demonstrations.
- 1 Introduction: GPT-Policy connects an off-the-shelf VLM to robot tools through context compilation, action proposals, constrained execution, outcome feedback, and replanning.The framework supports examining different context forms and models within a common execution interface.
- 1 Introduction: The framework evaluates five context families across imitation, contact-sensitive manipulation, goal-image following, exploration, and human-robot interaction.Matched model comparisons and controlled ablations measure task completion, decision count, and execution time.
- 1 Introduction: Task-relevant context can improve task completion while reducing decision count and execution time.The evaluation examines whether context changes both success and efficiency across the included task families.
- 1 Introduction: Human videos can improve task completion without robot action labels, while aligned action references provide additional benefits for contact-sensitive tasks.The examples span heterogeneous context, including human and robot videos, goal images, interaction, and self-interaction history.
- 1 Introduction: Context can guide a robot toward appropriate behavior while leaving critical execution failures unresolved across the perception-action loop.Dependable physical execution remains a separate challenge from contextual task understanding and adaptation.
2 Related Work
Related work places GPT-Policy at the intersection of general-purpose agentic models and demonstration-conditioned robot control. Prior approaches use language, vision, trajectories, geometric representations, and human demonstrations to specify robot behavior.
- 2 Related Work: General-purpose models support reasoning, tool use, and multi-step execution, while robotics systems apply them to affordances, program synthesis, and spatial motion objectives.This line of work connects broad agentic capabilities with perception and control interfaces.
- 2 Related Work: GPT-Policy complements dedicated robot policies by examining how fixed general agents use heterogeneous context to guide behavior without parameter updates.The supplied related-work passages motivate studying general-purpose models alongside existing demonstration-conditioned control paradigms.
- 2 Related Work: In-context robot policies use deployment-time examples to specify desired behavior through robot sensorimotor trajectories, geometric demonstrations, and human visual demonstrations.These approaches build on in-context learning and one-shot imitation.
3 Method
GPT-Policy connects a fixed VLM to constrained robot tools through a context-to-action interface that preserves references, executes requests, and feeds outcomes back for replanning.
- Context construction: GPT-Policy combines task instructions, observations, references, interaction history, system instructions, and tool schemas to produce VLM tool requests.The interface supports goal images, demonstration videos, recorded actions, robot states, and online history as context.
- Closed-loop formulation: The VLM selects parameterized tool actions from the current task context and preceding tool result, while its parameters remain fixed during execution.Completed or rejected requests produce feedback and updated observations for the next decision.
- Constrained execution: The execution harness resolves Cartesian targets, samples pose paths, checks inverse-kinematics residuals, time-scales joint references, and dispatches motion.Cartesian motion preserves gripper commands, which are changed through separate gripper tools.
- Constrained execution: Pose paths use linear position interpolation and quaternion SLERP along the shorter rotation arc before inverse-kinematics solving.The adapter samples the geometric path before converting sampled TCP poses and solving IK.
- Feedback and termination: Execution and rejection feedback update the context for replanning, while faults or operator interruption can terminate trials.Model-declared completion remains distinct from physical task success, with optional completion review extensions available.
4 Experiments
The experiments assess whether a general-purpose agent can adapt during closed-loop robotic-arm operation using different forms of context.
- Experimental goals: The evaluation tests context-based adaptation during closed-loop robotic-arm operation.It is designed to address three experimental questions.
- Experimental goals: The experiments vary the forms of context available to the general-purpose agent.The passage frames context variation as the basis for evaluating adaptation.
- Experimental goals: The study is organized around three questions about robotic adaptation during operation.The supplied passage introduces the questions but does not state their contents.
2. Can this information improve task completion rates and reduce the number of decisions?
Across real-robot tasks, deployment-time context improved task completion and often reduced decisions or execution time, with the strongest gains depending on context type and task.
- Human videos: 2/3 success with Human Video versus 0/3 with None on both towel and notebook pickup, while decisions and execution time also decrease.Towel decisions fall from 96.3 to 76.7 and time from 24.6 to 18.9 minutes; notebook decisions fall from 94.0 to 66.7 and time from 24.6 to 16.1 minutes.
- Robot videos and actions: Robot Video + Action reaches 3/3 bottle-opening and 2/3 plug-reinsertion success, outperforming None and Robot Video on both tasks.Action references provide time-aligned poses, gripper states, and commands, but the associated decision and time costs do not consistently decrease.
- Goal images: 3/3 success on both goal-image tasks demonstrates that a single target image can specify layouts more precisely than words alone.The target image jointly specifies object identity, relative position, and spacing.
- Interaction history and human interaction: Self-interaction history and online human interaction support 3/3 success on exploration, placement, turn-taking, and pointing tasks.The agent removes a towel to locate a hidden plate, avoids obstacles during exploration, and combines turn coordination with strategic move selection.
- Robot videos and actions: Action references improve alignment with demonstrated motion by supplying intermediate poses and gripper transitions between sparse video keyframes.Selected bottle-opening runs more closely match requested orientations and measured support posture with action references than with video alone.
5 Discussion
GPT-Policy shows that contextual information can improve robot task performance and efficiency, while reliable physical execution, safety, and broader validation remain unresolved.
- Comparison with Other Models: Human video increased GPT-6 Astra’s red towel pickup progress from 55% to 100%, with approximately 35.6% shorter run time and 58.9% lower estimated token usage.GPT-6 Astra succeeded in 2/3 Human Video trials, and the comparison does not establish a reliable model ranking.
- A Little Context Goes a Long Way: Context helps VLM agents handle fine manipulation and deformable-object tasks beyond their usual operating conditions.Human videos reveal task procedures, while action references provide more precise motion cues.
- Good Plans, Tricky Execution: Reliable deployment remains limited by precise pose generation, contact execution, outcome verification, and physical safety.The discussion distinguishes improved task understanding and planning from dependable physical behavior.
- Moves Take Time: VLM decisions can be slow and expensive, motivating hybrid systems that combine contextual reasoning with fast low-level controllers.Specialized VLA/WAM models may retain an advantage for fast, low-level control.
- Physical Safety for VLM-Driven Manipulation: Future work should address joint-arm collision safety and contact-aware grasping through trajectory checks, slip detection, force limits, and local recovery.These mechanisms are proposed to strengthen safe manipulation without requiring the agent to reason through every correction.
- Future Directions: Long-horizon adaptation may require composing subskills, updating physical-dynamics predictions from interaction history, and coordinating active perception with mobile manipulation.Suggested directions include retaining completed subgoals, adapting to friction and compliance, and using spatial memory with coordinated base–arm control.
- Scope and Limitations: Current evidence covers small task series under selected platforms, models, and context conditions, limiting causal and novel-skill claims.Transfer across embodiments and reliable autonomous operation require broader evaluation.
6 Conclusion
The study examines fixed general agents using contextual inputs for robotic tasks without parameter updates. GPT-Policy provides a shared interface for evaluating how context affects success, efficiency, recovery, and physical execution.
- 6 Conclusion: GPT-Policy studies fixed general agents using demonstrations, goal images, and interaction experience without parameter updates.The framework provides a shared context-to-action interface for examining these inputs.
- 6 Conclusion: Recorded behaviors illustrate goal grounding, operation-order changes, and online coordination, while contact execution and outcome verification remain distinct challenges.Task-level comparisons and reserved ablations support testing context effects across manipulation and mobile exploration.
A Method Details
The method converts planned Cartesian targets into checked, timed robot trajectories and executes them through feedback-aware synchronization. Its safeguards reject invalid plans before submission but do not establish collision-free motion or task success.
- IK Residuals and Backend Checks: IK residuals compare target and forward-kinematics poses, with execution tolerances of 0.002 m and approximately 1°.Numerical stopping tolerances are 10^-4 m and 5 × 10^-4 rad.
- IK Residuals and Backend Checks: The planner supports backend-specific IK refinement, joint bounds, residual acceptance checks, and sequential seeding from measured joint positions.ARX uses SDK IK with damped least-squares refinement, while YAM uses I2RT kinematics and effective SDK bounds.
- Timing, Synchronization, and Feedback: Trajectory timing stretches sampled references when velocity, acceleration, or jerk ratios exceed limits, while preserving joint samples.Derivatives scale by α^-1, α^-2, and α^-3; this checks the sampled reference rather than continuous-time physical jerk.
- Timing, Synchronization, and Feedback: Both arms are planned together and synchronized to the longer trajectory, with final holds and separate measured settling reports.ARX submits timestamped references, whereas YAM streams interpolated joint references at 100 Hz.
- Execution Checks: IK rejection returns feedback before submission, but the planner does not check collisions and accepted references do not establish task success.Physical task success remains distinct from reference acceptance and measured settling.
B Robot Configurations
The implementations differ in robot embodiment, sensing, motion limits, and execution checks. These checks provide provider-specific feedback, but settling and acceptance remain distinct from collision-free motion and physical task success.
- Robot and Sensing Configurations: YAM and ARX X5 use top, left-wrist, and right-wrist RGB views, while Morphi Kino uses head, chest, and wrist views without depth.Morphi Kino additionally has a mobile base, articulated waist, and head.
- Robot and Sensing Configurations: Table 3 compares YAM, ARX X5, and Morphi Kino across robot, sensing, planning, and execution configurations.YAM and ARX X5 use two six-joint arms; Morphi Kino uses two seven-joint arms with a mobile base, waist, and head.
- Motion Configuration: Morphi Kino’s rot6d orientation representation makes its component sampling budget non-angular, and its Ruckig timing does not impose explicit post-IK acceleration or jerk limits.At 10 Hz, the XYZ component budget implies 0.03 m/s per coordinate rather than a 0.03 m/s Euclidean TCP limit.
- Execution Configuration: Collision checking is absent from the Cartesian planner, so runtime diagnostics, reference acceptance, and settling do not establish collision-free motion or physical task success.YAM and ARX X5 report settling timeouts as unsettled results.
- Execution Configuration: Provider adapters use different IK and execution checks, including residuals, joint bounds, continuity, feedback freshness, tracking, and stopping.Morphi Kino returns eligible rejections or incomplete outcomes with fresh observations for replanning.
C Demonstration Data Details
The study compares human videos, teleoperated robot demonstrations, goal images, and robot videos with aligned action references. Demonstration content is selected, timestamp-matched, and retained as contextual reference rather than executable policy output.
- Human Demonstrations: Human demonstrations provide RGB frames of a person performing the task without numerical robot states or action labels.Selected frames show approach, object interaction, and outcome.
- Teleoperated Robot Demonstrations: Teleoperated robot demonstrations record timestamped images, measured joint and end-effector states, motion commands, and gripper commands through release and withdrawal.The plug task adds two wrist views to the overhead view.
- Goal Images: Goal images specify object identities, relative positions, spacing, and desired arrangements without prescribing an action sequence or recorded robot states.They are overhead screenshots or photographs supplied with the instruction and live observations.
- Robot Demonstration Content: Video and Video + Action share selected images, while only Video + Action includes measured states and recorded action segments.Table 5 distinguishes keyframes with state/action intervals from image-only inputs.
- Reference Construction: Automatic keyframe selection preserves chronological visual transitions, contact, release, arm-role changes, and initial and final states within bounded context size.Each keyframe uses nearest measured state within 0.1 s, with additional views matched to the overhead image at the same tolerance.
D Prompt Details
The prompt and tool formats organize demonstrations, observations, feedback, and action requests into a structured control interface. Instructions emphasize chronological evidence, explicit uncertainty, separate gripper decisions, and verification before advancing.
- D.1 Prompt Organization and Shared Instructions: Keyframe prompts preserve initial and final states, contact, grasp verification, release, handoffs, arm-role changes, and repeated twist order while removing redundant holds.Selection combines wrist close-ups with top views and records contact side and object-to-gripper orientation.
- D.1 Prompt Organization and Shared Instructions: Annotations must remain image-grounded, distinguish hand roles, report uncertainty or occlusion, and avoid treating gripper closure alone as proof of grasp.Historical actions are references, not pending commands.
- D.2 Context Formats: The context contains task instructions, current observations, references, and online interaction history, with goal images, videos, action sequences, and measured states included by condition.Inputs interleave labeled image blocks with text records and tool schemas.
- P3 Motion, grasping, and release instructions: Clear contact-free paths may use chunked motion without intermediate observations, but contact, gripper changes, occlusion, and tracking anomalies require observation and verification.check_path evaluates IK, joint limits, and timing without executing commands, yet acceptance does not certify collision clearance or physical tracking.
- P3 Motion, grasping, and release instructions: Before grasping or release, prompts require fresh visual and state evidence, support verification, and post-release observation rather than inferring physical success from commands or settled motion.The planned TCP path is not a measured trajectory, and joint torque is not a direct contact-force measurement.
- P3 Motion, grasping, and release instructions: Motion tools use complete absolute pose_xyzquat waypoints, preserve waypoint order, and change timing or sampling without rewriting the requested geometry.move_to handles a single TCP target, while move_eef_chunk handles ordered target sequences.
- P3 Motion, grasping, and release instructions: The controller separates gripper changes from arm motion and distinguishes requested, submitted, and measured values, with normalized opening from 0 closed to 1 open.After obstruction, the gripper reference is brought near the measured position.