Source-linked AI summary
Generalizing Manipulation Skills with a Local Coding Agent
Raman Talwar, Elias Nijs, Andreas Verleysen, Francis wyffels
TL;DR
Language-driven robots often rely on fixed interfaces or trained policies, while new objects can require programmer-supplied measurements and code. This paper tests a local coding agent on a UR3e and finds generalization across toy-task variations, with faster completion on resumed sessions.
Problem
Many language-driven robots use fixed action interfaces or trained policies, while new objects can require new programmer-supplied code and measurements.
Source: §I · p. 1Method
A locally served Qwen3.8-27B coding agent operates a UR3e, writing code above a service that separates hardware invariants from session-specific work.
Source: §I · p. 1, §III · p. 2Results
The agent completed 30 of 45 trials without intervention and resumed sessions completed tasks faster, with a second-pass time ratio of 0.48.
Source: C. Second Pass Results · p. 6, §VI · p. 7Takeaways & Limitations
The local coding agent shows promise for real-world applications, with further work aimed at addressing failure modes, speed, sensory information, and skill consolidation.
Source: §VI · p. 8The system is not yet fast enough, and the resumed-session gains reflect in-session reuse rather than demonstrated persistent learning.
Source: §VI · p. 8
Abstract
from arXiv · showhide
Today, progress in open-weight language models enables systems capable of writing, executing and debugging code while still running on a single workstation. Most language-driven robots give the model a fixed action interface or a trained policy. Generalizing to a new task therefore means more engineering effort or more data collection, both time-consuming. We investigate whether a local open-weight vision-language model can control a robot and one-shot generalize to new variations of a task without new human programming or training. We let a local open-weight VLM, Qwen3.8-27B, drive a UR3e robotic arm from a coding-agent harness. It writes and runs its own code above a service that implements kinematics, safety limits and classic computer vision techniques. We investigate if this system is capable of generalizing to unseen tasks. Specifically, we test it on nine tasks built from children's toys designed to probe generalization capability across various object characteristics: color, size, shape, and task variation of those. With five trials for each task, we observe generalization in 30 out of 45 trials with durations ranging from 3.4 to 67.5 minutes depending on task complexity. We further test if there is a speedup when an agent is asked to redo the task after successful completion. This resulted in a 50% reduction in duration, indicating that there is self-improvement over time. Finally, we expose the limitations of a local coding agent. We believe that solving those limitations combined with further investigation of self-improvement over time points at a direct path toward real-world deployment of a local coding agent.
I. INTRODUCTION
Fixed interfaces and programmer-supplied measurements constrain language-driven robots’ flexibility on new objects. This work investigates a local coding agent that writes robot code and tests its generalization across toy-task variations.
I. INTRODUCTION
- The work investigates real-hardware generalization and limitations of local coding agents serving as manipulation controllers, including task timings.Prior work had not tested generalization on real hardware with a locally running open-weight model.Source: §I · pp. 1–2
- A local coding agent controls the robot by writing code or making tool calls above a service that provides motion and vision primitives and enforces safety limits.The agent starts from a skill; the framework limits tool speed, gripper force, and workspace bounds.Source: §I · p. 2
- The evaluation protocol uses children’s toys to assess generalization across object and task variations.The tested variations include color, size, shape, and task requirements.Source: §I · pp. 1–2
- Resuming a session on a rearranged scene after successful completion halves tool calls and reduces output tokens to a third.The claim concerns reuse across passes, not a separate training procedure.Source: §I · p. 2
II. RELATED WORK
Prior robot-control approaches range from fixed skill inventories and APIs to code-generating agents and learned policies. This work focuses on generalization across object and task variations with a locally deployed model and fixed low-level interface and skills.
II. RELATED WORK
- Robot controllers have used fixed skill inventories, task-level APIs, generated low-level primitives, per-instance keypoints, and learned policies, each with different interface and data requirements.The related work also covers feasibility-constrained programs and frontier API-served models as direct controllers.Source: §II · p. 2
- Coding-agent research includes simulation studies and real-arm benchmarks, while recent manipulation work also examines agents discovering or refining skills.ALRM compares code-as-policy and tool-as-policy; CaP-X studies interface-abstraction tiers; ASPIRE tests text skills discovered from failures.Source: §II · p. 2
- This work tests generalization across object and task variations while holding the model, low-level interface, and skill documents fixed.The model is locally deployed on one workstation and writes additional perception and control code without additional human programming or model training.Source: §II · p. 2
III. METHOD
The framework separates hardware-owned invariants, configured agent resources, and per-session model work. This division keeps platform correctness independent of the model while allowing session-specific scene measurements and code.
III. METHOD
- The system separates a hardware-owning platform, a configured agent setup, and an isolated per-session workspace where the model reads, writes, and runs code.Kinematics, limits, and pixel-to-coordinate geometry stay in the platform; object-instance knowledge lives in skills, while scene measurements are remeasured during a session.Source: §III · p. 2
A. Platform layer
The platform provides shared robot operations and safety limits, while the agent setup supplies tools, skills, and a fresh workspace for each session. Skills document transferable procedures alongside measurements tied to a particular object instance.
A. Platform layer
- The hardware-owning platform exposes nine operations through a service, including pixel-to-coordinate mapping, color-blob detection, and guarded contact operations.Its generic detector requires color thresholds, which the model measures from the live frame; time-sensitive feedback loops run within the platform.Source: A. Platform layer · p. 2
- Every actuation request passes through safety limits, including workspace clamping and caps on linear speed, joint speed, and gripper force.The service boots into a motion lock.Source: A. Platform layer · pp. 2–3
B. Agent setup
- The model can call the platform operations directly or combine them in scripts, and its prompt directs it to remeasure object-specific values.The prompt distinguishes trusted workspace constants from measurements on the object.Source: B. Agent setup · p. 3
- Hand-crafted skills describe procedures step by step, with supporting measurements, verification steps, and authoring pitfalls; their measured values apply only to the documented instance.The per-session workspace is copied fresh from a template, so state does not cross between tasks.Source: B. Agent setup · p. 3, Table II · p. 3
IV. EXPERIMENTAL DESIGN
The experiments assess whether a local open-weight vision-language model can generalize to unseen object- and task-related variations on physical hardware. They measure task completion, duration, and system limitations across nine toy-based tasks.
IV. EXPERIMENTAL DESIGN
- The study tests whether a local open-weight vision-language model can control a robot and generalize without new human programming or training.Source: §IV · p. 3
- The evaluation covers nine toy-based tasks probing generalization across color, scale, shape, and simple task variations.Source: §IV · p. 4
- The evaluation measures whether unseen task variations can be completed, how long completion takes, and the system’s limitations.Source: §IV · p. 4
A. Games and Tasks
The study uses children’s toys to test transfer from documented skills to related tasks that vary in object properties or task requirements. The three games probe cup, peg-in-hole, and ring-stacking tasks.
A. Games and Tasks
- Children’s toys provide compact families of related tasks, while the study limits generalization tests to object and task variations.Material, lighting, clutter, workspace layout, task vocabulary, and robot morphology are outside the tested scope.Source: A. Games and Tasks · p. 4
- A heavy outline in the object illustration marks an object included in a skill.Source: Figure 3 · p. 4
- Cup tasks vary color, size, and task requirements, from stacking different cups to ordering four cups by size.The skill documents stacking a small blue cup on a larger blue cup; tested variations include other colors and sizes, three-cup stacking, and line ordering.Source: A. Games and Tasks · p. 4
- Sorter tasks test shape generalization by inserting pegs into matching holes, with difficulty increasing as rotational symmetry decreases.The provided skill covers a red cylinder; the other pegs are a cube, prism, and beam.Source: A. Games and Tasks · p. 4
- The Rings task asks the agent to stack five differently sized and colored rings on a rod, transferring from a skill for inserting the largest teal ring.Source: A. Games and Tasks · p. 4
B. Setup
The setup pairs a UR3e arm and wrist camera with a locally served vision-language model. The evaluation records trial outcomes and usage measures while permitting specified interventions for safety or object recovery.
B. Setup
- The robot setup uses a UR3e arm, a parallel gripper, and a wrist-mounted color-depth camera.The camera is eye-in-hand calibrated; the gripper is a Robotiq 2F-85.Source: B. Setup · p. 4
- Qwen3.8-27B runs locally on the workstation that also hosts the platform, using a 262,144-token context window.The model is served with vLLM 0.27.1 and uses low reasoning effort.Source: B. Setup · p. 4
C. Scoring and interventions
- Each task receives five trials, with success or failure, duration, token use, tool calls, and reasoning-token share recorded.Source: C. Scoring and interventions · p. 4
- Table III lists each task’s prompt and time cap, with bullets marking the generalization axes varied from the base task.Source: Table III · p. 5
- Interventions include restoring fallen or sideways objects and allowing protective or emergency stops for safety.A protective stop is triggered by the robot arm; an emergency stop is triggered by the operator.Source: C. Scoring and interventions · p. 5
D. Second Pass
The second-pass procedure resumes selected successful sessions after resetting the scene and asks the agent to repeat the task. The results section describes nine tasks, each run five times, and points to per-task results.
D. Second Pass
- For each task, the second-pass test resumes the fastest and slowest successful trial sessions and asks the agent to repeat the task.Source: D. Second Pass · p. 5
- The scene is reset with objects in new positions, and the same quantities as in the first pass are measured.Source: D. Second Pass · p. 5
V. RESULTS
- The results evaluation tests Qwen3.8-27B on nine tasks derived from three children’s games, with five trials per task.Per-task outcomes are reported in Table IV.Source: §V · p. 5
A. General Behavior
The agent completed 30 of 45 trials without intervention, using a measure–act–verify loop and sometimes recovering from errors. Failures clustered around scene misreadings, persistent faulty assumptions, and limited awareness of the robot’s body and nearby space.
A. General Behavior
- 30 of 45 trials ended without intervention; across the grid, the agent used 15.8 robot-hours, 3.18 million output tokens, 2,043 tool calls, and 1,034 motion commands.Six trials required a small intervention, one an emergency stop, and seven encountered protective stops.Source: A. General Behavior · p. 5
- The agent read the skill and manual, measured the scene with self-authored scripts, executed pick-and-place code, then verified from a predefined pose and remeasured after errors.Reasoning accounted for 71% to 85% of task-level median output tokens and its share increased with task complexity, increasing task duration.Source: A. General Behavior · p. 5
- Skill procedures and remeasurement helped the agent adapt instance-specific values, while verification exposed failed grasps and prompted retries.A stall width that failed to hold after lifting was one signal of a failed grasp.Source: A. General Behavior · p. 5
B. Failure Modes
- Three failure modes were observed: perception errors, anchoring on faulty assumptions, and unawareness of robot morphology and nearby space.Source: B. Failure Modes · p. 6
- Perception errors included confusing hole shapes or prism orientation and misreading ring size or color, sometimes leading to incorrect placements or stops.The authors attribute this category to classical computer vision limitations and suggest advanced deep-learning methods as a possible remedy.Source: B. Failure Modes · p. 6
- Faulty assumptions persisted in four failed runs, including repeated measurements that never challenged an incorrect label or pose.The agent recovered from a faulty assumption in 10 of 30 successful trials; the authors consider this failure category mostly intrinsic to their model.Source: B. Failure Modes · p. 6
- Limited body awareness led to tool or gripper collisions: platform checks covered only the tool center point, not the robot’s full morphology.The authors propose simulation including the robot’s morphology, possibly the scene, or a scene camera as possible solutions; Sorter: Prism was the worst-performing task, with all three failure categories involved.Source: B. Failure Modes · p. 6
C. Second Pass Results
Resuming an already-successful trial generally made the second pass faster and cheaper, while retaining some failure modes and reflecting in-session reuse rather than persistent learning.
C. Second Pass Results
- Across 17 resumed trials, duration fell to a 0.48 time ratio, tool calls to 354 against 734 (ratio 0.48), and output tokens to a 0.34 ratio.The figure compares first-pass durations with second passes resuming the fastest or slowest successful trial.Source: C. Second Pass Results · p. 6, Figure 4 · p. 7
- The agent retained first-pass findings and focused remaining effort on relocating objects; the faster first-pass resume was also faster in six of eight tasks with two resumes.The retained findings included color windows, object heights, matching-hole identity, and a successful wrist angle.Source: C. Second Pass Results · pp. 6–7
- Two resumed trials took longer after failure modes recurred, including a protective stop and five failed attempts to grasp the first ring; 16 of 17 passes completed without intervention.The two slower resumes were exceptions to the overall speedup.Source: C. Second Pass Results · p. 7
- The speedup reflects in-session reuse, not demonstrated persistent learning, because resumed sessions retain history, measurements, and scripts whose separate contributions were not isolated.The authors suggest investigating whether experience can be consolidated into skills for fresh sessions.Source: C. Second Pass Results · p. 7
VI. CONCLUSION
The local coding agent generalized across object and task variations on physical hardware, but its failures and runtime constrain deployment; the authors identify sensory improvements and skill consolidation as next steps.
VI. CONCLUSION
- The system completed 30 of 45 trials without intervention, including 20 of 25 cup trials across color and size, every unseen peg at least once, and three of five ring-tower rebuilds.The evaluation varied objects and task requirements within games while holding the interface, model, and skill text fixed.Source: §VI · p. 7
- A locally run 27B vision-language model adapted procedures for new object and task variations on one workstation, writing its own perception code using classical computer vision.The model, low-level interface, and skill documents were held fixed, with no additional human programming or model training.Source: §VI · p. 7
- Observed failures included wrist-camera scene misreads, repeated measurement instead of revisiting assumptions, and inadequate awareness of tool morphology; eight of eleven failures ended in protective stops.The platform guarded the tool center point, so rotated tools or wrongly opened grippers could still lead to a stop.Source: §VI · p. 7
- Cup tasks took four to fourteen minutes, sorter tasks half an hour to an hour, and 80% of model output tokens were reasoning; the authors recommend pre-deployment task learning.Task time was bimodal: correct first placement led to runs measured in minutes, while errors could lead to prolonged re-measurement.Source: §VI · p. 8
- The authors see promise for real-world applications but identify resolving failure modes and improving speed as prerequisites, with added sensory information and skill consolidation as next steps.Prior robot coding-agent work had not tested generalization on real hardware with a locally running open-weight model.Source: §VI · p. 8