PaperlayerView exact revision on arXiv

Source-linked AI summary

HexaShell: A Thin Harness for Exploring the Limits of VLM Agents as Robot Coding Agents

Hongcheng Gao, Alex Zhou, Jay Zhu, Zelin Zheng, Shijia Ge, Xiping Li, Di Wu, Yazhe Wang, Jianshu Zeng, Xuan Shangguan, Yongli Wang, Yexing Wan, Lingyu He, Zhiqi Jia, Sihang Wu, Xiao He

arXiv:2610.23479v1cs.RO

TL;DR

Robot coding agents raise the question of how far a VLM-based coding agent can go when controlling manipulation tasks. HexaShell gives the agent a thin robot interface and persistent tools repaired after failures, and reaches a new state of the art on RoboDojo while remaining weak on tight insertion.

  • Problem

    Coding agents that can revise software raise the question of how far they can go when controlling robots, where pixel observations must support accurate world positions for manipulation.

    Source: §1 · p. 2

  • Method

    HexaShell gives a coding agent a thin robot command interface and lets an optimizer use episode logs to write or fix persistent tools that measure positions from calibrated RGB observations.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    39.7% success and a progress score of 48.0 across all 42 RoboDojo tasks establish a new state of the art, with pushing and memory tasks stronger than tight insertion.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The results place the current boundary at contact: the agent often recognizes when a motion is failing but cannot make the motion work.

    Source: §7 · p. 15

    Tight insertion and pouring remain weak without depth because the hole and part must be measured together at millimetre-scale precision, while wrist views are difficult during a grasp.

    Source: §5.1 · p. 9

Abstract

from arXiv · show

Agents built on vision language models (VLMs) now compete with learned policies on robot manipulation leaderboards. We ask how far a coding agent can go when it controls a robot instead of a computer, and where it stops. Our system, HexaShell, keeps the harness thin: it only turns the agent's commands into robot motion. The agent drives a robot with two arms through one command line. It sees three RGB cameras and knows their calibration. After each failed attempt, the agent reads the log, finds the cause, and writes or fixes small programs called tools. In this way it keeps what it learns as code, as a coding agent does when it fixes a failing test. The tools may use only what the robot observes. They use triangulation to find where objects are, relative to each other and to the gripper. If two cameras see the same point, the point lies where the two camera rays meet. If a point lies on a known plane, such as the table or the top of a block, it lies where its ray meets that plane. When the head camera is not enough, the robot moves a wrist camera to get a second view. We build the tools on only ten layouts of one seed per task, then freeze them and test them on ten new layouts of the same seed. We use Codex with GPT-6-Astra at medium reasoning effort. On all 42 RoboDojo tasks, HexaShell reaches a success rate of 39.7% and a progress score of 48.0, averaged in the same way as the leaderboard. This is a new state of the art on RoboDojo. The limit is contact, not reasoning: pushing reaches 92.5% and memory tasks 65.0%, but tight insertion only 8.3%. When the agent fails, it usually sees the failure but cannot make the motion work. We open source the tools evolved so far at https://github.com/HexaFuture/HexaShell.

1 Introduction

HexaShell tests whether a coding agent can control a robot through a thin harness, retain repairs as tools, and generalize them to new layouts. Its results show strong performance on geometry-led tasks but a persistent limit at contact.

1 Introduction

  • HexaShell gives a coding agent one robot command line and retains learning as small tools that an optimizer diagnoses, writes, or fixes after failures.A repair round takes a median of 6.3 minutes and requires no robot data, training, or model change.Source: §1 · pp. 2–3
  • Calibrated-camera triangulation and known-plane intersections address pixel-to-position errors that monocular depth estimates leave substantial on RoboDojo scenes.Depth Anything V2 has median errors of 41 mm at the head camera and about 100 mm at the wrists, even after table-scale correction.Source: §1 · p. 2
  • 39.7% success and a progress score of 48.0 across all 42 RoboDojo tasks establish a new state of the art on the benchmark.Tools were built on ten layouts per task, frozen, and tested on ten new layouts.Source: §1 · p. 2
  • With the same harness, loop, model, and layouts, triangulation scores 8.7 points above Depth Anything V2, winning 57 paired layouts and losing 23.Second views cost at least 3.6% of executed steps; on layouts where the robot moved a camera, triangulation succeeds 39 times against 22.Source: §1 · p. 3
  • Pushing reaches 92.5% and memory tasks 65.0%, while tight insertion reaches only 8.3%; failures distinguish perception gaps from running out of steps.Across 420 retest episodes, performance follows the contact a task needs more than what the agent must read or remember.Source: §1 · p. 3
  • Seven case studies trace failed attempts through the optimizer’s diagnosis and fix to another attempt on the same layout.Source: §1 · p. 3

2 Related Work

Prior robot-language systems often generate per-instruction programs or rely on human-written primitives. HexaShell instead keeps task tools that an agent writes and repairs, and evaluates how far they reach on a full benchmark.

2 Related Work

  • HexaShell reverses the common division of labor: the agent writes persistent primitives, while its per-episode program stays short.Earlier approaches typically produce a program per instruction and discard it, with people writing the primitives.Source: §2 · p. 3
  • Unlike fixed-harness VLM policies, HexaShell keeps the model and benchmark but lets the agent write, retain, and repair tools.Source: §2 · p. 3
  • Among self-improving tool-building agents, HexaShell holds the scaffold, prompt, and base interface fixed, changing only task tools.Acceptance requires a fresh execution of the same layout rather than optimizer self-assessment, and delivered tools must run under the official client.Source: §2 · p. 3
  • HexaShell combines calibrated multi-view geometry, plane-sweep ideas, and active perception, charging extra camera views to the benchmark’s action budget.Source: §2 · p. 3
  • RoboDojo’s action budget, multiple seeds, and paired clean and cluttered layouts support separating geometric competence from appearance memorization.Source: §2 · p. 4

3 HexaShell

HexaShell uses a thin command harness, RGB-based geometric measurement, task-specific tools, and an evolution loop that turns failures into tool revisions. The interface restricts tools to observations and robot motion, while frozen tools are tested on new layouts.

3 HexaShell

  • The harness translates commands into motion without making decisions, isolates ground truth from the agent, and leaves strategy in tools.The benchmark’s own checker judges success, and the manual states interfaces only.Source: §3 · p. 4

3.1 The harness

  • The stock coding-agent CLI receives one executable and refreshed observations containing three calibrated RGB views, TCP poses, and joint angles, but no depth image.Base commands include motion, gripper, observation, status, and completion operations; tools add sub-commands to the interface.Source: §3.1 · p. 4

3.2 Measuring 3D from RGB images

  • RGB measurements combine table or other known-plane constraints with second-view ray intersections to recover 3D positions.The table plane is estimated by image warping; corresponding pixels come from the acting agent or tool-side matching and fitting.Source: §3.2 · p. 5

3.3 Tools

  • Tools expose task-specific sub-commands through an EpisodeAPI, but cannot access simulator state or object poses.The API supplies joint angles, TCP poses, calibrated RGB images, and the base motion primitives; camera-moving measurements consume steps.Source: §3.3 · p. 5, §3.2 · p. 5

3.4 The evolution loop

  • The optimizer diagnoses failures from logs and frames, revises tools, and retries the same layout; after ten layouts, tools are frozen for testing.Edits face a delivery check, may occur up to five times per layout, and final tests include ten new layouts.Source: §3.4 · p. 5

4 Experiments

Experiments evaluate frozen tools on new RoboDojo layouts and compare HexaShell with leaderboard systems, a depth-network variant, and cluttered-scene tests. Results show strong performance on some task types, while contact-heavy tasks and the cost of second views expose limits.

4.1 Setup; 4.6 What the agents built

  • The 42 evaluated tasks are grouped into five task types, and each task has three evaluation seeds; frozen RGB retests use ten new layouts while depth comparisons use development layouts.The system uses unmodified Codex with GPT-6-Astra at medium reasoning effort; its tools measure 3D from RGB images, with the model choosing pixels and code measuring world positions.Source: §4.1 · p. 6, §4.6 · p. 8

4.2 Main results

  • 39.7% success and a progress score of 48.0 put HexaShell above VPP2 (32.3%, 39.3) and PhysicalRSI (31.4%, 36.3) on the leaderboard metric.Memory reaches 65.0% and Open 50.0%, while VPP2 leads on Precision (35.2% versus 31.3%) and Long-Horizon (37.0% versus 23.8%).Source: §4.2 · p. 6

4.3 Clean versus cluttered layouts

  • On development layouts, HexaShell passes 35.0% of standard and 33.3% of cluttered Generalization layouts, retaining 95% of its success.The same model without tools retains 87%; fold clothes is an exception, falling from 4/5 to 2/5 because nearby objects disrupt colour-connectivity detection.Source: §4.3 · p. 6

4.4 Triangulation versus a depth network

  • Triangulation scores 8.7 points higher than Depth Anything V2 on the leaderboard metric, winning 57 layouts and losing 23 in paired comparisons.Depth-network median error is 41 mm at the head camera and about 100 mm at the wrists; rendered depth scores 10.6 points higher than triangulation, with most of the gap in tight insertion.Source: §4.4 · p. 7, Table 1 · p. 7

4.5 The cost of a second view

  • With triangulation, viewing uses 3.6% of executed steps versus 0.7% with the depth network, while the RGB agent succeeds on 39 of 180 viewing-move layouts versus 22 for depth.Viewing counts are lower bounds because viewing inside larger tools is omitted; the cost is concentrated in some tasks, including egg-carton measurement and coin-slot measurement, where none succeeds.Source: §4.5 · pp. 7–8, Table 2 · p. 7

5 Where the boundary lies

Across the retests, physical interaction was more decisive than decision load, with contact-intensive tasks especially difficult. Failures also distinguish execution problems from perception or task-specification gaps, and additional checker information alone did not resolve every capability limit.

5.1 A capability map

  • Pushing reaches 92.5% and rigid manipulation 50.0%, while articulated objects reach 7.5% and tight insertion 8.3%; memory tasks reach 65.0%.Seven of 12 tight-insertion variants scored zero.Source: §5.1 · pp. 8–9
  • VPP2 is weaker on pushing (24.7%) and rigid manipulation (30.0%), but stronger on insertion (28.1%), deformable material (49.4%) and articulated objects (21.6%).Source: §5.1 · p. 9
  • Tools add 33 points on pushing, 13 on rigid manipulation and 26 on memory, but fold clothes reaches 50% against 72% without tools.The tools commit to one interpretation and planned motion, while step-by-step action can still notice and correct.Source: §5.1 · p. 9
  • With rendered depth, insertion reaches 30.1% versus 8.3% for RGB tools, while pouring tasks drop from 59–95% to 0–10%.Insertion requires measuring the hole and part to a few millimetres; wrist views that resolve the hole must be made while holding the part.Source: §5.1 · p. 9

5.2 How episodes fail

  • Of 268 failed retests, 57% exhausted the budget, 30% were declared complete but rejected by the checker, and 13% ended with the agent giving up early.In deformable, articulated and rigid tasks, 60–77% of failures exhaust the budget; half of memory-task failures are declared successes, while moving-object failures often follow the object passing.Source: §5.2 · p. 9

5.3 Missing information or missing capability?

  • Checker notes lifted three tasks from 0–40% to 95–100% in an earlier depth run, but pour liquid into cup fell to one of ten layouts in RGB.Knowing the checker’s requirements was not enough: the tool also needed to measure the cup, which remained weak without depth.Source: §5.3 · p. 9

6 Case studies: from a failed attempt to a working tool

The case studies trace failed attempts to local tool edits and retests, showing repairs to measurement, timing, routing and grasp placement. These fixes improve individual attempts, but the cases also expose the boundary: the loop cannot by itself solve the contact demands of tight insertion.

6.1 What an optimizer round looks like

  • Across 27 tasks, 67 layouts failed and then passed after an edit; the seven selected cases were checked against logs and code, and five required no optimizer image inspection.The loop follows a failed attempt with an optimizer round and another attempt on the same layout; Table 4 summarizes seven turns and their frozen retests.Source: §6 · p. 9, §6.1 · p. 10, Table 4 · p. 10

6.2 Push T: a height check that compared unequal pixels

  • In Push T, a 15 mm height error shifted the projected outline by 10 mm; correcting the height check let layout 2 pass again and all ten development layouts succeed.The original area check rejected measurements across cameras with different resolutions; the fix compared each pixel’s footprint on the plane.Source: §6.2 · pp. 10–11

6.3 Swap T: a colour mask that mixed two faces

  • After surface_pair changed to match colours by hue and retain each view’s bright top face, frozen-tool layout 0 passed again in 275 steps with a 126.3-degree turn; all ten development layouts passed.Source: §6.3 · pp. 11–12

6.4 Play Xylophone: a second view of a held tool

  • For Play Xylophone, triangulating the held mallet from a second camera put its centre 2.7 mm from truth; the task then passed six of ten layouts, while the depth-network version passed none.The repaired tool fails rather than guessing when it cannot find a clear match.Source: §6.4 · p. 12

6.5–6.6 Timing and routing repairs

  • The conveyor tool lengthened infeasible pick schedules, enabling seven of ten layouts; in tic tac toe, combining gripper turns with travel saved steps and enabled nine of ten layouts.The conveyor retest passed in 460 steps; tic tac toe filled its board at step 1028 of 1100 and retested at step 1035.Source: §6.5 · p. 12, §6.6 · p. 13

6.7 Fold clothes: a grasp point on the edge

  • For cluttered Fold Clothes, requiring 12 mm of cloth around grasp points moved contacts inward; the frozen layout still failed, while the task passed six of ten layouts.The repaired points were 12.3 and 12.1 mm inside the edge, compared with 1.6 and 5.4 mm for the old code on the same frame.Source: §6.7 · pp. 13–14

6.8 Insert tubes: an early stop in the hardest class

  • In Insert Tubes, rechecking the original reachable path let the first tube reach its hole 1.5 mm from centre, but the task passed only three of ten layouts.The frozen retest layout passed without needing the recheck; the other seven ended with the agent quitting or reaching the step limit.Source: §6.8 · pp. 14–15

6.9 What the cases show

  • The cases show local edits repairing measurement, timing, grasp placement or tool stopping, but tight-insertion contact remained unresolved even when a tube entered by a drop.Source: §6.9 · p. 15

7 Conclusion

HexaShell combines a thin robot-control harness with tools repaired after failures, and reaches 39.7% success and a progress score of 48.0 across 42 RoboDojo tasks. Its performance is strongest on pushing and memory tasks, while tight insertion and contact remain difficult.

7 Conclusion

  • 39.7% success and a progress score of 48.0 on 42 RoboDojo tasks, using tools frozen after development and tested on new layouts.The conclusion identifies contact as the main limit: tight insertion remains low, and failures often occur despite the agent recognizing that a motion is not working.Source: §7 · p. 15
  • HexaShell gives an unchanged coding agent one robot command line and preserves what it learns as small tools repaired after failures.A repair round takes a few minutes.Source: §7 · p. 15
  • Pushing and memory tasks succeed most often, while tight insertion remains low and contact is the stated limit.The proposed next step is to measure continuously while the gripper holds a part and correct motion during contact rather than planning it in advance.Source: §7 · p. 15

A Per-task results

The per-task results are presented across task variants, with a figure comparing HexaShell against two leaderboard systems. HexaShell’s variants are evaluated on frozen RGB retests, while the comparison systems use leaderboard rollouts.

A Per-task results

  • Figure 11 presents success rates for every task variant, comparing HexaShell with VPP2 and PhysicalRSI.Variants are sorted by HexaShell’s success rate.Source: Figure 11 · p. 20, A Per-task results · p. 20
  • HexaShell’s Figure 11 results use RGB images and frozen retests on ten new layouts, numbered 10 to 19 of seed 0.Source: Figure 11 · p. 20
  • VPP2 and PhysicalRSI results in Figure 11 come from leaderboard rollouts with 50 episodes per seed.Source: Figure 11 · p. 20

B Tools of the RGB runs

The RGB-run tools are documented task by task, including their commands, implementation, optimization effort, and viewing-move costs.

B Tools of the RGB runs

C Example development log: push T

The push T development log traces how RGB tools replaced a depth-based registration and height measurement, after which all layouts passed on the first attempt. Figure 12 gives aggregate development-layout progress-score comparisons for triangulation, a depth network, and a model without tools.

C Example development log: push T

  • After RGB silhouettes and calibrated rays were used to rewrite registration, height measurement, arm handover, and release checks, every push T layout passed at its first attempt.Rounds 5 to 12 added playbook notes, and round 13 consolidated them.Source: C Example development log: push T · p. 20
  • The RGB push T run took 13 optimizer rounds and four hours over ten development layouts, then passed all ten frozen retest layouts.The earlier depth-network version passed 4 of 10 retest layouts.Source: C Example development log: push T · p. 20
  • The RGB rewrite corrected a two-view area check that rejected matching views because the cameras had different pixel scales.Source: C Example development log: push T · p. 20
  • 40.8 mean progress with triangulation exceeded 33.0 for the depth network and 29.8 without tools across 54 development variants.Source: Figure 12 · p. 21

D The earlier depth runs and the audit

The earlier depth-based runs tested rendered depth and a monocular depth network before the RGB runs began from the latter tools. An audit then replaced fixed object geometry with episode measurements and retested affected tasks.

D The earlier depth runs and the audit

  • The fully automatic rendered-depth run reached 38.0%; adding checker notes and manually rebuilding the build-tower tool brought the current tools to 48.8%.The notes described checker requirements for tasks where the first run lagged most.Source: D The earlier depth runs and the audit · p. 21
  • The Depth Anything V2 Base line reached 29.6% after developing 22 tasks and retaining new tools for ten; the RGB runs started from its tools.Tasks were developed for up to 24 hours each.Source: D The earlier depth runs and the audit · p. 21
  • By the RGB runs, 25 tasks had optimizer notes written from public task definitions, while the acting agent never saw them; nine tasks also began with a colleague's RGB tools.Source: D The earlier depth runs and the audit · p. 21
  • Seven tasks had fixed object geometry in their tools; after replacing constants with episode measurements, all seven scored 0/10 on retest, versus three prior passes across 70 layouts.The audit compared tool literals with task definitions and logs; the affected tools were retested without further development.Source: D The earlier depth runs and the audit · p. 21

E Interface given to the acting agent

The acting agent's shared manual defines the robot frame, commands, feedback, and image-based observations. The optimizer diagnoses failures and revises tools under constraints that restrict their inputs and prohibit evaluation runs.

E Interface given to the acting agent

  • The manual defines a shared right-handed world frame, the gripper TCP as the fingertip midpoint, and the gripper's opening value and reach envelope.The frame uses metres, degrees, and seconds; the table-plane tool measures the real table height.Source: E Interface given to the acting agent · p. 21
  • Base commands move, rotate, point, grip, home, and wait have execution limits; episodes cap commands and time, while feedback reports motion and planning outcomes.Motion planning knows the arm and table, not object locations.Source: E Interface given to the acting agent · p. 22
  • The agent receives three calibrated RGB views and robot state but no depth; it must infer object positions from images and calibration, using a measured plane or second view.Source: E Interface given to the acting agent · p. 22

F Optimizer role description

  • In failure mode, the optimizer inspects episode logs, trajectories, and head-camera frames, then chooses a diagnosis and may implement and enable a tool repair.Success mode adds at most 15 playbook lines; final mode rewrites the playbook, skill, and tool interfaces.Source: F Optimizer role description · p. 22
  • Tools may use only the EpisodeAPI and RGB geometry, must avoid simulator state and layout-specific values, validate arguments, and never raise; the optimizer runs no evaluations.The shared tool pool is read-only, and the optimizer records a short proposal at each round's end.Source: F Optimizer role description · p. 22

G Layout structure of RoboDojo seeds

RoboDojo seeds vary object poses and scene materials, with additional randomized distractors in cluttered layouts. The result tables distinguish frozen RGB retests from earlier depth and rendered-depth runs and describe per-task tool and evaluation details.

G Layout structure of RoboDojo seeds

  • Each seed contains 30–75 sampled layouts per task; standard layouts vary object positions and yaw plus table and ground materials, while random layouts add 13–18 distractors from 85 categories.Some tasks instead vary content, such as digits or object categories; development uses the first ten layouts of seed 0.Source: G Layout structure of RoboDojo seeds · p. 22
  • Table 5 covers all 42 tasks and distinguishes frozen RGB retests on new seed-0 layouts from depth-network development-layout retests and earlier rendered-depth results.It also identifies task interaction and decision types and specifies the leaderboard comparison scope.Source: Table 5 · p. 23
  • Table 6 reports each task's commands, tool-code size, optimizer rounds, viewing-step share, and development-layout retest; Table 7 covers rendered-depth runs with checker notes.Table 7 reports development-layout retests and success across seeds 0–2, with a parenthetical exception for fold clothes.Source: Table 6 · p. 24, Table 7 · p. 24

Related papers

Loading 2610.23479v1…