PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboArchitect: Automating the Design of Robotic Agents with Persistent Memory

Shengguang Wu, Suning Huang, Cherie Ho, Ria Doshi, Hao Zhu, Serena Yeung-Levy

arXiv:2610.42141v1cs.ROcs.AIcs.CV

TL;DR

Robot harnesses that mediate frozen policies are commonly designed by hand, despite the effort required to diagnose failures and revise control code. ROBOARCHITECT uses a coding model to iteratively refine the harness from rollouts while keeping policy weights fixed. Its evolved harnesses improve performance on held-out scenes and tasks across benchmarks and policies.

  • Problem

    Robot harnesses are still manually designed through a slow process of inspecting rollouts, diagnosing failures, and revising control code for each task and embodiment.

    Source: §1 · p. 1

  • Method

    A coding model iteratively diagnoses rollout failures and refines a harness around a frozen robot policy, with evaluation on held-out episodes separate from optimization.

    Source: §1 · p. 1

  • Results

    Across diverse benchmarks and frozen policies, evolved harnesses substantially improve performance on held-out scenes and tasks, and can outperform finetuned versions.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The findings suggest substantial robotic-manipulation performance headroom lies in the harness, whose design can be automated for novel setups without updating policy weights.

    Source: §1 · p. 2, §5 · p. 10

    The work evaluates a maximum of five optimization iterations; gains from further iterations and the point of saturation remain unexplored.

    Source: §6 · p. 10

Abstract

from arXiv · show

Recent work shows that a frozen robot policy performs far better inside an agent harness, where control code decides when to query the policy, consult vision tools, or command the arm directly. Yet today these harnesses are built by hand and redesigned for each new robot or benchmark. We show that harness design can be automated, offering a strong alternative to finetuning on a novel setup. In RoboArchitect, a coding model observes a robot's rollouts, diagnoses failures, and iteratively revises the harness around the frozen policy. The harness evolves on a set of episodes for its optimization and is evaluated on held-out scenes and tasks. Without updating any model weights, RoboArchitect raises average success from 9.7% to 43.3% on unseen scenes (4.5x) and from 12.7% to 40.2% on unseen tasks (3.2x), across multiple benchmarks and policies. The gains are largest where the benchmark is far from the policy's training data: wrapping a general-purpose policy, the evolved harness yields a specialist that often outperforms even the finetuned version. Intuitive control strategies are automatically designed, such as lifting a jammed arm clear or reaching in horizontally when an obstacle blocks the way from above. The harness also evolves its own persistent memory, carrying learned experience, such as effective grasp depths, into later decisions. These results suggest that substantial headroom in robotic manipulation lies in the agent harness, and its design can be effectively automated and optimized without weight updates.

1 INTRODUCTION

Robot harnesses can improve frozen policies, but they are still manually designed through a slow failure-diagnosis loop. ROBOARCHITECT automates harness refinement and adds persistent memory, improving held-out performance and adapting policies to specific setups without weight updates.

1 INTRODUCTION

  • Harness optimization is presented as an alternative to finetuning for adapting robot policies to new environments without updating weights.The introduction frames this as evidence that substantial manipulation-performance headroom lies in the agent harness.Source: §1 · pp. 1–2
  • ROBOARCHITECT automates the manual loop of inspecting rollout failures, diagnosing causes, and revising harness code for each task and embodiment.A coding model uses step-by-step rollout recordings to propose harness updates around a frozen policy.Source: §1 · p. 1
  • 9.7% to 43.3% on unseen scenes and 12.7% to 40.2% on unseen tasks: evolved harnesses raise held-out success, often beyond the same policy finetuned on the benchmark.The held-out episodes are not used for harness optimization; unseen scenes vary arrangements for shared tasks, while unseen tasks test transfer to distinct tasks.Source: Figure 1 · p. 2
  • Persistent memory lets the harness carry rollout-derived pragmatic clues, such as effective grasp depths, into later control decisions.The harness chooses what to store, how to represent it, and how to retrieve it without a specified schema or memory rules.Source: §1 · p. 2
  • A single harness optimized across diverse benchmarks can develop shared control strategies and memory structures for different robot platforms.The work uses harness evolution to build a unified agent across embodiments.Source: §1 · p. 2

2 RELATED WORK

Prior work establishes that agent interfaces around robot policies matter, while related approaches either specify these interfaces manually, optimize policy weights, or replace the policy with code. ROBOARCHITECT instead automates the harness around a frozen policy.

Vision-language-action policies

  • Out-of-distribution VLA performance can depend heavily on finetuning, which requires demonstrations for each new task or hardware configuration; finetuned skills may not generalize beyond their initial distributions.The cited works report these limitations as motivating context (Zang et al., 2025; Park et al., 2026; Zhou et al., 2025).Source: §2 · p. 2
  • Other frozen-policy approaches retrieve demonstrations or action segments or select candidate actions with a learned value function, whereas ROBOARCHITECT evolves complete agent interfaces fitted to each setup.Its interfaces can combine persistent memory, programmatic control, and policy orchestration.Source: §2 · p. 3

Robotic agents

Automated code optimization

  • ROBOARCHITECT optimizes an agent around a frozen policy, unlike robotics optimization loops that generate artifacts for finetuning or skill programs that replace the policy.It can evolve how control is divided between the policy and direct code commands.Source: §2 · p. 3

3 ROBOARCHITECT

ROBOARCHITECT iteratively evolves a program between observations and robot commands around a frozen policy. A coding model diagnoses rollout failures and refines the harness, which can use auxiliary tools, issue direct commands, and retain experience across episodes.

3 ROBOARCHITECT

  • The harness starts as a pass-through of the frozen policy and evolves to follow it, consult tools, command the arm directly, or use persistent memory as needed.The optimization loop evaluates the evolving harness on training episodes and reserves separate held-out episodes for final evaluation.Source: §3 · p. 3

Harness interface

  • At each control step, the harness receives environment observations and outputs low-level action vectors for robot execution.Inputs include camera images, depth maps, gripper pose and status, plus the task instruction; the initial harness forwards observations to the policy and returns its predicted actions.Source: §3 · pp. 3–4

Auxiliary tools

  • The harness can query pretrained language, vision-language, detection, segmentation, pointing, and grasp-pose tools, as well as geometric utilities for 3D projection and trajectory primitives.These provide alternatives to relying on the frozen policy alone.Source: §3 · p. 4

Optimization loop

  • A coding model reviews complete execution traces, diagnoses failures, and proposes revisions that are accepted only when training-set performance improves.The traces include camera frames, robot readings, and model and tool outputs; optimization is capped at a maximum number of iterations.Source: §3 · p. 4

Persistent memory

  • Persistent memory carries information across episodes and tasks, with its structure autonomously evolved from accumulated rollout evidence and fitted to each setup.The paper contrasts this with fixed hand-crafted memory schemes.Source: §3 · p. 4

4 EXPERIMENTS

Across multiple robot benchmarks and policies, ROBOARCHITECT optimizes harnesses around frozen policies and evaluates them on held-out scenes and tasks. The experiments show substantial transfer, targeted control routines, and persistent memories that carry experience across episodes.

4.1 SETUP

  • The setup spans LIBERO-Pro-10, RoboCerebra, VLABench, RoboCasa-365, and SimplerEnv, testing held-out scenes, held-out tasks, and a unified harness across robots.Held-out scenes share tasks but vary arrangements; held-out tasks differ while sharing manipulation skills.Source: §4.1 · pp. 4–5

4.2 HARNESS OPTIMIZATION UNLOCKS SUBSTANTIAL ROBOT MANIPULATION HEADROOM

  • +345% on unseen scenes and +216% on unseen tasks: the iteratively evolved harness raises mean held-out success over the bare policy.Table 1 reports these gains across the evaluated setups.Source: Table 1 · p. 6
  • Harness gains transfer positively to unseen tasks and generally to unseen scenes; the main exception is RLDX-1 on RoboCasa-365, which improved only on optimization episodes.On VLABench under both protocols and SimplerEnv WidowX under unseen scenes, the harness around the general policy outperforms finetuned weights; it falls short on WidowX unseen tasks.Source: §4.2 · pp. 5–6
  • Iterative refinement uses each harness’s execution evidence to repair newly exposed failures, improving held-out performance over one-shot designs in nearly every set.The exception is RLDX-1 on RoboCasa-365 under unseen scenes.Source: §4.2 · p. 8

4.3 EVOLVED CONTROL ROUTINES DRIVING THE PERFORMANCE GAINS

  • Evolved routines address failures with jam recovery, missing motion primitives, force rescaling, target redirection, and robot-speed-aware movement.On the slower Google robot, ending moves when the arm arrives contributed to raising training-pool success from 11.0% to 30.2%.Source: §4.3 · pp. 8–9

4.4 PERSISTENT MEMORY EMERGES FROM HARNESS EVOLUTION

  • Harnesses develop persistent memories for execution experience, task and environment clues, and embodiment calibration, without a predefined memory schema.Held-out episodes consult memory several times on average, and over 90% of reads find a stored entry; memories start empty at evaluation.Source: §4.4 · p. 9
  • Memory entries encode grasp heights relative to measured object height, successful grip styles, task clues, reset timing, and robot-specific finger orientation.Measured properties or object identity let grasp experience carry over to novel objects and new tasks.Source: §4.4 · p. 9

5 CONCLUSION

ROBOARCHITECT automates harness design and optimization around frozen robot policies, with the harness adapting policies to held-out scenes and tasks. The results suggest substantial headroom in robot manipulation lies in the harness and can be unlocked by evolving it for novel setups.

5 CONCLUSION

  • Across diverse benchmarks and frozen policies, evolved harnesses improve performance on held-out scenes and tasks, sometimes outperforming finetuned versions in out-of-domain settings.The harness can continue improving from prior versions’ behavior, while producing control routines, persistent memories, and a unified agent that adapts across embodiments.Source: §5 · p. 9
  • ROBOARCHITECT automates the design and optimization of agent harnesses around frozen robot policies.Its harness evolves without updating policy weights.Source: §5 · p. 9
  • The results suggest substantial headroom in robot manipulation resides in the agent harness and can be unlocked by automatically evolving it for novel setups.Source: §5 · p. 10

6 LIMITATIONS

The evaluation uses a fixed five-iteration optimization cap, while the point at which returns saturate remains unknown. Extending the work to richer-contact tasks and testing simulation-evolved harnesses on real robots are future directions.

Scaling cap of iterations

  • The optimization uses a fixed maximum of 5 iterations, and whether returns saturate with further iterations remains unexplored.Even with this moderate budget, the harness achieves gains that transfer to held-out settings.Source: §6 · p. 10

More complex manipulation

  • Extending ROBOARCHITECT to richer-contact tasks, including dexterous in-hand manipulation and tool use, remains future work.These tasks have a larger space of failure modes than pick-and-place, and the routines and memories evolution could discover there remain an open question.Source: §6 · p. 10

Sim-to-real transfer

  • All evaluations run in simulation; testing simulation-evolved harnesses on real robots and investigating further improvement iterations after deployment remain future work.The authors expect the interface to carry over because it reads standard inputs such as camera images, depth, and robot readings.Source: §6 · p. 10

A EVOLVED CONTROL ROUTINES AND PERSISTENT MEMORIES IN DETAIL

The appendix details auto-evolved control routines and persistent memories, with Figures 5–7 showing additional revisions.

A EVOLVED CONTROL ROUTINES AND PERSISTENT MEMORIES IN DETAIL

A.1 CONTROL ROUTINES

The evolved harnesses revise robot control in response to specific policy, motion, force, perception, and embodiment failures. Their routines range from recovering from jams to replacing an unusable policy with direct vision-based control.

A.1 CONTROL ROUTINES

  • When commanded and actual gripper movements persistently disagree during a jam, the harness lifts the arm clear before querying the policy again.This addresses the bare policy’s tendency to repeat similar commands when visual observations barely change during a jam.Source: §A.1 · p. 14
  • For an overhead obstruction, the harness uses depth maps to compute a horizontal approach beneath the obstacle rather than using the policy’s top-down motion.It also adds a wrist-rotation primitive for pouring and uses pointing and depth information to align and pull cabinet handles.Source: §A.1 · p. 14
  • On RoboCasa-365, the harness rescales action magnitudes while preserving direction when RLDX-1 pushes a toaster-oven door too gently to latch it.The door must close within 0.5% of its opening range.Source: §A.1 · p. 14
  • On RoboCasa-365, the harness anchors a mobile base near recent gripper activations, sets a driving budget, and retraces it to the latest anchor when that budget is spent.Source: §A.1 · p. 14, Figure 5 · p. 15
  • For spatially shifted LIBERO-Pro scenes, the harness queries a pointing model to locate the instructed target instead of reaching toward a memorized location.With RLinf-π0.5, it locates the instructed moka pot.Source: §A.1 · p. 14
  • When X-VLA on SimplerEnv WidowX provides no useful control signal, the harness disregards it and evolves into a vision-based picking agent that directly executes trajectories.It parses instructions, detects targets, estimates 3D poses from depth, and computes grasp vectors.Source: §A.1 · p. 14, Figure 6 · p. 15
  • When vision tools cannot reliably distinguish mahjong tile faces, the harness tries a ranked candidate list; for unreliable grasp checks, it uses fingertip depth to verify a grasp.Source: §A.1 · pp. 14–15, Figure 6 · p. 15
  • On the 3 Hz Google robot, the harness ends a move when the arm arrives or stops progressing rather than after a fixed time budget; the revision raised training-pool performance from 11.0% to 30.2%.It also reaches vertical-surface handles from the front along the measured surface normal.Source: §A.1 · p. 15

A.2 PERSISTENT MEMORIES

Persistent memory carries execution experience, task and environment clues, and embodiment-specific calibration into later decisions. These memories can support grasping novel objects, task-aware recovery, reset-aware execution, and grasp orientation across robots.

Learning execution parameters from experience

  • Successful grasp heights, grasp depths, and grip styles are stored against measured object properties or object identity to guide later grasps, including on novel objects.SimplerEnv WidowX records successful grasp heights by measured object height; LIBERO-Pro-10 stores successful grasp depth as a share of object height; OpenVLA-7B stores successful pinch or spread styles.Source: §A.2 · pp. 15–16

Tracking implicit task and environment clues

  • Task and environment memories enable empty-grasp recovery for picking but not pressing, anticipate RoboCerebra's 150-step resets, and cache instruction parses to avoid redundant computation.The harness associates instruction verbs with task needs and can take over placement before a known reset.Source: §A.2 · p. 16

Persistent self-calibration across embodiments

  • The unified harness calibrates each embodiment by observing empty-gripper motion in depth images, then uses the stored mapping to guide grasp orientation.It also records resting finger configuration as a reference for detecting empty grasps on robots that report finger positions.Source: §A.2 · p. 16

B IMPLEMENTATION DETAILS

The implementation evaluates harnesses across manipulation benchmarks using procedurally generated optimization episodes and benchmark-standard held-out episodes. A five-iteration evolution takes under six hours and costs under $150 on average.

Benchmarks

  • The test bed spans long-horizon LIBERO-Pro-10 sequences with swapped object locations, RoboCerebra scene resets, and visually grounded VLABench target selection.It also includes RoboCasa-365 kitchen navigation and arm control, plus SimplerEnv WidowX and Google evaluations under visual domain shift.Source: B IMPLEMENTATION DETAILS · pp. 16–17

Episode splits

  • The unified harness evolves on 2,201 procedurally generated episodes and is evaluated on 2,160 benchmark-standard episodes unseen during evolution.Each benchmark task receives the same number of trials, and held-out unseen tasks share underlying manipulation skills with optimization tasks.Source: B IMPLEMENTATION DETAILS · p. 17, Figure 9 · p. 17, Figure 8 · p. 17

Cost of automated harness optimization

  • A five-iteration evolution takes under 6 hours of coding-model time and costs under $150 on average, with no demonstrations collected or model weights updated.The authors describe the automated engineering loop as repeatable for each new robot setup.Source: B IMPLEMENTATION DETAILS · p. 17

C HARNESS EVOLUTION TRAJECTORIES

Across experimental settings, the trajectories show harness performance improving through iterative refinement and compare the final design with the bare policy and one-shot harness on held-out evaluations.

Unseen scenes

  • On unseen scenes, the final iteratively refined harness outperforms the one-shot harness on nearly every held-out set.The held-out episodes use the same tasks with different scene arrangements.Source: Figure 10 · p. 19

Unseen tasks

  • On unseen tasks, the final harness outperforms both the bare VLA policy and one-shot harness on every held-out set.The held-out tasks require underlying skills shared with the optimization tasks.Source: Figure 11 · p. 20

Unified agent design

  • Around general X-VLA, the unified harness improves on every training-pool benchmark, while around benchmark-finetuned specialists it leaves their scores intact.The figure compares bare policy, one-shot harness, and final harness on standard benchmark episodes unseen in the training pool.Source: Figure 12 · p. 21
Loading 2610.42141v1…