PaperlayerView exact revision on arXiv

Source-linked AI summary

HarnessPAI: An Evolving Harness for Physical AI

Xin Wang, Wenhao Wu, Menghao Zhang, Zhi Wang, Kun Shao, Jian Luan, Yang Li, Qing Li, Shangding Gu, Huichi Zhou, Shuqing Shi, Fei Ni, Shuo Lu, Weicheng Meng, Kang Li, Jin Wu, Kang Zhao, Shangmin Guo, Gen Li, Yongqiang Tang, Zhizhong Zhang, Yuan Xie, Heng Qu

arXiv:2609.29166v1cs.ROcs.AI

TL;DR

Physical AI research has centered on action models, while reliable behavior still requires perception, reasoning, and action to work together. HarnessPAI organizes and evolves action models through executable programs, improving performance across evaluated embodiments and benchmarks, including RoboCasa atomic tasks, where it raises WorldDreamer from 65.0% to 92.2%.

  • Problem

    Physical AI research has centered on learned action models, while reliably closing the perception-to-action loop under physical constraints remains challenging.

    Source: §1 · p. 3

  • Method

    HarnessPAI uses executable programs to organize perception, reasoning, and action-model invocation, executing open-loop within rollouts and evolving programs across rollouts using execution feedback.

    Source: §4.2 · p. 13, §7 · p. 31

  • Results

    Across evaluated embodiments and benchmarks, HarnessPAI improves on underlying action models; on RoboCasa atomic tasks, it lifts WorldDreamer from 65.0% to 92.2%.

    Source: §1 · p. 4, §7 · p. 31

  • Takeaways & Limitations

    The results indicate that executable code can be a practical interface for more robust and inspectable physical agents without replacing learned control.

    Source: §7 · p. 31

    HarnessPAI depends on the underlying model retaining atomic physical competence and cannot supply primitive abilities the model lacks.

    Source: §6.2 · p. 30

Abstract

from arXiv · show

Physical AI aims to build embodied agents that perceive the world, understand and reason about it, and decide how to act. Yet the field has focused primarily on the last component: the action model that maps observations to low-level controls. The prevailing training recipe can erode the perceptual and reasoning capabilities needed for robust behavior, leaving even strong action models vulnerable to scene perturbations and long-horizon tasks. We introduce HarnessPAI, a model- and embodiment-agnostic Harness framework for Physical AI that treats code as the executable and evolvable interface that organizes the underlying action primitive. The framework separates two timescales: within a rollout, it executes open-loop at the program level, with a fixed program guiding and checking execution; across rollouts, it evolves closed-loop, using execution feedback to revise the program and distill failures into reusable skills. Across desktop robot arms, household robots, a robot vacuum, and a legged walking agent, HarnessPAI improves on both pure action models and code-as-policy baselines without retraining the underlying model: a 61.6-point gain over $π_{0.5}$ on LIBERO-PRO and a 27.2-point gain over WorldDreamer on RoboCasa atomic tasks. Once a program is selected, rollout execution requires no online high-level LLM deliberation. Beyond execution, the converged program is also a cheap and reliable expert-data collector, and fine-tuning $π_{0.5}$ on collected expert data lifts success rate on LIBERO-PRO by 38.8 points. Our results suggest that the frontier of Physical AI depends not only on stronger action models, but also on executable harnesses that integrate perception, task understanding and reasoning, and action execution into a unified, verifiable, and feedback-driven system. Website: https://darwin-agent.github.io/HarnessPAI

1 Introduction

HarnessPAI organizes learned action models through executable programs that combine physical perception, reasoning, and verification. Programs execute without online high-level deliberation within rollouts and evolve from physical feedback across rollouts.

1 Introduction

  • HarnessPAI treats code as a model- and embodiment-agnostic executable interface that organizes perception, task reasoning, verification, and learned physical actions.The underlying action model remains the action primitive for contact-rich manipulation.Source: §1 · pp. 3–4
  • Programs execute open-loop without high-level deliberation within each rollout, then evolve across rollouts as feedback diagnoses failures and guides revisions.This shifts adaptation from transient online reasoning to persistent executable behavior that can be validated across scenes and reused.Source: §1 · pp. 3–4
  • HarnessPAI provides a fast, reproducible, verifiable physical execution substrate, while stronger models can aid code generation and diagnosis without replacing validated programs or accumulated repair knowledge.This complements foundation-model advances by turning transient reasoning into persistent physical capability.Source: §1 · p. 4
  • Across seven settings spanning four embodiment categories, HarnessPAI reaches 96.5% on LIBERO-PRO, 8.8 points above the strongest harness baseline; fine-tuning raises success from 34.9% to 73.7%.The collected trajectories improve the wrapped action model through post-training.Source: §1 · p. 4
  • A graph-structured, node-centric skill memory stores failure-to-repair knowledge and transfers it across tasks with shared workflow stages.Physical observations, execution traces, failures, and outcomes ground program revisions.Source: §1 · pp. 3–4

2 Related Work

Related work locates embodied adaptation in model parameters, executable programs, or runtime harnesses. HarnessPAI combines pretrained action-model competence with an evolvable executable layer.

2.1 Post-training VLAs and WAMs

  • VLAs map visual observations and language to actions, while WAMs add predicted visual futures; both still adapt through model training rather than HarnessPAI’s fixed-model program evolution.VLAs can remain fragile on tasks modestly outside their training distribution (Xie et al., 2026, Zhou et al., 2025).Source: §2.1 · pp. 4–5

2.2 Code as Policy

  • Code-as-policy systems compose inspectable programs from robot interfaces, but HarnessPAI uses code to organize and verify learned action models rather than relying solely on control APIs.Control APIs can provide precise geometric control but may lack learned adaptability and contact-rich robustness when scene conditions or intermediate states differ from assumptions.Source: §2.2 · p. 5

2.3 Harnesses for VLAs and WAMs

  • Prior VLA/WAM harnesses add memory, planning, verification, or recovery, whereas HarnessPAI makes a complete task program its adaptation unit.Validated programs execute directly on recurring tasks; evolving code can reuse structured skills and transfer across VLA, WAM, and other action backends.Source: §2.3 · p. 5

2.4 Sim-to-Real Transfer and Real-Robot Validation

  • Sim-to-real work uses randomization, simulator adaptation, and calibration, while HarnessPAI must validate executable assumptions under sensing error, latency, contact variation, and hardware faults.These transfer demands motivate safety-gated validation on real robots.Source: §2.4 · p. 6

3 Motivation

Section 3 motivates a harness by asking why action models alone are insufficient, why execution should be open-loop at the program level, and why code can organize the missing capabilities. The proposed approach complements action models with explicit grounding, checks, and feedback-driven program evolution.

3.1 Why Do VLA/WAM Models Need a Harness?

  • Action competence alone does not ensure reliable task completion under instruction changes, scene perturbations, or fine-manipulation demands; a harness can add grounding, reasoning, and execution checks.The cited LIBERO-PRO discussion reports that models above 90% on standard LIBERO lose most of their accuracy under scene perturbations.Source: §3.1 · pp. 7–8

3.2 Why Open-Loop Execution?

  • Because physical mistakes can persist or be irreversible, the program should remain fixed within each rollout while observations and predefined checks guide execution, with revisions between rollouts.Feedback-driven revisions aim to improve subsequent executions; real-world deployment cannot assume environment resets.Source: §3.2 · pp. 9–10

3.3 Why Use Code as a Physical AI Harness?

  • Code is executable, inspectable, and evolvable: it grounds tasks, invokes action models as primitives, checks success, and makes perception and recovery decisions readable and editable.It can wrap different VLA and WAM backbones as interchangeable primitives, or directly control an API when no learned action model is available.Source: §3.3 · pp. 10–11
  • HarnessPAI targets tasks executed repeatedly, refining programs across rollouts and reusing repair skills distilled from prior tasks to respond to failures in new scenes and instructions.Examples include re-localizing a target after a missed grasp, verifying a press, and re-grasping after a slip.Source: §3.3 · p. 11

4 Framework

HarnessPAI organizes perception, geometric reasoning, and action in executable programs that verify progress within rollouts. Across rollouts, it diagnoses and revises programs, reuses distilled repairs, and can turn successful program rollouts into training data for the action model.

4.1 Open-Loop Execution

  • HarnessPAI uses a fixed, self-checking program to coordinate scene perception, geometric reasoning, and action, treating the action model as one replaceable primitive rather than the agent itself.The program grounds the scene, reasons about where and how to act, then checks whether an outcome succeeded before proceeding; its decisions are readable, testable, and editable.Source: §4.1 · pp. 11–12

4.2 Closed-Loop Evolution

  • Every task enters a closed loop that builds, runs, diagnoses, and revises code, while optional embodiment-specific steps supply only capabilities the task requires.Optional steps generate a workflow graph from expert video, assess a VLA or WAM, or initialize perception code; errors trigger skill-based repair, while successful code and skills are saved.Source: §4.2 · pp. 12–13

4.3 Evolution Mechanism

  • Evolution uses a success contract and staged checks to detect failures, diagnose them, and promote only code that passes the full episode batch.Pre-flight catches execution failures before environment evaluation; probe rechecks three previously failed seeds before a full-batch rerun. A separate VLM diagnoses rollout videos, while gate reward provides a model-free fallback; candidates remain isolated until promotion.Source: §4.3 · pp. 14–15

4.4 Skill Library

  • The skill library indexes failure-to-repair notes by workflow node and links each note to maintained repair code, enabling targeted retrieval and transfer across tasks.Skills learned at shared stages such as grasp or transport can be reused in a new task, so each failure becomes a node-keyed repair for later runs.Source: §4.4 · pp. 15–17

4.5 Model Feedback

  • Successful rollouts from a converged harness provide demonstrations for supervised fine-tuning, but the harness is not re-optimized against the improved action model.This one-way feedback supplies trajectories for perturbed scenes without new human labels; fine-tuning is the evaluated offline route, while reinforcement learning is described as an alternative.Source: §4.5 · pp. 17–18

5 Experiment

The experiments evaluate HarnessPAI across seven benchmarks and varied action backends, covering performance, transfer, adaptation, evolution, and execution cost. They also examine fixed-program execution efficiency and how collected trajectories affect π0.5 performance after fine-tuning.

5.2.1 robosuite; 5.2.2 LIBERO

  • HarnessPAI achieves 96.3% average success on robosuite, above ASPIRE (81.0%), and 98.1% overall on LIBERO, 0.2 points above AtomVLA (97.9%).On LIBERO, HarnessPAI is the only comparison method in the top two across all three suites. Its 79.8-point advantage over OpenETA (18.3%) is a complete-system comparison, not an isolated program-evolution effect, because the systems use different action backends.Source: Figure 12 · p. 19, §5.2.1 · p. 19, Table 4 · p. 20

5.2.3 LIBERO-PRO

  • HarnessPAI reaches 96.5% overall on LIBERO-PRO, 8.8 points above Harness VLA CC (87.7%) and 24.8 points above ASPIRE (72%).It scores 95.7% on Swap and 97.4% on Task; the same program runs unchanged across the base suite and both perturbations.Source: Figure 13 · p. 21, §5.2.3 · p. 21

5.2.4 RoboCasa

  • On RoboCasa Atomic-Seen, HarnessPAI scores 92.2%, lifting WorldDreamer from 65.0% by 27.2 points.Harness VLA (Codex) reports 91.6% under a different evaluation protocol; WorldDreamer and HarnessPAI were evaluated under a unified setting.Source: Table 5 · p. 22, §5.2.4 · p. 22

5.2.5 BEHAVIOR-1K

  • On BEHAVIOR-1K, eight-round adaptation raises task success from 84% to 100% for radio pickup and from 72% to 92% for soda-can pickup.Navigation success remains 100% and 92%, respectively, so adaptation closes the navigation-to-completion gap on episodes the robot can reach.Source: Figure 14 · p. 22, §5.2.5 · pp. 22–23

5.2.6 VacuSim

  • On VacuSim, campaign evolution raises coverage to 63.76% on the unfurnished map and 53.92% on the furnished map.Diagnostics and painted coverage guide code revision, and the final policy runs without an LLM call on the critical path.Source: Figure 15 · p. 23, §5.2.6 · p. 23, Figure 16 · p. 24

5.2.7 MicroDuck

  • On MicroDuck, the harness keeps lateral error near zero and reaches the endpoint, while the same pure PPO model drifts up to 0.8 m over 4 m.Code measures heading error and steers the unchanged Walker policy; no retraining is used.Source: Figure 17 · p. 24, §5.2.7 · p. 24

5.3 Harness Zero-shot

  • A π0.5-evolved harness transfers zero-shot to DreamZero, raising LIBERO-PRO success from 0.6% to 73.4% on Swap and from 9.6% to 84.4% on Task.The harness is neither re-evolved nor retuned; its code supplies perception, geometric reasoning, and verification around the new action model.Source: §5.3 · pp. 24–25, Figure 18 · p. 25

5.4 Evolution Process; 5.5 Cost

  • With the skill library, all three illustrated LIBERO-Object tasks reach 100% within two evolution rounds, while no-skill search follows varied paths to convergence.The cost study measures complete 50-rollout runs for Harness VLA and HarnessPAI in serial, plus HarnessPAI parallel execution across 7 GPUs; no comparative cost values are supplied here.Source: Figure 19 · p. 25, §5.4 · p. 26, §5.5 · p. 27

5.5 Cost

  • 23.1× and 18.4× speedups over serial Harness VLA on Swap and Task rise to 147.4× and 117.8× in the seven-GPU configuration.HarnessPAI takes 49.2 s per Swap rollout and 53.2 s per Task rollout in serial execution; its parallel variant takes 7.7 s and 8.3 s, respectively.Source: §5.5 · p. 27

5.6 Harness Trajectories for Post-Training

  • Success-filtered harness trajectories cost roughly one second and $0 of API charges per episode, making the converged program an expert-data collector.A hard success gate retains successful rollouts and discards failures; fixed-program execution has no online high-level LLM API charge, but this does not mean perception, inference, simulation, hardware, or lifecycle costs are zero.Source: §5.6 · pp. 27–28, §5.5 · p. 27
  • Object, Spatial, and Goal suite fine-tuning raises LIBERO-PRO success from 21.5% to 88.0%, 50.0% to 82.5%, and 32.5% to 54.5%, respectively.Each suite’s best checkpoint within 15,000 training steps is reported; training uses that suite’s harness trajectories, with base, Swap, and Task conditions mixed equally.Source: Figure 22 · p. 28, §5.6 · p. 28
  • 34.9% to 73.7% for π0.5-LIBERO and near zero to 53.3% for π0.5-Base on LIBERO-PRO after fine-tuning on the full harness-trajectory mixture.Both checkpoints improve across all three suites and both perturbations; results use each checkpoint’s best point within 30,000 training steps, with test seeds held out from added fine-tuning data.Source: §5.6 · p. 28, Figure 23 · p. 29

6 Discussion

HarnessPAI divides work between code and a frozen action model, and its benefits depend on the model retaining atomic physical competence. Its task graph can be seeded from weak or related-task demonstrations, then successful converged rollouts can supply further training data.

6.1 Why the Framework Works

  • 18.3% is the code-only success rate on LIBERO; HarnessPAI assigns grounding, geometric reasoning, and verification to code and atomic physical acts to the frozen action model.The discussion says action training can degrade a VLA’s visual grounding and instruction understanding, while code alone cannot express contact-rich dexterity.Source: §6.1 · p. 30

6.2 When the Framework Works

  • HarnessPAI helps only when the VLA retains atomic skills; it cannot create failed primitives, and gains shrink on opaque WorldDreamer composite trajectories.On composite tasks, intermediate steps are harder for code to inspect and weaker to begin with, limiting the harness’s ability to steer them.Source: §6.2 · p. 30

6.3 Where the Expert Data Comes From

  • A task graph needs only phase structure and order, so a single successful video or a related-task demonstration can seed it without a human expert.The graph discards imperfect motion, and node-organized skills can transfer to novel tasks that share a phase.Source: §6.3 · p. 30
  • After convergence, successful harness rollouts become action-model training data, reducing reliance on human labels to sketching a graph and occasional successes or related-task examples.The policy generates trajectories in scenes where human demonstrations are absent.Source: §6.3 · p. 30

7 Conclusion

HarnessPAI uses code as an executable harness that organizes perception, reasoning, and action, evolving programs from execution feedback and reusing successful trajectories to train action models. It improves benchmark success while avoiding high-level model calls during rollout execution.

7 Conclusion

  • The paper argues that action-model-centric end-to-end policies can let perception, grounding, and planning atrophy, and that code can instead organize, check, and evolve learned control.The authors present code as a practical path to more robust and inspectable physical agents.Source: §7 · pp. 30–31
  • Within rollouts, HarnessPAI executes open-loop, grounding decisions and verifying steps; across rollouts, it evolves closed-loop by revising programs from feedback and distilling failures into transferable, node-keyed skills.The framework treats code as the harness organizing perception, geometric reasoning, and action-model invocation, not as a replacement for a VLA or WAM.Source: §7 · p. 31
  • After the harness converges, its successful trajectories can train the action model on tasks for which no expert demonstration exists.Source: §7 · p. 31
  • HarnessPAI reaches 98.1% overall on LIBERO, 96.5% on LIBERO-PRO, and raises WorldDreamer from 65.0% to 92.2% on RoboCasa atomic tasks.The evaluation spans robot arms, a humanoid, and a mobile robot, driven by GraspNet, π0.5, WorldDreamer, and a pure API without a learned model.Source: §7 · p. 31
  • Open-loop execution puts no model call on the critical path, making the reported gains orders of magnitude faster and free of per-run API cost.Source: §7 · p. 31

8 Future Work · Appendix

Future work proposes a uniform atomic-action interface for underlying models, less reliance on online perception, and complementing coding agents with reusable, validated HarnessPAI code packages.

8 Future Work

  • A standardized contract would let HarnessPAI compose models of any paradigm through one instruction-to-atomic-action interface, without reasoning about their internals.The proposed interface covers PPO-trained policies, VLAs, and WAMs.Source: §8 · p. 31
  • A uniform interface would let HarnessPAI benefit directly from progress across all underlying model families.Source: §8 · p. 31
  • HarnessPAI’s reliance on online perception is a source of fragility and recurring cost during program evolution.Source: §8 · p. 31
  • Building maps or scene models ahead of time, or using direct real-to-sim transfer, could make perception cheaper and more stable.The passage presents these as ways to offload reliance on online perception.Source: §8 · p. 31
  • Coding agents and HarnessPAI are complementary: agents provide open-ended reasoning, while HarnessPAI provides validated, reusable code packages.Source: §8 · p. 31
  • For long-horizon tasks, an agent could reuse or lightly adapt a similar code package and reserve reasoning for genuinely novel situations.This would avoid deliberating or invoking a model at every step.Source: §8 · p. 31

A Main Benchmark Results Table

The section describes benchmark tables reporting HarnessPAI comparisons on LIBERO-PRO and robosuite, including improvement over specified baselines.

A Main Benchmark Results Table

B Detailed Results · B.1 LIBERO

The detailed LIBERO results report per-task outcomes for HarnessPAI, its π0.5-LIBERO baseline, and DreamZero-LIBERO, supporting comparisons summarized in Table 4.

B.1 LIBERO

  • Table 8 reports HarnessPAI’s per-task results across three LIBERO suites, which underlie Table 4’s Spatial, Object, and Goal columns.Source: §B.1 · p. 37
  • Table 9 reports per-task results for the underlying π0.5-LIBERO model, the baseline measured against HarnessPAI in Table 8.Source: §B.1 · p. 37
  • Table 10 reports per-task DreamZero-LIBERO results from the authors’ evaluation on standard LIBERO, underlying its Table 4 row.Source: §B.1 · p. 37

B.2 LIBERO-PRO Perturbations

This section reports HarnessPAI’s per-task results under the LIBERO-PRO Swap and Task perturbations and compares them with the underlying π0.5 model. The supplied passages also report a Spatial subtotal of 486/500 (97.2%).

B.2 LIBERO-PRO Perturbations

  • Table 11 gives HarnessPAI’s per-task results for the Swap and Task perturbation axes, with Table 12 reporting the underlying π0.5 baseline.The passages do not provide the per-task comparison values from those tables.Source: §B.2 · p. 37
  • 486/500 (97.2%) is the reported Spatial subtotal in the LIBERO suite results.The evaluation covers 50 seeds per task.Source: §B.2 · p. 38, Table 8 · p. 38

LIBERO-Goal

The reported LIBERO-Goal results show high subtotals for π0.5 and DreamZero, while another reported Goal subtotal is 41.2%.

LIBERO-Goal

B.3 RoboCasa · B.4 Harness Zero-shot · C Skills Case

The appendices report per-task RoboCasa and zero-shot transfer results, then illustrate how a reusable skill diagnoses and repairs a basket-mouth alignment failure.

B.3 RoboCasa

  • Table 13 reports HarnessPAI per-task success rates on RoboCasa atomic tasks, and Table 14 reports the underlying WorldDreamer WAM baseline rates.These per-task results underlie the aggregate comparison in Table 5; the supplied passages do not provide the table values.Source: §B.3 · p. 43, Table 13 · p. 43, Table 14 · p. 43

B.4 Harness Zero-shot

  • Table 15 reports per-task LIBERO-PRO Object Suite success rates for DreamZero and DreamZero with HarnessPAI under Swap and Task perturbations.The harness evolved on π0.5 over LIBERO-Object and was applied unchanged to DreamZero.Source: §B.4 · p. 44, Table 15 · p. 44

C Skills Case

  • The failure arose because the all-points basket median included the angled view’s near wall, biasing target XY toward the camera while the percentile Z undershot the rim.The object-center servo aligned to the target, but the target represented the basket body rather than its mouth.Source: C Skills Case · p. 44
  • 5.3 cm X (depth) error separated the basket-body estimate from the mouth, leaving the object off the opening and clipped on the rim.The body estimate was [0.0585, 0.2653, 0.1321], while the mouth was [0.0056, 0.2708, 0.1386].Source: C Skills Case · pp. 44–45
  • The repair computes both target XY and Z from the upper-Z rim points, using their medians and falling back to all points if fewer than five rim points remain.The representative skill leaves the place servo unchanged; with the target set to the mouth, the object drops through the opening in one 3D move without a VLA.Source: C Skills Case · p. 45
  • The skill warns against mixing body-centroid XY with rim-percentile Z and treats the place target as a distinct failure surface from the object-center servo and grasp offset.The skill is a problem–repair note mounted on the task-graph node where the failure occurred.Source: C Skills Case · pp. 44–45
Loading 2609.29166v1…