PaperlayerView exact revision on arXiv

Source-linked AI summary

Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents

Yixian Zhang, Huanming Zhang, Feng Gao, Xiao Li, Zhihao Liu, Chunyang Zhu, Jiaxing Qiu, Yuchen Yan, Jiyuan Liu, Wenhao Tang, Zhengru Fang, Yi Nie, Changxu Wei, Yu Wang, Wenbo Ding, Chao Yu

arXiv:2607.08448v4cs.RO

TL;DR

Harness VLA addresses the mismatch between frozen VLAs’ local contact control and coding agents’ semantic, compositional reasoning for perturbed manipulation. It composes a frozen VLA contact primitive with fixed analytic primitives and planner-guided staging, achieving strong results across benchmark families while retaining the fixed primitive vocabulary.

  • Problem

    Frozen VLAs have strong local visuomotor skills but struggle with deployment perturbations, while analytic primitives struggle with irregular grasping, constrained placement, and articulated-object manipulation.

  • Method

    Harness VLA uses a memory-augmented planner to compose a frozen VLA contact primitive with fixed analytic primitives for grounding, staging, transport, navigation, posture, and release.

  • Results

    Harness VLA improves over the strongest relevant baselines by 38.6 percentage points on LIBERO-Pro and 27.1 percentage points on RoboCasa365, while reaching 58.4% on RoboTwin C2R.

  • Takeaways & Limitations

    The results support using pretrained VLAs as local contact specialists within an auditable planner that handles semantic grounding, non-contact execution, staging, retries, and verification.

  • Takeaways & Limitations

    The framework has an open feedback loop between its high-level planner and low-level VLA, lacks joint fine-tuning with environmental rewards and human preferences, and lacks fine-grained image captioning for highly cluttered long-horizon tasks.

Abstract

from arXiv · show

Language-conditioned manipulation requires both precise contact-rich control and robust reasoning over language, scenes, and long horizons. End-to-end Vision-Language-Action (VLA) models provide strong local visuomotor skills, but they are trained on in-distribution task trajectories and often fail under deployment perturbations such as semantic retargeting, goal re-binding, spatial-layout shifts, and unstable local contacts. LLM coding agents provide complementary semantic and compositional reasoning, but purely analytic primitives struggle with irregular grasping, constrained placement, and articulated-object interaction. We present Harness VLA, a memory-augmented agentic framework that exposes a frozen VLA as a retryable contact-rich primitive and composes it with a small fixed library of analytic primitives for grounding, staging, transport, navigation, and release. Rather than expanding the skill library, the harness learns the operating range of these fixed primitives from task-specific execution traces, global success rules, and failure models. By lifting semantic re-grounding, non-contact execution, and VLA re-staging to the planner while reserving the frozen VLA for local contact-rich phases, Harness VLA extends pretrained VLAs beyond their original trajectory distribution without finetuning. Across perturbed tabletop, household kitchen, and clean-to-randomized bimanual manipulation, Harness VLA improves over the strongest relevant baselines by 38.6 and 25.4 percentage points on LIBERO-Pro and RoboCasa365, respectively, and reaches 58.4% on RoboTwin C2R. Code is available at https://github.com/RLinf/RPent.

1 Introduction

Harness VLA combines analytic planning with a frozen VLA to handle deployment perturbations while preserving the VLA’s strength in local contact-rich control. Memory-guided orchestration and retryable staging extend this fixed primitive composition across long-horizon manipulation tasks.

  • Motivation: End-to-end VLAs provide strong local contact-rich control but often fail outside their training trajectories, whereas coding agents provide semantic composition but struggle with delicate physical interactions.The framework assigns non-contact reasoning and composition to analytic planning while retaining the VLA for difficult contact phases.
  • Framework: Primitive composition lets analytic controllers traverse perturbed configurations and reserve VLA ACT for local contact-rich regions where its training distribution remains informative.This decomposition avoids asking a direct VLA rollout to bridge the entire perturbed task space.
  • Memory-guided orchestration: Task-specific traces and global success rules or failure models teach the planner when and how to use the fixed primitives instead of expanding the skill library.The planner can stage a local observation, invoke VLA ACT, inspect the outcome, and retry through re-staging.
  • Framework: Harness VLA composes a frozen VLA contact primitive with fixed analytic primitives, extending pretrained control to perturbed, long-horizon tasks without fine-tuning or deployment-time vocabulary expansion.The planner handles grounding, transport, staging, navigation, release, and re-staging around local VLA calls.
  • Empirical results: 38.6 and 27.1 percentage points are the reported improvements over relevant baselines on LIBERO-Pro and RoboCasa365, while RoboTwin C2R reaches 58.4%.Harness VLA also preserves competitive standard LIBERO performance across tabletop, household kitchen, and clean-to-randomized bimanual settings.

2 The Harness VLA Framework

The Harness VLA framework exposes a fixed JSON primitive interface to an agentic planner that combines analytic controllers with a frozen VLA. Exploration stores reusable traces and heuristics, while deployment dynamically grounds, executes, verifies, and retries primitive sequences.

  • Perception and planning: RGB images support qualitative scene reasoning, while aligned depth and proprioception provide metric spatial information for localization and primitive argument binding.The planner uses these multimodal observations within the execution-feedback loop.
  • Exploration and memory: During bootstrapping, the planner explores primitive compositions with reset access and a generous budget, then serializes successful calls into parameterized JSONL traces for reuse across layouts.Concrete coordinates are replaced with symbolic perception queries in the stored traces.
  • Deployment evaluation: During deployment, reset is disabled and budgets are shortened; the planner retrieves traces, grounds them with live RGB-D observations, and executes them under the fixed vocabulary.The harness also logs traces, retrieves both memory stores, enforces reset and budget policies, and checks benchmark progress predicates.
  • Unified primitive interface: The planner selects primitives through a JSON interface, and each primitive executes to an internal post-condition before returning refreshed observations.The planner binds primitive arguments from language, RGB-D observations, proprioception, and memory rather than emitting low-level torques or joint targets.
  • Analytic primitives: Analytic primitives provide deterministic, model-based control for world-frame goals or individual channels such as pose, wrist orientation, gripper state, and base velocity.The library includes composite and atomic controllers, while RoboCasa365 adds mobile-base primitives for kitchen-scale staging.
  • VLA-backed contact primitive: VLA ACT maps a task-conditioned prompt and live camera inputs to action chunks for local contact-rich behavior, stopping at an early-return predicate or chunk limit.Semantic grounding, spatial rebinding, navigation, re-staging, and long-horizon composition remain under planner control.

3 Experiments

Harness VLA evaluates a frozen VLA exposed as a retryable contact-rich primitive within a fixed analytic library, using memory-backed planning across standard, perturbed, household, and transfer settings. It improves robustness through semantic re-grounding, planner staging, and repeated local contact attempts.

  • Few-shot evaluation: 96.0%: Harness VLA preserves competitive standard LIBERO performance, achieving 384/400 successes versus 95.3% for the frozen πRLinf checkpoint.This exposes the same policy through a controllable primitive interface for perturbed evaluations.
  • Few-shot evaluation: 38.6 percentage points: Harness VLA reaches 82.4% on LIBERO-Pro, exceeding RATS at 43.8% overall under instruction-redirection and position-swap perturbations.The direct πRLinf baseline reaches 50.0% under the same protocol.
  • Few-shot evaluation: 27.1 percentage points: Harness VLA reaches 57.1% on RoboCasa365 with Codex, exceeding the RLDX-1 baseline at 30.0%.The planner handles navigation, staging, and re-staging while the frozen VLA performs local contact-rich manipulation.
  • Zero-shot evaluation: 58.4%: Harness VLA reaches this success rate on RoboTwin C2R, raising the same frozen LingBot-VLA backend from 50.4% direct performance.The transfer uses a clean-setting task trace without randomized-setting bootstrapping, additional task-level exploration, or VLA fine-tuning.
  • Mechanism analyses: Planner-level semantic re-grounding handles target rebinding and scene changes, while the frozen VLA remains responsible for local contact-rich execution.This division of labor addresses task-perturbed behavior without altering the visuomotor backbone.
  • Mechanism analyses: Repeated planner-selected VLA invocations improve reliability: the planner stages the robot, observes contact outcomes, and re-stages failed local attempts.Aggregate results show additional allowed invocations further improve success on longer or more contact-heavy tasks.

4 Related Work

The paper situates Harness VLA between end-to-end robot foundation policies, multimodal LLM agents, and programmatic robot-control systems. Its position is to use a frozen VLA for contact-rich control while an agent composes explicit control primitives.

  • Positioning: Harness VLA differs from prior end-to-end policies and agentic control systems by surrounding a frozen VLA with explicit primitive composition rather than fine-tuning the policy or expanding the library.The framework combines learned contact-rich control with planner-driven semantic and spatial handling.
  • VLA Models: End-to-end VLA models map language and visual observations directly to low-level robot actions, with recent work scaling generalist policies through heterogeneous cross-embodiment data.The cited background places Harness VLA within the broader VLA model lineage.
  • VLA Models: VLA models are strongest in contact-rich visuomotor phases, whereas deployment outside their training distribution motivates additional reasoning and control structure.The supplied passage introduces this asymmetry as a limitation of the paradigm.
  • LLM-driven Multimodal Agents: Multimodal LLM agents extend language-model reasoning with perception, spatial reasoning, tool use, feedback, and persistent execution state.These capabilities support the complementary semantic and compositional role assigned to the planner.
  • Programmatic and Tool-Using Robot Agents: Programmatic robot-control systems recast control as executable program synthesis over perception and motion APIs, combining compositional generalization with deterministic low-level control.Harness VLA adopts this explicit-control perspective while retaining a learned primitive for difficult contacts.

5 Conclusion and Limitations

Harness VLA concludes that a frozen VLA can serve as a contact-rich primitive inside an auditable, memory-guided execution loop. The framework achieves robust benchmark performance, but remains limited by feedback, training, and cluttered-scene reasoning constraints.

  • Conclusion: Harness VLA achieves state-of-the-art robustness by isolating pretrained VLA control to contact-rich phases and delegating transport, posture, navigation, and release to the planner.The conclusion characterizes this asymmetric allocation as the central system-level finding.
  • Limitations: The framework remains limited by an open planner–VLA feedback loop, absent joint fine-tuning with environmental rewards or human preferences, and limited fine-grained image captioning in cluttered long-horizon tasks.The paper identifies sample-efficient reinforcement learning and skill discovery as future directions.
  • Execution and Memory: The execution loop uses serialized primitive commands, persisted observations, diagnostic logs, and memory while withholding privileged simulator state from the planner.This file-mediated design makes execution auditable through indexed records and controlled interfaces.
  • Execution and Memory: Task-specific memory stores procedural traces and semantic summaries, while global memory stores reusable success rules and failure models for future planning.The trace transfers solution structure rather than replaying reference-scene coordinates.
  • Execution and Memory: Memory is refined across attempts: recoverable failures remain documented, and later solutions can replace earlier traces when they are shorter or more reliable.This construction supports correction without simply accumulating prior executions.

B Primitive Vocabulary and Environment-Specific Extensions

The primitive vocabulary keeps shared interfaces across embodiments while allowing environment-specific availability, arm binding, and backends. VLA ACT is the sole learned primitive; analytic primitives handle transport, orientation, release, navigation, and base motion.

  • Vocabulary and Extensions: The same primitive names are used across benchmarks, with environmental differences expressed through availability, arm binding, and implementation backend rather than new primitive names.New names are introduced only when an embodiment exposes a new degree of freedom.
  • Analytic Primitives: Universal analytic primitives include MOVE TO, MOVE POSE, ROTATE WRIST, ROTATE PITCH, SET GRIPPER, and RELEASE, with backend-specific controllers hidden behind shared semantics.These primitives provide deterministic non-contact motion and release operations.
  • Environment-Specific Extensions: RoboCasa365 adds NAVIGATE TO and MOVE BASE for mobile-base staging, while RoboTwin C2R binds existing primitives to left-arm, right-arm, or bimanual patterns.The extensions adapt execution to embodiment without expanding the manipulation vocabulary unnecessarily.
  • Learned Primitive: VLA ACT is the single learned primitive, executing frozen-policy action chunks for grasping, placement, fixture actuation, insertion, and bimanual contact under planner-selected stop predicates.The interface preserves contact-rich learned control while exposing it as a planner-invoked primitive.
  • Evaluation Protocol: The evaluation uses benchmark-provided binary completion predicates, and primitive post-conditions only return control to the planner rather than determining task success.LIBERO, LIBERO-Pro, RoboCasa365, and RoboTwin C2R are the four benchmark families described for evaluation.

C.1 LIBERO Evaluation Benchmark

LIBERO evaluates language-conditioned manipulation across suites that vary spatial relations, object identity, goal predicates, and compositional horizon. Each suite uses one exploratory seed to construct memory and ten held-out seeds for reported evaluation.

  • Benchmark Structure: LIBERO contains four evaluated suites—SPATIAL, OBJECT, GOAL, and LIBERO-10—covering spatial relations, target-object identity, goal predicates, and longer-horizon composition.Each suite contains ten language-conditioned tasks.
  • Evaluation Protocol: Each LIBERO suite uses seed s0 only for exploration and Task Specific Memory construction, while seeds s1–s10 provide held-out evaluation under new initial states.The protocol reports ten evaluation seeds per task.
  • Evaluation Protocol: The benchmark reports predicate-based success rates for the held-out LIBERO evaluation.The cited protocol references the common predicate-based success rule.

C.2 LIBERO-Pro Evaluation Benchmark

LIBERO-Pro extends LIBERO with controlled instruction and spatial perturbations across four task families. Evaluation uses held-out seeds after Task Specific Memory is constructed from a reference seed.

  • C.2 LIBERO-Pro Evaluation Benchmark: LIBERO-Pro evaluates SPATIAL, OBJECT, GOAL, and LIBERO-10 task families under instruction-redirection and position-swap perturbations.T denotes task or instruction redirection, while S denotes object-position swaps or rearrangements.
  • C.2 LIBERO-Pro Evaluation Benchmark: The benchmark reports eight evaluation cells, each containing 10 tasks.The cells combine four task families with the T and S perturbation settings.
  • C.2 LIBERO-Pro Evaluation Benchmark: Evaluation uses seed s0 to construct Task Specific Memory and seeds s1–s10 for held-out evaluation, totaling 800 reported rollouts.Each cell contains 10 tasks × 10 evaluation seeds = 100 reported rollouts.

C.3 RoboCasa Evaluation Benchmark

RoboCasa365 evaluates kitchen manipulation across atomic and composite task groups, including composite templates unseen during pretraining. Split-specific held-out seeds test memory-grounded performance under new initial states.

  • C.3 RoboCasa Evaluation Benchmark: RoboCasa365 target50 contains ATOMIC-SEEN, COMPOSITE-SEEN, and COMPOSITE-UNSEEN kitchen task groups.The groups contain 18 atomic tasks, 16 seen composite tasks, and 16 composite tasks whose templates are held out from pretraining.
  • C.3 RoboCasa Evaluation Benchmark: The benchmark uses split-specific held-out seeds after seed s0 constructs Task Specific Memory for each split.ATOMIC-SEEN uses s1–s10, while both composite splits use s1–s5.
  • C.3 RoboCasa Evaluation Benchmark: ATOMIC-SEEN contributes 180 reported rollouts and COMPOSITE-SEEN contributes 80 reported rollouts under the stated seed protocol.These counts follow 18×10 and 16×5 task-seed combinations, respectively.

C.4 RoboTwin Clean-to-Randomized Evaluation Benchmark

RoboTwin C2R evaluates bimanual manipulation across 50 task types in a clean-to-randomized setting. Task-specific traces come from clean demonstrations, while evaluation uses randomized seeds verified for task feasibility.

  • C.4 RoboTwin Clean-to-Randomized Evaluation Benchmark: RoboTwin C2R covers 50 bimanual tasks spanning transport, handover, articulated-object interaction, pressing, rotation, scanning, and container placement.The task set also includes pick-and-place, stacking, and ordering behaviors.
  • C.4 RoboTwin Clean-to-Randomized Evaluation Benchmark: Task Specific Memory is obtained from one expert-verified clean demonstration seed and evaluated on five expert-verified randomized seeds per task.The expert verification step checks instance feasibility under the official task definition and is independent of the method.
  • C.4 RoboTwin Clean-to-Randomized Evaluation Benchmark: The evaluation comprises 50 × 5 = 250 reported rollouts in the demo randomized setting.All 50 tasks are evaluated on five seeds each.

C.5 Benchmark Summary

The benchmark suite combines held-out-seed LIBERO evaluation, split-specific RoboCasa testing, and clean-to-randomized RoboTwin testing with frozen VLA backends. Across these settings, the backends show strong local manipulation but substantial degradation under perturbations and compositional shifts.

  • C.5 Benchmark Summary: Across benchmarks, different VLA models are uniformly abstracted as the single VLA ACT contact-rich primitive within Harness VLA.πRLinf is used for LIBERO, RLDX-1 for RoboCasa365, and LingBot-VLA for RoboTwin.
  • C.5 Benchmark Summary: The frozen backends use multimodal observations and language to generate continuous or chunked actions for contact-rich manipulation.πRLinf uses hierarchical subtask prediction and low-level action generation, while RLDX-1 and LingBot-VLA use chunked continuous-action modeling.
  • C.5 Benchmark Summary: πRLinf achieves 95.3% on LIBERO but 50.0% on LIBERO-Pro, indicating sensitivity to instruction perturbations and compositional variation.πRLinf is used as the frozen VLA ACT contact-rich primitive for LIBERO and LIBERO-Pro.
  • C.5 Benchmark Summary: RLDX-1 achieves 60.0% on Atomic-Seen, 21.3% on Composite-Seen, 5.0% on Composite-Unseen, and 30.0% overall on RoboCasa365.The checkpoint is used fully frozen as the kitchen contact-rich execution primitive.
  • C.5 Benchmark Summary: LingBot-VLA reaches 50.4% as a direct frozen agent under RoboTwin randomized evaluation before agent-level decomposition or external planning.It serves as the frozen low-level contact-rich primitive for Harness VLA bimanual control.

E Agent Prompt Specification

The shared prompt specifies a closed-loop, file-mediated manipulation agent that combines RGB-based localization, fixed primitives, frozen-VLA contact actions, and structured memory. Benchmark-specific prompts preserve this common operating structure while supplying environment-specific interfaces and success signals.

  • E.1 Shared Prompt Core: Each benchmark prompt uses a shared agent template, filling environment-specific slots for success predicates, embodiment, cameras, primitives, VLA backend, memory paths, and recovery rules.
  • E.1 Shared Prompt Core: The agent is instructed to read task state, localize entities, select primitives, invoke the VLA for contact-rich behavior, and produce reproducible execution artifacts.
  • E.2 Perception and File-Mediated Control: Perception is isolated from privileged simulator state: RGB supplies semantic identity, while depth and world maps provide metric localization from selected surface pixels.
  • E.2 Perception and File-Mediated Control: The prompt requires robust pixel sampling, avoidance of unstable visual regions, and re-localization whenever the robot, camera, object, base, fixture, or grasp state changes.
  • E.2 Perception and File-Mediated Control: Execution follows a file-mediated REPL: issue one JSON primitive command, wait for the driver, inspect new state and perception outputs, then choose the next action.
  • E.3 Seed 0 Task Specific Memory: Task Specific Memory separates a seed rollout’s strategic audit from its executable command trace, allowing procedural structure to transfer without replaying scene-specific coordinates.
  • E.4 Global Memory: Global Memory supplies task-independent success rules, failure models, and recovery guidance for fixed primitives rather than an action trace to replay.
  • E.6 Compact Prompt Skeleton: The compact prompt skeleton makes the agent optimize the benchmark success signal, obey the allowed primitive vocabulary, use the frozen VLA for contact-rich behavior, and continue until success, exhaustion, or unrecoverability.

F Primitive Usage Statistics

Primitive usage statistics show a stable division of labor: analytic controls dominate reproducible geometry and staging, while VLA ACT supplies learned contact-rich interactions. The balance shifts across environments according to task demands.

  • 61.8% of LIBERO calls use MOVE TO and 15.8% use VLA ACT, consistent with analytic transport and release following learned contact interactions.
  • VLA ACT reaches 35.3% of RoboCasa365 calls and 47.4% in RoboTwin C2R, while analytic primitives still provide a slight majority in RoboTwin.
  • Across all three embodiments, the frozen VLA functions as a contact-rich primitive inside an analytic scaffold rather than as a monolithic end-to-end controller.
Loading 2607.08448v4…