PaperlayerView exact revision on arXiv

Source-linked AI summary

Guava: An Effective and Universal Harness for Embodied Manipulation

Haowen Liu, Xirui Li, Shaoxiong Yao, Peng Shi, Tianyi Zhou, Jia-Bin Huang, Furong Huang, Jiayuan Mao

arXiv:2606.18363v1cs.ROcs.AI

TL;DR

It remains unclear what makes an effective embodied-manipulation harness and whether it can work across reasoning models. Guava explores harness designs and distills tool-use capabilities into a 4B model, which achieves a 75.6% overall success rate across ID and OOD tasks while generalizing to unseen objects, instructions, and long-horizon scenarios.

  • Problem

    It remains unclear which harness design choices support effective embodied manipulation and whether harnesses can unlock capabilities across reasoning models, including small open-source models.

    Source: §1 · p. 2

  • Method

    Guava explores workflow, action, and observation designs, then pairs iterative reasoning, semantic tools, and multimodal input with 4B-model distillation from fewer than 2K simulation trajectories.

    Source: §1 · p. 2

  • Results

    75.6% overall success rate: Guava-Agent-4B leads across ID and OOD tasks, outperforming GPT-5.4 (70.2%) and CaP-Agent0 (62.7%), with strong generalization to unseen objects, instructions, and long-horizon scenarios.

    Source: §1 · p. 2, §4.2 · p. 6

  • Takeaways & Limitations

    The findings support Guava as a scalable, model-agnostic interface through which compact open-source models acquire strong embodied manipulation capabilities with minimal training data.

    Source: §1 · p. 2, §5 · p. 8

    Guava cannot handle dexterous manipulation because of its current action primitives.

    Source: §5 · p. 9

Abstract

from arXiv · show

Language models trained on large-scale vision-language data have demonstrated strong potential for embodied agents. Harnessing models through embodied tools use offers a promising alternative to end-to-end vision-language-action systems by combining high-level reasoning with external modules for perception, planning, and control. However, it remains unclear what makes an effective harness for embodied manipulation, and to what extent such a harness can unlock embodied capabilities in a wide range of reasoning models. In this work, we present Guava, a harness framework for embodied tool use developed through systematic exploration of the design space of agent workflows, action spaces, and observation spaces. Our study identifies three key ingredients for effective embodied agents: iterative perception-reasoning-action loops, semantic action abstractions, and multimodal observations. To understand whether these design principles are universal even to small models, we develop an end-to-end training pipeline that distills embodied manipulation capabilities into a 4B open-source model using fewer than 2K trajectories collected entirely in simulation. Experimental results in both simulation and real-world environments show performance comparable to frontier proprietary models while exhibiting strong generalization to unseen objects, novel instructions, and long-horizon tasks. Results suggest that a well-designed harness can serve as a scalable, model-agnostic interface for embodied manipulation, enabling strong emergent embodied capabilities in compact open-source models with minimal training data.

1 Introduction

Guava combines iterative reasoning-action loops, semantic manipulation tools, and multimodal observations in a harness for embodied manipulation. With fewer than 2K simulation trajectories, its training pipeline equips a 4B model with strong manipulation capabilities and transfer to real-world tasks.

1 Introduction

  • Vision-language-action policies directly generate robot actions from visual observations and language instructions but usually require costly, embodiment-dependent robot demonstrations that are difficult to scale.The paper identifies scaling to diverse objects, scenes, and long-horizon real-world tasks as a challenge.Source: §1 · p. 1
  • Harness-based manipulation remains poorly understood, while existing systems often rely on one-shot code generation or domain-specific pipelines tied to powerful frontier models.The paper frames robust long-horizon behavior, failure recovery, inference latency, and cost as challenges for such systems.Source: §1 · p. 2
  • Guava combines iterative ReAct loops, semantic action abstractions, and rich multimodal observations to support adaptation, task planning, and embodied reasoning.The loops enable adaptation to execution outcomes and failure recovery, while semantic tools let language models focus on task decomposition and planning instead of low-level control.Source: §1 · p. 2
  • A 4B model distilled from fewer than 2K simulation trajectories performs comparably to frontier proprietary models and generalizes to unseen objects, instructions, and long-horizon scenarios.The learned model also transfers zero-shot from simulation to the real world and shows robust recovery under execution failures; these results support Guava as a scalable, model-agnostic interface.Source: §1 · p. 2

2 Related Work

Related work spans foundation-model policies and tool-using harnesses for robotic manipulation. The paper positions Guava’s contribution as an iterative harness that interleaves perception, reasoning, and action execution, and as a potential interface for transferring capabilities to small open models.

Foundation Models for Robotic Manipulation

  • Foundation-model policies learn broad manipulation skills but may require substantial robot data, while structured spatial interfaces improve grounding yet depend on carefully designed interfaces or planning primitives.Action-generation models often use diffusion or flow-matching objectives for continuous actions, and their internal representations can impede explicit constraint checking and plan repair.Source: §2 · pp. 2–3
  • Explicit action-reasoning and hierarchical methods improve policy reasoning, but learned action heads can make plans difficult to inspect and iteratively repair (Lee et al., 2025; Fang et al., 2026; Huang et al., 2026; Li et al., 2025).MolmoAct and MolmoAct2 use interpretable spatial or language-conditioned representations; ThinkAct uses a visual-plan latent, and HAMSTER separates high-level reasoning from low-level control.Source: §2 · p. 3

Harnessing Agents for Robotic Manipulation

  • Language-model harnesses compose perception, control tools, and APIs into executable behavior, but most rely on one-shot execution; this work studies a ReAct-style iterative harness and transfer to small open models.Prior work includes Code-as-Policies and ProgPrompt (Liang et al., 2022; Singh et al., 2023), multimodal code generation in RoboCodeX (Mu et al., 2024), multi-robot coordination (Mandi et al., 2024; Chen et al., 2025), and tool-calling frameworks Maestro and Cap-X (Shi et al., 2025; Fu et al., 2026).Source: §2 · p. 3

3 Gauva: Harnessing VLM for Embodied Manipulation

Guava evaluates harness design for robust manipulation and uses simulation-generated demonstrations to distill embodied tool-use capabilities into a compact model.

3.1 Designing Effective Harness

  • Figure 2 shows iterative workflows, semantic action abstractions, and multimodal observations improve performance and robustness across six long-horizon Robosuite tasks with GPT-5.4.Closed-loop reasoning supports replanning after execution failures; semantic tools delegate motion planning to lower-level controllers, while visual and textual state observations provide complementary grounding.Source: §3.1 · pp. 3–4, Figure 2 · p. 4, Table 1 · p. 4

3.2 Learning Efficient and Generalizable Agentic Embodied Reasoning

  • Fewer than 2K simulation-only trajectories transfer embodied capabilities to a compact 4B model, with perturbed-state recovery demonstrations improving robustness.The data engine collects diverse skills and reasoning behaviors, including grasping, pushing, spatial reasoning, and task planning.Source: §3.2 · pp. 4–5
  • The two-stage training pipeline applies supervised finetuning on successful and recovery trajectories, then GRPO with sparse task-success rewards on challenging long-horizon tasks.The RL tasks require iterative planning, tool use, and adaptation to execution errors.Source: §3.2 · p. 5

4 Experiments

Distilled from fewer than 2K simulation trajectories, Guava-Agent-4B achieves 75.6% overall simulation success and the highest real-world success rates on ID and OOD tasks. RL post-training substantially improves performance on two challenging long-horizon tasks, while closed-loop execution supports failure recovery.

4.1 Setup

  • Guava-Agent-4B is trained from fewer than 2K simulation trajectories and evaluated in simulation and on a real robot across ID, OOD-object, OOD-prompt, and OOD long-horizon tasks.The evaluation reports task success rate, and comparisons use the same environments and, where possible, the same observation inputs and tool APIs; baselines include the untrained foundation model, GPT-5.4, and CaP-Agent0.Source: §4 · p. 5, §4.1 · pp. 5–6

4.2 Small VLM Achieves High Performance with Guava

  • 75.6% overall simulation success is Guava-Agent-4B’s highest rate, ahead of GPT-5.4 (70.2%) and CaP-Agent0 (62.7%).It also achieves 100% on several ID and OOD tasks, and 93.3% on both reported long-horizon tasks; performance remains weak on shell game (6.7%) and placing all red objects in a basket (0.0%).Source: §4.2 · p. 6, Table 2 · p. 6
  • 86% on ID and 92% on OOD tasks are Guava-Agent-4B’s highest overall real-world success rates, with no additional real-world fine-tuning.It achieves 60% on push basket versus GPT-5.4’s 40%, and 100% on move object away and 90% on set table.Source: §4.2 · p. 6, Figure 4 · p. 7
  • RL post-training raises success from 6.7% to 60.0% on shell game and from 0.0% to 93.3% on placing all red objects in a basket.These extended tasks require reasoning about progress and recovering from intermediate failures; the paper attributes the gains to training on challenging long-horizon tasks with sparse success rewards.Source: §4.2 · p. 7, Figure 6 · p. 7
  • Guava-Agent-4B interleaves observation, reasoning, and action execution, enabling it to detect failures, revise plans, and recover during long-horizon tasks.The paper contrasts this closed-loop execution with one-shot planning approaches such as CaP-Agent0.Source: §4.2 · p. 7
  • Guava-Agent-4B successfully recovers from unseen real-world execution failures, including joint limits and unreachable poses, by replanning or returning home before retrying.The paper suggests these behaviors emerge from reasoning over execution feedback rather than memorized correction patterns.Source: §4.2 · p. 8, Figure 8 · p. 8

5 Conclusion · Appendix

Guava identifies harness design principles for embodied manipulation and transfers capabilities to a 4B open-source model using fewer than 2K simulation trajectories. The conclusion also notes limitations in dexterity, tool-level correction, and single-view perception.

5 Conclusion

  • Guava identifies three ingredients for effective embodied agents: iterative reasoning, semantic action abstractions, and multimodal observations.The conclusion presents these as key harness design principles for embodied manipulation.Source: §5 · p. 8
  • With fewer than 2K simulation trajectories, Guava transfers embodied capabilities to a 4B open-source model with strong generalization, robust recovery, and competitive real-world performance.The results suggest effective harnesses can serve as transferable interfaces that let compact open-source models acquire embodied capabilities with minimal training data.Source: §5 · p. 8
  • The current action primitives do not support dexterous manipulation.The authors identify current action primitive choices as the cause of this limitation.Source: §5 · p. 9
  • The system cannot directly correct tool-level errors such as invalid grasp proposals or incorrect SAM3 segmentations, but can detect failures and attempt recovery through retries or alternative actions.Source: §5 · p. 9
  • The setup assumes a single-view image from a fixed camera, which can be limited by occlusion or perspective effects.Future work may explore multiple dynamic views, such as adding a wrist camera.Source: §5 · p. 9
  • The authors plan to scale the setup to a larger dataset, larger model, and broader toolset to handle more diverse tasks and improve performance.Source: §5 · p. 9

A Dataset for Fine-tuning Guava-Agent-4B · A.1 Construction of Trajectories

The dataset is built by running Guava with GPT-5.4 in RoboSuite, where a standardized tool interface supports closed-loop interaction. Starting from task prompts, the process produces varied manipulation trajectories with observations, tool calls, reasoning traces, and environment interactions.

A Dataset for Fine-tuning Guava-Agent-4B

  • Guava is deployed with GPT-5.4 in RoboSuite to generate the fine-tuning dataset.The standardized API exposes observations, action execution, and episode-level feedback for closed-loop simulation interaction.Source: A Dataset for Fine-tuning Guava-Agent-4B · p. 14

A.1 Construction of Trajectories

  • Each trajectory records observations, tool calls, model reasoning traces, and environment interactions across a complete task.GPT-5.4 executes actions in RoboSuite from a collection of task prompts, yielding an initial pool spanning diverse manipulation tasks.Source: §A.1 · p. 14
  • Pose, lighting, and camera-view parameters are randomized to improve trajectory diversity and generalization.These variations are applied while generating the initial candidate pool.Source: §A.1 · p. 14
  • Task-generation instructions present the controller with a scene image and task to complete using available tools.The prompt frames the model as a robot-arm controller.Source: §A.1 · p. 14
  • The interaction loop requires step-by-step reasoning before exactly one tool call per response, ending with task complete or task failed.The specified terminal conditions are task completion or an irrecoverable failure.Source: §A.1 · p. 14
  • At each turn, the controller reasons about scene state, object and gripper poses, task progress, and the outcome of its last action before choosing what to do next.The guidelines also ask it to identify remaining work and account for unexpected outcomes.Source: §A.1 · p. 14
  • The prompt supplies the gripper’s position, rotation, and opening at every turn for use in reasoning.Position is given as [x,y,z], rotation as [roll,pitch,yaw], and opening as a percentage.Source: §A.1 · p. 14

A.2 Data Processing … B Tools

The sections describe how Guava filters and augments its training data, then define its manipulation tools and compare semantic with low-level interfaces. The filtered dataset contains 1,934 trajectories, and the low-level interface performs worse than the semantic one.

A.2 Data Processing

  • Curation retains successful episodes, removes invalid-tool or bad-initialization and manually identified low-quality rollouts, then de-duplicates near-identical prompts or histories to reduce bias.Source: §A.2 · p. 14

A.3 Recovery Behavior Generation

  • Recovery data perturbs successful trajectories with common errors and restarts from sampled intermediate states; only valid counterfactuals are retained, then the same filtering is applied.Perturbations include missed grasps, dropped objects, and wrong alignment.Source: §A.3 · pp. 14–15

A.4 Summary

  • 1,934 trajectories across 237 unique task prompts form the filtered dataset: 1,191 (62%) successful executions and 743 (38%) recovery trajectories.Source: §A.4 · p. 15

B Tools

  • The semantic interface exposes grasping and relative alignment alongside object position and size queries and Cartesian end-effector movement.Grasping segments RGB-D observations and estimates a grasp pose; alignment uses discrete approach directions and clearance grounded in segmented object geometry.Source: B Tools · p. 15
  • The alternative low-level interface uses a fused pose-and-gripper movement primitive alongside unchanged position, size, and home-pose tools, requiring direct geometric and physical reasoning from the VLM.The model must determine grasp poses, end-effector orientations, clearances, and motion sequences, whereas semantic skills delegate motion planning and execution to lower-level controllers.Source: B Tools · pp. 15–16
  • Figure 2 shows the low-level tool interface consistently performs worse than the high-level semantic action interface, supporting semantic abstractions as a key ingredient for effective embodied performance.Source: B Tools · p. 16

C Training Details · C.1 Hyperparameters · C.2 RL Training

The training setup combines SFT with GRPO targeted exclusively at the two most challenging long-horizon manipulation tasks. This focuses reinforcement learning on sequential planning and recovery while limiting the cost of multimodal rollouts.

C.1 Hyperparameters

  • SFT uses a learning rate of 1 × 10−5, an effective batch size of 32, and 3 epochs.Source: §C.1 · p. 16
  • GRPO uses a learning rate of 5 × 10−6, an effective batch size of 12, 3 epochs, K = 4 rollouts per prompt, and a KL penalty of β = 0.04.Source: §C.1 · p. 16
  • Training runs on 8 NVIDIA H100 80GB GPUs with bfloat16, FlashAttention-2, DeepSpeed ZeRO-3, and ms-swift (ModelScope Community, 2025).Source: §C.1 · p. 16

C.2 RL Training

  • GRPO is applied exclusively to the two most challenging long-horizon manipulation tasks, which are the primary bottleneck for overall policy performance.These tasks require more multi-step reasoning, error recovery, and action planning than the remaining tasks.Source: §C.2 · p. 16
  • Restricting RL avoids rapidly growing on-policy trajectory costs from repeated multimodal inference and environment interaction, with diminishing returns expected on simpler SFT-learned tasks.Applying RL to the full task suite would require substantially more training resources.Source: §C.2 · p. 16
  • Targeted RL updates on challenging long-horizon tasks yield more favorable performance gains than uniformly allocating RL compute across all tasks.The strategy is intended to improve sequential planning, recovery from intermediate failures, and long-range task completion while remaining tractable under the training budget.Source: §C.2 · p. 16

D Additional Results

Additional tests report consistent performance from Guava with two frontier models, weaker performance from a small model, and lower token use by Guava-Agent-4B than GPT-5.4.

Guava Harness on Additional Models

  • Guava works consistently with Gemini-3.1-Pro (Fu et al., 2026) and Claude-Sonnet-4.6, while Qwen3.5-2B (OpenAI, 2026a) performs poorly.The paper attributes Qwen3.5-2B’s performance to its small size, weak instruction following and tool calling, hallucinated reasoning, and invalid tool calls.Source: D Additional Results · p. 16, Table 3 · p. 17

Efficiency

E Additional Discussions

The discussion identifies weak spatial reasoning as a major VLM bottleneck, while describing task-progress awareness and simulation-to-real transfer as strengths of the agent. It also cautions that simulation failures may not reflect real-world performance, making real-world evaluation essential.

Spatial Understanding in VLMs

  • Precise spatial tasks have the lowest success rates across models, as VLMs often rely on coordinate heuristics instead of inferring spatial relations from images.This shortcut can cause failures in tasks such as pushing and object arrangement, which require reasoning about direction, orientation, and relative position.Source: E Additional Discussions · p. 17

Awareness of Execution State

  • The agent recognizes completed subtasks after interrupted execution and resumes appropriately, such as inserting an already-grasped peg without repeating the grasp.Figure 10 illustrates the agent proceeding directly to insertion when the peg is already grasped.Source: E Additional Discussions · p. 17, Figure 10 · p. 18

Sim2Real for Embodied Agent

  • Separating high-level semantic planning from low-level perception and control supports simulation-to-real transfer and generalization to new scenes without task-specific robot data.Unlike robotics RL or IL policies described in the passage, this formulation can leverage pretrained VLMs' general visual understanding.Source: E Additional Discussions · p. 18
  • Imperfect simulated contact dynamics can create failures that do not reflect real-world performance, so real-world evaluation remains essential.The discussion calls for more accurate and efficient evaluation across both simulation and real-world settings.Source: E Additional Discussions · p. 18

F Examples

Example trajectories show Guava performing long-horizon planning, recovering from execution failures, and reasoning about spatial relationships in the scene.

F Examples

  • The examples demonstrate long-horizon planning for tasks involving multiple objects.Figure 11 shows an example task requiring the robot to pick up and place multiple objects.Source: F Examples · p. 18, Figure 11 · p. 19
  • The system detects grasping failures and retries execution to complete a task.This recovery behavior is illustrated in Figure 12.Source: F Examples · p. 18, Figure 12 · p. 20
  • The trajectories illustrate reasoning about spatial relationships, including pushing an object in the right direction.Figure 13 shows a pushing task requiring spatial understanding and a correctly directed push.Source: F Examples · p. 18, Figure 13 · p. 21

Related papers

Loading 2606.18363v1…