PaperlayerView exact revision on arXiv

Source-linked AI summary

Guava: An Effective and Universal Harness for Embodied Manipulation

Haowen Liu, Xirui Li, Shaoxiong Yao, Peng Shi, Tianyi Zhou, Jia-Bin Huang, Furong Huang, Jiayuan Mao

arXiv:2606.18363v1cs.ROcs.AI

TL;DR

It remains unclear which design choices make embodied manipulation harnesses effective and whether they transfer across models. Guava combines iterative reasoning, semantic actions, and multimodal observations, distilling tool-use behavior into a 4B model that performs competitively across simulation and real-world tasks with strong generalization.

  • Problem

    It remains unclear what makes an effective embodied manipulation harness and whether such a harness can support capable agents without extensive robot demonstrations.

  • Method

    Guava systematically evaluates embodied-agent design choices and combines iterative reasoning-action loops, semantic action abstractions, and multimodal observations in a unified harness.

  • Results

    75.6% overall success rate makes Guava-Agent-4B outperform GPT-5.4 at 70.2% and CaP-Agent0 at 62.7%, with strong generalization and real-world transfer.

  • Takeaways & Limitations

    Guava suggests that a well-designed harness can serve as a transferable interface for embodied manipulation in compact open-source models with minimal training data.

  • Takeaways & Limitations

    The method cannot handle dexterous manipulation because of its current action primitives.

Abstract

from arXiv · show

Language models trained on large-scale vision-language data have demonstrated strong potential for embodied agents. Harnessing models through embodied tools use offers a promising alternative to end-to-end vision-language-action systems by combining high-level reasoning with external modules for perception, planning, and control. However, it remains unclear what makes an effective harness for embodied manipulation, and to what extent such a harness can unlock embodied capabilities in a wide range of reasoning models. In this work, we present Guava, a harness framework for embodied tool use developed through systematic exploration of the design space of agent workflows, action spaces, and observation spaces. Our study identifies three key ingredients for effective embodied agents: iterative perception-reasoning-action loops, semantic action abstractions, and multimodal observations. To understand whether these design principles are universal even to small models, we develop an end-to-end training pipeline that distills embodied manipulation capabilities into a 4B open-source model using fewer than 2K trajectories collected entirely in simulation. Experimental results in both simulation and real-world environments show performance comparable to frontier proprietary models while exhibiting strong generalization to unseen objects, novel instructions, and long-horizon tasks. Results suggest that a well-designed harness can serve as a scalable, model-agnostic interface for embodied manipulation, enabling strong emergent embodied capabilities in compact open-source models with minimal training data.

1 Introduction

Guava identifies a harness design for embodied manipulation centered on iterative reasoning-action loops, semantic actions, and multimodal observations. It distills these principles into a 4B model using fewer than 2K simulated trajectories, enabling strong manipulation capabilities and real-world transfer.

  • Harness design: Harness-based systems let language models invoke external perception, planning, and control modules instead of internalizing all low-level manipulation capabilities.This modular design is presented as particularly well-suited to embodied manipulation.
  • Harness design: Guava identifies three critical principles for effective manipulation: iterative ReAct loops, semantic action abstractions, and rich multimodal observations.Iterative loops adapt to execution outcomes and recover from failures, while semantic actions support task decomposition and planning rather than low-level control.
  • Training and transfer: Fewer than 2K simulation-only trajectories distill embodied tool-use behaviors into a 4B open-source model.The end-to-end training pipeline tests whether the harness can function as a universal interface across models, including small open-source models.
  • Training and transfer: Guava acts as a scalable, model-agnostic interface that supports strong manipulation capabilities, robust failure recovery, and real-world transfer.These capabilities result from combining harness design with data-efficient post-training.

2 Related Work

Prior work builds generalizable robotic manipulation systems either by training foundation-model policies on large robot datasets or by harnessing language models with structured representations and callable tools. Recent approaches further make action reasoning explicit through interpretable representations, visual plan latents, or hierarchical separation of reasoning and control.

  • Foundation Models for Robotic Manipulation: Vision-language and vision-language-action models generate robot policies by adding action modules and training on large-scale trajectories, often with diffusion or flow-matching objectives.These models acquire broad manipulation skills but typically require substantial robot data.
  • Foundation Models for Robotic Manipulation: VLM-based methods ground high-level reasoning through structured spatial representations such as affordance maps and keypoint relations.This line of work aims to make model outputs usable for robotic manipulation without directly relying on opaque internal action representations.
  • Foundation Models for Robotic Manipulation: MolmoAct, MolmoAct2, and ThinkAct make action reasoning explicit using interpretable spatial or language-conditioned representations and visual plan latents.HAMSTER instead separates high-level reasoning from low-level control policies through a hierarchical design.
  • Harnessing Agents for Robotic Manipulation: Code-as-Policies and ProgPrompt harness language models to compose perception outputs, control primitives, and task-specific APIs into executable policies or situated task plans.These frameworks represent perception, planning, and control as callable tools, preserving the modularity of classical robotic systems.

3 Gauva: Harnessing VLM for Embodied Manipulation

Guava develops an embodied manipulation harness around iterative workflows, semantic action abstractions, and multimodal observations to improve robustness and grounding under stochastic execution. It also distills frontier-model tool-use behaviors into a compact 4B model using fewer than 2K simulated trajectories.

  • Harness design: Controlled ablations on six long-horizon Robosuite tasks evaluate workflow, action-space, and observation design choices for robust manipulation.The evaluation is based on frontier models.
  • Harness design: Iterative perception-reasoning-action workflows improve robustness by enabling replanning after execution failures and incorporating updated observations.This contrasts with single-turn planning under stochastic execution.
  • Harness design: Semantic action abstractions outperform low-level interfaces because they avoid requiring explicit geometric reasoning over poses, grasps, and trajectories.The semantic action space provides clear tool meaning while retaining fine-grained numerical parameters when necessary.
  • Harness design: Multimodal observations combine visual spatial information with textual robot-state and task-progress descriptions, improving grounding and reducing ambiguity.The two modalities provide complementary information during sequential decision making.
  • Policy distillation: Fewer than 2K simulated trajectories transfer frontier-VLM embodied tool-use capabilities to a compact 4B model.The data engine includes successful demonstrations and recovery trajectories generated from perturbed execution states, while training uses SFT followed by GRPO.

4 Experiments

Experiments show that Guava-Agent-4B transfers embodied manipulation capabilities to a compact model using fewer than 2K simulated trajectories, achieving strong simulation and real-world performance. Closed-loop execution and RL post-training improve long-horizon reasoning, failure recovery, and generalization beyond training failures.

  • Experimental setup: Guava-Agent-4B is distilled into a 4B-parameter VLM using fewer than 2K simulated trajectories and evaluated in simulation and the real world.The evaluation measures task generalization, failure recovery, and transfer from frontier models to compact open-source models.
  • Simulation results: Guava-Agent-4B achieves the highest overall simulation success rate at 75.6%, outperforming GPT-5.4 at 70.2% and CaP-Agent0 at 62.7%.It reaches 100% on place can in box, remove cube from tray, pick up carrot, lemon in bin, and stack cube reverse order, plus 93.3% on two long-horizon tasks.
  • Real-world transfer: Guava-Agent-4B achieves the highest real-world success rates on ID tasks at 86% and OOD tasks at 92%.It requires no additional real-world fine-tuning and reaches 100% on move object away and 90% on set table.
  • Long-horizon reasoning: RL post-training raises performance on shell game from 6.7% to 60.0% and on place all red objects in basket from 0.0% to 93.3%.These tasks require extended action sequences, recovery from intermediate failures, and reasoning over task progress.
  • Closed-loop execution: Closed-loop execution strengthens long-horizon manipulation by interleaving observation, reasoning, and action execution to detect failures, revise plans, and recover.This contrasts with one-shot planning approaches such as CaP-Agent0.
  • Failure recovery: Guava-Agent-4B successfully recovers from unseen real-world failures, including joint-limit and unreachable-pose control issues, by replanning or returning home before retrying.The observed corrective actions suggest recovery emerges from reasoning over execution feedback rather than memorization.

5 Conclusion · Appendix

Guava identifies three ingredients for effective embodied agents and transfers embodied capabilities to a compact 4B model using fewer than 2K simulation trajectories. The method generalizes strongly but remains limited by action primitives, tool-level errors, and fixed single-view observations.

  • 5 Conclusion: Guava’s three key ingredients are iterative reasoning, semantic action abstractions, and multimodal observations.
  • 5 Conclusion: Fewer than 2K simulation trajectories transfer embodied capabilities into a 4B open-source model.
  • 5 Conclusion: The resulting model achieves strong generalization, robust recovery, and competitive real-world performance.
  • 5 Conclusion: Effective harnesses can serve as transferable interfaces for embodied manipulation.
  • 5 Conclusion: The method cannot handle dexterous manipulation because of its current action primitives.
  • 5 Conclusion: The system cannot directly correct tool-level errors, including invalid grasp proposals or incorrect SAM3 segmentations.
  • 5 Conclusion: It can detect tool-level failures and attempt recovery through multiple retries or alternative actions.
  • 5 Conclusion: A single-view image from a fixed camera can limit performance because of occlusion or perspective effects.

A Dataset for Fine-tuning Guava-Agent-4B · A.1 Construction of Trajectories

Guava-Agent-4B’s fine-tuning data are generated by deploying the Guava harness with GPT-5.4 in RoboSuite, where a standardized API supports closed-loop simulated interaction. The resulting trajectories include multimodal observations, tool calls, reasoning traces, and environment interactions, with randomized scene parameters to improve diversity and generalization.

  • A Dataset for Fine-tuning Guava-Agent-4B: The dataset is constructed by deploying the Guava harness with GPT-5.4 in RoboSuite.This process uses the harness framework to generate data in simulation.
  • A Dataset for Fine-tuning Guava-Agent-4B: A standardized API exposes environment observations, action execution, and episode-level feedback for closed-loop interaction in simulation.The interface connects model decisions with environment feedback during episodes.
  • A.1 Construction of Trajectories: GPT-5.4 generates complete trajectories from task prompts by executing actions within RoboSuite.Each trajectory contains observations, tool calls, model reasoning traces, and environment interactions.
  • A.1 Construction of Trajectories: The initial candidate pool spans diverse manipulation tasks, with randomized poses, lighting, and camera views to improve diversity and generalization.Randomization is applied during trajectory generation from the task-prompt collection.
  • A.1 Construction of Trajectories: The generation prompt requires step-by-step reasoning inside <think></think> tags followed by exactly one tool call per response.Interaction ends when the task is fully complete or irrecoverable, with the model outputting Task complete or Task failed.
  • A.1 Construction of Trajectories: At every turn, the controller receives the gripper’s position, rotation, and opening percentage for use in reasoning.The provided state is [x,y,z] position, [roll,pitch,yaw] rotation, and gripper opening %.

A.2 Data Processing … B Tools

Guava curates successful, diverse manipulation data, augments it with valid recovery trajectories, and equips the agent with semantic tools that outperform a low-level geometric interface. The resulting fine-tuning set contains 1,934 trajectories, while the tool design shifts geometric execution burdens to lower-level controllers.

  • A.2 Data Processing: Guava retains successfully completed episodes, removes invalid-parameter and bad-initialization errors, and manually excludes low-quality samples with unrelated conversations.The pipeline combines automatic outcome filtering, error filtering, and manual inspection.
  • A.2 Data Processing: De-duplicating highly similar trajectories and repeated interaction patterns reduces task-instance over-representation and promotes behavioral diversity.The duplicates arise from near-identical prompts or execution histories.
  • A.3 Recovery Behavior Generation: Recovery data introduces predefined errors such as missed grasps, dropped objects, and wrong alignments, then continues interactive generation from perturbed states.Recovery generation starts from successful trajectories and manually applies sampled perturbations.
  • A.3 Recovery Behavior Generation: Randomized starting states reduce overfitting, only valid counterfactuals are retained, and unsuccessful or low-quality recovery rollouts undergo the same processing pipeline.For example, a pushing task cannot encounter a missing grasp.
  • A.4 Summary: 1,934 trajectories corresponding to 237 unique task prompts remain in Guava-Agent-4B’s fine-tuning dataset after filtering.The dataset includes 1,191 trajectories (62%) as successful executions and 743 trajectories (38%) as recovery trajectories.
  • B Tools: The semantic action space provides object interaction, geometric queries, Cartesian motion, rotation, gripper control, release, and home-pose recovery tools.Core tools include grasp, align, get_position, get_position_and_size, move, rotate, close_gripper, release, and home_pose.
  • B Tools: The semantic interface abstracts grasp poses, orientations, clearances, and motion sequences behind task-oriented skills handled by lower-level controllers.The alternative low-level interface requires explicit reasoning about these geometric and physical details.
  • B Tools: The low-level tool interface consistently performs worse than the high-level semantic interface, supporting semantic actions as a way to reduce VLM geometric and physical reasoning burdens.This comparison provides empirical evidence for semantic-level action spaces that emphasize task decomposition and decision making.

C Training Details · C.1 Hyperparameters · C.2 RL Training

Guava’s training combines specified supervised-fine-tuning and GRPO hyperparameters with a targeted reinforcement-learning strategy applied only to the two most challenging long-horizon manipulation tasks. This design aims to improve sequential embodied reasoning while keeping multimodal on-policy training computationally tractable.

  • C.1 Hyperparameters: 1 × 10−5 SFT learning rate, batch size 32, and 3 epochs define supervised fine-tuning, while GRPO uses 5 × 10−6, batch size 12, and 3 epochs.GRPO samples K = 4 rollouts per prompt and uses a KL penalty of β = 0.04.
  • C.1 Hyperparameters: Training runs on 8 NVIDIA H100 80GB GPUs with bfloat16 precision, FlashAttention-2, DeepSpeed ZeRO-3, and the ms-swift pipeline.These implementation choices support the SFT and GRPO stages described in the training setup.
  • C.2 RL Training: Unlike standard post-training recipes, GRPO is applied exclusively to the two most challenging long-horizon manipulation tasks rather than the entire supervised fine-tuning dataset.These tasks demand more multi-step reasoning, error recovery, and action planning than the remaining tasks.
  • C.2 RL Training: Long-horizon embodied RL is computationally expensive because each rollout repeatedly performs multimodal inference and environment interaction, with trajectory-collection costs rising rapidly with episode length.Applying RL across the full task suite would require substantially more training under this setup.
  • C.2 RL Training: Targeted RL focuses optimization on difficult long-horizon scenarios to improve sequential planning, recovery from intermediate failures, and long-range task completion within the training budget.The strategy is designed to remain computationally tractable while concentrating updates where they address the main performance bottlenecks.
  • C.2 RL Training: Concentrating RL updates on challenging tasks yields more favorable performance gains than uniformly allocating reinforcement-learning compute across all tasks.This empirical finding supports the targeted strategy’s training-effectiveness motivation.

D Additional Results

Additional evaluations show that Guava works consistently with several frontier and open-source models, while the small Qwen3.5-2B model performs poorly. Guava-Agent-4B also uses fewer tokens than GPT-5.4.

  • Additional Models: Guava’s harness works consistently with frontier models such as Gemini-3.1-Pro and Claude-Sonnet-4.6 in simulation.The evaluation also included several other frontier and open-source models.
  • Additional Models: Qwen3.5-2B performs poorly because its small size is associated with weak instruction-following and tool-calling abilities.It frequently hallucinates wrong reasoning and tool-use behavior.
  • Efficiency: Guava-Agent-4B uses fewer tokens than Guava’s harness with GPT-5.4.This comparison is reported in Figure 9.

E Additional Discussions

The discussion highlights persistent spatial-reasoning limitations in VLMs, alongside evidence that the agent tracks execution state and transfers from simulation to real-world scenes. It also cautions that simulation can introduce contact-dynamics failures, making real-world evaluation essential.

  • Spatial Understanding in VLMs: Spatially demanding tasks such as pushing and object arrangement have the lowest success rates across models in simulation and the real world.This reflects a persistent limitation in current VLMs’ spatial understanding.
  • Awareness of Execution State: After interrupted execution, the agent infers completed subtasks and avoids repeating actions, indicating an internal representation of task progress.For example, it proceeds to the next step when an object is already grasped.
  • Sim2Real for Embodied Agent: Separating high-level semantic planning from low-level perception and control enables transfer from simulation to real-world scenes without task-specific robot data.The abstraction leverages pretrained VLMs’ general visual understanding to generalize to new scenes.
  • Sim2Real for Embodied Agent: Imperfect contact dynamics in simulation can create failure modes that do not directly reflect real-world performance, so real-world evaluation remains essential.The discussion notes that simulation is not always an easier evaluation setting than the real world.

F Examples

The examples demonstrate Guava’s ability to plan over long horizons, recover from execution failures, and reason about spatial relationships during manipulation tasks.

  • F Examples: The example trajectories illustrate long-horizon planning, execution-failure recovery, and spatial reasoning in scenes.They include multi-object placement, retrying failed grasps, and pushing a basket in the correct direction.
  • F Examples: In a multi-object placement task, the reasoning orders placing the fork on the tray before placing the banana in the basket.The plan uses object locations, target locations, alignment, and object-specific clearance to sequence actions.
  • F Examples: For the pushing task, the reasoning approaches the basket from its left side, closes the gripper, lowers to contact, and moves rightward.The trajectory explicitly aligns to the basket’s left side and then applies a rightward move; the basket subsequently shifts right.
Loading 2606.18363v1…