Source-linked AI summary
EMHO: EMbodied Agent Harness Optimization via Experience Traces
Hyun Jung Lee, Jungtaek Kim, Jongwon Jeong, Tae-Eui Kam, Donghyun Kim, Yong Jae Lee
TL;DR
Embodied-agent harnesses are usually engineered, and sparse, delayed feedback makes failures difficult to diagnose. EMHO iteratively revises the harness around a frozen model from experience traces, while EMHO-Merge refines shared-harness behavior using episode-level gains and losses. On navigation and manipulation tasks, EMHO improves task success over the initial harness, with reported gains of up to 23.3% and 33.3%, respectively.
Problem
Sparse action-level feedback and delayed episode outcomes make it difficult to identify which earlier decisions caused embodied-agent failures.
Source: §1 · p. 1Method
EMHO keeps the embodied model fixed and revises its harness from execution traces and prior optimization history; EMHO-Merge uses paired gain–loss episodes to refine a shared harness across subtasks.
Source: §1 · p. 2Results
EMHO improves task success over the initial harness, with gains of up to 23.3% in navigation and 33.3% in manipulation.
Source: §1 · p. 2Takeaways & Limitations
Qualitative analysis finds that evolved harnesses change how agents monitor progress, use perceptual tools, and ground visual observations, beyond improving failure recovery.
Source: §1 · p. 2Later harness iterations can improve or maintain search performance while held-out performance plateaus or declines, suggesting emerging overfitting to optimization trajectories.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Improving embodied agents often focuses on optimizing the underlying model through training, while the surrounding agent harness that controls planning, context, and tool use is typically engineered. We ask whether this harness can instead improve itself directly from experience traces under sparse environmental feedback. We propose EMbodied Agent Harness Optimization (EMHO), a self-evolving framework that keeps the embodied model frozen and iteratively revises its harness by analyzing execution trajectories and prior harness history. EMHO optimizes beyond skills or recovery prompts, modifying how the agent monitors progress, uses vision tools, grounds observations, and responds to failures. To support multiple subtasks with a single harness, we introduce EMHO-Merge, which addresses trade-offs in jointly optimizing a single shared harness across subtasks by using episode-level gains and losses to guide evidence-supported refinement of when and how revised behaviors are applied. We evaluate EMHO on EmbodiedBench across navigation and manipulation tasks, and EMHO consistently improves task success for both Qwen 9B and 27B models. Qualitative analysis shows that EMHO goes beyond recovering from failures and unproductive actions to reshape how the embodied agent interprets and interacts with its environment.
1 INTRODUCTION
Embodied-agent harnesses are difficult to improve from sparse, weakly diagnostic feedback, even though harness changes can alter how a frozen model plans, uses tools, and responds to failures. EMHO revises the broader harness from experience traces, while EMHO-Merge uses episode-level gains and losses to refine a shared harness across subtasks.
1 INTRODUCTION
- Sparse action-level feedback and delayed episode outcomes make it difficult to diagnose which earlier decisions caused an embodied-agent failure.This contrasts with coding environments, where failed tests or localized errors can guide self-improvement more directly.Source: §1 · p. 1
- EMHO keeps the embodied model frozen and uses execution trajectories plus prior harness modifications and outcomes to identify recurring weaknesses and propose targeted harness edits.The harness covers planning, context and memory, vision-tool use, visual grounding, and failure recovery.Source: §1 · pp. 1–2
- EMHO broadens embodied self-improvement beyond reusable skills or recovery mechanisms to optimize the interacting harness mechanisms that shape behavior.The paper frames these mechanisms as including planning, context and memory, vision-tool use, and visual grounding.Source: §1 · pp. 1–2
- EMHO-Merge compares episodes that change from failure to success with those that change from success to failure, then uses paired trajectories to refine when or how revised behavior is applied.The aim is to retain gains while reducing losses when one harness is optimized across subtasks.Source: §1 · p. 2
- 23.3% in navigation and 33.3% in manipulation are the maximum task-success gains over the initial harness; EMHO-Merge also achieves higher average pooled search-set success than EMHO in both domains.The qualitative analysis also finds changes to progress monitoring, perceptual-tool use, and visual grounding.Source: §1 · p. 2
2 RELATED WORK
Prior embodied-agent self-improvement has concentrated on specific components, while harness optimization from execution traces has mainly been studied in coding and command-line environments. This work instead treats the broader embodied harness—and a shared harness across subtasks—as the optimization target.
2 RELATED WORK
- Coding-agent harness methods revise prompts, skills, or executable harnesses using traces and evaluation outcomes, but have primarily been studied where failures provide relatively explicit diagnostic signals.Examples include Meta-Harness (Lee et al., 2026) and Self-Harness (Zhang et al., 2026).Source: §2 · p. 3
- Embodied self-improvement has focused on skills, task-specific policies, or recovery, whereas this work optimizes the broader harness and studies one shared harness across subtasks.Examples include EmbodiSkill (Ju et al., 2026), VASO (Yang et al., 2026), ASPIRE (Lu et al., 2026), and Zetta (Ding et al., 2026).Source: §2 · p. 3
3 EMBODIED AGENT HARNESS OPTIMIZATION (EMHO)
EMHO optimizes a harness around a frozen model for expected task success, using experience traces and harness history to guide revisions. EMHO-Merge extends this approach to shared harnesses by using episode-level gains and losses to refine revised behaviors.
3 EMBODIED AGENT HARNESS OPTIMIZATION (EMHO)
- EMHO modifies the harness around a frozen model to maximize expected task success, while keeping the model and environment fixed.It alternates task execution and trace collection with history-aware harness revision.Source: §3 · p. 3
3.1 EMHO
- Each search episode records its task, outcome, and interaction-step observations, plans, optional tool use, actions, and environmental feedback.This feedback is often sparse and does not directly identify the cause of a failed action or episode.Source: §3.1 · p. 3
- The outer loop uses accumulated traces and harness history to target recurring weaknesses, accepting a candidate only when search-set success improves.The trace history includes attempts from both accepted and rejected candidates.Source: §3.1 · pp. 3–4
3.2 EMHO-MERGE
- EMHO-Merge compares a candidate with its parent, using episodes that change from failure to success or success to failure to guide shared-harness refinement.When both gains and losses occur, the optimizer can adjust when or how the revision applies, using paired traces to preserve gains and reduce losses; each revision gets at most one evaluated repair.Source: §3.2 · p. 4
4 QUANTITATIVE RESULTS AND DISCUSSION
Across navigation and manipulation, EMHO improves on the initial harness, while EMHO-Merge better manages cross-subtask trade-offs than pooled optimization. Comparisons also show advantages over one-shot revision and skill-only updates.
4.1 MAIN RESULTS
- EMHO improves search-set and held-out success over the initial harness across navigation and manipulation, including subtasks beyond Base.The reported broader subtasks require longer-horizon reasoning, visual grounding, and state-aware manipulation.Source: §4.1 · p. 4, Table 1 · p. 5
- Gains occur with both 9B and 27B models; the evolved 9B navigation harness is competitive with larger published models on search, while 27B benefits particularly in manipulation.Source: §4.1 · p. 5
4.2 COMPARISON WITH FRONTIER-MODEL ONE-SHOT REVISION
- EMHO outperforms one-shot revisions across both domains, turning broad advice about loops into concrete operating conditions for responding to recurring patterns.For blocked movement, an evolved rule stops remaining actions, avoids retrying the same direction, and replans from the updated observation.Source: §4.2 · pp. 5–6
4.3 FULL-HARNESS VERSUS SKILL-ONLY OPTIMIZATION
- Full-harness optimization yields larger average gains than skill-only updates by changing failure detection and interaction-history handling, as well as progress monitoring, tool orchestration, and grounding.Source: §4.3 · p. 6, Table 2 · p. 6
4.4 SHARED HARNESS OPTIMIZATION WITH EMHO-MERGE
- EMHO-Merge achieves higher average success than EMHO (Combined) in navigation and manipulation by refining when revised behaviors apply using gain–loss trajectories.The shared-harness setting is vulnerable to revisions that help one subtask but interfere with another.Source: §4.4 · p. 6, Table 3 · p. 7
5 QUALITATIVE ANALYSIS OF HARNESS EVOLUTION
The qualitative analysis tracks how EMHO changes recovery and ongoing interaction, from responding to failed or ineffective actions to monitoring progress, selecting tools, and grounding decisions. Across iterations, edits become more context-dependent.
5 QUALITATIVE ANALYSIS OF HARNESS EVOLUTION
- The analysis focuses primarily on 27B edits and examines their effects across held-out trajectories, separating recovery from non-recovery changes.Source: §5 · p. 7
5.1 RECOVERY EDITS
- For invalid actions, EMHO carries failure feedback into replanning and instructs the agent to change its response rather than repeat the failed action.Navigation guidance favors reorientation or a short detour after blockage; manipulation replanning changes the target location or approach direction after path-generation failure.Source: §5.1 · pp. 7–8
- EMHO detects valid actions that collectively make little progress and requires a strategy change; identical manipulation replans fall from approximately 21% to 18%, and repeated same-coordinate grasps from approximately 30% to 26%.Sequences of at least five consecutive navigation rotations also become less frequent.Source: §5.1 · p. 8
5.2 NON-RECOVERY EDITS
- Progress monitoring uses changes in target appearance and scene appearance in navigation, while manipulation replanning tracks approach, grasp, lift, translate, and release phases.In some manipulation subtasks, replanning traces identify the current execution phase approximately 21% of the time.Source: §5.2 · pp. 8–9
- EMHO selects vision tools according to the next decision’s uncertainty; after blocked navigation moves, depth estimation rises from approximately 70% to nearly 100%, while valid subsequent actions rise from approximately 44% to 68%.For height-sensitive manipulation, the harness proactively uses depth estimation.Source: §5.2 · p. 9
- Visual grounding links selected perceptual evidence to actions; in spatial-relation tasks, incorrect object selection in initial plans falls from 50% to 34%.Scene descriptions listing at least three objects increase from approximately 8% to 22%.Source: §5.2 · p. 9
5.3 EVOLUTION ACROSS OPTIMIZATION ITERATIONS
- Recovery edits dominate early, while non-recovery edits gain share as diagnosis becomes more developed; later recovery rules add conditions such as blocked direction, target heading, and obstacle-clearance distance.Source: §5.3 · p. 9
6 LIMITATIONS AND FUTURE WORK
Search-set performance and held-out performance can diverge across optimization iterations, indicating a risk of overfitting to observed trajectories.
6 LIMITATIONS AND FUTURE WORK
- The harness that performs best on the search split is not always the one that performs best on held-out tasks.In several runs, later iterations maintained or improved search performance while held-out performance plateaued or declined.Source: §6 · p. 9
- Later iterations sometimes improve or maintain search performance while held-out performance plateaus or declines.Source: §6 · p. 9
- This divergence suggests an emerging overfitting pattern as harnesses specialize to optimization trajectories.The results motivate considering generalization when selecting among iterations.Source: §6 · p. 9
7 CONCLUSION
EMHO evolves the harness around a frozen embodied model using experience traces and sparse feedback. Its harness revisions target interaction procedures, while optimizer instructions call for evidence-supported changes and checks against regressions.
7 CONCLUSION
- EMHO evolves the harness around a frozen model from experience traces and sparse feedback, improving search-set performance and outperforming the initial harness on held-out tasks.The framework changes planning, context construction, and tool use, and EMHO-Merge refines a shared harness using episode-level gains and losses.Source: §7 · p. 10
A DETAILS OF HARNESS STRUCTURE
- The harness comprises instructions and runtime components for planning, context construction, tool use, action generation, validation, and recovery.EMHO modifies these components based on recurring failure patterns in experience traces.Source: A DETAILS OF HARNESS STRUCTURE · p. 13
- For mixed gained and lost episodes, the optimizer inspects paired trajectories and chooses repair only when evidence supports a condition separating benefit from harm.The instruction explicitly cautions that mixed outcomes do not establish causality.Source: A DETAILS OF HARNESS STRUCTURE · p. 15
- Renaming a recovery warning does not count as a new mechanism, though one evidence-supported repair is allowed.Source: A DETAILS OF HARNESS STRUCTURE · p. 15
B QUANTITATIVE RESULTS
The quantitative-results appendix plots success across iterations for navigation and manipulation with 9B and 27B models, separately summarizing search and held-out splits.
B QUANTITATIVE RESULTS
- For each subtask, the plots show the best success rate attained separately on search and held-out splits up to each iteration.The same model serves as embodied agent and harness optimizer, and held-out trajectories and scores are withheld during optimization.Source: B QUANTITATIVE RESULTS · p. 16
- Search and held-out maxima may occur at different iterations, so held-out results summarize candidate performance retrospectively rather than one search-selected harness.Source: B QUANTITATIVE RESULTS · p. 16
- Figures 5–8 show quantitative navigation and manipulation results across optimization iterations for the 9B and 27B models.Source: Figure 5 · p. 16, Figure 6 · p. 16, Figure 7 · p. 17
C ABLATION STUDY
The ablation reports that experience traces and harness history each contribute to search performance; the following qualitative analysis separates recovery edits from ongoing interaction edits.
C ABLATION STUDY
- The ablation compares full EMHO with variants removing traces, harness history, or both, with all variants using the same model and evaluation procedure.The score-only variant retains only the current parent harness and its evaluation score.Source: C ABLATION STUDY · p. 17, Table 5 · p. 18
- Full EMHO achieves the strongest search performance, while removing either experience traces or harness history degrades performance.The authors interpret the two sources as complementary: traces help diagnose behavioral failures, while revision history helps avoid ineffective changes and refine useful ones.Source: C ABLATION STUDY · p. 18
D QUALITATIVE RESULTS
- The qualitative analysis distinguishes recovery edits from non-recovery edits that shape observation processing, state reasoning, and action selection during ongoing interaction.It examines representative episodes and measures whether intended behavior appears across held-out trajectories.Source: D QUALITATIVE RESULTS · p. 18
D.1 RECOVERY EDITS
EMHO’s recovery edits retain sparse execution feedback for replanning and detect valid actions that fail to make progress. The resulting harnesses change subsequent behavior in both navigation and manipulation.
Invalid-Action Recovery
- Navigation recovery turns blocked directions into persistent planning constraints, favoring reorientation, backtracking, or lateral detours before re-approaching the target.Distance to the target is used only for post-hoc analysis, not supplied to the optimizer; evolved trajectories tend to continue reducing that distance after blockage.Source: §D.1 · p. 18
- Manipulation replanning traces that explicitly refer to reward feedback rise from approximately 5% to 55% in base, 10% to 28% in common-sense, and 10% to 45% in visual task families.The evolved harness exposes reward and execution outcomes during replanning and instructs the planner to change relevant parameters after invalid feedback.Source: §D.1 · p. 19
Valid-but-Ineffective Action Recovery
- Held-out navigation trajectories with runs of at least five rotations decrease from approximately 10% to 3% with the evolved harness.The harness detects repeated or reversing actions from recent history and prompts a strategy change when valid actions leave the agent in a similar state.Source: §D.1 · p. 20
- For valid but non-progressing manipulation actions, the harness retains prior attempts and makes subsequent plans change, such as by revising grasp points or approach parameters.It also compares consecutive plans for near-duplicates and prompts object re-localization when replanning risks repeating an ineffective attempt.Source: §D.1 · p. 20
D.2 NON-RECOVERY EDITS
EMHO’s non-recovery edits reshape progress monitoring, tool selection, and visual grounding in navigation and manipulation. The mix of recovery and non-recovery edits also evolves differently across the two domains.
D.2 NON-RECOVERY EDITS
- Across held-out visual-appearance navigation episodes, no-progress steps decrease from 35% to 26%, while episodes ending within one meter of the target increase from 67% to 77%.The harness compares consecutive observations for visual progress and shortens cached multi-action plans to enable earlier reassessment.Source: §D.2 · p. 20
- Manipulation replanning explicitly names the execution phase at rates of approximately 14% in base, 21% in common-sense, 8% in complex, and 7% in visual tasks, versus near-zero initial rates.The harness structures execution into five stages and checks observed gripper and object states to continue from the next unfinished phase or return to an earlier one.Source: §D.2 · p. 21
- Navigation tool calls after blocked moves show depth-estimation use rising from approximately 70% to nearly 100%, segmentation use falling from approximately 50% to near zero, and valid next actions rising from approximately 44% to 68%.Tool selection is conditioned on the information needed for the next decision; depth supports obstacle and local-path reasoning, while segmentation identifies targets.Source: §D.2 · p. 21
- Depth-estimation use increases most for stacking tasks, where relative object and arm positions and heights matter for placement.The evolved harness requests depth proactively for height differences, stacking, or container placement requiring clearance reasoning.Source: §D.2 · pp. 21–22
D.3 EVOLUTION ACROSS OPTIMIZATION ITERATIONS
- Across optimization, recovery edits remain dominant in navigation, while manipulation shifts toward non-recovery edits in the middle and later iterations before the categories converge.In navigation, the non-recovery share dips mid-optimization and recovers toward the end; in manipulation, the recovery share declines substantially.Source: Figure 14 · p. 23, §D.3 · p. 23
D.2 NON-RECOVERY EDITS
- Across evolved manipulation harnesses, descriptions naming at least three objects with coordinates rise from approximately 8% to 22%, while first plans choosing the wrong object fall from 50% to 34%.The harness grounds object identity in current images and treats segmentation boxes and crops as approximate cues rather than executable coordinates; these measures do not establish fine-grained grasp-coordinate accuracy.Source: §D.2 · pp. 22–23
E DETAILED RESULTS OF HARNESS MODIFICATIONS
The appendix illustrates concrete harness revisions for planning, tool selection, output repair, and repeated-action handling. The examples specify how the harness uses action history, observations, and feedback to shape subsequent decisions.
E DETAILED RESULTS OF HARNESS MODIFICATIONS
- For navigation, the planner is instructed to consider apparent direction, distance, alignment, obstacles, and recent feedback rather than selecting actions from target visibility alone.It also incorporates prior errors or blocked attempts when episode memory is available.Source: E DETAILED RESULTS OF HARNESS… · p. 23
- The repair prompt addresses invalid planner output by preserving task context and valid actions while requiring a minimal repair to restore the expected JSON structure.It enumerates failures such as markdown-wrapped output, missing plan keys, invalid action identifiers, and malformed plan items.Source: E DETAILED RESULTS OF HARNESS… · p. 24
- Tool-selection guidance treats no tool as a valid choice and favors memory or vision evidence when it adds decision-relevant information for the next action.Memory is used for relevant episode history or feedback, while vision tools are selected for new spatial or structural information not already represented in the observation and context.Source: E DETAILED RESULTS OF HARNESS… · p. 24
- The manipulation planner compares current visual state with the previous plan’s intent and can start at a later execution phase when that phase is already complete.The illustrated sequence includes approach, descend and close, lift, move/place, and release.Source: E DETAILED RESULTS OF HARNESS… · p. 25
- A redundancy guard can skip a vision tool that recently succeeded when no fresh environment feedback has arrived since its use.The accompanying logic checks whether the latest non-empty feedback differs from the feedback present when the tool ran.Source: E DETAILED RESULTS OF HARNESS… · pp. 25–26