PaperlayerView exact revision on arXiv

Source-linked AI summary

Continual Harness: Online Adaptation for Self-Improving Foundation Agents

Seth Karten, Joel Zhang, Tersoo Upaa, Ruirong Feng, Wenzhe Li, Chengshuai Shi, Chi Jin, Kiran Vodrahalli

arXiv:2605.09998v1cs.LGcs.AI

TL;DR

Embodied agents lack harnesses for long-horizon decision-making under partial observability. Continual Harness enables reset-free self-refinement from minimal interfaces and substantially lowers gameplay cost while supporting sustained progress through model-harness co-learning.

  • Problem

    Embodied agents lack coding-style harnesses that provide tools, memory, and planning for long-horizon partial-observability decision-making.

  • Method

    Continual Harness alternates acting with online, reset-free refinement of the agent’s prompt, sub-agents, skills, and memory using trajectory data.

  • Results

    Across Pokémon Red and Emerald, Continual Harness substantially reduces button-press cost and recovers a majority of the gap to a hand-engineered expert, with capability-dependent gains.

  • Takeaways & Limitations

    The framework supports continual model-harness co-learning, with an open-source model showing sustained in-game milestone progress without resetting the environment.

  • Takeaways & Limitations

    A capability floor limits bootstrapping: Flash-Lite underperforms the minimalist baseline, and evaluated open-source models are not yet capable enough to serve as both teacher and trainee.

Abstract

from arXiv · show

Coding harnesses such as Claude Code and OpenHands wrap foundation models with tools, memory, and planning, but no equivalent exists for embodied agents' long-horizon partial-observability decision-making. We first report our Gemini Plays Pokemon (GPP) experiments. With iterative human-in-the-loop harness refinement, GPP became the first AI system to complete Pokemon Blue, Yellow Legacy on hard mode, and Crystal without a lost battle. In the hardest stages, the agent itself began iterating on its strategy through long-context memory, surfacing emergent self-improvement signals alongside human-in-the-loop refinement. Continual Harness removes the human fully from this loop: a reset-free self-improving harness for embodied agents that formalizes and automates what we observed. Starting from only a minimal environment interface, the agent alternates between acting and refining its own prompt, sub-agents, skills, and memory, drawing on any past trajectory data. Prompt-optimization methods require episode resets; Continual Harness adapts online within a single run. On Pokemon Red and Emerald across frontier models, Continual Harness starting from scratch substantially reduces button-press cost relative to the minimalist baseline and recovers a majority of the gap to a hand-engineered expert harness, with capability-dependent gains, despite starting from the same raw interface with no curated knowledge, no hand-crafted tools, and no domain scaffolding. We then close the loop with the model itself: an online process-reward co-learning loop, in which an open-source agent's rollouts through the refining harness are relabeled by a frontier teacher and used to update the model, drives sustained in-game milestone progress on Pokemon Red without resetting the environment between training iterations.

1 Introduction

The paper introduces Continual Harness, a reset-free framework that lets embodied agents refine prompts, sub-agents, skills, and memory online from a minimal interface. It formalizes human-supervised GPP harness refinement, improves Pokémon gameplay efficiency across Gemini 3 variants, and enables continual model-harness co-learning.

  • Motivation: No equivalent to coding-agent harnesses exists for embodied agents facing long-horizon, partially observable decision-making.Existing harnesses wrap foundation models with tools, memory, and planning for autonomous coding agents.
  • Gemini Plays Pokémon: Human-supervised harness refinement enabled GPP to iteratively transform a screenshot-and-buttons interface into a multi-agent system for Pokémon gameplay.Later runs gave the model meta-tools to construct sub-agents and reusable scripts during play.
  • Continual Harness: Continual Harness automates this refinement through reset-free online in-context learning, alternating environment actions with refinement of system prompts, sub-agents, skills, and memory.It starts from frame observations, an ASCII map of the visible area, and button inputs, while extending to joint training of model weights.
  • Empirical results: Across Pokémon Red and Emerald and three Gemini 3 variants, Continual Harness substantially reduces button-press cost versus the minimalist baseline and recovers a majority of the gap to a hand-engineered expert harness.The reported gains require no curated knowledge, hand-crafted tools, or domain scaffolding, and scale with model capability on Emerald’s cost-vs-completion Pareto plane.
  • Model-harness co-learning: Online process-reward co-learning relabels low-reward rollout windows with a frontier teacher and updates an open-source model via soft SFT, producing sustained Pokémon Red milestone progress across training iterations.The refined harness shapes trajectories, whose new failure modes inform subsequent harness refinement; both loops use the same trajectory data.

2 Preliminaries

The paper models embodied-agent interaction through a minimal visual-and-text interface and defines a harness as the scaffolding that mediates foundation-model behavior. Harnesses comprise prompts, sub-agents, skills, and persistent memory, which meta-tools can refine during play, including in Continual Harness.

  • Embodied-agent interface: At each timestep, the agent receives a rendered frame and ASCII text map, then selects a button input from a fixed action set A.The text map describes visible tiles and nearby walkable positions and is derived from game state readable from the screen.
  • Harness structure: A harness H mediates behavior between foundation model M and the environment through a system prompt, sub-agents, skills, and persistent memory.These components provide instructions, specialized task modules, reusable routines, and accumulated facts, strategies, and observations.
  • Harness refinement: Fixed meta-tools such as define_agent, run_code, and process_memory let the agent edit prompt, sub-agent, skill, and memory components in place.New skills can also be authored during play, alongside pre-built primitives such as press_buttons and get_game_state.
  • Harness variants: The minimalist harness Hmin exposes only the environment interface with a generic prompt, whereas the hand-engineered harness Hexpert manually populates all components.A meta-harness instead gives the model tools to construct sub-agents, skills, and memory entries during play.
  • Continual Harness: Continual Harness starts from a minimal harness and adds an automated Refiner that rewrites p, G, K, and M from trajectory analysis during an evolving run.The running harness state is denoted HCH and changes after each refinement cycle.

3 Methodology

Continual Harness performs reset-free online refinement of the full harness state during a continuous episode, alternating agent actions with edits driven by recent trajectory failures. Its training extension jointly updates harness state within iterations and model weights across iterations using frontier-teacher relabeling and soft SFT.

  • Online harness refinement: Continual Harness rewrites the full harness state from trajectory history online, unlike prompt-optimization methods that rewrite only the prompt after complete-episode resets.The LLM Refiner edits harness state H from the most recent trajectory window during a single continuous episode.
  • Online harness refinement: Every F steps after W warm-up steps, the Refiner reads a recent trajectory window and emits edits to the prompt, sub-agents, skills, and memory without resetting.The updated state Ht+1 = Ht ⊕ ∆ enters the agent’s context on the next step.
  • Refinement passes: The Refiner detects navigation loops, tool-call failures, stalled objectives, and missed exploration, then rewrites prompts, manages sub-agents, codifies skills, and updates memory.It can delete unproductive sub-agent entries, repair executable code exceptions, and demote stale memory areas.
  • Continual accumulation: Refinement information accumulates monotonically, allowing quality to compound over longer episodes and exposing deep-episode failures that reset-based methods cannot reach by construction.Earlier failure signatures remain available to subsequent refinement passes, including for late-game battles, multi-step puzzles, and dialogue chains.
  • Online process-reward co-learning: In the training loop, πθk runs for K=256 steps in a live-refining harness; low-reward windows are relabeled by a frontier teacher, and soft SFT produces θk+1.The trajectory distribution depends on both model weights θ and harness state Ht, which update across and within iterations, respectively.

4 Experiments

Experiments evaluate Continual Harness on Pokémon Red and Emerald using cumulative button presses to milestones, alongside GPP results and reset-free model–harness co-learning. Continual Harness reduces interaction cost relative to Hmin, recovers much of the expert-harness gap, and supports continued in-loop refinement and milestone progress.

  • Experimental setup: Experiments use Pokémon Red and Emerald with standardized PokeAgent Challenge milestones, measuring cumulative button presses to milestone across Gemini 3 variants and at least three seeds.Harness conditions range from Hmin’s raw interface to Hexpert’s hand-designed scaffold and three Continual Harness variants.
  • In-loop refinement: Yellow Legacy refinement remains recurrent rather than converging, with most updates concentrated in a small subset of navigation and battle components.The battle_strategist_agent prompt cycles between growth and simplification across successive Elite Four revisions.
  • GPP results: GPP completed Pokémon Blue, Yellow Legacy on hard mode, and Crystal without a lost end-game battle, making it the first AI system to complete multiple Pokémon RPGs.The project combined human-designed and agent-iterated harness components over thousands of hours of gameplay.
  • Continual Harness performance: Continual Harness substantially reduces button-press cost relative to Hmin on both games and recovers a majority of the Hmin-to-expert efficiency gap without Hexpert’s curated scaffolding.The remaining gap concentrates in dialogue-heavy gym interiors and multi-turn battle strategy; on Red, bootstrap-updating is more efficient than from-scratch at every milestone.
  • Continual Harness performance: On Emerald Pro, from-scratch HCH reaches 100% of milestones at a $130 median versus Hmin’s 98% at $215, a ∼40% cost reduction with no completion loss.The two Pro bootstrap variants reach 96–100% of milestones at $110–$140, while Flash results show high variance.
  • Model–harness co-learning: Reset-free DAgger+PRM co-learning advances Pokémon Red gameplay across training iterations from both game starts and mid-game checkpoints, while cross-family Qwen3.5 serves as a negative control.The training loop uses an open-source model, a frontier teacher, and batch size = 1; warm-up stages alone produce no meaningful milestone advancement.
  • In-loop refinement: Every Continual Harness condition invokes navigation skills hundreds of times in 24 hours, repairing failures within the same episode; bootstrap-updating matches or outperforms bootstrap-frozen.From-scratch path-cost deficits fall from a near-half-cost penalty to single digits early and remain there, demonstrating reset-free in-loop improvement.

5 Related Work

Continual Harness differs from prior coding, game-agent, prompt-optimization, and reflective methods by editing the full harness state online during a single partially observed episode without resets. It also automates human-supervised harness refinement and combines live harness refinement with teacher-relabeled online co-learning.

  • Online harness adaptation: Continual Harness edits the full harness state in place from partial trajectory windows mid-episode, unlike methods that optimize components or reflect between episodes.The harness state comprises p, G, K, and M.
  • Embodied RPG agents: Coding and assistant-task harnesses stall on embodied RPGs without domain scaffolding, while Continual Harness targets this gap from a minimal interface.The comparison covers agentic harnesses for coding and assistant tasks.
  • Game-agent harnesses: LLM game agents either develop tooling during play or use hand-designed planners; Continual Harness automates the human-supervised refinement demonstrated by GPP across multiple Pokémon RPGs.GPP runs across Blue, Yellow Legacy, and Crystal, while the PokeAgent Challenge supplies a canonical benchmark and expert harness.
  • Related adaptation methods: Continual Harness complements reset-free reinforcement learning, in-context reinforcement learning, and recursive language-model methods by writing structured edits to the full harness state at depth 1.These related approaches address no-reset environments, implicit improvement, or structured multi-call reasoning over context.
  • Process supervision and co-learning: Its co-learning pipeline combines SFT and offline GRPO warm-up with online teacher relabeling of low-reward rollout windows inside a live-refining harness for soft SFT updates.The teacher is a frontier model, and the rollouts come from the model’s own interactions.

6 Discussion · Appendix Contents · A Pokémon Environment

Continual Harness self-refines online from a minimal interface without resets, but requires sufficient model capability and has not been shown to converge. The appendix describes Pokémon environments, interfaces, maps, milestones, costs, and state access used in evaluation.

  • 6 Discussion: Continual Harness builds and refines scaffolding from a minimal interface without resets, recovering most of the gap to a hand-engineered expert harness.The same live-refining harness supports inference-time refinement and online co-learning during training.
  • 6 Discussion: Flash-Lite stalls below 20% on Emerald, and every Continual Harness variant underperforms the minimalist baseline.This establishes a capability floor below which the refinement loop cannot bootstrap.
  • 6 Discussion: The co-learning loop shows sustained milestone progress but no established convergence point, while reset-free training loads each iteration from the preceding emulator state.A comparison with traditional reset-based batch accumulation remains open.
  • A Pokémon Environment: Experiments use Pokémon Red, Crystal, and Emerald, all single-player, turn-based role-playing games with long-horizon navigation, dialogue, battles, inventory, and gated objectives.The games span Game Boy, Game Boy Color, and Game Boy Advance releases.
  • A Pokémon Environment: The emulator provides native-resolution screen buffers, eight discrete buttons, and observations every 120 frames so animations and battle text resolve between decisions.Red and Crystal render at 160×144, while Emerald renders at 240×160; frames are upscaled 2× for the vision-language model.
  • A Pokémon Environment: An ASCII text map supplements each frame with nearby tile structure, including walkable tiles, walls, interactables, NPCs, ledges, and the player’s position and facing.The map is derived from emulator memory to address fine-grained spatial-reasoning difficulties over pixel grids.
  • A Pokémon Environment: The evaluation follows canonical milestone sequences and measures cumulative button presses rather than tool calls; Emerald has 31 milestones and Red has 18 through the 3rd gym.A single press_buttons invocation may emit multiple button presses, so cumulative presses are the primary cost metric.
  • A Pokémon Environment: The emulator exposes structured RAM state, but Hmin and HCH access it only through text-map derivation and general primitives rather than pre-wired domain tools.The expert harness additionally uses an A∗ pathfinder, battle type chart, and damage calculator.

B Gemini Plays Pokémon: Additional Evidence · B.1 Yellow Legacy Battle-Agent Evolution Checkpoints · B.2 Crystal Battle Advisor Evolution Checkpoints

The appendix adds evidence that Pokémon agents’ performance and battle strategies diverged across models and evolved through iterative prompting, memory, and structural changes. It also documents recurring failure modes and the exploration demands motivating continual refinement.

  • B Gemini Plays Pokémon: Additional Evidence: The appendix supplements the main-text Yellow Legacy results with additional evidence from Gemini Plays Pokémon experiments.It specifically extends the reported harness updates and battle-agent structural-complexity analysis.
  • B Gemini Plays Pokémon: Additional Evidence: Emerald’s route spans 31 milestones and requires substantial exploration, backtracking, branching navigation, and nonlinear objective dependencies.The current world record completes this segment in 1:00:57 minutes; the Red route similarly spans 18 milestones and has a 43:04-minute world record.
  • B Gemini Plays Pokémon: Additional Evidence: Gemini 2.5 Pro became trapped by bad assumptions and spent 16,403 turns before obtaining the Fog Badge.This contrasted with Gemini 3 Pro’s handling of the Olivine Lighthouse pits.
  • B Gemini Plays Pokémon: Additional Evidence: GPP exposed unverified assumptions, brittle tool calls with missing parameters, and limited parallel goal pursuit as recurring human-repaired failure modes.These failures directly motivated Continual Harness’s mid-episode refinement targets.
  • B.1 Yellow Legacy Battle-Agent Evolution Checkpoints: Four graph metrics were tracked across 14 Yellow Legacy structural checkpoints during the Elite Four window.Mermaid-rendered decision graphs cover four canonical checkpoints and the remaining ten, with a shared renderer and palette isolating prompt-driven differences.
  • B.1 Yellow Legacy Battle-Agent Evolution Checkpoints: Yellow Legacy Elite Four retries used increasingly structured battle prompts and persistent written memory, yielding a text-encoded decision process across the gauntlet.The evidence is summarized as lifetime attempt totals accompanied by prompt and memory evolution.
  • B.2 Crystal Battle Advisor Evolution Checkpoints: Crystal’s Battle Tower attempt produced 10 structural checkpoints for the battle_advisor agent across turns 30k–36k.Figures divide these into six checkpoints from turns 30k–33k and four from turns 33k–36k.

B.3 Case Study: The Power Plant Route Loop

A Pokémon Yellow Legacy run exposed a 1,003-turn, 3.5-hour stagnation loop caused by a tool-schema mismatch, incorrect game knowledge, and failure to recognize execution feedback. The loop ended only after the agent cycled through all Fly destinations, corrected its assumption, and manually walked to Route 10.

  • Failure Loop: 1,003 turns over approximately 3.5 hours, the agent remained stuck near Cerulean City while attempting to reach the Power Plant.The incident documented runtime tool-generation failures, schema mismatch during prolonged stalls, and context-horizon limits for error recovery.
  • Tool Generation: The agent deleted get_next_pokemon_press, generated fly_menu_navigator with autopress_buttons enabled, and stored instructions to trust the replacement tool.This intervention was intended to bypass repeated overshooting in the party menu.
  • Schema Mismatch: The generated call required buttons_to_press=["tool"], but the agent supplied ["Down"], then repeated the same incorrect payload 842 times.The agent failed to detect that its logical action did not match the meta-harness execution schema.
  • Internal State vs. Execution Reality: The agent’s reasoning suggested active progress evaluation despite a largely static environment, while assuming the Power Plant was a valid Generation 1 Fly destination.The missing game knowledge and schema error reinforced one another, preventing effective error recovery.
  • Loop Termination: After 1,003 consecutive turns, the agent cycled through all Kanto Fly destinations, recognized the Power Plant was not directly reachable, stopped the custom tool, and walked to Route 10.The recovery followed the cursor returning to its starting location, Lavender Town.
  • Failure Modes: Beyond 500 turns in a stall, tool creation ceased and repeated execution continued, while persistent-memory rules remained vulnerable to schema mismatch and feedback blindness.Tool generation primarily occurred within the first 50-200 turns of a new bottleneck.

C Harness Ablations … C.2.1 Red Bootstrap-Updating Regression

The appendices attribute the Hmin-to-HCH progression gap and explain how reset-free refinement repairs skills, manages specialized sub-agents, reuses memory, and transfers harnesses across episodes. They also identify a Red bootstrap-updating regression caused by replacing inherited sub-agents with unrepaired newly authored ones.

  • C Harness Ablations: The appendix provides per-component attribution for the Hmin-to-HCH progression gap and reset-free bootstrap-transfer results.It supplies the full attribution behind the progression gap discussed in Section 4.6.
  • C.1 Mechanism Attribution: The appendix attributes the Hmin-to-HCH progression gap through component-level analyses of pathfinding, skill debugging, sub-agent handoffs, and memory reuse.These analyses form the mechanism-attribution portion of the harness ablations.
  • C.1.1 Pathfinding skills: Navigation efficiency is measured as issued button presses divided by a BFS-optimal path length over tiles observed across runs.Dialogue and battle presses are excluded, and ordering can invert inside gyms where dialogue and puzzle state dominate navigation.
  • C.1.2 Skill debugging: Reset-free refinement produces dramatic repairs on relied-upon skills within the same episode as the failure.Repairs are evaluated using rolling success rates before and after skills_updated events, alongside a create-to-succeed funnel.
  • C.1.3 Sub-agent handoffs: Sub-agent handoffs reduce per-step cost through narrow specialized contexts while allowing the orchestrator to resume its prior objective.Figure 17 tracks role-specific tokens, handoff counts, and task-type handoff success.
  • C.1.4 Memory reuse: The harness exposes the full memory catalog for free, but measures reuse through explicit entry reads, memorized-skill invocations, or entry-ID citations.The appendix identifies an explicit reuse prior as a natural next step.
  • C.2 Reset-Free Bootstrap Transfer: Bootstrap succeeds by leveraging inherited skills, sub-agents, and memory, while regressions appear when the agent stops using the inherited harness.Bootstrap runs preload the final artifacts from a successful from-scratch run and classify actual invocation origin across stores.
  • C.2.1 Red Bootstrap-Updating Regression: Around step 213, newly authored sub-agents overtake inherited ones in Red and regress because they lack the repair cycle observed for from-scratch skills.Proposed follow-ups include a reuse prior on sub-agent selection or deleting newly authored sub-agents covered by inherited ones.

D Training Setup and Results

The appendix details the open-source transfer pipeline’s training setup and results, covering SFT, offline GRPO, and online co-learning. It also presents the evaluation matrix across Gemma-4 sizes and co-learning training-curve diagnostics.

  • Training Setup and Results: The appendix provides full training details for the open-source transfer pipeline.The pipeline is described in Section 3.3.
  • Training Setup and Results: It reports hyperparameters for SFT, offline GRPO, and online co-learning.
  • Training Setup and Results: It includes the full evaluation matrix across Gemma-4 sizes and training-curve diagnostics for the co-learning stages.

D.1 Training Hyperparameters · D.2 Gemma-4 Full Eval Matrix · D.3 Training Curves and Reward Decomposition

The appendix specifies supervised fine-tuning, offline GRPO, and online co-learning configurations, then reports Gemma-4 evaluation failures across sizes and the reward structure used for online training. Smaller models collapse under the full harness prompt, while 31B SFT provides the viable Red initialization for online co-learning.

  • D.1 Training Hyperparameters: SFT fine-tunes Gemma-4 E2B, E4B, 26B MoE, and 31B dense variants with LoRA, bf16, 8K-token context, and one pass over teacher trajectories.LoRA uses r=256 and α=256; training uses learning rate 2 × 10−5, 3% linear warmup, and cosine decay.
  • D.1 Training Hyperparameters: Offline GRPO samples G=4 completions per teacher-visited state and scores them with a Gemini-3-flash-preview oracle for action correctness and format compliance.The composite reward weights action correctness at 0.6 and format compliance at 0.4; updates use group-normalized advantages, learning rate 1 × 10−6, and KL coefficient β=0.04.
  • D.1 Training Hyperparameters: Online co-learning performs K=256-step DAgger rollouts through the evolving Continual Harness on Pokémon Red.The harness evolves memory, skills, sub-agents, and prompts during each rollout.
  • D.3 Training Curves and Reward Decomposition: The online process reward combines trajectory progress (0.4), action correctness (0.3), reasoning quality (0.2), and format compliance (0.1).A Gemini-3-flash-preview pairwise process reward model scores transitions over a sliding window, with low-reward windows relabeled by a Gemini-based teacher process described in the passage.
  • D.2 Gemma-4 Full Eval Matrix: Smaller Gemma-4 models reach low SFT training loss but collapse to tool_format=0 on the real harness prompt.The passage attributes this to interaction between SFT signal strength and the 8K context needed for full state plus reasoning.
  • D.2 Gemma-4 Full Eval Matrix: The 31B SFT Emerald model underperforms the 26B SFT Emerald model on most metrics in the reported runs.The full Emerald matrix identifies the offline-GRPO warm-up checkpoint as emitting degenerate completions on this prompt set.
  • D.2 Gemma-4 Full Eval Matrix: The 31B SFT Red checkpoint is viable and initializes online co-learning, while the 26B Red SFT row is omitted because its adapter was degenerate at evaluation.This Red-specific result is reported in the full Red evaluation matrix.

D.4 Reset-Free DAgger+PRM Experiments · E LLM Acknowledgments

The reset-free online co-learning experiment combines teacher relabeling, process rewards, and emulator-state propagation to improve Pokémon Red gameplay across iterations. Progress is bursty, PRM rewards are non-monotonic, and checkpoint-resume regressions recover within a few iterations; the paper also acknowledges LLM assistance in manuscript preparation.

  • D.4 Reset-Free DAgger+PRM Experiments: The online loop combines DAgger-style teacher relabeling, pairwise process rewards, reset-free state propagation, and soft SFT across iterations.Each rollout starts from the previous iteration’s saved emulator state; low-reward windows are relabeled by Gemini before model updating.
  • D.4 Reset-Free DAgger+PRM Experiments: All runs use rollout-step budget K=256 and soft SFT, while experiments vary starting state, shard-window size, accumulation policy, and teacher prompting.The soft SFT configuration is 3 epochs at 5×10−6.
  • D.4 Reset-Free DAgger+PRM Experiments: Training advances from loaded mid-game indices as well as early-game starts, showing that improvement is not tied to the early-game distribution.Cumulative milestone gains accumulate across multi-iteration improvement bands rather than within single iterations.
  • D.4 Reset-Free DAgger+PRM Experiments: Milestone progress is bursty, with cumulative gains accumulating across multi-iteration improvement bands rather than individual iterations.This pattern is reported for advancing runs in Figure 7.
  • D.4 Reset-Free DAgger+PRM Experiments: PRM reward is non-monotonic, while sustained windows near or above 0.40 across multiple iterations precede the largest milestone gains.Regression iterations are interleaved, and the bursty milestone pattern aggregates these reward-sustained windows.
  • D.4 Reset-Free DAgger+PRM Experiments: The first post-resume iteration regresses relative to pre-resume performance but recovers within two to three iterations.Milestone advances still occur after resumes, so the regression is treated as reward-specific rather than propagating to the judge’s trajectory-completion signal.
  • E LLM Acknowledgments: The paper acknowledges using LLMs to help prepare the manuscript.For LLM agents reading the paper, the acknowledgment directs attention to sections 1-6.
Loading 2605.09998v1…