PaperlayerView exact revision on arXiv

Source-linked AI summary

EnvHarness: Awakening Static Worlds for Agent Learning

Chengsong Huang, Zifeng Wang, Rujun Han, Jun Yan, Yanfei Chen, Zoey CuiZhu, Ke Jiang, Peng Xia, Han Yu, Yufan Zhuang, Yifei Ming, Jiaqi Pan, Bhavana Dalvi Mishra, Jiaxin Huang, Burak Gokturk, Tomas Pfister, Chen-Yu Lee

arXiv:2608.19880v1cs.AIcs.CLcs.LG

TL;DR

Static, hand-built environments are rigid and domain-specific generation pipelines are costly to verify. EnvHarness wraps them with programmable components, while EnvRigger customizes environments from policy rollouts. Across five benchmarks, customized environments improved held-out performance by up to 9.0 points with 9.8% fewer interaction steps.

  • Problem

    Hand-built environments are rigid, while automated generation remains domain-specific and difficult to verify reliably across agent-learning settings.

  • Method

    EnvHarness wraps static environments with modular components, and EnvRigger diagnoses policy weaknesses from rollouts to synthesize and validate targeted customizations.

  • Results

    Across five benchmarks in four domains, EnvHarness-customized environments consistently outperformed original environments, with up to 9.0-point improvement on held-out tasks and 9.8% fewer interaction steps.

  • Takeaways & Limitations

    EnvHarness provides a cross-domain way to customize environments while preserving their original tasks and trusted verifiers.

  • Takeaways & Limitations

    EnvHarness requires resettable environments with a gym-style reset/step interface, excluding non-resettable live services and physical settings.

Abstract

from arXiv · show

LLM agents learn by interacting with environments, yet these environments are hand-built and static: blind to an agent's weaknesses, and quickly left behind as it improves. While recent environment generation methods attempt to address this, they require domain-specific pipelines, rely on expensive or unreliable verifiers, and still produce static environments. To alleviate the engineering burden of rebuilding environments from scratch, we propose Environment Harness (EnvHarness), a programmable layer of plug-in components that wraps a static environment to reshape its behavior without modifying the underlying logic. Operating through standard interfaces, EnvHarness applies across diverse domains while ensuring every reshaped environment retains its original verifier. To automate this process, we introduce EnvRigger, which treats the target policy as a black box, observing its execution trajectories to synthesize EnvHarness components targeting diagnosed flaws, and validating them via fresh rollouts. Across five benchmarks in four domains, EnvHarness outperforms both original environments and domain-specific environment generation pipelines, achieving up to a 9.0-point improvement on held-out instances with 9.8% fewer execution steps. Furthermore, EnvHarness provides a superior optimization signal for reinforcement learning, enabling continuous, targeted co-evolution of the policy and its environment.

1. Introduction

EnvHarness addresses the cost and rigidity of manually built or domain-specific generated environments by wrapping static environments with programmable plug-ins that customize behavior without changing their logic or verifiers. EnvRigger automates policy- and task-conditioned customization by diagnosing black-box execution failures and synthesizing validated EnvHarness components.

  • Motivation: EnvHarness addresses the substantial human effort required to hardcode interactive behavior and verifiers in learning environments.Automated generation improves scalability but remains inherently domain-specific across web navigation, programming, and tool use.
  • Environment Harness: EnvHarness transforms an existing static environment into a dynamically customized one without modifying its underlying environment logic.Its plug-in layer applies through standard interfaces while preserving the original environment’s tasks and verifiers.
  • EnvRigger: EnvRigger treats the target policy as a black box, diagnoses vulnerabilities from successful and failed trajectories, and synthesizes candidate EnvHarness components for validation.The configuration is tailored to each target policy and task rather than fixed across environments.
  • Evaluation: The evaluation spans five benchmarks across embodied tasks, web browsing, software engineering, and office work, covering skill-based learning and reinforcement learning.The benchmarks include ALFWorld (Shridhar et al., 2020), WebArena (Zhou et al., 2024), SWE-bench Verified, OfficeQA, and SpreadsheetBench (Ma et al., 2024).

2. EnvHarness

EnvHarness wraps static environments with programmable, modular interface-level transformations that customize behavior without changing the underlying simulator or verifier. Its Stage, Contract, and Chain components support composable changes to initialization, interaction, and environment structure.

  • Definition: EnvHarness is a programmable layer that customizes static environments through standard-interface information flow while leaving the underlying environment untouched.It applies the agent-harness idea to environments: Customized Env = Static Env + EnvHarness.
  • Definition: Interface-level interventions preserve the original ground-truth evaluation logic, so the unchanged verifier can still score each episode.Components can customize initial states, exposed action or observation spaces, and transition mechanics without modifying the simulator backend.
  • Components: Stage changes initial states, Contract rewrites actions, transitions, or observations, and Chain extends environments through a compositional logic.Stages can introduce obstacles or shorten horizons; Contracts can enforce preconditions, mask observations, or provide structured feedback; Chains can concatenate, interleave, or branch environments.
  • Composition: EnvHarness components compose through their shared interface, but nesting order is noncommutative and determines initialization and interaction constraints.A Stage, Contract, and Chain can jointly hide a mug, truncate observations, and append a follow-up task in one composite environment.

3. EnvHarness for Agent Learning

EnvHarness automatically reshapes a base environment into a task- and policy-conditioned environment that exposes a target agent’s weaknesses while preserving reusable, policy-agnostic components. EnvRigger implements this process by observing black-box rollouts, diagnosing failures, synthesizing components, and validating them through fresh executions.

  • EnvHarness objective: EnvHarness generates a modified environment E′ tailored to task t and target policy π by composing components that expose π’s unique flaws for targeted improvement.The transformation applies to the environment alone, so each component is policy-agnostic and reusable across policies, while component selection and parameterization are task- and policy-conditioned.
  • EnvRigger workflow: EnvRigger treats π as a black box, analyzes rollout trajectories, writes task-specific EnvHarness components, and validates candidates with fresh policy rollouts.Candidates are accepted when they provide an appropriate learning signal; unsuccessful candidates are rejected or revised.
  • Diagnose: EnvRigger diagnoses systemic weaknesses from failures and successes, then adjusts customization toward scaffolding missing steps or increasing difficulty when the environment is too forgiving.Example diagnoses include repetitive action loops, difficulty parsing long observations, and misread tool constraints; perfect success can trigger harder scenarios that expose latent flaws.
  • Write and Validate: EnvRigger can synthesize multiple coordinated components, such as a Stage and Contract, to target one flaw or block fragile shortcuts under specified conditions.Candidate components wrap the current environment to instantiate E′ before validation.

4. Experiments

Across five benchmarks in four domains, EnvHarness consistently improves skill-based agents over original environments, no-skill agents, and applicable specialized generators. Its verified environment modifications also reduce execution costs and generalize through a domain-agnostic interface.

  • Cross-Domain Generalization: EnvHarness applies across ALFWorld, WebArena, SWE-bench Verified, OfficeQA, and SpreadsheetBench through one domain-agnostic interface requiring only domain-specific prompt templates.Specialized generators are benchmark-specific and unavailable in other domains.
  • Experimental Setup: EnvRigger and the policy agent use the same model backbone within each benchmark, isolating environment reshaping as the source of performance gains.The experiments focus mainly on skill-based learning and additionally assess online reinforcement-learning compatibility in Section 5.
  • Primary Results: EnvHarness-customized environments outperform original environments on every benchmark, improving ALFWorld by up to 9.0 points and consistently exceeding the no-skill baseline.The write-and-validate loop commits only components verified by fresh policy trajectories, preventing static environments from reinforcing redundant or suboptimal behaviors.
  • Primary Results: EnvHarness surpasses specialized generators by 5.7 ALFWorld points on average, 8.5 points out-of-distribution, and 2.46 SWE-bench Verified success-rate points over SWE-smith.On SWE-bench Verified, it also requires 5.11 fewer execution steps per episode than SWE-smith.
  • Efficiency: EnvHarness reduces SWE-bench Verified average steps per episode from 53.6 to 49.6, while skills from unmodified environments increase them to 55.0.Contracts and Stages targeting repetitive action loops and verbose observations produce the efficiency gain.

5. Analysis

EnvHarness improves reinforcement-learning outcomes, scales training environments efficiently, and generalizes across policy backbones of varying capability. Its Chain component improves long-horizon efficiency, while EnvRigger can target explicit quantitative or natural-language constraints.

  • EnvHarness enables better RL: Training on EnvHarness environments outperforms training on original environments on three of four reinforcement-learning metrics, including ALFWorld in-distribution success of 87.9 versus 81.4.The comparison uses policies trained entirely on either original or EnvHarness environments and evaluates both on the same held-out instances.
  • The unique value of the Chain component: Combining Stage/Contract and Chain skills achieves the highest long-horizon success rate, 54.30, with 43.12 average steps, while Chain alone reduces average steps from 53.58 to 41.96.Chain alone has a standalone success rate of 49.63 versus the 49.88 baseline, reflecting its stringent condition that requires solving both halves.
  • EnvHarness enables efficient environment scaling: EnvHarness evaluates environment scaling under a fixed policy, environment budget, and skill-retrieval protocol while comparing EnvHarness, unmodified benchmark, and SWE-smith-generated environments.Each batch of 50 environments yields one skill bank, alternating between two and three skills per bank, for 15 skills total.
  • EnvHarness generalizes across different LLM backbones: EnvHarness skills outperform skills from real environments on all four tested policies by 2.7–3.7 absolute points, across skill-free success rates from 30.7 to 67.2.The gain is largely independent of policy strength, while EnvHarness helps the two weakest policies most relative to using no skills (+9.3 and +11.1 points).
  • EnvHarness produces environments on demand: EnvRigger can accept explicit user-defined constraints targeting quantitative objectives such as success rate or average steps, or capability weaknesses described in natural language.In standard settings, the loop autonomously identifies training targets through behavioral diagnosis.

6. Related Work

Related work scales or adapts learning environments and separately evolves agent components, but EnvHarness differs by reshaping the environment itself against a diagnosed weakness while the agent learns from it.

7. Conclusion … C.1. ActionableEnv: The Interactable-Environment Interface

EnvHarness converts static benchmarks into controllable, composable environments through a standard interface while preserving their trusted underlying logic and verifiers. Its ActionableEnv contract exposes validated interaction, safe state views, persistence, and optional lifecycle capabilities across heterogeneous runtimes.

  • 7. Conclusion: EnvHarness wraps frozen benchmarks with Stage, Contract, and Chain components that reshape task skills, horizons, or difficulty without modifying internal code.A single implementation can operate across domains while leaving original tasks unchanged.
  • A. The EnvRigger Prompt: EnvRigger emits rule hooks that transform actions, transitions, or observations, plus replayed in-environment actions that prepare the initial state.The levers can be used independently or composed, while the benchmark’s own verdict remains the success metric.
  • A. The EnvRigger Prompt: EnvRigger evaluates candidates using rollout statistics and refines perturbation magnitude when the mutation type moves success toward the target band.The prompt warns against unsolvable mutations and requires reversing or loosening restrictions when rollouts indicate impossibility.
  • B. Distinct Differences from Related Co-Evolution and Synthesis Frameworks: Unlike GenEnv (Guo et al., 2025), EnvHarness preserves native transitions and trusted verifiers; unlike EnvGen (Zala et al., 2024), it avoids benchmark-specific simulator modifications; and unlike Agent-World (Dong et al., 2026), it repurposes existing environments instead of synthesizing them from scratch.These alternatives are characterized as risking evaluation drift, requiring deep domain-specific engineering, or incurring substantial engineering overhead and logic errors.
  • C. Interface Protocol and Design Patterns: Its core commitment is interface uniformity: policies and component layers program against one abstract type whether or not an environment is wrapped.Figure 7 depicts heterogeneous Bridges, the EnvHarness decorator, and ordered component stacks over a Bridge.
  • C.1. ActionableEnv: The Interactable-Environment Interface: ActionableEnv standardizes reset(seed, options), step(action), and typed, validated EnvResponse data for Gymnasium-style interaction.Actions contain a tool name and JSON-serializable keyword arguments, while responses wrap the Gymnasium 5-tuple.
  • C.1. ActionableEnv: The Interactable-Environment Interface: get_env_state() exposes only plain, runtime-safe data to component hooks, enabling the same hook to operate across in-memory, containerized, and browser-backed environments.Hooks cannot access Docker handles, browser pages, sockets, or other underlying runtime objects.
  • C.1. ActionableEnv: The Interactable-Environment Interface: ActionableEnv keeps persistence environment-owned through save_state() and from_state(dict), and provides optional dense rewards, replay callbacks, task enumeration, and resource cleanup.Persistence may serialize full live state or only reset arguments when runtimes such as containers, browsers, or game engines cannot be cheaply cloned.

C.2. Bridges: Adapting Heterogeneous Benchmarks · C.3. EnvHarness: Components as Composable Decorators

EnvHarness standardizes heterogeneous benchmarks through shared Bridges while preserving runtime-specific execution, then reshapes each environment using composable decorator components. Setups, Rules, and Link support reachable state initialization, per-step transformations, and long-horizon composition.

  • C.2. Bridges: Adapting Heterogeneous Benchmarks: Seven Bridges adapt four runtime classes while sharing the policy loop, orchestrator, and component code verbatim across environments.Implemented runtimes include Toy24, ALFWorld via TextWorld, and per-instance Docker environments for SWE-bench, OfficeQA, and spreadsheetBench.
  • C.2. Bridges: Adapting Heterogeneous Benchmarks: Bridges expose typed action tools for universal schema generation while optionally dispatching through the registry according to runtime needs.Toy24 dispatches through the registry, whereas ALFWorld, SWE-bench, and WebArena directly drive engine handles that cannot pass through data-only state views.
  • C.2. Bridges: Adapting Heterogeneous Benchmarks: Bridges choose persistence granularity and publish env_state_schema(), which tells generated component code which exposed fields it may read.Toy24 uses full snapshots, while heavier runtimes use reset-arguments-only persistence; the schema is injected into the designer agent’s prompt.
  • C.3. EnvHarness: Components as Composable Decorators: Each environment contains an ordered EnvHarness decorator stack whose default delegation lets components override only the interfaces they modify.EnvHarness is both the abstract base for components and an ActionableEnv wrapping another ActionableEnv.
  • C.3. EnvHarness: Components as Composable Decorators: Component state is layered and checkpoints preserve the environment plus an ordered component list that the loader reconstructs inward-out.The shipped component types are Setups, Rules, and Link.
  • C.3. EnvHarness: Components as Composable Decorators: Setups create reachable initial states by resetting the inner environment and replaying an action list through its ordinary step interface.The resulting post-replay observation becomes the episode’s initial observation without privileged access to environment internals.
  • C.3. EnvHarness: Components as Composable Decorators: Rules interpose pure hooks that can rewrite actions, transitions, and observations, with generated Python source recompiled and executed in a per-episode subprocess.Hooks can block or rewrite actions, transform EnvResponses, and alter observations including the reset observation; default behavior is identity.
  • C.3. EnvHarness: Components as Composable Decorators: Link composes two ActionableEnvs into one episode through a per-step handoff hook supporting serial concatenation, outcome-conditioned branching, and mid-task switching.The work uses serial composition throughout, while the general mechanism can leave the agent in the current sub-environment or route it to another.

D. Concrete Implementation Examples of the Chain (Link) Operator · E. Experiment Details

The Link operator implements programmable environment composition by intercepting transitions after each step and selecting whether to continue or switch environments. Its examples cover sequential handoff, outcome-based branching, mid-task switching, and continuous interleaving.

  • D. Concrete Implementation Examples of the Chain (Link) Operator: Link overrides modify_transition to inspect each post-step transition and either retain the current environment or switch via self.switch_to().This hook provides the common mechanism for all composition modes.
  • D. Concrete Implementation Examples of the Chain (Link) Operator: Sequential concatenation automatically hands off from EnvA to EnvB when EnvA terminates, without custom transition logic.The default configuration uses a termination-based handoff.
  • D. Concrete Implementation Examples of the Chain (Link) Operator: Outcome-based branching routes the agent to AdvancedEnv after success or RemedialEnv after failure when the first task ends.The destination is selected by evaluating the completed task’s success.
  • D. Concrete Implementation Examples of the Chain (Link) Operator: Custom Link subclasses implement specialized routing by overriding modify_transition while preserving the shared transition-interception pattern.The examples include BranchOnOutcome and SwitchOnAction implementations.
  • D. Concrete Implementation Examples of the Chain (Link) Operator: Harness-controlled transitions can switch environments mid-task as soon as a specified condition is met, without waiting for task termination.A specific observed action can trigger an immediate switch to AdvancedEnv.
  • D. Concrete Implementation Examples of the Chain (Link) Operator: Because transition checks occur after every interaction, Link can alternate the agent continuously between two environments during execution.The example swaps between RedEnv and BlueEnv on every step.

E.1. Benchmark Splits … F.3. Skills from Chain Environments

The appendix specifies benchmark splits, baseline and EnvRigger protocols, reinforcement-learning settings, and skills extracted as EnvHarness co-evolves policies across ordinary and chained environments. These analyses show increasingly localized, constraint-driven skills and chain-specific adaptation to shared budgets and task handoffs.

  • E.1. Benchmark Splits: Training uses reshaped corpus tasks, whereas evaluation uses only original, unreshaped tasks, with benchmark-specific split definitions for SpreadsheetBench and ALFWorld.Table 7 lists the training and evaluation splits; ALFWorld In-Dist and OOD are its native seen and unseen splits, not newly constructed splits.
  • E.2. Baseline Details: GenEnv, VeriEnv, and SWE-smith are run with EnvHarness’s seed tasks, model, and environment count for controlled baseline comparison.The baselines respectively generate difficulty-calibrated tasks, executable website clones with programmatic rewards, and repository-level instances.
  • E.3. EnvRigger Hyperparameters: EnvRigger observes five baseline rollouts, proposes unconstrained component sets, validates each candidate on five fresh rollouts, and repeats write–validate at most five times per instance.Acceptance aggregates success rate, failure distribution, and timeout count rather than relying on a single trajectory; ambiguous candidates return to writing with validation trajectories attached.
  • F. Analyses Details: The supplementary analyses provide protocols, full results, additional experiments, and representative skills underlying the paper’s Section 5 analyses.Reinforcement-learning experiments train Qwen3-8B-base with GRPO on ALFWorld and Webshop using EnvHarness-integrated environments.
  • F.1. Experimental Details for Reinforcement Learning: RL uses EnvHarness environments with 50-step history and episode limits, trained on a single 8× NVIDIA H100 node with memory- and parameter-sharding optimizations.The configuration includes vLLM rollout generation, TP=1, 0.5 GPU-memory utilization, eager execution, FSDP, offloading, and gradient checkpointing.
  • F.2. Skills Across Co-evolution Rounds: Representative round-specific skills include fail-fast or targeted pytest execution, patch-based editing, programmatic pytest invocation, absolute interpreter paths, and grep-based implementation tracing.Each skill is tied to an accepted component that constrains the corresponding action surface and leaves the learned workaround as an escape.
  • F.3. Skills from Chain Environments: Chain environments elicit skills for treating joined tasks as one shared step budget and reorienting immediately after handoff by inspecting the new repository’s environment and tools.The examples show preserving steps for a second task and checking commands such as conda env list and which python when moving between repositories.

F.4. Cross-Model Results

Cross-model evaluation reports success rate and average episode length for four policy models under a shared protocol, revealing distinct efficiency regimes. EnvHarness’s additional successes generally occur within comparable or shorter execution budgets than original-environment skills.

  • Model-specific regimes: Qwen3.6 27B averages 69.8 steps bare versus 37.1 with skills, nearly halving execution length by replacing undirected trial and error with known procedures.Table 9 reports success rate and average episode length for all four policy models under the same protocol.
  • Efficiency across models: 3.7 extra steps for 3.7 extra points is the only case where EnvHarness uses more steps than original-environment skills, while other models achieve gains within essentially the same budget.The skill-source episode lengths are within one step on Flash-Lite and Sonnet, and EnvHarness is over five steps shorter on Flash.
  • Interpretation: Average episode length alone is not a quality signal: short episodes can reflect either efficient solutions or premature failure.The passages contrast efficient short episodes with short episodes that do not indicate quality.

G. Additional Analysis

Additional analyses show that EnvHarness transfers skills across ALFWorld task types, achieves comparable grounded compute to VeriEnv, and can target objective behavioral metrics through environment reshaping. Specified weaknesses can also be converted into targeted components and distilled skills across diverse tasks.

  • Generalizable Skills: 3.1 points on average: EnvHarness skills outperform original-environment skills on four of six held-out ALFWorld task types, with a 16.4-point gain on clean and an 8.7-point regression on heat.The leave-one-out evaluation extracts skills without exposing the held-out task type; results are reported in Table 10.
  • Practical Compute Overhead: 137.3M vs. 137.8M total tokens: EnvHarness and VeriEnv have essentially the same grounded compute footprint, while EnvHarness spends more design tokens to diagnose weaknesses from full trajectories.EnvHarness uses 1.46M design tokens versus 38K for single-pass baselines on ALFWorld, but rollouts dominate both budgets.
  • Objective Metric Targeting: 100 ALFWorld tasks are evaluated for objective targeting using success rate in [0.4, 0.6] and successful-episode steps in [25, 35], with each task measured over K=10 rollouts.Table 12 reports the percentage of tasks falling inside each target band before and after reshaping.
  • Teaching against a specified weakness: EnvHarness turns specified weaknesses into fatal task conditions, then distills skills from trajectories collected in the reshaped environments.Table 13 summarizes nine weaknesses, generated components, and the resulting distilled skills.
  • Teaching against a specified weakness: The demonstrated components address failures including inaccessible objects, inefficient container search, forgotten sub-goals, hidden viewport content, manual pagination, incorrect navigation, and context-poor code edits.Examples include pre-interaction state verification, semantic container prioritization, task-state verification, incremental viewport expansion, query-based filtering, search-first navigation, and context-aware code modification.

H. Limitations

EnvHarness has three limitations: its iterative design loop can require substantial time and inference compute, it depends on resettable gym-style environments, and its Chain supports only sequential subtask composition. These constraints limit applicability to non-resettable backends and workflows requiring semantic relationships, branching, or shared intermediate state.

  • Cost of the design loop: The iterative design loop can consume substantial time and inference compute because each proposed harness requires environment rollouts and weaker designers may need more iterations.This cost is paid once per environment rather than per training episode.
  • Requirement of a resettable, gym-style interface: EnvHarness requires a reset/step interface and excludes environments backed by live or otherwise non-resettable services.Stages must set chosen initial states, while Chains must restore known states between subtasks.
  • Purely sequential composition in Chain: Chain composes subtasks only by sequential concatenation, providing no support for semantic compatibility, branching workflows, or shared intermediate state.Adding semantic composition would require compatibility measures and a verifier.

I. Future Directions

Future work will broaden EnvHarness with new environment components, extend it beyond text-only settings, and replace Chain’s purely sequential composition with richer control flow while preserving trusted verification where possible.

  • New harness components: Future components could add stochasticity, partial observability, auxiliary feedback channels, or multi-agent interaction while preserving the reset/step interface.Stage, Contract, and Chain are presented as an initial rather than closed component set.
  • Beyond text-only environments: Extending EnvHarness to visual, GUI-driven, or embodied environments would test whether its wrapping abstraction survives non-symbolic observations and require state specifications and verification beyond text.The current system operates over textual actions and observations.
  • Purely sequential composition in Chain: Chain’s serial concatenation enables composite verification through conjunctive subtask verdicts but limits the richer control flow that could route among sub-environments.Each leg terminates independently and contributes a verdict, allowing reshaped tasks to inherit trusted human-built verification.
Loading 2608.19880v1…