PaperlayerView exact revision on arXiv

Source-linked AI summary

Adapting the Interface, Not the Model: Runtime Harness Adaptation for Deterministic LLM Agents

Tianshi Xu, Huifeng Wen, Meng Li

arXiv:2605.22166v2cs.AI

TL;DR

Many deterministic-agent failures arise from mismatches at the model–environment interface rather than model limitations. LIFE-HARNESS evolves reusable runtime interventions from training trajectories, improving frozen agents across diverse models and environments while offering a complementary alternative to model-centric training.

  • Problem

    In deterministic, rule-governed domains, agent failures often stem from mismatches in observations, tool contracts, executable actions, feedback, and trajectory control at the model–environment boundary.

  • Method

    LIFE-HARNESS evolves a lifecycle-aware runtime layer from training trajectories, adapting environment contracts, procedural skills, action realization, and trajectory regulation without changing model weights or evaluation environments.

  • Results

    LIFE-HARNESS improves 116 of 126 model–environment settings across seven environments and 18 backbones, with an average relative gain of 88.5%.

  • Takeaways & Limitations

    Runtime interface adaptation can mitigate many agent failures and serve as a complementary alternative to model-centric agent training.

  • Takeaways & Limitations

    The approach is evaluated in deterministic, rule-governed settings with relatively stable interfaces and remains challenging to extend to fully open-ended tasks.

Abstract

from arXiv · show

LLM agents are shaped not only by their language models, but also by the runtime harness that mediates observation, tool use, action execution, feedback interpretation, and trajectory control. While existing agent adaptation methods mainly update model parameters, many failures in deterministic, rule-governed domains stem from mismatches at the model--environment interface. We propose Life-Harness, a lifecycle-aware runtime harness that improves frozen LLM agents without changing model weights or evaluation environments. Life-Harness evolves from training trajectories by converting recurring interaction failures into reusable interventions across environment contracts, procedural skills, action realization, and trajectory regulation, and remains fixed for evaluation on unseen tasks. On seven deterministic environments from $τ$-bench, $τ^2$-bench, and AgentBench, Life-Harness improves 116 out of 126 model--environment settings across 18 model backbones, with an average relative improvement of 88.5%. Harnesses evolved only from Qwen3-4B-Instruct trajectories transfer to 17 other models, showing that Life-Harness captures reusable environment-side structure rather than model-specific behavior. These results position runtime interface adaptation as a complementary alternative to model-centric agent training. Code is available at https://github.com/Tianshi-Xu/Life-Harness.

1 Introduction

LLM agent behavior depends on the runtime harness as well as model parameters, motivating adaptation of the model–environment interface. LIFE-HARNESS evolves reusable runtime interventions from training trajectories and improves frozen agents across deterministic environments and diverse backbones.

  • Motivation: Runtime harnesses mediate observations, tools, actions, and feedback, so agent behavior is shaped beyond the language model itself.The harness governs how models observe environments, use tools, realize actions, and interpret feedback.
  • Motivation: Deterministic-domain failures often arise from mismatched observations, tool contracts, executable actions, feedback recovery, or trajectory dynamics.These mismatches expose stable environment-side structure that can be addressed at the interface.
  • Results: 116 of 126 model–environment settings improve across seven deterministic environments and 18 backbones, with an average relative gain of 88.5%.The evaluation spans household interaction, web shopping, OS control, database tasks, and policy-guided business workflows.
  • Results: Harnesses evolved only from Qwen3-4B-Instruct trajectories transfer to 17 other backbones, indicating reusable environment-side rather than model-specific structure.The results also position runtime interface adaptation as complementary to model training.
  • Approach: LIFE-HARNESS converts recurring trajectory failures into reusable interventions across environment contracts, procedural skills, action realization, and trajectory regulation.The harness adapts the runtime layer while leaving model parameters unchanged.

2 Related Work

Runtime-harness research improves frozen agents outside model weights, but existing approaches often target coding agents or model-coupled adaptations. LIFE-HARNESS instead uses a structured, lifecycle-organized interface for deterministic domains beyond coding.

  • Existing harness optimization: Recent harness methods optimize reasoning controllers, online workspaces, harness flags, or executable harness code around frozen LLM systems.These approaches establish runtime harness optimization as an alternative to model training.
  • Scope: Meta-Harness and AHE focus on automated harness engineering for coding agents, whereas LIFE-HARNESS targets deterministic domains beyond coding.Its scope includes household interaction, web shopping, database tasks, and policy-guided workflows.
  • Abstraction: LIFE-HARNESS treats adaptation as a structured runtime interface organized by the agent interaction lifecycle rather than free-form harness-code search or continuous editing.This distinguishes its abstraction from methods that treat harnesses as code artifacts or mutable state.
  • Model coupling: Prior adaptations can remain tightly coupled to specific model checkpoints and training distributions, while LIFE-HARNESS leaves model weights frozen and adapts the runtime interface.The proposed paradigm separates environment-side interface adaptation from model-specific training.

3 From Parameter Adaptation to Runtime Interface Adaptation

The paper models agents as stateful runtime systems whose trajectories are mediated by contracts, actions, environment execution, and feedback. Runtime interface adaptation keeps model parameters fixed while changing this mediation layer instead of repeatedly retraining model-specific weights.

  • Runtime system: An episode combines a task, environment, contract, and step budget, with the contract specifying tools, argument formats, feedback, answers, and policies.The environment may include both the environment and a user.
  • Runtime system: At each step, the trajectory records the contract, task, past actions, and observations, while actions may be tool calls, text commands, or final answers.Text actions can be represented as pseudo-tools for unified notation.
  • Runtime system: Unsupported or ineffective actions generate feedback that remains part of the trajectory, which continues until completion, termination, or budget exhaustion.This runtime view makes post-execution mediation part of agent performance.
  • Parameter adaptation: Conventional parameter adaptation absorbs task-specific structure into model weights and must be repeated when the model or environment changes.This makes the approach inherently model- and task-specific.
  • Runtime interface adaptation: Runtime interface adaptation keeps model weights and evaluation protocols unchanged while changing how the model interacts with the environment.The adapted harness is environment-specific but model-agnostic across backbones following the same interaction protocol.

4 Method: LIFE-HARNESS

LIFE-HARNESS adapts the model–environment interface through four lifecycle layers, targeting heterogeneous deterministic-agent failures without changing model weights or the evaluation environment. It evolves reusable interventions from training trajectories for contract clarification, procedural guidance, action validation, and trajectory recovery.

  • Failure Diagnosis: LIFE-HARNESS diagnoses four failure categories: action realization, environment contract mismatch, trajectory degeneration, and general reasoning failure.The dominant failure mode varies substantially across environments, motivating multiple intervention points.
  • Environment Contract Layer: The Environment Contract Layer makes tool-use rules, policy constraints, and common environment pitfalls explicit before interaction.It enhances the model-visible contract with updates derived from environment policies, API behavior, and recurring training failures.
  • Procedural Skill Layer: The Procedural Skill Layer retrieves reusable skills from a trajectory-derived library during task conditioning.Relevant skills are selected from the task description and inserted into the initial system prompt as non-parametric guidance.
  • Action Realization Layer: The Action Realization Layer validates model outputs before execution, canonicalizes unambiguous interface errors, and blocks deterministically failing actions.It uses tool schemas, admissible action sets, argument constraints, and task policies to map intended operations to executable actions.
  • Trajectory Regulation Layer: The Trajectory Regulation Layer monitors post-execution interaction patterns and triggers recovery for repetition, stagnation, or budget exhaustion.Its responses range from empty output and soft recovery messages to warnings or corrective directives when degradation is clear.
  • Lifecycle Integration: The four layers adapt the runtime interface across the agent lifecycle while keeping model weights and the evaluation environment unchanged.LIFE-HARNESS is evolved from frozen-model training trajectories, with interventions retained for evaluation on unseen tasks.

5 Experiments

Experiments evaluate LIFE-HARNESS across seven deterministic benchmarks and 18 model backbones, showing broad gains, cross-model transfer, iterative improvement, and complementarity with model training.

  • Experimental Setup: The evaluation covers seven deterministic task scenarios from τ-bench, τ 2-bench, and AgentBench across 18 model backbones.The model set includes instruction-tuned, reasoning, and agent-specialized backbones.
  • Main Results: LIFE-HARNESS improves 92% of model–benchmark settings, with relative gains of 10 ∼84% across all benchmarks.The complete results average performance over 18 models and provide a model-by-benchmark breakdown across seven benchmarks.
  • Evolution Dynamics: Training-set performance improves steadily across evolutionary rounds and eventually saturates, indicating practical iterative harness evolution.The four-layer design localizes updates to identifiable failure modes rather than rewriting the harness as an unconstrained whole.
  • Comparison with Prompt Evolving: LIFE-HARNESS adds an average relative improvement of 120% over prompt-only evolving in Pass@1 performance.The comparison attributes the gap to runtime mediation of tools, actions, feedback, and multi-step trajectories in addition to the initial prompt.
  • Ablation Study: Removing any of the four lifecycle layers causes substantial performance drops on some datasets, with different tasks benefiting from different layers.The leave-one-layer-out ablation evaluates the Environment Contract, Procedural Skill, Action Realization, and Trajectory Regulation layers.
  • Harnessing and Model Training: Harnessing complements model training: Qwen2.5-32B with LIFE-HARNESS surpasses xLAM-2-32B by 7.5 percentage points in-domain, while harnessing improves xLAM by 6.8 ∼28.9 percentage points.Specialized tool-use training underperforms its base model on τ 2-bench and AgentBench, indicating reduced OOD generalization.

6 Conclusion

LIFE-HARNESS adapts the runtime interface of frozen LLM agents through reusable interventions derived from training trajectories. Across seven environments and 18 backbones, it achieves broad gains while keeping model weights and evaluation environments fixed.

  • Conclusion: LIFE-HARNESS evolves reusable interface interventions across environment contracts, procedural skills, action realization, and trajectory regulation.The approach targets the runtime interface rather than updating model parameters.
  • Conclusion: Across seven environments and 18 model backbones, LIFE-HARNESS achieves broad performance gains with model weights and evaluation environments fixed.The conclusion presents runtime-interface adaptation as a way to mitigate failures in frozen LLM agents operating in rule-governed environments.

Limitations

The study focuses on deterministic, rule-governed environments with relatively stable interfaces, feedback rules, and evaluation criteria.

  • Limitations: The harnessing approach is evaluated in deterministic, rule-governed settings where interfaces, feedback rules, and evaluation criteria are relatively stable.Extending harness construction to fully open-ended tasks remains challenging because goals, tools, external resources, and success criteria can vary.

A.1 Failure Annotation Protocol

The protocol classifies each failed episode by the earliest primary failure among action realization, environment contract mismatch, trajectory degeneration, and residual reasoning. These categories guide targeted harness updates across four lifecycle layers while preserving the deterministic evaluation setting.

  • Failure Annotation Protocol: Each failed episode is classified trajectory by trajectory using a priority order: action realization, environment contract mismatch, trajectory degeneration, then residual reasoning.Codex reads the complete interaction trace and assigns one primary category.
  • Failure Categories: Action realization failures occur when the model’s intent is reasonable but its output is not executable by the environment.Examples include missing tool calls, invalid function names, malformed JSON, missing arguments, incorrect types, or non-executable SQL.
  • Failure Categories: Environment contract mismatches occur when executable calls violate tool purpose, boundaries, calling protocols, or argument semantics.Examples include wrong tools, invalid calling order, premature submission, and semantically incorrect answer formats.
  • Failure Categories: Trajectory degeneration covers executable actions that fail over long horizons through repetition, early strategic commitment, or repeated no-progress feedback.This category is considered only when action realization and environment contract mismatch are not the primary causes.
  • Failure Categories: Residual reasoning failures cover incorrect reasoning, computation, SQL, retrieval, or value selection after other failure categories are excluded.The category also includes retrieving the wrong answer or object when the error is not attributable to tool-protocol misunderstanding or trajectory loops.
  • Harness Evolution: The harness evolution process maps recurring, mechanically identifiable failures to the earliest lifecycle point where they can be detected or prevented.Updates are local, evidence-triggered, evaluation-preserving, robust to unseen tasks, and checked for over-triggering or negative side effects.

A.3 Final Evolved Harness Inventory

The evaluated harness inventory follows LIFE-HARNESS’s four lifecycle layers across the seven scenarios, with detailed component implementations provided in the codebase.

  • Final Evolved Harness Inventory: The harness inventory organizes components under four lifecycle layers across τ-bench, τ^2-bench, and AgentBench scenarios.τ-bench and τ^2-bench components appear in Table 5, while AgentBench components appear in Tables 6 and 7.

B Additional Experiments

The additional experiments evaluate held-out tasks across seven benchmarks and 18 model backbones, reporting consistent gains with LIFE-HARNESS. The evaluation uses explicit sampling, interaction-budget, and metric conventions.

  • Dataset Statistics and Splits: Evaluation separates raw training pools, harness-evolution subsets, and held-out test sets to keep evolution efficient while preventing evaluation-task reuse.“Train Used” counts examples sampled for each harness-evolution run, while “Used Total” and “Raw Pool Total” report experiment usage and full pool size.
  • Evaluation Configuration: The Procedural Skill Layer retrieves only the top-1 skill across experiments to prevent irrelevant skills from contaminating model context..
  • Main Results: The main results cover 18 model backbones and 7 benchmarks, showing consistent gains across diverse models and environments.Table 8 reports the full LIFE-HARNESS results.
  • Main Results: Table 8 reports pass@1 for AgentBench and Pass@1, Pass@3, and Pass^3 for τ-bench and τ^2-bench, comparing each model with and without LIFE-HARNESS.Bold values indicate improvement or a match relative to the corresponding no-harness result.
Loading 2605.22166v2…