Source-linked AI summary
StarHarness: Evolving Harnesses with Stratified Search for Enterprise Environments
Esakkivel Esakkiraja, Denis Akhiyarov, Vikas Yadav, Sai Rajeswar, Patrice Bechard, Sridhar Nemala, Sagar Davasam
TL;DR
Stateful enterprise agents face tool, state, dependency, and convention mismatches that make reliable task completion difficult. StarHarness evolves environment-specific harnesses around fixed model weights using stratified search and separated evaluation, improving benchmark performance while generalizing across excluded tasks and model families. The findings identify interface repairs, environment conventions, and operational knowledge as recurring forms of specialization.
Problem
Enterprise agents must operate across stateful backends, large tool surfaces, cross-step dependencies, and domain conventions that tool schemas may omit.
Method
StarHarness evolves frozen-model harnesses through failure-stratified task pools, proposer-visible search, hidden selection, and held-out evaluation.
Results
Across three enterprise benchmarks, evolved harnesses improve full-benchmark performance, generalize to excluded tasks, and transfer across GPT and Qwen models without re-evolution.
Takeaways & Limitations
Harness evolution offers a practical complement to model scaling for stateful enterprise agents by adapting record retrieval, tool use, dependency preservation, and side-effect verification.
Takeaways & Limitations
The search modes are sequential, so their complementarity is described rather than established through a causal head-to-head comparison.
Abstract
from arXiv · showhide
We present StarHarness, a framework for evolving environment-specific agent harnesses while keeping model weights fixed. The evolved harness can include prompt and task framing, tool interfaces, skills, MCP-backed providers, subagent structure, and agent-loop configuration. StarHarness constructs a compact evolution pool by stratifying tasks according to baseline failure behavior, separates proposer-visible search tasks from proposer-hidden selection tasks, and reserves held-out tasks for evaluating generalization. Across ITBench SRE, EnterpriseOps-Gym ITSM, and AutomationBench Finance, harness evolution improves full-benchmark performance by 20-35 percentage points over the default harness after 4-12 accepted changes per environment. These gains persist on tasks excluded from evolution and transfer without re-evolution across GPT and Qwen model families. Trace analysis links the improvements to interface repairs, environment conventions, and operational knowledge that compresses search, with fewer false-positive diagnoses and shorter trajectories in several settings. StarHarness therefore offers a practical way to reduce persistent model-environment mismatch in tool-rich enterprise tasks.
1 Introduction
StarHarness evolves environment-specific agent harnesses around fixed model weights for difficult, stateful enterprise tasks. Its stratified, separated evaluation protocol supports efficient search, generalization testing, and transfer across models.
- Enterprise agents must navigate stateful backends, broad tool surfaces, cross-step dependencies, and domain conventions that schemas may omit.
- StarHarness adapts the harness while keeping model weights fixed, including how agents frame tasks, use tools, and interpret environment state.
- Its protocol stratifies tasks by baseline failure behavior, separates proposer-visible search from hidden selection, and reserves held-out evaluation tasks.
- Across three enterprise benchmarks, evolved harnesses improve excluded tasks and transfer across GPT and Qwen models without re-evolution.
- Analysis identifies interface repair, environment conventions, and operational knowledge that compress search and improve agent precision, convergence, and efficiency.
2 Related Work
StarHarness extends harness evolution to adapting frozen models for stateful enterprise environments. It distinguishes its deployment scope and stricter generalization protocol from related prompt, architecture, and co-evolution approaches.
- Prompt optimization searches instructions and demonstrations, while agent-architecture methods also search executable modules and workflow structure.
- Recent harness-level systems edit executable scaffolding or co-evolve harnesses with model policies and weights.
- StarHarness instead adapts a frozen model’s harness to a stateful enterprise environment, with edits spanning prompts, tools, skills, MCP providers, subagents, and execution policy.
- Its task-level separation, proposer-hidden selection, and held-out evaluation distinguish search performance from generalization.
- The evaluation spans complementary settings including Kubernetes root-cause analysis, ITSM workflows, and multi-application finance workflows with state checks.
3 Method
StarHarness evolves a frozen model’s executable harness through bounded, validated interventions, using stratified task partitions and hidden evaluation to guide search. It supports both single-frontier hill climbing and tree-based exploration while enforcing guardrails against task-specific solutions.
- Overview: StarHarness evolves the executable scaffold around a fixed language model, including prompts, tools, skills, MCP providers, subagents, and execution policies.The optimizer modifies a separate harness while keeping model weights and the benchmark fixed.
- Evolution loop: The coding harness proposes one patch, validates scope and a smoke test, evaluates valid candidates, and accepts only deterministic improvements.A proposer-selected test flip can reject a candidate before expensive hidden selection evaluation; rejected, invalid, and crashed candidates revert to the prior frontier.
- Task partitioning: The protocol partitions reproducibly evaluated tasks into a stratified evolution pool, proposer-visible search tasks, hidden selection tasks, and untouched holdouts.Tasks are stratified using baseline failure mode and other baseline descriptors; holdouts never affect proposal or acceptance.
- Search strategies: Hill climbing keeps a single frontier, retaining a patch only when it improves selection score or ties while improving an available verifier metric.Otherwise, the system restores the previous harness.
- Search strategies: Tree search preserves alternative candidate nodes with parent links, cumulative patches, validation status, traces, and scores before promoting the best surviving node.This retains multiple hypotheses instead of committing after the first accepted edit.
- Search strategies: The sequential tree-then-hill design is an exploration–exploitation control experiment on EnterpriseOps-Gym, not a causal head-to-head comparison.Tree search explores hypotheses, while hill climbing exploits the best tree frontier through bounded local edits.
- Guardrails: Guardrails constrain candidates to scoped git diffs and prohibit task-ID branching, verifier content, hidden-state access, and benchmark-specific answer mappings.These constraints target reusable environment behavior rather than individual task solutions, with validation failures rejected before selection evaluation.
4 Experiments
StarHarness is evaluated across three enterprise benchmarks using frozen model weights, transferred models, and comparisons with multiple harness baselines. It is strongest across all three benchmarks, with lower estimated inference cost and transfer gains across GPT and Qwen models.
- Benchmarks: The evaluation covers ITBench SRE, EnterpriseOps-Gym ITSM, and AutomationBench Finance, using benchmark-specific full-benchmark scores.AutomationBench measures the share of domain objectives achieved, and guardrail violations score zero.
- Models and baselines: The study evolves harnesses with GPT-5.4 and evaluates the same frozen artifacts on additional GPT and Qwen models without re-evolution.The baseline is the unmodified Stirrup framework, with additional comparisons to Pi, Codex, and GEPA on Pi.
- Main results: StarHarness (Stirrup) is strongest on all three benchmarks, exceeding GEPA (Pi) by +13.8, +22.3, and +17.6 percentage points on ITBench, EnterpriseOps-Gym, and AutomationBench, respectively.The comparison is descriptive because the systems differ in prompts, tools, execution policies, and harness architecture.
- Main results: StarHarness reduces estimated GPT-5.4 inference cost per task by 17% on ITBench, 53% on EnterpriseOps-Gym, and 29% on AutomationBench.These estimates use published rates.
- Frozen cross-model transfer: The same evolved harness improves every transferred model in Table 1, including models from both GPT and Qwen families.Table 1 reports full-benchmark scores, with reasoning levels in parentheses when applicable.
- Reference results: External Claude reference scores are reported for EnterpriseOps-Gym but are not part of the controlled baseline–StarHarness comparison.The reported references are 48.1% for Fable 5, 35.9% for Sonnet 5, and 35.5% for Opus 4.8 max.
5 Analysis: What Harness Evolution Learns
Across the three benchmarks, accepted harness edits repaired interfaces, encoded environment conventions and operational knowledge, and compressed search. These changes coincided with improved task outcomes, shorter workflows, and reduced unsafe or false-positive behavior, although individual patch effects were not isolated.
- Accepted edits: 21 patches were accepted across the three evolution runs: 4 for ITBench, 12 for EnterpriseOps-Gym, and 5 for AutomationBench.EnterpriseOps-Gym accepted 8 patches during tree-search exploration and 4 during hill climbing.
- Interface repair: Interface repairs improved tool usability by correcting MCP arguments, preserving compound schemas, pruning misleading fields, and replacing fragile spreadsheet edits with structured row operations.These changes modified the existing interface without changing task data or the verifier.
- Environment conventions: Environment-specific procedures encoded execution contracts, coupled priority and impact/urgency updates, relationship preservation, sandbox-relative dates, and triage before mutations.The harness encoded these conventions while keeping model weights fixed.
- Operational knowledge and search compression: Operational knowledge compressed search by ranking upstream causes from evidence and delegating deterministic date, finance, arithmetic, and spreadsheet operations to specialized tools.These aids exposed derived information from the task environment without accessing labels or verifier state.
- Benchmark outcomes: ITBench score increased from 40.0% to 75.0%, with fewer turns and false positives, more true positives, and a comparable number of shell calls.The evolved agent reached more accurate conclusions while inspecting a comparable amount of raw evidence; regressions occurred when upstream search continued after a proximate cause should have ended it.
- Benchmark outcomes: EnterpriseOps-Gym evolution shortened workflows and increased verifier completion, while AutomationBench evolution reduced unsafe execution and improved partial completion.The reported comparisons cover the full EnterpriseOps-Gym benchmark and the full GPT-5.4 AutomationBench task set.
- Interpretation: Individual tool and patch contributions cannot be isolated from the paired benchmark comparisons.The AutomationBench records likewise cannot isolate any individual tool’s contribution.
6 Conclusion
StarHarness provides evidence that adapting a frozen model’s harness to a stateful enterprise environment can improve performance. The approach is presented as a practical complement to model scaling, while future work would jointly specialize harnesses and model weights.
- Conclusion: Adapting a frozen model’s surrounding harness to the environment can substantially improve performance across stateful enterprise tasks.The reported changes include tool-interface repairs, environment conventions, operational knowledge, and search-compression aids, with generalization and model-family transfer.
- Conclusion: Harness evolution is a practical complement to model scaling for stateful enterprise agents.Future work would co-evolve harnesses and model weights through reinforcement learning.