Source-linked AI summary
Ecdysis: Efficient and Effective Training of Runtime Harnesses for LLM Agents
Ruiqing Yue, Yu Cui, Zhuoyu Sun, Sicheng Pan, Xianhong Xue, Tingyu Li, Ting Li, Wenzhuo Zhu, Yi Chen, Yifei Liu, Baohan Huang, Zhe Cui, Haibin Zhang, Cong Zuo
TL;DR
Existing harness evolution repeatedly uses execution feedback but lacks principled attribution of whether failures arise from the model or the harness, risking overhead and poor generalization. ECDYSIS aggregates recurring cross-task failures and applies Failure-Driven Collaborative Refinement to guide systematic harness repair. It reports faster training, improved reasoning accuracy, stronger cross-LLM generalization, and reduced token consumption.
Problem
Existing harness evolution incurs repeated execution and code-modification overhead and may overfit observed failures because model-specific and harness-level deficiencies are difficult to distinguish.
Method
ECDYSIS aggregates recurring failure patterns across task instances and uses Failure-Driven Collaborative Refinement to produce structured harness modification specifications.
Results
1.84× speedup in harness training and 18.56% higher reasoning accuracy are reported, with stronger cross-LLM generalization and reduced inference-time token consumption.
Takeaways & Limitations
Cross-instance failure analysis and multi-role diagnosis support more efficient and generalizable runtime harness evolution.
Takeaways & Limitations
Cross-task recurrence is treated as an inductive bias rather than proof of causal failure attribution, and local failure-driven evolution can over-accommodate task-model limitations.
Abstract
from arXiv · showhide
Self-evolving runtime harnesses can substantially improve the capabilities of large language model (LLM) agents and provide a promising paradigm for optimizing agent execution. Existing harness evolution methods typically rely on iterative search, repeatedly evaluating and revising candidate harnesses based on execution feedback from task instances. While this paradigm enables continuous harness optimization, it incurs substantial time overhead due to repeated agent executions and code modifications, and may overfit to observed tasks and specific failure patterns, resulting in degraded generalization to unseen tasks. We identify the lack of principled failure diagnosis as a key bottleneck in harness evolution: an observed failure can reflect either model-specific deficiencies or systematic harness deficiencies, and directly optimizing against individual failures can lead to unnecessary model-specific accommodation. We therefore propose Ecdysis, an efficient and effective framework that distinguishes model-specific accommodation from harness-level repair and biases adaptation toward systematic harness deficiencies by identifying recurring cross-task failure patterns. Ecdysis adopts a batch-level cross-instance failure aggregation paradigm to jointly analyze failure evidence from multiple task instances and further introduces Failure-Driven Collaborative Refinement to diagnose failure causes and iteratively refine harness modification specifications. By combining cross-instance failure analysis with multi-role diagnosis, Ecdysis enables more effective harness evolution with lower training time. Experiments show that Ecdysis achieves up to a 1.84x speedup in harness training compared with existing harness evolution methods, while improving the reasoning accuracy of the resulting harnesses by 18.56%.
1 Introduction
Existing harness evolution repeatedly revises runtime code from execution failures, but lacks reliable failure attribution and can incur high training costs while overfitting observed tasks. ECDYSIS addresses this through cross-instance failure aggregation and collaborative diagnosis, improving efficiency, accuracy, and generalization.
- Motivation: Existing methods repeatedly execute agents, modify code, and verify results, creating substantial training-time overhead.They may also specialize to observed tasks and failure patterns, degrading generalization to unseen tasks.
- Motivation: Reliable failure attribution matters because an observed failure may reflect either model behavior or a systematic harness deficiency.Without this distinction, evolution can over-accommodate model-specific limitations and overfit failure patterns.
- ECDYSIS Training Framework: ECDYSIS aggregates recurring failure patterns across task instances to distinguish systematic harness deficiencies from model-specific behavior.Its batch-level analysis prioritizes failure evidence that provides stronger support for harness repair.
- ECDYSIS Training Framework: Failure-Driven Collaborative Refinement uses complementary roles and a moderator to refine causes and produce structured harness modification specifications.This multi-role diagnosis is combined with cross-instance failure analysis to reduce ambiguity in modifications.
- Empirical Results: 1.84× speedup in harness training and 18.56% higher reasoning accuracy are reported across multiple LLMs and benchmarks.The resulting harnesses also show stronger cross-LLM generalization and reduced inference-time token consumption.
- Empirical Results: ECDYSIS training results show substantial improvements in harness efficiency and reasoning performance, alongside stronger cross-model generalization and reduced token consumption.These outcomes summarize the framework’s reported benefits beyond its diagnosis procedure.
2 Related Work
Related work places runtime harness evolution within a broader progression from inference-time agent mechanisms to automated optimization of agent workflows and self-modifying programs.
- Related Work: Runtime harness research builds on tool interaction, deliberate planning, feedback adaptation, prompt optimization, agentic workflows, and self-modifying agent programs.Recent methods specifically evolve harnesses by proposing and validating modifications through execution feedback and iterative evaluation.
3 ECDYSIS
ECDYSIS evolves runtime harnesses by aggregating recurring cross-task failures, diagnosing them collaboratively, and conservatively validating candidate modifications. Its batch-level design reduces redundant modification calls while prioritizing systematic harness repair over isolated model-specific accommodation.
- 3.1 Preliminary Analysis: ECDYSIS distinguishes model-specific accommodation from systematic harness repair by analyzing recurring failure patterns across multiple task instances.Cross-task recurrence is treated as evidence favoring harness-level deficiencies, while single-task failures remain auxiliary evidence rather than mandatory repair targets.
- 3.2 Methodology: Each evolution round executes training tasks, aggregates failure evidence, generates a structured modification specification, modifies the current harness, and retains the candidate only when its overall training score improves.The previous harness is preserved when the candidate does not improve the overall training-set score.
- 3.3 Batch-Level Failure Aggregation: Failure records retain task identifiers, failure decisions, termination reasons, tool-call histories, and execution context derived from trajectories whose scores fall below a threshold.The fixed evaluation framework supplies the binary failure signal and remains unchanged.
- 3.3 Batch-Level Failure Aggregation: Batch-level aggregation reduces modification calls from one per failure record to one per nonempty evolution round, with a strict reduction whenever a round contains multiple failures.If a round contains n_i failure records, independent processing requires n_i calls whereas ECDYSIS uses one call for that nonempty round.
- 3.4 Failure-Driven Collaborative Refinement: Failure-Driven Collaborative Refinement uses Analyst, Critic, Engineer, and Moderator roles to refine evidence-based proposals into conservative harness modification specifications.The specification guides a separate coding agent and prioritizes recurring patterns while avoiding unnecessary changes.
- 3.5 Evaluation: ECDYSIS evaluates inference performance, training efficiency, and cost using metrics including accuracy, Pass@3, Pass^3, training time, evaluation tokens, cache hit rate, and API cost.Held-out evaluation reports the inference metrics and total task-model tokens consumed after training.
4 Experiments
The experiments evaluate ECDYSIS across five task models and multiple benchmarks, using controlled harness configurations and a shared candidate-evaluation protocol. Comparisons isolate harness optimization strategies while keeping the underlying models fixed.
- Models and Datasets: Five task LLMs are evaluated with AgentBench and the Airline and Retail subsets of τ 2-Bench.The study uses APIs, a fixed sampling temperature of 0.0, and OpenCode with DeepSeek-V4-Pro as the coding agent.
- Datasets: The benchmark suite combines relatively simple AgentBench tasks with more challenging τ 2-Bench tool-interaction tasks.Each τ 2-Bench subset includes 20 training tasks and 20 test tasks, with three independent held-out trials per test task.
- Baselines and Ablations: Five harness configurations cover direct, human-optimized, and three evolution-based strategies.The Human-Augmented Harness is the common starting point for all evolution-based methods.
- Controlled Comparison: Only the runtime harness is updated during training, keeping model parameters and the initial harness configuration controlled across evolution methods.This design isolates the effects of different harness optimization strategies.
- Evaluation Protocol: The three methods share training-time evaluation and candidate acceptance protocols, differing in how they organize failure evidence and construct modifications.Each method starts from the same initial harness and allows at most three candidate-generation rounds.
5 Results
ECDYSIS delivers the strongest aggregate task performance while also improving training efficiency over sequential evolution. Failure aggregation supplies the main efficiency gains, whereas FDCR adds accuracy at additional evolution-time cost.
- Overall Results: ECDYSIS with FDCR achieves the highest average accuracy across ten model–dataset combinations, outperforming SE by 27.1% and Human-Augmented Harness by 14.8%.It also improves Pass^3 by 55.2% relative to SE.
- Overall Results: ECDYSIS improves both task accuracy and inference efficiency relative to SE.The results section reports gains in training and inference efficiency alongside task-performance improvements.
- Ablation Results: Adding FDCR raises average task success from 54.67% without FDCR to 59.33%, while Pass@3 reaches 71.50% and Pass^3 reaches 45.00%.These gains follow improvements from SE to ECDYSIS without FDCR, showing contributions from both aggregation and collaborative refinement.
- Training Efficiency: ECDYSIS without FDCR completes training 1.42× faster on τ 2-Retail and 3.23× faster on τ 2-Airline than SE.With FDCR, speedups are 1.30× and 1.84×, respectively, indicating an accuracy-oriented cost for collaborative refinement.
6 Complete Evaluation Results
Complete held-out evaluation shows that ECDYSIS improves accuracy across models and datasets, including models not used during evolution. Its efficiency gains persist during inference, reducing token consumption and mean trajectory runtime.
- 6.1 Task Performance: Average accuracy rises from 58.67% with SE to 69.56% with ECDYSIS with FDCR, an 18.56% relative gain across five models and three datasets.The improvement includes held-out models not used during evolution, such as Qwen3-32B on τ 2-Airline.
- 6.2 Inference Efficiency: ECDYSIS reduces final evaluation tokens from 11.57M under SE to 10.16M with FDCR, a 12.19% relative reduction across ten model–subset combinations.The no-FDCR variant reduces tokens by 10.48%.
- 6.2 Inference Efficiency: ECDYSIS with FDCR lowers mean runtime from 126.08 to 118.69 seconds per trajectory.On Qwen3-14B over τ 2-Airline, runtime falls from 197.61 to 57.16 seconds, a 3.46× speedup.
- 6.1 Task Performance: Held-out task performance is reported across five LLMs and two datasets in Table 2.The table covers the complete cross-model, cross-dataset evaluation matrix.
7 Analysis of Evolution Process
ECDYSIS reduces harness-evolution cost by aggregating failures at the round level rather than generating modifications for individual failures. This lowers candidate-generation overhead and shortens end-to-end training time while preserving efficiency gains on held-out evaluation.
- Analysis Setup: The analysis measures actual execution paths using shared evaluation, validation, rollback, and early-stopping protocols rather than normalized theoretical budgets.This frames the reported efficiency results as observed resource usage during complete evolution runs.
- Candidate Generation: Round-level aggregation keeps candidate generation costs from growing with the number of failures within an evolution round.SE accumulates modifications for individual failures, whereas ECDYSIS generates candidates from aggregated round-level evidence.
- API Cost: Training API costs are divided into evaluation and evolution, with evaluation costing less than $1 for all three methods.Evolution includes candidate analysis, implementation, coding-agent calls, FDCR when enabled, and other model calls.
- End-to-End Evolution Time: 3.23× and 1.84× speedups on τ 2-Airline show that ECDYSIS shortens end-to-end training relative to SE.The corresponding τ 2-Retail speedups are 1.42× and 1.30× for ECDYSIS without and with FDCR, respectively.
8 Task Structure Analysis
ECDYSIS targets recurring interaction structures and failure mechanisms instead of memorizing isolated task failures. Its analysis links local failure-driven evolution to model-specific accommodation and motivates informative task selection for generalization.
- Task Structure: Recurring structures across Retail and Airline tasks include persistent goal tracking, conditional branching, state constraints, and cross-tool dependencies.These structures motivate harness changes that address runtime constraints rather than surface-specific failures.
- Failure-Aware Training Data: Aggregating failures across task instances helps identify shared harness deficiencies instead of repeatedly addressing different surface manifestations of the same mechanism.The proposed curation principle prioritizes informative tasks with novel execution paths and failure mechanisms over task quantity alone.
- Failure-Aware Training Data: The reduced-data experiment trains ECDYSIS with five failures, one-quarter of the original training set, and compares it with full-data and randomly selected samples.The passage specifies the experimental comparison but does not report its outcome.
- Beyond Local Failure-Driven Evolution: A local evolution process can promote one training-task answer into a global harness prohibition, shrinking the valid action space across models.This may improve the triggering evaluation case while impairing generalization.
- Beyond Local Failure-Driven Evolution: 60.0% of SE modification decisions accommodated task-model limitations versus 45.5% for ECDYSIS, a reduction of 14.5 percentage points.The analysis associates cross-task failure evidence with broader support before committing to harness-level modifications.
9 Conclusion
ECDYSIS reframes harness evolution around recurring cross-task failure patterns and multi-role diagnosis, addressing the risk that individual failures encode model-specific limitations. Across multiple models and benchmarks, it improves efficiency, reasoning performance, cross-model generalization, and token consumption.
- Conclusion: ECDYSIS shifts harness evolution from individual failures to recurring cross-task patterns and applies FDCR before committing to harness-level modifications.The framework seeks stronger evidence for systematic harness deficiencies and reduces model-specific accommodation.
- Conclusion: Across multiple LLMs and reasoning benchmarks, ECDYSIS improves harness training efficiency, agent reasoning performance, cross-model generalization, and token consumption.The conclusion links reduced model-specific accommodation with more generalizable harness evolution.
Ethical Considerations
The authors used AI assistants to polish the writing and assist with programming, and reviewed and verified all AI-assisted outputs.
- Ethical Considerations: AI assistants supported writing and programming, while the authors reviewed and verified all AI-assisted outputs.The disclosure covers both writing polish and programming assistance.