Source-linked AI summary
Task-CoEvolve: Efficient Harness Optimization via Adaptive Validation Task Selection
Atsuyuki Miyai, Kiyoharu Aizawa, Toshihiko Yamasaki
TL;DR
Existing harness optimization repeatedly evaluates full fixed validation sets, creating high costs as tasks become less discriminative. Task-CoEvolve adaptively samples informative tasks and estimates full-set scores from their inclusion probabilities, achieving comparable or better performance with substantially fewer evaluations.
Problem
Existing harness optimization evaluates every candidate on a fixed validation set at every iteration, although this is expensive and becomes less informative as the harness evolves.
Method
Task-CoEvolve uses variance-weighted adaptive task selection and sampling-aware full-set estimation to compare candidates evaluated on different task subsets.
Results
Task-CoEvolve approaches or matches full-set search across text classification and Terminal-Bench 2.1, while reducing evaluations or search costs by 67–80%.
Takeaways & Limitations
Adaptive validation task selection can reduce harness-optimization evaluation cost while retaining performance similar to full-set search.
Takeaways & Limitations
Task-CoEvolve fixes the number of tasks evaluated per candidate before observing results, preventing early stopping for clearly poor candidates or additional evaluation for difficult comparisons.
Abstract
from arXiv · showhide
We present a novel approach to efficient LLM harness optimization through adaptive validation task selection. Harness optimization iteratively rewrites the harness code based on validation performance, enabling substantial performance gains without updating the underlying model weights. Existing approaches, however, evaluate a fixed validation set in full at every iteration, incurring substantial evaluation costs even on tasks that become less discriminative as the harness evolves. We propose $\textbf{Task-CoEvolve}$, which co-evolves the validation tasks with the harness by addressing two challenges: selecting informative tasks and estimating full-set performance from partial evaluations. Task-CoEvolve builds on the observation that tasks on which candidate harnesses disagree are more informative for distinguishing among them than tasks that are consistently solved or failed. It uses variance-weighted sampling based on past outcomes to focus evaluation on tasks near the capability frontier, with the sampling distribution adapting as the harness evolves. It then estimates full-set scores from the sampled tasks by accounting for their sampling probabilities, enabling consistent comparisons across iterations despite evaluating different subsets. Experiments on online text classification and Terminal-Bench 2.1 show that Task-CoEvolve consistently outperforms subset-based baselines and matches the final performance of full-set search while reducing the number of evaluations during optimization by 80%. Code will be released at https://github.com/Agent4Science-UTokyo/Task-CoEvolve.
TASK-COEVOLVE: EFFICIENT HARNESS OPTIMIZATION VIA ADAPTIVE VALIDATION TASK SELECTION
The paper presents a figure comparing existing harness optimization with Task-CoEvolve. The listed authors include Atsuyuki Miyai, Kiyoharu Aizawa, and Toshihiko Yamasaki.
- Atsuyuki Miyai is listed among the paper’s authors.
- Kiyoharu Aizawa and Toshihiko Yamasaki are also listed among the authors.
- Figure 1 compares existing harness optimization with Task-CoEvolve.
- The passage identifies co-supervision of this work.
1 INTRODUCTION
The introduction motivates reducing the cost of harness evaluation by adapting which validation tasks are used. Task-CoEvolve selects discriminative tasks, estimates full-set performance from partial evaluations, and achieves strong performance with substantially fewer evaluations.
- Motivation: Existing harness optimization evaluates every candidate on the entire fixed validation set at each iteration, making costly task execution a major optimization expense.The introduction identifies both the expense of full-set evaluation and its static nature as limitations.
- Task-CoEvolve: Task-CoEvolve co-evolves the validation task set with the harness to select discriminative tasks and compare partial evaluations across iterations.Its task-selection criterion favors tasks where candidate outcomes differ, while its estimator uses task inclusion probabilities.
- Problem Setting: The proposed problem setting optimizes which tasks evaluate candidate harnesses, reducing evaluation cost per candidate rather than reducing the number of candidates.
- Task-CoEvolve: Task-CoEvolve combines adaptive task selection with full-set score estimation from partial evaluations.Adaptive selection concentrates evaluation near the capability frontier, and estimation provides a common criterion across subsets.
- Empirical Findings: On text classification, Task-CoEvolve approaches full-set search with 7% of the evaluation budget and surpasses it with 20%.
- Empirical Findings: On Terminal-Bench 2.1, Task-CoEvolve achieves comparable performance while reducing search costs by 67–80%.
2 RELATED WORK
Related work spans automated harness optimization, efficiency improvements for harness and program search, adaptive task selection, and sample-efficient model evaluation. The paper distinguishes its evaluation-focused task selection from methods that optimize learning tasks or selected test examples.
- Automatic Optimization of Harnesses: Automated harness optimization improves the harness while keeping the underlying model fixed through iterative proposal, evaluation, and selection.
- Efficient Harness Optimization: Recent efficiency methods target harness and program optimization through informative demonstrations or more efficient candidate generation and selection.DemoEvolve uses human demonstrations, while ShinkaEvolve and TurboEvolve improve sample efficiency in LLM-driven evolutionary search.
- Curriculum Learning and Adaptive Task Selection: Curriculum learning and adaptive task selection dynamically choose training tasks according to current capabilities, whereas harness optimization asks what to evaluate on.
- Sample-Efficient Model Evaluation: Sample-efficient model evaluation estimates performance from selected test examples using approaches such as importance weighting, compact subsets, or model-informed sampling.
3 METHOD
Task-CoEvolve optimizes harnesses under limited evaluation budgets by adaptively selecting informative validation tasks and estimating comparable full-set scores from partial evaluations.
- Problem Setting: The method evaluates candidate harnesses under a limited budget measured by total task executions, rather than using the full validation set at every iteration.Automated harness optimization proposes candidates iteratively and selects the candidate with the highest full-set score.
- Overall Procedure: Task-CoEvolve samples a new validation subset using past task outcomes and estimates full-set performance while accounting for task inclusion probabilities.These components provide a common comparison scale across candidates evaluated on different subsets.
- Initialization: The procedure reuses full-task-set outcomes from two initial harnesses to initialize each task’s historical success rate before subset sampling begins.This initialization is treated as part of the evaluations needed for harness optimization rather than as an additional method-specific cost.
- Variance-Weighted Task Selection: Variance-weighted sampling prioritizes tasks where previous harnesses disagree, while assigning additional weight to tasks with fewer observations.The Bernoulli variance is largest at a historical success rate of 0.5 and vanishes for tasks that are always solved or always failed; a floor preserves exploration of never-solved tasks.
- Sampling-Aware Full-Set Estimation: Full-set estimation uses each task’s inclusion probability, estimated by Monte Carlo simulation, to compare evaluations drawn from different subsets.The estimator is chosen according to benchmark task-pool structure, with Hájek estimation used when success rates are near 0 or 1.
- Final Selection: After optimization, Task-CoEvolve selects the candidate with the highest estimated full-set score, breaking exact ties in favor of the earliest iteration.The estimated score is intended to support selection despite candidates being evaluated on different task subsets.
4 EXPERIMENTS
Experiments evaluate Task-CoEvolve on online text classification and Terminal-Bench 2.1 against full-set and subset-search protocols. Across both settings, adaptive task selection preserves performance while substantially reducing evaluation and search costs.
- Online text classification: Task-CoEvolve nearly matches full-set search at ρ=7% and surpasses it at ρ=20% on online text classification.Table 1 reports held-out test accuracy under different validation-sampling budgets.
- Online text classification: 49.3% at ρ=20% and 47.6% at ρ=7%, exceeding Naive by 2.1 and 2.4 points, respectively.At ρ=7%, few-shot accuracy rises from 41.6% to 47.6% using 16 times fewer samples than full-set search.
- Online text classification: Task-CoEvolve outperforms Meta-Harness by about 1% at ρ=20%.The paper suggests that changing evaluation samples across iterations may reduce validation-set overfitting.
- Search cost and time: 67–80% lower search cost is achieved on Terminal-Bench 2.1, with final performance only 1.1 points below Full Search.For GPT-5.6 Luna, input tokens fall from 2,888M to 579M and time from 22.2 to 11.5 hours; for Qwen3.6, tokens fall from 741M to 246M and time from 38.0 to 20.5 hours.
- Search cost and time: Under the same 20% evaluation budget, Task-CoEvolve achieves final performance 3.3 points higher than Random-Resample.Variance-weighted selection uses 3.2M average input tokens per trial, compared with 0.7M for Random-Resample and 1.2M for Naive.
- Search cost and time: Search time is reduced by about half while performance remains close to Full Search.The reduction is limited by 10-way parallel execution and a fixed 2.4–3.3-hour candidate-proposal cost.
5 ANALYSIS
The ablation shows that resampling improves performance, while estimation and variance-weighted selection further improve comparability and final accuracy. Adaptive sampling targets changing task difficulty, but estimated-score ranking becomes weak at the smallest evaluation budget.
- Component ablation: Uniform subset resampling increases average performance from 47.2 to 48.2, the largest gain among the evaluated components.Repeatedly using the same subset appears to contribute substantially to Naive’s weaker performance.
- Component ablation: Adding full-set estimation and ˆS-max selection raises average performance from 48.2 to 48.8, providing a common comparison scale.
- Component ablation: Variance-weighted selection further increases performance to 49.3, achieving the best ablation result.
- Adaptive task selection: More than 70% of the task pool consists of extreme historical-accuracy groups that provide little information for ranking candidates.Tasks with divided candidate outcomes remain a relatively small fraction of the pool, so uniform sampling preserves an uninformative imbalance.
- Adaptive task selection: Task difficulty changes during optimization: nearly universally solved text-classification samples increase from 34 to 58, while unsolved Terminal-Bench tasks decrease from 32 to 21.These shifts motivate adapting evaluation samples as the harness evolves.
- Estimated-score accuracy: At ρ = 20%, ˆS has rank correlation 0.62 with true scores, and its selected candidate ranks 12th of 60.The selected candidate scores 51.3% versus 54.7% for the true best candidate, remaining within the top fifth.
- Estimated-score accuracy: At ρ = 7%, rank correlation falls to 0.13, but ˆS still selects the 10th-best candidate out of 60.The selected candidate scores 46.4% versus 47.8% for the true best, and the low correlation is attributed to limited sample size.
6 CONCLUSION, LIMITATIONS AND FUTURE WORK
Task-CoEvolve jointly optimizes harnesses and their validation tasks using adaptive selection and sampling-aware full-set estimation. It achieves performance close to full search at lower evaluation cost, while its estimator choice can materially affect results and its fixed per-candidate budget remains a limitation.
- Conclusion: Task-CoEvolve co-evolves validation tasks with candidate harnesses, combining discriminability-based selection with sampling-aware full-set performance estimation.
- Conclusion: On text classification, Task-CoEvolve approaches full-set search at a 7% evaluation budget and outperforms it at 20%; on Terminal-Bench, it reduces search cost by 67–80% with similar performance.
- Limitations and future work: Task-CoEvolve fixes the number of tasks evaluated per candidate before seeing results, preventing early stopping for clearly weak candidates or additional evaluation for ambiguous comparisons.
- Estimator analysis: Replacing the estimator with difference estimation lowers mean text-classification accuracy by 3.3 points at ρ=20% and 3.6 points at ρ=7%.With only two to ten sampled examples per dataset, subtracting the historical anchor reduces candidate differences and weakens comparison reliability.
- Estimator analysis: On Terminal-Bench, the H´ajek estimator can produce unstable estimates, including ˆS = 85.9% for a candidate with a raw score of 33.3%.The paper attributes the improved behavior of Equation (4) to using historical performance as an anchor.
B SENSITIVITY TO THE TIE-BREAK RULE
Tie-breaking affects baseline selections when small subsets create many exact ties, but it does not affect Task-CoEvolve and does not change the ordering of methods.
- Tie-breaking behavior: Task-CoEvolve is unaffected by tie-breaking because its continuous ˆS scores produced no tied maxima in the reported runs.
- Tie-breaking behavior: Selecting the latest tied candidate lowers Naive from 45.2 to 41.8 and Random-Resample from 47.0 to 45.6 at ρ=7%.
- Tie-breaking behavior: The tie-breaking choice does not change method ordering, so the paper uses the earliest candidate, favoring the baselines and following prior implementation.
- Tie-breaking behavior: At ρ=7%, 13–16 Naive candidates and 2–5 Random-Resample candidates share the highest score, compared with one Task-CoEvolve candidate.
C A STRONGER MODEL ON TERMINAL-BENCH 2.1
A stronger model leaves little room for harness optimization on Terminal-Bench, whereas the selected harnesses use benchmark-specific retrieval and terminal-wait strategies. These mechanisms target similarity signals and time-budget waste rather than model-weight updates.
- A stronger model: Under full-set search, DeepSeek-V4-Flash shows no harness improvement: Terminus 2 and the best candidate both reach 70.8%.The other nine candidates score between 64.0% and 69.7%.
- A stronger model: The authors attribute the lack of improvement to limited benchmark-and-model headroom, not to a Task-CoEvolve limitation.
- Selected harnesses: At ρ=20%, the selected text-classification harness retrieves examples with word-bigram and character n-gram TF-IDF indices, fusing their rankings.
- Selected harnesses: The fused retrieval uses word bigrams for legal descriptions and character n-grams for molecule strings and symptom text, avoiding a single-tokenizer compromise.
- Selected harnesses: The selected Terminal-Bench harness checks for a returned shell prompt after command keystrokes instead of always sleeping a fixed time.Fixed sleeps consumed a quarter to a third of the task budget, and about one fifth of trials timed out.
E IMPLEMENTATION DETAILS
The estimator is fixed using two starting harnesses evaluated on the full validation set before candidate generation. In this setting, the unanchored estimator is safe because low-inclusion-probability samples lie near the majority outcome.
- Two starting harnesses determine the estimator rule after full validation-set evaluation in Phase 0.This occurs before any candidate harness is generated.
- With two systems, ¯p can only be 0, 0.5, or 1, and dataset means lie near 0 or 1.
- Low-inclusion-probability samples correspond to majority-extreme outcomes close to their dataset mean, making the unanchored estimator safe here.
E.2 HYPERPARAMETERS OF EXPERIMENTS
The implementation details describe retrieval and prompt construction for online text classification, adaptive waiting for Terminal-Bench 2.1, and the use and review of LLM-assisted work. The experiments' hyperparameters are listed in Table D.
- HYPERPARAMETERS OF EXPERIMENTS: Table D lists the hyperparameters used in the experiments.
- Online text classification: Retrieval fuses word-bigram and character n-gram rankings, then greedily fills the prompt while penalizing labels already represented.
- Terminal-Bench 2.1: The Terminal-Bench harness cuts a post-keystroke wait short only when command echo, terminal quietness, and a prompt-ending line are all detected.
- Use of LLMs: The authors used LLMs for design discussions, coding assistance, and manuscript wording while retaining final scientific and implementation responsibility.