Source-linked AI summary
An Explicit Feedback Contract for Iterative ML-Agent Harnesses
Kehui Yao, Chuanbo Hu, Jiao Chen, Ramin Giahi, Reza Yousefi Maragheh, Lalitesh Morishetti, Ziwei Hong, Yiyun Su, Jianpeng Xu, Evren Korpeoglu, Kannan Achan
TL;DR
Iterative agents receive exact validation scores after every attempt, so adaptively selected scores can cease to represent held-out performance. The paper proposes a feedback contract that restricts what agents observe and releases scores only when improvements clear a noise-based threshold. Across tasks, the contract narrows real-data gaps and avoids reporting progress on a no-signal control.
Problem
Method
The contract restricts the agent's feedback and releases a new score only when a candidate beats the current best by at least the paired standard error.
Source: §1 · p. 2, §4.2 · p. 5, §7 · p. 9Results
0.0157 gain was reported by exact feedback on the no-signal control, while the contract's reported gain matched its test score; on real data the gap narrows at no cost.
Source: §5.2 · p. 8, §7 · p. 10Takeaways & Limitations
Runs under the contract end when further improvements cannot be certified rather than when the submission budget is exhausted.
Source: §1 · p. 2, §7 · p. 10The real-data evidence covers two tabular datasets and three seeds, which cannot distinguish a small effect from none.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Iterative ML agents develop models by editing code, running experiments, reading a validation score, and choosing what to try next. In current harnesses this loop is served by an exact validation score returned after every attempt, which leaves the validation split open to adaptive selection. In this work, we introduce an explicit feedback contract for ML-agent harnesses, which specifies what an agent may observe, when iteration should stop, and when the test split may be touched. The contract releases a validation score only when a submission clears a noise threshold computed from the paired standard error, so no margin has to be tuned. We ran experiments on click-through-rate prediction, learning to rank, and a control task with no signal to find. The control is the sharpest case, because the true gain there is zero by construction: an agent reading exact scores ended by reporting a gain of 0.0157, while under the contract it reported 0.0033. On both real tasks the contract narrows the validation-test gap, and test performance is slightly higher rather than lower. The largest difference is in when the agents stop. Exact feedback used its entire budget of thirty submissions on two of the three tasks, including the one where nothing can be learned, while the contract halted after four to twelve. A run that exhausts its budget is usually read as diligence. It is better read as a symptom: when the feedback channel is unbounded, noise keeps supplying improvements to chase.
1 Introduction
Iterative agents receive exact validation scores after every attempt, allowing adaptive selection to turn finite-sample noise into apparent gains. The paper proposes treating feedback as a protocol-design problem and reports that its contract narrows the validation–test gap while stopping earlier than unbounded feedback.
1 Introduction
- Repeatedly selecting edits that improve noisy validation scores makes the final score describe a particular split rather than the population.The same-size real improvement and random fluctuation look identical inside the loop, so the agent keeps either one.Source: §1 · p. 1
- 10.4 ± 4.4 rounds under the contract versus 30.0 ± 0.0 for Exact, illustrating earlier stopping.The contract narrows the validation–test gap without lowering final performance and uses a fraction of the submissions required by an unbounded interface.Source: Figure 1 · p. 2, §1 · pp. 2–3
- A single agent can query one validation split as many times in an afternoon as a competition receives from hundreds of teams over months.The harness has no rate limit or signal for when gains stop being real.Source: §1 · p. 2
- Existing threshold-based leaderboard mechanisms do not address agent file access, stopping rules, or leakage of validation predictions.The proposed contract specifies what agents may see, when they stop, and when the test set may be used, without parameter tuning.Source: §1 · p. 2
2 Related Work
Agentic ML systems commonly preserve an exact-score interface while replacing a competition submitter with an agent that can query it repeatedly. Existing adaptive holdout mechanisms offer privacy-budget perturbation or thresholded releases, but their assumption that submitters cannot access raw data does not hold for agents with filesystem access.
2.1 Agentic ML systems and their evaluation interfaces
- Agentic ML systems place agents inside evaluation loops that return exact metrics as often as requested, while benchmarks carefully specify splits and budgets.Source: §2.1 · p. 3
2.2 Mechanisms for adaptive reuse
- Adaptive holdout reuse can perturb answers under a privacy budget or threshold score releases, but both routes assume submitters cannot access raw data.That assumption breaks for agents with filesystem access.Source: §2.2 · p. 3
3 Problem Formulation
The formulation treats model development as an adaptive sequence of candidates chosen using prior harness feedback. It distinguishes population risk from noisy validation estimates and focuses on keeping released scores close to the best-so-far candidate’s risk.
3 Problem Formulation
- Each round submits a candidate for validation scoring, but the harness withholds the losses and releases only a feedback pair.The agent can use the training split freely, reaches validation only through the harness, and does not access the test split until after stopping.Source: §3 · pp. 3–4
- A sequence is admissible only when each candidate depends on training data, repository state, earlier edits, and prior feedback—not validation labels, per-example losses, prediction files, or test feedback.Source: §3 · p. 4
- Because candidates are chosen adaptively, validation estimates can become dependent on the validation sample, so the lowest observed score may describe the split rather than population risk.For a model fixed in advance, the validation estimate is unbiased; that guarantee lapses along an adaptively chosen path.Source: §3 · p. 4
- The harness aims to keep the released score close to the risk of the best candidate submitted so far, rather than promise accuracy for every released candidate.The release rule must also leave enough information for the agent to choose among its candidates.Source: §3 · pp. 4–5
4 The Feedback Contract: Thresholded Best-So-Far Release
The contract limits what leaves the harness, when iteration stops, and when the test set is used. It releases a new score only when a candidate’s improvement clears a data-dependent noise threshold, supporting best-so-far accuracy and finite stopping without tuning a fixed margin.
4.1 The contract: five clauses
- The contract keeps validation labels, per-example losses, prediction files, and the test split inside the harness, releasing only a changed-score indicator and score; rejected submissions reveal no miss size.Source: §4.1 · p. 5
- Iteration stops at a fixed budget or after p consecutive non-updates, and the final champion is evaluated once on the test split after it is named.Patience counts accepted updates, not score changes.Source: §4.1 · p. 5
4.2 A parameter-free release rule
- A candidate replaces the champion only when its improvement exceeds the standard error of paired per-example loss differences, so the threshold rises on noisy tasks and falls on clean ones.Accepted scores are rounded to multiples of 1/n.Source: §4.2 · p. 5
4.3 Adaptive validity of the released transcript
- For every admissible sequence, the fixed-margin variant achieves best-so-far public score error O(n^-1/3 log^1/3(kn)) with high probability when η has that order.The algorithm’s minimum-margin floor makes the transcript count finite while the margin above that floor remains data-dependent and untuned.Source: §4.3 · p. 6
- The dependence on the number of submissions k is logarithmic, so submitting more often does not move the released score.Source: §4.3 · p. 6
- With a fixed margin, a work batch has at most ⌈1/η⌉+1 accepted updates, bounding the stopping round by min{Tmax, p(⌈1/η⌉+2)}.Source: §4.3 · p. 6
5 Results
Across the three tasks, the feedback interface affected stopping behavior and validation–test outcomes. Sensitivity analyses examined public-validation size, update margins, and agent backbone.
5.1 Experimental Setup
- Experiments compare exact feedback with the contract on Criteo AUC, MSLR macro NDCG@10, and a permuted-label no-signal control.The control's base rate is population-optimal; for MSLR, the guarantee uses roughly 2,000 validation queries rather than rows.Source: §5.1 · p. 7
5.2 Feedback interfaces compared
- 0.0157: Exact's reported gain on the no-signal control, with a 0.0163 validation–test gap; the contract left no gap.On both real datasets, the gap narrowed by roughly a tenth and test performance was slightly higher.Source: §5.2 · p. 8
5.3 Sensitivity to public-validation size
- At every public size, Ours had a smaller signed gap and higher mean test AUC; both advantages were largest at 2,000 examples and shrank as size grew.At 20,000 examples, both methods recovered the main-table test AUC, with the training split fixed across sizes.Source: §5.3 · p. 8, Table 3 · p. 8
5.4 Fixed margins versus the paired standard error
- The paired-SE rule achieved the smallest gap and highest test AUC on the fixed-margin grid, without stopping sooner.The rows shared the harness, prompt, split, budget, and patience, differing only in update margin.Source: §5.4 · pp. 8–9, Table 4 · p. 8
5.5 Sensitivity to the agent backbone
- Under Codex, the contract cut mean stopping from the cap to twelve while slightly improving test AUC and reducing the gap; Claude stopped at nearly the same round under both interfaces.The Claude runs were not rerun under the current protocol and should be read separately.Source: §5.5 · p. 9, Table 5 · p. 9
6 Discussion and Limitations
Exact feedback leads agents to report progress even on data with no learnable signal and to keep using their full submission budget. The contract narrows the validation-test gap on both real datasets, but its guarantee and empirical evidence have important limits.
6 Discussion and Limitations
- With no signal to learn, exact-score feedback still led the agent to report an improvement, while exact feedback ran to the budget even on that control task.The paper presents the no-signal control as its sharpest evidence of the feedback problem.Source: §6 · p. 9
- The contract narrows the gap on both real datasets, with slightly higher rather than lower test performance, but the differences are small and three seeds cannot distinguish a small effect from none.Both real datasets are tabular.Source: §6 · p. 9
- The guarantee concerns the released transcript, not the final test score; the deployed paired-standard-error threshold is a heuristic, and thresholded feedback provides only a selection signal.Error analysis therefore has to use training data or a separately labelled sample.Source: §6 · p. 9
7 Conclusion
The conclusion frames exact validation feedback as selection against a fixed sample and presents a contract governing observations, stopping, and test access. It reports that the contract avoids apparent learning on a no-signal control, narrows the gap on real data, and stops runs when improvements can no longer be certified.
7 Conclusion
- The contract fixes what an agent may see on each split, when its loop should end, and when test data may be touched.It releases a new score only when a submission beats the current best by more than the noise in that comparison.Source: §7 · p. 9
- On the no-signal control, the contract reports nothing learned while exact feedback reports progress; on real data, it narrows the gap at no cost to final performance.Runs end when improvements run out rather than when the budget does.Source: §7 · p. 10