Source-linked AI summary
When Better Gets Worse: Improvement Fidelity for Self-Improving Agents in Adaptive Worlds
Ke Wang, Zijie Zhao, Zhiyi Yuan, Changlun Li
TL;DR
Self-improving agents can select proxy-improved updates that become worse when deployment changes the evaluation world. The paper formalizes this gap as Improvement Fidelity and introduces PIVOT-KG to target costly deployment evaluations; experiments support the update-level view.
Problem
Self-improving agents can choose proxy-improved updates that worsen after deployment changes their evaluation world, so policy-level accuracy alone does not establish reliable evaluation of proposed updates.
Source: §1 · pp. 1–2Method
PIVOT-KG is a paired, decision-aware validator that directs limited high-fidelity evaluation toward uncertainty that could change the replacement decision.
Source: §1 · pp. 1–2Results
Experiments across controlled, multi-agent, and reactive-driving simulations support the paper's update-level evaluation view.
Source: §7 · p. 8Takeaways & Limitations
Reliable self-improvement concerns whether proposed updates remain improvements after deployment, with scarce evidence allocated when it can affect replacement decisions.
Source: §1 · p. 2The response models do not cover all deployment dynamics, and validation depends on a calibrated correction model and sufficiently informative high-fidelity observations.
Source: §7 · p. 8
Abstract
from arXiv · showhide
Self-improving agents increasingly rely on proxy verifiers to choose policy updates, yet deployment can change the world in which those updates are evaluated. An update that looks better to the verifier can therefore become worse after deployment even when the verifier ranks policies well overall. We formalize this gap as Improvement Fidelity, which asks whether proxy improvements preserve the sign and ordering of deployment improvements over the updates an improvement process actually proposes. We show that global policy accuracy need not guarantee update fidelity: operator shift and deployment response can create update-level errors, while candidate margins determine whether those errors change the replacement decision. We introduce PIVOT-KG, a paired, decision-aware validator that allocates scarce high-fidelity evaluation according to the expected reduction in selection regret per unit cost. Across 90 held-out roots in Leduc, Kuhn, and Melting Pot, proxy and deployment optimal sets are disjoint in 51 cases. In an eight-candidate HighwayEnv stress test, PIVOT-KG reduces mean improvement-selection regret from 0.0435 under the exact Uniform validation rule to 0.0055 at the primary budget. Together, these results show why reliable self-improvement should evaluate proposed improvements in the worlds they induce, while providing a practical rule for allocating scarce deployment evidence when it can affect the replacement decision.
1 Introduction
Self-improving agents can select proxy-improved updates that become worse after deployment because an update may change the world in which it is evaluated. The paper defines Improvement Fidelity for these proposed transitions and introduces PIVOT-KG to direct scarce high-fidelity evaluation toward uncertainty that could change a replacement decision.
1 Introduction
- A verifier can rank policies well overall yet choose the wrong local replacement when deployment changes the evaluation world.Lane changes alter nearby trajectories, and actions in multi-agent games change how others respond.Source: §1 · p. 1
- Improvement Fidelity asks whether a verifier-side improvement remains an improvement after deployment in the world induced by the update.The criterion focuses on updates an improvement process actually proposes, rather than policy evaluation in general.Source: §1 · pp. 1–2
- PIVOT-KG allocates limited high-fidelity evaluation to uncertainty that could change the replacement decision, rather than global prediction error.It is the knowledge-gradient instantiation of PIVOT, a paired, decision-aware validator.Source: §1 · pp. 1–2
- The paper tests its predicted failure mechanisms across controlled response worlds, multi-agent environments, and reactive driving simulations.Its stated aim is to identify when targeted high-fidelity validation reduces selection regret.Source: §1 · p. 2
2 Related Work
Related work studies deployment-induced response, policy evaluation and safe improvement, verifier reliability, and decision-aware allocation. This paper instead focuses on whether proposed replacements remain improvements in their induced worlds and how to allocate deployment evidence for that decision.
2 Related Work
- Work on performative prediction and reinforcement learning examines how deployment-induced changes affect learning, stability, and optimality (Mandal et al., 2023; Rank et al., 2024; Mandal and Radanovic, 2025; Góis et al., 2025; Piliouras and Yu, 2022).This paper asks the local evaluation question of whether a proposed replacement remains an improvement under its induced response world.Source: §2 · p. 2
- Off-policy evaluation and safe policy improvement compare candidate performance with a baseline (Jiang and Li, 2016), while learned-simulator work studies policy values, value gaps, or ranking preservation (Quevedo et al., 2026; Dann et al., 2026).The paper focuses on whether a proposed transition produces the right replacement decision after deployment changes the evaluation world.Source: §2 · p. 2
- Prior self-improvement and verification research documents iterative improvement and verification regressions (Wang et al., 2026; Zhang et al., 2025; Wu et al., 2025; Guo et al., 2026), while reward-overoptimization work studies degradation from optimizing imperfect proxies (Gao et al., 2023; Laidlaw et al., 2025).The paper's complementary failure can occur with a fixed external verifier when deployment changes the evaluation world.Source: §2 · pp. 2–3
- Knowledge-gradient, ranking-and-selection, and best-arm methods allocate costly observations to decision-relevant alternatives.PIVOT-KG applies this allocation principle to proxy–deployment validation and replacement decisions.Source: §2 · p. 3
- The paper's gap is operator-relative and update-level: whether proposed replacements preserve improvement signs and ordering in the worlds they induce, and how to allocate scarce evidence when uncertainty can change the decision.This distinguishes the focus from policy-level accuracy or ranking alone.Source: §2 · pp. 2–3
3 Improvement Fidelity: Problem Formulation
The formulation distinguishes verifier-estimated improvement from deployment improvement in the world induced by a policy. It evaluates fidelity over the transitions an improvement operator proposes and defines metrics for transition errors, decision regret, and deployment response.
3 Improvement Fidelity: Problem Formulation
- For a replacement from π to π′, ∆V is the verifier-estimated improvement and ∆∗ is the deployment improvement in the world induced by the policy.The deployment world M[π] includes the environment response to policy π.Source: §3 · p. 3
- Improvement Fidelity error averages a transition-level loss over the proposed replacements generated by improvement operator A.The relevant population is the updates the process actually proposes, and smaller error indicates greater fidelity.Source: §3 · p. 3
- IDE measures absolute improvement error, ISC sign agreement excluding ties, IRR proxy-positive reversals, and ISR the consequence for replacement decisions.The metrics distinguish transition-level fidelity diagnostics from decision-level regret.Source: §3 · p. 3
- Direct response holds the incumbent world fixed, actor response lets the environment respond to the candidate, and strategic response additionally lets other agents adapt through R−i.Without other-agent adaptation, actor improvement equals deployment improvement.Source: §3 · p. 3
4 Properties of Improvement Fidelity
Improvement Fidelity concerns whether verifier improvements remain improvements after deployment in the world induced by an update. Global policy accuracy alone need not ensure update fidelity, and update errors affect replacement decisions relative to candidate margins.
From policy fidelity to update fidelity
- A uniform policy-value error bound of ε implies update error at most 2ε, preserving the sign when |∆∗| > 2ε.The bound makes uniform value fidelity sufficient for update-level fidelity.Source: §4 · p. 4
Global policy accuracy need not imply update fidelity
- Global policy MAE and Spearman rank error can be arbitrarily small while operator-induced updates have IRR = 1 and ISC = 0.An adjacent-policy value swap yields proxy-positive updates that are deployment-negative under the constructed operator.Source: §4 · p. 4
Decision preservation
- When the best deployment update is unique, a margin m > 2εC preserves the top-1 ordering under maximum update error εC.Replacement decisions depend on whether fidelity errors are large relative to candidate margins.Source: §4 · p. 4
Operator shift and deployment response
- Operator shift and deployment response are distinct sources of update-level error, and the actor–direct gap isolates candidate-value change caused by its induced world.Strategic evaluation additionally includes other-agent response through R−i.Source: §4 · p. 4
5 PIVOT-KG: Decision-Aware Validation
PIVOT-KG uses paired proxy–deployment validation to allocate scarce high-fidelity evaluations according to their expected effect on the replacement decision. Its fixed-budget procedure queries candidates sequentially and updates a calibrated correction model from paired observations.
PIVOT-KG: Decision-Aware Validation
- PIVOT-KG directs high-fidelity queries toward uncertainty that can change the replacement decision, using a calibrated correction model over candidate-wise values.The model's prior parameters and observation variances are estimated on a disjoint calibration set, so held-out roots start without using their HF outcomes.Source: §5 · p. 4
- Each query evaluates the incumbent and candidate under matched initial state, randomness, and opponent initialization, then updates the joint posterior by Gaussian conditioning.Correlated candidate corrections allow evidence to update unqueried candidates; each candidate is queried at most once in the fixed-budget procedure.Source: §5 · p. 5
- PIVOT-KG queries the feasible candidate with the largest expected reduction in selection regret per unit high-fidelity cost.The fixed-budget algorithm repeats this choice while a feasible candidate remains, then returns the posterior replacement decision.Source: §5 · p. 5
- The fixed-budget rule is the primary procedure; posterior-based early stopping is a secondary variant.Source: §5 · p. 5
6 Experiments
The experiments test how operator shift and deployment response affect update fidelity, whether those errors change replacement decisions, and whether targeted high-fidelity (HF) validation reduces selection regret. They evaluate the approach in controlled, multi-agent, and reactive driving simulations.
6 Experiments
- The experiments progress from isolating fidelity-failure mechanisms to testing decision consequences and budgeted validation, ending with a reactive-driving stress test.All evaluations are in simulation; held-out multi-agent studies include 30 roots per environment, and selection permits a no-update option.Source: §6 · p. 5, §6.1 · pp. 5–6
6.2 Improvement Fidelity under Operator Shift and Deployment Response
- The pooled high-minus-low contrast in operator-relative IDE is 0.2627 [0.1684, 0.3763], and local update error and reversals increase with designed shift intensity.Global policy rank remains a fixed calibration reference, so global accuracy can mask failure on proposed updates.Source: §6.2 · p. 6
- Strategic reversal is 0.9495, while proxy-positive reversal rises from 2.5% under one-step response to 51.7% under eight-step adaptation.These findings distinguish deployment response from operator shift as a source of update-level failure.Source: §6.2 · p. 6
6.3 Decision Relevance and High-Fidelity Resolvability
- Across 90 held-out roots, proxy and deployment optimal sets are disjoint in 51 cases, but disagreement alone does not determine validation value.At the primary long budget, PIVOT's query set intersects the ex-post decision-critical pair in 0.97 [0.92, 1.00] of roots; the model-assumed queried observation SD averages 13.99 [7.97, 22.90] times the proxy top-2 margin, so targeting need not resolve ordering.Source: §6.3 · p. 6, Figure 3 · p. 7
6.4 PIVOT-KG under Limited High-Fidelity Budgets
- PIVOT-KG substantially lowers ISR in Leduc, the contrast is small in Kuhn, and the primary Melting Pot comparison is unresolved.These are the primary held-out fixed-budget comparisons.Source: §6.4 · p. 7, Table 1 · p. 7
- PIVOT-KG and Uniform share the calibrated posterior, observation-update rule, and final selection rule, differing only in query allocation.A separate Leduc confirmation cohort reports a selected-gain contrast of +0.02989; its repeated-query, sample-size variant is specified in Appendix B.Source: §6.4 · p. 7
6.5 Reactive Driving Stress Test
- At B = 4, PIVOT-KG reduces ISR relative to the exact Uniform validation rule by 0.0380 [0.0300, 0.0474].The redesigned HighwayEnv stress test uses eight scheduled candidates and 120 disjoint test roots.Source: §6.5 · pp. 7–8
7 Conclusion
The conclusion frames reliable self-improvement as an update-level evaluation problem: errors matter when they can change replacement decisions. PIVOT-KG directs scarce HF evidence toward consequential uncertainty, while the study's response models, calibration, and driving evidence limit its scope.
7 Conclusion
- Operator shift and deployment response can create update-level errors, while candidate margins determine whether those errors change the replacement decision.PIVOT-KG allocates scarce HF evidence toward uncertainty that can affect the selected update.Source: §7 · p. 8
- The response models do not cover all deployment dynamics, and PIVOT-KG depends on a calibrated correction model and sufficiently informative HF observations.Noisy evidence or very small budgets can leave decision-critical uncertainty unresolved; held-out intervals also condition on the fitted calibration model.Source: §7 · p. 8
- Future work includes richer response and correction models, repeated self-improvement under co-adaptation, and intervention-based HF evaluation in more realistic deployment systems.The conclusion frames the central question as whether replacement remains an improvement in the world it creates.Source: §7 · p. 8
A Response Mechanism Analyses
The response analyses distinguish response levels and test whether longer deployment adaptation increases update-level reversals. In the Leduc confirmation cohort, reversal among proxy-positive candidates rises sharply under eight-step adaptation.
Response mechanisms
- The response-level analysis pools Performative Control, Congestion Resource, and frozen MPE2 with weights 0.4/0.4/0.2.The operator contrast is a descriptive shift diagnostic, not a measured χ2 divergence; the strategic reversal averages three adaptive families.Source: A Response Mechanism Analyses · p. 11
- Figure 5 compares response-disabled proxy, actor, and strategic evaluations, then shows strategic-minus-actor effects by opponent family and actor-versus-strategic improvement.Intervals use 30 matched seeds.Source: Figure 5 · p. 11, A Response Mechanism Analyses · p. 11
Response-horizon confirmation
- 51.7% under eight-step adaptation versus 2.5% under one-step response is the mean root-level reversal rate among proxy-positive candidates.A reversal means negative deployment gain; the comparison uses a separate Leduc confirmation cohort.Source: A Response Mechanism Analyses · p. 11
- Figure 6 reports proxy-positive reversal rates across 30 roots under one-step and eight-step adaptation, with 95% root-bootstrap intervals.A reversal is defined as negative deployment gain.Source: Figure 6 · p. 11
B Extended PIVOT-KG Evaluation
The extended evaluations examine whether targeted high-fidelity validation improves selection, how query targeting relates to resolution, and whether savings or benefits persist across settings. Results vary by environment and analysis: the Leduc findings are strongest, while other comparisons include small, unresolved, or mixed effects.
Decision diagnostics
- At the primary long budget, query-hit rates are 1.00 in Leduc, 1.00 in Kuhn, and 0.90 in Melting Pot; normalized gain is 0.113 [0.055, 0.180] among 87 hit roots and -0.195 [-0.584, 0.000] among three misses.The diagnostics distinguish successful targeting from successful resolution.Source: B Extended PIVOT-KG Evaluation · p. 12
Secondary and confirmation analyses
- Only Leduc meets the joint primary and allocation-interaction criterion, with designated interactions of 0.0365 [0.0145, 0.0637] in Leduc, 0.0020 [0, 0.0058] in Kuhn, and 1.4815 [−0.3930, 4.0956] in Melting Pot.The first cohorts use 12 calibration and 30 test roots with one frozen Uniform draw.Source: B Extended PIVOT-KG Evaluation · p. 12
Leduc confirmation cohort
- At 16,384 hands, the second Leduc cohort’s prespecified long-adaptation contrast against exact-subset expected Uniform is +0.02989.This cohort is a mechanism confirmation study, not a direct replication; Figure 7’s final row compares allocation contrasts across response horizons, not policy gains directly.Source: B Extended PIVOT-KG Evaluation · p. 12, Figure 7 · p. 13
Stopping analysis
- The secondary stopping rule reduces mean HF use from 16,384 to 12,083.2 hands, a 26.25% reduction, while matching the full-budget selected gain on all 30 observed roots.The paired HF-use difference is −4300.8 [−6007.5, −2662.4] hands.Source: B Extended PIVOT-KG Evaluation · p. 13
Closed-loop and numerical diagnostics
- Legacy batch-selector diagnostics use forced replacement and candidate-only CISRC, excluding no-update; negative improvement can therefore occur with zero regret.The query-count and regret/query diagnostics use common cost scaling with varying random draws and do not identify a cost-misspecification effect.Source: B Extended PIVOT-KG Evaluation · p. 13
Secondary analyses
- Additional response-mixture settings yield positive, null, and negative allocation effects, so deployment response alone does not determine validation value.Table 2 presents secondary held-out comparisons; positive Uniform −PIVOT denotes lower ISR for PIVOT-KG, with 95% root-bootstrap CIs.Source: Table 2 · p. 12, B Extended PIVOT-KG Evaluation · p. 13
C Reactive Driving Stress Tests
The reactive-driving tests include HighwayEnv cohorts and an eight-candidate redesign, alongside MetaDrive as a boundary case. The prespecified MetaDrive comparison is unresolved, while the evaluator-boundary analysis reports that a transition-specific learner does not outperform the global learner in its out-of-distribution comparison.
HighwayEnv
- The HighwayEnv cohorts use 20 calibration and 60 test roots each, a shared five-action panel, and B = 2 as the primary budget.The original and fresh-seed replication cohorts do not overlap; Uniform retains proxy estimates for unqueried candidates, whereas PIVOT-KG uses its calibrated posterior.Source: C Reactive Driving Stress Tests · p. 13
- The eight-candidate redesign uses eight scheduled candidates, 40 calibration roots, and 120 disjoint test roots, with B = 4 primary and B = 2 secondary.Uniform averages exact queried subsets; PIVOT-KG uses sequential Gaussian updates, and each query is a paired actor rollout.Source: C Reactive Driving Stress Tests · pp. 13–14
Legacy diagnostics
- Figures 8 and 9 report legacy forced-replacement trajectories and batch-selector diagnostics, respectively; Figure 9 varies Monte Carlo draws under common cost scaling.The Figure 9 diagnostic is not an identified cost-misspecification effect.Source: Figure 8 · p. 14, Figure 9 · p. 14
MetaDrive
- At MetaDrive’s prespecified B = 2 budget, the paired PIVOT-KG–Uniform contrast is +1.89 [−1.29, 5.49], leaving the primary comparison unresolved.The cohort has 12 profiles and 30 held-out roots sharing one fixed bottleneck geometry; B = 1 and B = 4 are secondary.Source: C Reactive Driving Stress Tests · p. 15, Figure 10 · p. 15
Evaluator boundary
- The transition-minus-global ISC contrast is -0.2590 [-0.3588, -0.1707] in the out-of-distribution comparison; the transition-specific learner does not outperform the global learner.Full ISC and IDE comparisons are reported in the supplement.Source: C Reactive Driving Stress Tests · p. 15