PaperlayerView exact revision on arXiv

Source-linked AI summary

Reliable Self-Evolution with Imperfect Proxy Rewards

Kangjun Noh, Soyu Kim, Kyungwoo Song

arXiv:2610.02975v1cs.AI

TL;DR

When accurate rewards are too expensive throughout iterative search, imperfect proxy rewards can admit false positives. CISE uses calibrated reward intervals for feedback and candidate selection; across material-search tasks, it returns smaller, more reliable sets than baselines, under stated coverage assumptions.

  • Problem

    High-fidelity evaluation of every candidate can be prohibitively expensive, so self-evolving search uses imperfect proxy rewards that may admit false positives.

    Source: §1 · p. 1

  • Method

    CISE combines conditional conformal inference and iteration-wise online density-ratio estimation to construct candidate-specific reward intervals for feedback and final selection.

    Source: §1 · p. 2

  • Results

    Across three material-search tasks, CISE returned smaller, more reliable candidate sets with no observed false positives among resolved DFT evaluations, while baselines included false positives.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    A smaller, more reliable shortlist can be valuable when downstream experimental validation is the bottleneck.

    Source: §5.2 · p. 8

    Coverage is fixed-iteration and depends on conditional i.i.d. calibration, an independent fresh draw, and bounded overlap under evolutionary covariate shift.

    Source: §4.3 · pp. 5–6

Abstract

from arXiv · show

Large language model (LLM)-based self-evolving search is a promising approach to scientific discovery. However, high-fidelity evaluation of every candidate is prohibitively expensive in some domains. Self-evolving systems in such settings therefore rely on low-cost but imperfect proxy rewards, which may assign high scores to infeasible candidates. These false positives may contaminate both the final output and the feedback used to guide subsequent generations. This motivates statistically calibrated reward intervals for more reliable self-evolving search. We propose Conformal Interval-Driven Self-Evolution (CISE), which constructs candidate-specific reward intervals using conditional conformal inference and iteration-wise online density-ratio estimation. CISE uses conservative interval-based rewards for evolutionary feedback and returns candidates only when all required property intervals lie entirely within their respective feasible regions. We derive fixed-iteration coverage results under explicit assumptions of independence and covariate shift. We evaluate CISE on three self-evolving search tasks in materials science. In our experiments, all candidates returned by CISE are true positives under high-fidelity evaluation, whereas the baselines return more candidates but include false positives. These results highlight the value of a smaller, more precise shortlist when downstream validation budgets are limited. Our repository is available at https://github.com/MLAI-Yonsei/CISE.git.

1 INTRODUCTION

Self-evolving search depends on reward reliability because evaluated candidates are reused to guide later generations. CISE addresses imperfect proxy rewards with calibrated intervals for both evolutionary feedback and final candidate selection.

1 INTRODUCTION

  • False-positive proxy evaluations can enter the archive as successful examples and steer later generations toward regions where the proxy is overly optimistic.High-fidelity evaluation of every candidate can be prohibitively expensive, motivating low-cost proxy rewards.Source: §1 · p. 1
  • Conditional conformal inference uses candidate characteristics to construct calibrated intervals, supporting conservative decisions about feedback and returned candidates.The motivation is to replace reliance on point-valued proxy rewards with uncertainty-aware estimates.Source: §1 · pp. 1–2
  • CISE uses candidate-specific calibrated reward intervals for evolutionary feedback and returns candidates only when their entire intervals satisfy task constraints.Iteration-wise online density-ratio estimation accounts for changes in the candidate distribution during evolution.Source: §1 · p. 2
  • CISE evaluates its approach on three material-property search tasks and reports higher observed DFT-validated output precision than baselines, which return false positives.Its fixed-iteration coverage analysis assumes independence and covariate shift.Source: §1 · p. 2

2 RELATED WORK

Prior work studies proxy-reward reliability through calibration, uncertainty estimation, and conformal inference. The paper applies conformal intervals to both evolutionary feedback and final selection in self-evolving search, targeting false-positive control under task-specific true-reward constraints.

2 RELATED WORK

  • Self-evolving search reuses evaluated candidates and rewards across domains, but proxy-favored candidates may not represent true improvements and can propagate errors across generations.This motivates accounting for proxy uncertainty in feedback and final selection.Source: §2 · p. 2
  • Earlier reward-calibration and uncertainty-aware methods improve interpretation or robustness, but do not statistically control false-positive rewards under task-specific constraints in iterative self-evolving search.The cited approaches include calibrated LLM-judge scores, confidence estimates, conservative reward-model objectives, and calibrated process rewards.Source: §2 · p. 2
  • Conformal inference provides prediction sets or intervals with finite-sample coverage under explicit assumptions, while conditional conformal methods address specified classes of covariate shift.The passage also notes applications to LLM-as-a-judge evaluation.Source: §2 · p. 2
  • CISE uses conformal inference to construct calibrated proxy-reward intervals for both evolutionary feedback and final candidate selection in self-evolving search.This builds on work applying conformal intervals to represent evaluation uncertainty in LLM-as-a-judge settings.Source: §2 · p. 3

3 PRELIMINARIES: SELF-EVOLVING AGENT FOR MATERIAL SEARCH WITH PROXY REWARD

The setup distinguishes true rewards from inexpensive proxy rewards and defines feasibility against task-specific constraints. The baseline search archives proxy-evaluated candidates and uses proxy-feasible candidates as outputs while selecting parents from the archive.

3 PRELIMINARIES: SELF-EVOLVING AGENT FOR MATERIAL SEARCH WITH PROXY REWARD

  • A false positive is a candidate that satisfies the constraints under the proxy reward but violates them under the true reward.The setup denotes the true-feasible and proxy-feasible candidate sets by Fr and Fˆr, respectively.Source: §3 · p. 3
  • At each iteration, the agent generates candidates from previous parents, evaluates them with the proxy reward, and adds candidate-reward pairs to the archive.The archive serves as retained search experience, distinct from the output set.Source: §3 · p. 3
  • Candidates satisfying proxy constraints are added to the cumulative output set, while the updated archive is used to select parents for the next iteration.The parent-selection rules are treated as given; the setup does not include a separate policy-learning step.Source: §3 · p. 3

4 METHODOLOGY

CISE uses candidate-specific reward intervals for both evolutionary feedback and output selection, adapting calibration to the current candidate distribution. Its coverage guarantees are iteration-wise and rely on explicit independence and covariate-shift assumptions.

4.1 PROBLEM SETUP

  • The method separates proxy-error quantile fitting, conformal calibration, and source-reference data for density-ratio estimation into D_fit, D_cal, and D_src.The proxy reward model, fitted quantile model, and candidate representation are fixed before evolution.Source: §4.1 · p. 3
  • CISE stores conservative interval-based rewards for parent selection but returns a candidate only when its entire valid, bounded interval satisfies the task constraints.A candidate can remain in the archive even if it fails the output criterion; lower- and upper-bound constraints use the respective interval endpoint, while range constraints require the full interval to fit.Source: §4.1 · p. 3

4.2 CONFORMAL INTERVAL-DRIVEN SELF-EVOLUTION UNDER EXCHANGEABILITY

  • The exchangeability analysis fixes the archive, selected parents, and generation configuration, and assumes the resulting candidate distribution is independent of D_cal.In this reference setting, the calibration candidates and fresh candidate share the same joint distribution over material structures and true rewards.Source: §4.2 · p. 4
  • Under exchangeability, the interval achieves iteration-wise marginal coverage when calibration and fresh pairs are independent draws from P_0 and the calibration space contains a constant function.The score and function space must be fixed independently of the pairs, and the augmented problem must be solved exactly with a permutation-symmetric dual-solution rule.Source: §4.2 · p. 5

4.3 CONFORMAL INTERVAL-DRIVEN SELF-EVOLUTION UNDER EVOLUTIONARY COVARIATE SHIFT

  • The covariate-shift setting allows material frequencies to change while preserving the conditional relationship between a structure and its true reward.The density ratio describes the change relative to the source distribution, under a bounded-overlap condition.Source: §4.3 · p. 5
  • At each iteration, CISE estimates the current density ratio from source candidates and unlabeled probes, then adds the estimated ratio as a calibration basis function for fresh candidates.Probe and fresh candidates are generated independently under the same fixed generation configuration; the augmented calibration objective retains its unweighted pinball loss.Source: §4.3 · pp. 5–6
  • Under evolutionary covariate shift, the fixed-iteration false-positive probability is bounded by α + ϵ_t under the theorem's stated conditions.Averaging over estimation data gives the corresponding marginal bound with E[ϵ_t]; when w_t ∈ G_t, the same 1−α coverage follows directly.Source: §4.3 · p. 6

5 EXPERIMENTS

The experiments compare CISE with existing material-discovery methods on three tasks, evaluating returned candidates with DFT. Results show a reliability–yield trade-off and compare methods both across task outputs and under an equal validation budget.

5.1 EXPERIMENTAL SETUP

  • The evaluation covers WBG, SSE, and PV multi-constraint tasks, compares CISE with MatterGen, LLMatDesign, and LLEMA, and validates returned candidates with DFT.Each method generates 200 structures; the solver LLM is gpt-5-mini.Source: §5.1 · pp. 7–8, Table 1 · p. 8

5.2 EMPIRICAL RESULTS

  • LLEMA returned 102 WBG and 104 SSE candidates, with 7.84% and 31.73% failing DFT, while CISE returned 41 WBG and 51 SSE candidates, all confirmed true positives among resolved evaluations.The comparison illustrates a smaller, more reliable shortlist for downstream validation.Source: §5.2 · p. 8
  • With the same six-candidate validation budget on PV, three selected LLEMA candidates were DFT false positives, while all six CISE candidates were true positives.LLEMA’s six candidates were ranked by proxy-predicted band-gap closeness to 1.2 eV before DFT evaluation.Source: Table 2 · p. 9, §5.2 · p. 9
  • CISE admits an output only when every required property interval lies entirely within its feasible region, so point-feasible candidates may still be excluded.Its archive-admission and output-selection steps modify LLEMA, while the generator and evolutionary operators remain unchanged.Source: §5.1 · p. 7, §5.2 · p. 9
  • On SSE, later LLEMA candidates increasingly fell above the DFT formation-energy threshold, while CISE showed no comparable accumulation and kept its output mean feasible throughout evolution.Figure 2 plots proxy rewards or CISE intervals alongside DFT evaluations over 20 iterations.Source: Figure 2 · p. 8, §5.2 · p. 9

6 CONCLUSION

The conclusion presents CISE as an approach to self-evolving search when obtaining true rewards throughout the search is too expensive. Across three material-discovery tasks, it reports smaller, more reliable DFT-validated outputs and improved reliability under a matched validation budget.

6 CONCLUSION

  • CISE replaces point-valued proxy rewards with calibrated intervals built using conditional conformal inference and iteration-wise online density-ratio estimation.The intervals guide evolutionary feedback and final candidate selection.Source: §6 · p. 9
  • Under explicit independence and covariate-shift assumptions, the paper provides fixed-iteration coverage guarantees and characterizes the effect of density-ratio estimation error.Source: §6 · p. 9
  • Across three material-discovery tasks, CISE returns smaller candidate sets with no observed false positives among DFT-validated outputs and improved reliability under a matched validation budget.Source: §6 · p. 9

A.1 IMPLEMENTATIONS

The implementation combines a fixed proxy model with conditional conformal calibration and probes that track changes in the candidate distribution during evolution.

A.1 IMPLEMENTATIONS

  • The calibration set contains 10,000 QETB material structures, separated from model-fitting data at the composition-group level.Source: §A.1 · p. 12
  • A conditional quantile regression model is trained on 423,205 QETB structures, and the proxy models remain fixed throughout evolution.Source: §A.1 · p. 12
  • At each generation context, eight valid probe structures generated with the main candidates’ parent structures and prompt context characterize the current candidate distribution for online density-ratio estimation.Source: §A.1 · p. 12
  • The conditional quantile model uses gradient-boosted regression with quantile loss to predict the conditional 0.95 quantile of absolute proxy error from 27 structural features and the proxy prediction.Source: §A.1 · p. 12
  • Conditional conformal inference uses structural features for heterogeneous uncertainty and incorporates the online density ratio into its finite-dimensional Gibbs calibration basis.The ratio is estimated from probe structures and a fixed source set.Source: §A.1 · p. 12
  • Selected structures are evaluated independently of proxy models using Quantum ESPRESSO with a PBEsol fixed-structure protocol and Modified GBRV PBEsol pseudopotentials.Source: §A.1 · p. 12

A.2 DETAILED TASK DESCRIPTIONS •

The detailed tasks impose different property constraints, with several requiring multiple predicted properties and others including composition-based criteria.

A.2 DETAILED TASK DESCRIPTIONS

  • WBG candidates must have Eg ≥2.5 eV and Eform ≤−1.0 eV/atom; the task represents proxy and DFT rewards by the corresponding band-gap and formation-energy pairs.Source: §A.2 · p. 12
  • SSE candidates must have Eform ≤−1.0 eV/atom, Eg ≥2.0 eV, and at least one mobile species from Li, Na, K, Mg, Ca, or Al.The mobile-species condition is determined directly from composition and does not require proxy calibration.Source: §A.2 · pp. 12–13
  • PV candidates must have 0.7 ≤Eg ≤2.0 eV and Eform ≤0.0 eV/atom, with compositions restricted to earth-abundant, non-toxic elements.The elemental restriction is evaluated directly from composition; Figure 3 illustrates the PV band-gap interval criterion.Source: §A.2 · p. 13, Figure 3 · p. 13

B ADDITIONAL EMPIRICAL RESULTS

Proxy-qualified false positives can be reused as feedback across generations, while CISE intervals reject uncertain successes and retain a candidate whose full interval is feasible.

B ADDITIONAL EMPIRICAL RESULTS

  • LLEMA repeatedly reuses proxy-qualified Na2O → Li2O → Na2O candidates, although DFT identifies all three as false positives.Recorded rationales refer to previous surrogate evaluations, showing that an erroneous reward becomes reusable feedback.Source: B ADDITIONAL EMPIRICAL RESULTS · pp. 13–14
  • CISE intervals extend beyond the feasible band-gap range for those candidates, while RbLiO’s full interval lies within the target range and DFT confirms it as a true positive.The examples illustrate suppressing uncertain proxy successes while retaining candidates whose interval rewards remain feasible.Source: B ADDITIONAL EMPIRICAL RESULTS · p. 14

C PROOFS

  • The proof setup fixes an iteration and assumes measurable scores, basis functions, and solution rules, with a symmetric dual-solution rule.It interprets the supremum in Equation 3 in the extended real line, taking sup ∅ = −∞.Source: C PROOFS · p. 14

C.1 PROOF OF THEOREM 1

Theorem 1’s proof uses an exchangeable augmented calibration problem and a symmetric dual rule to relate interval miscoverage to a bounded dual event.

C.1 PROOF OF THEOREM 1

  • At the true test score, the augmented calibration problem is exchangeable, and the symmetric dual-solution rule supports the coverage argument.The proof conditions on the independently fixed score and calibration space.Source: §C.1 · pp. 14–15
  • The pinball loss is represented as a maximization over η in [−α, 1−α], enabling a finite-dimensional linear-program dual formulation.The calibration space is finite-dimensional, and the stated constraints yield a nonempty compact dual feasible set with an optimal solution.Source: §C.1 · p. 14
  • When ηN < 1 − α, the true test score falls below the cutoff and the true reward lies in its prediction interval.The proof uses pointwise inequalities linking the miscoverage event to the dual boundary event.Source: §C.1 · p. 15
  • The dual argument also establishes that ηs(X′) is nondecreasing in the imputed score s, for any choices of optimal dual solutions.The result remains valid when G is replaced by a fixed Gt.Source: §C.1 · p. 15

C.2 PROOF OF THEOREM 2

Theorem 2’s proof compares the current-iteration target distribution with an auxiliary shifted law, then bounds the coverage change under covariate shift. With an exact ratio in the calibration space, the proof obtains coverage without that comparison penalty.

C.2 PROOF OF THEOREM 2

  • Under the auxiliary target law, changing only the fresh pair’s distribution gives interval miscoverage at most α while leaving the fitted score, ratio estimate, calibration space, and interval rule unchanged.The proof conditions on the current generation state and uses independent calibration pairs from P0.Source: §C.2 · pp. 15–16
  • Replacing the auxiliary target law with Pt increases miscoverage by at most ϵt, yielding the theorem’s conditional guarantee and a marginal bound involving E[ϵt].The comparison uses the total-variation distance between the two target laws.Source: §C.2 · pp. 16–17
  • If wt ∈ Gt, the weighted argument applies directly to Pt and gives coverage at least 1 − α without a total-variation comparison.When bwt = wt, ϵt = 0.Source: §C.2 · p. 17
  • The output rule bounds the joint event that a fresh proposal is accepted and is a false positive, not the false-positive probability conditional on acceptance.The result does not assert the same bound simultaneously over the full search trajectory.Source: §C.2 · p. 17
Loading 2610.02975v1…