PaperlayerView exact revision on arXiv

Source-linked AI summary

False Frontiers: Diagnosing and Mitigating Co-Cheating in Self-Evolving Search Agents

Meijia Chen, Hao Li, Zheng Lu, Hongshan Lin, Junbai Tian, Yichen Liu, Zijun Tian, Yufan Zou, Shuhan Sun, Hanxin Chen, Zeyu Zhang, Weizhi Du, Yueting Li, Tianyu Shi, Alaa Khamis

arXiv:2609.39102v1cs.CLcs.AIcs.LG

TL;DR

In self-evolving search, proposer–solver agreement can rise through shared errors rather than improved correctness. CrossFit changes proposer feedback by using auxiliary solvers trained on complementary source folds. It reduces false agreement and improves average performance across seven search benchmarks at both model scales.

  • Problem

    In proposer–solver self-evolution, agreement is an endogenous proxy for correctness, so shared incorrect labels can reinforce errors across rounds.

    Source: §1 · p. 2

  • Method

    CrossFit scores questions from each source fold with an auxiliary solver trained on the other fold, while the main solver continues training on all admitted questions.

    Source: §1 · p. 2

  • Results

    Across seven benchmarks, CrossFit improves average Cover-EM over Dr. Zero by 8.8 and 8.4 points at 4B and 9B, and over Search-R1 by 8.7 and 7.8 points.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    The findings motivate tracking how an evaluator acquired its supervision when designing self-generated curricula.

    Source: §7 · p. 10

    Shared pretraining errors, overlapping or connected evidence, and auxiliary cost remain limitations; lower end-to-end cost and robustness to connected sources are not established.

    Source: D DISCUSSION AND LIMITATIONS · p. 19

Abstract

from arXiv · show

Self-evolving search agents build their own training curricula by jointly optimizing a proposer that generates questions and a solver that answers them. This closed loop introduces a failure mode we call co-cheating: the proposer and solver increasingly agree on shared errors, so internal reward improves without a matching gain in external correctness. A post-hoc audit against source evidence shows co-cheating growing more severe over successive rounds of self-evolution, with pseudo-label correctness stagnating or declining even as the in-loop training signal improves. The most direct mitigation is to verify proposals before training: we introduce multi-sample verification (MSV), which queries the same model three times with the source and three times without it to decide task admission and replace unreliable pseudo-labels. MSV partially reduces false agreement but leaves substantial residual co-cheating and costs six extra labeler generations per candidate. These limitations motivate CrossFit, our main method: it partitions the proposer's source documents into groups A and B; questions generated from A are scored by an auxiliary solver trained only on B, and vice versa. The cross-fitted agreement determines proposer reward, so a same-source pseudo-label cannot be reproduced through the feedback solver, while the original solver's update rule is unchanged. Rerunning the loop with Qwen3.5-4B and Qwen3.5-9B, MSV reduces false-agreement mass from 6.1% to 5.7% and from 8.8% to 7.2%, whereas CrossFit reduces it to 3.0% and 3.7%. Replaying identical proposals with source-excluded feedback further reduces false agreement to 0.4% and 0.1%, isolating feedback ancestry from curriculum changes. Across seven downstream search benchmarks, CrossFit improves average performance over standard coupled self-evolution by 8.8 and 8.4 points and over Search-R1 by 8.7 and 7.8 points at 4B and 9B.

1 INTRODUCTION

Self-evolving search agents create their own curricula through a proposer–solver loop, but agreement can become an unreliable proxy for correctness. The paper introduces CrossFit to change the source of proposer feedback and reports reduced false agreement and improved search performance.

1 INTRODUCTION

  • In the proposer–solver loop, the solver's performance on new proposals determines proposer reward, creating an automated curriculum without a fixed human-authored training set.The proposer turns source documents into questions and pseudo-labels, and admitted pairs train the solver.Source: §1 · p. 2
  • Agreement can reinforce an incorrect pseudo-label across rounds, raising internal reward alongside false agreement as proposer and solver increasingly share errors.The paper calls this optimization outcome co-cheating and measures it as the fraction of evaluated pairs matching on the same incorrect answer.Source: §1 · p. 2
  • CrossFit scores questions from each source group with an auxiliary solver trained on the other group, changing proposer feedback while leaving the main solver's update rule unchanged.MSV instead verifies proposals with source-aware and source-blind samples, targeting label quality; CrossFit targets feedback provenance.Source: §1 · p. 2, Figure 2 · p. 3
  • CrossFit reduces false-agreement mass to 3.0% and 3.7% and improves performance over coupled self-evolution by 8.8 and 8.4 points at 4B and 9B.MSV reduces false-agreement mass to 5.7% and 7.2%; identical-proposal replay with source-excluded feedback reduces it to 0.4% and 0.1%.Source: §1 · p. 2

2 DIAGNOSING CO-CHEATING

The diagnosis combines an independent post-hoc audit with measures that distinguish correct agreement from shared mistakes. Across rounds, false agreement rises while lost credit falls, indicating a shift toward shared errors.

2 DIAGNOSING CO-CHEATING

  • Figure 4 depicts auxiliary solvers trained on one source fold scoring the other, while the main solver trains on all admitted questions.Source: Figure 4 · p. 4
  • An auditor checks saved questions and answers against source evidence without affecting admission, model updates, or reward.The audit saves the source document, adopted pseudo-label, and five solver responses at each scheduled step; unsupported cases remain unresolved.Source: §2 · p. 4
  • False agreement F counts pairs that match on the same incorrect answer; lost credit L counts correct solver responses denied credit by a wrong label.The audit checks the same five label–response pairs used to compute proposer reward rather than replacing them with a post-hoc majority.Source: §2 · p. 4
  • By round 3, false-agreement mass reaches 0.061 for Qwen3.5-4B and 0.088 for Qwen3.5-9B as lost credit falls.Figure 3 plots label truth, solver truth, and agreement across 129 steps, then false agreement against lost credit; rising false agreement with falling lost credit reveals shared-error accumulation.Source: Figure 3 · p. 3, §2 · p. 4

3 FROM VERIFICATION TO CROSS-FITTED FEEDBACK

The section moves from admission-time verification to CrossFit, which changes the solver supplying proposer feedback while leaving the main solver’s training on both folds intact.

3.1 MULTI-SAMPLE VERIFICATION

  • MSV draws three source-aware and three source-blind answers; compatible majorities replace the draft label and admit the task, otherwise the task is rejected.The two views test evidential support and answer stability, but same-model samples can share errors and the process adds six generations per candidate.Source: §3.1 · p. 5

3.2 CROSS-FITTED PROPOSER FEEDBACK

  • CrossFit assigns each source document to one fold and evaluates its questions with an auxiliary solver trained only on the complementary fold.Source-level assignment keeps related questions from one document together, and the evaluating solver has not trained on pseudo-labels from that source.Source: §3.2 · p. 5
  • CrossFit preserves the original feedback rule and frontier reward, changing only which solver supplies the proposer’s signal.The reward counts responses matching the adopted label; the main solver remains unsplit and trains on admitted questions from both folds.Source: §3.2 · p. 5
  • CrossFit prevents reward based solely on an evaluator reproducing a source-derived error, but auxiliary solvers may still share pretraining or overlapping-evidence errors.MSV changes which question–label pairs enter training; CrossFit changes which solver evaluates the next proposal.Source: §3.2 · p. 5
  • Table 1 reports downstream search performance after three rounds of self-evolution, with † marking Dr. Zero comparison baselines run on the same Qwen3.5 backbones and evaluated on the same 1,325-question set.Source: Table 1 · p. 6

4 MAIN EXPERIMENTS

Across seven search benchmarks, CrossFit improves downstream performance at both model scales; the experiments compare it with coupled self-evolution and other baselines under a shared evaluation setup.

4.1 EXPERIMENTAL SETUP

  • The evaluation uses Qwen3.5-4B and Qwen3.5-9B on the same fixed set of 1,325 questions from seven open-domain QA benchmarks.All treatments start from the same public checkpoint; no self-evolution treatment uses human-annotated QA training data.Source: §4.1 · p. 6
  • All rows use identical tool budgets, decoding, and answer extraction; self-evolution runs for three rounds and reports equal-weight average Cover-EM across seven benchmarks.The main solver is evaluated at each round end using one greedy search trajectory per question.Source: §4.1 · p. 6

4.2 MAIN RESULTS

  • CrossFit gains 8.8/8.4 points over Dr. Zero and 8.7/7.8 over Search-R1 at 4B/9B, respectively, with every benchmark improving at both scales.Average Cover-EM reaches 0.488 and 0.512; Table 1 reports downstream search performance after three rounds.Source: §4.2 · p. 6
  • Figure 5 summarizes agreement and solver truth by round, while its step-rate panels show all 129 steps and round means over 43 steps.The caption marks proposer phases, round boundaries, and step 62.Source: Figure 5 · p. 7
  • CrossFit’s gains average 10.0/10.9 points on four multi-hop tasks, versus 7.3/5.2 points across single-hop datasets at 4B/9B.The multi-hop tasks are HotpotQA, 2WikiMQA, MuSiQue, and Bamboogle.Source: §4.2 · p. 7
  • MSV alone adds only 0.7–0.8 average points over Dr. Zero; MSV + CrossFit reaches 0.491 and 0.515, only 0.3 points above CrossFit alone.The passage concludes that changing feedback provenance is more consequential than improving pseudo-label quality alone.Source: §4.2 · p. 7

5 ANALYSIS AND ABLATIONS

CrossFit reverses the divergence between agreement and truth, and its downstream advantage grows across rounds. Source-level exclusion and fixed-bank controls distinguish feedback provenance from evaluator duplication, curriculum selection, and extra updates; MSV adds complementary reliability gains but only modestly improves on CrossFit alone.

5.1 HOW DOES CROSS-FITTING CHANGE THE TRAINING TRAJECTORY?

  • After cross-fitted scoring begins in round 2, adopted-label truth rises, agreement stays at or below solver truth, and false agreement falls below half the coupled value by round 3.All arms share their first-round history, so the later divergence provides a within-run comparison of feedback provenance.Source: §5.1 · p. 7
  • By round 3, CrossFit’s downstream advantage over Dr. Zero reaches 8.8/8.4 points, while false-agreement mass falls from 0.061/0.088 to 0.030/0.037.CrossFit also raises adopted-label truth from 0.747 to 0.819 at 4B and from 0.737 to 0.851 at 9B.Source: §5.1 · pp. 7–8, Figure 6 · p. 8

5.2 WHY IS SOURCE-LEVEL EXCLUSION NECESSARY?

  • Same-source and full-data auxiliary solvers retain false-agreement mass near coupled controls, while source-ID splitting reduces it to 0.004/0.001 at 4B/9B.Random question-level splits reach only 0.050/0.062 because questions from the same document can enter both folds; the comparisons identify source exclusion as the effective component.Source: §5.2 · pp. 8–9, Figure 7 · p. 8

5.3 FIXED-BANK REPLAY SEPARATES FEEDBACK FROM CURRICULUM

  • On identical replay proposals, source-ID feedback reduces coupled false agreement from 0.058/0.073 to 0.004/0.001 at 4B/9B without changing admission or curriculum.A half-budget control matches the full source-ID result, ruling out extra auxiliary updates as the explanation.Source: Figure 7 · p. 8, §5.3 · p. 9

5.4 HOW DO VERIFICATION AND CROSS-FITTING INTERACT?

  • MSV reduces false-agreement mass to 0.057/0.072 at 4B/9B, CrossFit to 0.030/0.037, and their combination to 0.020/0.017.The combination improves downstream average performance by only 0.3 points beyond CrossFit alone at each scale; fixed-bank results likewise show larger reductions when source exclusion is added to MSV.Source: §5.4 · p. 9

6 RELATED WORK

Related work spans self-generated curricula, proxy-reward failures, and cross-fitting. This paper narrows its focus to the training-data ancestry of the solver used for proposer feedback.

Self-generated curricula and search

  • Prior self-play and corpus-based evolution work generates supervision, while Dr. Zero and Search Self-Play are direct search-agent comparators.SearchMaster offers a complementary diagnosis of misleading search self-play signals, and CAFE targets co-evolving feedback.Source: §6 · p. 9

Proxy rewards and self-confirmation

  • The paper links proxy-reward divergence and pseudo-label confirmation bias to the additional return path from a pseudo-label-trained solver into task generation.False agreement is treated as a diagnostic for that path, not by itself as causal proof of exploitation.Source: §6 · p. 9

Data exclusion versus better evidence

  • CrossFit borrows cross-fitting’s exclusion principle, not its asymptotic guarantees, and distinguishes source exclusion from evidence retrieval or majority stability.The adaptive curriculum lacks a demonstrated orthogonal score or independent sample structure; retrieval and stable majorities do not establish evaluator-label independence.Source: §6 · pp. 9–10

7 CONCLUSION

The conclusion presents CrossFit as a way to interrupt same-source feedback while leaving main-solver training unchanged. Results support its benefits at two model scales, with shared pretraining errors, overlapping evidence, and auxiliary cost remaining open limitations.

7 CONCLUSION

  • CrossFit scores each source with an auxiliary solver trained on the complementary fold, while the main solver continues learning from all admitted tasks.The evidence-backed audit separates internal agreement from correctness.Source: §7 · p. 10
  • Across Qwen3.5-4B and Qwen3.5-9B, CrossFit reduces final-round false agreement from 6.1%/8.8% to 3.0%/3.7% and improves seven-benchmark average Cover-EM over Dr. Zero by 8.8/8.4 points.The conclusion identifies shared pretraining errors, overlapping evidence, and auxiliary cost as open limitations.Source: §7 · p. 10

AI USE STATEMENT

The authors report using AI assistance for literature discovery, draft organization, consistency checks, and LaTeX authoring.

AI USE STATEMENT

  • A commercial-API LLM served as judge in the post-hoc audit, and the authors verified reported results, claims, citations, and implementation correspondence.The AI assistant also helped with literature discovery, draft organization, consistency checks, and LaTeX authoring.Source: AI USE STATEMENT · p. 11

A IMPLEMENTATION DETAILS

The implementation uses a sequence-normalized policy-gradient objective and fold-specific auxiliary solvers for proposer feedback. MSV and CrossFit add compute overhead, while the audit reports round-3 means of its statistics.

A IMPLEMENTATION DETAILS

  • The proposer and solver use a sequence-normalized policy-gradient objective over usable trajectories and scored assistant tokens.The advantage expression includes reward normalized by group mean and standard deviation, with a 10^-6 stabilizer.Source: A IMPLEMENTATION DETAILS · p. 16
  • Advantages are normalized within proposer task buckets or the five solver responses to one question, while prompt and tool tokens are masked.The proposer’s terminal answer is masked; solver answer tokens remain trainable.Source: A IMPLEMENTATION DETAILS · p. 16
  • The main solver trains on all admitted questions, while CrossFit’s auxiliary solvers train on assigned source folds and score the complementary fold only.The two auxiliary solvers produce proposer feedback and do not change the main solver’s training data or update count.Source: A IMPLEMENTATION DETAILS · p. 16
  • CrossFit adds 72% and 79% to the Dr. Zero budget; halving auxiliary updates lowers overhead to 36% and 40% with nearly unchanged replay false-agreement mass.MSV raises the reserved budget by about 90% at both scales and adds six labeler generations per proposal; combining both interventions costs 2.6 and 2.7 times the Dr. Zero budget.Source: A IMPLEMENTATION DETAILS · p. 16
  • Table 4 reports round-3 means of 43 step rates for audit coverage J/E and statistics TP, TS, A, F, and L.Figure 6 computes changes relative to Dr. Zero from unrounded means, so differences may vary by 0.1 percentage point from rounded table entries.Source: Table 4 · p. 17

B EVALUATION AND AUDIT DETAILS

Evaluation uses a fixed search-question manifest and a post-hoc evidence audit; the broader literature map distinguishes contextual research directions from matched performance comparisons.

B EVALUATION AND AUDIT DETAILS

  • Every checkpoint uses the same 1,325-question manifest, one greedy search trajectory, identical tool budget, and identical answer extraction.Cover-EM matches a nonempty normalized reference alias within the normalized prediction.Source: B EVALUATION AND AUDIT DETAILS · p. 17
  • The independent auditor judges saved labels and solver responses against evidence-backed references, leaves unsupported cases unresolved, and never affects training or reward.Table 4 reports each treatment’s round-3 mean audit statistics.Source: B EVALUATION AND AUDIT DETAILS · p. 17
  • Base micro averages are 0.382 on Qwen3.5-4B and 0.404 on Qwen3.5-9B; micro and macro averages order treatments identically in every round.Tables 5 and 6 report round-end Cover-EM and micro-averaged Cover-EM, respectively; each marks the best performance, and cross-fitted treatments share round 1 with their coupled counterpart.Source: B EVALUATION AND AUDIT DETAILS · p. 17, Table 6 · p. 18, Table 5 · p. 18

C BROADER LITERATURE MAP

  • Retrieval literature covers evidence representations, request strategies, alternative interfaces, and robustness; the study holds the retrieval backend fixed when examining source-excluded feedback.These directions include dense retrieval, query rewriting, hierarchical retrieval, and corrective retrieval.Source: C BROADER LITERATURE MAP · pp. 17–18
  • Search-agent systems are contextual references, not matched performance baselines in this manuscript; comparative claims require accessible implementations and matched protocols.The map also covers search optimization, difficult web exploration, tool use, and agent frameworks.Source: C BROADER LITERATURE MAP · pp. 17–18
  • Self-generated learning and reinforcement-learning work offers varied supervision and optimization choices; this paper’s distinctive variable is feedback-policy ancestry, not a generic policy-gradient estimator.The cited algorithms, training data, and capabilities are not claimed as reproduced.Source: C BROADER LITERATURE MAP · pp. 18–19

D DISCUSSION AND LIMITATIONS

Cross-fitting empirically mitigates co-cheating at two model scales, but its results do not establish lower end-to-end cost or robustness to connected sources. The complete audit trajectories retain scheduled steps and show correctness and coverage separately.

D DISCUSSION AND LIMITATIONS

  • The results establish empirical mitigation across two model scales, but not lower end-to-end cost or robustness to connected sources.Shared pretraining, overlapping evidence, semantically related sources, and an adaptive proposer may still induce correlated mistakes; lower false-agreement mass on accepted questions may also reflect rejection of difficult tasks.Source: D DISCUSSION AND LIMITATIONS · p. 19

E COMPLETE PER-STEP AUDIT TRAJECTORIES

  • Figures 8 and 9 plot all 129 scheduled steps, four treatments, and six metrics without smoothing; round summaries average 43 displayed step rates, not additional runs or uncertainty estimates.Separate axes keep false agreement and lost credit visible, while coverage is shown separately from correctness.Source: E COMPLETE PER-STEP AUDIT TRAJECTORIES · p. 19
  • Figures 8 and 9 show Qwen3.5-4B and Qwen3.5-9B audit trajectories, respectively, with shared treatment columns and metric layouts.Metrics are percentages; the coverage strip is separate, gray bands mark proposer phases, short dashes mark round boundaries, and the long dash marks step 62.Source: Figure 8 · p. 20, Figure 9 · p. 21
Loading 2609.39102v1…