PaperlayerView exact revision on arXiv

Source-linked AI summary

AdaptEvo: Adaptive Agent Learning with Evolving Supervision

Shijun Wan, Jiancong Xie, Hang Xu, Jin Duan, Qixiong Wang, Xi Xiang, Maofei Que, Yahui Liu, Zhongyu Wei, Mu Chuan

arXiv:2610.11354v1cs.CL

TL;DR

Rule-governed decisions require applying written rules to case-specific evidence, but guidance and process evaluation can have gaps and references vary in support. AdaptEvo combines confidence-adaptive training with evolving knowledge and rubrics, improving moderation accuracy over GRPO and retaining CA-GRPO gains under changed rules.

  • Problem

    Written rules can leave case-specific guidance and process-evaluation gaps, while reference judgments vary in their support from rules and available evidence.

    Source: §1 · p. 1, §3 · p. 3

  • Method

    AdaptEvo combines CA-GRPO, which balances outcome and process rewards by reference confidence, with cross-case synthesis of decision knowledge and rubric refinement.

    Source: §1 · p. 2, §4 · p. 3, §4.2 · p. 5

  • Results

    61.9% ELA and 72.2% BDA on In-Period exceed GRPO by 7.5 and 3.7 points; without injected knowledge, CA-GRPO retains Out-of-Period gains over the base model on both metrics.

    Source: §1 · p. 2, §5.2 · p. 6

  • Takeaways & Limitations

    Experiments on industrial multimodal moderation show improved decision accuracy and gains from evolved decision knowledge, while CA-GRPO retains ELA gains over the base model under changed rules.

    Source: §7 · p. 9

    Differing policy initializations and rubrics prevent isolating either failure mechanism from the reported curves alone.

    Source: §6.3 · p. 9

Abstract

from arXiv · show

Rule-governed contextual decision tasks require models to apply specified rules to case-specific context and evidence. Written rules can leave gaps in decision guidance and process evaluation, while reference judgments vary in their support from the rules and evidence. To address these challenges, we introduce AdaptEvo, a framework for learning under imperfect supervision that couples confidence-adaptive policy optimization with evolving decision knowledge and evaluation rubrics. Its Training module uses Confidence-Adaptive GRPO (CA-GRPO) to balance outcome and process rewards according to reference confidence. Its Evolution module synthesizes reusable decision knowledge from recurring failures across training cases and refines process rubrics to detect overlooked errors. To support empirical evaluation, we construct an industrial multimodal content moderation dataset comprising a training set and In-Period and Out-of-Period test sets, with the latter collected under changed rules. Using Qwen3.6-35B-A3B, AdaptEvo achieves 61.9% exact-label accuracy and 72.2% binary decision accuracy on In-Period, exceeding GRPO by 7.5 and 3.7 percentage points, respectively. On Out-of-Period, the policy trained with CA-GRPO retains exact-label accuracy gains over the base model across evaluated checkpoints without injected decision knowledge, while GRPO declines with continued training. CA-GRPO also outperforms the tested fixed reward mixtures on both Out-of-Period metrics.

1 Introduction

Rule-governed contextual decisions are difficult because written rules leave case-specific guidance and evaluation gaps, while reference judgments vary in their support. AdaptEvo combines confidence-adaptive training with evolving knowledge and rubrics, and improves moderation accuracy over GRPO.

1 Introduction

  • Written rules may omit case-specific evidence requirements, boundaries, and exceptions, making operational guidance and fixed-criteria process evaluation difficult.Reference judgments also vary in reliability, so equal reliance on all labels may teach decisions weakly supported by rules and evidence.Source: §1 · pp. 1–2
  • AdaptEvo couples CA-GRPO, which balances outcome and process rewards by reference confidence, with evolution of decision knowledge and evaluation rubrics.Evolution draws reusable guidance from recurring training failures and refines process rubrics to detect overlooked errors.Source: §1 · p. 2
  • 61.9% exact-label accuracy and 72.2% binary decision accuracy on In-Period exceed GRPO by 7.5 and 3.7 percentage points, respectively.On Out-of-Period, AdaptEvo reaches 59.5% ELA and 69.5% BDA, exceeding GRPO by 7.6 and 4.3 points.Source: §1 · p. 2
  • The paper contributes an industrial multimodal moderation dataset and reports state-of-the-art performance for AdaptEvo on that task.The dataset includes training and test sets from the training period and a later period with updated rules.Source: §1 · p. 2

2 Related Work

Related work covers imperfect supervision and operational decision criteria, self-improving agent updates, and changing evaluation signals. AdaptEvo combines confidence-adaptive training with updates to decision knowledge and process rubrics.

2 Related Work

  • AdaptEvo adjusts reward mixing to support for a specified reference and learns scoped decision guidance, but does not model the full distribution of perspectives or resolve undefined standards.Prior work discusses annotator disagreement as potentially reflecting valid perspectives [8] and the operationalization of abstract standards [10].Source: §2 · p. 2
  • Unlike AgentEvolver [19] and MemSkill, AdaptEvo couples confidence-adaptive training with updates to decision knowledge and evaluation rubrics.The two-stage evaluation covers one joint update, without establishing sustained recursive self-improvement.Source: §2 · p. 2
  • AdaptEvo revises textual process checks and scoring anchors while holding evaluator parameters fixed, rather than using model-generated preference labels as in Self-Rewarding Language Models [18].This work is motivated by concerns about reward overoptimization and judge biases [2] [22].Source: §2 · p. 2

3 Preliminaries

Rule-governed contextual decision tasks require applying explicit rules to case-specific context and evidence, including relevant conditions and exceptions. The paper formulates this as an agentic decision process under imperfect reference-label supervision.

3 Preliminaries

  • Rule-governed contextual decision tasks require applying explicit rules or guidelines to case-specific context and evidence, including relevant conditions and exceptions.A visible injury alone does not establish a moderation violation; severity, presentation, and exemptions may matter.Source: §3 · pp. 2–3
  • An agent may use tools to gather evidence before returning a label and explicit rationale, with actions and decisions conditioned on rules, context, observations, and injected knowledge.The predicted label should be evidence-supported and satisfy applicable rule conditions and exceptions.Source: §3 · p. 3
  • The learning objective is evidence-grounded decisions under the governing rules, with agreement to the reference label treated as an imperfect supervision signal.Reference judgments vary in support: among 3,395 cases, 20.9% had disagreement among four human annotations, while a model rated 33.1% ambiguous or unsupported.Source: §3 · p. 3
  • Written rules may leave conditions and exceptions implicit, so agents can miss required checks while rubrics still reward cue recognition and rule citation.This motivates jointly refining decision knowledge and evaluation rubrics while grounding both in governing rules and evidence.Source: §3 · p. 3
  • AdaptEvo addresses these challenges through training that adapts outcome–process feedback to reference support and evolution that revises decision knowledge and evaluation checks.Governing rules remain fixed throughout updates within a training run.Source: §3 · p. 3

4 Method

AdaptEvo alternates policy training with evolution of decision knowledge and process rubrics. CA-GRPO adapts outcome and process feedback to reference support, while evolution synthesizes guidance from recurring failures and screens rubric checks for discrimination.

4 Method

  • The state at each training stage comprises policy parameters, a decision knowledge library, and a process rubric; two training stages and one joint knowledge–rubric update are evaluated.Training holds knowledge and rubric fixed, while the stage-boundary evolution update freezes the policy and selects versions for the next stage.Source: §4 · pp. 3–4

4.1 Training: Confidence-Adaptive Policy Learning

  • CA-GRPO adjusts the balance of reference agreement and process feedback according to reference support, while retaining process feedback even at maximal confidence.The confidence measure combines human annotation agreement with an external support assessment and is distinct from task difficulty or calibrated correctness probability.Source: §4.1 · pp. 4–5

4.2 Evolution: Decision Knowledge and Evaluation Rubrics

  • Knowledge evolution groups recurring, evidence-attributed failures across training cases and synthesizes rule- and evidence-supported conditions, evidence requirements, boundaries, and exceptions.Unresolved conflicts remain flagged for review; resulting guidance is injected into the agent’s initial context under a fixed text budget.Source: §4.2 · p. 5
  • Rubric evolution targets checks with high scores and low within-case rollout variance across many cases, accounting for repeated-judgment noise before selecting alternatives.Candidates are validated on a separate training panel; dimensions and their weights remain fixed across rubric revisions.Source: §4.2 · p. 5
  • Knowledge candidates require fresh rollouts, while rubric candidates reuse fixed trajectories; selected knowledge and rubrics stay fixed through the next training stage.Final test data are excluded from diagnosis and update selection.Source: §4.2 · p. 5

5 Experiments

The experiments evaluate moderation methods on In-Period and later, changed-rule Out-of-Period test sets. AdaptEvo leads the evaluated configurations on In-Period, while CA-GRPO and the full AdaptEvo configuration show different gains on Out-of-Period.

5.1 Experimental Setup

  • The multimodal cases include posts, moderation queues, applicable rules, and evidence tools; the training set has 3,395 cases annotated by four reviewers.Human annotations disagree on 20.9% of cases, and the QC reviewer’s label and rationale serve as the reference judgment.Source: §5.1 · p. 6
  • Comparisons use Qwen3.6-35B-A3B as a shared backbone, reporting exact-label accuracy and binary decision accuracy on tests collected during training and later under changed rules.Both metrics measure reference agreement rather than independently verified rule compliance.Source: §5.1 · p. 6

5.2 Main Results

  • 61.9% ELA and 72.2% BDA make AdaptEvo the top In-Period configuration, exceeding GRPO by 7.5 and 3.7 percentage points, respectively.These gains reflect the complete configuration, including evolved knowledge, rubric revision, and continued training.Source: §5.2 · p. 6, Table 1 · p. 7
  • Adding evolved knowledge at inference improves CA-GRPO by 3.7 ELA and 3.2 BDA points; continued training with knowledge and an updated rubric adds 2.8 and 0.9 points.The latter increments reflect the combined effects of continued training and rubric revision.Source: §5.2 · p. 6
  • 59.5% ELA and 69.5% BDA put AdaptEvo above GRPO by 7.6 and 4.3 points on Out-of-Period; without injected knowledge, CA-GRPO also beats the base model on both metrics.AdaptEvo approaches Gemini3.8-Flash in ELA (59.5% versus 60.4%) and matches its BDA (69.5%).Source: §5.2 · pp. 6–7, Table 1 · p. 7

6 Analysis

The analysis finds that CA-GRPO generalizes differently from GRPO and compares adaptive reward mixing with fixed ratios. It also examines the effects of evolved knowledge and rubric updates, alongside process-evaluator reward trends.

6.1 Generalization to New Rules and Queues

  • GRPO’s training rewards rise, yet its Out-of-Period ELA/BDA fall by 2.6/0.8 points from steps 10 to 60; its final checkpoint is below the base model on both metrics.GRPO (High-Conf.) also falls by 2.2/1.3 points, and its final checkpoint is below the base model on both metrics.Source: §6.1 · p. 7, Figure 3 · p. 8
  • CA-GRPO maintains an ELA gain over the base model at every evaluated checkpoint, ending at 55.2% ELA and 67.3% BDA.Its final checkpoint exceeds the base model by 1.9/1.4 points and the final GRPO checkpoint by 3.3/2.1 points; this comparison does not isolate confidence adaptation from process supervision.Source: §6.1 · p. 7

6.2 Confidence-Adaptive Reward Mixing

  • Among fixed mixtures, 0.2:0.8 gives the highest ELA on both benchmarks, while 0.5:0.5 gives the highest BDA; moving to 0.2:0.8 trades BDA for ELA.The shift raises ELA by 1.4/0.8 points and lowers BDA by 0.6/0.4 points on In-Period/Out-of-Period.Source: §6.2 · p. 8, Table 2 · p. 8
  • 55.2% ELA and 67.3% BDA let CA-GRPO lead the best fixed mixture on Out-of-Period by 0.4 and 1.1 points, while ranking first in three of four comparisons.On In-Period, its ELA is 0.2 points below the process-heavy mixture and its BDA is 0.8 points higher.Source: §6.2 · p. 8, Table 2 · p. 8
  • With evolved knowledge, CA-GRPO reaches 59.1% ELA and 71.3% BDA, exceeding the best fixed mixture by 0.5 and 1.2 points.Its BDA gain from knowledge is 3.2 points, compared with 1.3–2.2 points for fixed mixtures.Source: §6.2 · p. 8

6.3 Knowledge and Rubric Updates

  • Adding evolved knowledge improves In-Period performance without policy updates; continued training with that knowledge and a revised rubric adds 2.8 ELA and 0.9 BDA points.Those latter gains combine continued training and rubric revision.Source: §6.3 · pp. 8–9
  • With two evaluators using a simplified rubric, process reward approaches 0.98 while reference-agreement outcome reward falls across the first and last 12-step windows.The other Kimi-K3 run improves on both rewards; differing policy initializations and rubrics prevent attributing the divergence to one failure mechanism from these curves alone.Source: §6.3 · p. 9, Figure 4 · p. 9

7 Conclusion

AdaptEvo combines confidence-adaptive policy optimization with cross-case knowledge synthesis and rubric screening to address uncertain references and gaps in rule application. The dataset analyses show that human agreement and assessed rule-and-evidence support provide distinct signals.

7 Conclusion

  • AdaptEvo combines confidence-adaptive policy optimization, cross-case knowledge synthesis, and rubric screening; under changed rules, CA-GRPO retains ELA gains over the base model while GRPO deteriorates.Experiments on industrial multimodal content moderation also demonstrate improved decision accuracy and gains from evolved decision knowledge.Source: §7 · p. 9

A.1 Moderation Task and Reference Supervision

  • The QC reviewer’s label is the reference regardless of annotator agreement; Kimi-K3 assesses its support against applicable rules and evidence without independently establishing correctness.Reference confidence combines human agreement with assessed support for the QC label and rationale.Source: §A.1 · p. 13

A.2 Human Agreement and Reference Support

  • 20.9% of 3,395 training cases had nonunanimous human labels, while 33.1% were rated ambiguous or unsupported; these groups overlap.Among 2,686 unanimous cases, 718 (26.73%) were rated ambiguous or unsupported; the intersection of unanimity and support defines a 1,968-case high-confidence subset.Source: §A.2 · p. 15

A.3 Training and Evaluation Protocol

The protocol compares shared-backbone training configurations on In-Period and Out-of-Period benchmarks, with checkpoint and knowledge-injection conditions specified separately. ELA requires an exact label match, whereas BDA evaluates the coarser pass-versus-violation decision.

A.3 Training and Evaluation Protocol

  • The evaluation covers In-Period and Out-of-Period benchmarks across configurations using Qwen3.6-35B-A3B, including three fixed outcome:process reward mixtures.The protocol evaluates six checkpoints per method on Out-of-Period at steps 10–60 without injected decision knowledge.Source: §A.3 · p. 16, Table 6 · p. 16
  • Filtering changes training-sample composition and exposure, so equal training steps alone do not establish equal data coverage.The logged configuration retains 32 prompt groups per step and eight trajectories per group; repeated prompt occurrences mean these are not unique posts.Source: §A.3 · pp. 16–17
  • The main comparison uses each trained method’s 60-step checkpoint; the knowledge-only condition adds evolved knowledge at inference to the CA-GRPO checkpoint.AdaptEvo continues training after a joint knowledge-and-rubric update, so its continuation comparison does not isolate additional training from rubric revision.Source: §A.3 · pp. 16–17

A.4 Metric Definitions

  • ELA requires an exact label match, while BDA counts a different violation subtype as correct when both labels indicate violation.Current single-run summaries establish neither stability nor statistical significance, and raw rubric scores have no common scale across versions.Source: §A.4 · p. 17

A.5 Evaluation with New Rules and Queues

The section reports training-reward trends alongside an Out-of-Period checkpoint evaluation without injected decision knowledge, while emphasizing limits of the aggregate evaluation.

A.5 Evaluation with New Rules and Queues

  • Table 7 evaluates six checkpoints at steps 10–60 on the Out-of-Period benchmark, with all agents running without injected decision knowledge.The table uses the same benchmark as Table 1 and the same Qwen3.6-35B-A3B base model.Source: §A.5 · p. 17, Table 7 · p. 18
  • 0.6135 to 0.6238 for GRPO and 0.7109 to 0.7550 for GRPO (High-Conf.) are the mean raw rewards in the initial and final 10-step windows.Neither trajectory is monotonic, and the methods train on different data subsets.Source: §A.5 · p. 17
  • The reported aggregate ELA and BDA do not establish per-queue effects, variability across independent runs, or sample-level uncertainty.Changed rules and new queues are evaluated jointly, so their effects are not separated.Source: §A.5 · p. 17

A.6 Reward-Mixture Comparison at 60 Training Steps

The section compares fixed outcome–process reward weights with CA-GRPO after 60 training steps and records run aliases and checkpoint conventions.

A.6 Reward-Mixture Comparison at 60 Training Steps

  • All Table 8 results use 60 training steps on the same In-Period and Out-of-Period benchmarks; the knowledge condition adds decision knowledge at In-Period inference.The CA-GRPO row uses the same 60-step checkpoint as the main table and Appendix A.5.Source: §A.6 · p. 18, Table 8 · p. 18
  • The 0.2:0.8 ratio has the highest fixed-mixture ELA in all three conditions, while 0.5:0.5 has the highest fixed-mixture BDA.The two metric-specific maxima are not from a single configuration.Source: §A.6 · p. 18

A.7 Run Aliases and Checkpoint Conventions

  • Equal training steps do not establish equal data exposure or total compute.Table 9 maps internal run aliases to compact main-table names and knowledge settings; the six shared-backbone configurations use Qwen3.6-35B-A3B.Source: §A.6 · p. 18, §A.7 · p. 18, Table 9 · p. 19
  • The main table reports trained methods at 60 steps, while Table 7 follows checkpoints from steps 10–60 without injected knowledge.These are run-local steps; AdaptEvo includes initial and continued training.Source: §A.7 · pp. 18–19

B.1 Method Protocol and Diagnostic Details

AdaptEvo alternates policy learning with failure diagnosis and knowledge–rubric selection; its guidance and rubric updates are structured, screened, and constrained by the evaluation protocol.

B.1 Method Protocol and Diagnostic Details

  • At predefined stage boundaries, AdaptEvo alternates policy updates, failure diagnosis and grouping, and knowledge–rubric selection.Failures are grouped across cases within moderation queues; repeated disagreement with a reference alone does not justify a knowledge entry, and accepted versions remain fixed for the next stage.Source: §B.1 · pp. 19–20

B.2 Decision Knowledge Entry Structure

  • Each per-queue knowledge entry records its scope, triggering conditions, required evidence, decision conditions, exceptions, and applicability limits.Supporting records preserve provenance and unresolved rule–reference conflicts outside the text injected to the agent.Source: §B.2 · p. 20

B.3 Rubric Checks and Scoring Anchors

  • Rubric versions preserve fixed process dimensions and weights while revising checks and scoring instructions.Candidate checks were screened on saved rollouts across 140 training cases and seven queues, then re-evaluated on another 140 cases; revised anchors use six levels from 0 to 1, without automatic zero for missing or unobservable information.Source: §B.3 · p. 20

C Additional Moderation Diagnostics

C.1 Historical Process–Outcome Reward Diagnostics

Historical runs show that process and outcome rewards can move in opposite directions, while runs using the same evaluator can also show different trends.

C.1 Historical Process–Outcome Reward Diagnostics

  • 0.936 to 0.981 process reward coincided with outcome reward falling from 0.569 to 0.455 in the Qwen-evaluator run.Both that run and the simplified-rubric Kimi-K3 run approached the process-reward ceiling as outcome rewards declined.Source: §C.1 · p. 21
  • 0.635 to 0.813 in process reward and 0.652 to 0.731 in reference-agreement outcome reward were the Kimi-K3 run’s respective increases across the first and last 12-step windows.The other Kimi-K3 run increased process reward from 0.914 to 0.975 while outcome reward fell from 0.615 to 0.464.Source: §C.1 · p. 21
  • The reward patterns motivate concern that evaluator limitations or permissive rubrics may leave scoring gaps, but the curves do not establish either cause.The runs differ in policy initialization and rubric configuration, and their aggregate curves do not establish a causal ranking of evaluators.Source: §C.1 · p. 21

Related papers

Loading 2610.11354v1…