Source-linked AI summary
Agon: Competitive Cross-Model RL with Implicit Rival Grading of Reasoning
Vladislav Beliaev
TL;DR
Outcome-only reinforcement learning does not grade reasoning traces, leaving hard-problem reasoning quality without process labels. Agon trains two competing models to grade each other implicitly, and on hard math its trained cascade nearly doubles the accuracy of an untrained Mixture-of-Agents pass while the ordering replicates on code and across model families.
Problem
Outcome-based RL grades final answers but not reasoning traces, leaving the quality of good thinking unlabeled on hard problems.
Method
Agon jointly trains two distinct policies through rotating draft-and-challenge roles, rewarding each for solving and out-reasoning a peer that has seen its work.
Results
Competition plus information exchange beats cooperation, self-refinement, and vanilla GRPO on hard math, with the ordering replicated on code and across model families.
Takeaways & Limitations
A competing peer can supply the implicit reasoning signal that outcome-only RL lacks, enabling co-improvement beyond a single self-graded policy’s plateau.
Takeaways & Limitations
Agon requires a verifier and reference problems, and its gains depend on matched strength and behavioral divergence between the two models.
Abstract
from arXiv · showhide
Reinforcement learning from verifiable rewards (e.g. GRPO) is the engine behind today's reasoning models, yet it grades only the final answer. On hard problems this trains models to write more rather than to think better, since the trace itself is never graded and no label for good thinking exists. We introduce Agon, which makes two competing models each other's graders. Both attempt the same problem; in alternating roles, one drafts a solution and the other reads it while solving, and each is rewarded for out-solving the other. To win, a model must out-reason a rival that has seen its work, so reasoning is judged implicitly during training, with no process labels and no reward model. Because both models are optimized, each faces a progressively stronger rival, which single-model RL cannot provide. The two need only be comparably strong and behaviorally different. At inference the pair deploys as it trains, a two-stage cascade in which one model drafts and the other answers after reading the draft. On the hard split of DeepMath with Qwen3, this doubles GRPO's pass@1, roughly eight times the gain of an untrained Mixture-of-Agents pass over the same base. The ordering replicates on competitive-programming code and across model families (Qwen3.5, Gemma 4). For now the models talk in text; the next step is to let them reason together in latent space.
1 Introduction
Agon treats reasoning quality as an implicit competitive signal: two distinct, comparably strong models alternately draft and challenge solutions, with each rewarded for out-solving the other. This trained draft-and-challenge cascade improves hard-math pass@1 over vanilla GRPO and untrained Mixture-of-Agents at the same two-pass inference budget.
- Motivation: Outcome-based GRPO rewards only final-answer correctness, leaving reasoning traces unscored and allowing models to write more without necessarily thinking better.Direct process grading lacks ground-truth labels for good thinking, while learned process reward models are expensive, brittle, and unverifiable.
- Results: Roughly 2× GRPO’s and 1.8× the untrained Mixture-of-Agents’ pass@1, Agon reaches these gains on a hard-math held-out set at the same two-pass inference budget.Under matched generation budgets, competition plus exchange also beats cooperation, self-refinement, and vanilla GRPO, with the largest gains on the smallest models.
- Agon: Competition works when comparable models have different blind spots; naive agreement or averaging can regress toward consensus, while near-identical models gain little from one another.The proposed setup places the pair under competitive pressure rather than merely letting them cooperate.
- Agon: Two distinct, comparably strong policies alternate drafting and challenging: the challenger reads the draft, seeks a correct solution, and earns a conversion bonus for out-solving it.Roles rotate every step so both models learn both roles, and inference uses the same two-stage draft-then-answer cascade.
- Motivation: Agon makes a competing peer the implicit grader, turning reasoning-quality assessment into a head-to-head reinforcement-learning game without process labels or a reward model.A different policy is intended to break the closed loop of self-auditing, whose shared blind spots can cause self-improvement to plateau.
2 Related Work
Prior work addresses outcome-only RL’s trace-length pathology, self-play’s self-signal ceiling, cooperative multi-model inference, and adversarial generator–verifier training. Agon is positioned as cross-model, symmetric, and rotational competition rather than these preceding approaches.
- Outcome-only RL and the length pathology: GRPO and the R1 recipe reward only verified final answers, contributing to uncontrolled trace growth as accuracy rises more slowly than token usage.Existing remedies target length directly through explicit penalties, length-controlled objectives, or dynamic sampling.
- Self-play and self-improvement: SPIN, Self-Rewarding LMs, Absolute Zero, and R-Zero improve through self-play or self-judging, but policies optimized on their own signals share a structural ceiling.These methods instantiate self-play through past generations, self-judgment, or proposer–solver co-evolution from one model.
- Multi-model collaboration, debate, and ensembles: Self-consistency, multi-agent debate, and Mixture-of-Agents improve robustness cooperatively at inference time, but consensus can dilute quality when models differ and adds little for near-identical members.These procedures operate over frozen models rather than training competing models against each other.
- Adversarial and generator–verifier training: Prior adversarial and generator–verifier methods range from GANs and prover–verifier games to adversarial language games, while debate framed judged competition as a proxy for supervising reasoning.Agon differs by making competition symmetric and rotational rather than role-fixed.
3 Preliminaries: GRPO and the Unscored Trace
GRPO rewards only verifier-checked final answers, so completions with identical answers receive identical advantages regardless of reasoning. On hard problems, this leaves traces unscored and encourages padded attempts, motivating a relational grader that compares which model out-reasons the other.
- GRPO: GRPO samples groups of rollouts and updates the policy toward above-average final-answer rewards.The verifier checks only the final answer of each completion; KL regularization toward a reference is omitted in the passage.
- GRPO: Zero-variance groups are dropped, while completions with the same answer receive the same advantage regardless of their reasoning.This occurs because the reward signal is entirely a function of the answer.
- The unscored trace: On hard problems, unscored traces encourage repeated restarts, case splits, and reconsiderations because each adds another chance to reach a correct answer.Padding is not penalized and is reinforced when a correct rollout contains it, producing length growth much faster than solve-rate gains.
- The missing grader: Agon addresses the missing process signal by having a peer policy read the worked solution and reward whichever model out-reasons the other.This relational reward distinguishes concise, sound reasoning from long, lucky reasoning without verifier-provided process labels or a process reward model.
4 Method: Agon
Agon trains two matched, behaviorally divergent policies in a rotating draft-and-challenge loop: one solves from the plain prompt, while the other reads its solution summary and is rewarded for outperforming it. The method combines competitive grading, compute-parity accounting, and divergent adapters, then deploys the same interaction as a two-stage cascade.
- Pair design: Pairing requires comparable strength and different failure modes, so each policy can identify errors the other misses rather than collapsing into imitation.Agon keeps the pair matched in capacity but divergent in behavior, making competitive interaction reward beating the peer rather than agreeing with it.
- Draft-and-challenge rollout: Agon alternates drafter and challenger roles each optimizer step, routing the drafter’s post-reasoning solution summary into the challenger’s context while withholding the final answer.The drafter receives a vanilla GRPO update from N plain-prompt rollouts; the challenger produces one paired rollout per draft, and both adapters are updated.
- Efficient instantiation: Both policies are instantiated as LoRA adapters over one frozen base, with πA = base+∆A and πB = base+∆B.The second policy adds one rank-16 adapter of approximately 10M parameters, about 2% of the base, plus optimizer state.
- Competitive reward: The drafter is trained with correctness-and-format rewards, whereas the challenger receives a competitive conversion bonus for solving cases where the paired draft is incorrect.The conversion term raises the effective correctness weight from 2 to 3 when the opponent fails, while the summary itself appears only in context, not in the reward.
- Interpretation and limitation: The reported adversarial gain is not uniquely attributable to game-theoretic pressure, because the conversion bonus also functions as difficulty-weighted reward shaping.The experiments do not distinguish these interpretations, so the competitive framing is retained as a constructive design rationale.
- Inference and compute: At inference, Agon preserves its training procedure as a two-stage cascade: one adapter drafts from the plain prompt, and the other reads the draft summary before answering.Generation count and length caps are equalized, although the challenger receives additional short-summary prefill compute that is not equalized.
5 Experiments
Experiments show that information exchange and competition each improve held-out reasoning accuracy, with Agon substantially outperforming GRPO and cooperative controls while producing shorter final-stage traces. The gains transfer across datasets, model scales, families, and verifiable code tasks, while role rotation contributes materially.
- Main comparison: The Agon cascade lifts vanilla GRPO by +31 pp, from 30 to 61 pass@1, roughly doubling its held-out accuracy.GRPO lifts the base by +7 pp, from 23 to 30, whereas Agon adds +31 pp over GRPO.
- Scaling and model families: Agon’s gain over vanilla GRPO remains positive as model size grows and appears across Qwen3.5 and Gemma 4, tracking zero-shot strength rather than family.On Qwen3, the 0.6B Agon cascade reaches 61 pass@1, exceeding zero-shot Qwen3-4B at 52 and vanilla GRPO on 4B at 59.
- Code transfer: On verifiable CodeContests programs, the Qwen3-1.7B experiment reproduces Table 3’s ordering: exchange beats baseline, competition beats cooperation, and traces shorten.Correctness is verified by unit tests, with the same protocol and generation budget as the math comparison.
- Ablations: Role rotation yields 61 pass@1 versus 52 with fixed roles, while the competition ablation reaches 32 versus 30 for cooperative exchange.Fixed roles give each adapter only one gradient stream; the shared-opponent comparison also removes visible exchange and competitive gradient.
- Trace length: Agon’s final challenger stage averages 3.5k tokens versus 8.1k for a single GRPO pass, although drafting-stage tokens are not reported.The comparison is per-stage rather than total cascade cost.
6 Conclusion
Agon uses cross-model competition to provide implicit reasoning feedback unavailable to outcome-only RL, enabling two policies to co-improve through rotating draft-and-challenge roles. Its gains require matched, behaviorally divergent models, while current evidence is limited by clean-reward domains and unquantified training variance; future work targets latent communication and multi-objective reasoning.
- Contribution: Agon’s competing peer reads and implicitly grades each policy’s solution, supplying a process signal without process labels, reward models, or verifier changes.Two distinct policies train head-to-head with rotating draft-and-challenge roles, implemented as two adapters over one frozen base.
- Limitations: Agon’s effectiveness depends on matched strength and behavioral divergence: excessive gaps resemble distillation, while insufficient gaps resemble self-play.Divergence is maintained heuristically through initialization and role rotation, and complementarity is not quantified beyond cascade win rate.
- Limitations: The evaluation requires a verifier and reference problems and covers math plus a smaller code study with clean programmatic rewards, so noisier domains may differ.The reported studies do not establish behavior in domains with noisier reward signals.
- Limitations: Training variance is unquantified because each trained result uses one run and one sampled rollout per problem, with the better cascade direction selected post hoc.Groups with zero within-group variance are dropped because their GRPO advantage standard deviation is undefined.
- Future work: Future work will replace serialized text exchange with hidden-state communication through KV-cache injection or gated latent bridges between adapters.The proposed channel could let models share information they cannot verbalize.
- Future work: Future work will also scale the auxiliary density lever into a multi-objective framework with an explicit compression tradeoff.The current lever is described as an initial step beyond accuracy.