Learning to Reason from Reference Samples Alone: Reinforced Adversarial Learning for Modern LLM Post-Training

1 INTRODUCTION

• High-quality answers may be recognizable but difficult to score reliably, while RL with verifiable rewards requires a checker and hand-designed rubrics can be difficult to specify.The introduction also notes that rubric-based reward signals can become brittle and exploitable under direct optimization.Source: §1 · p. 1
• Prior adversarial post-training approaches use explicit verifiers, reasoning traces, or teacher distillation, motivating the question of whether reference answers alone can train a judge reward.The proposed judge is independent of the generator and trained in parallel with it.Source: §1 · pp. 1–2
• RAL co-trains a judge to distinguish reference from generated answers and uses its reference probability as the generator’s sole reward, without verifiers or reasoning demonstrations.The authors propose that as superficial distinctions fade, the adversarial game can push the judge toward deeper properties of reference answers.Source: §1 · p. 2, Figure 1 · p. 2
• On multiplication and arithmetic-expression synthesis, judge-only training improves held-out correctness and can match or surpass exact-reward training; the paper reports the same pattern for SMILES-to-formula translation.These tasks have expected answers that leave little room for stylistic shortcuts; scientific abstract experiments extend the approach to open-ended generation.Source: §1 · p. 2
• Across arithmetic, code, and chemistry tasks, RAL matches or exceeds verifiable rewards; on NIH abstracts, it outperforms frontier-LLM baselines in fooling both LLM judges and human experts.The paper also reports that RAL matches reference domains in open-ended abstract generation without explicit domain instructions, while preserving diversity.Source: §1 · pp. 2–3

Inverse reinforcement learning

• TextGAIL shares RAL’s adversarial structure but evaluates text quality and diversity rather than recovery of exact-reward structure, and requires expert trajectories that include reasoning traces for modern reasoning models.Earlier language applications of adversarial imitation learning predate reasoning-oriented post-training in larger LLMs (Shi et al., 2018; Li et al., 2019; Wu et al., 2021).Source: §2 · p. 3

Reward models for LLM post-training

• Learned outcome reward models assign scalar scores to completed responses (Christiano et al., 2017; Ouyang et al., 2022), while RLVR uses rule-based correctness checks and process reward models provide step-level feedback.These approaches require human labels, rubrics, verifiers, or process annotations; RAL targets settings where outputs are easier to collect than score.Source: §2 · p. 3

Co-training reward models and policies

• PRIME, TANGO, and AIRL-S co-train reward models with policies, but all rely on explicit outcome-correctness supervision (Cui et al., 2025; Zha et al., 2026; Jin et al., 2025).RAL instead co-trains its judge and policy from reference answers without correctness labels for generated outputs.Source: §2 · p. 3

Adversarial LLM post-training from demonstrations

• GAD distills a stronger teacher, R-AIRL requires expert reasoning traces, VARL combines a discriminator with an explicit verifier, and RARO learns from demonstrations alone (Ye et al., 2025; Fanconi et al., 2025; Damani et al., 2026; Cai et al., 2025).RAL uses a separately scalable judge and continuous reference-probability reward, without verifiers, expert reasoning traces, or separately constructed negatives.Source: §2 · p. 4

3 METHOD

• The training data pairs task prompts with reference final answers from a reference distribution, without assuming a reward function or verifier for scoring arbitrary generations.The generator policy includes reasoning traces and final answers, while the judge receives only the prompt and answer, never the reasoning trace.Source: §3 · p. 4
• A separate judge estimates whether an answer came from the reference distribution, and its reference probability is the reward for valid generator rollouts; malformed rollouts receive zero reward.The judge trains on reference and valid generated answers, while RL updates the generator on the full reasoning-and-answer rollout.Source: §3 · p. 4
• GRPO normalizes each output’s reward within its sampled group, and a KL penalty regularizes the generator toward its initial policy.After warm-up, a judge update gate skips updates when moving-average batch accuracy exceeds a threshold, with a maximum skip streak ensuring periodic updates.Source: §3 · pp. 4–5
• RAL with 1.7B and 4B judges matches verifiable-reward training on integer multiplication.The figure reports generator accuracy, judge accuracy, relative error, and judge-score correlation with relative error; diamonds indicate held-out test performance, including SFT.Source: Figure 2 · p. 5
• The final experiment optionally uses replayed past generations to reduce forgetting and a Polyak-averaged reward judge to smooth reward changes.The reward judge tracks the training judge through soft updates rather than replacing its weights outright.Source: §3 · p. 5

4 EXPERIMENTS

• The experiments cover three tasks with correctness withheld during RAL training, biomedical abstract generation without domain instructions, and NIH abstract generation conditioned on proposal titles.Source: §4 · pp. 5–6

4.1 VERIFIER-HELD-OUT TASKS RESULTS

• On multiplication, larger judges’ scores correlate more strongly with lower relative error; on Countdown-Code, RAL moves error toward zero while verifier-based baselines retain higher error despite strong verifiable reward.Source: §4.1 · pp. 7–8
• On multiplication, 1.7B and 4B judges nearly match verifiable-reward baselines by 2,000 steps; on Countdown-Code, 32B surpasses both after 4,000 steps; on SMILES, 32B improves faster than binary rewards.Source: Figure 3 · p. 6, §4.1 · pp. 7–8, Figure 4 · p. 7

4.2 OPEN-ENDED PAPER ABSTRACT GENERATION RESULTS

• RAL raises biomedical-domain alignment toward one while recovering diversity to about its starting level; frozen-judge training instead sends 125 of 128 outputs to one out-of-domain theme.Source: Figure 5 · p. 8, §4.2 · p. 9

4.3 NIH GRANT PROPOSAL ABSTRACT GENERATION RESULTS

• At the final checkpoint, RAL reaches approximately 35% fooling against LLM judges versus at most 4% for baselines, and 55% against human experts versus 44% before training and 21–37% for frontier models.Source: §4.3 · pp. 9–10, Figure 6 · p. 10
• NIH training shows judge-accuracy cycles on current generations but sustained accuracy on replayed samples; later traces and abstracts develop more specific links among findings, hypotheses, alternatives, and experiments.Source: §4.3 · pp. 10–11
• The NIH qualitative analysis covers only a subset of generations, and the proposed explanation for lower LLM-judge fooling rates than human rates is not directly tested.Source: §4.3 · p. 11

5 LIMITATIONS AND FUTURE WORK

• Most hyperparameters were held fixed, and extensive task-specific sweeps were not performed; tuning by task and judge size may improve convergence and final performance.The study also reports NIH adversarial co-training fluctuations, while RAL repeatedly recovers and exceeds frontier-model baselines at multiple checkpoints.Source: §5 · p. 11
• NIH grant generation has adversarial co-training stability challenges, and the generator is conditioned on proposal titles rather than specific research results.The authors identify conditioning on real results and providing those results to the judge as a natural next step.Source: §5 · p. 11
• Larger judges learn better rewards, and some tasks require judges substantially larger than the generator to match verifiable-reward training, increasing training cost.The judge can be discarded at inference; better initialization, specialized pretraining, or more efficient architectures may reduce the capacity requirement.Source: §5 · p. 11

6 CONCLUSION

• Reference final answers alone can support effective RL post-training, with a co-trained judge's predicted reference probability serving as the sole reward without verifiers, reward labels, or expert reasoning traces.The judge is scaled independently from the generator, and useful reasoning is learned through outcome-level feedback.Source: §6 · p. 11
• On tasks with hidden verifiable rewards, RAL can match or exceed direct reward optimization across arithmetic, code, and chemistry.The conclusion also reports recovery of a target scientific domain from reference examples without explicit domain instructions, while preserving diversity.Source: §6 · p. 11
• On NIH grant generation, RAL surpasses strong frontier-LLM baselines in fooling both LLM judges and human experts.The paper identifies judge capacity and sufficient co-training as key to making the reward signal effective.Source: §6 · p. 11
• The results motivate further study in scientific and other high-value domains where good answers are easier to collect than to score.The authors view adversarial co-training as potentially providing a curriculum that moves beyond superficial cues.Source: §6 · p. 11

AI USE STATEMENT

• AI tools helped format and clean datasets, suggest Prime-RL code changes, find related papers, and support qualitative analysis of model traces.AI-generated synthetic data was used in the Countdown-Code training mix; reported performance metrics were computed with a deterministic verifier.Source: AI USE STATEMENT · p. 11
• The authors manually validated dataset samples, inspected and tested suggested code changes, read suggested papers, and state that AI did not generate hypotheses or interpret results.They accept responsibility for the final content and artifacts, including AI-assisted work.Source: AI USE STATEMENT · pp. 11–12

B.1 INTEGER MULTIPLICATION

• Integer multiplication uses sampled four-digit integer pairs and exact-product references; completions must place a nonempty integer in answer tags before the judge classifies them as reference or generated.The appendix also reports exact correctness and clipped relative error as diagnostic metrics.Source: §B.1 · p. 15

B.2 COUNTDOWN-CODE

• Countdown-Code asks for an expression using each provided number once to reach a target; references come from validated solutions, and gold pass rate checks correctness independently of submitted tests.The task uses 15,000 training and 800 evaluation examples; generated submissions are parsed from JSON files, and the judge sees the expression and whether test.py changed.Source: §B.2 · pp. 16–17

B.3 SMILES-TO-FORMULA

• SMILES-to-formula provides a SMILES string and uses the dataset solution as the reference molecular formula, including implicit hydrogens.The task has 17,938 training examples and 800 held-out evaluation examples; valid formulas are compared by atom-count L1 distance, with exact correctness also reported.Source: §B.3 · pp. 17–18

B.4 MEDICAL/BIOMEDICAL ABSTRACT GENERATION

• Biomedical abstract generation uses a 5,000-abstract medical and biomedical corpus; the generator receives no title and is prompted only to write a scientifically plausible abstract.The corpus is filtered for well-formed 200–350-word abstracts and deduplicated; evaluation reports fooling rates against held-out LLM judges and human evaluators.Source: §B.4 · pp. 18, 20

B.5 NIH GRANT PROPOSAL ABSTRACT GENERATION

• NIH grant abstract generation uses FY2026 project titles as inputs and project abstracts as references; the learned judge's reference-class probability proxy is the main reward.After basic cleaning, the corpus contains 45,718 title–abstract pairs.Source: §B.5 · pp. 20–21

• Table 2 presents the RAL digit-multiplication adversarial-training setup.Source: Table 2 · p. 22
• Table 3 presents the digit-multiplication SFT baseline setup.Source: Table 3 · p. 22

C.2 COUNTDOWN-CODE TRAINING HYPERPARAMETERS

• Table 4 presents the Countdown-Code adversarial-training setup.Source: Table 4 · p. 23
• Table 5 presents the Countdown SFT baseline setup.Source: Table 5 · p. 23

C.3 SMILES-TO-FORMULA TRAINING HYPERPARAMETERS

• Table 6 presents the SMILES-to-formula adversarial-training setup.Source: Table 6 · p. 24
• Table 7 presents the SMILES-to-formula SFT baseline setup.Source: Table 7 · p. 24

• Table 8 presents the medical/biomedical paper-abstract adversarial-training setup.Source: Table 8 · p. 25

• Table 9 presents the NIH grant-proposal abstract adversarial-training setup.Source: Table 9 · p. 26
• Table 10 presents the NIH grant-abstract SFT baseline setup.Source: Table 10 · p. 26

D.1 FROZEN-JUDGE TRAINING REWARD

• The frozen judge compares a real reference abstract with a generated abstract and gives the generator reward 1.0 when it selects the generated abstract as real.The prompt identifies the reference corpus as medical and biomedical.Source: §D.1 · p. 27

D.2 BIOMEDICAL DOMAIN EVALUATION

• Biomedical rate is the fraction of evaluated abstracts classified as clearly biomedical, using a separate domain-evaluation prompt.The prompt counts health- and disease-related fields as biomedical and excludes unrelated fields unless their biomedical connection is clear.Source: §D.2 · p. 27

D.3 FROZEN-JUDGE FAILURE MODE

• 125 of 128 untitled frozen-Haiku rollouts collapsed onto a lexically varied Pacific-oyster microplastics theme outside the intended biomedical domain.The rollouts were from the final checkpoint at step 2000.Source: §D.3 · pp. 27–28

E QUALITATIVE NIH GRANT ABSTRACT EXAMPLES

• Two RAL-trained NIH grant abstract examples at checkpoint 2099 each fooled all five LLM judges and a majority of human expert annotators.Each example is shown alongside its real NIH abstract.Source: E QUALITATIVE NIH GRANT ABSTRACT… · p. 29