PaperlayerView exact revision on arXiv

Source-linked AI summary

RLAC: Reinforcement Learning with Adversarial Critic for Free-Form Generation Tasks

Mian Wu, Gavin Zhang, Sewon Min, Sergey Levine, Aviral Kumar

arXiv:2511.01758v1cs.LGcs.AIcs.CL

TL;DR

Open-ended generation makes scalable RL rewards difficult because task-specific rubrics are numerous and hard to combine. RLAC jointly trains a generator and adversarial critic using external checks; it improves factual-generation outcomes and code-generation performance while reducing verification costs.

  • Problem

    Free-form generation requires diverse, prompt-dependent rubrics that are costly to enumerate and verify, making rubric-based RL post-training difficult to scale.

    Source: §1 · p. 1

  • Method

    RLAC jointly trains a generator and an adversarial critic that proposes likely-failing rubrics for external verification.

    Source: §1 · p. 2

  • Results

    Across factual text and code generation, RLAC outperforms competitive baselines with lower verification cost; on Qwen3-8B biographies it reaches FactScore 0.889 versus FactTune-FS’s 0.867.

    Source: §1 · p. 2, §6 · p. 11

  • Takeaways & Limitations

    The authors expect RLAC to generalize to open-ended tasks such as story or scientific text generation where exhaustive rubric verification is infeasible.

    Source: §6 · p. 11

    The formal setup assumes a binary verifier and counts an output as correct only if it satisfies every rubric.

    Source: §2 · p. 3

Abstract

from arXiv · show

Open-ended generation tasks require outputs to satisfy diverse and often implicit task-specific evaluation rubrics. The sheer number of relevant rubrics leads to prohibitively high verification costs and incomplete assessments of a response, making reinforcement learning (RL) post-training with rubric-based rewards difficult to scale. This problem is exacerbated by the fact that often the best way to combine these rubrics into one single reward is also highly prompt-specific. We propose Reinforcement Learning with Adversarial Critic (RLAC), a post-training approach that addresses these challenges via dynamic rubric verification. Our approach employs a large language model (LLM) as a critic that dynamically identifies only the most likely failure modes (e.g., a factual error or unhandled edge case), which are then verified by an external validator to optimize both generator and critic jointly. By training both the generator and the critic, this game enhances the critic's error detection and the generator's output quality while reducing required verifications. Our experiments demonstrate that RLAC improves factual accuracy in text generation and correctness in code generation, while also outperforming exhaustive verification and reward model methods. We show that dynamic critics are more effective than fixed critics, showcasing the potential of RLAC for scaling RL post-training to free-form generation tasks.

1 INTRODUCTION

RLAC targets the difficulty of scaling rubric-based rewards for free-form generation by training a critic to propose likely failures for external verification. The introduction reports gains in factual and code generation alongside reduced verification costs.

1 INTRODUCTION

  • Free-form outputs must satisfy diverse task-specific rubrics, making their enumeration and verification costly and their combination difficult.The best rubric combination can depend on the prompt and the model being optimized.Source: §1 · p. 1
  • Figure 1 contrasts costly exhaustive fact-checking, efficient learned scalar rewards prone to reward hacking, and RLAC’s externally verified likely-wrong rubric.In RLAC’s illustrated feedback, the critic is rewarded when the proposed fact fails and the generator otherwise.Source: Figure 1 · p. 2
  • RLAC jointly trains a generator and a critic that proposes likely-failing rubrics for external verification, avoiding exhaustive rubric checks.The critic is rewarded when its proposed rubric fails; the generator is rewarded when the critic cannot identify such a failure.Source: §1 · p. 2
  • On 8-sentence biography generation with Qwen3-8B, Reinforcement Learning with Adversarial Critic achieves FactScore 0.889, surpassing FactTune-FS’s 0.867, while reducing verification calls by 5.7×.Source: §1 · p. 2
  • Using only 9% of the training data, RLAC achieves the highest average code-generation scores on both base models and outperforms prior methods.Its averages are 53.2 on Qwen2.5-Coder-7B-Base and 56.6 on Qwen2.5-Coder-7B-Instruct.Source: §1 · p. 2

2 PRELIMINARIES

The preliminaries define generation correctness through task-specific verifiable rubrics and explain why checking every rubric becomes intractable for open-ended tasks. They contrast proxy reward models with rubric enumeration as existing approaches to the problem.

2 PRELIMINARIES

  • Each prompt has task-specific rubrics, and an output counts as correct only if it satisfies all of them; the objective is to maximize fully correct outputs.Examples include factual claims in biographies and edge cases in code.Source: §2 · p. 3
  • Standard RL can optimize a single well-defined rubric, but checking every rubric for every output becomes intractable when the set is extremely large or unbounded.The passage identifies PPO and GRPO as examples for constrained single-rubric settings.Source: §2 · p. 3
  • RLHF uses a proxy reward learned from offline preferences, which can misalign when generation explores beyond the preference data’s coverage.The passage notes that KL regularization can stabilize training while limiting exploration.Source: §2 · p. 3
  • Enumerating criteria can better reflect underlying rubrics, but assumes they can be listed exhaustively and makes verification computationally prohibitive.The passage gives all test cases for a nontrivial program as an example of unrealistic exhaustive enumeration.Source: §2 · p. 3

3 REINFORCEMENT LEARNING WITH ADVERSARIAL CRITIC

RLAC casts free-form generation training as an adaptive adversarial game: a critic proposes likely failure criteria, and an external validator checks them. The paper reformulates the objective around worst-case criteria and describes an online procedure for jointly updating the generator and critic.

3 REINFORCEMENT LEARNING WITH ADVERSARIAL CRITIC

  • RLAC makes verification adaptive: a learned critic proposes likely failure criteria for each generator output, and an external validator checks them to provide feedback for both models.The critic and generator are jointly updated, so verification targets the generator’s current weaknesses.Source: §3 · p. 3, §3.2 · p. 5

3.1 PROBLEM REFORMULATION

  • The reformulation treats satisfying every rubric as optimizing against the worst-case criterion, then uses a learned critic to avoid infeasible searches over large or infinite rubric sets.The paper states this critic-based min-max formulation has the same generator solution as the original objective while bypassing rubric enumeration.Source: §3.1 · pp. 3–4

3.2 PRACTICAL INSTANTIATION OF RLAC

  • The practical system uses a generator LLM, a fine-tuned LLM critic that proposes natural-language rubrics, and an external validator that checks whether outputs satisfy them.The validator may be implemented with domain-specific tools, such as rule-based checks or code test-case evaluation.Source: §3.2 · p. 4
  • Online training samples generator outputs and critic-proposed criteria, uses validator rewards to form preference data, and updates the generator; optional multi-criterion samples train the critic.The algorithm alternates policy evaluation and improvement, with DPO objectives for the generator and critic.Source: §3.2 · pp. 4–5

4 EXPERIMENTS

Across factual biography and code generation, RLAC is evaluated in settings where verification is costly or exhaustive checking is infeasible. It achieves strong task performance with fewer verification calls, while ablations examine validator reliability and critic adaptation.

4 EXPERIMENTS

  • The evaluation spans costly-but-possible biography fact checking and code generation with effectively unbounded edge cases; biography experiments use 170 topics, split into 120 training and 50 test topics.The factuality evaluation uses FactScore and tracks validator calls; code evaluation uses Pass1 and reports executed test cases as validator calls.Source: §4 · p. 5, §4.1 · p. 6, §4.2 · p. 8

4.1 FACTUAL TEXT GENERATION

  • 0.889 FactScore on Qwen3-8B eight-sentence biographies exceeds FactTune-FS’s 0.867 and ArmoRM’s 0.723, using 77k versus FactTune-FS’s 439k verification calls.FactTune-FS requires 4.4× more calls in the four-sentence setting (169k vs. 39k) and 5.7× more in the eight-sentence setting (439k vs. 77k).Source: §4.1 · p. 6, Table 1 · p. 6
  • 67K verification calls let RLAC reach 84% factuality, matching FactTune-FS at 368K calls by dynamically targeting high-risk errors.Figure 2 tracks training epoch, verification calls, and KL divergence from the base model; RLAC’s initial FactScore dips from 0.653 to 0.641 before later improvement.Source: §4.1 · pp. 6–7
  • RLAC’s rising KL accompanies FactScore gains from 0.653 to 0.889, whereas ArmoRM’s KL rises without corresponding factuality gains.The paper characterizes the latter pattern as evidence of reward hacking.Source: §4.1 · p. 7
  • With noisy validation, training falls below the base model; a static critic inflates FactScore by generating fewer facts, while the adapting critic increases correct facts and reduces errors.By round 16, static-critic validator outcomes reach about 0.81, compared with about 0.6 for the adversarial critic, which continues surfacing genuine errors.Source: Figure 3 · p. 7, §4.1 · p. 8

4.2 CODE GENERATION

  • 53.2 and 56.6 average scores across five code benchmarks, with 2,000 training problems, accompany 97.5% fewer test executions than AceCoder-Rule.The scores use Qwen2.5-Coder-7B-Base and Qwen2.5-Coder-7B-Instruct, respectively; RLAC required 192 thousand executions versus approximately 7.86 million for AceCoder-Rule.Source: Table 3 · p. 8, §4.2 · p. 9
  • On HumanEval, AceCoder-RM’s score falls from 91.5 to 89.0; despite noisy simulated validation, RLAC improves across all reported code benchmarks.The paper attributes the reward model’s fragility to noisy, incomplete preference data and its inability to adapt as generator outputs drift from its training distribution.Source: §4.2 · p. 9
  • Static-critic detection drops from 42.3% to 33.9% over three rounds, while the adapting critic remains above 39%; their final Pass@1 scores are 92.1% and 93.3%, respectively.The static critic’s test cases degenerate into minor variations, allowing the generator to avoid detection without truly fixing bugs.Source: §4.2 · p. 9, Figure 4 · p. 9

5 RELATED WORKS

Prior approaches use scalar or fixed-dimension reward models, enumerate rubrics, or use static judges. RLAC instead dynamically selects verifiable rubrics and adapts the critic within an adversarial training loop.

Reward models

  • Reward models compress criteria into a scalar from preferences or ratings, while multi-objective models use fixed dimensions such as truthfulness and honesty.RLAC instead learns a critic to propose a verifiable rubric per instance, with external-validator feedback.Source: §5 · pp. 9–10

Enumerative verifications for free-form generations

  • Enumerative verification checks fine-grained criteria for every output, so cost grows with the number of rubrics and unlisted error types may still be missed.RLAC dynamically selects a verifiable failure mode instead of exhaustively enumerating rubrics.Source: §5 · p. 10

LLM-as-a-Judge

  • LLM judges and generative verifiers produce fixed judgments or explanations, whereas RLAC trains a critic policy that proposes rubrics and updates from external validation.This makes the critic an active, on-policy agent that allocates verification effort per instance.Source: §5 · p. 10

6 CONCLUSION

RLAC trains a generator and critic in an adversarial min-max game, with the critic dynamically identifying and externally verifying a worst-case rubric for each output. On factual and code generation, it outperforms competitive baselines with lower verification costs; broader applicability is expected but evaluated only across two domains.

6 CONCLUSION

  • The adversarial min-max game jointly trains a generator and critic that dynamically identifies and externally verifies the worst-case rubric for each output.This bypasses exhaustive verification or manual reward design while providing adaptive learning signals that prevent reward hacking.Source: §6 · pp. 10–11
  • RLAC outperforms competitive baselines on factual and code generation with significantly lower verification cost.Ablations also confirm the critical role of components such as adversarial critic training.Source: §6 · p. 11
  • The authors expect RLAC to generalize beyond its two evaluated domains to open-ended tasks where exhaustive or rubric-by-rubric verification is infeasible.They give story or scientific text generation as examples.Source: §6 · p. 11

A PROMPTS

The appendix section lists the exact generator and critic prompts used for data creation and training, matching the input format described in Section 3.2.

A PROMPTS

  • The section lists the exact generator and critic prompts used for data creation and training, corresponding to Section 3.2’s input format.The passage does not provide the prompts themselves.Source: A PROMPTS · p. 18

B VALIDATOR IMPLEMENTATION DETAILS

This section concerns the validator design used in the training pipeline and points to Section 3.2 for its description.

B VALIDATOR IMPLEMENTATION DETAILS

  • The appendix section provides detailed validator design for the training pipeline, corresponding to Section 3.2.No specific validator design details are stated in this passage.Source: B VALIDATOR IMPLEMENTATION DETAILS · p. 19

B.1 FACTUAL TEXT GENERATION

Factual validation proceeds through an authenticity check, followed by external verification of eligible claims using FactScore’s atomic fact verifier.

B.1 FACTUAL TEXT GENERATION

  • The first validation stage checks whether the critic’s suspected fact appears in the specified sentence, using textual entailment to guard against information injection.The critic supplies both the suspected erroneous fact and the sentence number.Source: §B.1 · p. 19
  • Proposals that pass the authenticity check are verified claim by claim with FactScore’s component, which queries Wikipedia and returns a binary true-or-false result.Source: §B.1 · p. 19

B.2 CODE GENERATION

Because AceCoder lacks reference solutions, the authors build simulated ground truth for test-case validation and compare reference and generated-code outputs.

B.2 CODE GENERATION

  • Qwen2.5-Coder-7B-Instruct solutions filtered to 99.7% accuracy on original tests serve as simulated ground truth for validation.The dataset lacks reference solutions to prevent data contamination.Source: §B.2 · p. 19
  • Validation runs each critic test case on both the reference solution and generated code, then returns 1 for matching outputs and 0 for differences.Execution failures also indicate detected errors.Source: §B.2 · pp. 19–20
  • Noise in GPT-4o-generated AceCoder test cases may bias the reference-based validator, reflecting realistic imperfections in verification tools.Source: §B.2 · p. 20

Related papers

Loading 2511.01758v1…