Source-linked AI summary
Checklists Are Better Than Reward Models For Aligning Language Models
Vijay Viswanathan, Yanchao Sun, Shuang Ma, Xiang Kong, Meng Cao, Graham Neubig, Tongshuang Wu
TL;DR
Language models need to follow increasingly rich user instructions, but fixed or inferred reward criteria can be arbitrary or incomplete. The paper proposes Reinforcement Learning from Checklist Feedback, which extracts instruction-specific checklists and uses model or program-based evaluations as rewards. Across five benchmarks, RLCF improves both instruction following and general conversational assistance, with reported relative gains on FollowBench, InFoBench, and Arena-Hard.
Problem
Fixed reward criteria can be arbitrary, and even multiple criterion-specific prompts may not comprehensively capture response quality.
Method
RLCF extracts dynamic checklists from instructions, grades responses on checklist items using models or verification programs, and combines the scores into reinforcement-learning rewards.
Results
RLCF improves performance across all five evaluated benchmarks, including 5.4% relative improvement on FollowBench, 6.9% on InFoBench, and 6.4% on Arena-Hard over Qwen2.5-7B-Instruct.
Takeaways & Limitations
Checklist-based rewards provide a stronger learning signal than alternative feedback while remaining correlated with human preference judgments and requiring no additional data or human annotations.
Takeaways & Limitations
The paper notes that criterion-specific prompts may still fail to be comprehensive.
Abstract
from arXiv · showhide
Language models must be adapted to understand and follow user instructions. Reinforcement learning is widely used to facilitate this -- typically using fixed criteria such as "helpfulness" and "harmfulness". In our work, we instead propose using flexible, instruction-specific criteria as a means of broadening the impact that reinforcement learning can have in eliciting instruction following. We propose "Reinforcement Learning from Checklist Feedback" (RLCF). From instructions, we extract checklists and evaluate how well responses satisfy each item - using both AI judges and specialized verifier programs - then combine these scores to compute rewards for RL. We compare RLCF with other alignment methods applied to a strong instruction following model (Qwen2.5-7B-Instruct) on five widely-studied benchmarks -- RLCF is the only method to improve performance on every benchmark, including a 4-point boost in hard satisfaction rate on FollowBench, a 6-point increase on InFoBench, and a 3-point rise in win rate on Arena-Hard. These results establish checklist feedback as a key tool for improving language models' support of queries that express a multitude of needs.
1 Introduction
The paper proposes Reinforcement Learning from Checklist Feedback (RLCF), which uses instruction-specific checklists to automatically grade responses and provide reinforcement-learning rewards. RLCF improves instruction-following and conversational-assistance performance across five benchmarks, while alternative AI feedback methods produce mixed results.
- Motivation: RLCF addresses the limitations of arbitrary reward criteria and potentially incomplete criterion-specific prompts by making evaluation instruction-specific and comprehensive.The paper motivates checklist feedback as a way to reduce reward hacking and better capture the aspects of response quality relevant to each instruction.
- Approach: RLCF extracts dynamic rubrics from instructions and grades responses through atomic yes/no checklist questions answered by models or verification programs.Per-item scores are combined into a weighted checklist score used for reinforcement learning.
- Contributions: The authors introduce an algorithm for generating checklists at scale and WildChecklists, a synthetically generated dataset of 130,000 instructions with corresponding checklists.Checklist items may include verification programs for automatic evaluation.
- Results: RLCF improves Qwen2.5-7B-Instruct across constrained instruction-following and general conversational-assistance benchmarks, whereas alternative AI feedback methods yield mixed results.The five benchmarks are IFEval, InFoBench, FollowBench, AlpacaEval, and Arena-Hard.
- Results: 5.4% relative improvement in average hard satisfaction rate on FollowBench, 6.9% in overall requirement following ratio on InFoBench, and 6.4% on Arena-Hard are reported over Qwen2.5-7B-Instruct.The method can also be used off-policy, improving Llama 3.1 8B Instruct and OLMo 2 7B Instruct from samples collected with Qwen2.5-7B-Instruct.
2 Checklist Generation
The paper defines checklists as instruction-specific yes/no requirements and compares direct with candidate-based extraction. Candidate-based checklists improve objectiveness, atomicity, and overall quality, leading to stronger downstream RLCF performance.
- Checklist desiderata: A checklist pairs an instruction with yes/no requirements that a candidate response must satisfy completely.Requirements should be comprehensive, natural, objective, and atomic to support reliable evaluation.
- Extraction methods: The candidate-based method generates varied-quality responses and extracts their possible failure modes into checklist items.The direct method instead prompts an LM to extract criteria directly from the instruction.
- Evaluation: Directly generated checklists are more natural, while candidate-based checklists score better on objectiveness, atomicity, and overall quality.These trends are consistent across automatic and manual evaluations, despite absolute score differences.
- Downstream effect: Candidate-based checklist quality translates into stronger downstream RLCF performance than direct checklist generation.The comparison uses evaluations of four quality aspects and overall preference.
- Regularization: Two universal requirements regularize checklist optimization against long, off-topic preambles and other reward-hacking behavior.They require direct responses and appropriate matching to the instruction’s context.
- Dataset: The dataset WildChecklists contains 130,000 instructions with corresponding synthetically generated checklists.When applicable, checklist items are accompanied by verification programs for automatic evaluation.
3 Reinforcement Learning from Checklist Feedback
RLCF converts checklist-item judgments into preference data for reinforcement learning. It combines language-model scoring with verifier programs for suitable discrete criteria, then selects informative response pairs for preference tuning.
- Pipeline: RLCF begins by generating high-quality preference data from WildChecklists through a four-step process.The process includes response sampling, flexible scoring, and preference tuning.
- Sampling: Candidate responses are sampled from the base policy using temperature 1.3 and top-p 0.9.This sampling approach is presented as simpler than systematic response perturbation.
- Flexible scoring: Each checklist item is scored with a teacher language-model judge, averaging 25 numerical scores between 0 and 100.The score is computed for a given prompt, response, and individual checklist item.
- Flexible scoring: Verifier programs supplement AI judges for hard discrete criteria when exact checking is feasible.The program-generated score is averaged with the AI judge’s score, while the generator abstains on criteria it cannot verify exactly.
- Preference tuning: Responses are ranked by importance-weighted average checklist scores, and only the 40% most differentiated pairs become DPO preferences.This filters out pairs that provide too little useful pairwise signal.
4 Experimental Setup and Results
The experiments compare RLCF with instruction finetuning, reward models, and prompted AI judges across five constrained and general-purpose instruction-following benchmarks. RLCF improves across all five benchmarks, while its AI-judge averaging creates a computational bottleneck.
- Limitation: Averaging many AI-judge scores makes the judge component the computational bottleneck of the pipeline.The paper reports that reducing the number of samples can modestly reduce accuracy cost.
- Benchmarks: Five benchmarks measure constrained instruction following with IFEval, InFoBench, and FollowBench, plus general-purpose assistance with AlpacaEval and Arena-Hard.The latter two use naturalistic instructions collected from user queries.
- Baselines: RLCF is compared against instruction finetuning, specialized reward models, and prompted AI judges using single or mixed evaluation criteria.These baselines isolate the effects of feedback source and evaluation structure.
- Evaluation framing: Checklist feedback is framed as a very large mixture of prompted evaluators, with evaluators tailored to each instruction.Figure 3 describes the evaluator space as unbounded and instruction-specific.
5 Results
RLCF improves instruction-following performance across constrained and general benchmarks, while checklist quality and evaluator design strongly affect results. The method also has compute and domain-coverage trade-offs, and its RewardBench judging performance trails specialized reward models.
- 5.1 RL from Checklist Feedback consistently improves language models: RLCF improves performance across instruction-following and general conversational benchmarks, whereas off-the-shelf reward models produce mixed results.RLCF gives large gains on open-ended constraints and modest, consistent gains on general benchmarks; reward models can help on InFoBench but hurt on IFEval and FollowBench.
- 5.1 RL from Checklist Feedback consistently improves language models: 8.2% increase in FollowBench Constraint Satisfaction Level and 5.5% increase in average Hard Satisfaction Rate are reported for RLCF.Constraint Satisfaction Level measures the expected proportion of constraints satisfied, while average Hard Satisfaction Rate measures how often all constraints are satisfied.
- 5.1 RL from Checklist Feedback consistently improves language models: RLCF gives relative win-rate improvements of 2.8% to 8.4% over Qwen2.5-7B on general-use instruction-following benchmarks.Table 4 compares original and length/style-controlled metrics on Arena-Hard and AlpacaEval.
- 5.2 Comparing automatic evaluators: Specialized reward models outperform checklist scoring on RewardBench, but checklist scores remain competitive on Chat Hard and Reasoning.Checklist scores correlate especially well with preference annotations in the Chat and Chat Hard categories, while specialized models are better overall at predicting superior responses.
- 5.3 Learning from candidate-based vs directly-generated checklists: Candidate-based checklists outperform directly prompted checklists by 2% on IFEval and 2–3% on FollowBench, while matching them on InFoBench.The findings associate downstream performance with detailed, objective checklist construction.
- 5.5 Does RLCF lead to specialization at the expense of generality?: RLCF slightly reduces performance on underrepresented domains, including GSM8K and TruthfulQA, while reducing false refusals and slightly impairing true refusals.The paper reports 1–1.5% reductions on GSM8K and TruthfulQA and suggests expanding WildChecklists to a more diverse prompt distribution.
6 Related Works
The paper situates RLCF among AI-feedback and rubric-based alignment methods, contrasting checklist feedback with reward-model supervision and related checklist applications. It also reports off-policy results across model families and notes variance in IFEval.
- Evaluation caveat: The study reports moderate variance on IFEval, so slight differences between models are likely attributable to noise.This caveat applies to the trained models’ IFEval results.
- Off-policy generalization: Llama 3.1 8B Instruct improves strongly on InFoBench and FollowBench, while OLMo 2 7B Instruct improves strongly on IFEval in off-policy training.The models use Qwen2.5-7B-Instruct samples scored by Qwen2.5-72B-Instruct, with neutral results on the other reported benchmarks.
- AI feedback and rubrics: RLCF extends prior AI-feedback approaches by using checklists for reinforcement learning rather than only single prompts, global rubrics, or reward models.The paper relates its method to inference-time checklist use, rubric-based evaluation, and reward models used as synthetic preference annotators.
- Prior checklist work: RLCF is presented as the first work to apply a similar candidate-derived checklist approach to reinforcement learning.The comparison is with contemporaneous rubric-generation work that improves agreement between automatic evaluations and human judgments.
7 Limitations
The paper identifies three limitations: reliance on strong-to-weak generalization, evaluation only with preference-based reinforcement learning, and substantial scoring cost.
- Teacher dependence: RLCF relies on a larger Qwen2.5-72B-Instruct teacher to provide AI judgments for tuning a smaller model.The paper notes that RLCF still outperforms other methods that also use a 72B teacher.
- Training scope: The experiments explore only preference-based reinforcement learning, leaving policy-gradient training with checklist feedback for future work.The paper specifically identifies policy-gradient algorithms as an untested direction.
- Computational cost: Grading response pairs for 130k instructions takes roughly 4 days on eight 80GB H100 GPUs, making the scoring procedure computationally infeasible for many practitioners.The authors report that cost can be reduced by 50% with some loss of accuracy, while further optimization remains warranted.
8 Conclusion
The paper concludes that RLCF provides a detailed checklist-feedback framework, including rubric extraction and the WildChecklists dataset. Across the considered benchmarks, it reports uniform effectiveness for improving strong instruction-following models while identifying trainable judges as a future direction.
- Contribution: RLCF combines automatic rubric extraction, the WildChecklists dataset, and reinforcement learning from checklist feedback.The conclusion describes WildChecklists as a dataset of instructions and rubrics.
- Conclusion: RLCF uniformly improves strong instruction-following models on all benchmarks considered in the study.This is the paper’s broad conclusion about benchmark performance.
- Future direction: Combining checklist-style feedback with trainable judges is identified as an exciting future research direction.The current approach relies on prompt-based rubric generation and response grading under a rubric.
A The role of response pair mining
The response-pair mining study compares overall-score and single-aspect filtering. Filtering choice has little effect when retaining most pairs, whereas discarding 90% sharply harms both benchmarks, indicating that harder pairs are valuable.
- Pair mining: The method trains on the 40% of response pairs with the largest difference on at least one checklist criterion.This criterion-level selection differs from thresholding a single aggregated reward difference.
- Filtering comparison: When discarding 20% or 40% of pairs, overall-score and single-aspect filtering produce almost no performance difference.The comparison covers FollowBench and InFoBench.
- Filtering comparison: Discarding 90% of response pairs causes performance to plummet on both FollowBench and InFoBench.The result suggests that retaining some harder response pairs is beneficial regardless of filtering strategy.
B Prompt for Generating Verification Programs
The paper describes a prompt for generating programs that selectively verify responses.
- The verification-program prompt is used to generate programs for selectively checking responses.
- The prompt supports the paper’s approach to automatic response verification.
- The verification-program prompt is presented as part of the paper’s implementation description.
C Prompt for Scoring Semantic Criteria
The scoring procedure checks whether responses satisfy checklist requirements, using semantic assessment and selectively generated verification programs. Semantic criteria assess request fulfillment, relevance, and contextual fit.
- Requirement checking: The scoring prompt assesses whether a response satisfies specified requirements.
- Verification programs: Verification code is generated only when a requirement can be checked perfectly with a simple Python function; otherwise, it is deferred to a human collaborator.
- Semantic criteria: Semantic scoring evaluates request fulfillment, directness without unnecessary off-topic content, and fit with the instruction’s context and tone.