Checklists Are Better Than Reward Models For Aligning Language Models

1 Introduction

• Reinforcement learning remains difficult for ambiguous instruction-following tasks, while existing feedback approaches have limitations in coverage, reward reliability, or grading criteria.Verifiable-answer feedback misses subjective qualities; reward models can produce arbitrary rewards and invite reward hacking; preference distillation makes the larger model infer what to grade, reducing the generator-verifier gap.Source: §1 · pp. 1–2
• RLCF consistently improves Qwen2.5-7B-Instruct, with relative improvements of 5.4% in average hard satisfaction on FollowBench, 6.9% in overall requirement following ratio on InFoBench, and 6.4% on Arena-Hard.Source: Figure 1 · p. 2, §1 · p. 3
• RLCF extracts dynamic rubrics from instructions and turns response grading into yes/no checklist questions answerable by a model or verification program.The pipeline generates checklists, grades each response on every item, combines the scores into a weighted checklist score, and uses that score for reinforcement learning.Source: §1 · p. 2, Figure 2 · p. 2
• The work introduces checklist-generation and response-grading algorithms, WildChecklists with 130,000 instructions and synthetic checklists, and an RLCF-trained Qwen2.5-7B-Instruct model.When applicable, checklist items include verification programs; checklist grading uses language models and code and can rank responses for preference tuning.Source: §1 · p. 2
• RLCF also works off-policy and its checklist rewards correlate with human preference judgments while providing a stronger learning signal than alternatives.Llama 3.1 8B Instruct and OLMo 2 7B Instruct improve using samples collected with Qwen2.5-7B-Instruct; the approach requires a teacher model but no additional data or human annotations.Source: §1 · p. 3

2 Checklist Generation

• Checklist requirements are yes/no questions judged against a candidate response, and a response is acceptable only if every requirement is met; desirable checklists are comprehensive, natural, objective, and atomic.Objectivity and atomicity are intended to ease verification because false-positive rewards can be more detrimental to RL than false negatives [Huang et al., 2024].Source: §2 · p. 3
• Direct extraction prompts an LM to derive criteria from an instruction [Cook et al., 2024], whereas candidate-based extraction elicits failure modes from varied-quality responses and assigns each item an importance weight.Direct extraction risks repeating the instruction in its criteria, potentially limiting comprehensiveness and objectivity.Source: §2 · p. 3
• To curb reward-hacking preambles, all checklists receive universal requirements for directly addressing the request without excess and matching the instruction’s context, with total importance weight 100/100, following Sun et al. [2023].Source: §2 · p. 3
• Candidate-based generation created WildChecklists from 130,000 WildChat instructions; Qwen2.5-0.5B, -1.5B, -3B, and -7B supplied candidates, while Qwen2.5-72B-Instruct generated checklists for both methods.Source: §2 · p. 3
• Table 1 evaluates two checklist-generation methods across four quality aspects and overall preference, using manual evaluation on the first 50 InFoBench Easy rows and gpt-4o automatic evaluation on all 500 rows.Source: Table 1 · p. 4

3 Reinforcement Learning from Checklist Feedback

• RLCF generates high-quality preference data for RL from WildChecklists through a four-step process.The supplied passages describe candidate sampling, checklist-item scoring, and preference tuning.Source: §3 · p. 4

Sampling Candidate Responses

• Candidate response pairs are sampled from the base policy at temperature 1.3 and top-p 0.9, rather than systematically perturbed as in prior works [Sun et al., 2024, Dong et al., 2024].The sampling settings cite [Holtzman et al., 2019].Source: §3 · p. 4

Flexible Scoring

• For each checklist item, Qwen2.5-72B-Instruct scores a response from 0 to 100, using the average of 25 sampled numerical judgments.The score is assigned given the prompt, response, and individual checklist item.Source: §3 · p. 4
• For hard, discrete criteria, a generated verifier is used only when the model is confident it can check the requirement exactly, and its score is averaged with the AI judge’s.The generator abstains on criteria such as sentence coherence; the passage cites prior work [Dong et al., 2024, Zhou et al., 2023].Source: §3 · p. 4

Preference Tuning

• Responses are scored by criterion, averaged with criterion-importance weights, and the 40% of pairs with the greatest difference on at least one criterion are retained for direct preference optimization [Rafailov et al., 2023].The filtering removes pairs considered too similar to provide useful pairwise signal.Source: §3 · p. 4

4.1 Experimental Details

• Training uses filtered WildChat conversations and two Qwen2.5-7B variants, with Qwen2.5-72B-Instruct used for AI judgments or ground-truth responses.The WildChat set excludes non-English, toxic, and longer-than-two-turn conversations [Zhao et al., 2024]; finetuning uses DPO for two epochs, batch size 1024, and maximum sequence length 2048.Source: §4.1 · p. 4
• Sampling multiple judge responses makes the AI-judge component the pipeline’s computational bottleneck, though Section 5.7 reports that n can be reduced with modest accuracy cost.The approach follows prior work favoring mean over mode scores from an LM-as-a-judge model [Wang et al., 2025a].Source: §4.1 · p. 4
• Five benchmarks cover fine-grained constraint following (IFEval, InFoBench, FollowBench) and general-purpose instruction following (AlpacaEval, Arena-Hard).Source: §4.1 · p. 5

4.2 Baselines

• Baselines span instruction finetuning, specially trained single or mixed reward models, and prompted judges using single or mixed evaluation rubrics.Source: §4.2 · p. 5
• Instruction-finetuning distills a Qwen2.5-72B-Instruct teacher, while reward-model comparisons use Skywork-Reward-Gemma-2-27B and ArmoRM-Llama3-8B-v0.1.The distillation follows Hinton et al. [Hinton et al., 2015]; both reward models are described as highly rated on RewardBench [Lambert et al., 2024b]. The teacher is finetuned via LlamaFactory [Zheng et al., 2024].Source: §4.2 · p. 5
• Checklist feedback is framed as a very large mixture of prompted evaluators, with a unique subset selected for each instruction.Source: §4.2 · p. 5, Figure 3 · p. 5

5.1 RL from Checklist Feedback consistently improves language models

• RLCF improves IFEval loose metrics by 2.8–3.0% relative, FollowBench CSL by 8.2% and average hard satisfaction rate by 5.5%, and Qwen2.5-7B win rates by 2.8–8.4% on general-use benchmarks.On InFoBench, RLCF achieves results comparable to the best reward-model approaches while maintaining gains across constraint-based benchmarks.Source: §5.1 · p. 5, Table 2 · p. 6, Table 3 · p. 6

5.2 Comparing automatic evaluators

• Off-the-shelf reward models show mixed RLHF results: Skywork improves InFoBench, Arena-Hard, and AlpacaEval but regresses on IFEval and FollowBench, while ArmoRM regresses on IFEval.Skywork-guided RLHF notably outperforms RLCF on AlpacaEval; ArmoRM improves AlpacaEval and InFoBench, with mixed Arena-Hard and FollowBench results. Checklist scores correlate with RewardBench preferences, especially Chat and Chat Hard, but specialized reward models perform better there; RLCF is less correlated with Safety and is not designed to replace safety alignment.Source: §5.2 · pp. 6–7, Table 5 · p. 7

5.3 Learning from candidate-based vs directly-generated checklists

• Candidate-based checklists outperform directly prompted checklists by 2% on IFEval, match them on InFoBench, and lead by 2–3% on FollowBench.The authors attribute this to detailed, objective checklists providing more new information than checklists generated directly from the original prompt.Source: §5.3 · p. 7, Table 6 · p. 8

5.4 Where does checklist feedback help?

• On FollowBench, RLCF works best for content constraints, while prompt-based scoring helps with style or format constraints, suggesting checklists encourage attention to the full instruction.Qualitative examples show a single-rubric AI judge can miss major prompt changes, while Skywork can give different scores to responses with identical meaning; checklist-based judging combines a verifier program and AI judge to balance shortcomings.Source: §5.4 · pp. 7–8, Table 7 · p. 8

5.5 Does RLCF lead to specialization at the expense of generality?

• RLCF slightly reduces true refusals while reducing false refusals, and lowers GSM8K and TruthfulQA performance by 1–1.5% on domains underrepresented in WildChat.WildChat is primarily daily assistance, advice, and analysis (75.5%), with 12% factual information and mathematics; the authors suggest broader prompt coverage could mitigate these trade-offs.Source: §5.5 · p. 8, Table 9 · p. 9

5.6 RLCF improves other model families off-policy

• Off-policy RLCF yields positive results without regressions for Llama 3.1 8B Instruct and OLMo 2 7B Instruct, with strong gains on different benchmarks for each model.Llama improves strongly on InFoBench and FollowBench, while OLMo improves strongly on IFEval, suggesting the checklists capture criteria that transfer across model families.Source: §5.6 · p. 8

5.7 How much compute is required for producing checklist-based AI judgments?

• Grading with 3, 5, 10, and 25 judge samples takes 32, 40, 72, and 92 hours, respectively, while IFEval and InFoBench efficacy remains stable across sample sizes.Using fewer samples hurts FollowBench content and situation categories, indicating that cheap, high-variance scoring may suit simpler criteria but not difficult or ambiguous instructions; five samples retain much efficacy at 55% less clock time than 25.Source: §5.7 · p. 9, Figure 4 · p. 9

• RLCF extends AI-feedback RL using a single prompt/rubric [Tunstall et al., 2023] or rubric collections with instruction-specific checklists.Source: §6 · p. 9
• Checklist feedback is significantly more effective than UltraFeedback, which evaluates responses using four global principles.Source: §6 · p. 9
• Prior work uses reward models as synthetic preference annotators for RL [Sun et al., 2023], while Tables 2–4 demonstrate risks of reward-model-supervised RL [Liu et al., 2024b, Wang et al., 2024c].Source: §6 · p. 9
• For Llama 3.1 8B Instruct, off-policy RLCF is neutral on IFEval and improves InFoBench and FollowBench; for OLMo 2 7B Instruct, it improves IFEval and is neutral elsewhere.The trained models showed moderate IFEval variance, so slight differences there are likely noise.Source: §6 · p. 9, Table 10 · p. 10
• Unlike instruction-synthesis work using DPO on controlled candidate responses [Xu et al., 2023, He et al., 2024, Sun et al., 2024, Dong et al., 2024], RLCF scores student-sampled responses and can support RL algorithms such as GRPO [Shao et al., 2024].Source: §6 · p. 10
• Inference-time checklists [Cook et al., 2024; Saha et al., 2023], checklist evaluation [Saad-Falcon et al., 2024], and Wang et al. [2025b]’s criteria improve evaluation, but RLCF first applies a similar approach to RL.Wang et al. show their criteria improve agreement between automatic evaluations and human judgments.Source: §6 · p. 10
• Dineen et al. [2025] improve model safety with detailed rubrics and GRPO, but use globally shared criteria rather than RLCF’s instruction-specific checklists.Source: §6 · p. 10

7 Limitations

• Scoring 130k instructions takes roughly 4 days on eight H100 GPUs with 80GB memory, making the method infeasible for many practitioners.Section 5.7 reduces this cost by 50% at a slight cost to accuracy, but further optimization is warranted.Source: §7 · p. 10
• RLCF uses strong-to-weak generalization, with Qwen2.5-72B-Instruct judging responses to tune a smaller model, though RLCF also beats methods using a 72B teacher.The authors identify checklist-trained policy-gradient algorithms as a future research direction because their work explores only preference-based RL.Source: §7 · p. 10
• The study explores only preference-based RL; applying checklist feedback to policy-gradient algorithms remains future work.The authors describe this as an exciting research direction.Source: §7 · p. 10

8 Conclusion

• RLCF improves strong instruction-following models on all benchmarks considered.The conclusion describes the method as uniformly effective.Source: §8 · p. 10
• The study proposes an algorithm for automatically extracting instruction rubrics and uses it to build the WildChecklists dataset.Source: §8 · p. 10
• Combining checklist-style feedback with trainable judges is a future direction for research on reward models.The current approach uses prompt-based components for rubric generation and response grading; the authors suggest RLCF analysis may motivate better reward models.Source: §8 · p. 10

A The role of response pair mining

• With 20% or 40% of response pairs discarded, filtering method makes almost no difference; discarding 90% sharply reduces performance on both benchmarks.The results suggest that keeping some harder response pairs is beneficial, but aspect-based filtering is not the primary driver of improvement; checklist rewards capture instruction-relevant dimensions of quality.Source: Figure 5 · p. 17, A The role of response pair mining · p. 17
• RLCF trains on the 40% of response pairs that differ most on at least one criterion, rather than selecting pairs by a single scalar reward difference.A single scalar difference may aggregate multiple small differences across requirements.Source: A The role of response pair mining · p. 17

B Prompt for Generating Verification Programs

• The section presents the prompt for generating programs to selectively verify responses.The prompt is shown in Figure 6.Source: B Prompt for Generating Verification… · p. 18

C Prompt for Scoring Semantic Criteria

• The evaluator is prompted to verify whether responses satisfy various requirements, with the instruction beginning “Given.”The passage identifies this as the prompt used for requirement checking in Figure 7.Source: C Prompt for Scoring Semantic Criteria · pp. 18–19