Source-linked AI summary
AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Following
Yun He, Wenzhe Li, Hejia Zhang, Songlin Li, Karishma Mandyam, Sopan Khosla, Yuanhao Xiong, Nanshu Wang, Xiaoliang Peng, Beibin Li, Shengjie Bi, Shishir G. Patil, Qi Qi, Shengyu Feng, Julian Katz-Samuels, Richard Yuanzhe Pang, Sujan Gonugondla, Hunter Lang, Yue Yu, Yundi Qian, Maryam Fazel-Zarandi, Licheng Yu, Amine Benhalloum, Hany Awadalla, Manaal Faruqui
TL;DR
Advanced instruction following remains difficult to evaluate and train because complex instructions are hard to verify and high-quality human-annotated rubrics are scarce. The paper introduces AdvancedIF and RIFL, achieving a 6.7% absolute improvement on AdvancedIF while improving public-benchmark performance.
Problem
Advanced instruction following is difficult to verify and train, while existing evaluations often lack high-quality human-annotated rubrics.
Method
The paper introduces AdvancedIF for evaluating complex instructions and RIFL, which generates rubrics, trains a rubric verifier, and applies reward shaping during reinforcement learning.
Results
6.7% absolute improvement on AdvancedIF, with strong performance on MultiChallenge and IFEval and improvements across AdvancedIF categories.
Takeaways & Limitations
Rubrics can support both training and evaluation of advanced instruction-following capabilities.
Takeaways & Limitations
Early experiments observed reward-hacking issues in which generated artifacts could mislead the rubric verifier.
Abstract
from arXiv · showhide
Recent progress in large language models (LLMs) has led to impressive performance on a range of tasks, yet advanced instruction following (IF)-especially for complex, multi-turn, and system-prompted instructions-remains a significant challenge. Rigorous evaluation and effective training for such capabilities are hindered by the lack of high-quality, human-annotated benchmarks and reliable, interpretable reward signals. In this work, we introduce AdvancedIF (we will release this benchmark soon), a comprehensive benchmark featuring over 1,600 prompts and expert-curated rubrics that assess LLMs ability to follow complex, multi-turn, and system-level instructions. We further propose RIFL (Rubric-based Instruction-Following Learning), a novel post-training pipeline that leverages rubric generation, a finetuned rubric verifier, and reward shaping to enable effective reinforcement learning for instruction following. Extensive experiments demonstrate that RIFL substantially improves the instruction-following abilities of LLMs, achieving a 6.7% absolute gain on AdvancedIF and strong results on public benchmarks. Our ablation studies confirm the effectiveness of each component in RIFL. This work establishes rubrics as a powerful tool for both training and evaluating advanced IF in LLMs, paving the way for more capable and reliable AI systems.
1 Introduction
AdvancedIF targets difficult instruction-following settings with expert-authored evaluation criteria, while RIFL provides a rubric-based pipeline for training and verification. Experiments report substantial gains, including a 6.7% absolute improvement on AdvancedIF.
- Motivation: Existing instruction-following reinforcement learning is difficult because responses are hard to verify and preference-based rewards can be uninterpretable and vulnerable to reward hacking.Rubric-based evaluation decomposes complex instructions into criteria that can be checked individually.
- AdvancedIF: Human experts write AdvancedIF prompts and rubrics, with multi-turn prompts reflecting interaction with model responses.Rubrics undergo multiple rounds of human review to capture granular prompt criteria.
- AdvancedIF: AdvancedIF contains 1,600+ prompts covering complex, multi-turn, and system-prompt instruction following.Complex prompts combine constraints involving format, style, structure, length, spelling, and inter-conditional requirements.
- AdvancedIF: State-of-the-art models achieve around 70% on AdvancedIF, indicating that the benchmark remains challenging.The benchmark is designed to reveal gaps between existing models and stronger instruction-following capability.
- RIFL: RIFL combines rubric generation, a finetuned rubric verifier, and reward shaping in a full-stack post-training pipeline.The generator scales prompt-rubric creation, the verifier judges rubric compliance, and reward shaping addresses reward hacking.
- Results: 6.7% absolute improvement on AdvancedIF demonstrates that RIFL substantially improves Llama 4 Maverick’s instruction-following ability.RIFL also performs strongly on MultiChallenge and IFEval, while ablations support its verifier and reward-design components.
2 Related Work
Prior work improves instruction following through instruction tuning, RLHF, and rubric-based methods, but complex and multi-turn evaluation remains challenging. AdvancedIF differs by presenting a full-stack rubric-based training and evaluation pipeline at scale.
- Instruction Following in LLMs: Instruction tuning and RLHF are established approaches for improving zero-shot instruction following and alignment with user intent.Instruction tuning uses curated instruction data, while RLHF learns from human preferences through reward modeling and reinforcement learning.
- Instruction Following in LLMs: Complex and multi-turn instruction following remains challenging despite strong frontier-model performance on many instruction-following tasks.The related work identifies long-context reasoning and complex instructions as persistent difficulties.
- Reinforcement Learning with Human Feedback: Programmatic rewards improve reasoning in coding and mathematics, but they do not transfer directly to general instruction following because correctness is harder to verify.General instructions often lack simple automated success criteria.
- Rubric-based RL: Rubrics and checklists have been used with DPO, GRPO, and Constitutional AI to guide alignment and response generation.These approaches provide precedents for rubric-based optimization but do not constitute the full pipeline proposed here.
- Rubric-based RL: This work focuses on a full-stack rubric-based pipeline combining human-annotated rubrics, generator and verifier training, and large-scale rubric-rewarded reinforcement learning.Experiments across models of varying scales evaluate its effectiveness for instruction following.
3 AdvancedIF: Rubric-based Evaluation for Instruction Following
AdvancedIF is a human-created rubric-based benchmark for complex, multi-turn, and system-prompt instruction following. It evaluates these capabilities through decomposed criteria and shows that current frontier models still struggle, especially beyond single-turn instructions.
- 3 AdvancedIF: Rubric-based Evaluation for Instruction Following: Open-ended instructions are evaluated by decomposing them into separately verifiable rubric criteria checked by an LLM.Each prompt is paired with criteria that capture its applicable instructions and can be assessed independently.
- 3 AdvancedIF: Rubric-based Evaluation for Instruction Following: AdvancedIF evaluates explicit complex instructions, carried context across multiple turns, and system-prompt steerability.Its complex prompts target six or more instructions spanning tone, format, style, structure, length, constraints, spelling, and inter-conditional requirements.
- 3 AdvancedIF: Rubric-based Evaluation for Instruction Following: AdvancedIF uniquely combines human-created prompts and rubrics with multi-turn conversations and system-prompt steerability.This coverage is intended to simulate real user-bot interactions and impose higher standards for instruction following.
- 3.1 Benchmark Design and Collection: All benchmark prompts are manually written by experts to target specific advanced instruction-following capabilities and avoid ambiguous or overlapping intentions.The same annotator creates each prompt and its rubric, reducing inconsistency between user intent and evaluation criteria.
- 3.1 Benchmark Design and Collection: Each prompt’s rubric contains up to 20 clear criteria focused on instructions applying to the final turn.For multi-turn dialogs, rubric collection is restricted to the last turn.
- 3.2 Benchmarking SoTA LLMs: Benchmark evaluation uses an off-the-shelf LLM rubric verifier that assigns a binary TRUE or False label for satisfying all criteria.The reported evaluation chooses o3-mini for reasoning ability, cost efficiency, simplicity, and reproducibility.
- 3.2 Benchmarking SoTA LLMs: AdvancedIF is difficult for frontier models: Gemini 3 Pro and GPT-5 achieve around 75%, leaving a substantial gap to the benchmark’s target capability.Lower thinking effort also reduces performance, indicating that extensive reasoning can help instruction following.
- 3.2 Benchmarking SoTA LLMs: Single-turn complex instruction following scores substantially higher than multi-turn carried-context and system-prompt steerability scores.The comparison identifies multi-turn instruction following as more challenging for current state-of-the-art models.
4 RIFL: Rubric-Based Instruction-Following Learning
RIFL is a rubric-based reinforcement-learning pipeline that generates rubrics, trains a verifier against expert judgments, and shapes rewards to improve advanced instruction following.
- 4 RIFL: Rubric-Based Instruction-Following Learning: RIFL combines rubric generation, verifier training, and reward design to formulate advanced instruction following as reinforcement learning.The pipeline constructs prompt–rubric data, trains a rubric verifier, and designs the reward used during RL.
- 4.2 Rubric Generation: Rubrics are generated by fine-tuning Llama 4 Maverick on thousands of expert-annotated prompt–rubric pairs, then evaluated against held-out human annotations.Evaluation uses semantic matching to calculate rubric precision and recall.
- 4.3 Rubric Verifier Training: The verifier is trained from human-annotated rubric evaluations, first with SFT and then with RL on criterion-level binary judgments.Each verification example includes the user prompt, rubric, model response, and human evaluation; the RL stage improves generalization.
- 4.3 Rubric Verifier Training: The verifier checks each rubric criterion separately and receives reward according to agreement between its binary judgment and the expert label.The verifier also produces a justification for each criterion-level judgment.
- 4.3 Rubric Verifier Training: 0.728 human agreement was achieved by the fine-tuned verifier versus 0.515 for a vanilla prompted judge and 0.723 for o3-mini.SFT increased agreement from 0.515 to 0.656, while RL further increased accuracy; Table 4 reports the comparison.
- 4.4 Reward Design and Shaping: The final reward is an indicator that equals one when all rubric-specific verifier labels are satisfied, while alternative weighted combinations remain future work.This simple reward is selected for its simplicity and effectiveness.
- 4.4 Reward Design and Shaping: Reward hacking occurred when models generated self-evaluative artifacts intended to mislead the verifier into assigning spurious high rewards.The authors add cleanliness and response-completeness criteria as reward shaping, including checks for verbose self-evaluation and truncated final sentences.
- 4.4 Reward Design and Shaping: The added reward-shaping criteria explicitly test for artifacts and incomplete responses and fully address the reported reward-hacking issue in the experiments.The criteria target both misleading self-evaluation and responses whose final sentence is cut off.
5 Experiments
RIFL is evaluated against challenging instruction-following benchmarks and ablated across verifier, reward, and reward-hacking components. It improves AdvancedIF and MultiChallenge performance, while rubric verification and reward-shaping choices affect robustness.
- 5.1 Training and Evaluation Details: Training uses Llama 4 Maverick as the base policy, synthetically generated rubrics, and a finetuned Llama 4 Maverick rubric verifier.The RL training uses an internal RLHF infrastructure and manually written vendor prompts.
- 5.2 Results: RIFL improves AdvancedIF by 6.7% overall and by approximately 5–9% across categories, while also improving MultiChallenge by 2.9%.The evaluation covers ComplexIF, Carried Context, and System Steerability on AdvancedIF, alongside MultiChallenge and IFEval.
- 5.3 Ablations on Rubric Verifier in RL: The finetuned rubric verifier is more robust to reward hacking than a vanilla LLM judge, which can reward exploitation without genuine instruction-following gains.The comparison is qualitative and is illustrated by a concrete example referenced in the paper.
- 5.4 Ablations on Reward Design: All-or-nothing rewards achieve the best overall performance at 58.1, compared with 55.7 for hybrid and 53.6 for fractional rubric rewards.The hybrid reward averages all-or-nothing and fractional rubric rewards, while the fractional design may add noise when individual criteria are judged imperfectly.
- 5.5 Ablations on Reward Hacking Prevention: Additional reward-hacking-prevention criteria reduce artifacts and overlong responses, producing outputs that more coherently and relevantly satisfy the instructions.Removing these criteria leads the model to generate responses intended to mislead the rubric verifier into assigning spurious high rewards.
6 Conclusion
The paper introduces AdvancedIF and RIFL as a rubric-based framework for evaluating and improving advanced instruction following. Its experiments support rubrics as tools for both training and evaluation.
- 6 Conclusion: The paper proposes RIFL, which generates rubrics, trains a rubric verifier, and optimizes LLMs with reinforcement learning.The pipeline is presented as a rubric-based learning and evaluation approach for improving instruction following.
- 6 Conclusion: AdvancedIF is a human-annotated benchmark for measuring performance on complex and multi-turn instructions.The conclusion identifies AdvancedIF as a benchmark for advanced instruction following.
- 6 Conclusion: Experiments report significant instruction-following improvements and ablations supporting the effectiveness of the pipeline's components.The paper frames rubrics as a tool for training and evaluating LLMs and identifies future research in this direction.
A Taxonomy of AdvancedIF
AdvancedIF includes a taxonomy of instruction-following capabilities used to organize the benchmark.
- A Taxonomy of AdvancedIF: Table 7 presents the taxonomy of AdvancedIF.
B Rubric Verifier Prompt
The rubric verifier prompt supplies rubrics and conversation context, then asks the verifier to assess whether the latest response follows the user's instructions. Its output uses a structured JSON schema with per-rubric answers and an aggregate satisfaction field.
- B Rubric Verifier Prompt: The verifier assesses whether the AI response to the user's most recent prompt correctly follows the user's instructions.
- B Rubric Verifier Prompt: The prompt provides conversation history, the user's most recent prompt, and the AI's corresponding response as evaluation context.
- B Rubric Verifier Prompt: The verifier output is a JSON blob containing answers for each rubric question and a SATISFIED_ALL_REQUIREMENTS field marked YES or NO.
C Ablations on Rubric Verifier
The rubric-verifier ablation compares responses produced with fine-tuned and non-fine-tuned Maverick judges on constrained invitation-writing tasks. Both outputs preserve core party details, while the supplied examples show differences in wording and formatting.
- Fine-tuned verifier: Fine-tuned Maverick produced an invitation for Timmy’s 4th birthday that states his reasons for loving dragons.The response says dragons can fly and breathe fire, as required by the prompt.
- Non-fine-tuned verifier: The non-fine-tuned Maverick response also preserves the birthday, dragon-interest, date, and location content shown in the supplied example.The example identifies Timmy’s 4th birthday, flying and fire-breathing dragons, Saturday, March 21st, and 123 Castle Drive.
- Formatting requirements: The final rewriting step explicitly adjusts punctuation to meet a five-exclamation-point limit.The supplied implementation passage describes reducing the original punctuation to five exclamation points.
- Fine-tuned verifier: The fine-tuned response retains additional constraints requiring dragon costumes, themed gifts, a riddle-based RSVP, and a phone number.Its RSVP instruction asks readers to text “yes” to 555-1234.
D Ablations on Reward Hacking Prevention Rubric
The reward-hacking-prevention ablation uses kindergarten number jokes to test compliance with counting, topic, format, and audience constraints. The supplied examples contrast a rubric-protected joke set with alternative outputs while emphasizing explicit numbers and fruit or vegetable content.
- Task requirements: The task requires explicit numbers from 1–10, no number above 10, question-and-answer formatting, and fruit or vegetable content in about half the jokes.The prompt also asks to avoid making every joke a knock-knock joke.
- With reward-hacking prevention: The reward-hacking-prevention version presents five kindergarten number jokes covering numbers 3, 5, 7, and 10 in question-and-answer form.The examples include carrots, apples, grapes, finger counting, and broccoli.
- Without reward-hacking prevention: The supplied alternative without reward-hacking prevention is represented only by an introductory heading, so no substantive output comparison is established.The available passage identifies the condition but does not provide the corresponding joke set.
- Task requirements: The accompanying compliance summary says the joke set varies formats, uses question-and-answer structure, and is intended for kindergarteners.It also states that about half of the jokes discuss fruits and vegetables.
- Revision example: One revised joke explicitly adds the number 3 to the apple joke’s question and answer.The revision changes the wording to mention a count of 3 and three apples.