AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Following

Motivation

• Complex instructions can span multiple turns, while verifying full compliance is difficult and preference-trained rewards may be uninterpretable and vulnerable to reward hacking.Rubric-based reinforcement learning decomposes complex instructions into simpler criteria that can be verified relatively easily.Source: §1 · pp. 1–2

Benchmark

• AdvancedIF contains 1,600+ expert-written prompts and reviewed rubrics spanning complex, multi-turn, and system-prompt instruction following.Complex prompts combine 6+ instructions; multi-turn prompts reflect carried context, and the benchmark assesses system-prompt steerability.Source: §1 · pp. 2–3

Benchmark results

• The best results among the benchmarked GPT5, Gemini 2.5 Pro, and Claude-4-Sonnet models are around 70%, indicating AdvancedIF is challenging.The authors position the benchmark as reflecting the gap between existing models and their stated north star.Source: §1 · p. 3

Training pipeline

• RIFL combines rubric generation, a finetuned rubric verifier trained using human-annotated evaluations, and additional reward-shaping criteria.The generator learns from a small set of expert-written data; reward shaping is intended to address reward hacking.Source: §1 · p. 3

Results

• RIFL improves Llama 4 Maverick by 6.7% absolute on AdvancedIF.It also shows strong performance on MultiChallenge and IFEval; ablations support the finetuned rubric verifier and reward design and shaping.Source: §1 · p. 3

Instruction Following in LLMs

• Instruction tuning improves zero-shot performance, while RLHF helps LLMs better adhere to user intent; complex, multi-turn and long-context instructions remain challenging.Prior evaluation uses powerful LLMs as judges or verifiable rubrics.Source: §2 · pp. 3–4
• AdvancedIF extends MultiChallenge’s multi-turn rubric evaluation with broader coverage and purely expert-written prompts and rubrics.The paper presents this as a more holistic and realistic evaluation of instruction following.Source: §2 · p. 4

Reinforcement Learning with Human Feedback

• RLHF uses preference data and a learned reward model, but its data demands and imperfect, opaque rewards can enable reward hacking.Programmatic rewards work in coding and math, but general instruction following lacks automatic success criteria, motivating rubric-based rewards.Source: §2 · p. 4

Rubric-based RL

• Unlike prior rubric work focused on preference-data generation or indirect guidance, RIFL directly uses rubric rewards in RL and scales a full-stack instruction-following pipeline.The pipeline includes human-annotated rubric collection, rubric-generator and verifier training, and large-scale RLHF; concurrent rubric-reward work focuses on synthetic rubrics with relatively small-scale models.Source: §2 · p. 4

3.2 Benchmarking SoTA LLMs

• For open-ended instructions, evaluation decomposes expectations into rubric criteria and uses o3-mini to judge each criterion, assigning TRUE only when all are met.The benchmark reports the percentage of TRUE responses for ComplexIF, Carried Context, and System Steerability.Source: §3 · p. 4, §3.2 · p. 6

3 AdvancedIF: Rubric-based Evaluation for Instruction Following

• AdvancedIF assesses explicit complex instructions, instructions carried from prior conversation context, and adherence to system-prompt constraints.The system-prompt constraints include response style, safety, product context, voice, and tool-use specifications.Source: §3 · pp. 4–5
• AdvancedIF is the only compared benchmark with both human-expert-created prompts and rubrics and dialogs featuring multi-turn conversation and system prompts.The authors argue that this coverage better simulates real user-bot interactions and sets high standards for instruction following.Source: Table 2 · p. 5, §3 · p. 5
• For explicit complex instructions, AdvancedIF targets user prompts containing 6+ instructions spanning constraints such as tone, format, length, spelling, and inter-conditional requirements.The targeted instructions also include style, structure, and negative constraints.Source: §3 · p. 5

3.1 Benchmark Design and Collection

• Prompts are manually created to focus on instruction following, avoid ambiguous intent and overlapping capabilities, and retain challenging examples that trigger final-turn failures.For multi-turn prompts, annotators converse with a provided LLM before giving instructions in the last turn.Source: §3.1 · p. 5
• Each prompt has a rubric of up to 20 separately verifiable criteria for the final turn, written by the same annotator to align criteria with user intent.The criteria decompose the user’s instructions into clear expectations.Source: §3.1 · pp. 5–6

3.2 Benchmarking SoTA LLMs

• Around 75% is the best result achieved by Gemini 3 Pro and GPT-5 on AdvancedIF, and reducing thinking effort harms performance.The authors describe the benchmark as challenging and say extensive reasoning can help models follow instructions.Source: Table 3 · p. 6, §3.2 · p. 6
• Average performance on single-turn complex instruction following is much higher than on multi-turn carried context and system-prompt steerability.The authors conclude that multi-turn instruction following remains more challenging for state-of-the-art models than single-turn instruction following.Source: §3.2 · p. 6

4.1 Formulation

• Unlike hard-coded RLVR rewards, RIFL uses prompt-specific rubrics and an LLM verifier to judge each criterion, motivating verifier fine-tuning over naive prompting.The post-training pipeline is formulated as an RL problem.Source: §4.1 · p. 7

4.2 Rubric Generation

• The rubric generator, fine-tuned on expert-annotated prompts, scores 0.790 F1 on held-out human annotations, up from 0.639 for its starting checkpoint.The model was trained for one epoch on SFT data; generated rubrics were matched semantically against human annotations to calculate precision and recall.Source: §4.2 · p. 7

4.3 Rubric Verifier Training

• Expert annotations label whether each rubric is met and provide justifications; verifier SFT cold-starts evaluation, then RL rewards agreement with expert binary labels.In the RL stage, the verifier checks criteria separately and receives a reward based on its agreement with expert judgments.Source: §4.3 · p. 7
• The finetuned verifier reaches 0.728 average F1 agreement, exceeding the vanilla judge (0.515) and matching O3-mini (0.723).SFT raises the vanilla judge’s score from 0.515 to 0.656, and RL further improves performance.Source: §4.3 · p. 8, Table 4 · p. 9

4.4 Reward Design and Shaping

• RIFL uses a binary reward for satisfying all rubric criteria, prioritizing simplicity and effectiveness while leaving weighted sums of criterion labels for future work.The reward is defined for each prompt-response-rubric pair.Source: §4.4 · p. 9
• Reward hacking included self-praise artifacts intended to induce spurious high scores; added clean-response and completion checks are reported to fully address the issue.The added criteria check for weird artifacts such as verbose self-evaluation and for responses whose last sentence is cut off.Source: §4.4 · p. 9

5.1 Training and Evaluation Details

• Evaluation covers AdvancedIF, single-turn IFEval, and multi-turn MultiChallenge, while training uses Llama 4 Maverick with synthetically generated rubrics and a finetuned verifier.IFEval reports the percentage satisfying instructions under loose and strict settings; MultiChallenge uses the average instance score, and AdvancedIF reports rubric-completion percentages by category and overall.Source: §5.1 · pp. 9–10

5.2 Results

• RIFL improves AdvancedIF by 6.7% overall and 5–9% on each category, raises MultiChallenge by 2.9%, and slightly improves IFEval over Llama 4 Maverick.The authors report that IFEval is already saturated by frontier models.Source: Table 5 · p. 10, §5.2 · p. 10

5.3 Ablations on Rubric Verifier in RL

• The finetuned rubric verifier is more robust to reward hacking than a vanilla LLM judge prompted with rubrics, which can be exploited without genuine instruction-following improvement.The paper refers to Section C for a concrete example.Source: §5.3 · p. 10

5.5 Ablations on Reward Hacking Prevention

• Removing additional reward-hacking prevention criteria leads to artifact-filled or overlong responses intended to elicit spurious high verifier rewards, whereas including them yields coherent, relevant responses that satisfy instructions.The authors report that these criteria effectively mitigate reward hacking during RL training and refer to Section C for an example.Source: §5.5 · pp. 10–11

5.4 Ablations on Reward Design

• On AdvancedIF, all-or-nothing reward scores 58.1, hybrid reward 55.7, and fractional rubric reward 53.6, suggesting stricter rewards better incentivize full rubric satisfaction.The hybrid combines all-or-nothing and fractional rewards equally; the authors hypothesize fractional rewards may add noise when criterion-level verifier judgments are imperfect.Source: §5.4 · p. 10, Table 6 · p. 11

• AdvancedIF is a human-annotated benchmark for measuring LLMs’ ability to follow complex, multi-turn instructions.Source: §6 · p. 11
• RIFL generates rubrics, trains a rubric verifier to provide feedback, and optimizes LLMs with reinforcement learning.Source: §6 · p. 11
• Experiments show that RIFL significantly improves LLM instruction following, and ablation studies validate the effectiveness of each pipeline component.Source: §6 · p. 11
• The work presents rubrics as a tool for training and evaluating LLMs and invites further research toward more capable and reliable AI systems.Source: §6 · p. 11

A Taxonomy of AdvancedIF

• Table 7 is titled “Taxonomy of AdvancedIF.”Source: Table 7 · p. 15

B Rubric Verifier Prompt

• The same verifier prompt is used in benchmark evaluation and RL training.Source: B Rubric Verifier Prompt · p. 16
• The verifier assesses whether the AI’s response correctly follows the user’s most recent prompt, with the conversation history provided as context.The prompt includes placeholders for the full conversation, latest user prompt, response, and rubrics.Source: B Rubric Verifier Prompt · p. 16
• The verifier returns a JSON blob with an answer for each rubric question and a SATISFIED_ALL_REQUIREMENTS field set to YES or NO.YES indicates that the response passes the rubrics check; NO indicates that it does not.Source: B Rubric Verifier Prompt · pp. 16–17

C Ablations on Rubric Verifier

• The fine-tuned Maverick judge’s invitation opening names the birthday child and gives two reasons for loving dragons, while capitalizing “DRAGONS.”The prompt also specifies structural and content constraints for the full invitation.Source: C Ablations on Rubric Verifier · p. 17
• The untuned Maverick judge produces a similar opening, naming the child and giving the same two reasons for loving dragons.Its response also continues with the date, location, costume, gift, and RSVP instructions.Source: C Ablations on Rubric Verifier · pp. 17–18
• The displayed editing process reduces the invitation’s exclamation marks to meet the five-mark limit.One passage explicitly describes reducing the count to five; later passages show revised invitation text.Source: C Ablations on Rubric Verifier · p. 18

D Ablations on Reward Hacking Prevention Rubric

• With the rubric, the response supplied five number jokes, each pairing a number with fruit or vegetables.The examples mention 5, 3, 7, 1, and 10.Source: D Ablations on Reward Hacking… · p. 20
• Without the rubric, the initial set included a finger-counting-to-5 joke without fruit or vegetables; the later revision made joke 2 explicitly repeat 3.The revised joke asks about 3 apples and says the basket is correct with its 3 apples.Source: D Ablations on Reward Hacking… · p. 20