PaperlayerView exact revision on arXiv

Source-linked AI summary

RLBFF: Binary Flexible Feedback to bridge between Human Feedback & Verifiable Rewards

Zhilin Wang, Jiaqi Zeng, Olivier Delalleau, Ellie Evans, Daniel Egert, Hoo-Chang Shin, Felipe Soares, Yi Dong, Oleksii Kuchaiev

arXiv:2509.21319v3cs.CLcs.AIcs.LG

TL;DR

RLHF judgments often leave their quality criteria implicit, while correctness-focused verifiers cover a narrower set of tasks. RLBFF trains reward models to assess responses against explicit binary principles extracted from human feedback. The resulting models achieve strong reward-model benchmark results, and an open Qwen3-32B alignment recipe matches or exceeds named models on general alignment benchmarks at under 5% of their inference cost.

  • Problem

    Human-feedback judgments can lack explicit criteria, while verifiable rewards are typically limited to correctness-focused tasks, leaving a gap in interpretable and broadly applicable response evaluation.

    Source: §1 · p. 2

  • Method

    RLBFF extracts response-specific principles and trains reward models to judge whether a response satisfies each principle with a binary label.

    Source: §3 · p. 4

  • Results

    The RLBFF alignment recipe matches or exceeds o3-mini and DeepSeek R1 on MT-Bench, WildBench, and Arena Hard v2 at under 5% of their inference cost; its GenRM scores 86.2 on RM-Bench and 81.4 on JudgeBench.

    Source: §4.4 · p. 7, §7 · p. 9

  • Takeaways & Limitations

    Users can direct reward-model evaluation toward principles of interest, extending binary assessment beyond correctness-focused criteria.

    Source: §1 · p. 2, §4.3 · p. 6

    The consensus filter is intentionally high-precision and low-recall, so some correct principles are filtered out.

    Source: §3 · p. 5

Abstract

from arXiv · show

Reinforcement Learning with Human Feedback (RLHF) and Reinforcement Learning with Verifiable Rewards (RLVR) are the main RL paradigms used in LLM post-training, each offering distinct advantages. However, RLHF struggles with interpretability and reward hacking because it relies on human judgments that usually lack explicit criteria, whereas RLVR is limited in scope by its focus on correctness-based verifiers. We propose Reinforcement Learning with Binary Flexible Feedback (RLBFF), which combines the versatility of human-driven preferences with the precision of rule-based verification, enabling reward models to capture nuanced aspects of response quality beyond mere correctness. RLBFF extracts principles that can be answered in a binary fashion (e.g. accuracy of information: yes, or code readability: no) from natural language feedback. Such principles can then be used to ground Reward Model training as an entailment task (response satisfies or does not satisfy an arbitrary principle). We show that Reward Models trained in this manner can outperform Bradley-Terry models when matched for data and achieve top performance on RM-Bench (86.2%) and JudgeBench (81.4%, #1 on leaderboard as of September 24, 2025). Additionally, users can specify principles of interest at inference time to customize the focus of our reward models, in contrast to Bradley-Terry models. Finally, we present a fully open source recipe (including data) to align Qwen3-32B using RLBFF and our Reward Model, to match or exceed the performance of o3-mini and DeepSeek R1 on general alignment benchmarks of MT-Bench, WildBench, and Arena Hard v2 (at <5% of the inference cost). Models: https://huggingface.co/collections/nvidia/reward-models-10-2025

1 INTRODUCTION

RLBFF combines the versatility of human feedback with binary, explicit principles that make response judgments interpretable. It aims to retain broad evaluation coverage while making reward criteria more precise.

1 INTRODUCTION

  • RLBFF represents feedback as whether a response satisfies a stated principle, unlike RLVR’s correctness-only rewards or KTO’s unspecified mix of quality principles.The principle is provided alongside the prompt and response, making the judgment’s criterion explicit.Source: §1 · p. 2
  • Explicit principles can clarify the optimization objective because human preferences may reflect different qualities, such as humor on some subreddits and correctness on StackExchange-Math.The paper also motivates single-response judgments as more natural for textual feedback and binary labels as easier to calibrate than Likert scores.Source: §1 · p. 2
  • Binary principles preserve human feedback’s broad scope because they can assess any response aspect people value, beyond correctness alone.The paper contrasts this breadth with verifiable rewards typically used for math, coding, and precise instruction-following tasks.Source: §1 · p. 2
  • RLBFF is presented as more interpretable than Bradley-Terry scores, which are not globally calibrated across prompts and do not explain why a response received its score.The paper characterizes both verifiable rewards and binary flexible feedback as interpretable yes/no judgments.Source: §1 · p. 2
  • The approach aims to reduce reward hacking by modeling a specific feature and to recognize equivalent correct answers through training on pretrained models.The paper contrasts human-feedback precision problems with verifier recall problems, including failures to recognize equivalent answers such as 3 hours and 180 minutes.Source: §1 · pp. 2–3
  • RLBFF reward models reach 81.4% on JudgeBench and rank #1 on its leaderboard, while the Qwen3-32B recipe matches or exceeds named models on alignment benchmarks at under 5% of inference cost.The cited contribution also reports top performance on RM-Bench and PrincipleBench.Source: §1 · p. 3

2 RELATED WORK

Related work explores binary feedback, synthesized rubrics, and principle-following reward models. The paper distinguishes RLBFF by its larger, fine-grained principle inventory and binary scoring derived from human-written feedback.

2 RELATED WORK

  • RLBFF scales beyond prior safety and math approaches’ small, fixed inventories of about 10 principles to 1,000+ principles across general, STEM, code, and multilingual domains.The cited approaches use rule-based policy checks for safety or generative verifiers for math correctness.Source: §2 · p. 3
  • Unlike DeepSeek-GRM and RM-R1, which synthesize rubric criteria, RLBFF’s principles are user-controllable at evaluation time rather than fixed after generation.The passage says users of those self-generated-rubric models cannot specify or swap in custom principles during evaluation.Source: §2 · p. 3
  • RLBFF derives over 1,000 fine-grained principles directly from human-written feedback and uses binary yes/no scoring, contrasting with prior curated, consolidated, or synthetic criteria.The cited comparison includes RewardAnything’s 200 criteria and Likert-5 labels, R3’s pseudorubrics, and LMUnit’s combination of annotation criteria and generated principles.Source: §2 · p. 4

3 TRAINING DATA

The training data pipeline extracts binary response principles from human feedback, then filters for evidence support and cross-annotator agreement. It favors high precision over retaining every potentially valid principle.

3 TRAINING DATA

  • Principles are extracted from 40,821 HelpSteer3-Feedback samples using a zero-shot DeepSeek V3-0324 prompt that returns principles, supporting spans, and yes/no/partially labels.The data contain English feedback from more than 7,000 annotators across 80+ regions and General, STEM, Code, and Multilingual domains.Source: §3 · p. 4
  • To reduce unsupported extractions, the pipeline requests feedback evidence spans and removes principles whose cited spans fail a RapidFuzz match threshold.This filtering removed 2.2% of principles; the paper also excludes “helpfulness” because it denotes global quality rather than a specific principle.Source: §3 · p. 4
  • Partially fulfilled principles are excluded because natural-language extent markers lack a clean, consistent interpretation, leaving 35.4% no and 64.6% yes labels.The partially fulfilled category accounts for 13.8% of extracted principles.Source: §3 · p. 4
  • A high-precision consensus filter retains principles with a semantically similar match from every other annotator, using cosine similarity greater than 0.8.It keeps about 100,000 principles across three annotators, or 33,000 unique meanings, from an initial 1.2 million; the authors expect some correct principles to be filtered out.Source: §3 · p. 5
  • In a small verification exercise, majority annotator judgments agreed with extracted principles in 88.9% of cases, while inter-rater agreement was moderate at Fleiss’ κ = 0.447.Three volunteer annotators each reviewed 126 random samples, with each review taking less than one hour.Source: §3 · p. 5
  • The resulting corpus contains 33,000 principles representing 1,414 unique meanings, with most principles consisting of one or three words.Single-word principles account for 55.0% and three-word principles for 35.2%; the word distribution includes clarity, accuracy, and relevance among frequent terms.Source: §3 · p. 5, Figure 2 · p. 5

4 REWARD MODELING

The section evaluates scalar and generative reward models on general reward-model benchmarks and PrincipleBench, then compares their performance, bias, and efficiency. Flexible Principles models lead the reported results, with generative scoring improving benchmark performance and scalar scoring offering faster inference.

4.1 EVALUATION

  • PrincipleBench tests reward models on response qualities beyond correctness, including clarity, using human-labeled principles and examples spanning General, STEM, Code, and Multilingual domains.Its 487 samples include contexts of up to 10 prior turns; included labels require agreement from three annotators, and pairwise judgments must be correct in both response orders.Source: §4.1 · p. 6

4.2 BASELINES

  • The scalar baselines include a Bradley-Terry model trained on similar preference data, a fixed-helpfulness binary-feedback model, and two off-the-shelf Nemotron reward models.Generative baselines include Llama-3.3-Nemotron-Super-49B-GenRM and three related-work models.Source: §4.2 · p. 6

4.3 TRAINING

  • Flexible Principles ScalarRMs predict Yes or No from a prompt, response, and principle, scoring the response by the difference between the Yes and No log-probabilities.GenRMs reason over conversation history and a principle before judging Yes or No; this reasoning makes them approximately two orders of magnitude slower and more compute-intensive than scalar models.Source: §4.3 · pp. 6–7

4.4 RESULTS

  • RM-Bench (86.2) and JudgeBench (81.4) are achieved by Flexible Principles GenRM, surpassing the JudgeBench leaderboard top of 80.9 as of 24 Sep 2025.Flexible Principles ScalarRM scores 83.6 on RM-Bench, 76.3 on JudgeBench, and 91.6 on PrincipleBench.Source: Table 2 · p. 7, §4.4 · p. 7, Table 3 · p. 8
  • RewardAnything-8B-v1 scores 62.6 on JudgeBench, below the lowest-performing ScalarRM at 66.9; its score falls from 77.1 chosen-first to 65.1 rejected-first.Requiring correct judgments in both orders yields 62.6, which the authors interpret as evidence of substantial position bias in pairwise or n-wise GenRMs.Source: §4.4 · p. 7
  • Flexible Principles ScalarRM requires one generated token and finishes each task in <0.1 second, while comparable principle-grounded GenRMs take >100x longer.The authors report similar or better benchmark performance for the ScalarRM across RM-Bench, JudgeBench, and PrincipleBench.Source: §4.4 · p. 7

5 ABLATION STUDIES

The ablations examine how principle-consensus filtering and test-time principle choice affect reward-model performance. The default similarity threshold performs best on both reported benchmarks, while multi-principle training retains an advantage even when evaluation uses one principle.

5 ABLATION STUDIES

  • The cosine-similarity threshold of 0.8 achieves the best RM-Bench and JudgeBench performance among the tested thresholds.The authors interpret this as the best trade-off between data quantity and quality.Source: Table 4 · p. 8, §5 · p. 8
  • The training set contains 33,000 samples at threshold 0.8, compared with 95,000 at 0.7 and 11,000 at 0.9.Higher thresholds require greater semantic overlap between annotators' principles and remove principles attributed to one annotator's subjective preference.Source: §5 · p. 8
  • Fixing the test-time principle to Accuracy of Information yields +2.0% RM-Bench and -0.5% JudgeBench relative to training on a single fixed principle.The authors state that principle-following training can benefit users who evaluate with only one principle.Source: §5 · p. 8

6 MODEL ALIGNMENT

The alignment experiment trains Qwen3-32B with GRPO, using a Generative Reward Model to judge candidate responses against sample-specific principles. The open-source aligned model performs similarly to or better than several proprietary models across three benchmarks at less than 5% of the cheapest alternative's inference cost.

6 MODEL ALIGNMENT

  • Evaluation uses MT-Bench, WildBench, and Arena Hard v2, whose different prompts and stronger baseline make its scores not directly comparable with Arena Hard v0.1.Arena Hard v2 replaced v0.1 as the earlier benchmark approached saturation.Source: §6 · p. 9
  • GRPO training uses Qwen3-32B to generate candidate responses, then a GenRM judges them against the principle associated with each training sample.The policy optimizes the difference between the Yes and No log-probabilities, despite not being explicitly given a principle when generating candidates.Source: §6 · p. 9
  • The open-source RLBFF-aligned model performs similarly to or better than o3-mini, Claude-3.7-Sonnet (Thinking), and DeepSeek R1 across MT-Bench, Arena Hard v2, and WildBench, at <5% of the cheapest alternative's inference cost.The aligned model uses the Qwen3-32B architecture and is substantially ahead of the baseline BT-trained Qwen3-32B.Source: Table 5 · p. 9, §6 · p. 9

7 CONCLUSION

The paper proposes RLBFF to combine the advantages of RLHF and RLVR, reports strong reward-model benchmark performance, and uses it to align Qwen3-32B on general alignment benchmarks.

7 CONCLUSION

  • RLBFF-trained reward models achieve state-of-the-art performance on RM-Bench, JudgeBench, and PrincipleBench, while RLBFF alignment brings Qwen3-32B to comparable performance with o3-mini and DeepSeek R1 on MT-Bench, WildBench, and Arena Hard v2.The reported inference cost is less than 5% of those models' cost.Source: §7 · p. 9

REPRODUCIBILITY STATEMENT

This material describes reproducibility references, theoretical properties of the RLBFF reward, and prompt templates for extracting principles and judging responses. The reward places greater weight on samples near probability boundaries, and the templates elicit principle-specific judgments.

REPRODUCIBILITY STATEMENT

A ELABORATIONS ON THEORETICAL BASIS FOR RLBFF APPROACH

  • The reward log p − log(1 − p) increases monotonically with the probability of a “Yes” judgment, preserving the reward-only theoretical optimal policy.The discussion says this claim ignores regularization.Source: A ELABORATIONS ON THEORETICAL BASIS… · p. 16
  • Under the stated distributional assumptions, reward normalization can favor a minority of best responses when responses are mostly good and a minority of worst responses when they are mostly bad.The authors describe the resulting adaptive exploitation/exploration mechanism as potentially beneficial to training.Source: A ELABORATIONS ON THEORETICAL BASIS… · p. 16
  • RLBFF's reward gives more weight to samples near probability boundaries than using probability directly or sampling a binary reward from it.The comparison is reported in Table 6.Source: Table 6 · p. 17, A ELABORATIONS ON THEORETICAL BASIS… · p. 17

B PROMPT TEMPLATES

  • The principle-extraction template requests principles, supporting feedback spans, and yes/no/partially satisfaction labels, while the scalar reward-model template asks for a Yes-or-No judgment on a specified principle.The generative reward-model template asks for a final Yes-or-No judgment after reasoning about the conversation and principle.Source: B PROMPT TEMPLATES · pp. 17–18

C TRAINING DETAILS

The training details distinguish scalar reward models, generative reward models, and policy alignment by their training schedules and configurations. The reported compute requirements and optimal hyperparameters are measured in H100-node-hours.

C TRAINING DETAILS

  • Scalar reward models use one epoch, with Bradley-Terry trained on HelpSteer3-Preference and other scalar models trained on one epoch of the preprocessed dataset for comparison.The passage also specifies learning-rate search, AdamW, batch size, sequence length, and NeMo-Aligner.Source: C TRAINING DETAILS · p. 18
  • Generative reward models train for three epochs with 128 prompts and eight responses per rollout batch, using a 2e-6 learning rate and an optimal KL penalty of 1e-2.Training uses NeMo-RL; when “Yes” or “No” is absent from the top 20 log probabilities, the reward is set to -50.Source: C TRAINING DETAILS · p. 18
  • Model alignment trains for three epochs, with four generated responses per prompt and an optimal KL penalty of 0.01.The passage specifies a learning-rate search and NeMo-RL framework.Source: C TRAINING DETAILS · p. 19
  • Compute requirements and optimal hyperparameters are reported in H100-node-hours for experiments run on eight-H100-80GB-SXM-GPU nodes.Table 7 summarizes these requirements and hyperparameters.Source: Table 7 · p. 19, C TRAINING DETAILS · p. 19

D REWARD MODEL RESULTS

The reward-model results discussion corrects the reported RM-Bench overall score for RewardAnything-8B-v1, and the evaluation-details section describes benchmark judging choices and score reproducibility. It reports that style-controlled Arena Hard v2 scores varied substantially across environments, unlike non-style-controlled scores.

D REWARD MODEL RESULTS

  • RM-Bench contains 129 Chat, 529 Math, 228 Code, and 441 Safety samples, which the authors use to explain the sample-level averaging discrepancy.The reported domain scores for RewardAnything-8B-v1 are 76.7 for Chat, 90.3 for Math, 75.2 for Code, and 90.2 for Safety.Source: D REWARD MODEL RESULTS · p. 19
  • RewardAnything-8B-v1's RM-Bench overall score is treated as 83.1 rather than 86.4 because RM-Bench was intended to average domain scores, not samples.The authors describe 86.4 as an inaccurate sample-level average and use 83.1 for fair comparison.Source: D REWARD MODEL RESULTS · p. 19

E MODEL ALIGNMENT EVALUATION DETAILS

  • The evaluation details use GPT-4-0125-preview as the MT-Bench judge, GPT-4o-05-13 for WildBench, and Gemini-2.5-Pro for Arena Hard v2.For Arena Hard v2, the authors do not use style control because they found its scores were not reproducible.Source: E MODEL ALIGNMENT EVALUATION DETAILS · pp. 19–20
  • Style-controlled Arena Hard scores differed by about 8 points across environments, while non-style-controlled scores remained consistent at 48.0 for deepseek-r1 and 38.1 for QwQ-32B.The passage says the style-controlled differences were more than three times each model's standard deviation.Source: E MODEL ALIGNMENT EVALUATION DETAILS · p. 20

F SCALARRM ALIGNMENT EXPERIMENT

The ScalarRM-based Llama-3.3-70B-Instruct alignment performs substantially worse than GenRM across MT Bench, Arena Hard, and WildBench. The authors prioritize correctness-focused reward-model benchmarks for alignment evaluation and leave domain-specific alignment for future work.

F SCALARRM ALIGNMENT EXPERIMENT

  • ScalarRM-aligned Llama-3.3-70B-Instruct performs substantially worse than GenRM across MT Bench, Arena Hard, and WildBench.The largest reported gap is on Arena Hard V2: 11.9% for the ScalarRM model versus 55.6% for the Qwen3 RLBFF model.Source: Table 8 · p. 20, F SCALARRM ALIGNMENT EXPERIMENT · p. 20
  • The authors prioritize RM-Bench and JudgeBench because they focus on response correctness at the reward-model stage and better predict aligned-model performance.They note that general-domain benchmarks also assess aspects such as stylistic suitability.Source: F SCALARRM ALIGNMENT EXPERIMENT · p. 20
  • Domain-general alignment is the focus; alignment for settings such as character roleplay is left for future work.The authors identify those settings as ones where factual correctness may matter less.Source: F SCALARRM ALIGNMENT EXPERIMENT · p. 21

G SCALARRM EXPERIMENT WITH SMALLER BASE MODEL

With a smaller base model, Flexible Principles ScalarRM outperforms a same-data Bradley-Terry RM and an external baseline on Overall RM-Bench and Overall JudgeBench. Equal weighting slightly reduces RM-Bench performance while maintaining JudgeBench performance.

G SCALARRM EXPERIMENT WITH SMALLER BASE MODEL

  • Flexible Principles ScalarRM performs better than the same-data Bradley-Terry RM and Skywork-Reward-Llama-3.1-8B on Overall RM-Bench and Overall JudgeBench.The supplied passage reports the comparison but does not provide the score values.Source: Table 9 · p. 21, G SCALARRM EXPERIMENT WITH SMALLER… · p. 21

H SCALARRM EXPERIMENT WITH BALANCED WEIGHTED DATA

  • Equal weighting slightly reduced RM-Bench performance and maintained JudgeBench performance, supporting the authors’ view that the original imbalance had little impact.The passage reports this result in the balanced-weighting experiment.Source: Table 10 · p. 21, H SCALARRM EXPERIMENT WITH BALANCED… · p. 21
  • Equalizing Yes/No proportions required randomly subsampling each imbalanced principle’s majority class, retaining slightly under half the data, around 16,000 samples.The comparison evaluates the effect of this reweighting on RM-Bench and JudgeBench.Source: H SCALARRM EXPERIMENT WITH BALANCED… · p. 21

Related papers

Loading 2509.21319v3…