PaperlayerView exact revision on arXiv

Source-linked AI summary

Eat Your Precision: Precision Objective Mitigates Reward Hacking in Rubric-based RL

Hyun Ryu, Jong-Heon Kim, Yiqing Xie, Seungone Kim, Carolyn Rose, Eunho Yang

arXiv:2610.33512v1cs.CL

TL;DR

Rubric-coverage rewards help train models for non-verifiable tasks, but optimizing coverage can leave response necessity and factual correctness unmeasured. FORK combines coverage with unit-level precision assessments, and the authors report stronger response quality and preference while maintaining similar rubric coverage.

  • Problem

    Rubric-coverage rewards primarily optimize recall and do not penalize unnecessary or factually incorrect content unless rubrics specify those failures.

    Source: §1 · p. 1, §3.1 · p. 4

  • Method

    FORK combines rubric-coverage recall with precision requiring each response unit to be necessary and factually correct, assessed using rubric-driven necessity credits and an LLM grader.

    Source: §1 · p. 1, §5 · p. 9

  • Results

    FORK outperforms other rubric-based RL methods with Pareto-optimal performance; rubric-free win rate rises by 57.4 pp, from 10.6% to 68.0%, on RubricHub-Medical.

    Source: §4.2 · p. 7, §5 · p. 9

  • Takeaways & Limitations

    FORK balances rubric coverage with overall response quality, improving response precision and preference while maintaining similar rubric coverage.

    Source: §1 · p. 2

Abstract

from arXiv · show

Rubric-based rewards enable effective reinforcement learning on non-verifiable tasks. However, because these rewards are typically defined as a weighted measure of rubric coverage, models are trained primarily to maximize a recall objective. As a result, RL-trained models produce responses that are more complete but also more verbose and contain more unnecessary or factually incorrect content, leading them to be less preferred than responses from the base model. To address this issue, we present the FORK reward, which jointly optimizes recall and a novel precision objective, which requires each unit of a model response to be both necessary and factually correct, as measured by rubric-driven necessity credits and a factual-correctness grader, respectively. Our experimental results show that FORK outperforms existing rubric-based RL methods and achieves Pareto-optimal performance that balances rubric coverage and overall response quality. Compared to the original recall-based reward, FORK substantially improves precision and reduces response length while achieving similar rubric coverage, which leads to significantly higher win rate compared to the base policy (e.g., from 10.6% to 68.0% on RubricHub-Medical and from 22.1% to 74.1% on HealthBench).

1 INTRODUCTION

Rubric-coverage rewards can improve completeness while encouraging verbose, unnecessary, or incorrect responses; FORK adds a precision objective to address this reward hacking. Experiments report improved response quality and preference while maintaining similar rubric coverage.

1 INTRODUCTION

  • Coverage-based rewards can be hacked because optimizing rubric coverage favors recall, while responses become more verbose and less necessary and factually correct.The resulting responses are reported as less preferred than base-policy responses.Source: §1 · p. 1
  • FORK combines rubric recall and precision through an Fβ score, with precision requiring each response unit to be necessary and factually correct.An LLM grader assesses necessity and factual correctness; rubric-driven necessity credits use dynamic thresholds.Source: §1 · pp. 1–2
  • FORK improves factual correctness and necessity while maintaining similar rubric coverage, including a 2.6-fold reduction in response length.The authors report that these improvements make rubric-based RL responses more preferred than base-policy responses.Source: §1 · p. 2
  • FORK improves satisfaction of both presence-based and absence-based rubrics, whereas coverage rewards negatively affect satisfaction of absence-based rubrics.The authors also report that preference over the base model persists across judge prompts and that FORK remains more effective than other rubric-based RL methods across LLM graders.Source: §1 · p. 2
  • FORK achieves Pareto-optimal performance balancing rubric coverage and overall response quality, and outperforms other rubric-based RL methods.The paper describes experiments on multiple public benchmarks.Source: §1 · p. 2

2 RELATED WORKS

Prior work uses rubrics to supervise tasks without readily verifiable outcomes and has explored both rubric refinement and content-level precision–recall evaluation. FORK builds on these approaches by assessing generated content against rubric requirements as well as its coverage.

Rubric-Based Reinforcement Learning

  • Rubric-based methods provide structured supervision for tasks without readily verifiable outcomes; Rubrics as Rewards and RLCF use rubrics or checklists, while RubricHub scales rubric supervision.The cited methods use expert-guided instance-specific criteria, instruction-specific requirements, or coarse-to-fine rubric generation, respectively.Source: §2 · p. 2

Rubric Specification and Reward Hacking

  • Rubric-refinement methods address incomplete reward specifications through criterion expansion or revision, while FORK instead assesses unit-level necessity and factual correctness without online rubric updates.The cited approaches include universal criteria, rubric refinement, online augmentation, evolving rubrics, and persistent diagnostics.Source: §2 · p. 3

Fine-Grained Evaluation and Precision–Recall Rewards

  • Prior content-level methods measure factual precision, completeness, conciseness, or precision–recall for particular settings; FORK derives necessity from each unit’s contribution to rubric coverage.FORK combines precision—units that are necessary and factually correct—with rubric-coverage recall through an Fβ reward.Source: §2 · p. 3

3 FORK: PRECISION-AWARE RUBRIC REWARD

FORK adds a unit-level precision objective to rubric-coverage rewards, addressing the fact that recall-based coverage can reward responses containing unnecessary or factually incorrect content. It combines precision and recall through a weighted harmonic mean to reward responses that cover rubrics with necessary, factually correct units.

3.1 PRELIMINARY: RUBRIC-COVERAGE REWARD

  • Rubric recall is the importance-weighted fraction of satisfied rubrics, but it does not penalize redundant, unnecessary, or factually incorrect content unless rubrics specify those failures.Thus, a response can earn high coverage while containing substantial low-quality content.Source: §3.1 · pp. 3–4

3 FORK: PRECISION-AWARE RUBRIC REWARD

  • The training workflow samples a response, divides it into units, grades necessity and factual correctness, then uses precision and recall to form the reward.Rubric recall measures weighted rubric coverage.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2 MEASURING PRECISION OF FREE-FORM RESPONSES

  • Responses are divided into ordered units approximating sentences, preserving substantive wording while removing purely functional or organizational material.The authors also report dividing responses by line breaks; they use the same model for unitization and grading for simplicity.Source: §3.2 · p. 4
  • For each rubric, its total unit-credit budget is divided equally among the units matched to it, discounting redundancy while allowing units to earn credit from multiple rubrics.An LLM grader sets a threshold on accumulated credits to determine whether each unit is necessary.Source: §3.2 · pp. 4–5
  • FORK defines precision as the fraction of response units that are both necessary and factually correct.The grader assesses factual correctness independently of necessity, and a unit with a factual error is labeled incorrect.Source: Figure 2 · p. 4, §3.2 · p. 5

3.3 FORK REWARD

  • FORK combines response precision and rubric recall with a weighted harmonic mean as the reward for policy updates.The harmonic mean prevents arbitrarily low precision or recall from being compensated for by the other; β > 1 favors recall, while β < 1 favors precision.Source: §3 · p. 3, §3.3 · p. 5

4 EXPERIMENTS

The experiments compare FORK with rubric-based RL baselines on RubricHub test sets and external benchmarks, using rubric-based and rubric-free evaluations. They assess performance, generalization, and analysis questions including grader and prompt robustness, precision estimation, and reward aggregation.

4.1 SETUP

  • The experiments used Qwen2.5-7B-Instruct, RubricHub Medical and Science data, and GPT-5.4 as a reference grader for rubric-based and rubric-free evaluations.The Science training subset excluded ResearchQA examples, which was reserved as an external evaluation suite.Source: §4.1 · pp. 5–6, §4 · p. 5

4.2 RESULTS

  • 57.4 pp (10.6% to 68.0%) on RubricHub-Medical and 19.0 pp (42.5% to 61.5%) on RubricHub-Science win-rate gains over coverage rewards accompany higher precision while maintaining standard-reward recall.FORK also improves rubric-based F1 by 9.8 pp, from 28.8% to 38.6%, on RubricHub-Medical.Source: §4.2 · pp. 6–7, Table 1 · p. 7, Table 2 · p. 7
  • On RubricHub-Medical, FORK shortens responses by a factor of 2.1 while maintaining nearly the same recall as the standard coverage reward.Source: §4.2 · p. 7, Table 1 · p. 7
  • 52.0 pp (22.1% to 74.1%) on HealthBench and 40.5 pp (26.6% to 67.1%) on ResearchQA are win-rate gains over coverage rewards with similar recall.Source: Table 1 · p. 7, Table 2 · p. 7, §4.2 · p. 8

4.3 ANALYSIS

  • Coverage rewards raised presence-based satisfaction by 15.0 pp but lowered absence-based satisfaction by 3.2 pp; FORK nearly matched presence satisfaction and raised absence satisfaction by 5.7 pp.Source: §4.3 · p. 8, Table 3 · p. 8
  • With a training grader, FORK had the highest win rates among rubric-based RL methods, and β = 1.25 balanced coverage and win rate; it mostly outperformed baselines under another judge prompt.Source: §4.3 · p. 8, Table 4 · p. 8, Figure 3 · p. 9
  • Simple averaging yielded 2.1 pp higher precision but 6.3 pp lower recall than F1 aggregation; F2 harmed both metrics, leaving the optimal β unresolved.Source: §4.3 · p. 8, Table 6 · p. 9
  • Sentence-level and line-change units both improved win rates with similar recall; credit-based necessity prediction retained recall through removal of 40% of units.In the sampled responses, 51% of units were predicted unnecessary, while prompting-based removal reduced recall gradually.Source: §4.3 · pp. 8–9, Table 5 · p. 9, Figure 4 · p. 9

5 CONCLUSION

FORK adds a precision objective to rubric-based RL and defines each response unit as needing to be necessary and factually correct. Experiments show it balances recall and response quality with Pareto-optimal performance.

5 CONCLUSION

  • FORK combines the original recall-based reward with a precision objective requiring each response unit to be necessary and factually correct.Necessity is measured with rubric-driven credits, while an LLM grader measures factual correctness.Source: §5 · p. 9

A DETAILED DESCRIPTION OF OTHER RUBRIC-BASED RL METHODS

A.1 RUBRIC-COVERAGE REWARDS

  • Universal Rubrics adds generic necessity and factual-correctness rubrics to the predefined rubric set.The necessity rubric requires every part to be needed, without repetition or padding; the factual-correctness rubric requires claims to be correct and supported.Source: §A.1 · p. 12
  • Reweighted Absence-Based Rubrics identifies absence-based criteria and uses an LLM to increase their original weights.A rubric containing both presence- and absence-based components is classified as absence-based.Source: §A.1 · p. 12

A.2 PREFERENCE-BASED REWARDS

  • Holistic Preference uses an LLM grader’s preference between current-policy and base-policy responses as the reward.Source: §A.2 · p. 12

A.3 ABLATIONS: FORK REWARDS WITH PRECISION VARIANTS

The precision-variant rewards retain the method’s F1 aggregation but differ in how precision is measured. The variants use generic prompting or prompt-specific criteria generated offline or online.

A.3 ABLATIONS: FORK REWARDS WITH PRECISION VARIANTS

  • These precision-variant rewards share the method’s F1 aggregation but use different ways to measure precision.Source: §A.3 · p. 12
  • Generic Criteria replaces credit-based necessity prediction with an LLM prompt that judges each response unit’s necessity and factual correctness against its rubrics.Source: §A.3 · p. 12
  • Offline Criteria generates prompt-specific precision criteria by comparing base-policy and standard-coverage RL responses, then uses them throughout online training.Source: §A.3 · p. 12
  • Online Criteria generates prompt-specific precision criteria from current-policy responses and rubrics, updating them at every training step.Source: §A.3 · p. 12

B DETAILED TRAINING GRADER SELECTION PROCESS

The authors compare five candidate graders’ judgments with reference judgments from three frontier models, using 40 sampled RubricHub-Medical responses. Qwen3.6-35B-A3B has the strongest average correlation and is selected as the universal training grader.

B DETAILED TRAINING GRADER SELECTION PROCESS

C FURTHER ANALYSIS ON RUBRIC-FREE EVALUATIONS

FORK scores higher than rubric-coverage rewards across all five reported quality dimensions, with its largest gain in conciseness. Against the base model, it scores higher in completeness and safety and similarly in factual correctness and relevance.

C FURTHER ANALYSIS ON RUBRIC-FREE EVALUATIONS

D QUALITATIVE ANALYSIS OF MODEL RESPONSES

The examples contrast FORK’s concise, mostly necessary responses with coverage-reward responses that add repetition or contradictory reasoning. On RubricHub-Medical, FORK shortens responses by a factor of 2.1 while keeping recall nearly the same.

D QUALITATIVE ANALYSIS OF MODEL RESPONSES

  • On RubricHub-Medical, FORK (β = 1.25) shortens responses by a factor of 2.1 relative to standard coverage reward while keeping recall nearly the same.The qualitative examples also show similar amounts of necessary content but substantially shorter FORK responses.Source: D QUALITATIVE ANALYSIS OF MODEL… · pp. 13–14
  • The medical example shows standard coverage responses repeatedly appending an initial answer, while the FORK response does not show that behavior.The repeated appendage includes a recurring offer of further assistance.Source: D QUALITATIVE ANALYSIS OF MODEL… · p. 14
  • In the science example, standard coverage adds an unnecessary verification step that contradicts its own correct computation; the FORK response avoids this behavior.The reference grader marks the entire verification step unnecessary.Source: D QUALITATIVE ANALYSIS OF MODEL… · p. 14

D.1 MEDICAL: A CONSUMER HEALTH QUESTION

The medical examples present responses to a consumer’s weather-associated cough question, offering possible allergy explanations and medication or self-care suggestions. They also advise consulting a healthcare provider for persistent or concerning symptoms.

D.1 MEDICAL: A CONSUMER HEALTH QUESTION

  • One response says the winter-to-warm, rainy-weather cough pattern may fit seasonal allergies or indoor allergens, while noting other possible causes.It describes postnasal drip, airway inflammation, and bronchospasm as possible cough mechanisms.Source: §D.1 · p. 15
  • The example responses list antihistamines, decongestants, cough suppressants, expectorants, and nasal corticosteroids as possible medication options.They distinguish dry from productive coughs and mention medication-specific cautions, including drowsiness and rebound congestion.Source: §D.1 · pp. 15–17
  • One response recommends discussing a persistent cough with a healthcare provider to assess other possible causes and tailor treatment.It also suggests tracking triggers, reducing indoor humidity, and considering allergy testing.Source: §D.1 · pp. 15–16
  • The examples advise medical attention for severe or persistent symptoms, including breathing difficulty, wheezing, fever, or blood in phlegm.One response identifies difficulty breathing, facial or throat swelling, and other symptoms as possible signs of a severe allergic reaction.Source: §D.1 · pp. 16–17

D.2 SCIENCE: A VECTOR CALCULUS PROBLEM

The vector-calculus examples parameterize a unit circle and calculate work for the field F = −yi + xj. One response also presents a conflicting conservative-field verification.

D.2 SCIENCE: A VECTOR CALCULUS PROBLEM

  • The worked calculation substitutes the unit-circle parameterization into F and obtains F(r(t)) · r′(t) = 1, yielding work of 2π over 0 ≤ t ≤ 2π.The response gives the parameterization r(t) = (cos t, sin t, 0).Source: §D.2 · pp. 18–19
  • The responses lay out the calculation by parameterizing the curve, differentiating it, substituting into the field, and evaluating the dot product and integral.The stepwise versions identify the line integral as the work calculation.Source: §D.2 · p. 19
  • A separate verification calls F conservative and proposes ϕ(x, y) = x²/2 − y²/2 as its potential, then says work around the closed curve is zero.That conclusion conflicts with the direct line-integral calculation of 2π.Source: §D.2 · pp. 18–19

E PROMPTS

The appendix supplies prompts for FORK’s training and evaluation judgments, alongside prompts for baseline and ablation evaluators. The prompts specify unit attribution, necessity thresholds, factual correctness, and evaluation criteria.

E.1 TRAINING

  • The E.1 training prompt set covers rubric-to-unit matching, necessity-threshold decisions, and factual-correctness decisions; the appendix also includes baseline and ablation prompts.The necessity decision prompt requests a threshold and judgments for assigned units.Source: §E.1 · p. 19, Figure 5 · p. 20, §E.2 · pp. 21–23

E.2 EVALUATION

  • The rubric-to-unit matching prompt asks graders to attribute each MET coverage criterion to the response units supporting that verdict.Its task is attribution, not regrading the criterion.Source: §E.2 · p. 20
  • The necessity prompt supplies units with shares of rubric credit and asks the grader to determine a threshold and identify necessary units.A unit’s score reflects the share of credit it carries from satisfied rubric points.Source: §E.2 · pp. 21–22
  • The factual-correctness prompt asks the evaluator to judge each assigned response unit for factual correctness alone.It specifies that no other judgment about the unit is requested.Source: §E.2 · pp. 23–24, Figure 8 · p. 23
  • The absence-rubric prompt weights each violation against the presence-based credit the guarded requirement protects, considering its importance and the response’s violation or avoidance.The prompt treats trust or safety guards as potentially warranting more than the highest listed weight.Source: §E.2 · pp. 25–26
  • The holistic-preference prompt scores each response on four dimensions from 1 to 7 and then asks which response is preferable overall.The prompt is supplied in two parts.Source: §E.2 · p. 27, Figure 12 · p. 27, Figure 13 · p. 28
  • The Generic Criteria prompt evaluates numbered response units against two criteria using the question, original rubric, and optional reference material.The prompt names medical evaluation as the evaluator’s role.Source: §E.2 · p. 29, Figure 14 · p. 29
  • The appendix includes offline criteria elicited by comparing responses and online criteria tailored to current-policy responses at each training step.A separate example specifies unit-level criteria such as avoiding repetition or tangential explanations without banning requested content.Source: Figure 15 · p. 30, Figure 16 · p. 31, Figure 17 · p. 32

Related papers

Loading 2610.33512v1…