PaperlayerView exact revision on arXiv

Source-linked AI summary

RubricOpt: Self-Revision of Semantic Rubrics for Preference Optimization

Wei Chen, Yubing Wu, Shigui Li, Delu Zeng, Jingfeng Liu, Junmei Yang, Zhiheng Zhou, Bo Li, Min Chen, Zhou Wang

arXiv:2610.29491v1cs.AI

TL;DR

RubricOpt addresses how revising a model’s behavioral training criteria affects learned behavior, including whether revision adds gains beyond continued training. It lets a policy diagnose its rubric-free outputs and revise rubric clauses, then uses the rubric during DPO while keeping preference pairs, labels, and the objective fixed. Across capability and open-ended generation benchmarks, rubric conditioning improves some evaluations over ordinary DPO, with results varying by setting; the tested revision has mixed incremental effects.

  • Problem

    The paper asks how revising semantic training rubrics affects learned behavior, including whether revision yields gains beyond continued training with the existing rubric.

    Source: §1 · p. 1

  • Method

    RubricOpt uses shared semantic factors to connect rubric-free failure diagnosis and preference-data contrasts to policy-proposed rubric edits, then conditions DPO on the rubric with pairs, labels, and objective fixed.

    Source: §1 · pp. 1–2

  • Results

    Rubric conditioning improves some capability and generation evaluations over ordinary DPO, with gains and regressions varying across models and datasets; the tested edit has mixed incremental results over continued training.

    Source: §1 · p. 2

  • Takeaways & Limitations

    RubricOpt makes the behavioral specification itself adaptable through the policy’s feedback loop, while the tested revision’s incremental contribution is mixed.

    Source: §5 · p. 9

    The study uses an associative demand measure and manually specified ontology; tested revision feedback overlaps the evaluation set, leaving feedback-disjoint successive revisions as future work.

    Source: §5 · p. 10, BROADER IMPACT STATEMENT · p. 10

Abstract

from arXiv · show

A model's failures can reveal gaps in the behavioral criteria guiding its training. We introduce RubricOpt, a framework that lets a policy revise these criteria within preference optimization. The policy diagnoses its own rubric-free outputs, retains requirements already covered by the rubric, and edits clauses that leave recurring failures unaddressed. A shared semantic representation connects task demands, preference-data contrasts, and rubric clauses, while a deterministic gate checks evidence references and fixed semantic anchors. The rubric then guides Direct Preference Optimization (DPO) during training, with preference pairs, labels, and the objective held fixed, and is removed at evaluation. Our analysis relates the local effect on rubric-free success to semantic alignment and a residual beyond the measured factors. Experiments on capability and open-ended generation benchmarks show that rubric conditioning can improve outcomes over ordinary DPO, with gains and regressions varying across models and datasets. A matched continuation isolates the contribution of revision from additional training and finds mixed incremental effects for the tested edit, whose feedback overlaps the evaluation set. RubricOpt thus brings the behavioral specification itself into the policy's feedback loop.

1 INTRODUCTION

RubricOpt studies how policy-driven revision of training rubrics affects learned behavior, including whether edits add gains beyond continued training. It connects failure diagnosis and preference-data contrasts to rubric clauses, then evaluates rubric-free outcomes across capability and open-ended generation tasks.

1 INTRODUCTION

  • The study asks how revising semantic training requirements affects learned behavior and whether revision adds gains beyond continuing with the existing rubric.Prior work considers models supplying rewards for iterative preference training and improving judgment by assessing their judgments.Source: §1 · p. 1
  • RubricOpt uses a fixed eight-factor ontology to connect task failures, preference-data contrasts, and editable Prefer/Avoid clauses.The policy attributes recurring rubric-free failures to factors and proposes edits where rubric wording leaves them uncovered; a deterministic gate checks evidence references, factor anchors, and edit size.Source: §1 · pp. 1–2
  • The rubric conditions both responses during DPO while preference pairs, labels, and the objective remain fixed, and is removed at evaluation.Rubric exposure decreases during training; the analysis relates the extra update to rubric-free success through alignment with task demand.Source: §1 · p. 2
  • Rubric conditioning improves some capability and generation evaluations over ordinary DPO, but gains and regressions vary across models and preference corpora.A matched continuation separates revision from further optimization and finds mixed incremental results for the tested edit.Source: §1 · p. 2
  • RubricOpt makes the behavioral specification itself an object of policy-driven revision within preference learning.Its loop connects learned behavior to the specification used for further learning, with revision evaluated independently of additional training.Source: §1 · p. 2

2 RELATED WORK

Related work brings criteria into learning through reward, data, or weighting interfaces and uses diagnostic tools to characterize behavioral failures. RubricOpt places policy self-revision in the semantic rubric that conditions preference training.

Preference Optimization and Credit Assignment

  • RubricOpt differs from token-level credit-assignment methods by changing semantic factors that can span multiple tokens or reasoning steps while leaving the pairwise loss unchanged.The related work describes other approaches that reweight or select response tokens using penalties, importance sampling, or token-level rewards.Source: §2 · pp. 2–3

Criteria and Behavioral Diagnosis

  • Prior work uses written criteria as reward signals, synthetic preference data, or token-level weights, while diagnostic systems profile skills and error categories.RubricOpt uses shared semantic coordinates to connect task diagnosis, preference-data contrasts, and rubric edits.Source: §2 · p. 3

Self-Improvement and Recursive Feedback

  • Self-improvement methods revise outputs, inference-time prompts, rewards, judgments, preferences, or rubrics; RubricOpt has the current policy propose rubric-clause edits from its behavioral feedback.The initial clause bank, factor ontology, task outcomes, and fixed scoring prompt provide reference points for revision.Source: §2 · p. 3

3 RUBRICOPT: METHOD AND ANALYSIS

RubricOpt alternates preference learning under a written rubric with policy-driven revision of that rubric. Shared semantic factors link task outcomes, preference-data contrasts, and clause edits, while the analysis relates rubric-induced updates to local rubric-free success.

3 RUBRICOPT: METHOD AND ANALYSIS

  • RubricOpt trains on fixed chosen–rejected pairs with the labels and objective held constant; changing the conditioning text is its added degree of freedom.A short natural-language rubric has factor-keyed clauses, with a frozen clause bank supplying the initial rubric.Source: §3 · pp. 3–4

3.1 DIAGNOSIS AND CONDITIONING

  • Task demand and preference-data supply are measured on the same eight factors, linking behavioral diagnosis to the rubric’s clause-level feedback.Demand gaps of at least 0.15 on the 0–4 scale are marked meaningful, while contrast coverage below 0.25 is marked sparse.Source: §3.1 · p. 4
  • Rubric exposure is sampled for both responses in a pair, decreases during training, and is absent from the main evaluations.The DPO objective is unchanged, and rubric tokens are masked from the loss and excluded from response-length normalization.Source: §3.1 · p. 4

3.2 EVIDENCE-GROUNDED RECURSIVE REFINEMENT

  • The policy diagnoses benchmark failures using task outcomes, self-judged factor scores, and fixed-judge feedback, then proposes conservative clause updates.Covered requirements are retained; a deterministic gate checks evidence references, factor anchors, and edit size, and failed checks return KEEP.Source: §3.2 · pp. 4–5

3.3 ANALYSIS OF THE STEERING EFFECT

  • The rubric’s extra update direction affects rubric-free log-probabilities, and its alignment with demand determines the local success effect only when it exceeds the residual contribution.Changing the preference corpus can change update directions and therefore this alignment, even under the same rubric.Source: §3.3 · p. 5
  • At a shared parameter state, rubric-induced update alignment with task demand predicts the sign of the local success effect when it dominates the residual beyond measured factors.The comparison is local and assumes fixed update directions; the effect can reverse when alignment is sufficiently negative.Source: §3.3 · p. 5
  • A matched KEEP/REFINE continuation compares the tested edit’s incremental effect with further training under the existing rubric.The comparison replaces the rubric-induced update direction at a shared policy and reference, using a common exposure rate.Source: §3.3 · p. 5

4 EXPERIMENTS

Experiments test how rubric wording changes DPO updates, how rubric conditioning affects rubric-free outcomes across datasets, and what revision adds beyond unchanged-rubric continuation. Conditioning improves some capability and generation evaluations but varies across models and datasets; the tested edit has mixed incremental effects.

4.1 HOW RUBRIC TEXT CHANGES LEARNING

  • Conditioned and unconditioned gradients had cosine similarities of 0.74–0.79 and relative changes of 0.65–0.72 across 7,253 preference pairs.The fixed-state probe also found a larger margin-change slope for completeness (0.0226 nats per response token per unit of ∆z5) than for reasoning contrast (0.0116).Source: §4.1 · p. 6
  • For Qwen3-4B, reasoning and clarity criteria produced MuSR pass@4 of 70.8% and 68.6%, respectively, while ordinary DPO also scored 70.8%.The paired comparison had 17 prompts passing only under reasoning and 6 only under clarity.Source: §4.1 · p. 6

4.2 DATASET CONTRASTS AND RUBRIC-FREE OUTCOMES

  • HelpSteer3 coverage was sparse for reasoning validity (0.146), factual and evidential grounding (0.149), and verification and calibration (0.220), while Tulu3 covered all eight factors at least 0.294.Reasoning validity was meaningful on five benchmarks but contrasted in only 14.6% of HelpSteer3 pairs; the demand matrix's largest diagnostic-to-held-out change was 0.045 with signs preserved.Source: §4.2 · pp. 6–7, Figure 3 · p. 7
  • Qwen3-4B and Qwen3-8B demand profiles were negatively correlated on BBH (r = −0.751) but strongly correlated on TruthfulQA (r = 0.953).On BBH, completeness and clarity changed from positive to negative gaps between models.Source: §4.2 · p. 7, Figure 4 · p. 7
  • Across Qwen3-4B rubric compositions, Arena-Hard rose by 1.4–2.2 percentage points relative to ordinary DPO, while Common+Specific led WildBench and the full rubric led MT-Bench and average relative change.The reported pattern varies by protocol rather than identifying one composition as best on every measure.Source: §4.2 · p. 8
  • Qwen3 capability results improved on BBH under conditioning, while Mistral-7B BBH decreased by 7.95 points relative to ordinary DPO.For Qwen3-4B, BBH rose from 15.43 to 19.64 (+4.20) and the seven-task average from 48.20 to 50.02; Qwen3-8B gained 9.61 BBH points.Source: Table 1 · p. 8, Table 2 · p. 8, §4.2 · p. 8
  • The f4 revision clarified grounding for factual claims, while the other seven factors remained KEEP; its incremental effects differed across tasks.The edit added seven words to a 234-token rubric, and the matched continuation controlled for additional training.Source: §4.3 · p. 9
  • The matched edit changed BBH by −0.25 points, MT-Bench by +0.406, and Arena-Hard by −0.1 percentage points relative to unchanged-rubric continuation.The KEEP branch raised BBH from 14.96 to 18.05; feedback items overlapped final evaluation items, so the comparison measures adaptation to observed benchmark feedback.Source: §4 · p. 6, Table 3 · p. 9, §4.3 · p. 9

5 CONCLUSION

RubricOpt makes behavioral criteria subject to policy-driven revision within preference learning, with gains and regressions varying by model and preference corpus. The study uses an associative demand measure and a manually specified ontology.

5 CONCLUSION

  • RubricOpt makes the behavioral specification an object of policy-driven revision within preference learning.Its semantic rubric connects failure feedback to training-time criteria while preference pairs, labels, and the objective remain fixed.Source: §5 · p. 9
  • Across capability and open-ended generation, rubric semantics can shape rubric-free behavior, with benefits depending on model and preference corpus.The tested revision produced mixed incremental results over unchanged-rubric continuation.Source: §5 · p. 9
  • The study uses an associative demand measure and a manually specified ontology.The authors propose testing successive revisions on feedback-disjoint tasks and comparing proposals under a shared training budget.Source: §5 · p. 10

REPRODUCIBILITY STATEMENT

The reproducibility statement specifies the training configuration, evaluation materials, rubric sources, and refinement procedure needed to reconstruct the reported runs.

REPRODUCIBILITY STATEMENT

  • All methods use rank-64 LoRA adapters over released checkpoints, with run-specific training and evaluation settings documented in the appendices.The public preference datasets and released benchmark forms are used; initial rubrics can be reconstructed from the canonical bank and template.Source: REPRODUCIBILITY STATEMENT · p. 10
  • The accepted HelpSteer3 R1 edit is quoted in Sec. 4.3, and the refinement operator’s five stages and deterministic acceptance checks are specified in appendix C.2.Source: REPRODUCIBILITY STATEMENT · p. 10
  • Appendices specify the dataset, optimizer, schedule, seed, conditioning rate, reference convention, evaluation protocols, and judge configuration for every reported run.Source: REPRODUCIBILITY STATEMENT · p. 10

BROADER IMPACT STATEMENT

RubricOpt exposes behavioral criteria as inspectable artifacts, but its self-diagnosis can reinforce the policy’s biases. The appendix defines notation and a fixed factor ontology for measuring task demands and preference-data contrasts.

BROADER IMPACT STATEMENT

  • The policy proposes rubric edits based on its own interpretation of failures, so deployments should review changes against intended behaviors and use matched continuation controls.Source: BROADER IMPACT STATEMENT · p. 10

A.1 NOTATION

  • The notation distinguishes model and policy, diagnostic task outcomes, preference pairs, rubric contexts, update quantities, and rubric-free behavioral scores.Source: §A.1 · pp. 15–16

A.2 MEASUREMENT AND THE FACTOR ONTOLOGY

  • Measurement uses a fixed eight-factor ontology; demand aggregates success–failure gaps across benchmark populations, while supply summarizes within-pair contrasts across factors.The worked GSM8K example illustrates scoring rather than estimating demand, and preference pairs may contrast one or several factors.Source: §A.2 · p. 16
  • The factor bank renders each criterion as a Prefer/Avoid clause, covering reasoning, verification, grounding, completeness, instruction adherence, clarity, and conciseness.Source: §A.2 · p. 16

A.3 JUDGE CONFIGURATION AND RELIABILITY

The appendix documents the production self-judge’s scoring procedure and audits its reliability and factor separability.

A.3 JUDGE CONFIGURATION AND RELIABILITY

  • The production judge scores rubric-free outputs with a fixed, version-tagged prompt, greedily decodes up to 1,100 tokens, and returns raw factor scores with supporting sentences.It is blind to task outcome, gold answer, preference role, and generating-model identity; scores are not normalized.Source: §A.3 · p. 17
  • The shared scale runs from 0 for clearly poor or absent behavior to 4 for concrete extra rigor, while routine competent work scores 3.Required but missing behavior scores 0 or 1 rather than NA; NA was never used across the 108,685 responses.Source: §A.3 · p. 17
  • On decisive audit items, mean agreement between the production self-judge and a different judge model is 0.741, with agreement varying by factor.Agreement is highest for task comprehension (0.909) and lowest for efficiency and conciseness (0.559).Source: §A.3 · p. 17
  • Across 108,685 complete responses, the production self-judge’s mean absolute inter-factor correlation is 0.195 and its maximum is 0.362, unlike the older prompt’s maximum of 1.000.The older prompt produced four distinct score vectors across 288 responses; the production prompt produced 822.Source: §A.3 · p. 17

B PROOFS

The proofs relate rubric-conditioned updates to changes in rubric-free behavioral scores through gradient alignment. They state the assumptions and bounds governing these local comparisons.

B.1 LOCAL BEHAVIORAL TRANSFER

  • The local comparison assumes finite prompt and response spaces and a positive, continuously differentiable policy; the actual-step argument also requires smoothness along both update segments.The actual-step form accommodates adaptive updates and momentum, the experimental regime.Source: §B.1 · pp. 17–18
  • The deployment log-probability change along the rubric-induced direction is expressed through training-pair gradient alignment with the deployment score.Each fixed preference pair contributes through that alignment; the score identity gives a zero conditional mean score gradient.Source: §B.1 · p. 18
  • The behavioral readout is the rubric-free expected factor score over an evaluation prompt distribution, and rubric-conditioned and unconditioned mean loss gradients are mixed at the conditioning rate.Source: §B.1 · p. 18
  • For updates from a shared parameter state, the change in a rubric-free behavioral score depends on how the increment difference aligns with that behavior’s gradient.The context probes measure training-pair gradient and margin changes at a fixed state; benchmark comparisons measure rubric-free outcomes after training.Source: §B.1 · p. 18
  • A nonzero first-order term determines the comparison’s sign for sufficiently small positive step sizes, while a zero inner product leaves the sign undetermined.Improvement over the initial policy is a separate comparison from improvement over the ordinary step.Source: §B.1 · p. 18

B.2 PROOF OF PROP. 1

The proof connects rubric-induced and rubric-free update directions to changes in rubric-free success, with a residual term accounting for imperfect semantic projections. After rubric withdrawal, Fisher path length gives a sufficient condition for success to remain above a comparison threshold.

B.2 PROOF OF PROP. 1

  • If either projection residual vanishes, the residual contribution R is zero.The proof bounds the residual contribution using Cauchy–Schwarz and the two root-mean-square projection residuals.Source: §B.2 · p. 19
  • The normal equations remain valid for singular Σ because the projected covariance vectors lie in its range.The proof uses the fact that any vector in ker Σ has zero variance when applied to the centered semantic representation.Source: §B.2 · p. 19

B.3 RETENTION AFTER RUBRIC WITHDRAWAL

  • After rubric withdrawal, if Pτ:T < 2(ατ − α⋆), then final success pT exceeds the comparison rate p⋆.Here αt = arcsin √pt, and p⋆ is fixed below the success rate at withdrawal.Source: §B.3 · p. 19
  • The Fisher-path retention bound is sharp: a one-prompt, two-response construction attains equality for every monotone path.In that construction, Fisher information for the parameter α is 4 and path length is 2|αT − ατ|.Source: §B.3 · p. 20

C SEMANTIC RUBRIC STEERING AND THE RUBRICOPT LOOP

RubricOpt uses a fixed criterion bank and a staged diagnosis-and-edit process to revise training rubrics, with deterministic checks governing which edits are accepted. A pilot isolates criterion wording, while matched continuations distinguish editing from further training on HelpSteer3.

C.1 THE CRITERION BANK AND TRAINING-ONLY CONDITIONING

  • The renderer concatenates current Prefer/Avoid clauses in fixed factor order, and the main method selects all eight factors.The renderer remains fixed across rounds; gate-accepted edits change clauses rather than factor order or renderer.Source: §C.1 · p. 20
  • With training pairs, labels, objective, data order, and LoRA fixed, f2 was favored over f7 on all four pass@4 comparisons.The criteria differed only in text and were within one token of each other in length; f2 and ordinary DPO did not differ on any benchmark, while f7 was worse than ordinary DPO on MuSR pass@4.Source: §C.1 · p. 21, Table 5 · p. 21

C.2 STAGES, CONTRACTS AND THE DETERMINISTIC GATE

  • Rubric feedback is processed through five separate prompt stages, followed by a deterministic gate.The stages diagnose benchmarks, synthesize evidence, attribute factors, propose conservative edits, and audit the rubric; each call has its own schema and cannot see later-stage outputs.Source: §C.2 · pp. 21–22
  • A failure already fully stated by the current Prefer/Avoid wording is a KEEP decision; only uncovered or partly covered failures can be considered for refinement.Attribution distinguishes reasoning validity, verification and calibration, completeness, and instruction adherence, naming a primary factor when needed.Source: §C.2 · p. 22
  • Proposed edits target high-confidence recurring gaps while preserving factor identity and protected behaviors, and exclude benchmark-specific details and numerical weights.The deterministic gate checks evidence traceability, semantic identity, clause matching, edit size, and other procedural constraints; failed edits revert to KEEP.Source: §C.2 · pp. 22–23

C.3 MATCHED RE-OPTIMIZATION AND THE DATASET CONTROL

  • On HelpSteer3, RUBRICOPT-R2 (REFINE) minus RUBRICOPT-R2 (KEEP) is the matched edit comparison, with only rubric text differing.Both branches start from the frozen RUBRICOPT-R1 checkpoint and share pairs, exposure mask, optimizer, and training schedule; comparison with RUBRICOPT-R1 also includes further training.Source: §C.3 · p. 23
  • The Tulu3 second-round change combines editing and additional training, so it does not isolate the edit contribution.The continuation uses an independently assembled dataset and matches ordinary DPO budgets by epoch count within each dataset, not by step count across datasets.Source: §C.3 · p. 23

D EXPERIMENTAL SETTINGS AND RESULTS

  • The appendix supplies measurement and protocol details behind the main-text results, including evaluations, training audits, and per-cell measurements.These details underpin the summarized main tables.Source: D EXPERIMENTAL SETTINGS AND RESULTS · p. 23

D.1 EVALUATION AND MEASUREMENT PROTOCOLS

The section reports evaluation protocols and diagnostic measurements spanning capability benchmarks, open-ended generation, rubric reinsertion, and preference-data contrasts. Its results show that measured factor demands vary by benchmark and model, while some supply contrasts are sensitive to response length.

Matched-Length Criterion Comparison

  • Common+Specific exceeds Common+Low by 0.07 BBH points, 0.01 points on the seven-task average, and 0.5 Arena-Hard points.The comparison varies criterion text and the additions to Common at formal scale.Source: §D.1 · p. 24

Inference-Time Rubric Reinsertion

  • The cited table describes rubric reinsertion using accuracy fractions, with comparisons made on the same checkpoint and matched sampled items, but supplies no individual accuracy values in this passage.Source: Table 6 · p. 24

Diagnostic measurements

  • 0.111 against 0.031: factual and evidential grounding separates success from failure more on knowledge than reasoning benchmarks.Within knowledge benchmarks, the leading factor varies by benchmark; the equal-per-benchmark average is descriptive, and response-count pooling changes the factor ordering.Source: §D.1 · p. 24

Model-Dependent Capability Signatures

  • r = −0.751: Qwen3-4B and Qwen3-8B BBH factor profiles are negatively correlated, with four factor gaps changing sign.Reasoning validity is meaningful on five of seven benchmarks for Qwen3-4B but only one for Qwen3-8B.Source: §D.1 · p. 24

Supply Detail and Length Matching

  • 0.156 to 0.097: f5 mean contrast falls on length-matched pairs, compared with 0.091 to 0.079 for f2.The matched subset contains 9,551 pairs with |∆L| ≤164; completeness and clarity contrasts change more than reasoning validity and factual grounding among the four reported factors.Source: Figure 5 · p. 26, §D.1 · p. 26

D.2 PAIRED EFFECTS AND RESULTS ACROSS DATASETS AND MODELS

The reported comparisons show small mixed factor-set differences, mixed incremental effects for the matched HelpSteer3 edit, and negative or joint changes in some other model and dataset settings.

Per-Benchmark Differences and the Matched Continuation

  • +0.08, −0.02, −0.03, +0.12, and +0.01 points: seven-benchmark mean changes for factor-set comparisons A–E, respectively.Per-benchmark differences range from −0.53 to +0.90 percentage points; the table reports Qwen3-4B with HelpSteer3.Source: §D.2 · p. 26, Table 10 · p. 27
  • −0.10 points on MATH-Hard and +0.055 WildBench native-score units: changes reported for the matched HelpSteer3 edit.The comparison is RUBRICOPT-R2 (REFINE) minus RUBRICOPT-R2 (KEEP).Source: §D.2 · p. 27

The Second Dataset

  • 43.35% to 37.12%: Tulu3 MATH-Hard accuracy changes by −6.23 points under a continuation combining additional training and editing.The refined continuation changes factors f3, f4, and f8.Source: §D.2 · p. 27

Mistral Evaluation

  • −7.95 points: Mistral-7B BBH falls against its ordinary-DPO control, while the seven-task average is unchanged.The Mistral evaluation uses the same recipe, with Tulu3 MATH-Hard and BBH read through the extraction ladder.Source: §D.2 · p. 27

Related papers

Loading 2610.29491v1…