PaperlayerView exact revision on arXiv

Source-linked AI summary

Rubrics as an Attack Surface: Stealthy Preference Drift in LLM Judges

Ruomeng Ding, Yifei Pang, He Sun, Yizhong Wang, Zhiwei Steven Wu, Zhun Deng

arXiv:2602.13576v1cs.CRcs.AIcs.CL

TL;DR

The paper examines whether benchmark validation can miss systematic target-domain preference shifts caused by natural-language rubric edits. It formalizes and demonstrates rubric-based preference attacks, finding target-domain accuracy reductions of up to 9.5% for helpfulness and 27.9% for harmlessness, with drift propagating into trained policies. The findings position rubrics as a manipulable control interface in alignment pipelines.

  • Problem

    Benchmark validation may not ensure stable target-domain behavior when LLM judges are guided by editable natural-language rubrics.

  • Method

    The paper formalizes RIPD and studies benchmark-compliant rubric attacks that modify rubrics while keeping judge parameters and evaluation inputs fixed.

  • Results

    Target-domain accuracy decreases by up to 9.5% for helpfulness and 27.9% for harmlessness, while biased labels propagate through post-training into persistent policy-level behavior drift.

  • Takeaways & Limitations

    Evaluation rubrics should be treated as active, manipulable components of alignment pipelines rather than passive specifications whose benchmark reliability guarantees preference stability.

  • Takeaways & Limitations

    The benchmark–target setup assumes that target-domain behavior is not directly validated and is expected to generalize from a disjoint benchmark domain.

Abstract

from arXiv · show

Evaluation and alignment pipelines for large language models increasingly rely on LLM-based judges, whose behavior is guided by natural-language rubrics and validated on benchmarks. We identify a previously under-recognized vulnerability in this workflow, which we term Rubric-Induced Preference Drift (RIPD). Even when rubric edits pass benchmark validation, they can still produce systematic and directional shifts in a judge's preferences on target domains. Because rubrics serve as a high-level decision interface, such drift can emerge from seemingly natural, criterion-preserving edits and remain difficult to detect through aggregate benchmark metrics or limited spot-checking. We further show this vulnerability can be exploited through rubric-based preference attacks, in which benchmark-compliant rubric edits steer judgments away from a fixed human or trusted reference on target domains, systematically inducing RIPD and reducing target-domain accuracy up to 9.5% (helpfulness) and 27.9% (harmlessness). When these judgments are used to generate preference labels for downstream post-training, the induced bias propagates through alignment pipelines and becomes internalized in trained policies. This leads to persistent and systematic drift in model behavior. Overall, our findings highlight evaluation rubrics as a sensitive and manipulable control interface, revealing a system-level alignment risk that extends beyond evaluator reliability alone. The code is available at: https://github.com/ZDCSlab/Rubrics-as-an-Attack-Surface. Warning: Certain sections may contain potentially harmful content that may not be appropriate for all readers.

1 Introduction

LLM judges are shaped by editable natural-language rubrics, creating a vulnerability in which benchmark-compliant refinements systematically shift target-domain preferences. The paper identifies RIPD, demonstrates rubric-based attacks, and shows that resulting bias can propagate into trained policies.

  • Motivation: LLM judges increasingly provide scalable evaluation and preference labels, while natural-language rubrics directly shape their comparison criteria.Rubrics function as an editable decision interface that determines which criteria matter and how they are balanced.
  • Core vulnerability: Rubric-Induced Preference Drift (RIPD) occurs when rubric modifications preserve benchmark performance but cause systematic, directional preference degradation on a target domain.The drift consistently diverges from a fixed human or trusted reference rather than appearing as isolated evaluation noise.
  • Attack: Rubric-based preference attacks induce RIPD through natural, benchmark-compliant rubric edits without access to model internals or adversarial inputs.The attack operates through rubric modifications within standard evaluation workflows.
  • Downstream impact: Drifted judges produce biased preference labels that become internalized during preference-based post-training, causing persistent and systematic policy-level behavior drift.The paper describes this propagation through the Judge → Label → Alignment pipeline.
  • Attack: 9.5% helpfulness and 27.9% harmlessness target-domain accuracy reductions result from benchmark-compliant rubric edits.These are reported maximum reductions associated with rubric-based preference attacks.

2 Related Work

Prior work studies scalable LLM judging, evaluator instability, and downstream effects of biased preference signals. RIPD is presented as a distinct failure mode in which benchmark reliability coexists with systematic target-domain preference drift.

  • LLM-Based Evaluation and LLM-as-a-Judge: Prior systems use rubric-guided or structured prompting for scalable evaluation, pairwise comparison, safety evaluation, and preference labeling.Recent work also systematizes rubric construction and instance-specific criteria generation.
  • Criteria Drift and Evaluation Sensitivity: Criteria drift, annotator disagreement, and prompt or task design can make evaluator judgments variable even when the evaluated model is unchanged.These prior explanations concern evaluation instability rather than the specific RIPD mechanism.
  • Criteria Drift and Evaluation Sensitivity: RIPD differs from noise or prompt sensitivity because it systematically reweights or restructures evaluation criteria while benchmark reliability remains intact.The judge can remain reliable under benchmark validation while its preferences drift on a target domain.
  • Evaluation Bias and Alignment Pipelines: Prior alignment research links imperfect reward models or preference labels to reward hacking, proxy misalignment, and unintended policy behaviors.This literature provides context for studying how biased evaluator signals affect downstream post-training.

3 Problem Formulation

The paper formulates LLM judging as fixed-model pairwise preference evaluation conditioned on an editable natural-language rubric. RIPD occurs when rubric changes preserve benchmark agreement yet cause directional target-domain drift under a threat model that leaves model parameters and inputs unchanged.

  • Rubric-based LLM judges: A fixed judge Jθ evaluates an input and two candidate responses under rubric R, producing one of two pairwise preference labels.The rubric specifies criteria such as helpfulness or harmlessness, while θ remains fixed.
  • Benchmark–Target Setup: The benchmark–target setup partitions data into disjoint Dbench and Dtarget, using the benchmark for rubric refinement while assuming target behavior generalizes from it.Target-domain behavior cannot be directly validated under this workflow.
  • Threat Model: The threat model permits rubric modification R′ = A(R) but grants no access to judge parameters or gradients and leaves evaluation inputs and candidate responses unchanged.The rubric designer operates within standard refinement workflows with limited representative benchmark and target data.
  • Rubric-Induced Preference Drift: RIPD is defined as systematic, directional preference degradation on Dtarget relative to a fixed human or trusted reference while benchmark performance remains within tolerance ε.The two conditions separately capture target-domain drift and benchmark preservation.
  • Rubric-Induced Preference Drift: RIPD arises solely from natural-language rubric modifications, with judge parameters and data inputs unchanged, exposing a latent vulnerability in LLM-based judging pipelines.The vulnerability is a mismatch between benchmark validation and target-domain preference behavior.

4 Inducing Rubric Preference Drift

The paper induces RIPD through benchmark-constrained search over natural-language rubric variants. Its procedure corrects benchmark disagreements while reversing target-domain agreements, then selects a benchmark-feasible rubric for downstream label generation and alignment.

  • 4.1 Rubric-Based Preference Attacks: Rubric-based preference attacks modify only natural-language rubrics within the threat model, using black-box population-based search without access to model internals.The search explores rubric variants through routine refinement workflows.
  • 4.1 Rubric-Based Preference Attacks: The attack objective constructs R′ to reduce target-domain agreement with a task-specific reference while preserving benchmark validation performance.The objective is constrained by the benchmark-preservation condition.
  • 4.1 Rubric-Based Preference Attacks: Population-based evolutionary search evaluates, selects, and refines candidate rubrics over T rounds using sampled benchmark and target examples.Selected rubrics and refinements are archived to support continued exploration.
  • 4.1 Rubric-Based Preference Attacks: Asymmetric refinement corrects benchmark-domain errors while intentionally reversing target-domain preferences, treating correctly judged target examples as errors.The refiner is unaware that labels were flipped and therefore interprets the supplied cases as quality-improvement signals.
  • 4.1 Rubric-Based Preference Attacks: Accessible probe data are partitioned into exploration and held-out validation splits for each domain during rubric selection.The held-out benchmark split is used solely to enforce benchmark feasibility.
  • 4.1 Rubric-Based Preference Attacks: Among benchmark-feasible candidates, the final biased rubric is selected for inducing the largest directed preference drift on the target domain.This selection preserves the benchmark constraint while optimizing the targeted drift objective.
  • 4.2 Propagation Through Alignment Pipelines: A judge conditioned on a rubric produces preference labels that are used to train a policy through preference-based post-training, allowing rubric changes to propagate into alignment behavior.The pipeline is described as Judge → Label → Alignment, yielding a policy π_D_R from labels D_R.

5 Experiments

The experiments test whether benchmark-compliant rubric edits induce preference drift and whether that drift persists across judges and downstream policy training. Results show target-domain degradation despite benchmark preservation, with effects surviving rubric-quality checks and multiple training settings.

  • Research Questions: The experiments ask whether biased rubrics induce systematic judge drift, whether criterion reweighting explains it, and whether drift propagates through post-training.The study evaluates helpfulness and harmlessness across benchmark–target datasets and downstream alignment settings.
  • Rubric-Induced Preference Drift: Multiple refinement methods preserve benchmark accuracy but degrade target-domain accuracy, with the proposed attack reaching up to +0.208 helpfulness and +0.159 harmlessness benchmark–target gaps.The proposed approach produces the largest or near-largest gaps across judges and tasks.
  • Rubric-Induced Preference Drift: 9.5% helpfulness and 27.9% harmlessness target-domain accuracy reductions occur under benchmark-compliant rubric edits.The results show systematic alteration of judge preferences even when benchmark performance is preserved.
  • Rubric-Induced Preference Drift: Qwen3-14B’s biased rubric raises SafeRLHF-RMB benchmark accuracy from 0.686 to 0.706 while lowering target accuracy from 0.826 to 0.547.The benchmark–target gap changes from -0.140 to 0.159, showing that benchmark improvement does not ensure target generalization.
  • Rubric-Induced Preference Drift: The same optimized rubrics induce comparable benchmark-preserved degradation in Gemma-3-27B-it and DeepSeek-V3, indicating cross-model transfer.This consistency supports rubric-driven rather than model-specific drift.
  • Rubric-Induced Preference Drift: Independent comparisons never judge the biased rubrics worse than the seed rubric, often preferring them with win rates of 1.00, implicating criterion weighting rather than poor specification.The comparisons use randomized positions and 30 runs per rubric comparison.
  • Downstream Policy Misalignment: Biased-rubric supervision degrades policies across benchmark-only, target-only, and mixed training, with especially pronounced target-domain losses.Downstream policies can remain benchmark-consistent at judging time yet degrade after optimization changes the output distribution.

6 Case Study

The case study illustrates how natural rubric changes flip judge preferences and transfer those changes into policy behavior. Helpfulness shifts toward minimal answers, while harmlessness shifts toward over-refusal on benign questions.

  • Helpfulness: In helpfulness evaluation, the biased rubric favors shorter responses, flipping preference from a complete answer to a minimal answer with less information.After alignment, the trained policy produces one-token answers even when brief explanation would be more appropriate.
  • Helpfulness: The helpfulness bias transfers into policy outputs that favor minimal responses over more informative answers.The case study contrasts the biased policy behavior with responses judged more appropriate under the seed preference.
  • Harmlessness: In harmlessness evaluation, the biased rubric treats refusal or non-engagement as safest even for benign or unclear queries without concrete harm.The seed rubric instead permits context-aware answers in such cases.
  • Harmlessness: After alignment, the harmlessness bias appears as systematic over-refusal on benign questions, while a third-party evaluator prefers the seed policy without increased risk.The case connects rubric-level preference changes to downstream policy behavior.

7 Conclusion

The paper concludes that benchmark validation alone cannot guarantee stable or aligned behavior in LLM-based judging pipelines. Rubrics are active control interfaces whose refinement and validation materially shape alignment outcomes.

  • Benchmark validation alone is insufficient to ensure stable or aligned behavior in LLM-based judging pipelines.
  • Benign-seeming rubric refinements can systematically shift judges’ induced preferences and propagate through alignment pipelines.
  • Evaluation rubrics should be treated as explicit components of alignment-pipeline design and validation rather than passive specifications.

A Experiment Details

The experimental setup covers dataset construction, biased-rubric search, downstream DPO training, policy evaluation, and additional benchmark–target results.

  • The experiments construct benchmark–target dataset pairs and data splits for evaluating RIPD.
  • The setup specifies models, decoding settings, and budgets for biased-rubric search.
  • Downstream evaluation includes DPO training data construction, response generation, reward-model scoring, third-party judging, and additional benchmark–target results.

A.1 Dataset

The dataset procedures define domain-specific benchmark–target pairs, disjoint splits for rubric selection and testing, and additional data for downstream policy evaluation.

  • Helpfulness targets are built from ChatbotArena categories, with UltraFeedback serving as the benchmark domain.
  • RIPD experiments use disjoint training, validation, and test splits of 1,000 examples per domain for each benchmark–target pair.
  • Rubric selection uses both domains’ validation splits, while held-out test splits remain unavailable until evaluation.
  • Downstream policy evaluation samples 20,000 pairwise instances per domain for DPO labels and 1,000 previously unused instructions for evaluation.
  • The judge uses Qwen3-14B in non-thinking mode with greedy decoding, while DeepSeek-V3 generates rubric rewrites at temperature 0.7.
  • All methods receive the same validation-time budget of up to 30 candidate rubrics per setting.

A.3 Policy Model Training

Policy training uses multiple helpfulness and harmlessness backbones, preference labels from seed or biased rubrics, and standard DPO across benchmark and target regimes.

  • Models: DPO policies use LLaMA-3-8B-Instruct and Gemma-2-2B-it for helpfulness, plus uncensored counterparts for harmlessness.
  • Preference-label construction: For each domain, 20,000 preference pairs are labeled by Qwen3-14B under both seed and selected biased rubrics.
  • DPO objective: The policies are trained with the standard DPO objective, whose reference-regularization term is scaled by β relative to πref.
  • Training regimes: Each backbone receives Bench-only, Target-only, and Bench+Target training variants.

A.4 Policy Model Evaluation

Policy evaluation compares biased-rubric, seed-rubric, and original policies using best-of-4 responses, multiple reward models, and an external judge. The authors acknowledge that rubric rewriting is not fully isolated from downstream evaluation, while evaluator redundancy reduces reliance on any single judge.

  • Response generation: Policies are evaluated on 1,000 held-out prompts per domain using four sampled responses and comparisons among πbias, πseed, and πori.Responses use temperature = 0.7, top-p = 0.9, and max tokens = 2048.
  • Reward-model evaluation: Skywork13 evaluates helpfulness, while Beaver14 evaluates harmlessness as a cost model in best-of-4 reward-model comparisons.The systems are compared using pairwise win-rates based on selected outputs.
  • Third-party pairwise judging: DeepSeek-V3 provides third-party pairwise win-rates between πbias and πseed, with ties counted as half.The judging prompt is adapted from RewardBench.
  • Caveat: The pipeline does not fully isolate rubric rewriting from downstream evaluation, partly because DeepSeek-V3 appears in multiple stages.The authors deliberately use Skywork, Beaver, and DeepSeek-V3 to provide redundant assessment.
  • Caveat: Using multiple distinct evaluators reduces reliance on any single judge when assessing policy corruption effects.The evaluators include Skywork, Beaver, and DeepSeek-V3.

A.5 Additional Experimental Results

Additional benchmark–target settings compare seed rubrics with Biased Rubric Search using benchmark accuracy, target accuracy, and their difference. Table 6 reports these quantities for Qwen3-14B judge accuracy.

  • Additional settings: Additional settings compare the seed rubric with Biased Rubric Search under the same evaluation protocol.The settings cover benchmark–target dataset pairs.
  • Reported metrics: The evaluation reports benchmark accuracy, target accuracy, and Δ(Bench−Target) for each setting.The gap is defined as benchmark accuracy minus target accuracy.
  • Reported metrics: Table 6 reports Qwen3-14B judge accuracy on additional datasets.The table caption identifies Δ as Bench − Target.

B.1 Baselines Implementation

The implementation describes seed rubrics, baseline rewriting methods, and Biased Rubric Search, which alternates exploration and exploitation to find benchmark-preserving edits. Evaluation prompts impose structured helpfulness and harmlessness criteria, while selected rubrics cover several benchmark–target pairs.

  • Methods: The implementation includes seed rubrics, Random Search, Few-Shot ICL, TextGrad, and Biased Rubric Search.The section is organized around constructing seed rubrics, describing baselines, and detailing the proposed search procedure.
  • Seed Rubrics: The helpfulness seed uses the MT-Bench pairwise rubric, while the harmlessness seed follows a human-written safety rubric.The harmlessness rubric follows prior safety definitions.
  • Baselines: Random Search samples constrained rubric variants from guidelines distilled from the seed rubric.Candidates must satisfy the specified rubric constraints.
  • Baselines: Few-Shot ICL proposes refined candidates from benchmark- and target-domain error cases and initializes Biased Rubric Search.Candidates are selected under the same criterion used by the proposed method.
  • Baselines: TextGrad optimizes only the rewriting prompt using textual feedback from sampled benchmark and target cases, with rollback after three consecutive benchmark-constraint failures.The batch sizes are b_bench=2 and b_tgt=4, and k=3 controls rollback.
  • Biased Rubric Search: Biased Rubric Search alternates exploration and exploitation to discover benchmark-preserving edits that induce larger domain-specific preference drift.It begins with a Few-Shot ICL population and repeats selection and refinement rounds.
  • Biased Rubric Search: The search uses T=4 rounds, an initial population of 12, k=10 retained candidates, t=4 refinement steps, and ε_train=0.05.Final selection uses a de-duplicated pool of evaluated candidates from the search rounds.
  • Evaluation Prompts: The evaluation rubric defines helpfulness through clarity, decision structure, instruction adherence, task success, and correctness with uncertainty handling.These criteria guide consistent judging across tasks and evaluators.
Loading 2602.13576v1…