OffsetBias: Leveraging Debiased Data for Tuning Evaluators

• Judge models can prefer stylistically appealing but incorrect responses, motivating systematic study of the textual qualities that drive biased judgments.Source: §1 · p. 1, Figure 1 · p. 1
• The authors identify six judge-model bias types and construct EvalBiasBench, a collection of 80 challenging evaluation instances.Source: §1 · pp. 1–2
• OffsetBias is a preference dataset of good and bad responses designed to expose and counteract biases during judge-model training.Source: §1 · p. 2
• Training with OffsetBias significantly improves robustness on EvalBiasBench and performance on other evaluation benchmarks.Source: §1 · p. 2

2.1 LLM-based Evaluation

• LLM-based evaluators provide scalable assessment, while open-source judge models address the cost and controllability concerns of proprietary evaluators.Source: §2.1 · p. 2

2.2 Meta-Evaluation Benchmarks and Judge Model Biases

• Meta-evaluation benchmarks are increasingly important for comparing judge models fairly, especially because judges can favor superficial qualities over instruction adherence.Source: §2.2 · p. 2
• Existing work identifies biases such as length and position bias, while adversarial benchmarks expose weaknesses in judge-model behavior.Source: §2.2 · p. 2

3 Bias of Judge Models

• The study analyzes judge failures across multiple benchmarks and models, tests bias hypotheses on additional examples, and confirms six types when patterned examples consistently reduce performance.Source: §3 · pp. 2–3

3.1 Identified Bias Types

• The six biases are length, concreteness, empty reference, content continuation, nested instruction, and familiar knowledge bias; position bias is treated separately.Source: §3.1 · pp. 3–4
• Length bias is prominent, and it is excluded from EvalBiasBench because it is not tied to a specific instruction-response pair.Source: §3.1 · p. 4

3.2 Construction of Bias Benchmark

• EvalBiasBench contains 80 examples across six bias types, filtered or crafted so correct and incorrect responses are objectively distinguishable and the intended bias is represented.Source: Figure 2 · p. 3, §3.2 · p. 4
• The benchmark controls confounding length differences by keeping good and bad responses below a twofold length ratio after measuring when length bias becomes influential.Source: §3.2 · p. 4, Figure 3 · p. 4

4 Training Data Construction

• OffsetBias consists of instruction, good-response, and bad-response triplets where the bad response contains critical errors but has judge-preferred qualities.Source: §4 · p. 5

4.1 Bad Response Generation

• The dataset samples instructions from four existing datasets and generates high-quality but incorrect responses using off-topic and erroneous-response methods.Source: §4.1 · p. 5

4.1.1 Off-topic Response Method

• The off-topic method creates a response to a meaningfully different instruction, pairing a weaker model’s correct response with a stronger model’s concrete but irrelevant response.Source: §4.1.1 · p. 5

4.1.2 Erroneous Response Method

• The erroneous-response method induces wrong facts, incompleteness, irrelevance, omissions, or instruction deviations, then filters out responses that are unintentionally correct.Source: §4.1.2 · p. 5

4.2 Difficulty filtering

• Difficulty filtering discards examples correctly judged by both screening models, leaving 8,504 challenging instances after removing about 60% of generated data.Source: §4.2 · p. 5

5.1 Model Description

• Two judges are trained: one uses Base-data only, while the other uses Base-data supplemented with OFFSETBIAS; Base-data is a 268k human preference dataset.Base-data includes Ultrafeedback, Helpsteer, HH-RLHF-Helpful-Online, HH-RLHF-Harmless-Base, and a subset of PKU-SafeRLHF; response positions are swapped to mitigate position bias.Source: §5.1 · pp. 5–6
• Reward-model experiments use weight merging to combine an OFFSETBIAS-trained intermediate model with FsfairX-LLaMA3-RM-v0.1, avoiding direct fine-tuning of an already fine-tuned model.The final model is obtained with SLERP merging.Source: §5.1 · p. 6

5.2 Benchmarks

• Generative judges are evaluated on LLMBar, HHH-Alignment, MT-Bench Human Judge, and EVALBIASBENCH.LLMBar includes natural and adversarial subsets, while HHH-Alignment covers helpfulness, honesty, harmlessness, and related categories.Source: §5.2 · p. 6
• Reward models are evaluated on RewardBench, whose datasets are grouped into Chat, Chat Hard, Safety, and Reasoning criteria.The study also evaluates each model on the bias-specific EVALBIASBENCH test set.Source: §5.2 · p. 6

5.3 Baselines

• Baselines include proprietary GPT models, open-source generative judges, LLaMA-3-8B-Instruct, and reward models based on LLaMA, Mistral, and Yi.Original prompt templates are used for generative-model comparisons.Source: §5.3 · p. 6

6.1 Generative Model Results

• OFFSETBIAS improves the Base-data generative judge in every reported benchmark category, with especially large gains on LLMBar Neighbor, GPTInst, Manual, and HHH-Alignment Helpful and Honest subsets.These results are reported in Table 1; prior PandaLM, AutoJ, and PROMETHEUS-2 models score low on LLMBar.Source: §6.1 · p. 6, Table 1 · p. 7
• The OFFSETBIAS generative model achieves the highest average positional agreement rate, outperforming even proprietary models.Table 2 evaluates agreement when the two response positions are swapped; random guessing corresponds to 50% agreement.Source: §6.1 · p. 6, Table 2 · p. 7

6.2 Reward Model Results

• Reward-model results improve substantially on RewardBench Chat Hard and also improve Safety and Reasoning, but Chat performance decreases.The same mixed pattern appears in the EVALBIASBENCH results.Source: §6.2 · pp. 6–7, Table 3 · p. 7

6.3 EvalBiasBench Results

• OFFSETBIAS raises total average accuracy for both generative and reward models on EVALBIASBENCH, although reward-model scores decrease in categories such as Familiar Knowledge.Prior generative judges struggle with almost every bias type, while reward models perform relatively better before OFFSETBIAS is applied.Source: §6.3 · p. 7, Table 4 · p. 8

6.4 Ablation Study

• Position swapping does not significantly improve positional agreement, but it increases accuracy across all benchmark sets and contributes to generative-judge robustness.The ablation results are summarized with micro-average accuracy and agreement rate in Table 5.Source: §6.4 · p. 7, Table 5 · p. 8
• Removing ORM causes a significant LLMBar drop, while removing ERM lowers accuracy by 3 to 5 percentage points across all benchmark sets.ORM is mainly associated with adversarial cases, whereas ERM improves judgment in a more diverse setting.Source: §6.4 · p. 7

Coverage of Bias Types by OFFSETBIAS

• OFFSETBIAS construction is not designed to cover all six bias types, because it focuses on errors while allowing GPT-4 to naturally generate some bias-related qualities.The authors note that reduced Familiar Knowledge performance suggests incomplete coverage.Source: §7 · p. 8
• Improvements across most bias types may reflect training models to reject errors or to resist qualities naturally associated with GPT-4, rather than direct coverage of every bias.This interpretation is presented as an alternative explanation for the observed mitigation.Source: §7 · p. 8

Preference for GPT-4-Generated Responses

• Preference datasets dominated by GPT-4-generated chosen responses can encourage length and style preferences that harm performance on length-bias cases.OFFSETBIAS instead uses GPT-4 mostly for rejected responses and varied models for chosen responses.Source: §7 · p. 8

8 Conclusion

• The authors identify six judge-model bias types and propose EVALBIASBENCH to evaluate robustness against them.They also present de-biasing dataset construction methods and the OFFSETBIAS preference dataset.Source: §8 · pp. 8–9
• Mitigating identified biases with OFFSETBIAS significantly improves the overall performance of judge models.Source: §8 · p. 9

Universality of Identified Biases

• The six-bias taxonomy is empirical rather than exhaustive, and identified biases are inherently model-dependent.Some advanced proprietary models are more robust to these biases.Source: Limitations · p. 9

Applicability of EVALBIASBENCH

• EVALBIASBENCH has limited diversity and should support rather than replace other judge-model benchmarks.Its hand-crafted deceptive responses may not reflect most real-life evaluation outputs, confining applicability to hypothetical scenarios.Source: Limitations · p. 9

Usability of OFFSETBIAS Dataset

• OFFSETBIAS may introduce opposite-direction biases, making mitigation potentially unstable.The dataset is intended as a complement to other preference data rather than standalone training data.Source: Limitations · p. 9
• The study addresses pairwise preference evaluation, leaving biases in single-grading scenarios outside its scope.Source: Limitations · p. 9

Ethical Considerations

• Datasets generated with GPT-4-1106-preview or Claude-3-Opus may include potentially unhelpful or harmful instructions or responses.The authors based seed instructions on validated, published data to avoid ethical issues.Source: Ethical Considerations · p. 9

A.1 Generating Similar Instructions

• The appendix generates similar but distinct instructions, then asks whether instruction pairs request the same thing using a YES/NO judgment.Generated instructions should preserve superficial answer similarity while requiring different correct responses.Source: §A.1.1 · p. 12, §A.1.2 · p. 12

A.2 Generating Incorrect Responses

• Adversarial response prompts create seemingly correct but unhelpful outputs by inserting wrong facts, incompleteness, irrelevant material, omitted requirements, or instruction deviations.Examples include falsely stating that kookaburras are native to Korea, providing only a bird name, discussing Australian tourism, omitting bird-specific details, or defining birds generally instead of describing a type.Source: §A.2.1 · p. 12, §A.2.2 · p. 13, §A.2.3 · p. 13

A.2.6 Judging If the Response is Wrong

• A response judge answers YES or NO on whether an output correctly follows its instruction, with WRONG information requiring a NO judgment.The judging prompt provides the instruction and asks whether the response is correct.Source: §A.2.6 · p. 15

B Training Prompt Format

• Training uses four pairwise-preference prompt formats: General Single-turn, Safety Single-turn, General Multi-turn, and Safety Multi-turn.General prompts prioritize instruction or conversation execution, helpfulness, accuracy, and objectivity; safety prompts prioritize harmless execution and prohibit harmful assistance.Source: B Training Prompt Format · p. 15, §B.1 · p. 16, §B.2 · p. 16

B.1 General Single-turn / B.2 Safety Single-turn

• General single-turn prompts select the better of two outputs for an instruction, while safety single-turn prompts additionally prioritize safe and harmless execution.Both formats require outputting only Output (a) or Output (b) and treating response order as equally likely to be preferred.Source: §B.1 · p. 16, §B.2 · p. 16

B.3 General Multi-turn / B.4 Safety Multi-turn

• General and safety multi-turn prompts select the better assistant response for a conversation, with safety variants prioritizing harmlessness and forbidding harmful assistance.Both require a binary Output (a) or Output (b) answer without explanation or ties.Source: §B.3 · p. 17, §B.4 · p. 17

C.1 Generative Models

• OFFSETBIAS training adds 8.5k pairwise comparisons to base preference data, with response-position swapping augmenting pairwise examples.Generative judges use LLaMA-3-8B-Instruct; the base-data model combines direct scoring and pairwise data, while OFFSETBIAS extends it with the additional comparisons.Source: §C.1 · p. 18

C.2 Reward Models

• The reward-model experiment adds OFFSETBIAS to 71.5k supplementary examples, producing an 80k training set before merging the intermediate model with the original reward model.The intermediate model is trained from LLaMA-3-8B-Instruct and merged using SLERP with a ratio of 0.5.Source: §C.2 · p. 18

E OFFSETBIAS Examples

• OFFSETBIAS examples illustrate off-topic and erroneous-response construction methods, including wrong-fact and irrelevant-part prompts.The examples are presented in Figures 6–8.Source: Figure 6 · p. 21, Figure 7 · p. 22, Figure 8 · p. 22

F Length Distribution

• Existing judge-model training datasets generally contain more instances where chosen responses are longer, whereas OFFSETBIAS intentionally includes more lengthy rejected responses.A negative length difference indicates that the rejected response is longer than the chosen response.Source: F Length Distribution · p. 23

G Rejected Bias Hypotheses

• The appendix reports error cases for suspected bias hypotheses that were rejected, including preferences for bullet formatting, instruction-token overlap, and friendly tone.Figures 9–11 provide examples of these seemingly preferred response properties.Source: G Rejected Bias Hypotheses · p. 24, Figure 9 · p. 24, Figure 10 · p. 24

• Table 6 reports accuracy results for generative judge and reward models across EvalBiasBench categories, alongside a random-guessing null model.Source: Table 6 · p. 25
• Values significantly different from random guessing under a binomial test with p < 0.05 are highlighted.Source: Table 6 · p. 25
• Red indicates significantly lower scores and a tendency toward bias, whereas green indicates significantly higher scores than random guessing.Source: Table 6 · p. 25