Self-Taught Evaluators

• Human preference data is costly and time-consuming to collect, especially for challenging tasks such as coding and mathematics.Source: §1 · p. 1
• Synthetic iterative self-training generates contrasting responses, filters correct LLM-as-a-Judge reasoning and judgments, and trains an improved evaluator without human preference annotations.Source: §1 · p. 1
• 88.7 RewardBench accuracy with majority vote improves the Llama-3-70B-Instruct seed from 75.4 and exceeds the 85.6 achieved using 10k HelpSteer2 human annotations.Source: §1 · p. 1

• LLM-as-a-Judge systems can provide scalable and explainable proxies for human evaluation on open-ended or complex instructions.Source: §2 · p. 2
• Off-the-shelf evaluators show high variance across many tasks, indicating a need for methods that improve evaluation reliability.Source: §2 · p. 2
• Synthetic data supports customizable training and evaluation when real-world examples or correct annotations are difficult to obtain.Source: §2 · p. 2
• Prior synthetic-data applications include factuality, safety, coding, instruction following, and preference-pair construction for reward models.Source: §2 · p. 2

3 Method

• The judge receives an instruction, two responses, and an evaluation rubric, then produces a reasoning chain followed by an A-or-B preference.Source: §3 · p. 2
• The Self-Taught Evaluator iteratively annotates synthetic preferences with the current model and fine-tunes the next evaluator without human labels.Source: §3 · p. 3, §3.5 · p. 4

3.5 Model Fine-tuning (Iterative Training)

• As the evaluator improves, the number of accepted judgments is expected to increase, creating an automatic curriculum.Source: §3 · p. 3

3.1 Initialization

• The pipeline starts from human-written instructions and an initial seed LLM, with instructions potentially spanning knowledge, reasoning, coding, safety, and mathematics.Source: §3.1 · p. 3

3.2 Instruction Selection

• Instructions are categorized with an LLM so the training pool can emphasize useful or challenging categories while discarding others.Source: §3.2 · p. 3

3.3 Response Pair Construction

• Each synthetic pair combines a baseline response to the original instruction with a high-quality response to a semantically altered instruction, treated as inferior for the original.Source: §3.3 · pp. 3–4

3.4 Judgment Annotation

• For each example, multiple judgments are sampled and rejection sampling retains only judgments whose verdict matches the synthetic preference; examples without a correct judgment are discarded.Source: §3.4 · p. 4

3.5 Model Fine-tuning (Iterative Training)

• The accepted reasoning trace and verdicts form training examples for the next iteration, while each iteration reinitializes fine-tuning from the seed model.Source: §3.4 · p. 4, §3.5 · p. 4

4.1 Experimental Setup

• Training initializes from Llama3-70B-Instruct and repeatedly generates synthetic preferences and judgments before fine-tuning the evaluator.Source: §4.1 · p. 4
• 20,582 reasoning-category instructions are selected from WildChat because they are expected to provide challenging inputs.Source: §4.1 · p. 4
• N = 15 judgments are sampled per example, one positive judgment is retained, and labels are balanced between “A is better” and “B is better.”Source: §4.1 · p. 5
• Majority-vote inference selects the most common verdict among multiple sampled generations during final evaluation.Source: §4.1 · p. 5

4.2 Other Data Sources

• Synthetic judgments are also constructed from HelpSteer2, GSM8K, coding instructions, and hh_rlhf to test additional data sources.Source: §4.2 · p. 5

4.3 Evaluation

• Evaluation covers RewardBench, MT-Bench agreement with human judgments, and the HelpSteer2 validation split.Source: §4.3 · p. 5

5.1 RewardBench

• 88.3 on RewardBench at iteration 5 versus 75.4 for the seed, while outperforming the 85.6 obtained with labeled HelpSteer2 data.With 32-sample majority voting, the third iteration reaches 88.7.Source: §5.1 · p. 5, Table 1 · p. 6
• RewardBench gains are strongest in Chat Hard, Safety, and Reasoning, but performance is worse on the easier Chat category than for the seed model.The authors suggest the synthetic training data focused the model on harder examples.Source: §5.1 · p. 5
• Synthetic preference sources targeting coding, mathematical reasoning, and other skills all improve RewardBench categories related to their data distributions.The comparison uses supervised fine-tuning with synthetic preferences from different sources.Source: Table 4 · p. 6

5.2 MT-Bench

• On MT-Bench, the Self-Taught Evaluator outperforms the Llama3-70B-Instruct seed and is on par with or slightly better than GPT4-0125 on non-tie examples.Scores average both response orders because the models make binary judgments while MT-Bench includes ties.Source: §5.2 · pp. 5, 7, Table 2 · p. 6

5.3 HelpSteer2

• On HelpSteer2, synthetic iterative training improves both average accuracy and position-consistent accuracy over the Llama-3-70B-Instruct seed.Evaluation averages two response orders and three seeds, while position-consistent accuracy treats inconsistent orderings as incorrect.Source: Table 3 · p. 6, §5.3 · p. 7

6.1 Synthetic Data from Other Sources

• Synthetic preferences from coding, mathematical reasoning, and other skill-focused sources all improve evaluator performance, with gains generally concentrated in related RewardBench categories.Source: §6.1 · p. 7

6.2 Synthetic Bad Response Generation

• 83.8 versus 80.7 RewardBench overall score shows that constructing an inferior response through a similar instruction outperforms directly prompting for a bad response.Source: §6.2 · pp. 7–8

6.3 Comparison of Synthetic Data with Human Annotated Data

• Iterative training with HelpSteer2 labeled preferences improves RewardBench, but each iteration yields smaller gains and the final model does not outperform synthetic-preference training.Source: §6.3 · p. 8, Table 5 · p. 8

6.4 Iterative Training by Initializing from Labeled Data

• Initializing the pipeline from a model fine-tuned on HelpSteer2 performs well but does not clearly outperform iterative training using unlabeled data alone.Source: §6.4 · p. 8

6.5 Combining Synthetic and Human Labeled Preference Data

• Mixing synthetic preferences with HelpSteer2 data retains strong performance across data proportions and slightly improves overall accuracy.Source: §6.5 · p. 8, Table 6 · p. 8

6.6 Instruction complexity

• The curated instruction set emphasizes complex logical-reasoning and science tasks, while the full WildChat data contains more relationship and entertainment tasks and longer expected outputs.Most curated examples are under 500 input tokens, whereas the full dataset includes a cluster of very long instructions.Source: Figure 3 · p. 7, Figure 4 · p. 7, Figure 5 · p. 7

• Synthetic-preference training raises Llama3-70B-Instruct from 75.4 to 88.7 on RewardBench, establishing a reported state-of-the-art result for generative LLM-as-a-Judge methods.Source: §7 · p. 9

• The method uses relatively large 70B models, has not been tested on smaller models, and assumes the seed model can produce reasonable evaluations aligned with human or policy preferences.Source: §8 · p. 9
• Generative LLM-as-a-Judge models produce longer reasoning outputs and therefore have higher inference costs than reward models that output only a score.Source: §8 · p. 9
• The study evaluates pairwise response comparisons but leaves single-response scoring, such as ratings out of 5 or 10, for future work.Source: §8 · p. 9

• Prompt templates cover instruction selection, synthetic-preference judgment annotation, and ablation data generation with bad responses.Source: §A.1 · p. 11

A.2 More Training and Evaluation Details

• Training hyper-parameters and generation sampling parameters are provided in Tables 7 and 8.Table 7 covers finetuning, while Table 8 covers temperature and top-p settings during generation and evaluation.Source: §A.2 · p. 11, Table 7 · p. 12, Table 8 · p. 12

A.3 Position Order Evaluation on RewardBench

• Response-order sensitivity is evaluated because RewardBench randomly places winning and losing responses, which can change generative-model judgments.The winning response is denoted yw and the losing response yl.Source: §A.3 · p. 12
• 88.3 average accuracy is recommended for the fifth training iteration when RewardBench scores are averaged across both response orders.The evaluation compares shuffled order with cases where the winning or losing response always appears first.Source: §A.3 · p. 12, Table 9 · p. 13

A.4 Using Different Models for Training Data Generation

• Mixtral 22Bx8 Instruct-generated responses judged by Mixtral 22Bx8 Instruct produce the best RewardBench performance among tested training-data configurations.The resulting evaluator is finetuned and evaluated using Llama-3-70B-Instruct.Source: §A.4 · p. 13, Table 10 · p. 14
• The appendix also reports MT-Bench per-category results and iterative training from a model initially finetuned on labeled HelpSteer2 data.The MT-Bench table states that the synthetic-data evaluator performs on par with GPT-4 judgments.Source: Table 11 · p. 14, Table 12 · p. 14