PaperlayerView exact revision on arXiv

Source-linked AI summary

References Improve LLM Alignment in Non-Verifiable Domains

Kejian Shi, Yixin Liu, Peifeng Wang, Alexander R. Fabbri, Shafiq Joty, Arman Cohan

arXiv:2602.16802v1cs.CLcs.AIcs.LG

TL;DR

LLM alignment lacks reliable ground-truth verifiers, limiting the direct use of RL with verifiable rewards in this domain. The paper uses reference-guided LLM judges for self-improvement and reports gains over reference-free self-improvement, with performance comparable to trained reward models.

  • Problem

    The paper asks whether reference-guided LLM evaluators can act as soft verifiers for alignment, where reliable, verifiable rewards are difficult to design.

    Source: §1 · p. 2

  • Method

    The paper develops reference-guided LLM judges to supervise preference optimization in self-improvement, without requiring external human or AI feedback.

    Source: §1 · p. 2

  • Results

    +21.2 on AlpacaEval and +17.6 on Arena-Hard over DSV3-Distill are the largest reported gains; RefEval consistently outperforms RefFree and performs comparably to or better than ArmoRM.

    Source: §4.3 · p. 9

  • Takeaways & Limitations

    High-quality references can support effective semi-self-improvement for LLM alignment in non-verifiable domains, with performance comparable to trained reward models.

    Source: §5 · p. 10

    Most datasets lack human-written references, so the study uses GPT-4o-generated references and less capable LLMs as judges.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

While Reinforcement Learning with Verifiable Rewards (RLVR) has shown strong effectiveness in reasoning tasks, it cannot be directly applied to non-verifiable domains lacking ground-truth verifiers, such as LLM alignment. In this work, we investigate whether reference-guided LLM-evaluators can bridge this gap by serving as soft "verifiers". First, we design evaluation protocols that enhance LLM-based evaluators for LLM alignment using reference outputs. Through comprehensive experiments, we show that a reference-guided approach substantially improves the accuracy of less capable LLM-judges using references from frontier models; stronger LLM-judges can also be enhanced by high-quality (i.e., human-written) references. Building on these improved judges, we demonstrate the utility of high-quality references in alignment tuning, where LLMs guided with references are used as judges to self-improve. We show that reference-guided self-improvement yields clear gains over both direct SFT on reference outputs and self-improvement with reference-free judges, achieving performance comparable to training with ArmoRM, a strong finetuned reward model. Specifically, our method achieves 73.1% and 58.7% on AlpacaEval and Arena-Hard with Llama-3-8B-Instruct, and 70.0% and 74.1% with Qwen2.5-7B, corresponding to average absolute gains of +20.2 / +17.1 points over SFT distillation and +5.3 / +3.6 points over reference-free self-improvement on AlpacaEval / Arena-Hard. These results highlight the potential of using reference-guided LLM-evaluators to enable effective LLM post-training in non-verifiable domains.

1 INTRODUCTION

The paper investigates reference-guided LLM judges as soft verifiers for alignment, using them to evaluate outputs and supervise self-improvement without external human or AI feedback.

1 INTRODUCTION

  • RLVR’s verifiable-reward approach does not directly extend to alignment, where reliable rewards are hard to design and RLHF/RLAIF judges typically evaluate outputs without references.The authors motivate references as a way to extend reference-grounded supervision to non-verifiable tasks, especially when preference labels are costly or unavailable.Source: §1 · p. 2
  • Reference-guided prompting improved judges by 6.8% absolute over reference-free evaluation across 11 judges using GPT-4o outputs; high-quality human references also enhanced GPT-4o.The evaluation covered five benchmarks, and naive reference inclusion without explicit guidance yielded only modest improvements.Source: §1 · pp. 2–3
  • Training first applies SFT to high-quality DeepSeek-V3 references, then uses DPO with the model’s own reference-guided judge to provide supervision.The experiments fine-tune Llama-3-8B-Instruct and Qwen2.5-7B on UltraFeedback instructions; the reference-guided self-improvement stage further improves on SFT and needs no external human or AI feedback.Source: §1 · pp. 2–3
  • Llama-3 achieved 73.1 on AlpacaEval and 58.7 on Arena-Hard, while Qwen2.5 achieved 70.0 and 74.1, respectively.These models improved over reference-output SFT by up to 19.2 points on AlpacaEval and 16.5 on Arena-Hard, and matched or exceeded training with finetuned reward models. SFT distillation on references also outperformed DPO with ArmoRM-Llama3-8B.Source: §1 · p. 3

2 RELATED WORK

Prior work uses LLMs as judges and has begun incorporating references into evaluation, but reference-guided judging for alignment training remains less explored; this work extends the setting and studies it at larger scale.

LLM-as-a-Judge

  • LLM-as-a-Judge supports scalable instruction-following evaluation and DPO annotation, but faces positional and verbosity biases addressed through prompting, answer swapping, and robust protocols.This work instead uses external references to address knowledge gaps, complementing Trivedi et al. (2024), which uses internal self-rationalization to address reasoning gaps.Source: §2 · p. 3

The Role of References in LLM Evaluation

  • LLMBar (Zeng et al., 2024) and HREF (Lyu et al., 2024) explore references in LLM judging, but prior studies cover few models and limited dataset scale.Traditional NLG metrics such as BLEU and ROUGE use reference outputs, though single references are often insufficient for alignment evaluation.Source: §2 · p. 3

Reference-Guided Evaluation and Training

  • Unlike RevisEval (Zhang et al., 2025), which modifies references dynamically for static evaluation, this work extends reference guidance to model training and self-improvement across 5 datasets and 13 LLMs.Prior RL alignment studies use reference-based metrics such as BERTScore (Zhang et al., 2020) or BLEU; §4.3 reports the LLM-judge approach is more effective for alignment fine-tuning.Source: §2 · p. 3

3 DEVELOPING REFERENCE-GUIDED LLM-JUDGES

This section develops reference-guided LLM-judge prompts and evaluates them against established baselines. RefEval achieves the strongest average accuracy, while references also improve judge agreement and benefit less capable judges especially.

3.1 PRELIMINARY

  • The study focuses on pairwise comparison because it matches human-labeled alignment datasets and applies to preference optimization, measuring accuracy against human annotators’ preferred outputs.Pointwise scoring experiments are reported in Appendix B and also show gains from reference-guided evaluation.Source: §3.1 · p. 4

3.2 REFERENCE-GUIDED PROMPTING FOR LLM-JUDGES

  • RefEval explicitly asks judges to compare candidate outputs with the reference’s quality and content while still addressing the instruction, unlike prompts that merely include a reference with little usage guidance.The authors also introduce Ref-Free, which assesses instruction-following alongside factuality and verbosity, and RefMatch, which prioritizes semantic and stylistic similarity to the reference.Source: §3.2 · p. 4

3.3 EVALUATION SETUP

  • Evaluation uses human-annotated pairwise preferences across five datasets, averaging accuracy over swapped candidate orders; GPT-4o supplies references for 11 less-capable judges.The datasets cover instruction following, multi-turn conversation, controllable summarization, and human-written reference responses; the study also reports pointwise experiments in Appendix B.Source: §3.1 · p. 4, §3.3 · p. 5

3.4 OVERALL RESULT ANALYSIS

  • RefEval achieves the highest average accuracy at 79.1%, ahead of LLMBar-Base (72.3%), CoT (71.2%), HREF-Ref (74.8%), and LLMBar-Ref (74.0%); RefMatch ranks second at 77.7%.The average is across 11 open-source judges and five datasets; the results support grounding judgments with a strong reference.Source: Table 1 · p. 6, §3.4 · p. 6
  • RefEval improves average inter-judge agreement over the reference-free baseline (81.4% vs 76.6%), while its gains are more pronounced for weaker judges, including an approximately 17.4% absolute improvement over LLMBar-Base with Llama-3-8b.Qwen-2.5-72b also improves, reaching 84.6% with RefEval versus 79.4% with LLMBar-Base; the advantage generalizes to references from different frontier models.Source: §3.4 · pp. 6–7, Table 2 · p. 7
  • Human-edited oracle references on LLMBar-Adversarial produce consistent gains across all evaluated frontier judges.The case study is reported in Appendix A.8.Source: §3.4 · p. 7

4 REFERENCE-GUIDED SELF-IMPROVEMENT

The section evaluates reference-guided self-improvement, using high-quality outputs to guide an LLM judge that supplies preferences for the model’s own training. It compares this approach with reference-free self-improvement and other alignment-training baselines.

4.1 TRAINING PROCESS

  • Training first applies direct SFT to reference outputs, then on-policy DPO using preferences labeled by the reference-guided judge, without requiring separately preference-annotated data to train a reward model.Candidate pairs are sampled from the model being finetuned at the start of training and judged to select preferred and dispreferred outputs.Source: §4.1 · p. 7, §4 · p. 7

4.2 EXPERIMENTAL SETTINGS

  • Experiments use Llama-3-8B-Instruct and Qwen2.5-7B-SFT, 60K UltraFeedback instructions with DeepSeek-V3 references, and AlpacaEval and Arena-Hard evaluation.Benchmark outputs are compared with GPT-4 outputs using a strong LLM judge.Source: §4.2 · p. 8

4.3 RESULT ANALYSIS

  • RefEval consistently outperformed RefFree and ROUGE/BERTScore, matched or exceeded ArmoRM, and gained up to +21.2 on AlpacaEval and +17.6 on Arena-Hard over DSV3-Distill.The trained models also showed a clear advantage over the strong baselines compared in Table 5.Source: Table 4 · p. 9, Table 3 · p. 9, Table 5 · p. 9
  • RefFree improved over DSV3-Distill by +13.6 on AlpacaEval and +11.6 on Arena-Hard for Llama-3-8B-Instruct, and +16.3 and +15.3 for Qwen2.5-7B-SFT.Separately, DSV3-Distill generally outperformed ArmoRM-Base, showing that SFT on high-quality references was a strong starting point.Source: §4.3 · p. 9
  • RefEval improved Coding&Math performance for both models; its Creative Tasks benefit was less significant for Qwen2.5-7B-SFT but considerable for Llama-3-8B-Instruct.The authors suggest open-ended reference use is more challenging and may require more extensive post-training.Source: Figure 3 · p. 10, §4.3 · p. 10
  • With GPT-4o-mini references, RefEval still beat RefFree by +1.8 on AlpacaEval and +16.6 on Arena-Hard, though higher-quality references yielded better results.This indicates that reference-guided supervision provided benefits even when the reference generator was weaker than DeepSeek-V3.Source: Table 6 · p. 10, §4.3 · p. 10

5 CONCLUSION · A FULL EXPERIMENTAL RESULTS FOR EVALUATION PROTOCOLS

High-quality references consistently improve LLM-judge performance across five datasets and support alignment tuning through reference-guided judges. The appendix reports evaluation accuracy across prompting protocols, with averages over five core datasets.

5 CONCLUSION

  • Reference-guided LLM-judges enable semi-self-improvement with high-quality references, achieving performance comparable to trained reward models.The study investigates reference outputs for effective LLM alignment tuning.Source: §5 · p. 10
  • Across five datasets, high-quality reference outputs consistently improve LLM-judge performance.The findings highlight references’ potential for LLMs in non-verifiable domains and reducing the methodology gap between RLHF/RLAIF and RLVR post-training.Source: §5 · p. 10
  • Future work should explore references in specialized domains requiring domain expertise and knowledge.The conclusion also proposes developing reward models that specifically use references in these scenarios.Source: §5 · p. 10

A FULL EXPERIMENTAL RESULTS FOR EVALUATION PROTOCOLS

A.1 FULL STATISTICAL ANALYSIS … A.4 OVERALL PERFORMANCE WITH OPEN-SOURCE LLM JUDGES

The appendix reports statistically significant RefEval gains over reference-free baselines for every tested judge, describes reference-anchored evaluation protocols and variants, and provides full performance tables for GPT-4o and open-source judges.

A.1 FULL STATISTICAL ANALYSIS

  • RefEval significantly outperforms reference-free baselines for every one of the 11 tested LLM judges, with Table 7 reporting 95% bootstrap confidence intervals and paired significance tests.The table provides the detailed statistical analysis underlying the main-text summary in Table 2.Source: §A.1 · p. 16, Table 7 · p. 16

A.2 PROMPT PROTOCOL DESIGN AND VARIANTS

  • RefEval uses the reference output as a benchmark for judging candidates’ instruction-following, factual correctness, and overall quality.RefMatch instead asks judges to select the candidate more semantically and stylistically similar to the reference, using specified matching rules and its instruction-following pattern.Source: §A.2 · p. 16
  • Unlike approaches that treat references as supplementary information, RefEval and RefMatch explicitly make the reference a central anchor for judgment.The paper cites HREF-Ref (Lyu et al., 2024) and reference-augmented LLMBar prompts (Zeng et al., 2024) as examples of approaches that do not explicitly direct judges to ground decisions in the reference.Source: §A.2 · p. 16
  • Prompt variants add structured evaluation rules to RefEval, chain-of-thought reasoning to RefMatch, or comparison against three references.Multi-Ref Avg asks judges to assess overall similarity to the set, while Multi-Ref MAX asks them to choose the best match to any single reference; the paper also developed a strong reference-free baseline.Source: §A.2 · p. 16

A.3 PERFORMANCE WITH FRONTIER MODEL JUDGE

  • Table 8 reports full evaluation accuracy across five datasets using GPT-4o as judge and GPT-4o-generated references.The supplied passages do not provide the table’s cell values.Source: Table 8 · p. 17, §A.3 · p. 18

A.4 OVERALL PERFORMANCE WITH OPEN-SOURCE LLM JUDGES

  • Table 9 provides average evaluation accuracy for prompting protocols judged by 11 open-source models, using GPT-4o-generated references.The supplied passages identify the table’s scope but do not provide its cell values.Source: Table 9 · p. 17, §A.4 · p. 18

A.5 PERFORMANCE BREAKDOWN BY OPEN-SOURCE MODEL SCALE · A.6 FULL RESULTS FOR MULTI-REFERENCE STRATEGIES · A.8 HUMAN REFERENCES AS ORACLES

RefEval improves evaluation by smaller open-source judges on challenging datasets, while human-edited references also improve accuracy for stronger judges. The multi-reference results cover voting and prompt variants alongside single-reference baselines.

A.5 PERFORMANCE BREAKDOWN BY OPEN-SOURCE MODEL SCALE

  • RefEval shows clear advantages on LLMBar-Adversarial and HREF for larger judges, while remaining competitive elsewhere.Figure 4 reports aggregate performance across five datasets for larger and smaller model groups.Source: §A.5 · p. 18, Figure 4 · p. 18
  • 72.7% vs. 56.8% on LLMBar-Adversarial and 80.8% vs. 64.5% on HREF: RefEval outperforms LLMBar-Base for smaller judges.The breakdown covers smaller open-source judges with references by GPT-4o.Source: §A.5 · p. 18, Figure 4 · p. 18, Table 11 · p. 19

A.6 FULL RESULTS FOR MULTI-REFERENCE STRATEGIES

  • Multi-reference results average across 11 open-source judges and four datasets, comparing Multi-Voting and Multi-Prompt variants with single-reference RefEval and RefMatch.The single-reference methods use GPT-4o references; methods without “Vote” are single-reference unless reference-free, while Multi-Prompt is an exception.Source: Table 12 · p. 19, §A.6 · p. 20

A.8 HUMAN REFERENCES AS ORACLES

  • Human-edited Oracle references were created from a subset of GPT-4o references and tested on adversarial LLMBar-Adversarial with four judges using RefEval and RefMatch.The judges were GPT-4o, GPT-4.1, Qwen-2.5-72B, and Llama-3.1-70B; standard machine references could be flawed.Source: §A.8 · pp. 20–21, Table 13 · p. 21
  • 86.8% to 88.4%: GPT-4o’s RefEval accuracy rises with an Oracle GPT-4o reference rather than its self-generated reference.Similar improvements are reported across the models and protocols tested, particularly for adversarial or complex instructions where machine references may be flawed.Source: §A.8 · p. 21, Table 13 · p. 21

B ADDITIONAL RESULTS ON POINTWISE SCORING · C ADDITIONAL ANALYSES AND ABLATIONS

Reference-guided pointwise scoring outperforms reference-free pointwise scoring for both open-source and frontier model groups. The evaluation adapts pairwise protocols by rating single outputs and inferring preferences from scores.

B ADDITIONAL RESULTS ON POINTWISE SCORING

  • The evaluation adapts reference-based RefEval and reference-free LLMBar-Base into pointwise protocols named RefEval-point and Base-point.These protocols ask the LLM judge to rate a single model output on a Likert scale from 1 to 5.Source: B ADDITIONAL RESULTS ON POINTWISE… · p. 21
  • For pointwise accuracy, the output with the higher score is designated the winner, and that inferred preference is compared with the ground-truth human label.This uses the same accuracy metric as the main pairwise experiments.Source: B ADDITIONAL RESULTS ON POINTWISE… · p. 21
  • RefEval-point consistently outperforms Base-point for the average of 11 open-source models and the stronger GPT4o and GPT-4.1 group.Table 14 reports pointwise evaluation accuracy, with inferred preferences compared against human labels; the supplied passage gives no specific accuracy values.Source: B ADDITIONAL RESULTS ON POINTWISE… · p. 21, Table 14 · p. 21

C.1 IMPACT OF REFERENCE QUALITY · C.2 INTER-JUDGE AGREEMENT · C.3 MARGINAL GAINS OF MULTI-REFERENCE VOTING

Reference-guided evaluation and self-improvement remain effective with weaker references, while references increase judge agreement and voting accuracy with diminishing marginal gains. These findings support using references to improve alignment evaluation without requiring multiple references for every judgment.

C.1 IMPACT OF REFERENCE QUALITY

  • References from 7B models improve judge accuracy over the reference-free baseline, although stronger references yield the highest accuracy.The evaluation tests Llama-3.1-70B and Qwen-2.5-7B judges with references from Mistral-Nemo-12B and Tulu-2-7B.Source: §C.1 · pp. 21–22, Table 16 · p. 22
  • With GPT-4o-mini references replacing DeepSeek-V3 references, RefEval self-improvement outperforms its reference-free equivalent.This shows the training pipeline remains effective even with weaker references.Source: §C.1 · p. 22, Table 17 · p. 23

C.2 INTER-JUDGE AGREEMENT

  • RefEval significantly increases average pairwise agreement among 11 open-source judges across all datasets compared with Ref-Free.The authors suggest references reduce variance in subjective judgment.Source: §C.2 · p. 23, Table 18 · p. 23
  • The authors attribute reduced variance in subjective judgment to references improving consistency across judges.This interpretation follows the observed increase in agreement across all datasets.Source: §C.2 · p. 23

C.3 MARGINAL GAINS OF MULTI-REFERENCE VOTING

  • Adding references to voting ensembles consistently improves RefEval accuracy across 11 judges.The ensembles draw references from a pool of diverse frontier models.Source: §C.3 · p. 23, Table 19 · p. 23
  • The marginal accuracy gain diminishes as the number of references in the voting ensemble increases.This finding motivates the paper's focus on single-reference efficiency in its main experiments.Source: §C.3 · p. 23, Table 19 · p. 23

D DATASET DETAILS

The datasets evaluate LLM-as-a-Judge protocols using paired model outputs and human preference labels, with accuracy calculated under benchmark-specific tie handling and averaged across swapped output orders.

D DATASET DETAILS

  • Each instance pairs an instruction with two candidate outputs and a human preference label; accuracy excludes ties or follows benchmark win-rate rules and averages two passes with candidate order swapped.The two passes swap the order of candidate outputs.Source: D DATASET DETAILS · p. 23

D.1 CORE EVALUATION DATASETS · D.2 CREATION OF HUMAN ORACLE REFERENCES

The section describes four human-annotated evaluation resources spanning instruction following, dialogue, summarization, and reference-guided tasks, then explains how experts created blind Human Oracle references for adversarial evaluation.

D.1 CORE EVALUATION DATASETS

  • LLMBar-Natural (100 instances) tests objective quality differences, while LLMBar-Adversarial (319) uses superficially appealing but instruction-deviating outputs; agreement is 90% and 95%, respectively (Zeng et al., 2024).The study uses the datasets’ pairwise human preference labels.Source: §D.1 · p. 24
  • MTBench covers 80 prompts across eight categories; the study uses 200 expert-annotated pairwise comparisons with 81% inter-annotator agreement (Zheng et al., 2024).The comparisons exclude ties and come from publicly released data.Source: §D.1 · p. 24
  • Instrusum contributes 411 instruction-controllable summarization instances with 100% annotator agreement, and its instructions are longer and more complex on average than those in other datasets (Liu et al., 2024b).Preferences reflect instruction adherence and overall summary quality.Source: §D.1 · p. 24
  • The selected HREF human-agreement subset covers classification, closed QA, extraction, generation, and rewriting; summarization, brainstorming, and open QA are excluded (Lyu et al., 2024).HREF compares model outputs against human-written instructions and reference responses using human preference labels; exclusions avoid overlap or more subjective, potentially noisy categories.Source: §D.1 · p. 24

D.2 CREATION OF HUMAN ORACLE REFERENCES

  • The authors created Human Oracle references for a subset of LLMBar-Adversarial to test exceptionally high-quality references with strong judges on challenging tasks; they selected 23 cases where GPT-4o had erred (Zeng et al., 2024).The errors were against the dataset’s original human annotations, with GPT-4o using its own generated reference.Source: §D.2 · pp. 24–25
  • An NLP expert revised the 23 references using only each instruction, blind to candidate outputs and original preference labels, to optimize correctness, completeness, and instruction adherence.The expert was a co-author; the blind process made reference creation independent of the outputs later evaluated.Source: §D.2 · p. 25

E TRAINING DETAILS … H PROMPT TEMPLATES

The appendix specifies training settings, identifies the model registry and benchmark instruction categories, and documents prompt templates for the proposed methods and selected baselines.

E TRAINING DETAILS

  • Training used 2 epochs, batch size 128, a 5e-6 maximum learning rate, linear scheduling, 3% warmup, and 2048-token filtering, yielding 883K instances.Instances longer than 2048 tokens were filtered out.Source: E TRAINING DETAILS · p. 25

F MODEL REGISTRY

G DETAILS OF INSTRUCTION CLASSIFICATION FOR ALPACAEVAL AND ARENA-HARD

  • GPT-4o classified instructions into Coding&Math, Creative Tasks, Information Seeking, and Reasoning&Planning; AlpacaEval has more open-ended instructions, while Arena-Hard emphasizes coding and math reasoning.The appendix provides instruction-type distribution figures for both benchmarks.Source: G DETAILS OF INSTRUCTION… · p. 25, Figure 6 · p. 25, Figure 7 · p. 26

H PROMPT TEMPLATES

Related papers

Loading 2602.16802v1…