PaperlayerView exact revision on arXiv

Source-linked AI summary

References Improve LLM Alignment in Non-Verifiable Domains

Kejian Shi, Yixin Liu, Peifeng Wang, Alexander R. Fabbri, Shafiq Joty, Arman Cohan

arXiv:2602.16802v1cs.CLcs.AIcs.LG

TL;DR

RLVR does not directly address alignment because reliable verifiers are unavailable in this non-verifiable domain. The paper develops reference-guided LLM judges and uses them for self-improvement, finding gains over SFT and reference-free judges with performance comparable to finetuned reward models.

  • Problem

    Alignment lacks the reliable ground-truth verifiers required for directly applying RLVR.

  • Method

    The paper designs reference-guided LLM-judge protocols and uses the resulting judges to supervise self-improvement alignment tuning without external feedback.

  • Results

    Reference-guided self-improvement improves over SFT distillation and reference-free self-improvement, achieving performance comparable to ArmoRM-based training.

  • Takeaways & Limitations

    High-quality references can support effective LLM post-training in non-verifiable domains while narrowing the methodological gap between RLHF/RLAIF and RLVR.

  • Takeaways & Limitations

    Most evaluation datasets lack human-written references, so the evaluation uses GPT-4o as an oracle to generate them.

Abstract

from arXiv · show

While Reinforcement Learning with Verifiable Rewards (RLVR) has shown strong effectiveness in reasoning tasks, it cannot be directly applied to non-verifiable domains lacking ground-truth verifiers, such as LLM alignment. In this work, we investigate whether reference-guided LLM-evaluators can bridge this gap by serving as soft "verifiers". First, we design evaluation protocols that enhance LLM-based evaluators for LLM alignment using reference outputs. Through comprehensive experiments, we show that a reference-guided approach substantially improves the accuracy of less capable LLM-judges using references from frontier models; stronger LLM-judges can also be enhanced by high-quality (i.e., human-written) references. Building on these improved judges, we demonstrate the utility of high-quality references in alignment tuning, where LLMs guided with references are used as judges to self-improve. We show that reference-guided self-improvement yields clear gains over both direct SFT on reference outputs and self-improvement with reference-free judges, achieving performance comparable to training with ArmoRM, a strong finetuned reward model. Specifically, our method achieves 73.1% and 58.7% on AlpacaEval and Arena-Hard with Llama-3-8B-Instruct, and 70.0% and 74.1% with Qwen2.5-7B, corresponding to average absolute gains of +20.2 / +17.1 points over SFT distillation and +5.3 / +3.6 points over reference-free self-improvement on AlpacaEval / Arena-Hard. These results highlight the potential of using reference-guided LLM-evaluators to enable effective LLM post-training in non-verifiable domains.

1 INTRODUCTION

The paper asks whether reference-guided LLM evaluators can act as soft verifiers for alignment, a non-verifiable domain where RLVR cannot be directly applied. It develops reference-guided judges and uses them for self-improvement alignment tuning.

  • RLVR cannot be directly applied to alignment tuning because reliable, verifiable rewards are difficult to design.
  • Reference-guided LLM evaluators are proposed as soft verifiers that provide supervision for alignment without external human or AI feedback.
  • The paper develops targeted prompting strategies and systematically evaluates reference-guided judges across multiple judges and benchmarks.
  • Reference-guided judges are used in a two-stage self-improvement process combining SFT on high-quality references with subsequent preference optimization.
  • Reference-guided self-improvement outperforms SFT distillation and reference-free self-improvement, while achieving performance comparable to a strong finetuned reward model.
  • The method achieves 73.1 and 58.7 on AlpacaEval and Arena-Hard with Llama-3, and 70.0 and 74.1 with Qwen2.5, respectively.

2 RELATED WORK

Prior work studies LLM-as-a-Judge evaluation, reference-based prompting, self-improving models, and generative reward models. This paper extends reference-guided evaluation toward alignment training and conducts a broader investigation.

  • LLM-as-a-Judge: LLM-as-a-Judge supports scalable instruction-following evaluation and preference-optimization data annotation, but exhibits positional and verbosity biases.
  • LLM-as-a-Judge: Existing mitigation strategies include Chain-of-Thought prompting, answer swapping, and more robust evaluation protocols.
  • The Role of References in LLM Evaluation: Traditional NLG metrics use reference outputs, while reference roles in alignment-oriented LLM judging have been less explored because single ground-truth references are often insufficient.
  • The Role of References in LLM Evaluation: Prior reference-guided methods mainly target static evaluation, whereas this work applies reference-guided supervision to model training and self-improvement.
  • Self-Improving LMs and Generative RMs: Related research also examines self-improving language models and generative reward models that use LLMs as training-time evaluators.

3 DEVELOPING REFERENCE-GUIDED LLM-JUDGES

This section develops reference-guided LLM-judge methods for pairwise alignment evaluation, focusing on explicit reference utilization and comprehensive comparison across models and datasets. RefEval achieves the strongest reported average accuracy and improves agreement between judges.

  • Evaluation setup: Reference-guided evaluation is tested in pairwise comparison across five human-annotated datasets using 11 open-source LLM judges.The evaluation averages accuracy across swapped candidate orders and uses human annotations as ground truth.
  • Reference-guided prompting: RefEval instructs judges to select the candidate whose quality and content more closely align with the reference while still addressing the original instruction.The method adds explicit guidance because earlier reference-based prompts gave limited direction on how references should be used.
  • Overall results: 79.1% average evaluation accuracy makes RefEval the top method, surpassing reference-free and other reference-based prompting baselines.Ref-Free baselines LLMBar-Base and CoT achieve 72.3% and 71.2%, while HREF-Ref and LLMBar-Ref achieve 74.8% and 74.0%; RefMatch reaches 77.7%.
  • Overall results: 81.4% versus 76.6% average pairwise agreement shows that RefEval makes different judges align more closely than the reference-free baseline.The authors interpret this shared grounding as reducing variance in decision-making.
  • Overall results: Approximately 17.4% absolute improvement over LLMBar-Base is achieved by RefEval with Llama-3-8b, while qwen-2.5-72b reaches 84.6% versus 79.4%.Smaller, less capable models benefit more strongly, and references can bring small-model evaluation quality closer to that of larger models.
  • Overall results: Human-edited Oracle references produce consistent gains across all evaluated frontier judges on LLMBar-Adversarial.This case study extends the reference-guided evaluation analysis beyond frontier-model-generated references.

4 REFERENCE-GUIDED SELF-IMPROVEMENT

The paper uses reference-guided LLM judges to construct on-policy preference data for self-improvement, after distilling high-quality reference outputs. This pipeline consistently outperforms reference-free self-improvement and can approach finetuned reward-model performance.

  • Training Process: Reference-guided judges supervise DPO self-improvement without requiring external human or AI feedback.The judge evaluates on-policy candidate outputs, and its preferences supervise the model’s own training process.
  • Results: +13.6 on AlpacaEval and +11.6 on Arena-Hard are the gains of reference-free self-improvement over DSV3-Distill for Llama-3-8B-Instruct.For Qwen2.5-7B-SFT, the corresponding gains are +16.3 on AlpacaEval and +15.3 on Arena-Hard.
  • Results: +21.2 on AlpacaEval and +17.6 on Arena-Hard are the largest gains of RefEval over DSV3-Distill.RefEval consistently outperforms RefFree on both models and benchmarks, and also compares favorably with ROUGE, BERTScore, and ArmoRM.
  • Results: Higher-quality references yield better results, but reference-guided supervision still outperforms reference-free self-improvement when references come from GPT-4o-mini.This indicates a structural benefit from the supervision mechanism beyond dependence on frontier-model references.
  • Results: Reference-guided supervision substantially improves Coding&Math performance, while its Creative Tasks benefit is smaller for Qwen2.5-7B-SFT than for Llama-3-8B-Instruct.The authors suggest that leveraging references in open-ended tasks is more challenging and may require more extensive post-training.

5 CONCLUSION

The study finds that high-quality references improve LLM-judge performance and enable effective semi-self-improvement for alignment tuning in non-verifiable domains. It reports performance comparable to trained reward models and identifies specialized-domain references and reference-utilizing reward models as future directions.

  • High-quality references consistently improve LLM-judge performance across five datasets.
  • Reference-guided LLM-judges enable semi-self-improvement for alignment tuning using high-quality references.
  • Reference-guided alignment tuning achieves performance comparable to trained reward models.
  • Future work should test references in specialized domains and develop reward models that use references.

A.1 FULL STATISTICAL ANALYSIS

The appendix provides detailed statistical and protocol analyses of reference-guided evaluation. RefEval is presented as a reference-anchored protocol with statistically significant improvements over reference-free baselines across tested models.

  • Statistical analysis: RefEval significantly improves over reference-free baselines for every tested model.Table 7 reports 95% bootstrap confidence intervals and paired significance tests across 11 open-source models.
  • Evaluation protocols: Existing reference-guided approaches often treat references as supplementary information rather than explicitly grounding the decision in them.
  • Evaluation protocols: RefEval and RefMatch make the reference a primary anchor for judging candidate outputs.RefEval compares candidates with a reference for instruction-following, factual correctness, and overall quality; RefMatch evaluates semantic and stylistic similarity.
  • Experimental coverage: The appendix evaluates reference-guided and reference-free protocols across GPT-4o, 11 open-source judges, five datasets, and multiple protocol variants.It includes RefEval-Rules, RefMatch-Rules-CoT, Multi-Ref Avg, and Multi-Ref MAX.

A.5 PERFORMANCE BREAKDOWN BY OPEN-SOURCE MODEL SCALE

RefEval improves evaluation accuracy across model-scale groups and datasets, with especially large gains for smaller models on challenging benchmarks. Human-edited references also improve stronger judges on adversarial evaluation.

  • Model groups: The scale breakdown covers larger models above 9B parameters and smaller models at or below 9B parameters.
  • Aggregate performance: RefEval demonstrates consistent improvements across most datasets for both larger and smaller model groups.
  • Larger Models: RefEval shows clear advantages on LLMBar-Adversarial and HREF for larger models while remaining competitive elsewhere.
  • Smaller Models: 72.7% versus 56.8% on LLMBar-Adversarial and 80.8% versus 64.5% on HREF are reported for smaller models using RefEval versus LLMBar-Base.
  • Reference quality: Human-edited references increase evaluation accuracy for stronger judges on LLMBar-Adversarial.For GPT-4o with RefEval, accuracy rises from 86.8% using its self-generated reference to 88.4% using an Oracle reference.

B ADDITIONAL RESULTS ON POINTWISE SCORING

Additional experiments test pointwise reference-guided scoring as an alternative evaluation format. RefEval-point consistently outperforms the reference-free Base-point baseline across both open-source and stronger frontier judges.

  • Pointwise scoring: RefEval-point consistently outperforms Base-point for average open-source judges and stronger frontier models.Pointwise scores from 1 to 5 are converted into inferred pairwise preferences and compared with human labels.

C.1 IMPACT OF REFERENCE QUALITY

Reference-guided evaluation and self-improvement remain effective with weaker references, while stronger references generally yield higher accuracy. More references improve performance but with diminishing marginal gains, and reference guidance increases judge agreement.

  • Evaluation Robustness: Even references from 7B models improve evaluation accuracy over the reference-free baseline, although stronger references achieve the highest accuracy.The comparison uses Mistral-Nemo-12B and Tulu-2-7B references with Llama-3.1-70B and Qwen-2.5-7B judges.
  • Ablation Overview: Tables 16–19 report ablations of reference source quality, training robustness, inter-judge agreement, and reference-count effects.The tables summarize the experiments described in this section.
  • Training Robustness: Weaker GPT-4o-mini references still enable effective RefEval self-improvement and outperform the reference-free equivalent.This tests the training pipeline after replacing DeepSeek-V3 references.
  • Judge Agreement: Reference-guided RefEval significantly increases average pairwise agreement among 11 open-source judges across all datasets compared with Ref-Free.The result suggests that references reduce variance in subjective judgment.
  • Reference Count: Adding references consistently improves ensemble accuracy, but marginal gains diminish as the reference count increases.This motivates focusing on single-reference efficiency in the main experiments.

D DATASET DETAILS

The evaluation datasets contain instructions, two candidate outputs, and human preference labels, with accuracy computed under benchmark-specific tie handling.

  • Dataset Structure: Each dataset instance contains an input instruction, two candidate model outputs, and a human preference label.The label identifies the superior output or a tie.
  • Evaluation Metric: Primary evaluation accuracy typically excludes ties or handles them according to the original benchmark protocol for pairwise win-rate calculations.
  • Purpose: The datasets are used to evaluate LLM-as-a-Judge protocols.

D.1 CORE EVALUATION DATASETS

The core evaluation suite spans instruction-following, multi-turn conversation, controllable summarization, and human-response-guided tasks, with selected exclusions to preserve diversity and reduce subjectivity.

  • Instruction Following: LLMBar-Natural and LLMBar-Adversarial meta-evaluate instruction following using objective quality differences and superficially appealing adversarial outputs.LLMBar-Natural has 100 instances, while LLMBar-Adversarial has 319.
  • Multi-turn Conversation: MTBench evaluates pairwise responses to 80 multi-turn prompts spanning eight categories, using 200 expert-annotated comparisons excluding ties.
  • Controllable Summarization: Instrusum evaluates instruction-controllable summarization using 411 instances with perfect inter-annotator agreement.Instances vary in summarization length, style, or focus.
  • Human-Response-Guided Evaluation: HREF provides human-written instructions and reference responses for human-annotated pairwise comparisons across five selected task categories.
  • Dataset Selection: Summarization, brainstorming, and open-QA categories were excluded to avoid overlap, maintain diversity, or reduce subjectivity and noise.
  • Dataset Summary: Table 20 lists the number of instances for each evaluation dataset.

E TRAINING DETAILS

The appendix specifies training hyperparameters, instruction-category analyses, and prompt templates for reference-free, reference-guided, and related evaluation methods.

  • Training Configuration: Training uses 2 epochs, batch size 128, maximum learning rate 5e-6, linear scheduling, 3% warmup, and a 2048-token sequence limit.Examples exceeding the length limit are filtered, leaving 883K training instances.
  • Instruction Classification: AlpacaEval contains more open-ended instructions, whereas Arena-Hard emphasizes coding and mathematical reasoning.Instruction categories are classified with GPT-4o into Coding&Math, Creative Tasks, Information Seeking, and Reasoning&Planning.
  • Prompt Templates: The appendix provides prompt templates for Ref-Free, RefEval, RefMatch, HREF-Base, HREF-Ref, Multi-Ref Avg, and Multi-Ref Max.
  • Variants and Related Methods: Additional templates cover rule-based variants, metric-plus-reference prompting, Prepair’s pointwise and pairwise stages, and related self-reference prompting.
  • RefEval: RefEval instructs judges to prioritize precise instruction execution, then consider helpfulness, accuracy, detail, and harmlessness.It also uses a reference output to characterize successful instruction following and compare candidate effectiveness.
Loading 2602.16802v1…