PaperlayerView exact revision on arXiv

Source-linked AI summary

From Preferences to Principles: Rubric-Based Alignment for Grounded Knowledge Answers

Aman Saini, Priyanshu Kumar, Eric Peng, Kai Yuan, Harsh Girase, Wanming Chen

arXiv:2608.23812v1cs.CL

TL;DR

Open-domain QA needs reward signals that capture multiple dimensions of answer quality rather than relying on coarse holistic preferences. The paper introduces query-specific rubrics grounded in retrieved evidence and decomposed into quality dimensions for post-training. Across evaluation datasets, the approach consistently improves performance, with retrieval conditioning strengthening grounding and decomposition improving composition and instruction-following.

  • Problem

    Open-domain QA is difficult to optimize because high-quality responses are multi-dimensional and coarse preference signals can encourage reward hacking, verbosity inflation, or superficial optimization.

  • Method

    The framework generates query-specific rubrics from queries, retrieved passages, and reference answers, decomposes them into quality dimensions, and uses rubric judgments as reinforcement-learning rewards.

  • Results

    The combined retrieval-conditioned and multi-dimensional approach consistently improves over the instruction-tuned baseline across the evaluation suite, with the largest gains on Search Arena.

  • Takeaways & Limitations

    Grounded rubrics improve factual grounding and response faithfulness, while structured decomposition further improves composition and instruction-following.

  • Takeaways & Limitations

    The approach is evaluated on open-domain QA, relies on LLM-based evaluation proxies, uses single training and evaluation runs, and assumes reasonably relevant retrieved evidence.

Abstract

from arXiv · show

Designing effective reward signals for open-domain question answering is challenging because high-quality responses must simultaneously satisfy multiple aspects of answer quality that are difficult to capture with a holistic scalar objective. We introduce a rubric-based reward framework that generates query-specific rubrics grounded in retrieved evidence and decomposed into multiple quality dimensions, providing fine-grained supervision during post-training. Averaged across three evaluation axes (composition, grounding, and instruction-following), our approach improves over the instruction-tuned baseline by 6.5% and over flat rubric variants by 4%, with consistent gains across all evaluation datasets. Conditioning rubrics on retrieved evidence improves factual support, while decomposing rubrics into quality-specific dimensions further improves coherence, organization, and adherence to query requirements. Our results show that grounded, multi-dimensional rubrics provide more effective reward supervision for complex open-domain question answering.

1 Introduction

Open-domain QA is difficult to optimize because answer quality is multi-dimensional and coarse preference signals can encourage superficial behavior. The paper addresses this with query-specific rubrics grounded in retrieved evidence and decomposed across quality dimensions.

  • Open-ended QA remains challenging because answer quality is inherently multi-dimensional and difficult to capture through coarse preference signals.
  • Opaque or weakly specified rewards can produce reward hacking, verbosity inflation, and optimization toward superficial patterns rather than true answer quality.
  • Existing rubric approaches typically use flat or weakly structured criteria and do not explicitly ground rubric generation in retrieved evidence.
  • The proposed framework generates query-specific, evidence-grounded, multi-dimensional rubrics for fine-grained and controllable supervision.
  • The framework analyzes retrieval conditioning and rubric decomposition as distinct components of reinforcement-learning supervision.

2 Related Work

Prior work replaces holistic preference rewards with structured rubrics or checklists, but retrieval-grounded and dimension-controllable supervision remains underexplored. This paper targets that gap for RAG systems.

  • Open-ended language tasks challenge reinforcement learning because answer quality is subjective, multi-faceted, and difficult to capture with coarse rewards.
  • Rubrics as Rewards and checklist-based feedback use structured, instance- or instruction-specific criteria to provide more informative and interpretable supervision.
  • Recent work studies rubric robustness, scalable generation, and rubric-based RL, including mitigation of reward overoptimization.
  • The paper addresses limited prior emphasis on retrieval-grounded rubric generation and controllable supervision across distinct quality dimensions in RAG systems.

3 Problem Setup

The task is grounded answer generation for knowledge-seeking queries in a RAG setting. Responses must answer the query using retrieved evidence and explicitly support factual claims with citations.

  • Each example pairs a user query with retrieved passages, and the policy generates a helpful response faithful to that evidence.
  • Unlike standard QA, the model must rely only on information supported by retrieved passages rather than unstated or potentially stale external knowledge.
  • Every factual claim should have at least one passage citation, making the objective more demanding than correctness alone.
  • The alignment pipeline has two stages: offline rubric generation followed by online reinforcement learning with rubric rewards.
  • Candidate answers are judged against associated rubrics, and the resulting per-item judgments are aggregated into a scalar reward for KL-regularized GRPO optimization.

4 Method

The method constructs query-specific rubrics from queries, retrieved passages, and reference answers, then scores responses through weighted rubric satisfaction. Rubrics are decomposed into quality buckets and normalized into a scalar RL reward.

  • Rubric Generation: Rubric construction uses the user query, retrieved passages, and reference answer to capture factual grounding, query intent, and crucial response aspects.
  • Rubric Structure: The fixed buckets are accuracy, composition, safety, and general, representing distinct principles used to guide reinforcement-learning supervision.
  • Rubric Generation: Rubrics are designed to be query-specific, atomic, testable, evidence-grounded, and discriminative.
  • Rubric Structure: Rubric items are categorized as Essential or Optional and assigned weights from 1 to 5 according to relative importance.
  • Reward Aggregation: An external LLM judge produces binary per-item satisfaction judgments, which are aggregated into weighted bucket scores and a scalar reward.
  • Reward Aggregation: Normalization by each example’s maximum achievable weighted score ensures the final reward lies in [0, 1].

5 Experiments

The experiments train rubric-rewarded models with GRPO and evaluate them on fixed-retrieval open-domain QA benchmarks using composition, grounding, and instruction-following metrics.

  • Training setup: The training setup uses GRPO with Qwen2.5-14B-Instruct as the policy model, sampling K = 4 candidate completions under a KL-regularized objective.Training uses the TRL framework.
  • Training data: The synthetic training set contains 4.7k English knowledge-seeking queries designed for realistic open-domain RAG information needs.The queries target diverse, ambiguous, and open-ended assistant-style requests.
  • Training data: Queries cover multi-aspect reasoning, comparisons, ambiguity, tradeoff analysis, decision-making, and location- or time-dependent requests.They also include partial specifications, implicit context, and varied user intent.
  • Rubric construction: Retrieved passages support rubric construction, while GPT-4o-generated reference answers provide grounded examples rather than ground truth.The retrieval setup uses the Brave Search API and top-k = 10 passages.
  • Evaluation: Evaluation fixes retrieved passages and measures generation quality across Search Arena, RAGBench, and FACTS Grounding Public.This isolates the model’s ability to produce grounded, well-structured, and useful responses from retrieval effects.
  • Evaluation: The evaluation judge scores Composition, Grounding, and Instruction-Following using externally established or custom LLM-as-a-judge procedures.Composition covers seven presentation dimensions, while Grounding is assessed sentence by sentence and Instruction-Following uses three issue categories.

6 Results and Research Questions

Rubric-based training consistently outperforms the instruction-tuned baseline, with the strongest results from combining retrieval conditioning, multi-dimensional decomposition, and optional rubric items.

  • Main results: Rubric-based training consistently improves over the instruction-tuned baseline across the evaluation suite, with the largest gains on Search Arena.Gains on RAGBench are smaller in absolute terms because its baseline performance is already high.
  • Retrieval conditioning: Conditioning rubrics on retrieved evidence improves Grounding across all datasets compared with checklist-style rubrics.Both single-dimensional and multi-dimensional grounded variants show stronger factual support and more consistent evidence attribution.
  • Multi-dimensional decomposition: Rubric decomposition into answer-quality buckets achieves the best performance across all metrics and datasets.The largest improvements over the single-dimensional grounded variant occur on Composition and Instruction-Following.
  • Multi-dimensional decomposition: Multi-dimensional grounded rubrics improve coherence, synthesis, completeness, and adherence to query requirements beyond grounded single-dimensional rubrics.Qualitative inspection also finds better aggregation and presentation of information.
  • Optional rubrics: Including Optional rubrics improves performance across all metrics and datasets beyond using Essential rubrics alone.Optional items add preferences that distinguish satisfactory responses from high-quality responses.
  • Overall takeaway: Retrieval conditioning and multi-dimensional decomposition each contribute gains, while their combination yields the strongest results.The authors frame this as evidence for structured, principle-based supervision on complex open-domain tasks.

7 Rubric Quality Analysis

The rubric quality analysis compares rubric-derived preferences with LLM-based preferences and finds that retrieval conditioning most improves agreement, while decomposition increases discriminativeness.

  • Evaluation protocol: Rubric-derived preferences are evaluated against LLM-based preference labels using ties, agreement, and egregious errors.The comparison uses query, retrieved passages, and pairs of candidate responses from OpenGenAlign.
  • Preference alignment: 43.4% agreement is achieved by checklist-style rubrics, indicating weak preference-estimation signal from unstructured criteria.This establishes the starting point for successive rubric design changes.
  • Preference alignment: 10.6% is the agreement increase from retrieval conditioning, rising from 43.4% to 54.0%.Egregious errors simultaneously decrease from 22.6% to 18.2%.
  • Preference alignment: Rubric decomposition reduces ties from 27.8% to 23.8%, producing more discriminative rubric-derived preferences while maintaining comparable agreement.The authors hypothesize that fewer tied rewards may benefit GRPO, but do not directly verify this mechanism.

8 Conclusion

The framework combines retrieval conditioning with structured rubric decomposition for open-domain question answering. Retrieval-conditioned rubrics improve grounding and faithfulness, while decomposition improves composition and instruction-following.

  • Retrieval-conditioned rubrics improve factual grounding and response faithfulness.
  • Structured rubric decomposition further improves composition and instruction-following.
  • The results support moving from flat reward formulations toward structured, principle-based reward design for complex question answering.

9 Limitations

The evaluation is limited to open-domain question answering, uses LLM-based evaluation as a proxy for human judgment, and reports single training and evaluation runs. Rubric quality also depends on reference-answer quality, while retrieval failures may weaken supervision.

  • The approach is evaluated only on open-domain question answering, leaving other domains and multilingual settings unvalidated.
  • LLM-based evaluation serves as a proxy for human judgment, so broader human evaluation could strengthen the conclusions.
  • Single training and evaluation runs mean small absolute differences may reflect run-to-run variance.
  • Generated-rubric quality depends on reference-answer quality, diversity, and coverage; using one reference may introduce stylistic or content biases.
  • Retrieval failures or incomplete evidence may reduce rubric quality and downstream supervision, and fixed retrieval evaluation does not fully capture end-to-end retrieval errors.

10 Ethical Considerations

The framework relies on LLM-generated rubrics, answers, and judgments, creating risks from inherited bias, synthetic-data distributional bias, and rubric over-optimization. Imperfect retrieval or rubric design may still incentivize undesirable behavior or factual errors.

  • LLM-generated rubrics, reference answers, and evaluation judgments may inherit biases or inaccuracies from their underlying models.
  • Synthetic data may introduce distributional biases or fail to capture the diversity of real-world user behavior.
  • Rubric-based optimization risks over-optimizing rubric satisfaction rather than broader helpfulness or truthfulness.
  • Imperfect retrieval, rubric design, or alignment principles may still incentivize undesirable behaviors or factual errors.
  • Human oversight and careful evaluation remain important for real-world deployment.

A Additional Training Details

The appendix describes a length penalty, fixed-retrieval evaluation datasets, preference-label evaluation, and examples of structured rubric construction. These details specify how verbosity is regularized and how varied question-answering settings are assessed.

  • A.1 Length Penalty: A length-based penalty regularizes rubric rewards by discouraging overly verbose responses while preserving rubric-based supervision.Responses near the reference length are not penalized; moderately longer responses receive a small penalty, while excessively long responses receive a capped larger penalty.
  • A.1 Length Penalty: The penalty uses the ratio between generated and reference answer lengths, with thresholds and coefficients selected empirically.
  • A.1 Length Penalty: The piecewise penalty is subtracted from the normalized rubric-based reward to produce the final reward.
  • Datasets: The selected datasets vary in reasoning complexity, query source, context length, and expected answer structure.They range from single-hop factoid responses in MS MARCO to longer, more detailed answers in ExpertQA.
  • Datasets: The evaluation uses fixed query-document pairs from RAGBench-derived datasets without modifying retrieval, isolating generation quality conditioned on retrieved passages.The combined subset contains 1.7k examples from MS MARCO, HotpotQA, HAGRID, and ExpertQA.
  • Preference Evaluation: Rubric-derived preferences are evaluated on 500 OpenGenAlign test samples using labels generated by majority voting over three independent judgments.The reported agreement with human annotations on the WebGLM subset is 77%.
  • Rubric Examples: Rubric examples decompose criteria across quality dimensions, classify items as Essential or Optional, and weight them from 1–5.The examples cover comparison reasoning, tradeoff analysis, and open-ended decision-making.

D Qualitative Examples

Qualitative examples compare baseline, checklist-style, single-dimensional grounded, and multi-dimensional grounded answers across language-learning and framework-selection queries. The multi-dimensional grounded variant most consistently combines evidence support with clearer organization, synthesis, and usability.

  • Qualitative Examples: The examples compare baseline, checklist-style, single-dimensional grounded, and multi-dimensional grounded responses on language-learning and FastAPI-versus-Django queries.The comparisons illustrate how grounding and rubric dimensionality affect factual support, structure, and synthesis.
  • Framework-Selection Query: For FastAPI versus Django, the multi-dimensional answer structures the comparison around each framework’s strengths and project requirements.It presents Django as full-stack with built-in features and FastAPI as API-focused, high-performance, and asynchronous.
  • Qualitative Examples: The single-dimensional grounded model improves factual support by aligning claims with supporting citations and adding evidence-backed details.
  • Qualitative Examples: The multi-dimensional grounded model improves structure, readability, synthesis, and answer organization while maintaining strong grounding.
  • Language-Learning Query: For language learning, the multi-dimensional grounded answer organizes recommendations into coherent categories, reduces redundancy, and integrates supporting evidence more effectively.Its categories include input and activation, practice and immersion, cultural engagement, structured learning, conversations, and personalization.
  • Prompt Templates: The prompt templates operationalize rubric generation, grounded evaluation, reference-answer generation, and composition assessment across the training and evaluation pipeline.The composition prompt evaluates clarity, conciseness, structure, completeness, tone, usefulness, and formatting, while grounding prompts test evidence support and contradiction avoidance.
Loading 2608.23812v1…