PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Relevance-Centric Retrieval: Rubric-Oriented Document Set Selection and Ranking

Kailin Jiang, Lei Liu, Jian Xi, Hui Xu, Junlin Liu, Baochen Fu, Bin Li, Vichwang, Yu Lu, Haibo Shi

arXiv:2607.19747v2cs.CL

TL;DR

Existing retrieval evaluation scores documents independently, overlooking interactions within document sets that matter for downstream generation. This paper introduces rubric-based set evaluation and selection, finding that Rubric4Setwise delivers the best generation performance across short- and long-form scenarios with fewer documents and search rounds.

  • Problem

    Existing evaluation systems score documents independently and aggregate them, overlooking redundancy, conflict, complementarity, and overall document-set quality.

  • Method

    The paper builds a three-level, nine-dimension rubric benchmark and converts query-specific rubric criteria into adaptive document-set selection signals.

  • Results

    Rubric4Setwise achieves the best downstream generation performance across both short-form and long-form scenarios, while using fewer documents and search rounds.

  • Takeaways & Limitations

    Rubric-based evaluation signals can guide document-set optimization beyond relevance-centric reranking across short-form and long-form retrieval scenarios.

  • Takeaways & Limitations

    Rubric4Setwise operates under an oracle setting because its rubrics are generated with reference answers, establishing an empirical upper bound.

Abstract

from arXiv · show

As large language models and AI agents become the primary consumers of search results, document set quality determines the upper bound of downstream generation. Yet existing evaluation systems remain confined to scoring documents independently and aggregating via nDCG, ignoring inter-document interactions (redundancy, conflict, complementarity) and unable to answer what makes one document set better than another. To address these issues, we propose a complete evaluate-diagnose-optimize framework. We design SetwiseEvalKit, a three-level, nine-dimension document set evaluation benchmark covering both short-form and long-form scenarios, comprising approximately 28K high-quality evaluation rubrics. We systematically evaluate 12 rerankers: even the best method achieves no more than 45% coverage, cross-document coordination dimensions are universally weak, and no single method maintains top performance across both settings. Building on this, we propose Rubric4Setwise, a training-free method that converts rubric-based evaluation criteria into document set selection signals, achieving the best downstream generation performance with fewer documents and search rounds. It is the only method that maintains state-of-the-art results across both scenarios, validating the effectiveness of closing the loop from evaluation to optimization.

1 Introduction

As retrieval shifts from human browsing to LLM-based generation, document sets must be evaluated and optimized holistically rather than by document relevance alone. The proposed framework addresses this gap with multidimensional set evaluation, diagnosis, and optimization.

  • Motivation: RAG systems inject retrieved document sets directly into LLM context, making set quality central to downstream generation.Human users can filter redundancy, resolve conflicts, and identify gaps; LLM-based generation requires stronger document-set evaluation.
  • Problem: Even with nDCG@5 = 100%, relevance-only selection can produce redundant, conflicting, and incomplete document sets.These defects include overlapping documents, factual errors, and missing key information.
  • Problem: Existing evaluation systems score documents independently, overlooking inter-document interactions and holistic document-set quality.This structural blind spot spans classic retrieval benchmarks, LLM-Judge methods, and recent retrieval evaluations.
  • Contributions: SetwiseEvalKit provides three-level, nine-dimension evaluation across Short-form and Long-form scenarios, producing approximately 28K high-quality hybrid rubrics.Its query-specific rubric generation pipeline supports multidimensional document-set assessment.
  • Contributions: 12 rerankers were systematically evaluated, yet the best method achieved no more than 45% overall coverage.Rubric coverage scores were strongly predictive of downstream generation quality, supporting the benchmark’s credibility.

2 Related Work

Existing retrieval benchmarks primarily score documents independently, aggregating relevance while overlooking inter-document relationships and broader set-level properties. This work addresses that gap by transferring structured rubrics to document-set evaluation and training-free set selection.

  • Retrieval Evaluation Benchmarks: Existing benchmarks such as TREC-DL, BEIR, MTEB, and BRIGHT annotate per-document relevance and aggregate scores with nDCG.RAGAS and ARES add LLM-based judges but still reduce evaluation to binary per-document judgments.
  • Retrieval Evaluation Benchmarks: Existing retrieval benchmarks evaluate documents individually and primarily cover relevance, leaving complementarity, redundancy, and conflict unexamined.The paper characterizes these as two structural blind spots and introduces a framework spanning three granularity levels and nine dimensions.
  • LLM-based Document Reranking: LLM-based rerankers have progressed from pointwise document scoring toward listwise and setwise comparison and set-composition optimization.The cited progression includes reasoning-enhanced methods as a further development in LLM-based document reranking.
  • Rubric-based Evaluation: RUBRIC4SETWISE transfers structured rubrics from answer evaluation to nine-dimensional set evaluation and uses them as training-free set-selection signals.This bridges passive assessment and active guidance without requiring training.

3 Rubrics-Oriented Evaluation and Set Selection

This section introduces rubric-oriented evaluation that scores document sets across nine dimensions, capturing inter-document interactions and holistic utility. It then presents RUBRIC4SETWISE, a training-free selector that converts query-specific rubrics into adaptive set-selection guidance.

  • 3.1 Task Formulation: Query-specific rubrics generated from the query and reference answer define nine quality dimensions spanning document, set, and global levels.The rubric set is R = {r1, r2, . . . , rK}, with criteria constructed from (q, a).
  • 3.1 Task Formulation: Traditional reranking independently scores selected documents and aggregates them, overlooking inter-document interactions such as redundancy and conflict.Each method selects Sm ⊆ C with |Sm| ≤ k, while conventional aggregation implicitly equates set quality with summed document quality.
  • 3.1 Task Formulation: The proposed evaluator scores each selected set holistically with an LLM, producing Score(Sm) = E(q, Sm, R) ∈ R9 across the rubric dimensions.The evaluator comprehensively reads the set and scores its quality along each dimension.
  • 3.2 SETWISEEVALKIT: SETWISEEVALKIT organizes evaluation into three levels and nine dimensions: document attributes, inter-document relationships, and holistic set utility.The dimensions cover relevance, authenticity, quality, complementarity, redundancy, conflict, completeness, density, and reachability.
  • 3.2 SETWISEEVALKIT: The rubric dimensions assess document-level relevance, authenticity, and quality; set-level complementarity, redundancy, and conflict; and global completeness, density, and reachability.Global reachability asks whether the full reasoning chain can be completed using only the selected set.
  • 3.2 SETWISEEVALKIT: Approximately 70% of aggregated rubrics were judged Highly or Critically discriminative, versus 37% for GPT 5.1 and 43% for Gemini 3.1-Pro-Preview, with an approximately 8% waste rate.Approximately 77% of aggregated rubrics caused Moderate or Significant ranking changes when removed.
  • 3.3 RUBRIC4SETWISE: RUBRIC4SETWISE is a training-free method that converts evaluation criteria into selection guidance and selects S∗ = arg maxS⊆C f(S; q, R).The subset size is adaptively determined by rubric satisfaction rather than fixed in advance, with Qwen3-8B modeling f through chain-of-thought prompting.

4 Experiments

Experiments evaluate 12 rerankers with rubric-based LLM judging across short-form and long-form scenarios, establishing that rubric coverage strongly predicts downstream generation quality. Results show scenario-dependent leaders, motivating RUBRIC4SETWISE, which turns evaluation criteria into training-free document-selection signals.

  • Evaluation Setup: 12 mainstream rerankers are evaluated on SETWISEEVALKIT, spanning adhoc, reasoning-enhanced, and setwise reranking paradigms.The evaluation includes cross-encoders, sequence-to-sequence models, LLM-based listwise methods, and reasoning-enhanced approaches.
  • Benchmark Credibility: Rubric coverage and downstream generation performance show a strong positive correlation, with Pearson r = 0.92 and p = 0.0013.This consistency analysis supports the benchmark’s practical credibility as a measure of document-set quality.
  • Short-form Scenario: Setwise reranking methods dominate nearly all short-form dimensions, with SetR and Rank4Gen achieving significantly higher Overall scores than adhoc and reasoning-enhanced methods.Setwise methods lead at both Doc-Level and Set-Level, indicating benefits from optimizing document quality and inter-document relationships together.
  • Long-form Scenario: Reasoning-enhanced methods lead the long-form scenario: ReasonRank and Rearank rank first and second overall, while previously dominant Setwise methods fall to mid-tier.The reversal suggests that multi-turn cumulative retrieval depends more on reasoning about the marginal value of newly retrieved documents.
  • Cross-scenario Analysis: No existing reranker consistently achieves the best performance across both scenarios: Setwise methods prevail short-form, whereas reasoning-enhanced methods dominate long-form.This scenario-dependent leadership reveals a generalization gap in current rerankers.
  • Rubric-guided Optimization: RUBRIC4SETWISE converts evaluation criteria into selection signals without additional training, achieving the best short-form performance while using only 2.66 documents on average.It outperforms the second-best SetR by 0.97 ↑ on EM and 0.62 ↑ on F1, while conventional rerankers use a fixed top-5.

5 Case Study and Qualitative Analysis

The case study shows that keyword-overlap retrieval can produce redundant, contradictory documents that fail to jointly reconstruct the reasoning chain. It demonstrates why traditional nDCG cannot capture these set-level coordination deficiencies.

  • Only Doc [11] directly describes the crowning event, while the remaining documents are retrieved through surface keyword overlap with “Miss USA” and “crowned.”
  • Redundancy (2/4) and Conflict (2/4) expose repeated yet contradictory claims about the crowning identity.
  • Reachability (1/4) confirms that the documents cannot jointly reconstruct the full reasoning chain, a deficiency traditional nDCG cannot capture.

6 Conclusion and Limitation

The paper presents an evaluate-diagnose-optimize framework for document set quality, centered on SETWISEEVALKIT and RUBRIC4SETWISE. RUBRIC4SETWISE achieves the best downstream generation with fewer documents and search rounds, but operates under an oracle setting using reference answers.

  • Conclusion: SETWISEEVALKIT is a set-level evaluation benchmark spanning three granularity levels and nine dimensions.It reveals that existing rerankers are universally weak on cross-document coordination.
  • Conclusion: No single reranking method maintains top performance across both settings.This finding accompanies the benchmark’s diagnosis of universal weakness in cross-document coordination.
  • Conclusion: RUBRIC4SETWISE converts rubric signals into selection guidance and achieves the best downstream generation with fewer documents and search rounds.Its rubric-based signals provide a basis for targeted document-set optimization.
  • Limitation: RUBRIC4SETWISE operates under an oracle setting in which rubrics are generated with reference answers.The result establishes an empirical upper bound while validating rubric-as-signal effectiveness.

A Implementation Details of SETWISEEVALKIT · A.1 Data Source · A.2 Generator and Search Agent

SetwiseEvalKit draws on established short-form multi-hop QA benchmarks and the long-form ResearchQA resource, then evaluates retrieval with dedicated short- and long-form generation agents. The implementation uses moderate-scale instruction-tuned models and rubric-oriented scholarly queries to emphasize document-set quality and doctoral-level information needs.

  • A.1 Data Source: Short-form evaluation adopts four established multi-hop question-answering benchmarks.These datasets require reasoning across multiple supporting documents.
  • A.1 Data Source: Over 112K crowdsourced question-answer pairs comprise HotpotQA, which is based on Wikipedia.HotpotQA requires models to reason across multiple supporting documents to derive answers.
  • A.1 Data Source: Approximately 192K samples comprise 2WikiMultihopQA, constructed from Wikipedia and Wikidata with explicit evidence paths in triples.
  • A.1 Data Source: ResearchQA provides 21K long-form scholarly queries paired with 160K rubric items distilled from survey articles across 75 research fields.Its rubrics specify query-specific criteria such as citing papers, making explanations, and describing limitations.
  • A.1 Data Source: Validation by 31 Ph.D. annotators found that 96% of ResearchQA queries reflect doctoral-level information needs.
  • A.2 Generator and Search Agent: The short-form scenario uses Llama-3.1-8B-Instruct to generate concise answers conditioned on each reranker’s top-k documents.The moderate-scale instruction-tuned model is intended to make downstream differences primarily reflect document-set quality rather than generator capacity.
  • A.2 Generator and Search Agent: The long-form scenario uses DR.Tulu-8B, an open-source deep research agent built on Qwen3-8B and trained end-to-end via Reinforcement Learning with Evolving Rubrics.DR.Tulu alternates internal planning, tool invocation, and final-answer generation in an autonomous multi-turn search loop, using web search, web browsing, and paper retrieval.

A.3 Reranker · A.4 Rubric Statistics · A.5 Human Study Interface

The paper compares diverse pointwise, pairwise/listwise, and prompting-based rerankers, reports rubric-generation statistics for SETWISEEVALKIT, and validates rubric quality through a structured human-annotation interface. The interface evaluates rubric discriminativeness and the effect of rubric omission on document-set rankings across short-form and long-form scenarios.

  • A.3 Reranker: BGE-Reranker-Large uses bidirectional cross-encoder attention and approximately 560M parameters, while MonoT5 predicts relevance through “true” token probabilities.BGE-Reranker-Large is based on XLM-RoBERTa; the study uses monot5-3b-msmarco-10k for MonoT5.
  • A.3 Reranker: RankT5 directly produces scalar relevance scores from decoder logits and supports pointwise, pairwise, and listwise ranking losses.The evaluated checkpoint is rankt5-base fine-tuned on MS MARCO.
  • A.3 Reranker: RankLLaMA applies a linear projection to the final-token hidden state for scoring, using pairwise training with BM25 and dense-retriever hard negatives.It is fine-tuned on MS MARCO and built from LLaMA-2-7B.
  • A.3 Reranker: RankVicuna and RankZephyr are fully open-source 7B-scale listwise rerankers trained with permutation distillation, with RankZephyr additionally using direct preference optimization.RankVicuna is built on Vicuna-7B, while RankZephyr is built on Zephyr and distills GPT-3.5/4 outputs.
  • A.3 Reranker: Setwise presents candidate documents simultaneously for comparative selection, whereas Rank1 uses chain-of-thought reasoning distilled from DeepSeek-R1 on over 600K MS MARCO query-document pairs.Setwise uses Heapsort-based selection to trade off ranking effectiveness and computational cost; Rank1 outputs binary relevance judgments after step-by-step reasoning.
  • A.4 Rubric Statistics: 33,522, 20,845, and 24,052 Short-form rubrics were generated by GPT 5.1, Gemini 3.1-Pro-Preview, and DeepSeek-V4 Pro, averaging approximately 15, 9, and 11 per query.These averages span all nine evaluation dimensions.
  • A.5 Human Study Interface: Annotators evaluate each rubric on 6-point Likert scales for Discriminative Importance and Omission-based Change in Set Ranking.Samples include the query, gold or reference answer, and rubric items grouped by evaluation dimension; ranking-improves indicates that a rubric introduces noise.
  • A.5 Human Study Interface: The Short-form interface shows the query and short answer, while the Long-form interface adds a cited full reference answer and preserves original rubric-dimension groupings.This design reflects the richer information structure of multi-turn research scenarios and supports contextual, consistent scoring.

A.6 Inter-Annotator Agreement Numbers of Human Study

Human judgments showed almost perfect inter-annotator agreement: Krippendorff’s α = 0.81 and ICC = 0.84. These metrics are appropriate for three annotators using a 6-point ordinal Likert scale and confirm reliable annotation.

  • Agreement metrics: Krippendorff’s α (ordinal) accounts for distances between rating levels and supports multi-rater ordinal data.The study uses this metric because three annotators rate items on a 6-point ordinal Likert scale.
  • Agreement metrics: ICC(3,1) treats ratings as continuous values and measures the absolute consistency of annotator scores.For this study, k = 3 annotators; ICC > 0.75 indicates excellent reliability.
  • Agreement results: 0.81 Krippendorff’s α and 0.84 ICC indicate almost perfect agreement across annotators.The reported α exceeds the 0.800 high-agreement threshold, while ICC exceeds the 0.75 excellent-reliability threshold.

A.7 Experiment Resources

Experiments run on NVIDIA H20 GPUs with 96GB HBM3 memory, with each reranking model deployed on a single GPU for inference.

  • A.7 Experiment Resources: Experiments use NVIDIA H20 GPUs with 96GB HBM3 memory, and each reranking model runs on a single GPU for inference.This setup applies across the experiments.

B More Experimental Results · B.1 Reproducibility of Rubric Scoring Across Two Independent Passes

Rubric scoring is highly reproducible across independent passes, especially for Doc-Level dimensions, while ordering becomes fragile when reranker scores are narrowly separated. Long-form evaluation is consistently less stable than Short-form evaluation, but key leaderboard rankings remain robust.

  • B.1 Reproducibility of Rubric Scoring Across Two Independent Passes: Pearson’s r measures magnitude agreement between two scoring-pass reranker-mean vectors, while Kendall’s τ measures their ordering agreement; both range from −1 to 1.Values near 1 indicate reproducibility in magnitude or ordering.
  • B.1 Reproducibility of Rubric Scoring Across Two Independent Passes: Doc-Level dimensions are most reproducible: Short-form Relevance, Authenticity, and Quality reach τ ≥0.74 and r = 1.00, while Long-form values remain τ ∈[0.56, 0.85] and r ∈[0.79, 0.95].Short-form item-level weighted Cohen’s κ is ∈[0.92, 0.94].
  • B.1 Reproducibility of Rubric Scoring Across Two Independent Passes: Density shows extreme ordering fragility despite high item agreement: %±1 agreement is 99.7% Short-form and 90.9% Long-form, but τ is 0.54 and 0.35.Rerankers cluster within a narrow Density interval, allowing residual noise to change their ordering.
  • B.1 Reproducibility of Rubric Scoring Across Two Independent Passes: Set-Level Complementarity and Redundancy maintain r≥0.85 across scenarios with τ from 0.56 to 0.87, whereas Conflict has the weakest ordering agreement.Conflict reaches τ = 0.56 Short-form and τ = 0.38 Long-form because reranker means occupy a ≤0.35-wide interval.
  • B.1 Reproducibility of Rubric Scoring Across Two Independent Passes: Every dimension loses 6–20 percentage points on r and τ from Short-form to Long-form evaluation, reflecting larger document sets, open-ended queries, and accumulated sub-rubric noise.Long-form document sets are 2–8 times larger than Short-form sets.
  • B.1 Reproducibility of Rubric Scoring Across Two Independent Passes: Despite lower Long-form stability, the top-3 Relevance rerankers agree exactly across independent passes, and 6 of 9 dimensions retain robust leaderboard ordering.The passage reports this robustness alongside the uniform Long-form decline in agreement.
  • B.1 Reproducibility of Rubric Scoring Across Two Independent Passes: Across both scenarios, absolute rubric scores remain reproducible even when ordering is fragile, with item-level κ ∈[0.77, 0.94] and %±1 agreement ≥90%.Ordering fragility is concentrated in Density and Conflict, whose reranker separation is intrinsically narrow.

B.2 Detailed Per-Round Analysis in Long-form Scenario

Long-form reranker quality systematically declines across search rounds, especially at Doc-Level and Global-Level, while Set-Level coordination remains relatively stable. ReasonRank is most resilient overall, but universally low Reachability reveals a bottleneck that motivates rubric-guided, gap-aware selection.

  • All rerankers score lower in the last round than the first, with the best Overall score dropping from 27.92 (Setwise) to 23.10 (ReasonRank).The shift reflects later-round retrieval dynamics: early rounds have broader information acquisition and richer candidate pools.
  • Doc-Level averages drop approximately 3–5 points, while Global-Level Reachability also declines sharply; Set-Level scores remain relatively stable.Examples include BGE-Reranker Doc-Level 20.67→13.70, Setwise Reachability 10.13→8.94, and minimal variation in Redundancy and Conflict.
  • Rank4Gen achieves the highest Redundancy scores across all rounds—87.32, 88.56, and 88.78—but the lowest Complementarity scores—11.99, 9.03, and 9.69.Its generator-preference-aligned training improves deduplication while sacrificing information diversity.
  • ReasonRank leads average-all-rounds Overall at 23.69 and Doc-Level average at 18.35, while its Overall decline is only 3.74 points.Its Overall score changes from 26.84 to 23.10, suggesting reasoning capabilities help preserve selection quality as candidate pools become sparser.
  • Reachability remains extremely low across all methods and rounds, ranging from 4.38–11.60 and limiting complete end-to-end reasoning chains.This bottleneck motivates RUBRIC4SETWISE, which uses rubric signals for gap-aware document selection.

B.3 Correlation Between Rubric Dimensions and Downstream Generation Quality · B.4 Performance across Fine-Grained Knowledge Types

Document-level rubric dimensions are the strongest predictors of downstream generation quality, while current rerankers show weak, statistically non-significant differentiation on set-level coordination. Performance varies substantially across reasoning types and research fields, leaving clear improvement opportunities in difficult slices.

  • B.3 Correlation Between Rubric Dimensions and Downstream Generation Quality: Relevance (r = 0.94, p < 0.001), Quality (r = 0.90, p = 0.002), and Authenticity (r = 0.88, p = 0.004) strongly predict downstream generation quality.All three correlations are statistically significant at p < 0.01.
  • B.3 Correlation Between Rubric Dimensions and Downstream Generation Quality: Density correlates significantly with generation quality (r=0.83, p=0.010), whereas Completeness (r = 0.52, p = 0.191) and Reachability (r = 0.43, p=0.292) do not.Completeness and Reachability show positive directional trends but lack statistical significance.
  • B.3 Correlation Between Rubric Dimensions and Downstream Generation Quality: Complementarity (r = 0.51, p = 0.193), Redundancy (r = 0.63, p = 0.093), and Conflict (r = 0.23, p = 0.590) show weak, non-significant correlations.Conflict scores span only 92.5 to 95.0 across current rerankers, limiting observable variance.
  • B.3 Correlation Between Rubric Dimensions and Downstream Generation Quality: Existing methods are highly homogeneous and deficient in cross-document coordination, so set-level weakness does not imply that coordination dimensions are unimportant.No method achieves meaningful differentiation on set-level objectives.
  • B.4 Performance across Fine-Grained Knowledge Types: Short-form analysis partitions 2 061 questions into eight reasoning types across Bridge Family and Non-Bridge pillars, while Long-form analysis spans 16 research fields.Bridge types include Br, PeC, SpC, and MiC; Non-Bridge types include Cmp, BrC, Inf, and Set.
  • B.4 Performance across Fine-Grained Knowledge Types: On easier Short-form Br and Cmp types, SetR and Rank4Gen lead Only Retrieval by +8 to 19 points, but the advantage narrows to +3 to 5 points on harder PeC, SpC, and MiC types.The harder types also show tight clustering among rerankers.
  • B.4 Performance across Fine-Grained Knowledge Types: Long-form rerankers separate clearly on Health fields—PH, EM, and EP—but nearly overlap on Materials Engineering and Sustainable Development.Ranker separation is therefore dependent on the evaluation slice.
  • B.4 Performance across Fine-Grained Knowledge Types: Person-attribute chains, spatial/geographic chains, and sustainability-related fields remain slices with substantial headroom for current rerankers.These slices are identified across the Short-form and Long-form analyses.

C Case Study · D Model Prompts

The case studies show that rubric-based evaluation exposes redundancy, complementarity, and cross-document quality patterns that document-level relevance misses. The model prompts operationalize these rubrics for passage selection, rubric generation, and scoring across document and set levels.

  • C Case Study: Four cases from HotpotQA, TriviaQA, and ResearchQA use nine-dimensional rubrics to provide document-, set-, and global-level diagnostic scores.Each case reports the query, reference answer, rubric text, selected documents, and corresponding scores.
  • C Case Study: Case 2 shows redundancy and complementarity explicitly: duplicate British Leyland answer sentences reduce Red to 2, while complementary ChopValue process and product documents yield Cmp = 3 and Con = 4.The ChopValue set also contains duplicated interview coverage, with Red = 1.
  • C Case Study: Case 1 shows that individually relevant documents can still be incomplete: MQL sources are authentic and moderately relevant, but none comprehensively explains the mechanism or quantitative benefits.Documents D1–D4, D6, and D8 average near 1.5 to 2, while D5, D7, and D9 score 0; the set has Con = 4.
  • C Case Study: Across the cases, doc-level relevance is insufficient, whereas Red and Cmp reveal redundant top-ranked documents and poorly complementary sets.The takeaways identify these patterns in short-form Case 2 and long-form Cases 1 and 2.
  • D.1 RUBRIC4SETWISE Prompt: Rubric4Setwise prompts map each rubric item to satisfying passages, then select the minimum passage set that jointly covers all items while retaining closest matches for unmet items.Every selected passage must contribute at least one rubric item not covered by another selected passage.
  • D.2 Rubric Generation Prompt for Short-form Scenario: Doc-Level: The short-form doc-level prompt requires binary, query-specific, answer-grounded rubrics for Relevance, Authenticity, and Quality, phrased as judgments of each document.It also limits generation to at most three nonredundant questions per dimension and requires concrete entities, facts, dates, or numbers.
  • D.3 Rubric Generation Prompt for Short-form Scenario: Set-Level: The short-form set-level prompt generates binary, query-specific rubrics for Complementarity, Redundancy, and Conflict that evaluate the document set as a whole.These dimensions assess joint coverage, duplicated core information, and mutually contradictory factual statements.
  • D.6 Rubric Scoring Prompt for Short-form Scenario: The scoring prompts demonstrate how rubric judgments distinguish direct from indirect evidence, such as scoring a document about the losing Titans lower for a query about the champion Rams.The long-form scoring example similarly evaluates documents against query-specific dimensions for strain-softening behavior in sensitive and non-sensitive clays.
Loading 2607.19747v2…