PaperlayerView exact revision on arXiv

Source-linked AI summary

Two-Level Meta-Rubrics for Evaluating Open-Ended Generation: GAMUT, a Benchmark for Factual Completeness

Xilun Chen, Zhaleh Feizollahi, Ross Goodwin, Seungwhan Moon, Scott Yih, Pinar Donmez, Babak Damavandi, Luna Dong

arXiv:2607.19322v2cs.CL

TL;DR

Evaluating open-ended factuality with flat binary checks misses the structure of complete answers. This paper introduces a two-level meta-rubric framework and Gamut, which remains challenging and discriminative, with the strongest model reaching 58.7%.

  • Problem

    Existing factuality evaluation often reduces answers to independent boolean fact-checks, missing open-ended coverage and order-sensitive structure needed to assess factual completeness.

  • Method

    A structured meta-rubric represents open-ended sets, ordered processes, relationships, and importance tiers, then mechanically converts them into a flat binary rubric for LLM judging.

  • Results

    Gamut is challenging and highly discriminative, with stable model rankings across judges; the strongest model reaches a Gamut score of only 58.7%.

  • Takeaways & Limitations

    The meta-rubric provides a general recipe for evaluating open-ended generation beyond long-form factuality when flat binary criteria fall short.

Abstract

from arXiv · show

Rubric-based evaluation of open-ended generation faces a fundamental tension between expressiveness and reliability. Authoring a faithful rubric requires expressing the structure of the space of good answers: open-ended sets of acceptable options, ordered processes, and the relative importance of facts. Grading with the rubric requires a judge to score consistently, and judges are far more reliable on flat, binary checks than on rich structure. We resolve this tension with a two-level meta-rubric framework. A structured meta-rubric captures the grading criteria at authoring time, and fixed mechanical rules compile it into a flat checklist of binary, machine-gradable checks that an LLM judge scores reliably at evaluation time. We instantiate the framework as Gamut (Grounded Assessment of Multimodal Factuality), a benchmark for factual completeness in long-form generation. Gamut comprises 1,813 questions grounded in real wearable imagery across 10 diverse domains, each paired with an evidence-backed rubric verified by expert human annotators. Evaluating 14 frontier and open-weight models, we find Gamut genuinely challenging (best score 58.7% from Gemini 3.1 Pro), highly discriminative, and robust to the choice of judge.

1 Introduction

The paper argues that factual completeness is an understudied evaluation target inadequately captured by independent fact checks, and introduces a two-level rubric framework to combine expressive annotation with reliable automatic grading. It instantiates this framework in Gamut, a multimodal-first benchmark that is challenging, discriminative, and robust to judge choice.

  • Motivation: Existing factuality evaluation emphasizes precision, while factual completeness—whether responses include all information a complete answer should—has received comparatively little attention.The dominant decompose-search-verify paradigm decomposes responses into atomic claims and verifies them against retrieved web documents.
  • Motivation: Open-ended questions, ordered processes, and relative fact importance expose limitations of fixed lists of independent binary checks.Different valid answers may contain different acceptable facts, while process questions require order-sensitive evaluation.
  • Framework: Gamut reconciles expressiveness and grading reliability by mechanically compiling a structured meta-rubric into a flat checklist of binary, machine-gradable items.Rubric construction and verification occur in the structured space, while evaluation uses the converted binary space.
  • Benchmark: Gamut contains 1,813 questions across 10 domains grounded in realistic everyday deep-research scenarios involving wearable imagery and multi-step web research.Question and rubric creation pair frontier web-search-capable LLMs with expert human annotators over multiple rounds.
  • Results: 58.7% was the strongest Gamut score, achieved by Gemini 3.1 Pro among 14 evaluated frontier and open-weight models.The benchmark is described as genuinely challenging and highly discriminative, with rankings aligning closely with independently understood model capabilities.
  • Results: The evaluation is robust to judge choice, and its text-only variant indicates that visual identification raises scores by a roughly constant amount across models.This supports interpreting multimodal rankings as differences in knowledge completeness rather than perception alone.

2 Related Work

Prior long-form factuality work has emphasized precision through claim verification, while a thinner line measures recall by checking coverage of reference facts. Evaluation has progressed from holistic LLM judgments toward explicit rubrics and binary checklists, motivating Gamut’s structured-authoring, flat-grading design for open-ended factual completeness.

  • Factuality evaluation: Most long-form factuality methods measure precision with decompose-search-verify pipelines that verify atomic claims against retrieved web documents.A thinner line instead evaluates recall by checking whether responses cover the reference facts required for a complete answer.
  • Factuality evaluation: Holistic Likert judgments are noisy, prompting a shift from rubric-based criteria to binary checklists that reduce score variance and increase agreement.This progression moves evaluation toward narrow, independently gradable checks rather than broad overall judgments.
  • Gamut’s positioning: Gamut uses a two-level meta-rubric that preserves structured authoring while mechanically compiling criteria into a flat checklist of narrow, self-contained checks.The framework separates expressive rubric construction from decomposed grading.
  • Gamut’s positioning: Unlike multimodal retrieval and agentic research benchmarks with single short verifiable answers, Gamut asks open-ended questions and scores factual completeness against reference-fact rubrics.It is also distinguished from multimodal deep-research benchmarks that grade generated-report quality rather than factual completeness.

3 A Two-Level Meta-Rubric Framework for Evaluating Open-Ended Generation

Gamut separates rubric authoring from grading: structured meta-rubrics represent coverage, ordering, importance, and relationships, then fixed rules compile them into auditable binary checks and tier-weighted scores.

  • 3 A Two-Level Meta-Rubric Framework for Evaluating Open-Ended Generation: Gamut uses a two-level representation because holistic judgments struggle to preserve the coverage and ordering that define good open-ended answers.A structured meta-rubric is compiled into a flat pass-or-fail checklist.
  • 3.1 Meta-Rubrics: Meta-rubric items classify answer content as Answer-Critical, Valuable, or Context and specify what is essential, acceptable, ordered, or supplementary.This structure mirrors how knowledgeable reviewers reason about answer quality.
  • 3.1 Meta-Rubrics: The framework supports Simple Knowledge, Strict List, Flexible List, Process, and Relationship items, each capturing a distinct factual structure.Flexible lists require minimum coverage without requiring specific options, while processes encode meaningful step order.
  • 3.1 Meta-Rubrics: Meta-insights add case-specific synthesis across list or process items, such as an overarching trend or categorization that no single item captures.Their importance may range from essential to secondary.
  • 3.2 From Meta-Rubrics to Binary Rubrics: Fixed mechanical rules deterministically compile each meta-rubric into a flat binary rubric, with flexible structures receiving most of the design attention.The resulting rubric remains auditable while preserving open-set coverage and ordering requirements.
  • 3.2 From Meta-Rubrics to Binary Rubrics: Relationships become entity-and-aspect checks, meta-insights become single checks, and secondary checks drop one importance tier from their parent requirement.Simple knowledge and strict lists become one check per fact or list item, respectively.
  • 3.2 From Meta-Rubrics to Binary Rubrics: Flexible lists generate baseline thresholds plus broader-coverage credit, while processes generate step-presence checks and sequence checks with tier-dependent treatment of optional content.Short lists use one check per item for additional coverage; long lists use a few higher thresholds.
  • 3.3 Scoring: Gamut scores each check independently, maps meets, partially meets, missing, and contradicts to tier scores, then combines tiers using wAC = 0.6, wV = 0.3, and wC = 0.1.Contradictions receive µ = −2 and partial satisfaction λ = 0.5, making falsehoods cost more than omissions.

4 Dataset Construction

Gamut’s dataset construction combines frontier-LLM generation with iterative human verification to create image-dependent, research-intensive questions and evidence-backed rubrics compiled into binary checks. Starting from 1,938 wearable images, the process corrected upstream errors and produced 1,813 questions with complete, human-verified rubrics.

  • Dataset Construction: Gamut is built in two stages: generating deep-research questions from real wearable imagery, then constructing structured meta-rubrics and compiling them into binary scoring rubrics.Both stages combine frontier-LLM proposals with human review and revision over multiple rounds.
  • Question Construction: 1,938 images from CRAG-MM span 10 everyday-life domains and commonly depict small, rotated, truncated, occluded, or poorly lit objects.The images cover head-, torso-, and tail-distribution entities across plants, food, animals, vehicles, places, objects, and products.
  • Question Construction: Candidate questions require multi-step web research, multi-paragraph answers, concise natural phrasing, and references to entities only through image-dependent terms.Automatic filtering rejects leading questions, including those that fail the stranger test and make the image decorative.
  • Question Construction: Every question is independently reviewed by two annotators, with a third adjudicating disagreements, and revision or regeneration continues until each image has an accepted question.Candidates may be accepted, revised into later rounds, or discarded before replacement generation.
  • Rubric Construction: Rubric generation researches each question, builds a cited structured meta-rubric, compiles its binary rubric, and then self-refines evidence coverage before human revision.Auditing re-verifies sources and flags incomplete lists or processes, while independent search retrieves missing evidence rather than editing the draft in place.
  • Rubric Construction: 12 upstream entity or question errors were corrected by hand, and discarded cases left 1,813 questions with complete, human-verified rubrics.Annotators revised the binary rubric directly while checking its conversion from the structured representation.

5 Evaluating Models on Gamut

Gamut is challenging and discriminative across 1,813 multimodal questions, while rankings remain robust across judges. A text-only conversion preserves model ordering and reveals a broadly uniform visual-identification cost.

  • Multimodal evaluation: 58.7% is the best Gamut score, and models meet at most 55% of rubric elements across 1,813 multimodal questions.These results indicate substantial headroom in model performance.
  • Multimodal evaluation: Gamut’s ranking recovers expected regularities, with newer and larger models outperforming older and smaller ones.The metric is described as highly discriminative.
  • Judge agreement: Gemini 3.1 Pro and Claude Opus 4.8 produce identical rankings, with per-model scores within 1.8 points and self-scoring differences within 0.3 points.Qwen3-VL 235B is more lenient but preserves the same broad ordering.
  • Text-only evaluation: The image-independent meta-rubric supports a text-only variant that rewrites questions to name subjects explicitly while keeping the judge and scoring unchanged.The conversion removes the Answer-Critical item identifying the named subject because it becomes trivially satisfied.
  • Text-only evaluation: 10 to 20 points is the typical Gamut-score increase after removing images, with every model scoring higher and the ordering broadly preserved.The increase is largely independent of multimodal score, indicating a broadly uniform visual-identification cost.

6 Dataset Analysis

Gamut’s questions are diverse, everyday, and non-templated across 10 domains, while its rubrics reveal pervasive, deep structure that a flat checklist would misrepresent. The rubrics average 15.2 evidence-backed binary checks, enabling fine-grained completeness evaluation.

  • Question diversity: Questions are short but diverse in wording and content, with domain-specific topics rather than a few repeated templates.The median question length is 23 words, the 25 most common opening phrases cover under 60%, and about one in six questions lack a standard interrogative opening.
  • Rubric depth: Each rubric averages 15.2 binary checks, split across 5.9 Answer-Critical, 5.8 Valuable, and 3.5 Context checks.Rubrics draw on about 10 human-verified web snippets each and cite over 9,400 distinct web pages across the benchmark.
  • Structured meta-rubrics: 98% of questions contain at least one structured component beyond Simple Knowledge, and 86% include a flexible-list structure.More than half require two or more structures, showing that the meta-rubric structure is pervasive rather than incidental.
  • Structured meta-rubrics: Structured components are deep: flexible-list pools have a median of four options, processes have a median of four steps, and 13% of questions include a meta-insight.Among 318 processes, 182 mix optional and mandatory steps; 357 flexible lists are long enough to use tiered coverage thresholds instead of per-item checks.
  • Design implication: For 98% of questions, a flat rubric of independently required checks would be wrong because it cannot represent open-ended coverage, process ordering, or graded importance.The two-level design instead specifies jointly required facts, open sets needing coverage, ordered requirements, and helpful information before compiling them into reliably gradable checks.

7 Conclusion

Gamut introduces a two-level meta-rubric framework that preserves expressive answer structure while mechanically producing reliable binary grading checks. The benchmark applies this framework to 1,813 image-grounded questions, remains far from solved, and supports broader rubric-based evaluation of open-ended generation.

  • Benchmark: 1,813 questions comprise Gamut, grounded in real wearable or public-source images and paired with evidence-backed rubrics verified by expert annotators.The dataset targets long-form, open-ended factual completeness.
  • Results: 58.7% is the strongest reported Gamut score, achieved by Gemini 3.1 Pro, showing that the benchmark remains far from solved.Evaluations also show stable rankings and large margins across current systems.
  • Generality: The two-level meta-rubric is presented as a general recipe for open-ended generation evaluation when flat binary criteria are insufficient.Although developed for long-form factuality, the framework is intended beyond that specific setting.
  • Release: The authors release Gamut, a text-only variant, and evaluation scripts.These releases support use of the benchmark and framework beyond the original multimodal setting.

Appendix · A Prompts

Appendix A provides the full prompts used to construct Gamut, covering question generation and selection, rubric generation, LLM judging, and text-only question conversion. The prompts enforce image dependence, research depth, factual rubric alignment, structured grading, and explicit handling of subject-identification mismatches.

  • A Prompts: The appendix collects the complete prompts used throughout Gamut’s construction and evaluation pipeline.These prompts cover question generation, question selection, rubric generation, LLM judging, and text-only question conversion.
  • A.1 Question Generation: Question generation requests 10 open-ended questions requiring multi-step research, careful reasoning, and answers longer than one or two paragraphs.Questions should be concise, natural, and plausible for a user addressing a smart assistant.
  • A.1 Question Generation: Generated questions must depend on the image as much as possible and avoid assuming knowledge beyond the original question.The prompt requires questions to be unanswerable without seeing the image and prohibits leading questions or excessive textual context.
  • A.1 Question Generation: Question generation outputs one question per line, excluding the original question and all additional explanation.The format constraint is explicit and requires no other text.
  • A.2 Question Selection: Question selection may reject all candidates, otherwise choosing the best question after testing research depth, non-leading phrasing, image dependency, and unambiguous reference.The selection prompt also proposes five improved questions and applies a Stranger Test to ensure questions can be asked from the visible image alone.
  • A.2 Question Selection: Among valid candidates, selection prioritizes a succinct research-intensive question that depends on the image, is instance-specific, and offers an original angle.Boilerplate questions are deprioritized, while proposed replacements must satisfy all hard requirements and be diverse from existing candidates.
  • A.3 Rubric Generation: Rubric generation uses three phases beginning with factual knowledge gathering anchored to the question’s actual subject and supported by the image, user question, and ground-truth entity.The prompt warns that the ground-truth name may mismatch the question’s target, requiring the target to remain the strict search anchor.
  • A.5 LLM Judge: The LLM judge evaluates every rubric element independently and holistically with four verdicts: meets, partially_meets, missing, or contradicts.The prompt distinguishes insufficient specificity from absence and reserves contradiction for substantive factual errors rather than trivial rounding or unit differences.

B. KNOWLEDGE SEARCH & SNIPPET CREATION · Rubric Refinement Prompt

The framework first gathers authoritative, evidence-backed snippets and organizes the knowledge required for a perfect answer into typed meta-rubrics. It then audits and refines draft rubrics through phased, mechanically specified transformations into atomic, self-contained binary checklist elements.

  • B. KNOWLEDGE SEARCH & SNIPPET CREATION: Web searches identify all critical, valuable, and contextual information needed to formulate a perfect answer to the user’s question.
  • B. KNOWLEDGE SEARCH & SNIPPET CREATION: Each search-result snippet records an exact URL, page title, page substring of at most 40 words, unique ID, and preferably an authoritative source.
  • B. KNOWLEDGE SEARCH & SNIPPET CREATION: Meta-rubrics decompose every distinct part of a perfect answer, assigning each a description, importance level, target type, and cited knowledge components.
  • B. KNOWLEDGE SEARCH & SNIPPET CREATION: Target types distinguish discrete facts, mandatory-item lists, volume-based option pools, chronological processes, and entity relationships.
  • B. KNOWLEDGE SEARCH & SNIPPET CREATION: Flexible lists use baseline thresholds for minimum recall, while short lists receive item-level bonus checks and long lists receive at least 2-3 upper-tier thresholds.The final upper-tier threshold represents a perfect answer, usually requiring all items unless the question caps the requirement.
  • B. KNOWLEDGE SEARCH & SNIPPET CREATION: Final rubric elements translate facts, lists, processes, and relationships into dedicated binary checks, with exhaustive option expansion and explicit sequence checks for processes.Process rubrics prohibit thresholds and separately evaluate mandatory-step order and the full chronological flow.
  • B. KNOWLEDGE SEARCH & SNIPPET CREATION: Cascading category logic assigns final elements according to parent importance: critical essentials remain Answer_Critical, while optional or upper-tier content drops to lower categories.Context meta-rubrics produce only Context elements and never Answer_Critical elements.
  • Rubric Refinement Prompt: The refinement prompt reviews and restructures a draft rubric by correcting factual errors, filling knowledge gaps, enforcing threshold curves, and requiring atomic, self-contained formatting across four ordered phases.Phase 1 audits snippets and rubrics, identifies the targeted subject and entity, and prioritizes the ground-truth name while recognizing possible mismatches.

B. SNIPPET AUDIT · C. DRAFT RUBRIC AUDIT

The process audits every draft snippet for validity, relevance, and exact quotation, then reviews rubric elements for correctness, completeness, categorization, and binary-check formatting. It flags partial lists or procedures for exhaustive follow-up and independently searches for missing knowledge to avoid anchoring bias.

  • B. SNIPPET AUDIT: Each snippet is verified for URL existence, subject alignment, and exact quotation matching.The audit also requires the Text field to be an exact quote of no more than 40 words.
  • B. SNIPPET AUDIT: Uncited snippets are assessed as either missing information or irrelevant hard negatives.The audit determines whether each uncited snippet contributes missing content or should be treated as irrelevant.
  • B. SNIPPET AUDIT: Each snippet receives a KEEP, DELETE, EDIT, or SPLIT action based on validity and quotation quality.DELETE removes invalid or irrelevant snippets, EDIT fixes quotations, and SPLIT separates content when needed.
  • C. DRAFT RUBRIC AUDIT: Every draft rubric element is reviewed for correctness, relevance, and formatting.The review checks whether the element is factually correct, properly categorized, and expressed as a strict, self-contained binary check.
  • C. DRAFT RUBRIC AUDIT: A rubric testing one item from a broader category or one step from a procedure is flagged as a partial list or process.Examples include testing only one affiliation or one procedural step, which triggers further completeness work.
  • C. DRAFT RUBRIC AUDIT: Flagged partial lists and procedures are used in Phase 2 to locate the exhaustive list or complete sequence.The flag directs the subsequent search toward missing siblings and the full structure.
  • C. DRAFT RUBRIC AUDIT: Phase 2 independently searches for missing knowledge to avoid anchoring bias on a flawed draft.The search is explicitly performed independently rather than relying only on the initial rubric.

A. SEARCH STRATEGY

The search strategy aims to construct a complete answer from scratch while explicitly addressing incompleteness identified in the initial draft.

  • A. SEARCH STRATEGY: The strategy covers every aspect of a perfect answer and targets Phase 1C completeness flags, especially incomplete lists or sequences.It requires searching for omitted elements when the draft is incomplete.

B. NEW SNIPPET CREATION · B. RESOLVE THE MOST-SPECIFIC TRUE ENTITY (use the rubrics)

The workflow resolves the most-specific true entity and converts image-grounded questions into standalone ambiguity-checked questions, then builds verified meta-rubrics and compiles them into binary checklist elements. It distinguishes target types, completeness thresholds, process ordering, relationships, and cascading categories while preserving exhaustive, self-contained grading criteria.

  • B. NEW SNIPPET CREATION: The process searches for missing information, adds uniquely identified snippets, and builds a verified blueprint that supersets the draft rubric and newly discovered knowledge.New snippets retain distinct IDs, and the blueprint is based on verified knowledge rather than a one-to-one draft mapping.
  • B. NEW SNIPPET CREATION: Meta-rubrics assign descriptions, importance levels, target types, and cited knowledge components, with completeness resolved through exhaustive Strict List, Flexible List, or Process structures.Standalone facts use Simple Knowledge, while relationships use entities and relationship nature.
  • B. NEW SNIPPET CREATION: Strict and Flexible Lists must contain homogeneous conceptual items, while concepts combining mandatory core items with optional valid alternatives must be split into separate Strict List and Flexible List meta-rubrics.Processes represent chronological steps, with mandatory and optional status explicitly marked.
  • B. NEW SNIPPET CREATION: Flexible Lists use baseline thresholds for minimum recall and bonus handling based on list length and importance, with long or Context lists requiring upper-tier thresholds whose final tier represents a perfect answer.Short Critical or Valuable lists map remaining items individually, whereas long lists avoid rubric bloat through aggregate thresholds.
  • B. NEW SNIPPET CREATION: Final elements map facts and list items to binary checks, expand thresholds into exhaustive option lists, and represent processes with presence checks plus mandatory and overall sequence checks.Process grading uses no thresholds, and relationship rubrics are decomposed into atomic entity and relationship elements.
  • B. NEW SNIPPET CREATION: Element categories cascade from meta-rubric importance: Critical parents make strict items, mandatory steps, and baselines Answer_Critical, while optional items and upper tiers drop to Valuable.Valuable parents assign Valuable to strict items, mandatory steps, and baselines, and Context parents assign Context to all resulting elements.
  • B. RESOLVE THE MOST-SPECIFIC TRUE ENTITY (use the rubrics): The workflow identifies the real entity from the identification rubric, combines brand, model, or generation details when needed, and uses that resolved entity as the strict anchor for subsequent processing.If no identification rubric is clear, it falls back to the most specific name in the first Answer_Critical element and then the ground-truth name.
  • B. RESOLVE THE MOST-SPECIFIC TRUE ENTITY (use the rubrics): Questions are rewritten as natural standalone text by removing image-only identification requests and replacing deictic or image references with the resolved entity, then classified as AMBIGUOUS or UNAMBIGUOUS based on remaining visual dependence.Spatial relations, transient visual states, and unspecified image text remain ambiguous, while factual history, specifications, processes, and general traits are unambiguous.

B Full Examples

Two complete Gamut examples show how structured meta-rubrics are compiled into tiered binary checklists. Together, the jollof rice and Tang sancai examples exercise every meta-rubric type and conversion rule described in Section 3.

  • Overview: The two complete examples pair questions and human-verified evidence with structured meta-rubrics and compiled binary checklists organized by importance tier.Each figure reproduces the full authoring and compilation process.
  • Jollof rice example: The jollof rice example demonstrates Flexible List demotion and separate mandatory-versus-full ordering checks for a Process.Its Answer-Critical baseline remains Answer-Critical, per-item checks drop to Valuable, and the two sequence checks occupy Answer-Critical and Valuable tiers.
  • Tang sancai example: The Tang sancai example adds a Relationship, places a meta-insight inside a Strict List, and uses all five meta-rubric types in one question.Together, the examples cover Simple Knowledge, Strict List, Flexible List, Process, Relationship, and meta-insights.

C Annotator Training and Auditing

Annotators were trained through guidelines, examples, scored qualification tasks, and targeted instruction before contributing to question or rubric creation. Completed jobs underwent 10% audits, feedback, and multi-annotator review or adjudication.

  • Question creation: Question annotators received guidelines and worked examples, then had to pass scored gold jobs at 95% before entering the live task.Guidelines covered naturalness, multi-step search requirements, and image dependence.
  • Auditing: Completed jobs were audited at 10%, with feedback and additional review rounds including third-annotator adjudication.This auditing procedure applied to both question and rubric creation workflows.
  • Rubric creation: Rubric annotators completed a 10-item two-part guidelines quiz and advanced after achieving a 90% passing score.Each quiz item paired a true/false question with a related multiple-choice question.
  • Rubric creation: Rubric annotators then completed practice examples and two 45-minute trainings on meta-rubrics and binary rubrics, each with Q&A.A chat channel connected annotators with the authors for ongoing clarification.
Loading 2607.19322v2…