PaperlayerView exact revision on arXiv

Source-linked AI summary

Precision Is Not Faithfulness: Coverage-Aware Evaluation of Grounded Generation with a Complete Oracle

Juan S. Santillana

arXiv:2606.09376v2cs.CL

TL;DR

Precision-only faithfulness metrics can reward models for saying little, so this paper introduces coverage-aware evaluation using complete oracles and finds that requiring coverage reverses system rankings across domains.

  • Problem

    Precision-only faithfulness metrics reward abstention, while open-domain benchmarks lack complete, enumerable relevant-fact sets needed to measure recall.

  • Method

    The paper uses deterministic, complete Formula 1 oracles to enumerate relevant facts and evaluates grounded explanations with precision-plus-recall coverage.

  • Results

    0.46 of relevant facts were covered by the highest-precision frontier model, and requiring coverage changed system rankings across Formula 1 and weather.

  • Takeaways & Limitations

    Complete oracles make faithfulness comparisons sensitive to both factual precision and coverage rather than precision alone.

  • Takeaways & Limitations

    Faithfulness coverage is bounded by the claim ontology, and human extraction or judging remains future work.

Abstract

from arXiv · show

Reference-free faithfulness metrics verify each atomic claim a model makes against ground truth, and are increasingly used to evaluate grounded generation. We show they share a blind spot: they measure only precision -- are the stated claims supported? -- and therefore reward abstention, since a model can score near-perfect faithfulness by saying almost nothing. We make this measurable using Formula 1 telemetry, a domain where strategic ground truth is derived deterministically and, crucially, completely: for each decision we know the full set of facts that mattered. This completeness -- absent in open-domain faithfulness benchmarks -- lets us measure recall (coverage of the relevant facts) exactly, alongside precision. On a multilingual (EN/ES/PT) benchmark of 7,253 decision instances spanning 157 races, the most precise frontier model covers under half of the relevant facts and ranks last by F1, so requiring coverage reorders the systems; the same effect reappears in a second complete-oracle domain (NOAA weather forecasts). Fine-tuning small models (1B-7B) on the complete oracle closes the precision-recall gap entirely (F1 ~0.98), beating every zero-shot frontier system regardless of scale. We pair faithfulness with coverage into a single score, validate the metric (controlled perturbation; agreement across a model-free regex extractor and a cross-family LLM extractor, system-level Spearman 1.0), and give a verifier-guided generation method that improves precision and recall without references. We release the benchmark, structured annotations, metric, baselines, and an interactive demo.

1 Introduction

Precision-only faithfulness metrics reward abstention because they verify only whether stated claims are supported, not whether models cover all relevant facts. A complete oracle makes coverage measurable, revealing that requiring recall changes model rankings across Formula 1 and weather.

  • The abstention problem: Precision-only faithfulness metrics reward abstention: a model can state one safe fact, omit everything else, and still score near-perfect.These metrics measure the proportion of generated claims supported by ground truth, but not omitted relevant facts.
  • Complete-oracle evaluation: Complete-oracle evaluation adds recall by enumerating the facts that mattered, overcoming open-domain benchmarks’ inability to define a complete relevant-fact set.Formula 1 telemetry deterministically yields strategic ground truth, including pit laps, compounds, undercuts, and defenses.
  • Empirical finding: 0.46 is the relevant-fact coverage of the highest-precision frontier model, which ranks differently once coverage is required.The benchmark is multilingual and evaluates grounded Formula 1 strategy explanations rather than race-outcome prediction.
  • Empirical finding: Ranking inversion is replicated across two unrelated complete-oracle domains: Formula 1 and weather.The paper frames this as evidence that precision-only evaluation can mischaracterize grounded generation beyond a single domain.
  • Contributions: The work contributes a 7253-instance multilingual benchmark, a precision+recall metric, extractor-based validation, and verifier-guided generation that improves both precision and recall.The benchmark covers EN/ES/PT grounded Formula 1 decisions; validation uses controlled perturbation and agreement between model-free and cross-family extractors.

2 Related Work

Prior work spans structured data-to-text generation, fact-level accuracy scoring, and faithfulness evaluation, but most faithfulness metrics emphasize precision. Recall-oriented approaches use incomplete or estimated fact denominators, whereas this work uses a deterministic complete oracle to make precision–recall ranking inversion measurable.

  • Data-to-text generation: Structured data-to-text research includes sports, weather, and controlled table-to-text generation, with manual fact-level scoring as the closest prior accuracy practice.Examples include RotoWire, SportSett:Basketball, weather forecasts from records, ToTTo, and Thomson and Reiter’s manual gold-standard methodology.
  • Faithfulness evaluation: Faithfulness and hallucination evaluation methods are precision-oriented, scoring whether generated claims are supported rather than whether relevant facts are covered.The cited approaches include FActScore, QA-based consistency, NLI-based inconsistency, consistency benchmarks, sampling-based detection, and attribution or revision.
  • Recall of facts: Recall-oriented methods such as PARENT, SAFE, and RAGAS evaluate coverage or factual recall, but their denominators rely on references, retrieved context, or sampled facts that may be incomplete.SAFE reports factual precision and recall using F1@K, while PARENT requires a reference text and a possibly incomplete table.
  • Recall of facts: A deterministic, complete oracle supplies an exact recall denominator, enabling measurement of the precision–recall ranking inversion.The paper contrasts this complete denominator with recall estimates based on retrieved or sampled facts.

3 Task

The task evaluates whether models can explain Formula 1 strategic decisions with factual claims grounded in structured race contexts. It covers five decision types and prompts in English, Spanish, and Portuguese.

  • Task: Each instance pairs structured race context with an EN/ES/PT decision prompt requiring every factual claim to be verifiable against that context.Contexts include driver stints, tyre compound and age, pit stops, gaps, and safety-car/VSC status.
  • Task: The benchmark covers five decision types: tyre strategy, undercut, overcut, on-track defense, and race summary.On-track defense includes keeping a faster pursuer behind for several laps, while race summary requires a grounded recap of results and key moments.

4 Dataset

The dataset combines deterministic Formula 1 telemetry and strategic events with multilingual decision prompts and structured ground truth. Its complete oracle enumerates every checkable fact relevant to each decision, enabling exact coverage measurement.

  • Data extraction: FastF1 telemetry and deterministic rules derive stints, degradation slopes, pit stops, and strategic events without broadcast access.Degradation fits use green-flag laps only, excluding in/out laps and laps affected by yellow flags, SC, or VSC.
  • Dataset composition: 7,253 instances span 157 races across eight seasons and five decision types in EN/ES/PT, with season-based train/test splits.The dataset contains 6,004 training instances from 2018–2024 and 1,249 held-out test instances.
  • Release: The release includes derived structured ground truth, annotations, and code, while raw FOM data is not redistributed.The implementation materials are released in Section 10.
  • Complete oracle: The complete oracle enumerates the full set of checkable facts that each explanation should cover, including stops, pit laps, compounds, positions, moves, outcomes, and time gained.This enumerable fact set provides the denominator for recall.

5 Metric: Precision and Recall

The metric verifies typed atomic claims against structured ground truth rather than gold text, measuring both reference-free precision and oracle-enabled recall. Their harmonic mean, F1, penalizes terse explanations that omit relevant facts while rewarding explanations that are accurate and informative.

  • Claim verification: The reference-free metric labels each typed claim as supported, contradicted, or unverifiable against the structured data provided to the model.Claim types include pit lap, compound change, stop count, stint compound, final position, undercut/overcut, outcome, and time gained.
  • Recall (coverage): Recall measures the fraction of enumerable facts that mattered for an instance that the model correctly stated, penalizing omissions because the oracle is complete.A supported claim counts as correctly stating the corresponding fact.
  • F1: F1 is the harmonic mean of precision and recall, requiring explanations to be both accurate and informative.Reporting precision without recall is the quantified blind spot addressed later in the paper.
  • Two extraction backends: A dependency-free regex extractor supports fast, transparent English validation, while an LLM extractor handles free-form output in any language with the same typed-claim schema.Verification is identical across the two extraction backends; the LLM backend scores Spanish and Portuguese in cross-lingual comparisons.

6 Method: Verifier-Guided Generation

Verifier-guided generation iteratively edits an explanation using verifier feedback on both contradicted claims and omitted ground-truth facts. Because the oracle is complete, this reference-free loop jointly targets precision and recall and works with any LLM backend.

  • Verifier-Guided Generation: The method runs a verifier on each generated explanation, then feeds contradicted claims back as targeted instructions to fix them.The process repeats for a few rounds.
  • Verifier-Guided Generation: The verifier also identifies uncovered ground-truth facts and instructs the model to add them, targeting omitted information.This feedback is available because the oracle is complete.
  • Verifier-Guided Generation: The reference-free loop jointly targets precision and recall using only the structured verifier and supports any LLM backend.Its feedback addresses both false claims and missing relevant facts.

7 Experiments

Experiments show that precision-only faithfulness rewards abstention and can reorder system comparisons, while complete-oracle coverage exposes both undercoverage and imprecision. Fine-tuned small models reach near-perfect coverage F1, and the metric remains robust across languages, extractors, and tasks.

  • RQ1: Precision, coverage, and ranking: In PT, grok-4.3’s precision of 0.89 coexists with recall of 0.46, placing it last by F1 at 0.61, whereas claude-sonnet-4-6 leads at F1 0.83.Coverage against the complete oracle changes which systems look best.
  • RQ2: Multilingual evaluation: Coverage ranking changes persist across EN/ES/PT, while DeepSeek recall falls from 0.76 in EN to 0.50 in PT and grok recall from 0.84 to 0.46.Claude’s recall instead rises to 0.83/0.84 in ES/PT from 0.73 in EN through more verbose generations.
  • RQ3: Fine-tuning and verifier guidance: Fine-tuned 1B–7B models reach coverage F1 of ∼0.98, exceeding every zero-shot frontier system, including Claude Sonnet at 0.805 EN and grok-4.3 at 0.858 EN.The fine-tuned 3B model is both accurate and complete on this distribution by reproducing deterministic grounded templates.
  • Metric robustness: A deterministic regex extractor agrees with the LLM extractor on system ranking at Spearman 0.80 and instance-level scores at Pearson 0.50 across N = 564.The study re-scores identical generations with independent extractors to test possible same-family scoring inflation.
  • Open-ended summaries: On open-ended race summaries, DeepSeek-V3.2 reaches recall 0.42 at precision 0.49, showing that verbose recaps can be imprecise and that precision and F1 rankings disagree.The oracle scores winner, finishing positions, battles, and defenses.

8 Is It Just Prompting?

The section tests whether low coverage results from under-prompting by comparing the neutral default prompt with an explicit cover-all instruction. Asking models to state every supportable fact does not close the coverage gap, although the supplied passage truncates before reporting the recall result.

  • 8 Is It Just Prompting?: The authors test whether low coverage is an artifact of under-prompting.They compare the default prompt with an explicit cover-all prompt.
  • 8 Is It Just Prompting?: The default prompt asks models to explain using only the data, without a length instruction.It is deliberately neutral rather than explicitly requiring exhaustive coverage.
  • 8 Is It Just Prompting?: The cover-all prompt asks models to state every supportable fact, including pit laps, compounds, stops, moves, outcomes, time gained, and positions.Table 3 compares this instruction with the neutral default prompt.
  • 8 Is It Just Prompting?: Asking for completeness does not close the coverage gap.The passage states that mean recall does not rise, but the supplied text truncates before giving the quantitative result.

9 Generalization: A Second Oracle

The abstention problem generalizes beyond Formula 1 to NOAA weather forecasts with a complete oracle, where precision and F1 rankings again disagree. The milder weather effect indicates that coverage penalties scale with how much a faithful answer should contain, confirming the gap is a property of precision-only metrics.

  • NOAA Weather Forecasts: NOAA weather forecasts replicate the abstention problem using complete, enumerable fact sets for temperature, wind, precipitation chance, and sky.The benchmark uses 150 records per language, frontier models, and the same precision/recall machinery.
  • NOAA Weather Forecasts: Precision and F1 rankings disagree in the weather domain, though the effect is milder because each record contains fewer facts to omit.This moderation is itself informative: the coverage penalty scales with how much a faithful answer should contain.
  • Cross-Domain Generalization: The precision/coverage gap across Formula 1 and NOAA indicates that abstention is a property of precision-only metrics rather than one dataset.Only complete structured oracles make recall measurable, enabling this cross-domain comparison.

10 Ethics and Licensing

The paper respects usage restrictions on F1/FOM timing data by withholding raw feeds while releasing derived resources, and frames the benchmark as strategy explanation rather than prediction.

  • F1/FOM timing data carries usage restrictions, so the authors do not release raw broadcast or telemetry feeds.
  • The release includes code, derived structured data, and annotations instead of restricted raw data.
  • The benchmark evaluates strategy explanation, not betting or outcome prediction.

11 Limitations

The evaluation measures recall only over a high-precision, enumerable oracle, so claims outside its ontology remain unpenalized and applicability is limited to domains with complete fact sets. Additional limitations include template-mimicry risk, incomplete task coverage, and the need to audit the measurement pipeline itself.

  • Oracle scope: Recall is defined relative to the deterministic extractor’s enumerable fact set, which is high-precision but not exhaustive because event detection uses heuristics.The oracle is complete only for derived facts, not every potentially salient detail.
  • Training and evaluation: Fine-tuning uses silver deterministic templates, creating a template-mimicry risk that motivates stronger supervision and out-of-template evaluation.The reported model comparison uses a stratified, held-out sample across EN/ES/PT, but generalization beyond the template distribution remains unresolved.
  • Metric scope: Faithfulness coverage is bounded by the claim ontology, so unsupported details that the extractor never recognizes are never penalized.One model received a perfect supported fraction after naming a protected teammate and attributing a title-fight swing absent from the supplied context.
  • Measurement pipeline: The measurement pipeline requires auditing because extractor choices and output-token budgets can affect evaluation outcomes.Reasoning-model outputs required a higher token budget to prevent answer truncation, and the thesis’s measurement concern applies reflexively to the pipeline itself.
  • Task coverage: The headline comparison covers strategy, undercut, and overcut decisions, while full evaluation of on-track defense remains future work.The open-ended race-summary task is evaluated separately, but newly recovered brief, overtake-ending holds are not fully assessed.
  • Generalizability: A complete oracle restricts applicability to domains where the full fact set can be enumerated, motivating retrieval-delta evaluation for open-domain settings.The proposed relaxation would compare faithfulness with and without grounding context and validate its relationship to oracle-measured recall.
Loading 2606.09376v2…