PaperlayerView exact revision on arXiv

Source-linked AI summary

How Correct Is Your Answer? A Semantic Correctness Framework for Open QA Evaluation

Elitsa Yotkova, Violeta Kastreva, Petar Velkov, Hristo Boyanov, Dimitar Dimitrov, Ivan Koychev, Preslav Nakov

arXiv:2609.01369v1cs.CL

TL;DR

Open-ended QA evaluation must distinguish semantically different answer behaviors that lexical, binary, and judgment-based metrics may collapse. The paper introduces an eight-class taxonomy, datasets, and CAP, a bidirectional-NLI scorer over question-conditioned statements; CAP better aligns metric scores with the taxonomy’s ordering, while remaining limited by reference-based NLI and benchmark construction choices.

  • Problem

    Open QA answers can be correct with different wording or can be incomplete, unsupported, contradictory, or otherwise difficult for coarse metrics to distinguish.

  • Method

    The paper introduces an eight-class correctness taxonomy, CAP-Correctness, CAP-Statements, and CAP, which compares question-conditioned declarative statements using bidirectional NLI.

  • Results

    CAP achieves stronger semantic ranking alignment, higher pairwise ordering accuracy, and fewer monotonicity violations than similarity-based alternatives.

  • Takeaways & Limitations

    The taxonomy and monotonicity protocol provide a diagnostic structure for evaluating whether metrics distinguish increasingly less-correct Open QA answers.

  • Takeaways & Limitations

    CAP can invert partial and overinclusive-valid distinctions, underspecify alternative-correct answers, and depend on NLI world knowledge.

Abstract

from arXiv · show

Reliable evaluation of open-ended question answering remains a bottleneck for measuring answer correctness of modern LLMs. Unlike multiple-choice tasks, free-form answers may be correct in many surface forms and may fail in qualitatively different ways, including incompleteness, contradiction, overgeneration, and endorsement of false premises. Existing judgment-based and similarity-based metrics often collapse these distinctions. We address this gap with three reusable contributions. First, we introduce a semantic correctness taxonomy that assigns open-ended answers to eight ordered classes, separating verbose-but-correct answers from those contaminated by hallucinated content. Second, we release CAP-Correctness, an 8.8k-example benchmark spanning widely used QA datasets, and CAP-Statements, an 11k-example dataset for converting question-answer pairs into declarative statements for natural language inference (NLI) training and statement-based evaluation. Third, we introduce CAP (Context-Aware Precision), a reference-based metric that scores question-conditioned statements using bidirectional NLI. Under a monotonicity protocol testing whether metrics respect the taxonomy's intended ordering, CAP outperforms established baselines.

1 Introduction

Open QA evaluation is difficult because free-form answers can be correct despite different wording and can fail through incompleteness, unsupported additions, or contradiction. The paper proposes an eight-class taxonomy, reusable datasets, and CAP, then evaluates metrics by whether scores follow the taxonomy’s ordering.

  • Free-form Open QA outputs make automatic correctness evaluation substantially harder than multiple-choice evaluation.
  • Similarity-based metrics can penalize paraphrased correct answers while rewarding lexically similar answers that omit information, add unsupported claims, or contradict references.
  • Binary correctness labels obscure qualitatively different errors, including incompleteness, overinclusion, unsupported content, contradiction, and false premises.
  • The proposed taxonomy defines eight ordered semantic classes, enabling diagnostic evaluation of how answers relate to questions and references.
  • CAP reformulates question–answer pairs as declarative statements and compares them with bidirectional NLI to produce a continuous score in [0, 1].
  • CAP-Correctness contains 8.8k examples, while CAP-Statements contains 11k examples for QA-to-statement generation and evaluation.
  • Across comparisons, CAP achieves stronger semantic ranking alignment, higher pairwise ordering accuracy, and fewer monotonicity violations than similarity-based alternatives.

2 Related Work

Prior Open QA evaluators rely on lexical overlap, embeddings, learned scores, or coarse semantic judgments that do not fully distinguish answer relations. This paper frames evaluation around an eight-class taxonomy and a monotonicity benchmark, with CAP using bidirectional NLI to address these gaps.

  • NLI-based evaluation has been used to assess generated answers, but earlier approaches often treat correctness as an entailed-or-not decision.
  • The monotonicity benchmark tests whether metric scores respect the ordering induced by the correctness taxonomy, enabling systematic comparison across evaluation methods.
  • Existing metrics can misclassify paraphrased correct answers, reward surface-similar incorrect answers, and conflate correctness with faithfulness.
  • Prior graded-correctness work addresses answer equivalence, specificity, or partial credit, but remains tied in some cases to intermediate reasoning or token overlap.
  • The proposed taxonomy separates correct valid elaboration from correctness contaminated by hallucinated content, a distinction identified as important for verbose LLM answers.
  • CAP extends the taxonomy to eight classes, adds overinclusive-valid and overinclusive-invalid categories, and derives a continuous score from bidirectional NLI probabilities.

3 Proposed Taxonomy for OpenQA

The paper defines semantic correctness as the relation between a predicted answer and a gold answer under the same question context. Its taxonomy distinguishes exact, equivalent, alternative-correct, partial, overinclusive, and contradictory responses, ordered by expected quality.

  • Semantic correctness is defined as the relation between a predicted answer and a gold answer under the same question context.
  • Exact answers repeat the gold answer verbatim, while equivalent answers express the same meaning with different wording.
  • Alternative-correct answers satisfy the question with a valid response that may differ substantially from the single reference answer.
  • Partial answers provide only some required information, whereas overinclusive-valid answers add accurate and relevant information beyond the request.
  • Overinclusive-invalid answers contain correct content plus unsupported or false claims, while other predictions may directly contradict the gold answer.
  • The taxonomy orders answer relations by expected correctness so metric scores should decrease as answers become less correct.
  • Neighboring relations such as partial and overinclusive-valid may be adapted to application-specific preferences.

4 CAP-QA Framework

CAP reformulates gold and predicted answers as question-conditioned statements, then scores their semantic relationship with bidirectional NLI. Its directional and combined scores produce a bounded measure that can distinguish equivalent from incomplete answers.

  • An NLI classifier maps ordered statement pairs to entailment, neutral, and contradiction probabilities used for directed inference.
  • CAP converts a question with gold and predicted answers into declarative statements and compares them using bidirectional entailment scoring.
  • The directional score incorporates neutral-class weighting through λ ∈ [0, 1], while CAP combines two directional scores.
  • Bidirectional scoring distinguishes semantically equivalent from semantically incomplete answers by checking entailment in both directions.
  • CAP uses α = 0.85 and λ = 0.30, selected on held-out CAP-Correctness data because asymmetric scoring and moderate neutral weighting jointly maximize semantic ranking.
  • CAP's statement reformulation uses a fine-tuned mT5-based sequence-to-sequence model, providing a cheaper, standardized, and reproducible component.

5 Dataset Construction

The framework is supported by CAP-Correctness, a correctness-annotated QA benchmark, and CAP-Statements, a dataset for reformulating QA pairs into declarative statements.

  • CAP-Correctness and CAP-Statements provide complementary resources for semantic evaluation and NLI-based statement reformulation.
  • CAP-Correctness contains 8,827 labeled examples from OpenBookQA, AI2 ARC, and MMLU, spanning elementary through undergraduate-level questions.
  • The benchmark removes multiple-choice-specific questions and gold answers such as “all of above” and “none of above.”
  • Synthetic correctness labels reliably track human judgments, with Cohen’s κ = 0.779 and quadratic-weighted κ = 0.879.
  • CAP-Statements contains 11,000 question–answer–statement triples whose statements preserve the semantic content of their source pairs.
  • Long-form items with multi-sentence context are the hardest reformulation regime because several sentences must be compressed into one declarative claim.

6 Experiment Design

Experiments test whether CAP and established metrics preserve the taxonomy’s semantic-correctness ordering, separate neighboring classes, and generalize to LLM-generated answers.

  • The evaluation compares CAP with BLEU, ROUGE-L, METEOR, BERTScore (F1), and COMET across lexical, contextual-embedding, and learned-regression families.
  • Monotonicity asks whether metrics assign higher mean scores to every more-correct taxonomy class than to each less-correct class.
  • Local separability tests whether metrics distinguish neighboring taxonomy classes that challenge surface- and embedding-based scorers.
  • Generalization tests whether CAP preserves these properties on free-form answers generated by state-of-the-art LLMs rather than only semi-synthetic variants.
  • Rank correlation uses Spearman’s ρ and Kendall’s τ, while pairwise ranking accuracy measures correctly ordered class pairs against a random baseline of 0.5.
  • Headline comparisons use the untouched 7,827-example CAP-Correctness test split after selecting hyperparameters and the NLI backbone on 1,000 validation examples.

7 Results

CAP aligns more closely with the taxonomy’s intended correctness ordering than established metrics, while remaining imperfect on difficult neighboring classes and largely generalizing to human-labeled LLM outputs.

  • 7.1 CAP against Established Metrics: CAP achieves the strongest rank correlation and clearly informative pairwise accuracy, while lexical metrics and BERTScore remain near the 0.5 baseline.COMET performs better than the near-random baseline but recovers only part of the intended ordering.
  • 7.1 CAP against Established Metrics: CAP is near-perfect on distant class pairs and degrades smoothly as class pairs become more local.This profile places CAP’s gains on a difficulty axis rather than treating all pairwise distinctions as equally challenging.
  • 7.2 Monotonicity Analysis: Across 25 strictly ordered class pairs, CAP reduces class-mean ordering inversions to 4/25, compared with 9/25 for COMET and roughly half for lexical metrics and BERTScore.The remaining CAP inversions concentrate on alternative-correct and the partial/overinclusive-valid pair.
  • 7.3 Hard Neighboring-Pair Evaluation: Most baselines collapse to or below chance on neighboring class pairs, indicating that their global performance is driven by easy extreme contrasts rather than difficult middle distinctions.The hard comparisons include equivalent versus invalid and exact versus contradictory, but the genuinely informative failures occur among neighboring categories.
  • 7.3 Hard Neighboring-Pair Evaluation: All metrics except METEOR fall below chance on overinclusive-valid versus partial, with CAP showing the largest reversal.CAP’s asymmetric bidirectional formulation, using α = 0.85, produces this inversion even though the asymmetry helps elsewhere.
  • 7.4 Evaluation against LLM outputs: Human-labeled responses from GPT-4o, Gemini 2.0 Flash, and Qwen3-8B-Instruct populate all eight classes, while CAP’s class-level ordering largely generalizes to these outputs.CAP incurs 4 violations for GPT-4o, 6 for Gemini Flash, and 2 for Qwen3-8B-Instruct across 25 ordered class pairs.

8 Discussion

The paper evaluates CAP against an explicit, human-validated taxonomy ordering rather than taxonomy-free ground truth, and proposes reusing the benchmark and protocol for future metrics and calibrated classification.

  • 8 Discussion: CAP’s empirical claim is alignment with the taxonomy’s semantic correctness ordering, whose class labels were validated by human annotators.Monotonicity, pairwise accuracy, and ranking measures therefore assess alignment with this benchmark target.
  • 8 Discussion: The benchmark ground truth inherits the taxonomy’s design choices and the subjectivity of human annotation, so CAP’s alignment should not be treated as taxonomy-independent correctness.The framework instead provides an explicit, inspectable target ordering for comparing candidate metrics.
  • 8 Discussion: CAP’s continuous score can be calibrated into taxonomy class boundaries, yielding an evaluator that returns both a semantic score and a class label.The paper gives example intervals such as [0, 0.125) for contradictory and [0.125, 0.25) for invalid.
  • 8 Discussion: The evaluation protocol can extend beyond CAP by reusing CAP-Correctness, the taxonomy ordering, and calibrated metric-specific thresholds for shared comparison.Ambiguous neighboring pairs can be reordered according to application-specific evaluation needs.

9 Conclusion and Future Work

The paper presents an eight-class semantic correctness framework, reusable datasets, and CAP, while identifying extensions for finer-grained, stronger, and multilingual evaluation.

  • The taxonomy replaces binary correctness with eight ordered classes covering equivalence, incompleteness, valid elaboration, hallucinated additions, and contradiction.Its ordering enables monotonicity tests that assess whether metric scores decrease as answers become less correct.
  • CAP roughly doubles the rank correlation of the strongest tested baseline against the taxonomy ordering.The framework also supports class diagnostics and threshold-calibrated label-free classification.
  • Future work includes subcomponent-level entailment, stronger and multilingual NLI models, and revising the taxonomy as language-model behavior evolves.The current eight classes are explicitly not treated as final.

Limitations

CAP is constrained by whole-statement reference-based NLI, English educational data, synthetic labeling, and computational and representational limitations.

  • Whole-statement NLI can invert partial and overinclusive-valid answers because entailment is asymmetric across the two directions.CAP may also under-score alternative-correct answers that satisfy the question without entailing the single reference answer.
  • The benchmark is limited to English educational QA derived from multiple-choice sources, with synthetic candidates and closed LLM-assisted labels.Only a small subset is human-labeled, examples receive single annotations, and inter-annotator agreement is not estimated.
  • CAP is more computationally expensive than lightweight metrics because scoring requires statement generation and two NLI passes.Its single label per answer also cannot represent outputs combining multiple correctness dimensions.

Ethics and Broader Impact

The paper describes educational, non-sensitive source material, controlled annotation and generation procedures, and evaluation protocols for CAP and its datasets.

  • Ethics and Data Privacy: The source datasets contain educational question–answer content, with no personal, sensitive, or personally identifiable information reported.The paper states that the material poses no privacy risk.
  • Human Annotation: Human annotation validates CAP-Correctness, CAP-Statements, and generated answers, but each example receives a single label from one annotator.The same two annotators performed all three tasks, and human–LLM agreement rather than inter-annotator agreement is reported.
  • Human Annotation: Annotators received taxonomy guidelines, worked examples, and calibration, and were compensated at three times the regional average pay after providing consent.The annotated material consisted exclusively of educational questions and candidate answers without sensitive or distressing content.
  • CAP-Correctness: CAP-Correctness uses class-conditioned generation to produce an approximately balanced 8,827-example corpus, with no class exceeding approximately 13%.This reduces the risk that downstream metric comparisons are dominated by one correctness category.
  • CAP-Correctness: Human validation of CAP-Correctness yields κ = 0.779 unweighted, κ = 0.825 linearly weighted, and κ = 0.879 quadratically weighted.The weighted estimates indicate substantial agreement and suggest disagreements often involve nearby taxonomy categories.
  • CAP-Statements: CAP-Statements generation is generally reliable overall, with 90.0% of statements labeled correct and 7.5% labeled wrong.Long-question items are substantially harder: 32.9% are labeled wrong and 9.6% semantic, with errors propagating into downstream CAP scores.
  • Evaluation Measures: The monotonicity protocol defines a violation when a less-correct class has a mean metric score at least as high as a more-correct class.CAP’s best setting combines λ = 0.3 neutral weighting with α = 0.85 asymmetric scoring and matches the lowest violation count.

D.2 Monotonicity Analysis

CAP largely preserves the taxonomy’s intended class ordering, outperforming baselines in monotonicity and producing more separated per-class score distributions. Its residual errors concentrate on partial and alternative-correct answers, while overinclusive-valid is specifically pushed too low by asymmetric NLI weighting.

  • Monotonicity results: 4/25 CAP monotonicity violations, versus 9/25 for COMET, with CAP correctly ordering 21 of 25 class pairs.Lexical metrics and BERTScore invert roughly half of the expected orderings.
  • Monotonicity results: CAP forms progressively lower, largely distinct score bands across taxonomy classes, whereas COMET distributions heavily overlap.Only COMET’s exact class separates cleanly; most other classes occupy similar score ranges.
  • Residual errors: CAP’s four inversions include alternative-correct scoring below other top-tier classes because valid alternatives may be mutually non-entailing.The issue is amplified by uneven pretrained NLI world knowledge across CAP-Correctness topics.
  • Label-source stability: CAP remains stable across synthetic and human labels, with higher human-label performance: ρ 69.89 vs. 66.87, τ 57.34 vs. 54.69, and PairAcc 87.16 vs. 84.87.Both label sources recover the same broad class structure, although partial and alternative-correct show the largest shifts.
  • Residual errors: Partial answers score too high because α = 0.85 weights gold-to-partial entailment more than the reverse direction, which correctly detects missing content.This places partial answers near the top group rather than their mid-taxonomy position.
  • Residual errors: Overinclusive-valid answers score too low because asymmetric weighting down-weights the strong answer-to-gold entailment and treats valid extras as mostly unsupported.This reverses the expected ordering between overinclusive-valid and partial classes.
Loading 2609.01369v1…