Source-linked AI summary
Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation
Jannis Bulian, Christian Buck, Wojciech Gajewski, Benjamin Boerschinger, Tal Schuster
TL;DR
QA systems are often evaluated against finite reference-answer sets, limiting coverage and causing token-level EM and F1 to miss valid equivalences. The paper defines asymmetric Answer Equivalence, releases human annotations, and trains BERT matching to approximate it. BEM better reflects human judgments and QA performance, while AE and BEM also support smaller accurate prediction sets.
Problem
Finite reference sets and token-level EM and F1 provide limited coverage for evaluating QA answers and miss important equivalence relations.
Method
The paper defines asymmetric Answer Equivalence, collects human annotations, and trains a BERT matching measure to approximate those judgments.
Results
BEM provides significantly better QA performance assessment than token-level and other similarity measures and generalizes to new QA models and questions.
Takeaways & Limitations
AE and BEM enable more inclusive QA evaluation and practical prediction sets with fewer answers while retaining high-probability coverage of a correct answer.
Takeaways & Limitations
The main experiments focus on machine reading models, while extensions to NQ-Open show only promising initial results.
Abstract
from arXiv · showhide
The predictions of question answering (QA)systems are typically evaluated against manually annotated finite sets of one or more answers. This leads to a coverage limitation that results in underestimating the true performance of systems, and is typically addressed by extending over exact match (EM) with pre-defined rules or with the token-level F1 measure. In this paper, we present the first systematic conceptual and data-driven analysis to examine the shortcomings of token-level equivalence measures. To this end, we define the asymmetric notion of answer equivalence (AE), accepting answers that are equivalent to or improve over the reference, and publish over 23k human judgments for candidates produced by multiple QA systems on SQuAD. Through a careful analysis of this data, we reveal and quantify several concrete limitations of the F1 measure, such as a false impression of graduality, or missing dependence on the question. Since collecting AE annotations for each evaluated model is expensive, we learn a BERT matching (BEM) measure to approximate this task. Being a simpler task than QA, we find BEM to provide significantly better AE approximations than F1, and to more accurately reflect the performance of systems. Finally, we demonstrate the practical utility of AE and BEM on the concrete application of minimal accurate prediction sets, reducing the number of required answers by up to x2.6.
1 Introduction
QA evaluation commonly relies on token-level EM and F1, but these measures miss important answer equivalences and depend heavily on available references. The paper introduces AE annotations and BEM to better assess candidate answers and support practical QA applications.
- Motivation: Token-level EM and F1 can underestimate candidate quality, depend strongly on reference-answer diversity, and fail to distinguish significant from insignificant span differences.The paper frames these limitations as a coverage problem in finite reference sets and motivates evaluation beyond token matching.
- Contributions: The paper defines Answer Equivalence (AE) to capture the desired relation between candidate and reference answers and releases a large human-annotated dataset.The dataset contains annotations comparing gold answers with model predictions on SQuAD examples.
- Analysis: F1 gives a false impression of gradual rating, with non-zero F1 answers showing a similar ratio of equivalent to non-equivalent cases.The analysis also finds that both EM and F1 underestimate answer quality and rely heavily on the number of available references.
- Learned evaluation: BERT matching (BEM) learns to approximate AE and correlates better with human judgments than F1 across experiments with three QA systems.BEM is proposed as a learned equivalence measure rather than another token-matching rule.
- Application: AE and BEM reduce the number of answers needed in prediction sets while retaining arbitrarily high probability of including a correct answer.The paper presents this as a practical application of equivalence-aware QA evaluation.
2 Common token-level metrics: EM & F1
EM and F1 are the dominant token-level QA metrics, but they often misrepresent answer quality because token overlap cannot reliably capture meaningful equivalence.
- Metric definitions: EM tests whether a candidate exactly matches any reference, while Token F1 takes the maximum token-wise F1 against the available references.Stop words and punctuation are commonly removed before either metric is computed.
- Limitations of EM and F1: Both metrics can be too strict: minor surface variations may receive EM and F1 scores of 0.0.The problem is especially pronounced when only a few gold answers are available.
- Limitations of EM and F1: Both metrics also fail at granularity, allowing a completely wrong candidate to receive a relatively high F1 score of 0.67.Token overlap does not distinguish significant from insignificant span differences.
- Limitations of EM and F1: Token matching mishandles equivalent numerical answers expressed in different units, approximate quantities, percentages, absolute values, or spelled-out numbers.Equivalence can hold when the question does not require a specific unit.
- Further limitations: F1 is symmetric, so it cannot reward candidates that add relevant information or remove irrelevant information, even when those changes improve the reference answer.The paper therefore argues that candidate quality can be asymmetric rather than strict equivalence.
- Further limitations: Whether two answers are equivalent can depend on the question, the context, or both, as illustrated by “secondary school” and the metaphorical “Queen Bees.”These cases show why token overlap alone cannot determine answer equivalence.
3 The AE task definition and dataset
The paper defines asymmetric Answer Equivalence (AE) as a question- and context-dependent relation that credits candidates matching or improving on references, then collects human ratings for SQuAD predictions. These annotations expose systematic weaknesses of token-level metrics, including underestimation, reference dependence, and misleading F1 gradation.
- 3.1 Answer Equivalence: AE credits a candidate that preserves at least the reference’s relevant information without adding misleading or excessive content, while allowing relevant improvements.The relation is explicitly asymmetric and conditioned on both the question and context.
- 3.2 Data collection: 14,170 annotations cover 8,565 question-context-reference-candidate tuples from 4,369 non-exact-match SQuAD predictions.Predictions came from held-out Albert models for training data and BiDAF, XLNet, and Luke for development data.
- 3.3 Dataset statistics: 55% of non-exact-match pairs were judged equivalent overall, rising to 69.9% for SQuAD training candidates and varying across QA systems.BiDAF candidates were less often judged equivalent than XLNet or Luke candidates.
- 3.4 Annotation quality: Human ratings show strong agreement: over 88% of multiply annotated examples had full agreement, with Krippendorff’s α = 0.84.A random pair of ratings agreed 92% of the time.
- 3.7 Quantifying limitations of F1: EM and F1 underestimate prediction quality and depend strongly on the number of available references, whose expansion is expensive and laborious.More references make F1 a better estimator of human judgment, but additional annotations have diminishing practical feasibility.
4 Predicting Answer Equivalence
The paper trains BERT Matching (BEM) to approximate human AE judgments because collecting ratings is expensive, especially for open-domain or generative QA. BEM improves over token-level and other similarity metrics, with question input helping and context input degrading performance.
- 4.1 Model: BEM predicts AE for a question, reference answer, and candidate answer using a BERT classifier trained on human AE annotations.The task is simpler than QA because it verifies a candidate against an already given answer.
- 4.1.1 Comparing input variations: Adding the question to the two answers improves classification accuracy, whereas adding the context degrades performance relative to the question-and-answers variant.The authors attribute the context degradation to sparse pertinent-context examples, harder and noisier assessments, and irrelevant input length.
- 4.1.2 Model performance: BEM achieves high accuracy and correlation with human equivalence ratings, significantly improving over F1, EM, BERTScore, BLEURT, and LERC baselines.The gain in accuracy is consistent across QA systems.
- 4.2 AE for QA performance evaluation: BEM-based accuracy is closest to true model performance, while EM is significantly lower and F1 remains about 2–3 points below the human-derived score.The comparison uses predictions from Luke, XLNet, and BiDAF on the SQuAD development set.
- 4.2 AE for QA performance evaluation: BEM is more robust to reference count than F1 and remains much closer to human judgment even with a single reference.This robustness can improve distinctions between closely ranked systems when few gold references are available.
- 4.3 Generalization: On 300 non-exact-match NQ-Open predictions, BEM agreed with an independent manual assessment in 87% of examples.This provides evidence of generalization beyond SQuAD models and to a more challenging dataset.
5 Example application: Returning small and accurate prediction sets
The paper applies expanded admission conformal prediction to construct smaller prediction sets that retain user-specified accuracy. AE labels substantially reduce set size, while BEM provides similarly effective approximate calibration when AE labels are unavailable.
- Motivation: Smaller prediction sets are practically useful because they retain accuracy while reducing the number of candidate answers that must be returned.The application measures accuracy by whether a correct answer appears in the predicted set and calibrates set size to the desired accuracy.
- Conformal prediction framework: Conformal prediction dynamically determines each set’s size through instance-wise hypothesis testing while marginally satisfying the user-specified target accuracy.Calibration uses nonconformity scores from an exchangeable set of question–correct-answer pairs, with negative predicted probability as the score.
- Expanded admission: Expanded admission supports multiple answers by using answer equivalence to reduce conformal prediction-set size while preserving the accuracy guarantee.This extends the original single-label conformal prediction setting.
- Expanded admission with AE: 11.31 to 4.31 answers per question at 90% accuracy when calibration changes from SQuAD labels to AE labels.All methods empirically meet the target accuracy, while AE identifies higher-ranked correct answers for threshold calibration.
- Approximate admission with BEM: BEM is as effective as AE labels for reducing required answers across target accuracy levels when approximate admission is needed.BEM predictions are used after false-positive-rate correction, and Figure 2 reports comparable efficiency gains over the full target-accuracy range.
6 Related Work
Related work spans answer-equivalence annotation, learned evaluation metrics, text-similarity measures, and manual evaluation of answers outside gold sets. This paper differs by targeting asymmetric equivalence for extractive QA and by focusing on answers equivalent to a gold reference.
- Answer equivalence: The paper targets asymmetric answer equivalence conditioned on the answer, unlike Risch et al.’s symmetric string-similarity task.Risch et al. annotate answer pairs and train a classifier over concatenated strings.
- Related QA evaluation: MOCHA also learns an answer metric, but it addresses generative QA with different candidate collection, datasets, and rating procedures.Other work uses natural language inference to verify QA predictions, potentially enabling comparison with gold answers.
- Text similarity: Token F1 is established for QA evaluation but fails to account for the greater surface-form variability of generative QA.Other text-comparison metrics, including BLEU, ROUGE, and METEOR, have been tried but are not widely used for QA.
- Text similarity: Prior work adapts lexical metrics or expands entity aliases to improve answer comparison, but often focuses on restricted question types or exact-match reliability.Yang et al. address yes-no and entity questions, while Si et al. add Freebase aliases to gold answers.
- Manual evaluation: Unlike the EfficientQA competition, which rewards correct answers absent from gold answers, this paper rewards only answers equivalent to a gold answer.The distinction concerns the scope of accepted answers rather than the use of manual annotation itself.
- Learned metrics: Learned metrics such as BLEURT and COMET have improved correlation with human judgments in other NLG tasks, while prior QA metrics have not surpassed F1 for extractive QA.Chen et al.’s question- and context-conditioned BERTScore variant did not improve over F1.
7 Conclusion
The paper analyzes shortcomings of EM and F1, introduces answer-equivalence judgments and the BEM metric, and demonstrates their value for QA evaluation and practical prediction-set construction. It releases data intended to support further development of QA metrics and usability.
- Contribution: The paper provides a systematic data-driven analysis of EM and F1 shortcomings when QA systems are evaluated against reference-answer sets.This analysis forms the paper’s central evaluation contribution.
- Contribution: AE directly captures the desired candidate–reference relation, supported by extensive annotations for evaluation, model training, and QA-system assessment.The annotations are designed to evaluate and train equivalence models while measuring system performance.
- Results: BEM generalizes to new QA models and evaluation questions and provides significantly better system-performance assessment than token-level and other similarity measures.The paper also demonstrates AE’s value in an application beyond direct QA performance assessment.
- Broader impact: The released data and experiments are intended to support further development of better metrics and improve QA-system evaluation and usability.This is presented as the paper’s anticipated broader contribution.
Ethical Considerations
The study reports participant-protection procedures and limits its empirical scope to English data and experiments. It does not claim that the findings generalize across languages.
- Data protection: All annotators provided informed consent, no personally identifiable information was collected or released, and the data were screened for PII and offensive language.Screening used heuristics and random sampling.
- Annotation procedure: Annotators received fair local-market compensation without speed- or accuracy-based incentives intended to avoid distorting motivation.The paper connects intrinsic motivation with higher-quality results through Gneezy and Rustichini (2000).
- Scope limitation: Because the released data and experiments are in English, the paper does not claim cross-language generalization.The authors suggest applying the methods in other languages using suitable corpora, but do not establish that transfer empirically.
Limitations
The framework is limited by its focus on short, primarily machine-reading QA answers and by settings where context, language, task format, or temporal and spatial metadata complicate equivalence.
- Scope: The approach focuses on short answers and is not directly applicable to long-answer datasets, although BEM could support aggregated comparison methods.The authors suggest summarizing long answers with a machine reading model before comparing the summaries.
- Context: The final BEM excludes context, which improves portability to context-unavailable settings but may reduce AE performance in domains where context helps.
- Language: The work studies English datasets, while multilingual equivalence may require more than translation because semantics can differ across languages.
- Task coverage: Experiments mainly target machine reading models, with only promising initial extensions to NQ-Open.
- Generative QA: Generative QA models may add unsupported details or hallucinations, and BEM trained on extracted answers generally accepts such specific answers as equivalent.
- Question setting: Temporal or spatial answer equivalence can depend on when and where an answer is given, while conversational QA may require different question and context processing.
B BEM training details
BEM is trained by fine-tuning a published BERT-Base checkpoint on the collected training examples with fixed training settings and no hyperparameter search.
- Training setup: BEM fine-tunes the published BERT-Base checkpoint for one epoch using a JAX-based BERT implementation.
- Optimization: The training uses batch size 64, learning rate 1e-4, and the Adam optimizer without searching for optimal hyperparameters.
- Compute: Training on a TPU v2 takes less than 5 minutes.
C Conformal prediction sets experimentation details
The experiments evaluate conformal prediction sets under different answer-equivalence terms, measuring whether sets meet target accuracy while minimizing their size.
- Method: The calibration experiments use expanded admission conformal prediction, which leverages equally correct answers to improve calibration efficiency.
- Experimental setup: The setup uses the top 20 Luke predictions per SQuAD question, 50 calibration trials, and average results with 16th- and 84th-percentile visualizations.
- Evaluation goal: Prediction sets are evaluated for marginal validity at a user-defined target accuracy while minimizing the number of included answers.
- Equivalence conditions: Exact admission compares original SQuAD labels with AE annotations, whereas approximate admission compares F1 with BEM and corrects approximation errors statistically.