Semantic Answer Similarity for Evaluating Question Answering Models

1 Introduction

• String-based metrics can score slightly different short answers or longer, more elaborate answers as wrong despite their semantic similarity.Exact match is binary; F1 provides a continuous score, while top-n accuracy checks answer positions in the source context rather than comparing answer strings.Source: §1 · p. 1
• When only one of several semantically equivalent answers is annotated, existing metrics may not reliably evaluate predictions without all correct answers being annotated.For multiple ground-truth answers, the maximum prediction similarity across them is used.Source: §1 · p. 1
• SAS uses a cross-encoder to estimate semantic answer similarity, addressing the limits of metrics based on lexical comparison.The paper also compares SAS with traditional lexical metrics and two metrics developed for semantic textual similarity.Source: §1 · p. 2
• The authors release an annotated dataset and trained model under the Creative Commons Attribution-ShareAlike 4.0 International License.The experiments use two newly created three-way annotated datasets and one dataset from related work.Source: §1 · p. 2

2 Related Work

• To the authors’ knowledge, no dataset had been created specifically for semantic answer similarity, although STS has a benchmark dataset.STS concerns semantic similarity between text inputs; approximate paraphrases convey similar meaning without requiring strict semantic equivalence.Source: §2 · p. 2
• NLG evaluation includes lexical metrics such as BLEU, ROUGE, and METEOR, as well as semantic metrics BERTScore and BLEURT (Gehrmann et al., 2021).The cited GEM benchmark comprises eleven datasets, and BERTScore and BLEURT are BERT-based metrics tailored to NLG.Source: §2 · p. 2
• BLEU, ROUGE, and METEOR rely on lexical overlap, and modified versions for yes-no and entity answers retain those shortcomings (Yang et al., 2018).METEOR uses WordNet and is therefore limited to English; the passage describes BLEU through n-gram overlap and ROUGE through n-gram overlap or longest common subsequence.Source: §2 · pp. 2–3
• BERTScore compares contextual token embeddings, while the authors argue that models should learn which words and phrases matter when comparing answer meaning (Zhang et al., 2020).Chen et al. (2019) applied BERTScore to QA and reported that METEOR correlated more strongly with human judgment.Source: §2 · p. 3
• Other QA evaluation work considers consistency and model properties, inference-based correctness checks, and modifications to string metrics that emphasize relevant content (Ahmad et al., 2019; Chen et al., 2021; Nema and Khapra, 2018).The section also notes reported annotator bias in evaluation and anecdotal cases where lexical metrics score correct answers as wrong (Geva et al., 2019; Ko et al., 2020; Wadhwa et al., 2018).Source: §2 · p. 3

3 Approach

• The study evaluates four approaches: a bi-encoder, cross-encoder SAS, vanilla BERTScore, and trained BERTScore.The methods differ in whether they encode the answers separately, process them together, or compare contextual token embeddings.Source: §3 · p. 4
• The bi-encoder separately encodes each answer and scores the pair with cosine similarity, allowing ground-truth embeddings to be precomputed and reused.The same multilingual model is used across the three datasets and was fine-tuned on English and machine-translated German STS data.Source: §3 · p. 4
• SAS concatenates the two answers with a separator token and processes them together using a cross-encoder rather than separate embeddings.The English model is trained on the STS benchmark; because an English cross-encoder was available but a German one was not, the authors trained a German STS cross-encoder.Source: §3 · p. 4
• Vanilla BERTScore matches contextual token embeddings from task-agnostic models, whereas trained BERTScore uses a multilingual STS model.The vanilla approach uses bert-base-uncased for English and deepset/gelectra-base for German; it extracts embeddings from the second layer, compared with the trained model’s last layer.Source: §3 · p. 4

4 Experiments

• The experiments assess how well automated metrics estimate answer-pair semantic similarity by correlating their scores with human judgments on three datasets.The section covers dataset creation, experiment setup, and results.Source: §4 · p. 4

4.1 Datasets

• The datasets pair answers with three-way similarity labels: dissimilar, approximately similar, or equivalent.The experiments use processed, hand-annotated subsets of SQuAD, GermanQuAD, and NQ-open.Source: §4.1 · pp. 4–5
• The SQuAD subset contains 942 answer pairs, including 566 with no lexical overlap, labeled by two authors with a third resolving disagreements.The labels use the annotators’ majority vote.Source: §4.1 · p. 5
• GermanQuAD contributes 425 pairs after removing questions whose three answers are identical; 137 pairs have no lexical overlap.Pairs were labeled in the same way as the SQuAD subset.Source: §4.1 · p. 5
• NQ-open contributes 3,658 pairs of predictions and ground-truth answers, using existing correctness annotations as similarity labels.The subset filters to questions with exactly one ground-truth answer; 3,118 pairs have no lexical overlap.Source: §4.1 · p. 5

4.2 Results

• Across SQuAD, GermanQuAD, and NQ-open, the cross-encoder consistently has the strongest correlation among semantic metrics, except for slightly lower τ than trained BERTScore on English pairs with no lexical overlap.All four semantic approaches outperform traditional metrics; without lexical overlap, ROUGE-L and METEOR correlate very weakly with human judgment.Source: §4.2 · p. 5, Table 2 · p. 6
• Figure 2 shows that the last layer performs best for trained BERTScore, while the first two layers give vanilla BERTScore its strongest correlations.Layer choice affects trained BERTScore much more than vanilla BERTScore; the cross-encoder has no layer-selection option.Source: §4.2 · p. 6

• SAS uses a cross-encoder and transformer language models pretrained on STS datasets not previously used in QA.Source: §5 · p. 6
• Across three datasets, SAS outperforms four lexical-based and three semantic similarity metrics in correlation with human judgments.Source: §5 · p. 6
• Analyzing answer pairs where SAS differs from human judgment is proposed as a path toward a dataset tailored to training semantic answer similarity models.Source: §5 · p. 6