Source-linked AI summary
Towards a Better Metric for Evaluating Question Generation Systems
Preksha Nema, Mitesh M. Khapra
TL;DR
AQG systems are increasingly used to generate questions from documents, knowledge bases, and images, but it is important to assess whether their evaluation metrics capture answerability as well as textual similarity. The paper evaluates existing metrics against human judgments and proposes adding a learnable answerability component. The modified metrics correlate better with human judgments, though the authors note scope for improvement, especially for document QA and visual QA.
Problem
Existing n-gram metrics are used to evaluate AQG, but whether they account for answerability—having relevant information such as question type, entities, and relations—needs examination.
Source: §1 · pp. 1–2Method
The authors collect human answerability judgments on systematically noised questions across document, knowledge-base, and visual QA, then propose adding task-adjustable, learnable answerability weights to existing metrics.
Source: §1 · p. 2Results
Takeaways & Limitations
The proposed metric weights can be tuned by source using released human judgments, and the authors propose extrinsic evaluation of metrics for selecting AQG systems for QA training data.
Source: §1 · p. 2, §7 · p. 9The modified metrics still leave scope for improvement, especially for document QA and visual QA.
Source: §7 · p. 9
Abstract
from arXiv · showhide
There has always been criticism for using $n$-gram based similarity metrics, such as BLEU, NIST, etc, for evaluating the performance of NLG systems. However, these metrics continue to remain popular and are recently being used for evaluating the performance of systems which automatically generate questions from documents, knowledge graphs, images, etc. Given the rising interest in such automatic question generation (AQG) systems, it is important to objectively examine whether these metrics are suitable for this task. In particular, it is important to verify whether such metrics used for evaluating AQG systems focus on answerability of the generated question by preferring questions which contain all relevant information such as question type (Wh-types), entities, relations, etc. In this work, we show that current automatic evaluation metrics based on $n$-gram similarity do not always correlate well with human judgments about answerability of a question. To alleviate this problem and as a first step towards better evaluation metrics for AQG, we introduce a scoring function to capture answerability and show that when this scoring function is integrated with existing metrics, they correlate significantly better with human judgments. The scripts and data developed as a part of this work are made publicly available at https://github.com/PrekshaNema25/Answerability-Metric
1 Introduction
The introduction motivates automatic question generation (AQG) as a way to create more QA training data, then questions whether n-gram metrics adequately evaluate generated questions. It argues that evaluation should account for answerability and the task-dependent importance of question components.
1 Introduction
- AQG generates questions from documents, knowledge bases, or images to create additional training data for QA systems.The introduction notes that creating or augmenting domain-specific datasets can be tedious, time-consuming, and expensive.Source: §1 · p. 1
- A question can have a high BLEU score yet remain unanswerable when it lacks relevant information such as question type, entities, or relations.The introduction identifies answerability as a dimension for evaluating AQG beyond n-gram similarity.Source: §1 · p. 2
- 81.9 BLEU3 for “Who was the director of?” exceeds 36.8 BLEU3 for “director of Titanic?”, although the latter contains the relevant information and is more answerable.The example illustrates how n-gram similarity can reward a question that omits important content.Source: §1 · p. 2
- Which question components matter can vary by source: question words can distinguish image questions, while a document title may supply a missing entity.The introduction gives examples involving cats drinking milk and a passage titled “Matt Damon.”Source: §1 · p. 2
- The proposed metric combines answerability with n-gram similarity, using adjustable weights for question type, content words, function words, and named entities.The weights may be learned from a small amount of human-annotated data and may differ across tasks.Source: §1 · p. 2
2 Related Work
The related-work survey covers question-generation systems and studies of NLG evaluation metrics. It reviews varied approaches to question generation alongside criticism of n-gram-based evaluation.
2 Related Work
- The survey organizes prior work into question-generation systems and analyses of NLG evaluation metrics..Source: §2 · p. 2
- Question-generation research spans rule-based and neural text approaches, QA-coupled models, and systems generating questions from images or knowledge bases.Some work uses QA performance as a question-quality indicator or jointly trains QA and question generation.Source: §2 · pp. 2–3
- NLG metrics such as BLEU, METEOR, ROUGE, and NIST primarily measure n-gram similarity between generated and reference sentences.The passage notes criticism of these metrics across NLG and dialog evaluation, and argues for suitable AQG metrics before systems are compared using them.Source: §2 · p. 3
3 Current Evaluation Metrics
This section outlines BLEU, METEOR, NIST, and ROUGE, which use different variants of precision, recall, matching, and sequence overlap. Their shared purpose is to compare generated text with a reference.
3 Current Evaluation Metrics
- BLEU combines exact n-gram precisions across n with a geometric mean and applies a brevity penalty to hypotheses that are too short.For each n, precision is the fraction of hypothesis n-grams that match an n-gram in the reference.Source: §3 · p. 3
- METEOR combines precision and recall using exact, stemmed, synonym, and paraphrase matches, with a fragmentation penalty for gaps and word-order differences.Its parameters, including weights for match types and function versus content words, are tuned to maximize correlation with human judgments.Source: §3 · p. 3
- NIST weights less frequent n-grams more heavily and averages n-gram precisions arithmetically rather than geometrically.It also modifies the brevity penalty to reduce the impact of small hypothesis-length differences.Source: §3 · p. 3
- ROUGE-L uses an F-measure based on the longest common subsequence, which preserves word order but does not require contiguous matches.For example, “the boy home” is the longest common subsequence of “the boy went home” and “the boy will go home.”Source: §3 · p. 3
4 Human Judgments For Answerability
The study collected human answerability judgments for systematically altered questions from three QA sources and compared them with existing metrics. Human agreement was moderate or strong depending on the dataset, while metric correlations varied across sources.
4.1 Creating Noisy Questions
- The study used 1,000 questions each from SQuAD, WikiMovies, and VQA, then created noisy versions by dropping function words, named entities, or content words, or changing question type.Content words typically capture relations between entities.Source: §4.1 · p. 4
- The researchers chose simulated noisy questions because annotators found existing AQG outputs too poor for systematically assessing the importance of different words.They had first asked evaluators to analyze 200 questions generated by an existing system.Source: §4.1 · p. 4
4.2 Instructions
- Annotators rated answerability on a 1–5 scale without seeing the source, because source context could let them infer missing information.Showing the image or document led annotators to mark some questions answerable even when relevant information was missing from the question.Source: §4.2 · p. 4
4.3 Human-Human Correlation
- Human ratings showed strong inter-annotator agreement for WikiMovies and moderate agreement for SQuAD and VQA.The study used Pearson correlation for the linear relationship between ratings and Spearman correlation for monotonic association; Spearman was slightly lower because agreement was stronger at ratings 1 and 5.Source: §4.3 · p. 5
4.4 Correlation between human scores and existing evaluation metrics
- For SQuAD and VQA, NIST correlated better with human scores than other metrics; for WikiMovies, METEOR did.For SQuAD and WikiMovies, BLEU1 correlation was higher than that of other metrics, and humans could answer some questions that were not well formed.Source: §4.4 · pp. 5–6
5 Modifying existing metrics for AQG
The proposed Q-Metrics augment n-gram similarity with a weighted answerability score based on question elements. Their weights are tuned from human judgments, and the resulting metrics correlate better with human judgments than existing metrics, though improvement remains possible.
5 Modifying existing metrics for AQG
- The answerability score weights precision and recall across relevant content words, named entities, question types, and function words.These weighted components can be combined with an existing metric, such as BLEU4, to form Q-Metrics including Q-NIST and Q-METEOR.Source: §5 · p. 6
5.1 Tuning the weights wi’s and δ
- Weights tuned separately for each source were stable across repeated samples, and their relative importance varied by dataset.Named entities had the highest weight for WikiMovies, while content words were most important for VQA; the base metric also received comparable weight for SQuAD and VQA.Source: §5.1 · pp. 6–7
5.2 Correlation between Human scores and different Q-METRICs
- Modified Q-Metrics correlate much more highly with human judgments than existing metrics on held-out noisy questions.The reported correlations are statistically significant; the authors note that VQA and SQuAD correlations remain below human-human correlations.Source: §5.2 · p. 7, Table 6 · p. 7
5.3 Qualitative Analysis
- Positive qualitative examples show Q-BLEU scores close to human answerability scores.The examples are categorized by whether human and Q-BLEU answerability scores are similar.Source: §5.3 · p. 7
6 Extrinsic evaluation
The extrinsic evaluation tests whether metrics for noisy question training data align with the downstream performance of QA models. In general, model performance aligns better with Q-BLEU4 than with BLEU4.
6 Extrinsic evaluation
- The evaluation trains task-specific QA models on noisy SQuAD, WikiMovies, and VQA training data, then tests them on the original datasets’ test sets.The models are Seo et al. (2016) for SQuAD, Miller et al. (2016) for WikiMovies, and Ben-younes et al. (2017) for VQA.Source: §6 · p. 8
- Model performance generally aligns better with Q-BLEU4 than BLEU4 across the noisy-question training comparisons.The authors note that BLEU4 gives substantial importance to stop words, which have little influence on final model performance.Source: §6 · p. 8, Table 8 · p. 8, Table 9 · p. 8
- The comparison considers how noisy-question quality scores under BLEU4 and Q-BLEU4 relate to downstream model performance.The tables organize results by the manner in which noisy training data was created.Source: §6 · p. 8
7 Conclusion
The paper finds that existing n-gram metrics do not align well with human judgments of answerability across document, knowledge-base, and visual QA. It proposes source-tunable learnable weights and reports better correlation, while noting scope for further improvement.
7 Conclusion
- Across document, knowledge-base, and visual QA, human answerability scores did not correlate well with existing metrics, while modified metrics correlated better.The modification uses learnable weights tuned by source using human judgments.Source: §7 · pp. 8–9
- The authors identify further improvement, especially for document QA and visual QA, as an open need.They propose exploring better answerability metrics and nonlinear combinations of Q-Metric components.Source: §7 · p. 9