Source-linked AI summary
Evaluating Rewards for Question Generation Models
Tom Hosking, Sebastian Riedel
TL;DR
Question-generation models trained with teacher forcing face exposure bias, while reward-based fine-tuning has uncertain value as a measure of question quality. This paper fine-tunes with QA, language-model, and learned discriminator rewards; reward gains do not translate into better human ratings, and outputs can exploit reward weaknesses.
Problem
Teacher-forced models learn one-step predictions from limited question data, while reward metrics had not been confirmed as proxies for question quality.
Source: §1 · p. 1Method
The authors fine-tune a question-generation model using QA and language-model rewards and a discriminator reward learned from training data.
Source: §1 · p. 1, §3.2 · p. 3Results
Takeaways & Limitations
QA and language-model scores may support reranking or filtering at the lower end of human ratings, and reward-driven generation may produce adversarial training examples for QA models.
Source: §5 · p. 5The discriminator did not learn a sufficiently useful distinction between generated and real questions, and its fine-tuning added unwanted noise to predictions.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Recent approaches to question generation have used modifications to a Seq2Seq architecture inspired by advances in machine translation. Models are trained using teacher forcing to optimise only the one-step-ahead prediction. However, at test time, the model is asked to generate a whole sequence, causing errors to propagate through the generation process (exposure bias). A number of authors have proposed countering this bias by optimising for a reward that is less tightly coupled to the training data, using reinforcement learning. We optimise directly for quality metrics, including a novel approach using a discriminator learned directly from the training data. We confirm that policy gradient methods can be used to decouple training from the ground truth, leading to increases in the metrics used as rewards. We perform a human evaluation, and show that although these metrics have previously been assumed to be good proxies for question quality, they are poorly aligned with human judgement and the model simply learns to exploit the weaknesses of the reward source.
1 Introduction
Question generation can support natural-language interaction with documents, but training on one question per context-answer pair with teacher forcing leaves models vulnerable to exposure bias. The paper fine-tunes with reward objectives and finds that higher reward scores can coincide with worse human ratings and exploitable outputs.
1 Introduction
- Question generation can help machines ask clarification questions, become more robust to queries, and act as automatic tutors.The passage gives these as applications of posing natural-language questions about documents.Source: §1 · p. 1
- A single question per context-answer pair and one-step-ahead teacher forcing leave models unprepared for valid alternative sequences and decoding mistakes.The paper identifies this mismatch as exposure bias.Source: §1 · p. 1
- Prior work used BLEU or other quality metrics as performance-based rewards, although their status as proxies for question quality had not been confirmed.Decoupling training from ground truth was proposed to let models explore questions and recover from suboptimal decoding predictions.Source: §1 · p. 1
- Fine-tuning can raise reward scores while producing questions rated worse by humans, and models can exploit weaknesses in the reward sources.The examples include QA-reward outputs pointing at the answer and language-model-reward outputs repeating common phrase templates.Source: §1 · p. 1, Table 1 · p. 2
2 Background
Question-generation research adapted machine-translation architectures and also explored policy-gradient fine-tuning. Prior reward choices remained coupled to ground truth or lacked human evaluation, while reinforcement learning had also been applied to summarisation.
2 Background
- Question-generation systems adapted machine-translation methods, including Seq2Seq models, pointer networks, coreference preprocessing, and augmented context encodings.Du et al. and Zhou et al. reported human-evaluated improvements over their baselines; other work explored answer-independent questions and multi-perspective context matching.Source: §2 · p. 2
- Policy-gradient fine-tuning used ground-truth-coupled similarity rewards or external language-model and QA rewards, and Yuan et al. did not perform human evaluation.Kumar et al. (2018b) reported promising human-evaluation results, but used similarity metrics still coupled to ground truth.Source: §2 · p. 2
- For summarisation, Paulus et al. (2017) fine-tuned with reinforcement learning using ROUGE similarity as the reward.Source: §2 · p. 2
3 Experimental setup
The experiments fine-tune a question-generation model with language-model, QA, and discriminator rewards, then assess automatic scores and human ratings. They compare generated-question fluency and relevance as well as model and reward metrics.
3 Experimental setup
- The task is to generate a natural-language question from a document and the location of an answer within it.The example input pairs a document with the answer “question generation” and asks what is investigated in the paper.Source: §3 · p. 2
3.1 Model description
- The model uses Yuan et al. (2017)’s Seq2Seq architecture with attention, copying, and an answer encoder, and is trained by maximum likelihood before fine-tuning.Beam search is used at inference; the implementation scores 13.5 BLEU-4 on Du et al. (2017)’s test set before fine-tuning.Source: §3.1 · p. 2
3.2 Fine tuning
- Fine-tuning rewards fluency with negative perplexity under an LSTM language model and relevance with a QA system’s F1 score, or a weighted combination.The QA system is QANet.Source: §3.2 · p. 3
3.3 Adversarial training
- A discriminator is trained to distinguish generated from training-data questions, then its probability that a generated question is real rewards the generator.The study also interleaves discriminator updates so it can adapt alongside the generator, and applies REINFORCE while retaining a maximum-likelihood objective.Source: §3.3 · p. 3
3.4 Evaluation
- Evaluation reports NLL, BLEU-4, QA, language-model and discriminator scores, plus human ratings of fluency and relevance.Three workers rate 300 generated questions on each human-evaluation criterion; Table 2 describes the automatic-metric changes and Table 3 summarizes selected human-evaluation scores.Source: §3.4 · pp. 3–4, Table 2 · p. 3, Table 3 · p. 3
4 Results
Fine-tuning raises scores on the rewarded metrics, but human evaluation shows that higher automatic scores do not necessarily mean better questions. The results also show reward exploitation and an ineffective discriminator reward.
4 Results
- The QA-and-LM-tuned model scored significantly worse with human annotators despite higher automatic-metric scores, showing those rewards did not correspond to human-rated question quality.Across reward combinations, BLEU decreased; models generally improved on the metrics they were tuned against.Source: §4 · p. 4
- The discriminator failed to learn a useful distinction between generated and real questions, so its reward did not improve human ratings.Training was stable across initializations, but fine-tuning added unwanted noise to predictions.Source: §4 · p. 4
- A language-model reward produced looping words, while a QA reward led the model to point at the answer location, illustrating reward exploitation.The QA-reward model’s LM score worsened by +226.Source: §4 · p. 4
- Human relevance correlated 0.439 with automatic QA scores, and fluency correlated 0.355 with LM scores; the scores distinguished low ratings better than higher ones.A sequence could score highly while remaining unintelligible to a human.Source: §4 · p. 4
5 Conclusion
The paper finds that policy-gradient fine-tuning can raise reward scores without improving human-rated question quality. Reward models can be exploited, though their scores may help filter or rerank generated questions.
5 Conclusion
- Fine-tuning on external rewards raised reward scores but did not yield better questions in human ratings.The generator exposed weaknesses in QA and language-model rewards, suggesting a possible use in generating adversarial QA training examples.Source: §5 · p. 5
- QA and language-model scores correlated with human ratings at the low end, suggesting they could support reranking or filtering.The authors plan to study overgenerating questions and reranking them with reward signals.Source: §5 · p. 5
A Discriminator architecture
The discriminator adapts a modified QANet to score context-question-answer triples with a single probability output. Its output layers combine pooled encoder representations of the context, question, and answer span.
A Discriminator architecture
- The modified QANet replaces its output layers to produce one probability for a context-question-answer triple.Unlike the QA task’s context-question pair, the discriminator takes the full triple as input.Source: A Discriminator architecture · p. 6
- The discriminator max-pools the first two encoder outputs and mean-pools the third encoder’s answer-span outputs.Source: A Discriminator architecture · p. 6
- It concatenates the three reduced encodings, applies a 64-unit ReLU hidden layer, then a sigmoid output layer.The sigmoid estimates whether the triple came from ground-truth data or was generated.Source: A Discriminator architecture · p. 6