Source-linked AI summary
Ask the Right Questions: Active Question Reformulation with Reinforcement Learning
Christian Buck, Jannis Bulian, Massimiliano Ciaramita, Wojciech Gajewski, Andrea Gesmundo, Neil Houlsby, Wei Wang
TL;DR
Complex questions can require reformulation and evidence aggregation, but conventional QA systems are treated as fixed environments. AQA trains an agent with policy gradient to probe a black-box QA system using multiple natural-language reformulations and select the best answer. On SearchQA, it outperforms BiDAF and heuristic benchmarks, while learning interpretable strategies unlike ordinary paraphrases.
Problem
Complex question answering may require multiple reformulations and evidence aggregation, but the QA system is treated as a black box that the agent must probe through question strings.
Method
AQA combines a reinforcement-learning sequence-to-sequence reformulator, a black-box QA environment, and an answer selector trained to aggregate returned evidence.
Results
11.4% absolute F1 improvement over BiDAF on SearchQA, with additional gains over competitive heuristic query-reformulation benchmarks.
Takeaways & Limitations
Successful reformulations diverge from natural-language paraphrases and learn interpretable operations resembling term re-weighting, tf-idf, and stemming.
Takeaways & Limitations
AQA can converge to sub-optimal deterministic or minimally changed policies, while excessive entropy regularization can produce random policies.
Abstract
from arXiv · showhide
We frame Question Answering (QA) as a Reinforcement Learning task, an approach that we call Active Question Answering. We propose an agent that sits between the user and a black box QA system and learns to reformulate questions to elicit the best possible answers. The agent probes the system with, potentially many, natural language reformulations of an initial question and aggregates the returned evidence to yield the best answer. The reformulation system is trained end-to-end to maximize answer quality using policy gradient. We evaluate on SearchQA, a dataset of complex questions extracted from Jeopardy!. The agent outperforms a state-of-the-art base model, playing the role of the environment, and other benchmarks. We also analyze the language that the agent has learned while interacting with the question answering system. We find that successful question reformulations look quite different from natural language paraphrases. The agent is able to discover non-trivial reformulation strategies that resemble classic information retrieval techniques such as term re-weighting (tf-idf) and stemming.
1 INTRODUCTION
AQA addresses complex question answering by learning to reformulate questions, query a black-box QA system repeatedly, and aggregate evidence. On SearchQA, it improves substantially over BiDAF and heuristic reformulation benchmarks while learning non-paraphrastic information-retrieval strategies.
- Complex questions often require multiple search iterations, critical assessment, and synthesis rather than simple factual retrieval.
- AQA places a reinforcement-learning agent between users and a black-box QA system to generate reformulations and aggregate returned evidence.The agent uses question strings alone to probe the environment.
- 11.4% absolute F1 improvement over BiDAF demonstrates AQA’s advantage on SearchQA.This corresponds to a 32% relative F1 improvement; AQA also outperforms competitive heuristic query-reformulation benchmarks.
- AQA reformulations diverge significantly from natural-language paraphrases while discovering term re-weighting and morphological simplification resembling tf-idf and stemming.
2 RELATED WORK
Related work connects AQA to paraphrase generation, reinforcement learning for language and QA, graph-based question answering, and semantic query perturbation. AQA differs by perturbing questions directly at the natural-language surface to improve interaction with a QA environment.
- Prior QA research has used dependency patterns, machine-translation paraphrases, latent paraphrase augmentation, and multilingual pivoting to generate question variants.
- Reinforcement learning has been applied to textual game control, dialogue generation, machine translation, semantic parsing, and query reduction for multi-hop reasoning.
- Graph-based QA learns to navigate linked documents toward answers, whereas AQA learns question reformulations for a black-box QA system.
- Active QA resembles fact-checking methods that perturb queries, but it perturbs semantic questions directly in surface natural language.
3 ACTIVE QUESTION ANSWERING MODEL
The AQA model repeatedly reformulates an input question, sends the variants to an opaque QA environment, and selects the best returned answer. Its reformulator is a pretrained sequence-to-sequence model adapted through environment rewards, while a selector ranks observed answers.
- AQA starts with an original question, generates reformulations, queries the environment with them, and selects the best candidate answer.
- The agent-environment setup uses a downward pass for reformulation and querying, followed by an upward pass that selects the final answer.
- BiDAF is an opaque extractive QA environment that returns answers and token-level F1 rewards, while withholding parameters, activations, and gradients from the agent.Rewards are computed against the original question, and unavailable gradients make reformulation feedback noisy.
- The reformulator is a same-language sequence-to-sequence model pretrained on multilingual translation and adapted using interaction signals from the QA environment.
- At test time, the selector chooses among observed answers by predicting each answer’s F1 difference from the variants’ average F1.Its inputs include query, rewrite, and answer representations processed with one-dimensional CNNs and max-pooling.
4 TRAINING
The reformulation policy generates questions from an initial question and is optimized with policy gradients using rewards from an opaque QA environment. Training combines multilingual translation pre-training with reinforcement-learning signals, while entropy regularization addresses policy collapse.
- Environment: The QA environment is fixed after training, and the agent learns to communicate with it without access to its parameters, activations, or gradients.The environment is BiDAF, whose parameters are not updated during agent training.
- Policy objective: The reformulation policy generates questions token by token and maximizes the expected reward of the answer returned by the environment.The reward is computed against the original question, while the answer is produced for the reformulated question.
- Policy optimization: Because the expected reward has no closed-form computation, training estimates it with Monte Carlo samples and uses REINFORCE for gradient computation.The estimator’s variance is reduced with a baseline reward estimated by sampling from the policy.
- Policy optimization: Entropy regularization is introduced to counter observed collapse onto a sub-optimal deterministic policy.The regularization weight is denoted by λ.
- Pre-training: The policy is pre-trained with multilingual translation and then adapted using interaction signals because high-quality English-English paraphrase data is scarce.The multilingual model enables zero-shot English-English translation through language tokens, providing the initial reformulation model.
5 EXPERIMENTS
Experiments evaluate AQA on SearchQA with BiDAF and multiple reformulation baselines, using candidate generation and answer selection variants. The full AQA system improves substantially over BiDAF, while the results also quantify dataset difficulty, training behavior, and available rewrite headroom.
- Dataset and setup: SearchQA contains over 140k question/answer pairs and 6.9M snippets from Jeopardy! clues, with predefined training, validation, and test splits.The experiments use the training split for model development and report validation and test performance.
- Dataset and setup: BiDAF uses the top 10 snippets as context, leaving no answer in that shorter context for 10% of questions.Those questions are counted as losses.
- Training behavior: Using the SearchQA training data three times produces only a small additional generalization gap, rising from 3.4 F1 for BiDAF to 3.9 F1 for AQA-Full.The authors conclude that this reuse causes very little additional overfitting.
- Compared systems: AQA is compared with ASR, BiDAF without reformulation, MI-SubQuery, and Base-NMT, alongside TopHyp, Voting, MaxConf, and CNN selection variants.The AQA experiments generate N = 20 reformulations per query, matching the benchmark setting.
- Results: SearchQA is difficult: BiDAF drops by 40 F1 points relative to SQuAD but improves over ASR by 13.7 F1 points.The reported metrics are evaluated on the full validation and test sets using token-level exact match and F1.
- Results: 11.4 F1 points, or 32% relative, is the full AQA improvement over BiDAF on the original questions.AQA TopHyp alone improves test F1 by 2.2, while learned CNN selection provides the larger overall gain; differences versus Base-NMT and MI-SubQuery are statistically significant at p < 10^-4.
- Results: Oracle performance reaches near 50 Exact Match and close to 58 F1 across the available reformulation sources.The result measures the highest answer F1 obtainable by selecting among answers returned for all reformulations.
6 ANALYSIS OF THE AGENT’S LANGUAGE
The agent’s reinforcement-learned reformulations differ from fluent paraphrases and exploit query statistics, repetition, morphology, and QA-system-specific mechanisms. Despite sub-optimal policies and lower linguistic fluency, AQA reformulations improve QA performance and resemble information-retrieval strategies.
- Policy behavior: AQA can converge to meaningless deterministic reformulations or minimal edits, while entropy regularization can prevent deterministic behavior but excessive regularization may produce random policies.Minimal edits can remain competitive because the environment was trained only on original questions.
- Input and rewrite statistics: SearchQA inputs average 9.6 words, have mean term frequency 1.03, and are evaluated using document frequency and query clarity.Document frequency measures term informativeness in the answer context; query clarity is based on relative entropy between query and collection language models.
- Paraphrasing quality: Base-NMT rewrites average 6.3 words, are mostly well-formed questions, and are 50% more likely under a language model than the original questions.They reinsert function words and wh-phrases, but their lower document-frequency terms and degraded query clarity reflect transduction effects and domain mismatch.
- AQA reformulations: AQA-QR rewrites alone outperform original SearchQA queries by 2% on the test set, with 99.8% beginning with “What is name.”These rewrites are analyzed separately from answer selection and differ from both SearchQA and Base-NMT reformulations.
- AQA reformulation properties: AQA-QR rewrites are less fluent but contain more repeated terms, average 11.9 words, and include more informative context terms than SearchQA questions.Their average term frequency is 1.2, and their reformulations also modify surface forms through morphological operations.
- Discussion: AQA learns to re-weight informative and query-specific terms, duplicate terms, and alter surface forms in ways resembling tf-idf weighting, stemming, and morphological analysis.Some strategies exploit character-based encoding and attention: novel variants can remain usable, while repetition can increase alignment chances.
7 CONCLUSION
The paper presents active question answering as a framework in which an agent reformulates questions and aggregates answers from a black-box QA system. Future work targets sequential, iterative information seeking as an end-to-end reinforcement-learning problem.
- Active question answering systematically perturbs input questions to improve question answering.
- The proposed system combines a question reformulator, a black-box QA system, and a candidate answer aggregator.
- Future work will investigate sequential, iterative information seeking and connect reformulation with selection through end-to-end reinforcement learning.
A REFORMULATION EXAMPLES
Table 2 presents qualitative SearchQA reformulation examples, pairing original Jeopardy! questions with reference answers and other questions with BiDAF answers.
- Table 2 reports qualitative analysis results on SearchQA.
- For original Jeopardy! questions, the table provides the reference answer.
- For other questions, the table provides the answer produced by BiDAF.
B EXAMPLES OF RANKING LOSSES
Table 3 gives examples where none of the methods returns the correct answer, while an Oracle model can.
- Table 3 lists queries that none of the methods answers correctly.
- The examples concern cases where an Oracle model can produce the right answer.
- The table contrasts ordinary method outputs with Oracle-model performance on these queries.
C PARAPHRASING EXAMPLES
Table 4 presents paraphrasing examples using MSCOCO image captions.
- Table 4 contains paraphrasing examples from MSCOCO captions.
- The examples use captions from the MSCOCO dataset.
- The table provides concrete instances for examining paraphrasing behavior on captions.