PaperlayerView exact revision on arXiv

Source-linked AI summary

A Discrete Hard EM Approach for Weakly Supervised Question Answering

Sewon Min, Danqi Chen, Hannaneh Hajishirzi, Luke Zettlemoyer

arXiv:1909.04849v1cs.CLcs.AI

TL;DR

Weakly supervised QA leaves ambiguity about which mention or derivation yields the answer. The paper uses precomputed candidate sets with hard EM, achieving 2–10% absolute gains across six tasks and state-of-the-art results on five datasets.

  • Problem

    Weak supervision makes QA data easier to gather but leaves many spurious answer mentions or derivations, complicating learning.

  • Method

    The paper precomputes task-specific solution sets and trains a discrete latent-variable QA model with hard-EM updates selecting and reinforcing the most likely solution.

  • Results

    2–10% absolute gains were achieved across six QA tasks, with state-of-the-art results on five datasets.

  • Takeaways & Limitations

    Hard updates encourage models to assign much higher probability to the correct solution.

  • Takeaways & Limitations

    The approach assumes a precomputed solution set contains one desired solution, potentially alongside spurious options.

Abstract

from arXiv · show

Many question answering (QA) tasks only provide weak supervision for how the answer should be computed. For example, TriviaQA answers are entities that can be mentioned multiple times in supporting documents, while DROP answers can be computed by deriving many different equations from numbers in the reference text. In this paper, we show it is possible to convert such tasks into discrete latent variable learning problems with a precomputed, task-specific set of possible "solutions" (e.g. different mentions or equations) that contains one correct option. We then develop a hard EM learning scheme that computes gradients relative to the most likely solution at each update. Despite its simplicity, we show that this approach significantly outperforms previous methods on six QA tasks, including absolute gains of 2--10%, and achieves the state-of-the-art on five of them. Using hard updates instead of maximizing marginal likelihood is key to these results as it encourages the model to find the one correct answer, which we show through detailed qualitative analysis.

1 Introduction

The paper frames weakly supervised QA as discrete latent-variable learning over precomputed candidate solutions, then trains with hard EM and reports strong gains across six datasets.

  • Motivation: Weak supervision makes QA scalable but creates ambiguity because documents can contain multiple answer mentions or many equations yielding the same answer.Only one mention or derivation may actually answer the question, while others are spurious.
  • Approach: The paper formulates diverse weakly supervised QA tasks as discrete latent-variable problems with task-specific solution sets containing the correct option and spurious alternatives.Solutions can be answer spans or equations, and the set is precomputed for each task.
  • Approach: Hard EM alternates between selecting the most likely candidate solution and updating parameters to increase its likelihood.The updates are intended to enforce the prior that there is a single correct solution and can work with any compatible architecture.
  • Results: Hard updates encourage models to assign much higher probability to the correct solution.The paper supports this claim through detailed qualitative analysis.

2 Related Work

Prior reading-comprehension methods often resolve weak supervision with span heuristics or marginal likelihood, while semantic parsing commonly uses marginal-likelihood or reward-based learning.

  • Reading Comprehension: Reading-comprehension research often chooses the first or a random matching span instead of modeling which mention is relevant as a latent choice.Multiple mentions are frequently false positives, although a single best-fitting span often exists.
  • Reading Comprehension: Maximum marginal likelihood sums probability over multiple candidate spans, but its benefit over heuristic supervision remains unclear.This uncertainty motivates alternatives that explicitly distinguish the correct solution from spurious candidates.
  • Semantic Parsing: Semantic parsing commonly learns from question–answer pairs without observing the logical form that computes the answer.MML maximizes probability over candidate logical forms, while reward-based methods update parameters using a prior reward function.
  • Connection to Prior Work: Unlike semantic-parsing methods that repeatedly recompute beam-based logical-form sets, this approach targets tasks whose solution sets can be precomputed.The targeted tasks include reading comprehension, open-domain QA, and SQL-based semantic parsing.

3 Method

The method treats weakly supervised QA as discrete latent-variable learning over candidate solutions and trains with hard EM to select and reinforce one likely solution.

  • Setup: Each input has a finite candidate set Z of solutions whose deterministic outputs match the observed answer y.When the full space is large or infinite, Ztot can be approximated with high coverage before filtering candidates by f(z) = y.
  • Setup: At inference, the model selects a solution from Ztot and returns its deterministic answer f(z), because the gold answer needed to construct Z is unavailable.
  • Maximum marginal likelihood: MML can reward probability assigned to multiple spurious candidates and mismatches training, which sums candidate probabilities, with inference, which chooses the maximum-probability solution.
  • Hard-EM learning: Hard EM alternates between selecting the highest-likelihood candidate in Z and optimizing negative log likelihood as if that candidate were the true solution.
  • Hard-EM learning: Replacing MML’s sum with a max produces a hard-EM objective that directly reinforces the currently selected solution.

4 Task Setup

The paper instantiates candidate-solution construction across multi-mention reading comprehension, discrete arithmetic reasoning, and SQL generation, where matching outputs can include spurious derivations.

  • Overview: The task setup covers multi-mention reading comprehension, discrete reasoning, and semantic parsing, with task-specific solution sets constructed from input and answer.
  • Multi-mention reading comprehension: Multi-mention reading comprehension represents each exact or similar answer mention as a document span candidate.Exact matching applies when the answer is guaranteed to appear in the document; free-form answers use ROUGE-L matching.
  • Reading comprehension with discrete reasoning: Discrete reasoning represents each candidate as an executable arithmetic equation using two numeric values and one operation.The finite approximation covers 93% of development examples, and f executes each equation to obtain its answer.
  • Multi-mention reading comprehension: Only one of six matching ‘Robert Schumann’ spans is question-relevant, illustrating why answer-string matching creates spurious candidates.
  • Reading comprehension with discrete reasoning: Many equations can produce the answer 4, but only ‘40-37’ answers the question, demonstrating answer-equivalent false positives.
  • Semantic parsing: SQL generation treats each non-nested query with at most three conditions as a candidate, while supervision provides only the query result.The approximation covers 99% of development examples, and f is an SQL executor.

5 Experiments

Experiments across weakly supervised reading comprehension, discrete reasoning, and SQL generation compare the proposed hard-EM learning strategy with heuristic supervision, MML, and reward-based methods. The method consistently improves performance, including 2+ metric points across four datasets and a 10% gain over prior weakly supervised WIKISQL methods.

  • Experimental setup: The experiments cover multi-mention reading comprehension, discrete reasoning, and weakly supervised SQL query generation across several datasets and model architectures.The study includes five datasets in the reading-comprehension and discrete-reasoning comparison and WIKISQL for semantic parsing.
  • Experimental setup: The study compares First-Only, MML, hard EM, and reward-based weak-supervision algorithms, including REINFORCE, iterative ML, and MAPO.For WIKISQL, comparisons exclude execution-guided decoding and use single models for fairness.
  • Reading-comprehension results: 2+ F1/ROUGE-L/EM points separate the proposed method from other methods consistently across four reading-comprehension datasets.The method reaches state of the art on NARRATIVEQA, TRIVIAQA-OPEN, and NATURALQUESTIONS-OPEN, and is comparable to state of the art on TRIVIAQA.
  • Solution-set analysis: The proposed method consistently outperforms MML as the solution-set size varies at test time and across WIKISQL training subsets.The training-subset experiment uses five values of |Z| and evaluates every subset on the same original development set.
  • SQL query-generation results: 10% gain over the previous state of the art is reported on WIKISQL, where the method also outperforms all compared weakly supervised algorithms.The method does not require SQL executions during training and remains on par with or better than most published fully supervised results.

6 Analysis

The analyses examine how solution-set size and noise affect training and inference, and how hard updates shape model predictions and solution quality. Across these settings, the hard-EM method is consistently more robust or accurate than MML.

  • Model predictions over training: Hard-EM models gradually shift from a near-uniform distribution over Z toward favoring the true solution during training.In the DROPnum example, the model temporarily predicts the spurious solution ‘37-36’ before returning to a true solution.
  • Model predictions over training: The temporary ‘37-36’ prediction reflects confusion between Rob Bironas’s 40- and 37-yard field goals, rather than arbitrary behavior.The example’s answer is 4, obtained from the longest Bironas field goal and Carney’s 36-yard field goal.
  • Quality of the predicted solution: Manual analysis finds that 98% of correct NARRATIVEQA cases and 92% of correct DROPnum cases produce the correct underlying solution.The analysis distinguishes predicting a correct derivation from merely executing a derivation that yields the right answer text.
  • Robustness to the noise in |Z|: Increasing NARRATIVEQA’s noisy solution set from 4.3 to 7.1 lowers MML from 56.07 to 51.14, but lowers hard-EM performance only from 58.77 to 57.97.The result suggests that MML is more sensitive to noise in Z than the proposed method.

7 Conclusion

The paper concludes that weakly supervised QA can be formulated using precomputed discrete solution sets containing a correct option, and trained with discrete latent-variable updates. This approach improves performance across six QA tasks, with 2–10% absolute gains and new state-of-the-art results on five datasets.

  • 7 Conclusion: Precomputed discrete solution sets can contain one correct option for many QA tasks supervised only by answer text.The paper treats spans, equations, and other derivations as candidate solutions.
  • 7 Conclusion: The proposed algorithm alternates between predicting the most likely candidate solution and increasing that solution’s likelihood.This is the paper’s discrete latent-variable learning procedure.
  • 7 Conclusion: 2–10% absolute gains across six QA tasks accompany new state-of-the-art results on five well-studied datasets.The reported tasks include reading comprehension, open-domain QA, discrete reasoning, and semantic parsing.

A Model details

The model details describe task-specific architectures that estimate probabilities for candidate spans, using BERT representations and paragraph selection where needed. The implementation reuses established base models while changing the learning objective.

  • A Model details: The base model estimates P(z|x; θ), the probability of a task-specific solution given the input.The solution may be a span or another task-specific derivation.
  • A Model details: The implementation uses BERT-based models for multi-mention reading comprehension and sequence tagging, while retaining original QANet and SQLova architectures apart from the objective.The code is implemented in PyTorch and uses uncased BERT-base.
  • A Model details: For extractive reading comprehension, BERT processes each question–paragraph pair, and the system selects the best paragraph before outputting its span.The input combines the question and paragraph with a [SEP] token.
  • A Model details: The span probability is computed for a span beginning at position s and ending at position e in a selected paragraph.The model assigns probability to a specific span z defined by its paragraph and boundary positions.

Sequence Tagging model for discrete reasoning

The discrete reasoning model represents each candidate equation as two operators and two numbers drawn from the question, paragraph, or predefined special numbers. BERT encodings provide the representations used to assign probabilities to these equation components.

  • Sequence Tagging model for discrete reasoning: Each reasoning solution is an equation z = (o1, n1, o2, n2) with operators drawn from {+, −, ∗0.01}.The two numbers are selected from numeric values in the paragraph or question and from predefined special numbers.
  • Sequence Tagging model for discrete reasoning: The equation’s numbers come from NP ∪ NQ ∪ S, where NP and NQ are paragraph and question numbers and S contains predefined special numbers.The same candidate representation is used to compute a probability for each equation.
  • Sequence Tagging model for discrete reasoning: BERT encodes the question and paragraph after concatenation with a [SEP] token, using m and n for their lengths and h for hidden dimension.These encodings form the basis for subsequent component representations.
  • Sequence Tagging model for discrete reasoning: Operator-specific indicator vectors mark the positions of the selected numbers, assigning α(o1) and α(o2) at their locations and zero elsewhere.The operator indices are encoded as 1 for ‘+’, 2 for ‘-’, and 3 for ‘*0.01’.

B Annealing

The method gradually shifts training from the MML objective to the paper’s objective, preventing early model decisions from dominating optimization. This annealing improves performance and is insensitive to τ.

  • B Annealing: Training uses the MML objective with probability min(t/τ, 1), otherwise using the paper’s objective.τ controls the transition schedule.
  • B Annealing: Annealing improves performance without making results sensitive to the hyperparameter τ.Ablations and dataset-specific τ choices are reported in Table 6.

C Examples

The qualitative analyses show that hard-EM training selects solutions that answer the question, concentrates probability on the best derivation, and often produces correct SQL even when it differs from annotation. These examples illustrate how weak supervision can admit multiple textual spans or derivations, while the model identifies a suitable one.

  • TRIVIAQA: In TriviaQA, the proposed training method selects the span that answers the question rather than another span containing the same answer text.The example concerns the repeated answer text “Montgomery.”
  • DROPnum: 59 vs. 36 with ϵ = 10^-3, and 54 vs. 17 with ϵ = 10^-4, are the reported sparsity values for the proposed method versus MML on DROP.These values quantify the greater sparsity attributed to the proposed method.
  • WIKISQL: 7110 of 8421 WIKISQL development examples execute the correct answer, including 6296 predictions exactly matching the annotated queries.The remaining correct executions include predictions that differ from the annotated SQL.
  • WIKISQL: WIKISQL examples show that different predicted and annotated SQL queries can both be correct, with some predictions making more sense than the annotations.Table 9 presents four such examples.
Loading 1909.04849v1…