PaperlayerView exact revision on arXiv

Source-linked AI summary

NumNet: Machine Reading Comprehension with Numerical Reasoning

Qiu Ran, Yankai Lin, Peng Li, Jie Zhou, Zhiyuan Liu

arXiv:1910.06701v1cs.CL

TL;DR

Existing MRC models often do not explicitly address numerical reasoning, although such skills are needed in reading-comprehension problems. NumNet encodes number relations in question and passage text as a graph and reasons over it with a numerically-aware graph neural network. It outperforms strong baselines on DROP, while its predefined graph limits cases requiring derivation of intermediate numbers.

  • Problem

    Most existing MRC models do not explicitly account for numerical reasoning skills such as addition, subtraction, sorting, and counting.

    Source: §1 · p. 1

  • Method

    NumNet encodes numerical relations among numbers in the question and passage as a graph and uses a numerically-aware graph neural network to reason over it.

    Source: §5 · p. 9

  • Results

    NumNet outperforms strong baselines by a large margin on the DROP dataset.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The paper integrates numerical reasoning, described as a special case of symbolic reasoning, into traditional MRC systems.

    Source: §5 · p. 9

    Because its numerically-aware graph is predefined, NumNet does not apply when reasoning requires deriving an intermediate number through arithmetic.

    Source: §4.9 · p. 9

Abstract

from arXiv · show

Numerical reasoning, such as addition, subtraction, sorting and counting is a critical skill in human's reading comprehension, which has not been well considered in existing machine reading comprehension (MRC) systems. To address this issue, we propose a numerical MRC model named as NumNet, which utilizes a numerically-aware graph neural network to consider the comparing information and performs numerical reasoning over numbers in the question and passage. Our system achieves an EM-score of 64.56% on the DROP dataset, outperforming all existing machine reading comprehension models by considering the numerical relations among numbers.

1 Introduction

Existing MRC models often handle numerical reasoning poorly, despite its relevance to questions requiring sorting or comparison and to questions where comparison determines what should be counted. NumNet explicitly incorporates numerical comparison into MRC and improves results on DROP.

1 Introduction

  • Most existing MRC models remain weak at addition, subtraction, sorting, and counting, skills needed when reading financial news and scientific articles.The cited work identifies numerical reasoning as an under-addressed MRC skill.Source: §1 · p. 1
  • Numerical comparison can directly answer ranking questions, such as finding the second-longest field goal from ordered distances.The example requires knowing that 49 > 47 > 36 > 31 > 22.Source: §1 · p. 1
  • Some questions require comparing values to a threshold before counting, as when identifying age groups exceeding 7% of the population.Here comparison helps determine which groups qualify for the count.Source: §1 · p. 1
  • NumNet combines question-and-passage encoding with a numerically-aware graph neural network that incorporates comparisons into representations used to infer answers.The described encoder includes convolution, self-attention, feed-forward, and passage-question attention layers.Source: §1 · pp. 1–2
  • NumNet significantly and consistently improves over all baseline methods on DROP, including for questions requiring sorting with a multi-layer NumGNN.The model performs numerical reasoning over numbers in the question and passage.Source: §1 · p. 2

2 Related Work

Prior MRC models focus mainly on matching, summarization, and logic, while numerical reasoning remains challenging; numerical-answer extensions such as NAQANet do not explicitly reason numerically during encoding or inference. Arithmetic word-problem solvers face limits in data scale and transfer to real-world MRC passages, motivating a numerical-reasoning approach for MRC.

2.1 Machine Reading Comprehension

  • Neural MRC systems use multi-layer reading and attention to focus on question-relevant document content, but the cited review finds numerical reasoning remains a weakness.The review lists matching/paraphrasing, summary, logic, external knowledge, and numerical reasoning as question types.Source: §2.1 · p. 2
  • Most existing MRC models focus on matching, summarization, and logic, and struggle with numerical reasoning; NAQANet supports counting and arithmetic answers but lacks explicit numerical reasoning during encoding or inference.The paper describes its work as the first to incorporate numerical reasoning explicitly into MRC.Source: §2.1 · p. 2

2.2 Arithmetic Word Problem Solving

  • NumNet represents numbers as graph nodes and numerical relations as edges, then uses a numerically-aware graph neural network to reason over them.An edge from 6 to 5 denotes that 6 is greater than 5.Source: Figure 1 · p. 3
  • Arithmetic word-problem systems handle varied operations, but their small training benchmarks and the complexity of real-world MRC passages limit straightforward transfer to MRC.The paper therefore focuses on enhancing MRC models with numerical reasoning abilities.Source: §2.2 · p. 3

3 Methodology

NumNet adds a numerically aware graph reasoning module between MRC encoding and prediction. The graph represents numerical relations among question and passage numbers, which the model uses to reason about answers.

3.1 Framework

  • NumNet uses NAQANet-style encoding and prediction around a NumGNN reasoning module, then combines number representations with passage features for span answers.Its prediction layers handle passage spans, question spans, counts, and arithmetic expressions.Source: §3.1 · pp. 3–4

3.2 Numerically-aware Graph Construction

  • The graph represents each question and passage number as a node, with directed greater-than and lower-or-equal edges; duplicate occurrences remain distinct to preserve potentially different facts.Multiplication and division expression handling is left for future work because few DROP samples require it.Source: §3.2 · pp. 4–5

3.3 Numerical Reasoning

  • NumGNN weights nodes to bypass irrelevant numbers, propagates relation-specific messages among neighbors, and fuses each message with its node representation.Question and passage numbers can play different roles, so message transforms distinguish node types and numerical relations.Source: §3.3 · p. 5
  • Multi-step reasoning lets NumGNN infer relations among multiple nodes, which may be required for tasks such as sorting.The graph topology encodes numerical relations, making the reasoning process numerically aware.Source: §3.3 · pp. 5–6

4 Experiments

The experiments use DROP, a crowd-sourced numerical MRC dataset, and evaluate models with Exact Match and numerically-focused F1. Comparisons include semantic-parsing, traditional MRC, and numerical MRC baselines.

4.1 Dataset and Evaluation Metrics

  • DROP contains 77,409 training, 9,536 development, and 9,622 testing examples built from Wikipedia passages with questions requiring numerical reasoning.The crowd-sourced questions may require addition, counting, or sorting over passage numbers.Source: §4.1 · p. 6
  • Evaluation uses Exact Match and numerically-focused F1, which is set to zero when a prediction mismatches a numeric gold answer.The metrics follow Dua et al. (2019).Source: §4.1 · p. 6

4.2 Baselines

  • Semantic-parsing baselines include KDG with Stanford dependency, OpenIE, or semantic-role-labeling sentence representations.These variants are listed as Syn Dep, OpenIE, and SRL.Source: §4.2 · p. 6
  • Traditional MRC baselines include BiDAF, QANet, and BERT.The passage describes their attention, convolution/self-attention, and pretrained Transformer approaches, respectively.Source: §4.2 · p. 6
  • Numerical MRC comparisons include NAQANet and NAQANet+, an enhanced implementation adding real numbers, richer arithmetic expressions, and data augmentation.The listed enhancements are also used in NumNet.Source: §4.2 · p. 6

4.3 Experimental Settings

The authors tune the model on the development set with grid search and use fixed representation dimensions and training settings.

4.3 Experimental Settings

  • The model is tuned on the development set with grid search; representation dimensions are 128 and the reasoning step count K defaults to 3.Other parameters follow Dua et al. (2019) because they have little effect on results.Source: §4.3 · p. 6
  • Training uses Adam with learning rate 5 × 10^-4, L2 weight decay 10^-7, gradient clipping at 5, and batch size 16 for 40 epochs.The optimizer uses β1 = 0.8, β2 = 0.999, and ϵ = 10^-7.Source: §4.3 · p. 6
  • Inputs are trimmed to 400 passage tokens and 50 question tokens during training, and 1,000 passage tokens and 100 question tokens during prediction.An exponential moving average with decay 0.9999 is applied to trainable variables.Source: §4.3 · p. 6

4.4 Overall Results

On DROP, NumNet outperforms semantic-parsing, traditional MRC, and numerical MRC baselines on both development and test sets. The authors attribute its performance to using numerical comparison information from numbers in the question and passage; their implemented NAQANet+ also improves on original NAQANet.

4.4 Overall Results

  • NumNet achieves better results than semantic-parsing, traditional MRC, and numerical MRC baselines, including NAQANet and NAQANet+, on DROP development and test sets.The authors attribute the gains to NumGNN using numerical comparison information over numbers in both the question and passage.Source: §4.4 · pp. 6–7, Table 2 · p. 7
  • Table 2 reports development and test results using metrics maximized over a golden answer set; results other than NAQANet+ and NumNet come from Dua et al. (2019).Source: Table 2 · p. 7
  • The implemented NAQANet+ performs much better than original NAQANet, supporting the effectiveness of the authors’ baseline enhancements.Source: §4.4 · p. 7

4.5 Effect of GNN Structure

The GNN-structure ablations favor NumGNN over a traditional GNN, especially on comparison questions. Layer experiments show that two layers work best for comparison questions, while deeper networks improve overall development-set performance with instability at four or more layers.

4.5 Effect of GNN Structure

  • NumGNN significantly improves over traditional GNN on EM and F1, especially for comparison questions.Table 3 evaluates Comparison, Number, and the full development set; ablations replace NumGNN with a fully connected graph or omit question numbers or either relation-edge type.Source: Table 3 · p. 7, §4.5 · p. 7

4.6 Effect of GNN Layer Number

  • Overall development-set performance improves consistently as GNN layers increase, but gains are unstable for K ≥4, which the authors attribute to GNN over-smoothing.The authors note that numerical questions involving many passage numbers can benefit from multi-step reasoning.Source: Figure 2 · p. 7, §4.6 · p. 8
  • The 2-layer NumNet performs best on comparison questions; most require at most two reasoning steps, while a third layer adds complexity without gains.Source: Figure 2 · p. 7, §4.6 · pp. 7–8

4.7 Case Study

The case study contrasts NAQANet+ errors with NumNet’s correct answers on two numerical reasoning examples. The examples involve comparing percentages and identifying a second-longest field goal.

4.7 Case Study

  • NumNet answers both cases correctly, while NAQANet+ makes errors on the age-group and second-longest-field-goal questions.Source: §4.7 · p. 8
  • For the age-group case, NAQANet+ cannot distinguish which percentage is larger, 10.1% or 56.2%.Source: §4.7 · p. 8
  • For the field-goal case, NAQANet+ fails to identify the second-longest field goal as 22 yards, while NumNet answers correctly.Source: §4.7 · p. 8

4.8 Error Analysis

NumNet answers about 76% of sampled sorting/comparison questions correctly, while many remaining errors involve nonadjacent answer spans or comparisons with arithmetic-derived intermediate numbers. Its numerically-aware graph supports comparison and sorting, but the predefined graph limits reasoning over numbers not already represented.

4.8 Error Analysis

  • NumNet answers about 76% of sampled sorting/comparison questions correctly.The authors interpret this as evidence of numerical reasoning ability.Source: §4.8 · p. 8
  • Among incorrect sorting/comparison answers, 26% have multiple nonadjacent spans as gold answers, while 19% require comparison with an arithmetic-derived intermediate number.Source: §4.8 · pp. 8–9

4.9 Discussion

  • The numerically-aware graph represents numbers as nodes and their relationships as edges; one-step NumGNN reasoning handles comparison and numerical conditions, while multiple steps support sorting.Source: §4.9 · p. 9
  • Because the reasoning graph is predefined, NumNet cannot handle cases requiring an intermediate number derived during reasoning, such as through arithmetic.The authors identify this as a major limitation.Source: §4.9 · p. 9

5 Conclusion and Future Work

NumNet encodes numerical relations between question and passage numbers in a graph and uses a numerically-aware graph neural network for reasoning. It outperforms strong baselines on DROP, while future work targets graph flexibility, richer arithmetic, and broader symbolic reasoning.

5 Conclusion and Future Work

  • NumNet outperforms strong baselines by a large margin on the DROP dataset.Source: §5 · p. 9
  • NumNet encodes relations among question and passage numbers as graph topology and uses a numerically-aware graph neural network to reason over them.Source: §5 · p. 9
  • The predefined reasoning graph cannot handle processes involving intermediate numbers absent from that graph; incorporating a dynamic graph is proposed as future work.Source: §5 · p. 9
  • The authors propose combining NumNet's natural-language understanding with arithmetic-word-problem methods' ability to handle richer arithmetic expressions.Source: §5 · p. 9
  • Incorporating more sophisticated symbolic reasoning into MRC is identified as another future direction.The paper characterizes numerical reasoning as a special case of symbolic reasoning.Source: §5 · p. 9

Appendix: Baseline Enhancements

Ablations indicate that real-number and richer-arithmetic-expression support are important for numerical questions, while passage preference and data augmentation improve comparing questions. The appendix describes these baseline enhancements and their evaluation on the development set.

Appendix: Baseline Enhancements

Loading 1910.06701v1…