PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-granularity hierarchical attention fusion networks for reading comprehension and question answering

Wei Wang, Ming Yan, Chen Wu

arXiv:1811.11934v1cs.CL

TL;DR

Reading comprehension requires answering questions from narrative passages while locating the correct answer span. The paper proposes hierarchical multi-granularity attention and fusion over question–passage representations, achieving state-of-the-art results across SQuAD, TriviaQA, AddSent, and AddOneSent, including top SQuAD leaderboard positions at the time of writing.

  • Problem

    Reading comprehension requires systems to understand a passage and question and identify an answer, motivating richer question–passage alignment than earlier approaches provide.

  • Method

    The model encodes question and passage representations, applies hierarchical co-attention and self-attention with multi-level fusion, and progressively focuses on the answer span.

  • Results

    The method outperforms previous state-of-the-art methods across SQuAD, TriviaQA, AddSent, and AddOneSent, with 82.4 EM and 88.6 F1 for its SQuAD ensemble.

  • Takeaways & Limitations

    Hierarchical attention and fusion provide a robust reading-comprehension model that retains global passage information while refining attention toward relevant answer spans.

Abstract

from arXiv · show

This paper describes a novel hierarchical attention network for reading comprehension style question answering, which aims to answer questions for a given narrative paragraph. In the proposed method, attention and fusion are conducted horizontally and vertically across layers at different levels of granularity between question and paragraph. Specifically, it first encode the question and paragraph with fine-grained language embeddings, to better capture the respective representations at semantic level. Then it proposes a multi-granularity fusion approach to fully fuse information from both global and attended representations. Finally, it introduces a hierarchical attention network to focuses on the answer span progressively with multi-level softalignment. Extensive experiments on the large-scale SQuAD and TriviaQA datasets validate the effectiveness of the proposed method. At the time of writing the paper (Jan. 12th 2018), our model achieves the first position on the SQuAD leaderboard for both single and ensemble models. We also achieves state-of-the-art results on TriviaQA, AddSent and AddOne-Sent datasets.

1 Introduction

Reading comprehension requires understanding a passage and question before identifying an answer span. The paper proposes hierarchical attention and fusion to progressively connect question and passage representations and locate the answer.

  • Reading comprehension asks systems to understand a text and answer questions from it, requiring natural-language understanding, inference, and reasoning.
  • Prior work used end-to-end methods on cloze-style datasets, but those datasets have limitations for broader reading comprehension.
  • The proposed model follows a human-like process that scans the passage, relates it to the question, locates a rough answer span, and refines the selection.
  • The architecture combines language-model-based encoding, hierarchical multi-granularity attention, and pointer-network answer-boundary prediction.
  • The attention layer combines co-attention and self-attention while repeatedly fusing aligned representations with global information to maintain focus on the relevant passage.
  • 79.2% EM and 86.6% F1 are achieved by the single model on SQuAD, while the ensemble reaches 82.4% EM and 88.6% F1.

2 Related Work

Attention-based reading-comprehension systems evolved from one-way matching toward richer question–passage interactions. This paper adds hierarchical co-attention, self-attention, and fusion to preserve both aligned details and global passage context.

  • Early reading-comprehension systems relied on linguistic annotation, semantic parsing, and structured knowledge, while later datasets enabled end-to-end neural methods.
  • CNN/Daily Mail and Children’s Book Test use cloze-style missing-word prediction, whereas SQuAD extracts an arbitrary answer span from the passage.
  • Typical attention models encode question and passage representations before computing attention to align them and identify answers.
  • One-way attention summarizes the question into a single vector, while two-way attention computes interactions across question and passage word embeddings.
  • Prior work also introduced co-attention, bidirectional attention flow, and self-attention to couple or refine question–document representations.
  • The proposed model combines co-attention and self-attention across layers, fusing aligned and original representations so it retains global passage context while focusing on relevant regions.

3.1 Task Description

Machine reading comprehension takes a passage and question as input and predicts an answer span within the passage. The task formalizes answer selection as maximizing the probability of a candidate answer given both inputs.

  • A machine comprehension system reads an evidence text and question, then predicts a span within the evidence that answers the question.
  • The passage and question are represented as token sequences, with n denoting passage length and m denoting question length.
  • In SQuAD, the answer is guaranteed to be a continuous span in the passage.
  • The task learns f(q, p) = arg maxa∈A(p) P(a|q, p) from question, passage, and answer tuples.

3.2 Encode-Interaction-Pointer Framework

The framework uses layered semantic processing to encode inputs, model question–passage relations, match representations, and predict answer boundaries. Its main contribution is hierarchical attention with fine-grained fusion.

  • The framework contains four layers that learn progressively different concepts of semantic representation.
  • Encoder Layer: The encoder uses contextual cues from surrounding words to convert passage and question tokens into semantic representations.
  • Attention Layer: The attention layer captures question–passage relations, merges aligned context with contextual embeddings, and forms a multi-level working memory.
  • Match and Output Layers: The match layer computes question–passage relevance at the span level, while the output layer uses a pointer network to search for the answer span.
  • The central contribution is hierarchical attention that progressively clarifies answer boundaries together with fine-grained fusion across contextual representations.

3.3 Hierarchical Attention Fusion Network

The framework uses fine-grained fusion and multi-granularity attention to progressively narrow the potential answer domain and locate discriminative content.

  • The design first identifies a rough answer domain, then progressively locates its most discriminative parts.
  • The framework combines shallow co-attention fusion, deep self-attention fusion, and memorywise bilinear alignment.
  • Fine-grained fusion blends attention vectors to improve modeling of the question–passage relationship.
  • Word- and sentence-level attention helps the model attend to important content when constructing question and passage representations.

3.4 Language Model & Encoder Layer

The encoder converts question and passage tokens into contextual continuous representations by combining pretrained word and character embeddings with a shared Bi-LSTM.

  • The encoder transforms discrete question and passage tokens into sequences of continuous vector representations.
  • It combines pretrained GloVe word embeddings with ELMo-based character embeddings.
  • A shared Bi-LSTM models temporal interactions among words to incorporate contextual cues from surrounding words.
  • The model concatenates word and character embeddings before contextual encoding for a fuller representation of each word.
  • The encoder additionally concatenates contextual Bi-LSTM outputs with pretrained character embeddings as a residual connection across representation levels.

3.5 Hierarchical Attention & Fusion Layer

The hierarchical attention and fusion layer aligns question and passage information through co-attention, self-attention, multi-level fusion, and contextual refinement.

  • The attention layer links and fuses question and passage representations to locate the passage span most relevant to the question.
  • A soft-alignment matrix computes shallow semantic similarity between question and passage representations.
  • Passage-to-question attention identifies relevant question words for each passage word, while question-to-passage attention highlights passage words most similar to question words.
  • Fusion combines original contextual representations with aligned representations using a kernel based on concatenation, difference, and element-wise product.
  • Gating incorporates projected representations with original contextual representations and supports scalar-, vector-, and matrix-based sigmoid functions.
  • Self-attention refines question-aware passage representations, after which a Bi-LSTM produces the final contextual passage representation.

3.6 Model & Output Layer

The model and output layer use bilinear matching to combine question and passage information and predict a constrained answer span from start and end distributions.

  • A top-level bilinear match function captures the semantic relation between the question and contextual passage representation through multi-hop matching.
  • The top layer uses previous-layer interactions to locate the correct answer span after co-attention and self-attention produce progressively refined representations.
  • Bilinear matching produces separate start and end distributions over passage words.
  • Pointer networks predict the answer's start and end positions because the answer is a passage sub-phrase.
  • Training minimizes the summed negative log probabilities of the true start and end indices using cross entropy.
  • During prediction, dynamic programming selects the span maximizing ps · pe subject to s ≤ e ≤ s + 15.

4 Experiments

Experiments evaluate the model on SQuAD, TriviaQA, and adversarial or ablation settings, showing strong benchmark performance and sensitivity to key architectural choices. The studies compare fusion kernels and attention configurations to identify effective components.

  • Datasets and evaluation: The evaluation covers SQuAD, TriviaQA, adversarial SQuAD, and AddSent/AddOneSent settings using benchmark comparisons and component analyses.SQuAD is described as a 100,000+ question span-extraction dataset, while TriviaQA contains over 650K context-query-answer triples with substantially longer contexts.
  • Main results: 82.4 EM and 88.6 F1 are achieved by the SLQA+ ensemble on the SQuAD hidden test set, outperforming previous approaches.The reported comparison is on the hidden test set and is presented as evidence for the hierarchical attention and fusion structure.
  • Ablations: Nearly 15% performance drops when bilinear alignment plus passage-question fusion is removed, while removing ELMo reduces performance by about 5%.Replacing multi-hop fusion with a standard LSTM reduces performance by nearly 5% on both metrics.
  • Fusion functions: The vector-based fusion kernel performs best among the compared fusion methods with a moderate parameter size.The comparison is conducted on the SQuAD development set.
  • Attention hierarchy and function: Bilinear attention with a post-linear ReLU performs significantly better than the other tested attention functions.The model also shows a steep and steady accuracy rise as attention layers increase from N = 1 to 3.
  • Robustness and transfer: The proposed model obtains state-of-the-art performance on the more complex TriviaQA dataset and remains superior to competing approaches under adversarial SQuAD evaluation.The adversarial setting appends misleading sentences to the original SQuAD contexts.

5 Conclusions

The proposed hierarchical attention network outperforms previous state-of-the-art methods across multiple reading-comprehension datasets. Its attention and fusion operate horizontally and vertically across layers at different granularities.

  • The hierarchical attention network conducts attention and fusion horizontally and vertically across layers at different granularity levels between question and paragraph.
  • The method outperforms previous state-of-the-art methods on SQuAD, TriviaQA, AddSent, and AddOneSent.
  • Figure 2 presents learning curves for F1 and EM scores on the SQuAD development set.
Loading 1811.11934v1…