PaperlayerView exact revision on arXiv

Source-linked AI summary

FusionNet: Fusing via Fully-Aware Attention with Application to Machine Comprehension

Hsin-Yuan Huang, Chenguang Zhu, Yelong Shen, Weizhu Chen

arXiv:1711.07341v2cs.CLcs.AI

TL;DR

FusionNet targets the limited use of multi-level information in attention-based machine comprehension. It introduces history-of-word representations, an efficient scoring function, and fully-aware multi-level attention, then achieves leading performance on SQuAD and adversarial SQuAD datasets.

  • Problem

    Existing approaches do not capture the full information available across representation levels in the question and context, while using all layers can make models harder to train.

  • Method

    FusionNet combines history-of-word representations, an attention scoring function designed to use them efficiently, and fully-aware multi-level fusion applied layer by layer.

  • Results

    FusionNet ranked first in both SQuAD single-model and ensemble categories and outperformed prior models on both adversarial SQuAD datasets.

  • Takeaways & Limitations

    The results support FusionNet as a general and improved attention mechanism for machine reading comprehension across original and adversarial SQuAD settings.

Abstract

from arXiv · show

This paper introduces a new neural structure called FusionNet, which extends existing attention approaches from three perspectives. First, it puts forward a novel concept of "history of word" to characterize attention information from the lowest word-level embedding up to the highest semantic-level representation. Second, it introduces an improved attention scoring function that better utilizes the "history of word" concept. Third, it proposes a fully-aware multi-level attention mechanism to capture the complete information in one text (such as a question) and exploit it in its counterpart (such as context or passage) layer by layer. We apply FusionNet to the Stanford Question Answering Dataset (SQuAD) and it achieves the first position for both single and ensemble model on the official SQuAD leaderboard at the time of writing (Oct. 4th, 2017). Meanwhile, we verify the generalization of FusionNet with two adversarial SQuAD datasets and it sets up the new state-of-the-art on both datasets: on AddSent, FusionNet increases the best F1 metric from 46.6% to 51.4%; on AddOneSent, FusionNet boosts the best F1 metric from 56.0% to 60.7%.

1 INTRODUCTION

FusionNet addresses the challenge of fully using question and context information in machine reading comprehension. It combines history-of-word representations, a more trainable attention score, and fully-aware multi-level attention, achieving leading results on SQuAD and adversarial datasets.

  • Machine reading comprehension requires a system to understand a context and question well enough to find the correct answer.The task demands comprehension, inference, and reasoning capabilities.
  • Existing attention models typically transfer either word-level question embeddings or high-level question representations into the context.
  • FusionNet uses information from word embeddings through the highest-level representations to capture complete information in both question and context.Its design includes history-of-word representations, an attention scoring function that uses them, and fully-aware multi-level attention applied layer by layer.
  • 78.8% EM and 85.9% F1 were achieved by the ensemble model on SQuAD, where FusionNet ranked first in both single-model and ensemble categories at the time of writing.

2 MACHINE COMPREHENSION & FULLY-AWARE ATTENTION

FusionNet frames machine comprehension as fusing information between context and question, then extends fusion with history-of-word and fully-aware attention across representation levels. Its architecture combines word-level, high-level, and self-boosted fusion processes to preserve both detail and broader contextual meaning.

  • Machine Comprehension: Machine comprehension provides a context and question, requiring the system to understand the context and find the answer.In SQuAD, the answer is a contiguous span in the context.
  • Conceptual Architecture: Existing MRC architectures fuse information from one vector set into another, with recent improvements centered on fusion-process design.Fusion commonly uses attention, though some mechanisms do not.
  • Conceptual Architecture: Recent fusion mechanisms include word-level, high-level, and self-boosted fusion, each supplying different contextual or semantic information.Word-level fusion preserves direct lexical clues, high-level fusion supplies semantic information, and self-boosted fusion connects distant context regions.
  • History-of-Word: History-of-word concatenates all representations generated for a word, from word embeddings through intermediate and output hidden vectors.This representation is intended to capture multiple levels of contextual information for fuller text understanding.
  • Fully-Aware Attention: Fully-aware attention replaces standard attention-score computation with history-of-word information and provides a lightweight implementation.The paper reports that an ablation study found this enhancement improves performance.
  • FusionNet Architecture: FusionNet applies fully-aware attention between context and question, then performs multi-level question fusion and self-attention over the context.Its architecture concatenates context concepts with multi-level question information, processes them with BiLSTMs, and then incorporates context self-attention.

3 FULLY-AWARE FUSION NETWORK

FusionNet fuses information from one text into another using representations spanning word-level, contextual, and understanding-level information. Its fully-aware multi-level mechanism applies separate attention weights to different information levels.

  • FusionNet fuses information from text B to text A and generates vector representations for both texts.
  • Input Vectors: Each context and question word is represented using GloVe, contextualized vectors, and task-specific features for SQuAD.The input vectors combine 300-dimensional GloVe embeddings, 600-dimensional contextualized vectors, POS and NER embeddings, and normalized term frequency for context words.
  • Fully-Aware Multi-level Fusion: Word-level fusion separately informs context representations about question words, while higher-level fusion combines contextual information from the question.
  • Reading: Separate bidirectional LSTMs form low-level and high-level concepts for the context and question.The resulting low- and high-level concepts are created for each word and then used to derive question understanding vectors.
  • Fully-Aware Multi-level Fusion: Fully-aware higher-level fusion attends from the question to the context with independently parameterized functions for low, high, and understanding levels.The three attention functions use the proposed symmetric form with nonlinearity and attend to different regions for each level.

2. High-level fusion: ˆh

After fusing question information into context representations, FusionNet applies self-boosted fusion and constructs understanding vectors used to predict answer spans. The architecture combines context and question information through recurrent processing and span-specific attention.

  • A new bidirectional LSTM produces a context representation fully fused with information from the question.
  • Self-Boosted Fusion: Fully-aware self-boosted fusion uses attention over the context to incorporate information from distant context parts.
  • FusionNet outputs understanding vectors for both context and question after its fusion components.
  • Prediction: For span prediction, a summarized question understanding vector attends to span starts, while a GRU incorporates the predicted start information before span-end attention.
  • Training: Training maximizes the log probabilities of the ground-truth answer span's start and end positions.

4 EXPERIMENTS

Experiments evaluate FusionNet on SQuAD and adversarial variants, alongside comparisons of attention functions and architectural configurations. FusionNet achieves strong benchmark and adversarial results, while ablations support fully-aware attention and multi-level history-of-word fusion.

  • Main Results: 46.2 / 51.4 EM / F1 is achieved by FusionNet (E) on AddSent, exceeding the listed competing systems.
  • Main Results: FusionNet outperforms previous models by more than 5% in EM score on the adversarial datasets.
  • Attention Functions: The proposed symmetric attention form consistently outperforms the alternatives on the SQuAD development set.The authors attribute the gain to using a single large parametric matrix, while the nonlinear variant retains additional performance benefits.
  • History-of-Word Ablations: Replacing standard attention with fully-aware attention improves the High-Level model's performance by more than 8%.Fully-aware self-boosted fusion also provides considerable improvement over normal self-boosted fusion in the improved fusion architecture.
  • History-of-Word Ablations: FA Multi-Level improves over the naive FA All-Level configuration by attending to different parts of the question history independently.
  • History-of-Word Ablations: The ablations conclude that taking all levels of understanding together is crucial for better text understanding.

5 CONCLUSIONS

FusionNet combines history-of-word representations, improved attention scoring, and fully-aware multi-level fusion. It was evaluated against published models on SQuAD, with performance shown across training epochs.

  • FusionNet combines history-of-word representations, an improved attention scoring function, and fully-aware multi-level fusion.These components use information from word-level embeddings through high-level semantic representations.
  • FusionNet was compared with published state-of-the-art architectures on the SQuAD development set.The comparison included R-net, Reinforced Mnemonic Reader, MEMEN, ReasoNet, DrQA, DCN, BiDAF, and Match-LSTM variants.
  • Figures 5 and 6 report FusionNet’s SQuAD development-set EM and F1 performance under different training epochs.Each epoch processes all training examples once; one epoch took roughly 20 minutes on a single NVIDIA GeForce GTX Titan X with batch size 32.

B DETAILED CONFIGURATIONS IN THE ABLATION STUDY

The ablation configurations progressively add fully-aware attention, full history-of-word representations, and layer-specific attention weights. These variants produce understanding vectors for context and question before answer prediction.

  • General configuration: All configurations generate understanding vectors for context and question before applying the shared machine-comprehension output architecture.The resulting vectors are used to find an answer span in the context.
  • High-Level: High-Level fuses question information with high-level context representations and passes the concatenated vectors through two BiLSTM layers.The context understanding vectors are the final BiLSTM hidden vectors, while the question vectors are its high-level representations.
  • FA High-Level: FA High-Level replaces the high-level attention with fully-aware attention while leaving the remaining configuration unchanged.The shared history-of-word is used for both context and question, and the paper reports significant improvement from this change.
  • FA All-Level: FA All-Level applies fully-aware attention to the entire history-of-word and mixes attended information with two BiLSTM layers.The context understanding vectors combine original and attended history-of-word information; the question representation is constructed similarly.
  • FA Multi-Level: FA Multi-Level assigns different attention weights to different history-of-word layers while retaining awareness of the complete history.Its self-boosted fusion variants include None, Normal, and Fully-Aware settings; fully-aware attention makes self-boosted fusion useful in this approach.

C ADDITIONAL ABLATION STUDY ON INPUT VECTORS

Additional ablations examine GloVe and CoVe input vectors on SQuAD and adversarial datasets. FusionNet remains stronger than previous best results, while CoVe removal is slightly better on the adversarial sets.

  • Adversarial datasets: Figure 7 shows single-model AddSent performance across 10 training runs, with a dashed vertical line marking previous best performance.
  • SQuAD input vectors: FusionNet outperforms previous state-of-the-art by +2% in EM on SQuAD with and without CoVe embeddings.Fine-tuning the top-1000 GloVe embeddings is reported to be slightly helpful.
  • Adversarial datasets: FusionNet single and ensemble models, with or without CoVe, all exceed previous best performance on AddSent and AddOneSent by a significant margin.The ten original FusionNet training runs show mostly similar performance, with a few slightly worse runs.
  • Adversarial datasets: Removing CoVe is slightly better on the adversarial datasets.The authors attribute this to the datasets targeting over-stability and CoVe being based on a two-layer BiLSTM output.

D APPLICATION TO NATURAL LANGUAGE INFERENCE

FusionNet’s fully-aware attention is also applied to natural language inference by enhancing ESIM. On MultiNLI, fully-aware attention yields the largest improvement, with further gains from multi-level fusion.

  • Task and dataset: MultiNLI evaluates entailment, contradiction, and neutral relations across in-domain and cross-domain settings.Its ten genres include spoken and written text, while training covers five genres.
  • ESIM baseline: ESIM uses BiLSTM encoders, standard bidirectional attention, inference BiLSTMs, and an MLP classifier for natural language inference.The implementation uses two-layer BiLSTMs with shortcut connections for inference.
  • Attention enhancements: Fully-aware attention is added to ESIM by replacing standard attention and using history-of-word representations.The multi-level variant also changes the inference BiLSTM input while keeping parameter size similar to or lower than standard ESIM.
  • Results: Fully-aware attention produces the biggest improvement in ESIM accuracy, and multi-level fusion provides further improvement on MultiNLI.Experiments with and without CoVe show similar observations.
  • Conclusion: The natural-language-inference experiments support the conclusion that using all levels of understanding together is crucial for better text understanding.
  • Adversarial analysis: On AddOneSent, FusionNet is not confused by most questions that BiDAF answers correctly.Among the 3.3% answered correctly by BiDAF but not FusionNet, approximately 1.6% involve the added sentence, 1.2% differ slightly from the ground truth, and 0.5% are completely incorrect.

F.1 FUSIONNET ANSWERS CORRECTLY WHILE BIDAF IS INCORRECT

FusionNet answers several adversarial or difficult questions correctly where BiDAF fails, but both models remain vulnerable when answers require broader context understanding or common knowledge.

  • FusionNet correctly predicts effective planning, while BiDAF predicts “Confusion.”
  • FusionNet correctly identifies New York Times as the publication, while BiDAF predicts “The Start Industries.”
  • FusionNet correctly predicts 587,000 square kilometres, while BiDAF predicts 187000.
  • FusionNet correctly answers “contemporary accounts were exaggerations” and identifies the church conference, whereas BiDAF gives incorrect alternatives.
  • Both models are confused by difficult or adversarial contexts, including a Rhine question and a question requiring common knowledge about political interference.
  • Both models fail when answering requires whole-context interpretation or common knowledge, including identifying E.I. du Pont and comparing the ages of empires.

G MULTI-LEVEL ATTENTION VISUALIZATION

The visualization shows that low- and high-level attention capture different matches between adversarial context sentences and questions. Combining low-level attention with high-level attention helps reject a misleading lexical match.

  • Figures 9 and 10 visualize attention between an added adversarial sentence in the context and the question at multiple representation levels.
  • High-level attention matches “The proclamation of the Central Park abolished protestantism in Belgium” with a question about France, producing an incorrect answer.
  • Low-level attention distinguishes “in Belgium” from “in France,” showing why incorporating lower-level information can reduce the misleading match.
Loading 1711.07341v2…