PaperlayerView exact revision on arXiv

Source-linked AI summary

Iterative Alternating Neural Attention for Machine Reading

Alessandro Sordoni, Philip Bachman, Adam Trischler, Yoshua Bengio

arXiv:1606.02245v4cs.CLcs.NE

TL;DR

Machine comprehension asks models to answer questions about text, and Cloze-style benchmarks provide a setting for evaluating that task. The paper uses iterative alternating attention to explore query and document information without collapsing the query into one vector. It reports state-of-the-art results on CNN and CBT-CN, while CBT-NE test performance is on par with the best baseline.

  • Problem

    Machine comprehension requires systems to answer questions about a text, and existing models can compress the query into a single representation rather than explore it in detail.

    Source: §1 · pp. 1–2

  • Method

    The model iteratively alternates attention between query and document, using each search result to guide the next step before predicting the answer.

    Source: §1 · p. 1, §3 · p. 2

  • Results

    The model reports state-of-the-art results on CNN and CBT-CN; on CBT-NE, its test accuracy is on par with the best baseline.

    Source: §5.1 · p. 5, §5.2 · p. 6, §7 · p. 8

  • Takeaways & Limitations

    The iterative process continually refines the model’s view of the query and document while aggregating information for answer prediction.

    Source: §7 · p. 8

    Across CBT and CNN examples, most questions can be answered from words neighboring the placeholder, so these datasets do not require deeper query exploration.

    Source: §5.3 · p. 7

Abstract

from arXiv · show

We propose a novel neural attention architecture to tackle machine comprehension tasks, such as answering Cloze-style queries with respect to a document. Unlike previous models, we do not collapse the query into a single vector, instead we deploy an iterative alternating attention mechanism that allows a fine-grained exploration of both the query and the document. Our model outperforms state-of-the-art baselines in standard machine comprehension benchmarks such as CNN news articles and the Children's Book Test (CBT) dataset.

1 Introduction

Machine comprehension research advanced through deep learning and Cloze-style benchmarks, which support rapid cycles of model development and evaluation. The paper introduces an iterative alternating-attention model that explores query and document information without compressing the query to one representation.

1 Introduction

  • Deep learning techniques support language processing beyond surface forms and generalization from relatively small amounts of labeled data.Cloze-style benchmarks enable fast integration loops between model conception and evaluation (Hill et al., 2015; Hermann et al., 2015).Source: §1 · p. 1
  • Cloze-style questions remove a word from a statement or document-derived query, requiring prediction of the missing word from context (Hill et al., 2015; Hermann et al., 2015).Document-derived questions may draw on broader context, while questions formed from summaries can demand comprehension of the original text.Source: §1 · p. 1
  • The model alternates attention between query and document, preserving a fine-grained query representation within fixed computation time.Each document match feeds the next inference step, and the process summary is used to predict the answer.Source: §1 · pp. 1–2

2 Task Description

The task evaluates predicting a missing word from a document, using CBT excerpts and CNN news articles as distinct query-document settings.

2 Task Description

  • CBT uses 20-sentence book excerpts as documents and forms queries from the 21st sentence by replacing one word with a placeholder.The evaluation focuses on CBT-NE and CBT-CN; the verb and preposition subsets are described as relatively simple (Hill et al., 2015).Source: §2 · p. 2
  • CNN queries replace a named entity in a bullet-point summary, paired with the full news article as the document.Source: §2 · p. 2
  • Each example pairs a query and document with a candidate-answer set containing the correct answer.The task replaces the correct answer in the query with a placeholder.Source: §2 · p. 2
  • Table 1 reports statistics for CBT-NE, CBT-CN, and CNN.Source: Table 1 · p. 2

3 Alternating Iterative Attention

The model encodes query and document words, then iteratively alternates attention over both to gather evidence for answer prediction. A gated inference GRU updates its state from these reads, and final document attention weights estimate answer probability.

3 Alternating Iterative Attention

  • The workflow encodes the query and document, alternates attention over them during inference, then uses gathered information to predict the answer.The repeated attentive reads are conditioned on the previous inference state.Source: §3 · p. 2, §3.2 · p. 3

3.1 Bidirectional Encoding

  • Bidirectional GRUs map each input token to a contextual representation using both preceding and future sequence information.The forward and reverse hidden states are concatenated for each token.Source: §3.1 · pp. 2–3

3.2 Iterative Alternating Attention

  • At each inference step, the model attends to the query, then uses that query glimpse and the previous state to select relevant document information.A gate can reset the current query and document glimpses when attention is nearly uniform or unfruitful, and their multiplicative interaction helps assess their match.Source: §3.2 · pp. 3–4

3.3 Answer Prediction

  • After a fixed number of steps, the final document attention weights estimate answer probability using the positions where that answer occurs in the document.The model maximizes log probability over the training corpus.Source: §3.3 · p. 4

4 Training Details

Training uses ADAM with validation-based learning-rate decay and a hyperparameter search. The paper reports computation comparable to prior bidirectional-encoder models, which it outperforms on the tested datasets.

Training Details

  • Training used ADAM with initial learning rate 0.001 and batch size 32, decaying the rate when validation accuracy failed to improve after a half-epoch.The half-epoch was 2000 batches for CBT and 5000 for CNN.Source: §4 · p. 4
  • Table 2 reports results for CBT-NE and CBT-CN, with results marked 1 from Hill et al. (2015) and marked 2 from Kadlec et al. (2016).Source: Table 2 · p. 5
  • The selected settings were embedding regularization 0.0001, T = 8, d = 384, h = 128, and s = 512.These settings were reported to work robustly across datasets.Source: §4 · pp. 4–5

Computational Complexity

  • The method’s computation cost is comparable to prior bidirectional-encoder models, while outperforming them on the datasets tested.The attention mechanism uses a fixed T = 8 steps, far fewer than typical document or query lengths; repeated softmaxes over about 1000 locations are typically fast on recent GPUs.Source: §4 · p. 5

5 Results

The model reports benchmark results on CBT-CN, CBT-NE, and CNN, including gains over prior systems on CBT-CN and CNN. Discussion examines how attention evolves during inference and notes limits related to local-context questions and a fixed number of steps.

5.1 CBT

  • On CBT-CN, the model gains 3.6 validation and 5.6 test accuracy points over the best baseline, AS Reader.On CBT-NE, its 1.4-point validation gain does not translate to better test performance; it ties the best baseline.Source: §5.1 · p. 5
  • The five-model ensemble reaches 76.9 validation and 72.0 test accuracy on CBT-NE, and 74.1 validation and 71.0 test on CBT-CN.The authors report that it outperforms the AS Reader ensemble on both CBT settings.Source: §5.1 · pp. 5–6

5.2 CNN

  • On CNN, the model improves state-of-the-art accuracy over the AS Reader by 4 percentage points absolute on validation and 3.4 on test.It slightly improves over Stanford AR by 0.2 percentage points on validation and 0.9 on test; the authors note that different training and initialization strategies may influence this comparison.Source: §5.2 · p. 6, Table 3 · p. 6
  • Ensembles achieve state-of-the-art test performance, with 75.2 on validation and 76.1 on test, respectively, outperforming previously published results.The passage does not specify the metric label.Source: §5.2 · p. 6
  • CNN category analysis suggests neural models perform similarly on local-context categories, while iterative alternating attention seems better on Ambiguous/Hard examples.Coreference errors account for about 25% of the corpus and may make questions unanswerable; the authors say this could place the 76.1% ensemble near-optimal if the estimate is accurate.Source: §5.2 · pp. 6–7, Table 4 · p. 7

5.3 Discussion

  • In an illustrated CNN example, attention shifts from the query placeholder toward contextual terms as document attention becomes more confident about “Italian.”The query attention remains at the same location while document attention evolves across inference timesteps.Source: §5.3 · p. 7, Figure 2 · p. 7
  • Across CBT and CNN examples, query attention often stays near the placeholder, and most questions can be answered using neighboring words; the authors call for harder examples to require deeper query exploration.They fix the number of inference steps at T=8, which works consistently across tested datasets, and hypothesize that harder examples may benefit from more steps and easier ones from fewer; dynamically selecting steps is proposed as an extension.Source: §5.3 · pp. 7–8

6 Related Works

The model builds on neural attention and iterative memory-network ideas while distinguishing itself through separate, repeated attention to query components and document content. It also contrasts this approach with prior query-attention methods that process documents repeatedly or compress queries into one vector.

6 Related Works

  • Attention models use learned policies to access memory states; here, that memory is the set of document and query contextual encodings.The model repeatedly integrates query and document attention rather than compressing the query into one representation.Source: §6 · p. 8
  • Unlike Kadlec et al. (2016), the model alternates query and document attention instead of using one attention step and a single-vector query representation.The repeated integration lets it dynamically select query components and focus on document parts salient to them.Source: §6 · p. 8
  • Unlike Hermann et al. (2015), the model encodes the query and document once, then attends to those encodings in a fixed number of steps.Hermann et al. process the document once per query word, which may be computationally intractable for large documents; the proposed approach may be more scalable.Source: §6 · p. 8
  • Unlike Memory Networks, which iteratively update the query representation without separately attending to its components, the model uses a GRU update.The GRU gating made multiple attention steps possible and propagated the learning signal back to the first timestep.Source: §6 · p. 8

7 Conclusion

The paper concludes that its iterative neural attention model achieves state-of-the-art results on three datasets by refining its view of the query and document while aggregating information for answers. It identifies dynamic step selection and stochastic attention as future directions, and suggests information retrieval as another possible application.

7 Conclusion

  • State-of-the-art results on three datasets support the iterative neural attention model’s performance on machine comprehension tasks.The conclusion describes the architecture as tightly integrating successful ideas from prior machine-reading work.Source: §7 · p. 8
  • The alternating attention mechanism continually refines the model’s view of the query and document while aggregating information needed to answer a query.The model was applied to machine comprehension tasks.Source: §7 · p. 8
  • Future directions include selecting the number of inference steps per example, using stochastic attention, and applying the model to information retrieval.The paper presents these as research directions and describes the model as suspected to be fully general.Source: §7 · p. 8

Related papers

Loading 1606.02245v4…