PaperlayerView exact revision on arXiv

Source-linked AI summary

Linguistic Knowledge as Memory for Recurrent Neural Networks

Bhuwan Dhingra, Zhilin Yang, William W. Cohen, Ruslan Salakhutdinov

arXiv:1703.02620v1cs.CL

TL;DR

Long-range dependencies are difficult for RNNs to learn without explicit signals. This paper uses typed linguistic links as memory in recurrent models, achieving state-of-the-art results on three text-comprehension benchmarks.

  • Problem

    RNNs and memory-augmented models can struggle to learn long-range dependencies without an explicit signal for which distant elements to use.

    Source: §1 · p. 1

  • Method

    MAGE-RNN augments sequences with typed linguistic edges, decomposes the resulting graph into forward and backward DAGs, and uses those links as explicit memory in recurrent models.

    Source: §1 · pp. 1–2, §3.1 · p. 3

  • Results

    MAGE-GRU achieves state-of-the-art performance on bAbi QA, LAMBADA, and CNN, with consistent gains over vanilla GRU and a coreference-input-feature baseline.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The learned representations encode fine-grained information about entities across a document.

    Source: §1 · p. 2

    The authors hypothesize that explicit linguistic knowledge will benefit data-scarce settings most, while purely data-driven learning is feasible when training data is large.

    Source: §5 · p. 8

Abstract

from arXiv · show

Training recurrent neural networks to model long term dependencies is difficult. Hence, we propose to use external linguistic knowledge as an explicit signal to inform the model which memories it should utilize. Specifically, external knowledge is used to augment a sequence with typed edges between arbitrarily distant elements, and the resulting graph is decomposed into directed acyclic subgraphs. We introduce a model that encodes such graphs as explicit memory in recurrent neural networks, and use it to model coreference relations in text. We apply our model to several text comprehension tasks and achieve new state-of-the-art results on all considered benchmarks, including CNN, bAbi, and LAMBADA. On the bAbi QA tasks, our model solves 15 out of the 20 tasks with only 1000 training examples per task. Analysis of the learned representations further demonstrates the ability of our model to encode fine-grained entity information across a document.

1. Introduction

Long-range dependencies are difficult for recurrent models to learn without an explicit signal. The paper uses symbolic linguistic relations to direct memory in a recurrent model, then applies it to coreference-aware text comprehension.

1. Introduction

  • RNNs are difficult to train for long-range dependencies, and even memory-augmented models may focus only on immediately preceding steps.The paper notes that LSTMs and GRUs extend recurrent models, but long-range training remains difficult.Source: §1 · p. 1
  • Coreference and hypernymy links can point to distant tokens, giving the RNN explicit locations for memories of linked elements.The linked element's representation is the memory content, while edge labels distinguish memory types.Source: §1 · pp. 1–2
  • MAGE-RNN decomposes typed-edge sequence graphs into two directed acyclic graphs and encodes them with separate representations for each edge type, touching each node once.With at most one incoming edge of each type per node, it reduces to a memory-augmented RNN whose access is determined by symbolic signals.Source: §1 · p. 2
  • MAGE-GRU achieves state-of-the-art performance on bAbi QA, LAMBADA, and CNN, and consistently improves over vanilla GRU and a coreference-input-feature baseline.The learned representations also encode fine-grained entity information across a document.Source: §1 · p. 2

2. Related Work

Related work includes graph and tree architectures for structured inputs, as well as methods that use coreference or other symbolic knowledge. The paper distinguishes its approach through typed edges integrated into recurrent representations.

2. Related Work

  • Graph neural networks update nodes from neighbors over multiple time steps, with complexity O(NT), versus O(N) for RNNs; full propagation requires T at least the graph width.Figure 1 depicts coreference, hypo/hypernymy, and sequential links as distinct edge types.Source: §2 · p. 2, Figure 1 · p. 3
  • Tree-Structured LSTMs focus on dependency parses and ignore sequence order, whereas MAGE-RNN incorporates sequential links and multiple edge types.MAGE-RNN is described as an extension of Tree-Structured LSTMs.Source: §2 · p. 2
  • DAG-based image RNNs do not handle typed edges; MAGE-RNN uses separate parameter matrices and split hidden states for them.The paper identifies these design choices as key to improved performance.Source: §2 · p. 2
  • Prior coreference features improved reading-comprehension models, while MAGE-RNN modifies the reader structure and produces interpretable representations; its chains can interact unlike recurrent entity networks.Local context around a mention can inform its coreference representation and vice versa.Source: §2 · p. 2
  • Prior knowledge-integrating language models use RNN outputs to select between full and restricted vocabularies, a strategy specific to next-token prediction.This paper instead targets more general text representations; the cited prior work is Yang et al. (2016) and Ahn et al. (2016).Source: §2 · pp. 2–3

3. Methods

The methods turn typed relations between sequence elements into acyclic graphs and adapt recurrent updates to use graph-structured memory. For multiple sequences, a random ordering keeps graph traversal cost at that of separate processing.

3.1. From Sequences to DAGs

  • Typed relation edges are added to sequential links, then original and inverse edges are split by sequence order into forward and backward DAGs with available topological orders.Source: §3.1 · p. 3

3.2. MAGE-GRUs

  • MAGE-GRU processes forward and backward DAGs with separate networks, updating edge-specific hidden states from all incoming edges at each node.Source: §3.2 · p. 3
  • Concatenated edge-specific states form each timestep's output, while prior outputs M_t are the recurrent model's available memory.Source: §3.2 · p. 4, Figure 2 · p. 4
  • For relations with at most one incoming edge of a type, the DAG decomposes into independent chains whose updates combine into a regular GRU update.The recurrent state is replaced by a graph-derived state for the update.Source: §3.2 · p. 4

3.3. Multiple Sequences

  • For multiple sequences, one random permutation yields forward and backward DAGs that traverse each edge twice without extra cost versus separate processing; multilayer extensions permit information flow along arbitrary graph paths.Source: §3.3 · p. 4

4. Experiments

The experiments evaluate coreference-augmented recurrent readers on text-comprehension benchmarks, comparing graph-based memory with recurrent and GA baselines. Results include standard benchmarks, a longer-distance bAbi variant, and analyses of learned representations.

4.1. Text Comprehension with Coreference

  • The reader uses bidirectional GRUs for document and query representations; experiments replace the GA Reader’s bi-GRUs with MAGE-GRU and compare against a one-hot coreference-feature baseline.The one-hot baseline appends a vector indicating each token’s coreference chain, if any, to the GRU input.Source: §4.1 · p. 5

4.2. Performance Comparison

  • MAGE improves bAbi results over strong baselines, raises LAMBADA’s GA performance by 2%, and raises CNN GA test performance by 0.7%, setting new state-of-the-art on the latter two datasets.The bAbi results also report new state-of-the-art performance; on CNN, GA+MAGE adds no information beyond what is already available in the anonymized dataset.Source: §4.2 · pp. 5, 7, 8
  • 72.8% to 0.3%: on bAbi-Mix Task 3, MAGE reduces the error rate for reasoning over three supporting facts; both MAGE variants outperform QRN across the mixed-story tasks.bAbi-Mix interleaves two independent stories in random order, extending the distance over which entity information must be tracked.Source: §4.2 · pp. 5–6, Table 2 · p. 6
  • GA’s candidate-answer distribution stays roughly constant across the story, whereas GA+MAGE’s distribution reflects the story state at each sentence boundary.The analysis links GA+MAGE’s strongest gains to tasks requiring entity-state tracking across a story.Source: §4.2 · p. 6, Figure 4 · p. 7
  • Adding coreference edges improves the simple bi-GRU architecture by 1.7%, while the one-hot baseline does not improve it.On the multi-layer GA architecture, the coreference edges improve performance by 2%; the one-hot baseline performs comparably there.Source: §4.2 · p. 7
  • On 100 manually labeled validation instances, GA+MAGE consistently outperforms GA across categories, with the largest gains for single-name cue, semantic trigger, coreference, and external knowledge labels.The authors caution that the small sample makes strong conclusions difficult.Source: Table 4 · p. 7, §4.2 · p. 7
  • For LAMBADA passages averaging 75 tokens, the authors suggest multi-layer models can propagate reference information when given an explicit input signal.The one-hot baseline performs comparably on this dataset.Source: §4.2 · p. 7

5. Conclusions

MAGE-RNN uses typed linguistic-relation edges as explicit memory in recurrent networks, improving results across three benchmarks. The authors identify data-scarce settings and broader linguistic knowledge as directions for further study.

5. Conclusions

  • Across three widely studied benchmarks, MAGE-RNN consistently improves results for simple and sophisticated architectures, with its best results setting a new state of the art on all three.The conclusion reports no benchmark-specific values here.Source: §5 · p. 8
  • MAGE-RNN adds typed relation edges to chain-structured RNNs, parameterizes each edge type separately, and maintains distinct edge-specific hidden states at each node.The graph structure dictates memory access.Source: §5 · p. 8
  • The authors hypothesize explicit linguistic knowledge may help most when data is scarce, and note that coreference and entity-linking tools are widely available.They describe models that exploit those tools as valuable.Source: §5 · p. 8
  • Future work proposed includes attention over edge types and testing additional knowledge sources, including dependency parses, semantic roles, frames, ontologies, and databases.The authors describe MAGE-RNN as a framework capable of integrating these sources.Source: §5 · p. 8

Related papers

Loading 1703.02620v1…