Source-linked AI summary
Long Short-Term Memory-Networks for Machine Reading
Jianpeng Cheng, Li Dong, Mirella Lapata
TL;DR
The paper addresses recurrent networks’ limited ability to handle structured input and long sequences. It proposes LSTMN, which embeds an attention-addressed memory network within an incremental LSTM reader and extends it to encoder-decoder settings. Across language modeling, sentiment analysis, and natural language inference, the model achieves performance comparable to or better than state-of-the-art systems.
Problem
The paper addresses recurrent networks’ difficulty representing structured input and retaining information under recursive compression over long sequences.
Method
LSTMN replaces the LSTM memory cell with an internal memory network that stores token representations and uses neural attention for shallow relation reasoning, including in encoder-decoder models.
Results
Across language modeling, sentiment analysis, and natural language inference, LSTMN achieves performance comparable to or better than state-of-the-art models.
Takeaways & Limitations
The model provides an incremental recurrent reader that combines adaptive memory usage with weakly supervised induction of token relations.
Takeaways & Limitations
The experiments focus on LSTMs, while future work aims to develop architectures that reason over nested structures and learn compositionality with weak or indirect supervision.
Abstract
from arXiv · showhide
In this paper we address the question of how to render sequence-level networks better at handling structured input. We propose a machine reading simulator which processes text incrementally from left to right and performs shallow reasoning with memory and attention. The reader extends the Long Short-Term Memory architecture with a memory network in place of a single memory cell. This enables adaptive memory usage during recurrence with neural attention, offering a way to weakly induce relations among tokens. The system is initially designed to process a single sequence but we also demonstrate how to integrate it with an encoder-decoder architecture. Experiments on language modeling, sentiment analysis, and natural language inference show that our model matches or outperforms the state of the art.
1 Introduction
The paper develops a general-purpose machine reader that processes text incrementally while addressing recurrent networks’ difficulty with long sequences and structured input. LSTMN combines recurrent processing with internal memory, attention, and shallow relation induction, achieving competitive results across three tasks.
- 1 Introduction: The paper asks how sequence-level networks can induce latent relations during incremental text processing.It frames machine reading as a general-purpose alternative to focusing on a single task.
- 1 Introduction: The proposed reader processes text from left to right, reflecting the incremental nature of human language comprehension.The model is designed as a reading simulator rather than a task-specific system.
- 1 Introduction: Recurrent networks face training difficulties, inefficient memory use across sequence lengths, and limited mechanisms for handling input structure.These limitations motivate a model that preserves recurrent, incremental, generative processing while improving memory and structural handling.
- 1 Introduction: LSTMN replaces a single recurrent memory cell with an internal memory network that uses attention to connect current processing with past memories.This design supports adaptive memory usage and shallow reasoning over token relations.
- 1 Introduction: The model learns which previous tokens relate to the current token and induces undirected token relations as an intermediate representation-learning step.The attention mechanism is trained end-to-end with task-specific supervision.
- 1 Introduction: LSTMN performs comparably to or better than state-of-the-art models in language modeling, sentiment analysis, and natural language inference.It also outperforms vanilla LSTMs across these evaluations.
2 Related Work
The related work situates LSTMN among structured and memory-augmented neural models. Its distinctive approach keeps memory and soft attention inside the recurrent architecture to induce differentiable token relations.
- 2 Related Work: LSTMN is a recurrent machine reader with incremental processing and shallow structure reasoning over input streams.It is presented as a sequence-processing model rather than a task-specific architecture.
- 2 Related Work: LSTMN adds memory and attention within a sequence encoder to uncover lexical relations between tokens.This places the relational mechanism inside the larger representation-learning network.
- 2 Related Work: Prior work introduced structural bias through recursive networks, modular recurrent models, and external memory systems.These approaches target improved representation of structured input or longer contexts.
- 2 Related Work: Unlike shift-reduce models with hard intermediate decisions, LSTMN uses soft, differentiable attention to induce token relations.The relations are learned as part of a larger neural network.
3 The Machine Reader
The LSTMN extends an LSTM with an attention-addressed memory tape, preserving incremental processing while enabling shallow relation analysis over tokens.
- 3 The Machine Reader: The machine reader processes structured input incrementally while retaining the recurrent network’s left-to-right behavior.Its core is an LSTM unit with an extended memory tape that simulates a human memory span.
- 3 The Machine Reader: An LSTM’s single-state recursive compression can fail to preserve information in long sequences and lacks explicit structure modeling.Sequential aggregation provides no explicit mechanism for reasoning over structure or modeling token relations.
- 3 The Machine Reader: LSTMN replaces the LSTM memory cell with a memory network whose slots store contextual representations of input tokens.The memory grows with time until reaching an upper bound defined by the memory span.
- 3 The Machine Reader: At each time step, attention computes relations between the current input and previous hidden states, producing adaptive summaries for recurrent updates.These summaries are used to update both the memory and hidden representations.
- 3 The Machine Reader: Attention in LSTMN induces soft, differentiable token relations that are optimized within a downstream representation-learning network.The paper also describes stacking memory and hidden layers as a possible route toward more structured relational reasoning.
4 Modeling Two Sequences with LSTMN
LSTMN can be integrated with encoder-decoder models for two-sequence tasks by combining intra-sequence reasoning with source-target inter-attention.
- 4 Modeling Two Sequences with LSTMN: The encoder-decoder integration addresses tasks that model a source sequence and a target sequence, such as machine translation and textual entailment.The LSTMN supplies intra-relation reasoning while the encoder-decoder supplies inter-alignment between sequences.
- Shallow Attention Fusion: Shallow attention fusion uses LSTMN modules for both encoder and decoder, while decoder attention aligns target tokens with the source.The inter-attention is triggered as the decoder reads each target token.
- Deep Attention Fusion: Deep attention fusion combines inter- and intra-attention when computing recurrent state updates.Its target memory stores source inter-alignment, target intra-relations, and new input information through gating.
- Deep Attention Fusion: Deep fusion recurrently stores the inter-alignment vector in target memory so the target network can review source information.This recurrent storage is the major architectural difference from shallow fusion.
5 Experiments
The experiments evaluate LSTMN on language modeling, sentiment analysis, and natural language inference, with comparisons against recurrent, recursive, convolutional, and other baselines. Across tasks, LSTMNs outperform LSTM baselines or achieve comparable state-of-the-art performance, while attention visualizations show learned lexical and sentiment relations.
- Experimental Setup: The experiments evaluate LSTMN on language modeling, sentiment analysis, and natural language inference.The study uses task-specific supervision for all three tasks.
- 5.1 Language Modeling: Single-layer LSTMN outperforms KN5, RNN, and LSTM by a significant margin on Penn Treebank language modeling.Among deep architectures, the three-layer LSTMN performs best.
- 5.1 Language Modeling: In language modeling, attention focuses mainly on recent memories but also captures some valid lexical relations.Figure 4 visualizes attention scores for sentences sampled from the Penn Treebank validation set.
- 5.2 Sentiment Analysis: LSTMNs outperform LSTM baselines on fine-grained and binary sentiment classification while achieving accuracy comparable to state of the art.The two-layer LSTMN performs close to the best system, T-CNN, on both tasks.
- 5.2 Sentiment Analysis: In sentiment analysis, intra-attention associates sentiment-important words such as “though” with “fantastic” and “not” with “good.”The attention examples are shown for sentiment words in Figure 5.
- 5.3 Natural Language Inference: For natural language inference, deep fusion yields state-of-the-art performance and performs better than LSTMs with and without attention.The comparison includes independently encoded, shared, attention-based, and matching LSTM variants.
6 Conclusions
The paper introduces a machine reading simulator that embeds a memory network within LSTM recurrence to process structured input. Across three tasks, the model achieves performance comparable or superior to state of the art, while attention is trained through downstream supervision.
- The LSTMN embeds a memory network within a Long Short-Term Memory architecture to store contextual token representations without recursively compressing them.
- An intra-attention mechanism addresses memory slots and weakly induces undirected relations among tokens without direct attention supervision.
- Experiments across three tasks show performance comparable or superior to state of the art.
- The experiments focused on LSTMs, while the authors suggest the structure-aware approach can extend to other network types.