Source-linked AI summary
ReasoNet: Learning to Stop Reading in Machine Comprehension
Yelong Shen, Po-Sen Huang, Jianfeng Gao, Weizhu Chen
TL;DR
Machine comprehension requires answering questions from documents despite variable reasoning difficulty and limited fixed-depth approaches. ReasoNet uses multi-turn attention with a learned termination decision, trained by reinforcement learning, and achieves superior results across unstructured and structured comprehension datasets.
Problem
Machine comprehension requires models to answer questions from documents, while fixed reasoning depth does not account for variation in question and document difficulty.
Method
ReasoNet repeatedly attends to documents and uses reinforcement learning to learn a termination state that dynamically decides whether to continue inference or answer.
Results
ReasoNet achieves superior results on CNN, Daily Mail, Stanford SQuAD, and Graph Reachability machine comprehension datasets.
Takeaways & Limitations
Dynamic termination allows ReasoNet to adapt inference depth to individual documents and queries across unstructured and structured tasks.
Takeaways & Limitations
Training is challenging because the discrete termination state is not connected to the final output and prevents direct use of canonical back-propagation.
Abstract
from arXiv · showhide
Teaching a computer to read and answer general questions pertaining to a document is a challenging yet unsolved problem. In this paper, we describe a novel neural network architecture called the Reasoning Network (ReasoNet) for machine comprehension tasks. ReasoNets make use of multiple turns to effectively exploit and then reason over the relation among queries, documents, and answers. Different from previous approaches using a fixed number of turns during inference, ReasoNets introduce a termination state to relax this constraint on the reasoning depth. With the use of reinforcement learning, ReasoNets can dynamically determine whether to continue the comprehension process after digesting intermediate results, or to terminate reading when it concludes that existing information is adequate to produce an answer. ReasoNets have achieved exceptional performance in machine comprehension datasets, including unstructured CNN and Daily Mail datasets, the Stanford SQuAD dataset, and a structured Graph Reachability dataset.
1 INTRODUCTION
Machine comprehension requires models to infer answers from natural-language documents, but existing reasoning depth is fixed despite large variation in question and document difficulty. ReasoNet addresses this with dynamically learned termination and reinforcement-learning training.
- Motivation: Machine reading comprehension asks models to answer questions from provided passages, requiring thorough document understanding and sophisticated inference.
- Prior approaches: Single-turn attention models focus on query-relevant document regions, whereas multi-turn models repeatedly infer relations among queries, documents, and answers.
- Research gap: Fixed-hop multi-turn models ignore that question and document difficulty varies, although readers may stop when existing information is sufficient or continue reading.
- ReasoNet: ReasoNet dynamically decides whether to continue inference or terminate, modeling reasoning turns from both the document and query.
- Training: Reinforcement learning trains ReasoNet because its discrete termination state is not directly compatible with canonical back-propagation.
2 RELATED WORK
Prior cloze-style comprehension models use either single-turn attention or iterative multi-turn reasoning. ReasoNet extends multi-turn reasoning with a termination module that adapts inference depth to document and query complexity.
- Model categories: Cloze-style models are categorized by inference strategy into single-turn and multi-turn reasoning.
- Single-turn reasoning: Single-turn models attend to query-relevant document sections and score answer candidates using weighted document representations.
- Multi-turn reasoning: Multi-turn models revisit documents and combine query information with material digested during previous iterations.
- ReasoNet: ReasoNet combines attention-sum candidate aggregation and multiple turns with a termination module that selects whether to continue or stop.
- ReasoNet: The number of reasoning turns is dynamically modeled from document and query complexity rather than fixed in advance.
3 REASONING NETWORKS
ReasoNet repeatedly attends to an external memory while updating an internal question state, then stochastically chooses whether to continue or answer. Its termination and answer actions are trained with expected reward over inference episodes.
- Architecture: ReasoNet repeatedly reads a document with attention on different parts until a satisfying answer is found, using a stochastic inference process.
- Core components: External memory stores contextual word vectors, while the internal state represents the question and is updated by an RNN from attention vectors.
- Core components: At each step, the binary termination gate either stops inference and triggers answer selection or generates attention for the next state update.
- Inference process: An episode terminates at step T when tT = 1, with the answer action aT taken at that step; the termination step varies by instance.
- Training objective: The model parameters are trained by maximizing expected total reward, with reward received only at final termination when the answer is correct.
- Training details: The instance-dependent baseline averages rewards over possible episodes for each training instance, supporting different baselines as instance complexity varies.
4 EXPERIMENTS
ReasoNets are evaluated across natural-language comprehension and structured graph-reachability tasks, using attention-based multi-turn reasoning with dynamic termination. Results show strong performance and instance-dependent reasoning depth, while graph experiments indicate that longer or more difficult paths require more reasoning steps.
- Datasets: ReasoNets are evaluated on CNN, Daily Mail, SQuAD, and Graph Reachability datasets.The first three are natural-language comprehension benchmarks, while Graph Reachability tests structured inference.
- Model configuration: The model combines query and passage memories, projected-cosine attention, a GRU state controller, termination logistic regression, and an answer projection.The final attention vector concatenates query and passage attention, and the answer module predicts entity candidates from GRU outputs.
- Experimental settings: Tmax = 5 is used for CNN and Daily Mail experiments, with ADAM optimization, batch size 64, and gradient clipping within 0.001.CNN and Daily Mail use 101k and 151k vocabulary words, respectively, excluding entities and a placeholder marker.
- CNN and Daily Mail results: ReasoNet outperforms fixed-step multi-turn baselines on CNN and Daily Mail and obtains comparable results with AoA Reader on the CNN test set.AS Reader is treated as a special case with Tmax = 1, while ReasoNet dynamically selects reasoning steps for different test cases.
- CNN and Daily Mail results: Around 70% of CNN instances terminate at the last step, while an example shifts attention toward the correct clue before terminating with higher probability.The example initially attends to incorrect entities, then focuses on the right clue in the second and third steps; query attention remains on the placeholder token.
- SQuAD and Graph Reachability results: ReasoNet outperforms published SQuAD approaches, while graph experiments show that dynamic termination converges faster and harder instances require more reasoning steps.For Graph Reachability, ReasoNet and ReasoNet-Last outperform the single-turn ReasoNet-Tmax = 2 model; larger BFS-Steps require more reasoning steps.
5 CONCLUSION
ReasoNets dynamically decide whether to continue or terminate inference in machine comprehension tasks and achieve superior results across unstructured and structured datasets.
- ReasoNets dynamically decide whether to continue or terminate the inference process in machine comprehension tasks.
- The model achieves superior results on CNN, Daily Mail, Stanford SQuAD, and Graph Reachability datasets.