PaperlayerView exact revision on arXiv

Source-linked AI summary

Natural Language Inference over Interaction Space

Yichen Gong, Heng Luo, Jian Zhang

arXiv:1709.04348v2cs.CL

TL;DR

Natural Language Inference asks whether a hypothesis is entailed, contradicted, or neutral given a premise, requiring complex semantic understanding. The paper introduces Interactive Inference Network architectures that hierarchically extract features from interaction tensors, including the DIIN instantiation. DIIN achieves state-of-the-art performance on multiple datasets, while the full potential of interaction space remains open for exploration.

  • Problem

    NLI requires determining entailment, contradiction, or neutrality between sentence pairs involving lexical, compositional, and commonsense understanding.

  • Method

    Interactive Inference Network hierarchically extracts semantic features from word-by-word interaction tensors constructed from premise and hypothesis representations.

  • Results

    DIIN achieves new state-of-the-art performance on SNLI, MultiNLI, and Quora Question Pair datasets.

  • Takeaways & Limitations

    Interaction tensors contain semantic information for NLI, and denser representations support extracting richer semantic features.

  • Takeaways & Limitations

    The full potential of interaction space is not yet clear, and incorporating external commonsense knowledge remains a research goal.

Abstract

from arXiv · show

Natural Language Inference (NLI) task requires an agent to determine the logical relationship between a natural language premise and a natural language hypothesis. We introduce Interactive Inference Network (IIN), a novel class of neural network architectures that is able to achieve high-level understanding of the sentence pair by hierarchically extracting semantic features from interaction space. We show that an interaction tensor (attention weight) contains semantic information to solve natural language inference, and a denser interaction tensor contains richer semantic information. One instance of such architecture, Densely Interactive Inference Network (DIIN), demonstrates the state-of-the-art performance on large scale NLI copora and large-scale NLI alike corpus. It's noteworthy that DIIN achieve a greater than 20% error reduction on the challenging Multi-Genre NLI (MultiNLI) dataset with respect to the strongest published system.

1 INTRODUCTION

NLI requires reasoning over entailment, contradiction, and neutrality, including lexical, compositional, and commonsense phenomena. The paper proposes interaction-space architectures that extract semantic features from dense sentence-pair alignments and achieve state-of-the-art results.

  • 1 INTRODUCTION: NLI classifies the logical relationship between a premise and hypothesis as entailment, contradiction, or neutral.MultiNLI examples illustrate the full complexity of lexical and compositional semantics involved.
  • 1 INTRODUCTION: Attention weights encode cross-sentence word relationships as alignment matrices, while multi-head attention represents multiple alignments.The paper extends this idea toward denser, higher-order interaction representations.
  • 1 INTRODUCTION: The interaction tensor is a word-by-word, dimension-wise alignment representation that encodes high-order relationships between sentence pairs.Experiments associate its rich semantic features with performance on paraphrases, antonyms, and overlapping words.
  • 1 INTRODUCTION: Interactive Inference Network (IIN) is introduced as a framework for extracting semantic features from interaction space end-to-end.The paper presents DIIN as one instance of this framework and evaluates it across NLI and related tasks.
  • 1 INTRODUCTION: DIIN achieves new state-of-the-art performance on SNLI, MultiNLI, and the Quora Question Pair dataset.The Quora dataset contains over 400k real-world question pairs, extending evaluation beyond standard NLI corpora.

2 RELATED WORK

Prior NLI research used conventional methods and benefited from large annotated datasets such as SNLI. Representation learning techniques, attention variants, memory, parsing, and convolutional structures broadened the modeling approaches.

  • 2 RELATED WORK: Early NLI work relied mainly on conventional methods and small-scale datasets, while SNLI provided 570k human-annotated sentence pairs.SNLI became an important benchmark for sentence understanding.
  • 2 RELATED WORK: Researchers studied attention, memory, and parse structure as representation-learning techniques for NLI on SNLI.These methods targeted sentence understanding from complementary structural and contextual representations.
  • 2 RELATED WORK: Attention variants included hard, self-, multi-hop, bidirectional, and multi-head attention for increasingly complex tasks.Before this work, neural attention was mainly used to align representations and focus on selected parts.
  • 2 RELATED WORK: Convolutional structures were explored in NLU because they offer parallel computation and support multiple granularities.Applications included machine translation, sentence classification, text matching, and sentiment analysis.

3 MODEL

IIN is a hierarchical, five-component architecture that extracts semantic features from an interaction tensor between premise and hypothesis representations. DIIN instantiates this framework with dense interactions and convolutional feature extraction for NLI classification.

  • Interactive Inference Network: IIN comprises embedding, encoding, interaction, feature extraction, and output layers in a hierarchical multi-stage process.The framework permits different implementations for each component, including non-neural alternatives in principle.
  • Interactive Inference Network: The embedding layer maps words or phrases to vectors and constructs sentence representation matrices.Possible inputs include pretrained word representations and linguistic preprocessing features.
  • Interactive Inference Network: The encoding layer incorporates context and additional features using mechanisms such as bidirectional recurrence, recursive networks, or self-attention.Different encoder components can be combined to improve sentence-matrix representations.
  • Interactive Inference Network: The interaction layer builds a word-by-word tensor from premise and hypothesis matrices using pairwise similarities or outer products.Outer products provide a route to higher-order interaction tensors.
  • Densely Interactive Inference Network: DIIN is presented as a relatively simple IIN instantiation that achieves state-of-the-art performance on multiple datasets.The paper also notes that the architecture can operate without recurrent structure in its reported comparison.
  • Densely Interactive Inference Network: DIIN combines word, character, and syntactical features, then applies highway encoding, self-attention, a fuse gate, dense interaction, and DenseNet feature extraction.Its output layer classifies the flattened features into three NLI classes; DenseNet is selected partly for parameter efficiency.

4 EXPERIMENTS

The experiments evaluate DIIN on SNLI, MultiNLI, and Quora question pairs, combining benchmark comparisons, ablations, error analysis, and hidden-representation visualization. DIIN achieves strong results without recurrent encoders, while analyses identify the contributions and behavior of its interaction-space features.

  • Experimental setting: The evaluation covers accuracy on SNLI, MultiNLI, and Quora question pairs, with Quora duplicates interpreted as entailment and non-duplicates as neutral.The datasets contain 570k SNLI pairs, 433k MultiNLI pairs, and over 400k Quora question pairs.
  • Quantitative evaluation: DIIN achieves 80.0% on MultiNLI, exceeding the current state-of-the-art performance by more than 5%.The authors also report that out-of-domain performance is consistently lower than in-domain performance, partly because parameters were selected using in-domain development accuracy.
  • Ablation study: Removing the exact-match feature reduces MultiNLI development accuracy to 78.2 matched and 78.0 mismatched.The ablation associates the exact-match feature with improved sentence understanding, consistent with observations from reading comprehension.
  • Ablation study: Removing the convolutional feature extractor yields 73.2 matched and 73.6 mismatched, while removing the encoding layer yields 73.5 matched and 73.2 mismatched.Without convolution, the model becomes sentence-encoding based and uses [p; h; |p −h|; p ◦h] for classification.
  • Error analysis: DIIN performs better on WORD OVERLAP, ANTO, LONG SENTENCE, PARAPHRASE, and BELIEF examples by a large margin.The authors report that removing exact-match features lowers ANTO accuracy by about 10%, while performance on PARAPHRASE does not worsen.
  • Visualization: Interaction-tensor channels activate for different word and phrase pairings, while dense-block feature maps reveal different semantic features across positions and channels.The visualization includes activations associated with “referendum,” “supreme court,” “ban,” and “powerless over.”

5 CONCLUSION AND FUTURE WORK

The paper concludes that interaction tensors contain semantic information for NLI, while DIIN achieves state-of-the-art performance across multiple datasets. It identifies the full potential of interaction space as an open direction for future work.

  • IIN extracts semantic features from interaction tensors end-to-end for NLI and NLI-like tasks.
  • DIIN achieves state-of-the-art performance on multiple datasets.
  • The full potential of interaction space remains unclear, motivating further exploration and possible integration of external commonsense knowledge.

A SUPPLEMENTARY MATERIAL

The supplementary study examines how dimensionality and parameter count affect performance in the model’s highway-network variant. Higher dimensionality helps below a threshold, but additional parameters do not help beyond it, with different needs for SNLI and MultiNLI.

  • The dimensionality study varies model dimension, which also determines the interaction tensor’s channel number, and reports parameter counts.The default dimensionality is 448: 300 word-embedding, 100 character, 47 part-of-speech, and 1 exact-match feature dimensions.
  • The highway-network variant enables different output sizes from the input dimensionality.
  • Higher dimensionality improves performance below a threshold, while larger dimensionality and parameter counts do not help beyond it.
  • 250D suffices for good SNLI performance, whereas 350D is required for competitive MultiNLI performance.
  • The authors fail to reproduce their best MultiNLI performance with the new structure, indicating that the additional highway layer does not help convergence.
Loading 1709.04348v2…