PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning applied to NLP

Marc Moreno Lopez, Jugal Kalita

arXiv:1703.03091v1cs.CL

TL;DR

CNNs have been highly successful in computer vision, but their application to NLP requires understanding how convolutional architectures represent language. This paper surveys CNN basics and variants, reviews their NLP applications, and concludes that results are promising but have not yet matched computer vision’s advances.

  • Problem

    CNNs are strongly associated with computer vision, while their effectiveness and uses in NLP remain an emerging research area.

  • Method

    The paper surveys CNN architectures and subtypes and reviews their applications across NLP and related language tasks.

  • Results

    Reviewed CNN applications achieve improvements over previous state-of-the-art techniques across tasks including event detection, dependency parsing, and speech recognition.

  • Takeaways & Limitations

    CNNs provide promising approaches for NLP by learning task-relevant features and reducing reliance on manually engineered resources.

  • Takeaways & Limitations

    Deep-learning results in NLP remain promising rather than as substantial as the breakthroughs achieved in computer vision.

Abstract

from arXiv · show

Convolutional Neural Network (CNNs) are typically associated with Computer Vision. CNNs are responsible for major breakthroughs in Image Classification and are the core of most Computer Vision systems today. More recently CNNs have been applied to problems in Natural Language Processing and gotten some interesting results. In this paper, we will try to explain the basics of CNNs, its different variations and how they have been applied to NLP.

I. INTRODUCTION

NLP applies deep learning to analyze human language, while CNNs adapt neural-network ideas by learning local textual features through convolution and pooling. In NLP, sentences and documents are represented as token matrices, and filters typically span full token vectors across short word windows.

  • NLP and deep learning: NLP analyzes and derives meaning from human language for tasks including summarization, translation, entity recognition, sentiment analysis, and speech recognition.Human language is challenging because it can be ambiguous and depends on slang, dialects, and social context.
  • CNN basics: CNNs use local convolutional connections and nonlinear activations instead of connecting every input neuron to every output neuron.Pooling layers commonly follow convolution and subsample the resulting feature maps.
  • CNN basics: Pooling produces fixed-size outputs and retains salient information while reducing dimensionality for classification.Fixed-size outputs allow variable-length sentences and variable-size filters to feed a classifier.
  • CNNs for NLP: NLP inputs are sentence or document matrices whose rows represent tokens using word embeddings or one-hot vectors.A 10-word sentence with 100-dimensional embeddings forms a 10x100 matrix.

II. MOTIVATION

The paper motivates CNNs for NLP by examining their use in text regression and their interpretability on limited data. It also highlights methods for identifying influential n-grams and nonlinear word interactions.

  • Text regression: The reviewed work develops a convolutional network for predicting movie box-office takings from reviews and movie attributes.It compares CNN efficiency with an ANN-based text-regression approach.
  • CNNs with limited data: CNNs can be trained on about a thousand movie-review examples when model design and regularization are carefully handled.The task predicts future movie box-office takings from critic reviews and movie attributes.
  • Interpretability: A method quantifies how text n-grams affect ANN prediction outputs, enabling identification of important textual inputs and nonlinear word interactions.The analysis examines these effects across different data instances.

III. TYPES OF DEEP NEURAL NETWORKS

Deep neural architectures differ in how they represent sequence information: recurrent networks reuse computations over time, bidirectional networks incorporate both contexts, and deep recurrent networks stack layers to increase capacity.

  • Recurrent neural network: Recurrent neural networks process each sequence element using previous computations, providing a memory of information calculated so far.Although they can theoretically use arbitrarily long sequences, practical performance is limited to looking back only a few steps.
  • Bidirectional RNN: Bidirectional RNNs compute outputs from both previous and future sequence elements by combining the hidden states of two stacked RNNs.This supports tasks such as predicting a missing word from left and right context.
  • Deep RNN: Deep bidirectional RNNs add multiple layers at each time step, increasing learning capacity while requiring substantial training data.They extend bidirectional recurrence by stacking layers per time step.

3) LSTM networks:

LSTMs modify recurrent hidden-state computation with memory cells that selectively retain or erase information, while recursive and dependency-based networks represent variable-length linguistic structures. These architectures support long-distance, semantic, and syntactic modeling for NLP tasks.

  • LSTM networks: LSTM cells use the previous state and current input to decide what information to retain or erase from memory.The resulting units are described as efficient for capturing long-term dependencies.
  • Recursive and dependency-based networks: Recursive neural networks repeatedly apply shared weights over tree structures to produce representations for phrases, sentences, or structured predictions.They are deep in structure because recursive connections are applied repeatedly.
  • Recursive and dependency-based networks: Dependency-based recurrent architectures model interactions between a head word and its children, combining convolution and pooling to capture semantic and structural information.They can represent variable-length sequences as fixed-length vectors and support joint learning with NLP tasks such as classification.
  • Dependency-based convolution: A dependency-based convolution model can stack LSTM processing with variable-size convolutional filters and max pooling to extract task-specific features.For documents, additional LSTM processing encodes dependencies across subsentences before convolution.

D. Dynamic k-max pooling neural network

Dynamic k-max pooling extends max pooling by retaining a dynamically selected number of salient features. CNN variants use this operation and multiple convolutional columns to model position-independent features, question aspects, or context-dependent sentence meaning.

  • Dynamic k-max pooling: Dynamic k-max pooling returns the k maximum values in sequence order, with k chosen dynamically from network or input properties.It generalizes max pooling both by retaining a subsequence rather than one value and by allowing the pooling size to vary.
  • Convolutional feature extraction: Convolutional filters scan sentence features across positions, and dynamic pooling produces feature maps that retain salient patterns while reducing positional dependence.Higher-level filters can capture relations between noncontinuous phrases far apart in the input.
  • Multi-column CNN: Multi-column CNNs share word embeddings while analyzing different question aspects, such as answer path, answer context, and answer type.For each candidate answer, the model predicts a score indicating whether it is correct.
  • Context-dependent CNN: A context-dependent CNN summarizes source and target phrase embeddings through repeated convolution and pooling into fixed-length vectors, then computes a matching score for translation.The score is added as a feature to a conventional statistical machine translation system.

1) A Re-ranking Model for Dependency Parser with Recursive Convolutional Neural Network:

Recursive convolutional networks build representations over dependency trees by composing each node with its children. Applied to parser reranking, RCNN improves parsing accuracy and outperforms state-of-the-art models on two benchmark datasets.

  • RCNN architecture: RCNN captures syntactic and compositional-semantic representations and supports k-ary parsing trees, making it suitable for dependency parsing.The architecture is proposed by Zhu et al. as a general recursive convolutional model.
  • RCNN architecture: Each RCNN unit combines a node with its children, pools the most informative features, and recursively produces a fixed-length root representation.The output at each node becomes input to its parent until the dependency tree is reduced to one vector.
  • Parsing application: RCNN improves the accuracy of a base parser when used for parsing reranking.The model makes parsing decisions using representations recursively constructed from dependency-tree structure.
  • Results: RCNN outperforms state-of-the-art models on two benchmark datasets.The cited passage reports this as the experimental result of the paper.

4) Dependency Sensitive Convolutional Neural Networks for Modeling Sentences and Documents:

The reviewed CNN variants target sentence and document modeling through hierarchical representations, dynamic pooling, recurrent features, and context-aware combination. Reported applications include event detection, relation classification, and models that avoid parser-dependent processing.

  • Dependency-sensitive CNN: DSCNN builds hierarchical sentence and document representations from pretrained word embeddings processed by LSTMs and convolution operators.It is presented as a general-purpose classification system for both sentences and documents.
  • Dependency-sensitive CNN: DSCNN does not rely on parsers or expensive phrase labeling, unlike recursive tree-based models.This design is described as avoiding restriction to sentence-level tasks.
  • Dynamic multi-pooling CNN: DMCNN uses dynamic multi-pooling because ordinary CNNs may miss valuable facts in multiple-event sentences.The model combines a word-representation model with a multi-pooling CNN for sentence-level clues.
  • Event detection: CNNs for event detection automatically learn sentence features, reducing dependence on supervised toolkits and manually engineered resources.The reported domain-adaptation advantage is attributed to mitigating preprocessing error propagation and using word embeddings for trigger candidates.
  • Relation classification: A combined CNN-RNN system uses extended middle context, bidirectional recurrent models with ranking loss, and voting to achieve state-of-the-art results on SemEval 2010.The extended context includes the relation arguments and the sentence regions surrounding and between them.

4) Comparing Convolutional Neural Networks to Traditional Models for Slot Filling:

CNNs are applied to slot filling, document-level entity linking, and translation selection by learning position-independent patterns, multi-granular topic semantics, and context-dependent phrase similarity. The reviewed systems combine CNN representations with task-specific context or complementary models.

  • Slot filling: CNNs recognize phrase patterns independently of sentence position and use word embeddings that reflect word similarity for relation classification.Adel et al. compare CNNs with pattern matching and support vector machines for slot filling.
  • Slot filling: Splitting context at relation arguments improves handling of slot-filling sentences and outperforms a state-of-the-art piecewise CNN.The work also reports genre as an important confounding variable and provides a benchmark for comparison.
  • Document modeling: Extended dynamic convolution represents documents in a low-dimensional space while preserving word and sentence order distinctions.The model learns sentence- and document-level filters hierarchically from lexical features to semantic concepts.
  • Translation selection: A context-dependent translation model uses CNNs to score phrase-pair similarity while encoding source-language context.It trains with curriculum learning by progressing from easy to medium and difficult examples.

2) Encoding Source Language with Convolutional Neural Network for Machine Translation:

CNN-based approaches encode source language by guiding convolution with target information or by extracting multiple aspects of questions. These models use relational and multi-column representations to support translation and question answering.

  • Target-guided source encoding: Target-guided convolution and gating summarize source information according to the target word being predicted.The architecture identifies relevant source-sentence regions and combines them with the full-sentence context.
  • Multi-aspect question analysis: Multi-column CNNs extract answer types, relations, and context information from question words using shared embeddings.Question paraphrases are also used in a multi-task learning setup to generalize to unseen words.
  • Relational question-answer modeling: Question-answer CNNs encode word matches as tunable embedding dimensions to capture interactions between the paired sentences.The model combines similarity scores, intermediate representations, and additional pair-specific features.

F. Speech recognition

CNNs are applied to speech recognition through frequency-oriented convolution, limited weight sharing, filter analysis, and end-to-end phoneme recognition. The reviewed approaches report improved recognition behavior and reveal how learned filters represent speech.

  • CNN architectures for speech recognition: CNNs improve speech recognition by convolving along frequency and using limited weight sharing to accommodate speech variability.The approach reports about 6-10% relative error reduction versus standard DNNs with similar numbers of weight parameters.
  • Efficiency and pooling: Limited weight sharing reduces pooling-layer units, model size, and computational complexity compared with full weight sharing.Performance is sensitive to pooling size but insensitive to overlap between pooling units.
  • Interpreting learned filters: Filter analysis on TIMIT indicates that first-layer CNN responses model the envelope of 2-4 ms sub-segmental speech signals.The study examines mean frequency responses associated with vowel inputs.
  • End-to-end phoneme recognition: An end-to-end system combines CNN frame-level classification with recurrent decoding and Connectionist Temporal Classification for phoneme sequences.The system uses mel-filter bank features and avoids a traditional Hidden Markov Model decoder.

4) Applying Convolutional Neural Networks Concepts to Hybrid NN-HMM Model for Speech Recognition:

CNNs are adapted to speech and relation classification by exploiting local spectral structure, pooling, distributed class representations, and ranking-based objectives. The reviewed systems reduce reliance on conventional architectures or handcrafted features while reporting strong task performance.

  • Hybrid NN-HMM speech recognition: Hybrid NN-HMM speech recognition uses local filtering and max-pooling in the frequency domain to normalize speaker-related spectral variation.The method aims to enforce feature locality and speaker invariance.
  • Relation classification: CR-CNN represents relation classes and text segments as distributed vectors, then scores them with a pairwise ranking loss.The ranking formulation reduces the impact of artificial classes.
  • Relation classification: CR-CNN outperforms the state of the art on SemEval-2010 Task 8 without costly handcrafted features.The reported evidence also favors CR-CNN over CNN followed by a softmax classifier.
  • Language modeling: genCNN combines local and long-range language structures for conditional-probability modeling, text generation, and language modeling.It is reported as empirically superior to traditional n-grams and its RNN or FFN counterparts on both tasks.

C. A Convolutional Neural Network for Modelling Sentences (P14-1062)

CNN-based sentence models construct representations over variable-length text and combine them with recurrent context for sequential classification. Across the reviewed work, CNN applications in NLP are promising, but their gains remain less dramatic than in computer vision and their goals are not yet unified.

  • Sentence modeling: Dynamic CNNs model variable-length sentences by stacking convolution and dynamic-pooling operations into a feature graph.The graph is designed to capture both short- and long-range relations.
  • Sequential short-text classification: A recurrent-CNN model classifies a current short text using its representation together with representations of several preceding texts.The architecture supports sequential short-text classification for dialog act detection.
  • Survey contribution: The paper surveys CNN variants and their applications across NLP problems, including reported results.Its stated contribution is an overview of CNN subtypes and solved problems using these technologies.
  • Conclusion and limitations: NLP deep-learning results are promising, but they have not produced a breakthrough as large as the one achieved in computer vision.The paper presents this as a limitation of the current state of CNN applications to NLP.
  • Future work: The review identifies a lack of common goals for CNN use in NLP and calls for better-defined objectives and adaptations beyond image-oriented design.It notes that CNNs were originally designed for images and that a 2D-3D component is missing.
Loading 1703.03091v1…