PaperlayerView exact revision on arXiv

Source-linked AI summary

A Primer on Neural Network Models for Natural Language Processing

Yoav Goldberg

arXiv:1510.00726v1cs.CL

TL;DR

Neural NLP methods offered promising alternatives to traditional sparse linear models, but their principles and terminology created a barrier for practitioners. This tutorial surveys neural-network architectures, input representations, and computation graphs in a unified, practical framework. It aims to help NLP researchers understand and apply these models while remaining explicitly limited in scope.

  • Problem

    Neural techniques were promising for language processing but presented a substantial barrier of entry to NLP practitioners seeking to understand and apply them.

  • Method

    The tutorial provides a self-contained survey of NLP input encoding, feed-forward, convolutional, recurrent, and recursive networks, plus computation graphs for automatic gradient computation.

  • Results

    The primer presents neural networks as applicable learning models spanning nonlinear classification and non-Markovian modeling of sequences and trees.

  • Takeaways & Limitations

    The exposition is intended to help NLP researchers incorporate neural-network models into their work and use their capabilities for language-processing tasks.

  • Takeaways & Limitations

    The tutorial excludes comprehensive treatment of language modeling, acoustic modeling, machine translation, multimodal applications, attention, efficient large-vocabulary training, and several unsupervised approaches.

Abstract

from arXiv · show

Over the past few years, neural networks have re-emerged as powerful machine-learning models, yielding state-of-the-art results in fields such as image recognition and speech processing. More recently, neural network models started to be applied also to textual natural language signals, again with very promising results. This tutorial surveys neural network models from the perspective of natural language processing research, in an attempt to bring natural-language researchers up to speed with the neural techniques. The tutorial covers input encoding for natural language tasks, feed-forward networks, convolutional networks, recurrent networks and recursive networks, as well as the computation graph abstraction for automatic gradient computation.

1. Introduction

This introduction positions neural networks as nonlinear, dense-input alternatives to traditional sparse linear NLP models and presents the primer as a self-contained guide for applying existing techniques. Its scope is practical rather than comprehensive, with several neural NLP areas explicitly excluded.

  • Traditional NLP relied heavily on linear models trained over very high-dimensional, sparse feature vectors.
  • Neural NLP replaces sparse linear inputs with nonlinear models over dense representations, but the techniques can present a substantial barrier to entry.
  • The primer unifies neural NLP approaches, notation, tools, and methodology in a self-contained resource for practitioners and newcomers.
  • Scope: The tutorial excludes comprehensive treatment of language modeling, acoustic modeling, machine translation, multimodal applications, attention, efficient large-vocabulary training, and several unsupervised approaches.
  • In this primer, “feature” denotes a concrete linguistic input, whereas “input vector” denotes the vector supplied to the neural classifier.

2. Neural Network Architectures

The primer organizes neural NLP architectures around feed-forward, convolutional, recurrent, and recursive networks, each suited to different forms of prediction or structure. Feed-forward and convolutional models emphasize classification and salient clues, while recurrent and recursive models preserve sequence or tree structure.

  • The tutorial distinguishes feed-forward networks from recurrent and recursive networks, while allowing these architectures to be mixed and matched.
  • Feed-forward networks: Fully connected feed-forward networks provide nonlinear replacements for linear learners in binary, multiclass, and structured prediction tasks.
  • Convolutional and pooling networks: Convolutional and pooling networks target classification tasks where informative local clues may occur at different positions in the input.
  • Natural-language sequences and trees can be encoded as fixed-width vectors, but convolutional and pooling architectures sacrifice much of their structural information.
  • Recurrent and recursive networks: Recurrent and recursive architectures preserve more structural information while modeling sequences and trees, respectively.
  • Prior work reported strong recurrent-network results for language modeling and sequence tagging, and state-of-the-art or near-state-of-the-art recursive results across parsing and related tasks.

3. Feature Representation

Neural NLP systems transform linguistic features into vectors before applying nonlinear classifiers, with dense representations sharing information among similar features. For variable-length inputs, continuous bag-of-words representations combine feature embeddings into fixed-size vectors.

  • Feed-forward classification pipeline: A feed-forward NLP classifier extracts core linguistic features, retrieves their vectors, combines them into x, and feeds x to a nonlinear network.
  • Dense and one-hot representations: One-hot representations assign each feature its own dimension, so their dimensionality equals the number of distinct features and features remain independent.
  • Dense and one-hot representations: Dense representations map each feature to a low-dimensional vector, allowing similar features to have similar vectors and share information.
  • Dense and one-hot representations: Dense vectors can improve generalization by sharing statistical strength between related but differently observed words such as “dog” and “cat”.
  • Variable number of features: Continuous bag of words represents an unbounded number of features with a fixed-size vector by summing or averaging their embedding vectors.
  • Variable number of features: Weighted CBOW assigns each feature a relative-importance weight, such as a word’s TF-IDF score in document classification.
  • Feature extraction: Neural feature extraction focuses on core features rather than manually specified feature combinations, while embeddings are trained as model parameters.

4. Feed-forward Neural Networks

Feed-forward neural networks transform fixed-dimensional inputs through affine layers and nonlinear activations, producing flexible classifiers. Their representation power is substantial, but practical learning considerations still motivate deeper architectures.

  • Architecture: Neural networks consist of connected computational units whose weighted inputs are combined and passed through nonlinear functions.The network’s weights determine how inputs are transformed and its output is used for prediction.
  • Multi-Layer Perceptrons: An MLP adds nonlinear hidden layers between linear transformations, allowing the network to represent functions that a purely linear model cannot.Without the activation function, successive linear transformations remain linear.
  • Activation Functions: The activation function is crucial to representation power, with common choices including sigmoid, tanh, hard tanh, and ReLU.ReLU clips negative values at zero and often performs well, especially with dropout.
  • Architecture: A fully connected layer maps a d_in-dimensional input to a d_out-dimensional output through a weight matrix and bias vector.For l(x) = xW + b, x is 1 × d_in, W is d_in × d_out, and b is 1 × d_out.
  • Representation Power: An MLP with one hidden layer is a universal approximator for broad families of functions, but this theorem does not establish practical learnability or suitable hidden-layer size.The tutorial notes that finite data and optimization constraints can still make more complex architectures useful.

5. Word Embeddings

Word embeddings represent features as low-dimensional vectors, and their values can be learned directly with supervised training or initialized through pre-training.

  • Embedding Sources: Embeddings represent each feature as a vector in a low-dimensional space.The section surveys common approaches for obtaining these vectors.
  • Embedding Sources: With sufficient supervised data, embedding vectors can be randomly initialized and tuned alongside the other model parameters.The network-training procedure learns vectors suited to the supervised task.
  • Initialization: Random initialization requires care, with word2vec using uniformly sampled values whose range depends on the embedding dimension.Xavier initialization is presented as another option.
  • Initialization: In practice, common features may use random initialization, while rare features such as individual words often use supervised or unsupervised pre-training.Pre-trained vectors may be fixed or further updated during network training.

5.2 Supervised Task-specific Pre-training

Task-specific pre-training uses a data-rich auxiliary task to learn word vectors before training a target task with limited labeled data.

  • Supervised Task-specific Pre-training: Word vectors can be pre-trained on an auxiliary task with more labeled data and then used to train a target task with limited supervision.For example, part-of-speech tagging can provide data for vectors later used in syntactic parsing.

5.3 Unsupervised Pre-training

Unsupervised pre-training creates training instances from raw text to learn embeddings, especially when annotated auxiliary data is unavailable. Its usefulness depends on whether the induced similarities support the intended task.

  • Motivation: When no sufficiently large auxiliary annotated dataset exists, embeddings can be pre-trained from huge amounts of unannotated text.The approach is also useful for bootstrapping auxiliary-task training with better vectors.
  • Training Objective: Unsupervised embedding methods create practically unlimited supervised instances from raw text by designing proxy prediction tasks.These proxy tasks are intended to approximate the final task of interest.
  • Training Objective: Distributional methods learn similar vectors for words that occur in similar contexts, typically by predicting words from contexts or contexts from words.The choice of auxiliary problem includes both what is predicted and which context is used.
  • Benefits: Unannotated-data training supplies vectors for words absent from supervised data, helping models generalize to unseen events when related words receive similar representations.The desired similarity is the aspect of similarity useful for the intended task.
  • Algorithms: Common algorithms include word2vec, GloVe, and Collobert–Weston embeddings, which use stochastic-gradient methods and connect to matrix-factorization approaches.The tutorial emphasizes auxiliary-problem choices over training-method details.
  • Applications: Unsupervised embeddings have applications beyond initializing the embedding layer of a neural-network model.The tutorial notes these broader applications without developing them further.

5.4 Training Objectives

Training objectives create word representations by asking models to predict words from contexts or distinguish observed from randomly paired word-context examples.

  • Each word is represented as a d-dimensional vector initialized randomly, and auxiliary-task training produces embeddings that relate words to their contexts.Vectors for similar words become similar through successful auxiliary-task training.
  • Word2vec is a software package containing multiple training objectives, optimization methods, and hyperparameter choices rather than one single algorithm.
  • Language-modeling-inspired methods formulate word prediction probabilistically by modeling the conditional probability P(w|c).
  • Binary-classification approaches distinguish observed word-context pairs in D from randomly constructed pairs in ¯D.Methods differ in how ¯D is constructed, how the classifier is structured, and which objective is optimized.

5.5 The Choice of Contexts

Context design determines what relationships word vectors capture: window size shifts similarities from functional and syntactic toward topical, while alternative contexts encode structure, translation, or morphology.

  • Context Choices: Context words may be drawn from a short window, the same sentence, paragraph, or document, or from syntactic neighborhoods produced by parsing.
  • Window Approach: Sliding-window objectives use a focus word and surrounding context words, predicting the focus from all contexts or pairing it separately with each context.
  • Effect of Window Size: Larger sliding windows tend to produce topical similarities, whereas smaller windows tend to produce functional and syntactic similarities.Examples include topical groupings such as “dog”, “bark” and “leash”, versus syntactic groupings such as inflected forms.
  • Positional Windows: Positional contexts distinguish context words by their distance from and direction relative to the focus word.
  • Syntactic Contexts: Syntactic contexts produce functional similarities and group words that can fill the same sentence role, including words sharing an inflection.
  • Character-Based Contexts: Character-based representations can help with unknown words, support syntactic tasks, reduce model size, and combine word-specific and morphological vectors when available.Character vocabularies are much smaller than word vocabularies, although character-level modeling is challenging.

6. Neural Network Training

Neural networks are trained by minimizing losses with gradient-based updates, while computation graphs organize forward prediction and backward gradient calculation for flexible architectures.

  • Training Objective: Neural-network training repeatedly estimates error, computes gradients with respect to parameters, and moves parameters in the gradient-defined update direction.Parameters include transfer matrices, biases, and embedding matrices.
  • Stochastic Gradient Descent: Stochastic gradient descent minimizes a parameterized function’s loss using desired input-output pairs, commonly through online or minibatch updates.
  • Minibatch Training: Minibatch SGD estimates corpus-wide gradients from m examples: larger minibatches improve estimates, while smaller minibatches permit more updates and faster convergence.The minibatch size ranges from m = 1 to m = n.
  • Automatic Gradients: Automatic computation-graph tools avoid cumbersome manual derivatives and enable arbitrary networks, forward evaluation, and backward gradients for scalar losses.
  • Computation Graphs: Computation graphs represent mathematical computations as directed acyclic graphs whose nodes perform operations or hold variables and whose edges carry intermediate values.The DAG structure captures dependencies, including shared results used by multiple operations.
  • Computation Graphs: A concrete MLP graph supports prediction after inputs are bound, but training additionally requires the expected output and a scalar loss node.Once constructed, the graph supports forward computation and backward gradient computation.
  • Framework Trade-offs: Theano compiles graphs for efficient CPU or GPU execution with fixed structure, but compilation can be costly and the interface cumbersome for dynamic recurrent models.

7. Cascading and Multi-task Learning

Computation graphs and automatic gradients support composing neural components, sharing parameters, and combining related supervision signals through cascading and multi-task learning.

  • Overview: Online training combined with automatic computation-graph gradients enables model cascading, parameter sharing, and multi-task learning.
  • Model Cascading: Model cascading composes smaller networks into larger ones, allowing hidden representations from one task to serve as inputs for another.A part-of-speech network can provide an encoding for syntactic chunking or parsing.
  • Model Cascading: Component networks can be pretrained on relevant tasks before integration and further tuning, helping address vanishing gradients and use additional training material.
  • Model Cascading: Cascading is common in convolutional, recursive, and recurrent networks, including recurrent sentence encoders feeding fixed-sized vectors to another network.
  • Multi-task Learning: Multi-task learning combines related prediction tasks whose information may be useful across tasks, even when their outputs do not feed directly into one another.Examples include chunking, named entity recognition, and language modeling.
  • Multi-task Learning: Separate task losses are summed into one loss for gradient computation, and training can alternate examples with different supervision signals across corpora.

8. Structured Output Prediction

Structured prediction applies neural networks to outputs such as sequences, trees, and graphs through greedy, search-based, probabilistic, reranking, and hybrid formulations. These approaches provide flexible nonlinear scoring but sacrifice convexity and can increase computational cost.

  • Structured prediction covers sequence tagging, segmentation, and parsing, where outputs are sequences, trees, or graphs rather than class labels.
  • Greedy Structured Prediction: Greedy models decompose structure prediction into local decisions, then apply a neural classifier sequentially at test time.
  • Greedy Structured Prediction: Greedy prediction suffers from error propagation because early mistakes influence later decisions; easy-first and dynamic-oracle training can mitigate this problem.
  • Search Based Structured Prediction: Search-based models decompose structures into parts, score parts separately, and use inference to find the highest-scoring valid structure.
  • Search Based Structured Prediction: Neural structured models lose linear-model convexity and formal guarantees, while evaluating networks across parts can make training and inference slower.
  • Search Based Structured Prediction: CRF objectives maximize conditional log likelihood, with dynamic programming enabling polynomial-time partition computations when an appropriate algorithm exists.

9. Convolutional Layers

Convolutional networks transform sliding windows of word embeddings into learned representations and pool them into fixed-size vectors for prediction. Pooling variants trade positional sensitivity against retaining multiple or increasingly large patterns.

  • Basic Convolution + Pooling: Convolutional text models apply learned nonlinear filters to k-word sliding windows, then pool the resulting vectors into one d-dimensional representation.
  • Basic Convolution + Pooling: Training propagates prediction-loss gradients through pooling, convolution, and embedding layers, tuning filters to highlight task-relevant information.
  • Basic Convolution + Pooling: A 1d convolution reuses the same filter across sentence windows, producing one vector per window from concatenated word embeddings.
  • Basic Convolution + Pooling: Max pooling selects the most salient information across window positions, producing a fixed-size vector for downstream prediction.
  • Dynamic, Hierarchical and k-max Pooling: Position-aware pooling partitions windows into groups before pooling, preserving domain-relevant positional distinctions such as regions around related entities.
  • Dynamic, Hierarchical and k-max Pooling: Hierarchical pooling stacks convolution and pooling stages to gain sensitivity to increasingly larger structures.
  • Dynamic, Hierarchical and k-max Pooling: k-max pooling retains the k most active indicators in their original order, preserving feature order while remaining insensitive to exact positions.
  • Dynamic, Hierarchical and k-max Pooling: Parallel convolutional layers with window sizes 2–5 capture n-grams of varying lengths before concatenating their pooled representations.

10. Recurrent Neural Networks – Modeling Sequences and Stacks

Recurrent neural networks represent arbitrarily sized structured inputs with fixed-size states that preserve sequence order, then support acceptor, transducer, and encoder-decoder prediction setups. Their unrolled computation graphs enable shared-parameter training through backpropagation through time, while bidirectional and deeper variants extend the available context or depth.

  • 10.1 The RNN Abstraction: RNNs encode arbitrarily sized structured inputs in fixed-size vectors while preserving their structured properties.The network processes input vectors sequentially, maintaining a state vector that summarizes computation as the sequence unfolds.
  • 10.1 The RNN Abstraction: A recursively defined transition R maps each state and next input to a new state, while O maps each state to an output; both functions are shared across positions.The state dimension is tied to output dimension in popular Simple RNN, LSTM, and GRU architectures.
  • 10.1 The RNN Abstraction: For finite sequences, unrolling exposes a deep computation graph with shared parameters, allowing training by adding a loss node and applying backpropagation through time.The state at position i depends on the entire prefix x1, ..., xi, although later inputs may have stronger effects on the final state unless the transition is designed otherwise.
  • 10.2 RNN Training: RNNs support acceptors that supervise only the final output, transducers that produce outputs across sequence positions, and encoder-decoders that pass an encoded vector to another RNN.Acceptor examples include word-level part-of-speech prediction, sentiment classification, and noun-phrase validity; encoder-decoder supervision is applied to the decoder while gradients reach the encoder.
  • 10.2 RNN Training: RNN-based language models provide much better perplexities than traditional language models, while transducer-style conditioning can capture properties beyond n-gram histories.Character-level models condition each character on previous characters and show sensitivity to line lengths and nested-parenthesis balancing.
  • 10.3 Multi-layer (stacked) RNNs / 10.4 BI-RNN: Deep RNNs work better than shallower ones on some tasks, and bidirectional RNNs extend sequence-tagging context arbitrarily into both the past and future.A four-layer architecture was reported as crucial for good machine-translation performance in one encoder-decoder framework, while BI-RNNs relax fixed window-size assumptions.

11. Concrete RNN Architectures

The tutorial presents Simple RNNs, LSTMs, GRUs, and related recurrent variants as concrete sequence-modeling architectures. Gated designs address Simple RNN training difficulties and support longer-range dependencies, while simpler variants can remain competitive.

  • Simple RNN: Simple RNNs combine the current input with the previous state through a nonlinear activation, with the hidden state serving as the output.They provide strong results for sequence tagging and language modeling.
  • LSTM: The Simple RNN is hard to train effectively because vanishing gradients prevent error signals from reaching earlier inputs and hinder long-range dependency capture.LSTMs address this by adding memory cells whose access is controlled by gates.
  • LSTM: LSTMs use memory and output states plus input, forget, and output gates to regulate updates and preserve information across time.Gate values are computed from the current input and previous state, while the memory component carries information forward.
  • LSTM: LSTMs are described as the most successful RNN architecture, responsible for many state-of-the-art sequence-modeling results.The tutorial identifies the GRU as their main competitor and notes practical recommendations such as initializing forget-gate biases near one.
  • GRU: GRUs use fewer gates than LSTMs and no separate memory component, interpolating the previous state with a proposed update.They are effective in language modeling and machine translation, but the comparison among GRUs, LSTMs, and alternatives remains open.
  • Other Variants: Gated LSTM and GRU architectures alleviate Simple RNN vanishing gradients and capture dependencies spanning long time ranges.A simpler variant splits the state into slow- and fast-changing components and achieves competitive language-modeling perplexities with the more complex LSTM.

12. Modeling Trees – Recursive Neural Networks

Recursive neural networks generalize recurrent processing from sequences to binary trees by composing child representations into vectors for their parent nodes. These representations support node-level prediction, structural scoring, and encoding, while the best composition function remains an open question.

  • 12. Modeling Trees – Recursive Neural Networks: Recursive neural networks generalize RNNs from sequences to binary trees, representing each tree node with a state vector.The vectors can support predictions or quality scores for nodes and semantic representations of rooted spans.
  • 12.1 Formal Definition: A recursive composition function combines the d-dimensional vectors of two children into a d-dimensional vector representing their parent subtree.This parent vector encodes the entire subtree rooted at that node.
  • 12.1 Formal Definition: Figure 14 depicts bottom-up composition: V and NP1 form VP, then VP and NP2 form S.The diagram illustrates how representations are propagated through a binary parse tree.
  • 12.1 Formal Definition: The RecNN takes word vectors and a parse tree of production rules as input and returns corresponding inside state vectors for the tree nodes.Its recursive computation defines each node’s inside vector from the inside vectors of its direct children.
  • 12.1 Formal Definition: Composition functions may ignore labels, incorporate label embeddings, or use separate matrices for different nonterminal combinations.Label-aware formulations are useful when parse-tree labels are available and reliable, especially when the label inventory is relatively small.
  • 12.2 Extensions and Variations: Existing recursive composition functions inherit the Simple RNN vanishing-gradient problem, motivating tree-shaped LSTMs and leaving optimal tree representation open.Other extensions include recursive matrix-vector and tensor-network models with more expressive composition mechanisms.
  • 12.2 Extensions and Variations: Recursive networks are trained by defining losses, constructing computation graphs, backpropagating gradients, and optimizing parameters with SGD.Losses may be attached to the root, individual nodes, or sets of nodes, and inside vectors can also serve as encodings for another network.

13. Conclusions

The tutorial presents neural networks as powerful learners for NLP, spanning nonlinear classification and non-Markovian modeling of sequences and trees. It aims to help NLP researchers incorporate these models into their work.

  • 13. Conclusions: Neural networks support applications ranging from nonlinear classification to non-Markovian modeling of sequences and trees.The tutorial’s goal is to help NLP researchers incorporate these models and use their capabilities.
Loading 1510.00726v1…