PaperlayerView exact revision on arXiv

Source-linked AI summary

Fast and Accurate Entity Recognition with Iterated Dilated Convolutions

Emma Strubell, Patrick Verga, David Belanger, Andrew McCallum

arXiv:1702.02098v3cs.CL

TL;DR

Sequence-labeling systems based on recurrent feature extraction and structured inference underuse GPU parallelism, motivating faster methods for large-scale NLP. The paper introduces ID-CNNs, which use dilated convolutions, parameter sharing, and training procedures to aggregate broad context; they achieve comparable accuracy with substantially faster decoding, including nearly 8× speedups for document-level prediction.

  • Problem

    Recurrent feature extraction and structured inference limit GPU parallelism in accurate sequence-labeling models, creating a need for faster methods for large-scale NLP.

  • Method

    ID-CNNs repeatedly apply shared blocks of increasingly dilated convolutions to preserve token resolution while aggregating broad context, with independent or CRF-based prediction.

  • Results

    More than 14× faster greedy decoding matched Bi-LSTM-CRF accuracy on CoNLL-2003, while document-level prediction was nearly 8× faster and more accurate.

  • Takeaways & Limitations

    ID-CNNs provide fast token encoders that aggregate broad context without losing resolution, particularly when processing entire documents.

  • Takeaways & Limitations

    The experiments use simpler word-embedding-only inputs and leave character representations and lexicons for future work.

Abstract

from arXiv · show

Today when many practitioners run basic NLP on the entire web and large-volume traffic, faster methods are paramount to saving time and energy costs. Recent advances in GPU hardware have led to the emergence of bi-directional LSTMs as a standard method for obtaining per-token vector representations serving as input to labeling tasks such as NER (often followed by prediction in a linear-chain CRF). Though expressive and accurate, these models fail to fully exploit GPU parallelism, limiting their computational efficiency. This paper proposes a faster alternative to Bi-LSTMs for NER: Iterated Dilated Convolutional Neural Networks (ID-CNNs), which have better capacity than traditional CNNs for large context and structured prediction. Unlike LSTMs whose sequential processing on sentences of length N requires O(N) time even in the face of parallelism, ID-CNNs permit fixed-depth convolutions to run in parallel across entire documents. We describe a distinct combination of network structure, parameter sharing and training procedures that enable dramatic 14-20x test-time speedups while retaining accuracy comparable to the Bi-LSTM-CRF. Moreover, ID-CNNs trained to aggregate context from the entire document are even more accurate while maintaining 8x faster test time speeds.

1 Introduction

Sequence-labeling models are accurate but underuse GPU parallelism because recurrent and structured inference steps process sequences sequentially. ID-CNNs address this with dilated convolutions, shared blocks, and broad context while preserving token resolution.

  • Motivation: RNN feature extraction and structured inference require sequential computation across input length, limiting GPU speed.These models are expressive and accurate but fail to fully exploit available parallelism.
  • Motivation: CNNs parallelize filters across sequences, but ordinary stacked convolutions require depth that grows linearly with sequence length to capture full context.Pooling would avoid this scaling but reduces output resolution, making it unsuitable for sequence labeling.
  • Approach: Dilated convolutions expand effective input width exponentially with depth without losing resolution; four width-3 layers cover 31 tokens.This exceeds the average 23-token Penn TreeBank sentence length.
  • Approach: ID-CNNs repeatedly apply the same dilated-convolution block, sharing parameters to reduce overfitting and support supervision on intermediate activations.The architecture supports both independent token prediction and Viterbi inference in a chain-structured model.

2 Background

The paper formulates sequence tagging with either conditionally independent token predictions or a linear-chain CRF. Independent prediction is parallelizable, whereas CRF decoding uses global Viterbi search and models neighboring tag interactions.

  • Conditional Probability Models for Tagging: The input is a token sequence x=[x_1,...,x_T] with per-token tags y=[y_1,...,y_T], and prediction selects the most likely tag sequence.The tag domain has size D.
  • Conditional Probability Models for Tagging: Conditionally independent tags given features permit O(D) prediction that is parallelizable across the sequence.Feature extraction may still be sequential when it uses RNN-based features.
  • Conditional Probability Models for Tagging: The linear-chain CRF couples all tags through local and pairwise factors and normalizes scores with a partition function.The pairwise factor is timestep- and input-independent in these experiments.
  • Conditional Probability Models for Tagging: CRF prediction requires global Viterbi search with O(D^2T) complexity.This search explicitly models neighboring tag interactions and can guarantee constraints such as valid IOB tagging.

3 Dilated Convolutions

One-dimensional CNN layers transform sliding windows of token vectors. Dilated convolutions widen the effective context by skipping inputs, and exponentially increasing dilation rates aggregate broad context with relatively few parameters while preserving resolution.

  • Convolutional Layers: A one-dimensional NLP convolution applies an affine transformation to a sliding window of token vectors.The operator produces an output vector for each token, and bias terms are omitted from the displayed formulation.
  • Dilated Convolutions: Dilated convolutions use the same operation while skipping inputs at intervals set by the dilation width δ.A dilation width of 1 is equivalent to a simple convolution.
  • Dilated Convolutions: With the same parameter dimensionality, δ>1 dilated convolutions incorporate broader context than simple convolutions.The wider effective input does not require adjacent inputs at every step.
  • Multi-Scale Context Aggregation: Stacking dilated convolutions with exponentially increasing rates expands effective input width exponentially while parameter count grows linearly with depth.A dilation-1 first layer ensures no positions within the effective width are excluded.

4 Iterated Dilated CNNs

ID-CNNs repeatedly apply a shared block of dilated convolutions to aggregate broad context while supporting parallel token prediction. The architecture combines parameter sharing and intermediate supervision to improve generalization and training.

  • 4 Iterated Dilated CNNs: With a radius of 2 and 8 dilated-convolution layers, the effective input width exceeds 1,000 tokens, enough to encode a full newswire document.
  • 4 Iterated Dilated CNNs: ID-CNNs repeatedly apply the same small stack of dilated convolutions, broadening context without adding parameters.Repeated parameter use provides broad effective input width and desirable generalization capabilities.
  • 4.1 Model Architecture: The network maps a sequence of T input vectors to per-class scores for each token, used either for independent conditional distributions or CRF local factors.
  • 4.1 Model Architecture: ID-CNNs support both independent token prediction and Viterbi inference in a chain-structured graphical model.
  • 4.2 Training: Intermediate supervision averages losses after each block application, training later blocks to refine initial predictions and helping reduce vanishing gradients.

5 Related work

Prior sequence-labeling systems commonly use recurrent feature extraction or structured inference, while this work applies dilated convolutions directly to sequence labeling. The paper also introduces a loss and parameter-sharing scheme intended for smaller datasets and focuses on simpler word-embedding inputs.

  • 5 Related work: State-of-the-art sequence-labeling models use chain-structured inference or beam search, outperforming comparable systems with independent local predictions.
  • 5 Related work: Bi-LSTM-CRF models achieved state-of-the-art results across multiple languages and sequence-labeling tasks, while related systems added character and lexicon features.
  • 5 Related work: The paper focuses on simpler word-embedding-only inputs, leaving character representations and lexicons for future work.
  • 5 Related work: The work applies dilated convolutions to sequence labeling and introduces a novel loss and parameter-sharing scheme for smaller datasets.
  • 5 Related work: ID-CNNs aggregate document-level context through broad effective input width, extending earlier fixed-window and whole-document structured approaches.

6 Experimental Results

Experiments on English NER show that ID-CNNs achieve accuracy comparable to recurrent baselines while substantially reducing decoding time. Document-level context further improves performance and preserves a large speed advantage, though results vary across datasets and decoding strategies.

  • Evaluation setup: Experiments use CoNLL-2003 and OntoNotes 5.0 English NER, comparing ID-CNNs with Bi-LSTM, Bi-LSTM-CRF, and non-dilated CNN baselines.Evaluation uses segment-level micro-averaged F1, with sentence- and document-level settings.
  • Sentence-level prediction: 0.11 points of F1 separate the ID-CNN-CRF and Bi-LSTM-CRF averages in opposite directions, while greedy ID-CNN outperforms the Bi-LSTM and 4-layer CNN.The ID-CNN-CRF exceeds the Bi-LSTM-CRF by 0.11 F1 points on average, whereas the Bi-LSTM-CRF exceeds greedy ID-CNN by 0.11.
  • Sentence-level prediction: More than 14 times faster test-time decoding is achieved by greedy ID-CNN versus Bi-LSTM-CRF with comparable accuracy.The ID-CNN is nearly 50% faster than the Bi-LSTM, while ID-CNN-CRF is about 30% faster than Bi-LSTM-CRF.
  • Regularization: Expectation-linear dropout regularization improves F1 for every evaluated model, supporting its use on the relatively small CoNLL-2003 dataset.The paper recommends this regularizer for practitioners training neural networks for NLP.
  • Document-level prediction: 90.65 average F1 is reached by document-level greedy ID-CNN on CoNLL-2003, where document context improves every model and decoding remains almost 8 times faster than Bi-LSTM-CRF.The authors attribute the broader-context improvement partly to the ID-CNN’s feature function and tied parameters with its training objective.
  • OntoNotes 5.0: On OntoNotes, longer entities benefit more from Viterbi constraints, while document context boosts greedy ID-CNN performance but reduces Bi-LSTM-CRF performance.The ID-CNN outperforms other greedy methods, whereas the gap between greedy and Viterbi decoding is wider than on CoNLL.

7 Conclusion

The paper presents ID-CNNs as fast token encoders that aggregate broad context without losing resolution, especially for sequence labeling over entire documents.

  • ID-CNNs efficiently aggregate broad context without losing resolution.

A.1 Optimization and data pre-processing

The models use end-to-end optimization with dropout and word dropout, pretrained embeddings, and sentence-model initialization for document models.

  • Models are trained end-to-end with mini-batched Adam and multiple dropout regularizers.
  • Word dropout is crucial for learning high-quality representations of out-of-vocabulary words.
  • 100-dimensional pretrained skip-n-gram embeddings are used in all experiments after outperforming 50-dimensional alternatives in initial experiments.
  • Sentence-model parameters initialize document models to significantly accelerate document-model convergence.

A.2 Data details

The evaluation uses NER datasets with different entity-type and genre coverage, summarized in a dataset-statistics table.

  • CoNLL-2003 labels entities with four types distributed fairly evenly across the corpus.
  • Table 8 is titled “Statistics of NER datasets.”
  • OntoNotes contains 19 entity types and broader genres, including telephone conversations, web text, broadcast news, and translated documents.

A.3 Evaluation

Evaluation selects hyperparameters through repeated grid search and measures test-time labeling speed under controlled GPU experiments across batch sizes.

  • Hyperparameters are selected through increasingly fine-grained grid search, with the final settings averaged over three validation runs.
  • Timing experiments use top-performing models on an nVidia Titan X GPU with a 2.4GHz Intel Xeon CPU.
  • Reported timing covers producing labels from word and capitalization inputs after burn-in, averaged over 20 runs for batch sizes from 1 to 10,000.
Loading 1702.02098v3…