Source-linked AI summary
Natural Language Processing (almost) from Scratch
Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, Pavel Kuksa
TL;DR
NLP systems traditionally rely on task-specific, hand-designed features, creating a separate engineering burden for each task. This paper proposes a unified neural architecture that learns representations from largely unlabeled data, achieving performance in the ballpark of existing approaches across four tasks and yielding a fast, efficient tagger.
Problem
Traditional NLP depends on task-specific hand-designed features, requiring separate empirical feature engineering for each new task.
Method
A unified multilayer neural network learns shared internal representations from large unlabeled datasets while minimizing task-specific engineering across NLP tasks.
Results
Neural networks trained out-of-the-box lagged benchmark systems but remained in the performance ballpark, while semi-supervised initialization significantly improved generalization on every task.
Takeaways & Limitations
The work supports a fast, efficient all-purpose NLP tagger based on learned representations rather than extensive task-specific feature engineering.
Takeaways & Limitations
Training is computationally expensive, taking about three days for semantic role labeling.
Abstract
from arXiv · showhide
We propose a unified neural network architecture and learning algorithm that can be applied to various natural language processing tasks including: part-of-speech tagging, chunking, named entity recognition, and semantic role labeling. This versatility is achieved by trying to avoid task-specific engineering and therefore disregarding a lot of prior knowledge. Instead of exploiting man-made input features carefully optimized for each task, our system learns internal representations on the basis of vast amounts of mostly unlabeled training data. This work is then used as a basis for building a freely available tagging system with good performance and minimal computational requirements.
1. Introduction
Because complete semantic representations remain unresolved, NLP systems target restricted syntactic and semantic representations for practical tasks. This paper seeks strong performance across multiple benchmarks using one learning system that discovers internal representations instead of relying on task-specific engineered features.
- Motivation: Complete, unambiguous representations of natural-language meaning remain unresolved, so NLP research targets restricted syntactic or semantic representations.Examples include part-of-speech tagging, chunking, parsing, word-sense disambiguation, semantic role labeling, named entity extraction, and anaphora resolution.
- Limitations of prior systems: Most state-of-the-art NLP systems apply linear statistical models to ad-hoc, task-specific features, often derived from preexisting systems and dependent on substantial linguistic engineering.These engineered intermediate representations are effective but can create complex runtime dependencies.
- Contribution: The authors aim to excel on multiple benchmarks with a single learning system that discovers adequate internal representations rather than using task-specific engineering.They view benchmark performance as an indirect measurement of representation quality and posit that the learned representations are more general than individual benchmarks.
- Paper organization: The paper evaluates supervised benchmark models, transfers representations learned from approximately 852 million unlabeled words, and investigates multitask supervised training.The introduction previews benchmark tasks, a unified model, language-model pretraining, transfer to supervised models, and multitask training.
2. The Benchmark Tasks
This section defines four benchmark NLP tasks—POS, chunking, NER, and SRL—and summarizes their standard datasets, evaluation setups, and established comparison systems. It also notes that more complex tasks generally require more engineered features than simpler ones.
- Benchmark scope: The paper benchmarks part-of-speech tagging, chunking, named entity recognition, and semantic role labeling under standard experimental setups summarized in Table 1.State-of-the-art systems and evaluation metrics are reported in Table 2.
- Chunking: Chunking labels sentence segments as syntactic constituents such as noun or verb phrases, using CoNLL 2000 with WSJ sections 15–18 for training and section 20 for testing.The cited systems use word, POS, and tag context with SVMs, voting, dynamic programming, or second-order random fields, reporting up to 95.23% F1.
- Named Entity Recognition: NER labels tokens as entity categories such as PERSON or LOCATION under the CoNLL 2003 Reuters setup, which provides training, validation, and test sets.Cited systems combine words, POS and chunk tags, affixes, gazetteers, classifier outputs, or unlabeled data, with reported performance up to 89.31% F1.
- Semantic Role Labeling: SRL assigns semantic roles such as ARG0–5 to predicate arguments, and state-of-the-art systems use parsing, argument identification, classification, and numerous parse-tree features.The cited SRL system obtains 77.30% F1 with SVM classifiers using the two parse trees provided for the task.
- Benchmark selection and task complexity: To ensure fair comparisons, benchmark systems are established top systems that avoid external data, except that the selected NER benchmark uses additional unlabeled data only.Across the four tasks, more complex tasks with lower accuracies tend to rely on more engineered features; SRL is described as the most feature-intensive.
3. The Networks
The section replaces task-specific hand-engineered features with an end-to-end multilayer neural network that learns task-relevant representations by backpropagation. Its architecture extracts word-level features, then models local windows or whole-sentence sequence structure before applying standard neural-network layers.
- Motivation and approach: The proposed approach minimizes preprocessing and trains a multilayer neural network end to end, learning task-relevant feature representations through backpropagation.This contrasts with traditional NLP systems that use empirically selected, task-dependent hand-designed features and can incur substantial computational costs.
- Architecture: The network’s first layer extracts features for each word, while its second layer captures either a word window or the whole sentence as local and global sequence structure.The sentence is treated as a sequence rather than a bag of words.
- Architecture: The architecture is summarized by separate window-based and sentence-based network diagrams in Figures 1 and 2.The window approach is shown in Figure 1, and the sentence approach is shown in Figure 2.
- Architecture: After the word-level and sequence-processing layers, the networks use standard neural-network layers.The section introduces these components within a general feed-forward network notation.
3.1 Transforming Words into Feature Vectors
The architecture represents words primarily through trainable lookup-table feature vectors, allowing it to operate effectively on almost raw words. Optional discrete features can be concatenated into richer representations that feed subsequent neural layers.
- 3.1 Transforming Words into Feature Vectors: Trainable lookup tables map dictionary word indices into dense feature vectors, making effective processing of almost raw words central to the architecture.The method relies on learning useful word representations rather than encoding substantial task-specific input information.
- 3.1 Transforming Words into Feature Vectors: Applying the same lookup operation across a sequence produces a matrix of word vectors for downstream neural-network layers.Each word receives a vector of user-specified dimension, and the resulting sequence matrix is passed to further layers.
- 3.1 Transforming Words into Feature Vectors: The lookup-table features are trainable representations that support extracting patterns over words and ultimately representing complete sentences.The authors used lowercasing and capitalization encoding as preprocessing, while learning directly from letter sequences was left beyond the work’s scope.
3.2 Extracting Higher Level Features from Word Feature Vectors
The architecture combines word feature vectors through either fixed windows or a convolutional sentence network, using nonlinear feature extraction and task-specific tag scoring. The sentence approach is needed for tasks such as semantic role labeling, where relevant predicates may lie outside a local window.
- Architecture: The model offers two ways to combine word feature vectors for tagging: a fixed-size window around each target word or a convolutional sentence approach over the complete sentence.Both approaches feed extracted features into subsequent neural-network layers that produce a tag decision for each word.
- Architecture: Stacked affine and HardTanh layers extract increasingly nonlinear features before the final layer assigns one score to each possible tag.HardTanh is cheaper to compute than the exact hyperbolic tangent while leaving generalization performance unchanged.
- Convolutional sentence approach: A window approach performs well for most targeted NLP tasks but fails for SRL when the relevant predicate falls outside the local window, motivating whole-sentence processing.For SRL, the sentence operation is performed for each word and each verb, with additional input markers designating the word being tagged.
- Convolutional sentence approach: The convolutional approach processes the complete sentence, extracts local features with shared weights, and combines them into a fixed-size global vector for standard affine layers.Convolutional layers must be followed by nonlinearities when stacked; a Max layer can aggregate variable-length sequences into fixed-size representations.
- Output tagging: POS assigns labels directly to words, whereas the other tasks label sentence segments using boundary-aware schemes such as IOB, IOE, or IOBES.The text notes that no tagging scheme is clearly best in general.
3.3 Training
The networks are trained by maximizing likelihood with stochastic gradient ascent, using either independent word-level tags or sentence-level tag paths. Sentence-level training models tag dependencies and supports efficient dynamic-programming inference, while benchmark results remain behind established systems.
- Likelihood training: Training maximizes the dataset log-likelihood over network parameters through stochastic gradient ascent.Each update selects a random training example and takes a step controlled by the learning rate λ.
- Word-level likelihood: The word-level approach converts network tag scores into conditional probabilities with a softmax but treats each word independently.This cross-entropy criterion discards correlations between neighboring tags that occur in chunking, NER, and SRL.
- Sentence-level likelihood: The sentence-level approach adds learned transition and initial scores, normalizes over complete tag paths, and trains against the true path.The path normalization is computed in linear time by recursion, while Viterbi decoding finds the best path by replacing logadd with max.
- Optimization and implementation: Gradients are obtained by applying the chain rule through the network and sentence recursion, with nondifferentiable updates skipped when necessary.The modular implementation lets network modules independently compute derivatives with respect to inputs and trainable parameters.
- Benchmark comparison: Vanilla neural networks using word-level and sentence-level likelihoods trail benchmark systems on POS, chunking, NER, and SRL.Performance is reported as per-word accuracy for POS and F1 for the other tasks.
3.4 Supervised Benchmark Results
The supervised networks were evaluated on POS, chunking, NER, and SRL using task-specific architectures and raw word features, but remained behind baseline systems while broadly matching existing approaches. Sentence-level likelihood improved chunking, NER, and SRL more than POS, motivating later use of unlabeled data to improve embeddings.
- Input features and training: All networks used lowercase words and a capital-letter feature as their two raw text inputs, with experiments comparing word-level and sentence-level log-likelihood.Lowercasing limited dictionary size, while the capitalization feature retained upper-case information.
- Overall results: The neural networks were behind baseline benchmark systems but remained within the performance range of existing approaches across the evaluated tasks.Results used per-word accuracy for POS and F1 for the other tasks.
- Training criterion: Sentence-level likelihood boosted chunking, NER, and SRL performance, while providing little advantage for POS.POS, chunking, and NER used the window architecture, whereas SRL used the sentence approach.
- Transition: The next section improves the word embeddings with unlabeled data, which the authors expect to boost performance on all tasks.This marks the transition from supervised benchmark evaluation to leveraging unlabeled data.
- Training time: Training required about one hour for chunking and NER, a few hours for POS, and about three days for SRL.The authors retained a small learning rate rather than optimizing for training speed and suggested larger learning rates or second-order methods as possible accelerations.
4. Lots of Unlabeled Data
The authors improve word embeddings by training window-based neural language models on large unlabeled English corpora with a pairwise ranking objective. Initializing supervised task networks with these embeddings significantly boosts generalization across tasks, with larger unlabeled data yielding further gains.
- Unlabeled corpora: The authors train language models on about 631 million Wikipedia words and an expanded corpus containing 221 million additional Reuters words.The second corpus also expands the dictionary from 100,000 to 130,000 words to test whether more unlabeled data improves performance.
- Language-model training: The models use the window architecture and train word embeddings with a pairwise ranking criterion that scores legal phrases above incorrect phrases.Training uses stochastic gradient minimization while sampling sentence-word pairs.
- Embedding quality: The resulting embeddings exhibit syntactically and semantically related nearest neighbors and are more appealing than embeddings learned purely from supervised benchmark tasks.Table 7 reports nearest neighbors for randomly selected query words from the LM1 model.
- Results: Initializing supervised networks with the learned embeddings significantly boosts generalization performance on every task, and the larger language model performs even better.The result suggests that still larger unlabeled datasets could provide additional benefits.
- Limitations: The approach aims to learn hierarchical syntactic and semantic information from unlabeled data without parse trees, but its restrictive language-model structure currently exploits only the learned word embeddings.The authors therefore view the ranking criterion as conceptually capable of capturing rich information while acknowledging limitations in the current architecture.
5. Multi-Task Learning
This section applies multi-task learning by jointly training POS, CHUNK, NER, and SRL models with shared parameters, seeking common representations across tasks. The unified architecture performs well, but yields no or only marginal gains over separately trained task-specific architectures.
- Architecture: The models were jointly trained with shared lookup-table parameters, while task-specific output layers produced predictions for each task.The neural multitasking design shares the lookup tables and first hidden layer, with the last layer remaining task-specific.
- Experimental setup: SRL required the sentence-level network because its predicate-related dependencies are long-range, whereas POS, CHUNK, and NER were jointly trained with the window network.The window model enlarged its shared first hidden layer to n1_hu = 500, while the embedding dimension remained d0 = 50.
- Training: Joint training minimizes the average loss across tasks by alternating task examples and updating both task-specific and shared parameters, giving every task equal weight.Training and test sets do not overlap across tasks, and each task’s generalization is evaluated on its traditional test data.
- Results: Multi-task learning produced a single unified architecture that performed well across POS, CHUNK, NER, and SRL, but delivered no or only marginal improvements over separate task architectures.The separate architectures still used semi-supervised learning, identified as the most important multi-task component.
6. The Temptation
The section tests how task-specific engineering changes the largely generic neural architecture, finding that suffixes, related-task tags, and parse-tree information can improve performance. It also notes that the approach is only “almost” from scratch because of basic preprocessing and remains computationally lightweight.
- Scope: The method is “almost” from scratch because it performs basic raw-input preprocessing; a completely from-scratch system would use letters, speech, or optical recognition instead of words.The qualification explains the “almost” in the article’s title.
- Word features: Adding the final two characters of each word as discrete suffix features yields a small POS improvement, with a suffix dictionary of 455 entries.Porter-stemmer suffixes perform identically to two-character suffixes.
- Related-task features: Adding gold POS tags to CHUNK and NER, and gold CHUNK tags to SRL, produces moderate improvements across the corresponding tasks.The experiments use POS and CHUNK tags supplied by the CoNLL challenge rather than the neural POS system’s more accurate predictions.
- Parse-tree information: Parse-tree leaves outperform chunking features for SRL, while feeding the network’s own PT0 predictions performs similarly to Charniak predictions and consistently better than CHUNK.The leaves and chunking have identical labeling when BIES segmentation prefixes are ignored, but parse trees provide richer information.
- Efficiency: Despite added task-specific features, the systems remain lightweight: POS and SRL taggers use 32MB and 120MB of RAM, respectively, and run considerably faster than cited alternatives.The Shen and Toutanova taggers slow substantially below 2.2GB and 800MB, while the Koomen tagger requires at least 3GB.
7. Critical Discussion
The authors present their work as a step toward “NLP from scratch” while acknowledging criticisms of both their goal and their reliance on multilayer neural networks. They defend task-general learning and neural networks’ ability to discover representations with scalable stochastic learning.
- The authors acknowledge that both their “NLP from scratch” objective and the means used to pursue it can be criticized.
- Critics may question abandoning NLP feature-engineering expertise to acquire the skills needed to train large neural networks.The authors argue that no single NLP task fully covers the goals of NLP, motivating methods that avoid task-specific engineering.
- The authors chose multilayer neural networks because they can discover hidden representations through stochastic learning that scales linearly with the number of examples.They characterize this technology as roughly twenty years old and note that much of the necessary neural-network technology was described a decade earlier.
8. Conclusion
The paper presents a fast, accurate multilayer neural network for multiple NLP tasks that minimizes task-specific engineering by learning useful representations from large unlabeled datasets.
- 8. Conclusion: The proposed multilayer neural network handles multiple NLP tasks with both speed and accuracy while minimizing task-specific engineering.It relies on large unlabeled datasets and lets the training algorithm discover internal representations useful across the tasks.
- 8. Conclusion: The learned representations support a fast and efficient all-purpose NLP tagger.
Appendix A. Neural Network Gradients
The appendix derives stochastic-gradient training for the network by expressing feed-forward computation as layered functions and applying generalized chain-rule backpropagation. It then specializes gradient computation to the network layers and training criteria used in the paper.
- General setup: Training minimizes a cost with respect to network parameters using stochastic gradients, including the negated likelihood and ranking criterion.The cost is written as C(fθ(·)) after negating the likelihood.
- General setup: Generalized chain-rule backpropagation partitions parameters by layer, initializes the gradient at the final output, and recursively propagates gradients through each layer.Each layer computes gradients for its parameters and inputs from the gradient arriving at its output.
- Layer gradients: The appendix derives layer-specific gradients for lookup-table, linear, window-based, and parameter-free layers, including accumulation across sequence windows.Lookup-table updates omit columns for indices absent from the sequence, while the parameter-free layer requires only input gradients and ignores non-differentiability points.
- Training criteria: For word-level and sentence-level likelihoods, gradients are obtained from tag scores and through the sentence-level logadd recursion, while the ranking criterion uses a hinge-style gradient ignoring the zero-point non-differentiability.Sentence-level differentiation accumulates gradients for network inputs and transition scores during the recursion.