Source-linked AI summary
Pointing the Unknown Words
Caglar Gulcehre, Sungjin Ahn, Ramesh Nallapati, Bowen Zhou, Yoshua Bengio
TL;DR
Rare and unknown words challenge NLP systems because shortlist vocabularies discard or poorly represent them. The paper introduces Pointer Softmax, which adaptively chooses between shortlist prediction and attention-based copying from context, and reports improvements in translation and summarization. Evaluation includes a sampled Gigaword test set rather than the unavailable public test set used in prior work.
Problem
Rare and unknown words limit neural NLP systems because shortlist vocabularies poorly represent infrequent words and map other words to UNK, losing information.
Method
Pointer Softmax jointly learns whether to use shortlist softmax or attention-based location softmax to copy a word from a variable-length context.
Results
Pointer Softmax improves performance over NMT baselines on machine translation and summarization, including about 3.6 BLEU points on the reported translation baseline.
Takeaways & Limitations
The model generalizes to unseen words and handles rare words more efficiently while remaining compatible with techniques such as the large vocabulary trick.
Takeaways & Limitations
For summarization, evaluation used 2,000 validation examples sampled as a test set because the prior work’s test set was unavailable.
Abstract
from arXiv · showhide
The problem of rare and unknown words is an important issue that can potentially influence the performance of many NLP systems, including both the traditional count-based and the deep learning models. We propose a novel way to deal with the rare and unseen words for the neural network models using attention. Our model uses two softmax layers in order to predict the next word in conditional language models: one predicts the location of a word in the source sentence, and the other predicts a word in the shortlist vocabulary. At each time-step, the decision of which softmax layer to use choose adaptively made by an MLP which is conditioned on the context.~We motivate our work from a psychological evidence that humans naturally have a tendency to point towards objects in the context or the environment when the name of an object is not known.~We observe improvements on two tasks, neural machine translation on the Europarl English to French parallel corpora and text summarization on the Gigaword dataset using our proposed model.
1 Introduction
NLP systems need broad word coverage, but shortlist-based neural models struggle with rare words and unseen words. The paper proposes learning when to copy words from context, motivated by human pointing behavior.
- Broad word coverage is important for robust NLP, but languages contain many named entities and continually evolving vocabulary.
- Shortlist softmax limits vocabulary to frequent training words, mapping others to UNK because high-dimensional softmax is expensive.
- Rare words are difficult to represent, while mapping distinct words to UNK loses information and cannot prevent unknown test-time words.
- Humans use pointing to communicate about objects whose names are unknown, providing motivation for context-based word copying.
- Pointer Softmax chooses between shortlist generation and copying from context, learning both whether and where to point.
2 Related Work
Prior approaches address rare and unknown words by enlarging vocabulary computation, using context information, or changing the word representation. Context-based methods are closest to Pointer Softmax but differ in flexibility and attention use.
- Pointer networks dynamically compute output dimensions over input observations but cannot choose whether to point or not.
- Softmax-acceleration methods support larger vocabularies but mitigate unknown words without solving the rare-word problem.
- Context-based methods copy information from inputs, forming the category that includes Pointer Softmax.
- Luong et al.’s positional model uses fixed pointing ranges and lacks attention, limiting application when context lengths and locations vary widely.
- Character- and bytecode-based representations reduce rare or unknown-word issues but make training harder by substantially lengthening sequences.
3 Neural Machine Translation Model with Attention
The baseline neural machine translation model encodes the input bidirectionally and uses attention to build decoder context at each target timestep. A vocabulary softmax then predicts target-word probabilities.
- The baseline NMT model jointly learns soft alignment and translation.
- A bidirectional RNN encoder produces annotation vectors whose representations incorporate each input word’s surrounding context.
- At each timestep, an MLP-based soft-alignment mechanism scores annotation relevance using the annotation, previous decoder state, and previous output.
- Normalized relevance scores weight annotation vectors to form the context vector for the current target word.
- The decoder updates its hidden state from the previous state, current context vector, and previous output, then uses vocabulary softmax to predict word probabilities.
4 The Pointer Softmax
The Pointer Softmax (PS) addresses rare and unknown words by jointly learning whether to generate from a shortlist or copy from a variable-length context sequence. It combines shortlist and location softmaxes with an MLP switching network that selects the generation source at each timestep.
- Core idea: The Pointer Softmax jointly learns when to point to the context and which context location to copy, extending handling of rare and unknown words.The method explicitly targets both deciding whether pointing is needed and selecting a location in contexts whose lengths vary across examples.
- Two softmax layers: The model uses a shortlist softmax for vocabulary words and a location softmax whose output dimensions correspond to positions in the context sequence.The location softmax can reuse attention distributions in attention-based models or use a pointer network to predict the copied location.
- Adaptive switching: At each timestep, a switching network chooses shortlist generation or context copying, with the latter producing a location for the word to copy.The binary variable z_t indicates the selected source: z_t = 1 uses the shortlist softmax, while z_t = 0 uses the location softmax.
- Adaptive switching: The switching network is an MLP with sigmoid output conditioned on the context representation and the decoder’s previous hidden state.Its scalar output represents the conditional probability of switching between the location and shortlist distributions.
- Output: The final pointer-softmax output concatenates the switch-weighted shortlist and location vectors, and test-time prediction selects the highest-probability word or location.The model computes probabilities for all shortlist words and context locations before choosing the maximum-probability option.
5 Experiments
Experiments evaluate Pointer Softmax on rare-word detection, Gigaword summarization, and Europarl English–French translation, using baselines and task-specific pointer data. Pointer Softmax improves performance, reduces rare-word error, and converges faster than shortlist softmax.
- 5.1 The Rarest Word Detection: 17.4% error with Pointer Softmax versus 48.2% with softmax over all 600 tokens in rare-word detection.The synthetic task uses sequences of length 7 and vocabulary size 600, with pointers assigned to the rarest 60 words.
- 5.2 Summarization: Pointer Softmax improves over baseline NMT on both UNK-pointer and entity-pointer Gigaword data.Entity improvements were not larger, possibly because anonymization reduced data sparsity by replacing entities with shared integer IDs.
- 5.2 Summarization: Pointer Softmax improves all reported Gigaword summarization scores when evaluated using recall.The comparison follows the evaluation setup used by Rush et al. (2015).
- 5.2 Summarization: The Gigaword summarization test set uses 2,000 validation texts sampled without replacement because the reference test set was unavailable.This sampling choice limits direct comparability with results based on the original test set.
- 5.3 Neural Machine Translation: Pointer Softmax yields about 3.6 BLEU improvement over the Europarl English–French NMT baseline.It also reaches validation NLL 63.91 after 200k updates and converges faster than shortlist softmax.
6 Conclusion
Pointer softmax extends attention-based shortlist softmax with input-sequence pointers, improving rare-word handling across machine translation and summarization. It also improves convergence in neural machine translation and remains effective alongside the large-vocabulary trick for Gigaword summarization.
- Pointer softmax extends traditional attention-based shortlist softmax with pointers over the input sequence.
- The model can be trained jointly with a single objective function.
- Pointer softmax improves results over baselines on machine translation and summarization tasks.
- On Gigaword summarization, pointer softmax improves results even when combined with the large-vocabulary trick.
- For neural machine translation, pointer softmax also improves training convergence speed.