Generating News Headlines with Recurrent Neural Networks

• RNNs learn from input-output sequences and generate outputs for previously unseen inputs, supporting text transduction tasks such as machine translation and speech recognition.Source: §1 · p. 1
• Reading-comprehension applications train models to recall facts or statements from input text.Source: §1 · p. 1
• Prior headline generation on the same dataset used attention but not recurrent encoding, instead relying on a simpler attention-based model.Source: §1 · p. 1

2.1 Overview

• The encoder processes article words into hidden representations, and the decoder generates the headline one word at a time from the encoder state and attention.Each generated word is fed back as input for the next decoding step.Source: §2.1 · pp. 1–2
• Training uses log loss and randomly feeds generated rather than expected words 10% of the time to address the teacher-forcing mismatch with testing.At testing, a beam-search decoder extends the highest-probability sequences.Source: §2.1 · p. 2

2.2 Attention

• Attention assigns weights across input words for each output word and uses them to form a weighted-average context for the decoder’s softmax.The attention weights sum to 1.Source: §2.2 · p. 2
• The complex mechanism uses the same hidden units to compute attention weights and context.Source: §2.2 · pp. 2–3
• Simple attention splits the final hidden layers into 50 units for attention weights and 550 for context or output, while retaining the corresponding formulas.This separates the units used to compute attention weights from those used for context computation.Source: §2.2 · p. 3

3.1 Overview

• The processed Gigaword training set contains 5.5M news articles and 236M words from several years of reporting by six major news agencies.Source: §3.1 · p. 3

3.2 Preprocessing

• Preprocessing retains only each article’s first paragraph, filters examples exceeding 25 headline or 50 text tokens, and keeps the 40,000 most frequent words.Text and headlines are lowercased and tokenized, and rare words are replaced with <unk>.Source: §3.2 · p. 3
• The holdout uses the last month of data, with the preceding month excluded from both splits to reduce near-duplicate articles across training and holdout.Source: §3.2 · p. 3

3.3 Dataset Issues

• The dataset includes malformed examples and headlines that do not summarize their texts, and these examples were not filtered.Some headlines contain coded forms or service-client notices.Source: §3.3 · pp. 3–4

• Model performance is assessed using training and holdout loss and BLEU over 384 holdout examples for efficiency.Source: §4 · p. 4
• BLEU measures the fraction of expected-headline n-grams output by the model and compares generated and expected word counts across the 384 examples.The computation is over the examples collectively rather than separately for each example.Source: §4 · p. 4

5.2 Understanding how the attention weight vector is computed

• The simplified attention mechanism performed significantly better than the complex mechanism during training, as shown by loss and BLEU over epochs.The authors suggest that separating attention-weight computation from context computation may reduce noise in both.Source: Figure 4 · p. 4, Figure 5 · p. 4, §5.2 · p. 7

5 Analysis

• On articles from familiar newspapers, the model generally captured the gist and paraphrased headlines, sometimes with new words, but examples 2, 4, and 7 contained mistakes.The analysis presents seven randomly selected held-out examples.Source: §5 · p. 4, Table 1 · p. 5
• On unfamiliar sources, generated headlines were mixed: the model did well on BBC, Wall Street Journal, and Guardian articles but poorly on Huffington Post and Forbes, especially Forbes.The authors suggest Forbes articles differ substantially in writing style from the training articles.Source: §5 · p. 4, Table 2 · p. 6

5.1 Understanding information stored in last layer of the neural network

• The analysis examined the words most associated with decoder hidden states and attention context, including context contributions from individual input positions.The context is a weighted sum over decoder hidden layers.Source: §5.1 · p. 5
• Hidden-state analysis suggests that the network can take multiple processing steps to encode a word, complicating interpretation of what it attends to.For example, the hidden state after “and” was closest to several words that appeared earlier in the input.Source: §5.1 · p. 6

5.2 Understanding how the attention weight vector is computed

• The hypothesis that decoder and encoder hidden units share a space for matching remembered and input words did not hold; analysis instead found key units for computing attention weights.The simplified mechanism was easier to inspect, and a smaller model used 20 units for attention-weight computation whose functions were catalogued.Source: §5.2 · p. 7, Table 3 · p. 7
• The neuron analysis was shallow because of time constraints, and the authors suspected some units detect more complex phenomena such as sentence structure.Encoder and decoder neurons interact, with unit 12 changing from attending to the object to attending to the subject during decoding.Source: §5.2 · pp. 7–8

5.3 Errors

• The model sometimes invents missing details, particularly with few decoding beams, or produces unrelated headlines, particularly with many beams.The authors used two decoding beams for BLEU evaluation and proposed scheduled sampling as one possible avenue for addressing errors.Source: §5.3 · p. 8

6 Future Work

• The authors propose pretraining a recurrent neural network on Gigaword, then fine-tuning it for a task such as part-of-speech tagging with less data.They note that large training datasets are usually unavailable for real-world NLP problems.Source: §6 · p. 8
• A bidirectional RNN could provide following-word information that may help the attention mechanism choose values for input-word neurons.The current model makes that decision using only the current and previous words.Source: §6 · p. 8
• The authors also identify scheduled sampling as a previously discussed direction for improving the model.Source: §6 · p. 8

7 Conclusion

• Using the first 50 words of news articles, the model usually generates a valid, grammatically correct concise summary of those words.Source: §7 · p. 8
• The model performs less well on general text, which the authors take as evidence that many Gigaword articles follow a particular form.Source: §7 · p. 8
• The simplified attention mechanism does better on the evaluation metrics.Source: §7 · p. 8
• The authors study two attention mechanisms and find that the network detects linguistic phenomena including verbs, subjects, objects, names, negations, and noun-phrase boundaries.The simplified mechanism uses a small set of neurons for attention weights, making the network easier to study.Source: §7 · p. 8