Soft Layer-Specific Multi-Task Summarization with Entailment and Question Generation

1 Introduction

• A good abstractive summary should preserve salient information, follow logically from the document, and avoid redundancy, but prior coverage models addressed redundancy while salience detection and logical inference remained to be taught.Abstractive summarization rewrites rather than selects text, and neural sequence-to-sequence models had enabled substantial progress on the task.Source: §1 · p. 1
• Question generation teaches the model to identify salient details by learning what questions to ask, while entailment generation teaches it to produce summaries that logically follow from the document.Question generation uses SQuAD (Rajpurkar et al., 2016); entailment generation converts the SNLI task (Bowman et al., 2015) into generation (Pasunuru and Bansal, 2017).Source: §1 · pp. 1–2
• The architecture shares higher-level semantic layers across the three tasks while leaving lower-level lexico-syntactic layers unshared, and soft parameter sharing outperforms hard sharing.The paper also explores different ways to optimize the shared parameters.Source: §1 · p. 2
• The model achieves statistically significant improvements over the state of the art on CNN/DailyMail and Gigaword and performs significantly better on DUC-2002 transfer.The authors also report improvements on the auxiliary generation tasks, reduced multitask training time through pretrained baseline initialization, human evaluations, and analyses of learned saliency and inference skills.Source: §1 · p. 2

Summarization research

• Earlier summarization progressed from extractive and compressive approaches to graph-, concept-map-, discourse-, syntax-, and AMR-based abstractive methods (Dohare and Karnick, 2017).Recent neural sequence-to-sequence work explored hierarchical, distractive, saliency, and graph-attention modeling (Rush et al., 2015; Nallapati et al., 2016), reinforcement learning (Paulus et al., 2018), and pointer-copy with coverage (See et al., 2017).Source: §2 · p. 2

Multi-task learning

• Multi-task learning shares parameters or representations across related tasks to improve generalization, and prior neural work examined shared versus unshared parameters (Caruana, 1998; Argyriou et al., 2007; Kumar and Daumé III, 2012).This paper investigates soft parameter sharing and high- versus low-level layer-specific sharing.Source: §2 · p. 2

Contributions

• The paper reports that its soft, layer-specific multi-task model improves over pointer-plus-coverage baselines on CNN/DailyMail, Gigaword, and DUC, alongside human evaluation and analyses of learned skills.It also reports improvements on the auxiliary question- and entailment-generation tasks over their respective previous state of the art.Source: §2 · p. 2
• Compared with Pasunuru et al. (2017), this work adds question generation for saliency, high-level layer-specific sharing, soft parameter sharing, and initialization from pretrained baselines.The paper reports that high-level sharing outperforms alternative sharing approaches, soft sharing significantly improves over hard sharing, and pretrained initialization speeds experimentation.Source: §2 · p. 2

Auxiliary tasks

• Question generation uses SQuAD (Rajpurkar et al., 2016) to generate a question from a sentence containing the answer span; entailment generation uses the generation version of RTE.Earlier summarization work used RTE for graph-based redundancy detection and sentence selection, whereas this work uses multi-task learning.Source: §2 · p. 3

3.1 Baseline Pointer+Coverage Model

• The baseline combines a 2-layer bidirectional LSTM encoder and 2-layer unidirectional decoder with attention, a source-copy pointer, and coverage to reduce repetition.Coverage sums prior attention distributions, and the training loss combines cross-entropy and coverage loss.Source: §3.1 · p. 3

3.2 Two Auxiliary Tasks

• Question generation trains the model to identify salient, answer-bearing information in sentences, using SQuAD sentence-question pairs and shared high-level representations to support multiple document-level facts.Generating all questions from a full document performed poorly because the task was challenging and training data were limited.Source: §3.2 · pp. 3–4
• Entailment generation trains a decoder to produce hypotheses logically following SNLI premises, sharing representations with summarization to discourage contradictions and unrelated or extraneous content.The section reports that the multi-task model is better entailed by its source and has fewer extraneous facts.Source: §3.2 · p. 4

4.1 Layer-Specific Sharing Mechanism

• Higher-level encoder and decoder layers and attention are shared across the three tasks, while lower-level layers remain task-specific.This design reflects the premise that the tasks can share high-level semantic representations while retaining distinct low-level representations.Source: §4.1 · pp. 4–5

4.2 Soft vs. Hard Parameter Sharing

• Soft sharing penalizes L2 distances between shared parameters, loosely coupling task representations rather than forcing hard parameter identity.The primary summarization-task loss uses γ as a hyperparameter; θ_s and ψ_s denote the shared parameter subsets.Source: §4.2 · p. 5

4.3 Fast Multi-Task Training

• Training alternates task mini-batches according to a tunable mixing ratio and starts summarization and auxiliary models from 90%-converged baselines.A fully converged summarization baseline can trap training in a local minimum, while scratch-initialized auxiliary models can pull shared parameters toward random initializations.Source: §4.3 · p. 5

Datasets

• Summarization uses CNN/DailyMail (Hermann et al., 2015; Nallapati et al., 2016) and Gigaword (Rush et al., 2015); auxiliary tasks use SNLI (Bowman et al., 2015) and SQuAD (Rajpurkar et al., 2016).CNN/DM-trained models are also evaluated on DUC-2002 for transfer.Source: §5 · p. 5

Evaluation Metrics

• Summarization results use full-length F1 ROUGE, following Chopra et al. (2016) and Nallapati et al. (2016), and METEOR, following See et al. (2017) with the MS-COCO script (Chen et al., 2015).METEOR is attributed to Denkowski and Lavie (2014).Source: §5 · p. 5

Human Evaluation Criteria

• Amazon MTurk annotators compare anonymized model summaries for relevance and readability, judging relevance by salient, correct, nonredundant content and readability by fluency, grammar, and coherence.Annotators were US-located, had approval rates above 95% and at least 10,000 approved HITs; comparisons included the article and ground-truth summary.Source: §5 · pp. 5–6

Training Details

• Soft, hard, and layer-specific sharing decisions are made using the validation/development set.RNN hidden-state sizes, Adam settings, and mixing ratios are provided in the supplementary material.Source: §5 · p. 6

6.1 Summarization (Primary Task) Results

• The pointer-coverage baseline matches or exceeds See et al. (2017), while entailment- and question-generation multitask variants significantly improve their baselines on both datasets.For entailment generation, improvements are significant on every metric for both datasets; for question generation, they are significant on CNN/DailyMail ROUGE-1, ROUGE-L, and METEOR and on every Gigaword metric.Source: §6.1 · p. 6
• The full three-task model achieves the best CNN/DailyMail and Gigaword scores, significantly improving all metrics on both datasets.Significance is p < 0.01 on CNN/DailyMail ROUGE-1, ROUGE-L, and METEOR, and p < 0.02 on ROUGE-2; on Gigaword it is p < 0.01 on all metrics. It also significantly outperforms See et al. (2017)’s publicly available pretrained result and exceeds its higher-reported results on ROUGE-1 and ROUGE-2.Source: Table 2 · p. 6, §6.1 · p. 7

6.2 Human Evaluation

• Human evaluations favor the three-task model over the baseline on CNN/DailyMail relevance and readability, and over See et al. (2017) on relevance and aggregate scores.Against See et al. (2017), readability is slightly lower, which the authors suggest may reflect more abstractive rewriting encouraged by entailment generation; on Gigaword, the model scores better than its baseline on both relevance and readability.Source: §6.2 · p. 7, Table 3 · p. 7, Table 4 · p. 7

6.3 Generalizability Results (DUC-2002)

• On DUC-2002, the three-task model significantly outperforms the strong baseline on ROUGE-1 and ROUGE-L and See et al. (2017) on all metrics.Models trained on CNN/DailyMail were tested directly on DUC-2002, with beam size retuned using DUC-2003 as validation; the authors report significant improvements over See et al. (2017) even without retuning.Source: §6.3 · p. 7, Table 5 · p. 7, §6.4 · p. 7

6.4 Auxiliary Task Results

• In separate auxiliary-task evaluations, pointer-based entailment generation benefits from copying the premise, while its two-layer model is comparable to the one-layer model.The question-generation model is evaluated against Du et al. (2017), but the supplied passages do not report its comparative outcome.Source: §6.4 · p. 8, Table 6 · p. 8, Table 7 · p. 8

7 Ablation and Analysis Studies

• The final E2+Attn+D1 model significantly outperforms hard-sharing and alternative layer-sharing methods across all metrics.Soft-sharing beats hard-sharing with p < 0.001 in all metrics; the final model beats the alternative sharing methods with p < 0.05.Source: Table 8 · p. 8, §7 · p. 8

Quantitative Improvements in Entailment

• The 2-way entailment-generation model significantly improves entailment classification over baseline and reduces extraneous named entities by 17.2%.Entailment-classification improvement is significant at p < 0.001.Source: Quantitative Improvements in Entailment · p. 8, Table 9 · p. 8
• The 2-way question-generation model significantly improves salient-word matching and yields 93 more cases detecting at least two additional salient keywords than the pointer baseline.The salient-word matching improvement is significant at p < 0.01.Source: Quantitative Improvements in Entailment · p. 9, Table 10 · p. 9
• In qualitative examples, the 3-way model avoids unsupported phrases and recovers salient information missed by See et al. and the baseline.It recovers “josh meekings,” “leigh griffiths,” and “hoops,” while the other models produce “john hartson” and “hampden injustice,” absent from the input.Source: Quantitative Improvements in Entailment · p. 9
• The EG + QG multitask model is more abstractive than See et al. (2017) by the novel n-gram measure.Abstractiveness is measured as the number of novel n-grams between the output summary and source document.Source: Quantitative Improvements in Entailment · p. 9, Table 11 · p. 9

• The multi-task approach adds question generation and entailment generation to help summaries capture salient information and remain logically entailed by the document.Source: §8 · p. 9
• The proposed architecture uses soft and high-level semantic layer-specific parameter sharing.Source: §8 · p. 9
• The approach achieves significant improvements over the state of the art on two popular datasets and in the DUC-2002 transfer setup.Source: §8 · p. 9

A.1 Dataset Details

• CNN/DailyMail uses the original non-anonymized See et al. (2017) version, with 287,226 training, 13,368 validation, and 11,490 test pairs; average source and summary lengths are 781 and 56 tokens.Source: §A.1 · p. 12
• Gigaword uses Rush et al. (2015)’s annotated corpus, treating each article’s first sentence as input and its headline as summary, with about 3.8 million training samples and a 50,000-word vocabulary.Source: §A.1 · p. 12
• For transfer evaluation, DUC-2002 provides 567 documents with one or two human reference summaries; DUC-2003’s 624 documents were used for beam retuning.Source: §A.1 · p. 12
• SNLI supplies entailment-generation data using Pasunuru and Bansal (2017)’s re-splits to prevent train-test overlap and support multiple references; 145,822 unique premise pairs are drawn from 190,113 pairs.Source: §A.1 · p. 12
• For question generation, SQuAD comprehension-question examples are converted into sentence-question pairs by pairing each question with the sentence containing its answer span, following Du et al. (2017).Source: §A.1 · p. 12
• The resulting SQuAD question-generation data contains around 100K sentence-question pairs from 536 articles.Source: §A.1 · p. 12

A.2 Training Details

• All models use shared encoder-decoder embeddings, 256-dimensional LSTM-RNN hidden states, 128-dimensional word embeddings, gradient clipping at 2.0, and Adam with learning rate 0.001.They use no dropout or other regularization, and tuning decisions are made on the appropriate validation/development set. Adam follows Kingma and Ba (2015).Source: §A.2 · p. 12
• CNN/DailyMail models use maximum encoder and decoder lengths of 400 and 100 steps, respectively, and mini-batches of 16.They use uniform initialization in [−0.02, 0.02] and λ = 1.0 for joint cross-entropy and coverage loss; coverage is added only after the attention-and-pointer model converges, reducing the learning rate to 0.0001. Multi-task coverage applies only to summarization, while soft-sharing γ is 5 × 10−5 for 2-way and 1 × 10−5 for 3-way models; inference beam size is 4, following See et al. (2017).Source: §A.2 · p. 12
• Gigaword models use maximum encoder and decoder lengths of 50 and 20 steps, mini-batches of 256, and beam size 5 following Nallapati et al. (2016).LSTM-RNNs are initialized uniformly in [−0.01, 0.01], and Gigaword models do not use coverage.Source: §A.2 · pp. 12, 16
• For CNN/DM-to-DUC transfer, beam sizes tuned on DUC-2003 are 10 for the multi-task model, 4 for the baseline, and 3 for See et al. (2017).The models’ beam sizes are individually retuned for the DUC transfer experiments.Source: §A.2 · p. 16

Multi-Task Learning with Question Generation

• For question generation, the summarization-to-SQuAD mixing ratio is 3:2 for CNN/DailyMail and 100:1 for Gigaword.Source: Multi-Task Learning with Question… · p. 16
• For summarization and entailment generation, the soft-sharing coefficient γ was set to 5 × 10−5 for CNN/DailyMail and 1×10−5 for Gigaword.Source: Multi-Task Learning with Question… · p. 16
• For entailment generation, the summarization-to-SNLI mixing ratio is 3:2 for CNN/DailyMail and 20:1 for Gigaword.Source: Multi-Task Learning with Question… · p. 16
• For entailment generation, the soft-sharing coefficient γ is 5 × 10−5 for CNN/DailyMail and 1 × 10−5 for Gigaword.Source: Multi-Task Learning with Question… · p. 16
• For joint question and entailment generation, CNN/DailyMail uses a 4:3:3 mixing ratio and γ = 5 × 10−5, while Gigaword uses 100:1:5 and γ = 1.5 × 10−6.Source: Multi-Task Learning with Question… · p. 16

A.3.1 Entailment Generation Examples

• Figure 6 presents example outputs from the entailment generation model.Source: §A.3.1 · p. 16

A.3.2 Question Generation Examples

• Figure 7 presents example outputs from the question generation model.Source: §A.3.2 · p. 16