PaperlayerView exact revision on arXiv

Source-linked AI summary

Meta-Transfer Learning for Low-Resource Abstractive Summarization

Yi-Syuan Chen, Hong-Han Shuai

arXiv:2102.09397v2cs.CLcs.LG

TL;DR

Low-resource abstractive summarization faces costly labeling and potential domain shifts when transferring to novel tasks. MTL-ABS combines pre-trained models, adapter-based meta-learning, and selected source corpora; it achieves state-of-the-art results on six corpora with 0.7% of previous work’s trainable parameters.

  • Problem

    Abstractive summarization requires costly, sometimes domain-expert annotations, while transfer to novel tasks may face domain shifts and limited target labels.

    Source: Introduction · p. 1

  • Method

    MTL-ABS uses adapters to combine pre-trained models with meta-learning and selects multiple source corpora using similarity criteria.

    Source: Introduction · p. 2, Meta-Datasets · pp. 4–5

  • Results

    MTL-ABS achieves state-of-the-art on 6 corpora in low-resource abstractive summarization with only 0.7% of previous work’s trainable parameters.

    Source: Introduction · p. 2, Conclusion · p. 7

  • Takeaways & Limitations

    The paper provides general criteria for source-corpus choice and reports that the approach outperforms the state of the art on six diverse corpora.

    Source: Introduction · p. 2, Conclusion · p. 7

    The evaluated low-resource setting limits target-corpus labeled training examples to under 100.

    Source: Methodologies · p. 3

Abstract

from arXiv · show

Neural abstractive summarization has been studied in many pieces of literature and achieves great success with the aid of large corpora. However, when encountering novel tasks, one may not always benefit from transfer learning due to the domain shifting problem, and overfitting could happen without adequate labeled examples. Furthermore, the annotations of abstractive summarization are costly, which often demand domain knowledge to ensure the ground-truth quality. Thus, there are growing appeals for Low-Resource Abstractive Summarization, which aims to leverage past experience to improve the performance with limited labeled examples of target corpus. In this paper, we propose to utilize two knowledge-rich sources to tackle this problem, which are large pre-trained models and diverse existing corpora. The former can provide the primary ability to tackle summarization tasks; the latter can help discover common syntactic or semantic information to improve the generalization ability. We conduct extensive experiments on various summarization corpora with different writing styles and forms. The results demonstrate that our approach achieves the state-of-the-art on 6 corpora in low-resource scenarios, with only 0.7% of trainable parameters compared to previous work.

Introduction

Low-resource abstractive summarization must contend with costly annotations, domain shifts, and limited target labels. MTL-ABS combines pre-trained models with meta-learning over multiple corpora and reports state-of-the-art results on six corpora with 0.7% of previous work’s trainable parameters.

Introduction

  • Low-resource abstractive summarization seeks to use past experience to improve performance with limited target-corpus labels, which are costly and may require domain expertise.Articles also vary in writing style and form, including news, social media posts, and scientific papers.Source: Introduction · p. 1
  • Jointly using pre-trained models and meta-learning raises challenges from large-model overfitting, meta-learning gradient problems, and the need to construct diverse source tasks.The paper notes that suitable source corpora for summarization meta-datasets remain an open question.Source: Introduction · pp. 1–2
  • MTL-ABS meta-learns a limited number of inter-layer parameters and uses small-initialized skip connections to reduce interference and model complexity.The paper contrasts this design with adding consecutive randomly initialized layers, which may reduce performance and be difficult to recover from with limited target labels.Source: Introduction · p. 2
  • Using multiple corpora can diversify source tasks, but inappropriate choices may cause negative transfer, so the paper studies similarity criteria and offers general selection criteria.The authors also present MTL-ABS as a combination of transfer learning and meta-learning for low-resource summarization.Source: Introduction · p. 2
  • MTL-ABS achieves state-of-the-art results on 6 corpora in low-resource abstractive summarization with only 0.7% of previous work’s trainable parameters.The paper’s stated result compares its trainable-parameter count with previous work.Source: Introduction · p. 2

Related Works

Prior work transfers pre-trained language models to summarization, including through task-specific layers and specialized pre-training objectives. In low-resource settings, existing approaches have relied on a single large corpus, while this paper’s framework uses adapters and multiple corpora.

Related Works

  • Adapters offer a compact alternative to full-model fine-tuning: only adapter modules are learnable in the BERT transfer approach described.The cited PAL approach instead uses multi-head attention layers residually connected to the base model.Source: Related Works · p. 2
  • Abstractive summarization pre-training has used encoder language modeling followed by extractive fine-tuning and decoder training, as well as decoder pre-training with Gap Sentence Generation.The Gap Sentence Generation objective reconstructs important sentences masked from the article.Source: Related Works · p. 2
  • Earlier low-resource approaches described here use a single large corpus, whereas MTL-ABS uses limited trainable parameters and multiple corpora selected by proposed criteria to address domain shift and overfitting concerns.Figure 1 places adapters after every encoder and decoder feed-forward layer; during meta-transfer learning, only adapters and layer normalization layers are learnable.Source: Related Works · p. 2, Figure 1 · p. 3

Methodologies

The paper defines low-resource abstractive summarization as learning from limited target-corpus examples plus indirect experience, and develops MTL-ABS around a pretrained summarizer and meta-learned adapters. It forms meta-datasets from multiple source corpora, with selection criteria informed by observed relationships between source choice and performance.

Methodologies

  • Low-resource abstractive summarization combines limited target-corpus article-summary pairs with indirect experience, such as pretrained models or related corpora, and is evaluated with ROUGE.The considered target setting has fewer than 100 labeled training examples.Source: Methodologies · p. 3

Base Model

  • The base summarizer uses a Transformer encoder-decoder with BERT initialization for the encoder, which is first fine-tuned on the chosen pretraining corpus with an extractive objective.The encoder uses one self-attention layer and the decoder uses two.Source: Base Model · p. 3

Adapters

  • Adapters are bottleneck feed-forward modules with a skip connection, inserted after every feed-forward layer in the encoder and decoder to restrict meta-trainable parameters.The restriction is intended to prevent overfitting and gradient instability when applying MAML to a large pretrained model.Source: Adapters · p. 3

Meta-Transfer Learning for Summarization

  • After base-model training, source-corpus examples are sampled into balanced tasks; inner-loop adaptation and outer-loop testing-loss optimization update task parameters and the meta-learner.Separate inner-loop optimizers are used for tasks from different corpora because shared momentum-based optimization could slow convergence.Source: Meta-Transfer Learning for Summarization · p. 4

Meta-Datasets

  • Source corpora are ranked by cosine similarity, ROUGE precision, and article length; observed results favor statistical over semantic similarity, high information density over coverage, and similar article lengths.Corpora from the same domain do not always provide better knowledge.Source: Meta-Datasets · pp. 4–5

Experiments

Low-resource experiments compare MTL-ABS with PEGASUS and examine parameter efficiency, overfitting, gradient stability, and source-corpus selection. Results vary across corpora and similarity criteria.

Implementation Details

  • The experiments use a meta-batch of three tasks, four examples per task, four inner gradient steps, and 6K meta-steps.The encoder has 12 Transformer layers, the decoder has 6, and adapter hidden size is 64.Source: Implementation Details · p. 5

Corpora

  • Targets are excluded from source corpora, which span news, multi-document, social-post, and scientific summarization.CNN/DailyMail is used to pre-train the summarization framework.Source: Corpora · p. 5

Low-Resource Performance

Preventing Overfitting Problem

Preventing Gradient Problem in Meta Learning

  • Full-model meta-transfer training has unstable gradient norms and fails after 1,500 meta-steps, while MTL-ABS uses adapter modules with skip-connections.Figure 2 compares gradient-norm dynamics when all parameters or only adapters are trainable.Source: Preventing Gradient Problem in Meta… · p. 6, Figure 2 · p. 7

Impact on the Choice of Source Corpora

  • Source candidates for AESLC are ranked by embedding, cosine, length, ROUGE-2 recall, and ROUGE-2 precision criteria to form meta-datasets.The study examines rankings at different positions along each criterion.Source: Impact on the Choice of Source Corpora · pp. 6–7
  • For AESLC, source performance is more correlated with word overlap and article length than embedding similarity; ROUGE-2 precision is a better criterion than recall.The authors note that some similarity-based choices perform worse than random selection.Source: Impact on the Choice of Source Corpora · p. 7
  • For Gigaword as target, WikiHow and AESLC yield ROUGE-2 of 11.77, versus 10.23 for Multi-News and NEWSROOM.The authors conclude that same-domain sources may not always provide better transfer knowledge.Source: Impact on the Choice of Source Corpora · p. 7

Conclusion

MTL-ABS combines transfer learning and meta learning through adapter modules, and outperforms the state of the art on six diverse corpora. The paper also provides criteria for choosing source corpora and identifies future directions for adapting to distant sources and novel tasks.

Conclusion

  • MTL-ABS outperforms the state of the art on 6 diverse corpora.Source: Conclusion · p. 7
  • Adapter modules serve as a bridge for combining transfer learning and meta learning in low-resource abstractive summarization.Source: Conclusion · p. 7
  • The paper investigates and provides general criteria for choosing source corpora, a topic not previously studied in meta-learning for NLP.Source: Conclusion · p. 7
  • Future work will explore better use of distant source corpora and modulation of pre-trained parameters for adapting to novel tasks.The authors note that similar source corpora are not guaranteed to be available.Source: Conclusion · p. 7

Related papers

Loading 2102.09397v2…