PaperlayerView exact revision on arXiv

Source-linked AI summary

Prompt-based Pseudo-labeling Strategy for Sample-Efficient Semi-Supervised Extractive Summarization

Gaurav Sahu, Olga Vechtomova, Issam H. Laradji

arXiv:2311.09559v3cs.CLcs.AI

TL;DR

Semi-supervised learning is relatively underexplored for extractive text summarization. The paper uses LLM prompting to score and relabel pseudo-labels, outperforming prior semi-supervised methods across three datasets and achieving strong results against fully supervised methods in both data-scarce and data-abundant settings.

  • Problem

    Semi-supervised learning is relatively underexplored for extractive text summarization.

    Source: §1 · p. 2

  • Method

    The approach uses teacher confidence and LLM prompting to select and relabel pseudo-labels for semi-supervised extractive summarization.

    Source: §1 · p. 2, §3.2 · p. 3, §3.4 · p. 5

  • Results

    Across three datasets, the method outperforms prior semi-supervised methods on ROUGE-1, ROUGE-2, ROUGE-L, and L-Eval; it is competitive with fully supervised methods using 6× fewer labeled examples and outperforms them in data-abundant settings.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The method achieves performance competitive with fully supervised counterparts while using 6× fewer labeled examples in low-resource summarization setups.

    Source: §6 · p. 8

    The approach relies on LLM pseudo-label scoring and relabeling, which cannot currently be judged automatically and requires a human to assess pseudo-label quality.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Semi-supervised learning (SSL) is a widely used technique in scenarios where labeled data is scarce and unlabeled data is abundant. While SSL is popular for image and text classification, it is relatively underexplored for the task of extractive text summarization. Standard SSL methods follow a teacher-student paradigm to first train a classification model and then use the classifier's confidence values to select pseudo-labels for the subsequent training cycle; however, such classifiers are not suitable to measure the accuracy of pseudo-labels as they lack specific tuning for evaluation, which leads to confidence values that fail to capture the semantics and correctness of the generated summary. To address this problem, we propose a prompt-based pseudo-labeling strategy with LLMs that picks unlabeled examples with more accurate pseudo-labels than using just the classifier's probability outputs. Our approach also includes a relabeling mechanism that improves the quality of pseudo-labels. We evaluate our method on three text summarization datasets: TweetSumm, WikiHow, and ArXiv/PubMed. We empirically show that a prompting-based LLM that scores and generates pseudo-labels outperforms existing SSL methods on ROUGE-1, ROUGE-2, and ROUGE-L scores on all the datasets. Furthermore, our method achieves competitive L-Eval scores (evaluation with LLaMa-3) as a fully supervised method in a data-scarce setting and outperforms fully supervised method in a data-abundant setting.

1 Introduction

Extractive summarization faces costly annotation, yet SSL remains underexplored and teacher confidence is not tuned to judge pseudo-label accuracy. This work uses LLM prompting to score and relabel pseudo-labels, reporting results across three benchmark datasets.

1 Introduction

  • The work addresses costly annotation and the underexplored use of SSL for extractive summarization.Source: §1 · pp. 1–2
  • LLMs score shortlisted pseudo-labels because teacher classifiers are not specifically tuned to measure pseudo-label accuracy.Source: §1 · p. 2
  • It is competitive with fully supervised models using 6× fewer labels in data-scarce settings and outperforms them in data-abundant settings.Source: §1 · p. 2
  • The method outperforms prior semi-supervised approaches on ROUGE-1, ROUGE-2, ROUGE-L, and L-Eval across three benchmark datasets.Source: §1 · p. 2
  • Prompt-based LLM relabeling generates high-quality summaries for unlabeled examples.Source: §1 · p. 2

2 Related Work

Prior extractive summarization methods span surface-feature approaches, supervised models requiring annotated data, and domain-tailored semi-supervised methods. This work positions prompt-based LLM pseudo-labeling as a more sample-efficient approach that can adapt across domains.

2 Related Work

  • Earlier unsupervised extractive methods used graph ranking, Wikipedia knowledge, and fuzzy logic, but relied on surface features and could not handle dangling anaphoras.Source: §2 · p. 2
  • Supervised methods need costly annotated datasets, while Zhuang et al. (2023) target dialog summarization with a less flexible, domain-tailored SSL objective.Source: §2 · p. 2
  • Unlike fully supervised extract-then-generate work and semi-supervised dialog summarization, this method uses prompt-based LLMs across domains with fewer labeled and unlabeled examples.Source: §2 · pp. 2–3

3 Methodology

The method iteratively trains a teacher-student summarization pipeline, then uses prompted LLMs to relabel and score teacher-generated pseudo-labels. The highest-scoring examples are added to training for the next cycle.

3.1 Stage 1: Training the Teacher Model

  • PreSumm trains a teacher on available labeled examples and selects the highest-probability sentences in their original order to form extractive summaries.Source: §3.1 · p. 3

3.2 Stage 2: Generating Pseudo-labels using the Teacher Model

  • The teacher generates unlabeled-set summaries, then ranks them by average probability of selected sentences and shortlists 50 pseudo-labels.Source: §3.2 · pp. 3–4

3.3 Stage 3: LLM Relabeling of Teacher’s Pseudo-labels

  • A prompted LLM generates replacement summaries for each shortlisted teacher pseudo-label.The generation prompt asks the LLM to output sentence probabilities, which are used to form the extractive summary.Source: §3.3 · p. 4

3.4 Stage 4: LLM Scoring of Pseudo-labels

  • LLaMA-3 scores relabeled summaries against criteria including concision, key-point coverage, and extractiveness; the five highest-scoring examples join the labeled set.Source: §3.4 · pp. 4–5
  • The four stages repeat for multiple cycles, and the final model generates summaries for unseen test examples.Source: §3.4 · p. 5

4 Experimental Setup

The experiments cover three summarization datasets with different document types and labeling conditions. Models are trained and evaluated under two labeled-data settings and compared with supervised, semi-supervised, and pipeline variants.

4.1 Datasets

  • TweetSumm covers customer-service conversations, while WikiHow and ArXiv/PubMed provide article and scientific-paper summaries; the latter two lack extractive labels and use sampled unlabeled pools.Source: §4.1 · p. 5

4.2 Training and Evaluation Details.

  • Training uses 50 labeled examples in the data-scarce setting or 500 in the data-abundant setting, with five pseudo-labels added per cycle for 50 cycles; evaluation reports ROUGE and L-Eval.Source: §4.2 · pp. 5–6

4.3 Baselines

  • Baselines include PreSumm at 50 and 300 labeled examples, teacher-student learning, confidence-only selection, random selection, and variants that use GPT-4 or LLaMA-3 for relabeling or scoring.Source: §4.3 · p. 6

5 Results

The results compare pseudo-label selection and relabeling strategies, then show that the proposed approach is competitive with 300-label supervision using 50 labels and outperforms fully supervised models in data-abundant settings.

5.1 Comparison of Pseudo-label Selection Strategies

  • Teacher confidence gives slight gains, GPT-4 scoring improves results further, and combining GPT-4 scoring with relabeling greatly raises selected pseudo-label quality.The qualitative comparison evaluates the top five labels by ROUGE-2 in the data-scarce setup; LLaMA-3 shows similar trends for scoring.Source: Figure 4 · p. 6, Table 2 · p. 7, §5.1 · pp. 7–8

5.2 Effect of Relabeling

  • BERTbase R-1 rose from 52.3 to 58.9 on TweetSumm, 25.2 to 26.1 on WikiHow, and 27.7 to 29.1 on ArXiv/PubMed with relabeling.Relabeling also helps without GPT-4 scoring, with smaller relative gains at 500 labels; on TweetSumm, the R-1 difference between the two leading relabeling methods was significant (Welch’s p-value < 0.016).Source: Table 2 · p. 7, §5.2 · p. 8

5.3 On Sample Efficiency of Different Approaches

  • 50-label GPT-4 relabeling and scoring is competitive with 300-label PreSumm; in data-abundant settings, LLaMA semi-supervision outperforms fully supervised models on ROUGE and L-Eval.The 50-label method has better L-Eval on WikiHow and ArXiv/PubMed and is competitive on TweetSumm.Source: Table 3 · p. 7, §5.3 · p. 8
  • GPT-4 scoring and relabeling boost ROUGE-1 and peak and stabilize under 20 cycles (100 pseudo-labels) across datasets, while random selection is highly unstable.Each cycle adds five pseudo-labels; confidence plus GPT-4 scoring is more stable than confidence alone.Source: Figure 5 · p. 8, §5.3 · p. 8

6 Conclusion

The paper proposes prompt-based pseudo-labeling for semi-supervised extractive summarization, combining teacher confidence with GPT-4 relabeling and ranking to improve performance with fewer labeled examples.

6 Conclusion

  • The paper proposes a prompt-based pseudo-labeling strategy for semi-supervised extractive text summarization.It combines teacher confidence with LLM knowledge to improve pseudo-labeling in low-resource summarization setups.Source: §6 · p. 8
  • A GPT-4 critic relabels and ranks pseudo-labels, producing highly performant summarization models.The conclusion describes this as part of the proposed approach to improve pseudo-label quality.Source: §6 · p. 8
  • The resulting models are competitive with fully supervised counterparts while using 6× fewer labeled examples.Source: §6 · p. 8

Limitations

The authors identify two limitations: LLM-generated pseudo-labels cannot yet be judged automatically, and the relabeling stage may use resources on summaries that are already good. Both constrain the method’s practical efficiency and oversight.

Limitations

  • LLM pseudo-label scoring and relabeling are imperfect, and the authors have no automatic way to judge their quality.They say human review is needed to assess the pseudo-labels.Source: Limitations · p. 9
  • Stage 3 relabels all 50 pseudo-labels selected by PreSumm, although this is cheaper than relabeling every pseudo-label.The authors note that the initial shortlist contains some good-quality summaries.Source: Limitations · p. 9
  • A strategy to decide which pseudo-labels need relabeling could avoid relabeling already good summaries and further reduce costs for closed-source models such as GPT-4.Source: Limitations · p. 9

Ethics Statement

The method uses GPT and LLaMA-3 to generate examples, but their generations may contain biases despite instruction tuning. The authors recommend human monitoring in real-world use and suggest debiasing models for specific applications.

Ethics Statement

Related papers

Loading 2311.09559v3…