PaperlayerView exact revision on arXiv

Source-linked AI summary

Impossible Distillation: from Low-Quality Model to High-Quality Dataset & Model for Summarization and Paraphrasing

Jaehun Jung, Peter West, Liwei Jiang, Faeze Brahman, Ximing Lu, Jillian Fisher, Taylor Sorensen, Yejin Choi

arXiv:2305.16635v4cs.CLcs.AIcs.LG

TL;DR

Paraphrase and sentence-summarization models need high-quality data, but human supervision is hard to scale and prior approaches use capable large teachers. Impossible Distillation exploits paraphrastic proximity in low-quality LMs to create task data and models, with strong results across tasks and reported dataset measures exceeding those of a much larger corpus.

  • Problem

    High-quality data is central to compact paraphrase and summarization models, but human supervision is difficult to scale and earlier approaches rely on capable, often very large teachers.

    Source: §1 · p. 1

  • Method

    Impossible Distillation uses paraphrastic proximity in a low-quality LM to generate candidate pairs, filter them with critics, and train then self-distill a student.

    Source: §1 · pp. 1–2

  • Results

    Across three tasks, distilled models outperform competitive baselines, while the synthetic dataset beats ParaBank on all reported measures despite being 13 times smaller.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework produces high-quality paraphrase data and models from small, low-quality LMs without strong LLMs or human-authored references.

    Source: §1 · p. 1, §6 · p. 9

    Experiments are limited to sentence paraphrasing and summarization, and the fixed off-the-shelf filters may not be accessible for a wider range of tasks.

    Source: E Limitations and Future Work · p. 15

Abstract

from arXiv · show

We present Impossible Distillation, a novel framework for paraphrasing and sentence summarization, that distills a high-quality dataset and model from a low-quality teacher that itself cannot perform these tasks. Unlike prior works that rely on an extreme-scale teacher model (e.g., GPT3) or task-specific architecture, we hypothesize and verify the paraphrastic proximity intrinsic to pre-trained LMs (e.g., GPT2), where paraphrases occupy a proximal subspace in the LM distribution. By identifying and distilling generations from these subspaces, Impossible Distillation produces a high-quality dataset and model even from GPT2-scale LMs. We evaluate our method on multiple benchmarks spanning unconstrained / syntax-controlled paraphrase generation and sentence summarization. Our model with 770M parameters consistently outperforms strong baselines, including models distilled from ChatGPT, and sometimes, even ChatGPT itself. Also, we find that our distilled dataset from 1.5B LMs exhibits higher diversity and fidelity than up to 13 times larger datasets.

1 Introduction

The introduction frames compact paraphrase and summarization models as constrained by scalable, high-quality data, then proposes distillation from low-quality GPT2-scale LMs through paraphrastic proximity. It reports strong cross-task model results and a synthetic dataset with better reported measures than a corpus 13 times larger.

1 Introduction

  • High-quality data is central to compact performant models, but human supervision is difficult to scale and unsupervised back-translation or auto-encoding can lack diversity and be noisy.Prior LLM distillation instead relies on already capable, often multibillion-scale teachers and can require substantial data-collection budgets.Source: §1 · p. 1
  • Paraphrastic proximity is the premise that a sentence and its paraphrases occupy a proximal subspace in a pretrained LM’s distribution.Source: §1 · p. 1
  • The framework samples context-constrained generations, filters source-paraphrase pairs with off-the-shelf critics, fine-tunes a student, then self-distills it.The resulting student supports paraphrasing; redefining the filters extends the framework to sentence summarization.Source: §1 · p. 2
  • The synthetic dataset from 1.5B LMs scores better on all reported measures than ParaBank, which is 13 times larger.Source: §1 · p. 2
  • Across three tasks, the model distilled from a 1.5B LM outperforms competitive baselines, and the 770M model is human-preferred over ChatGPT-distilled models and sometimes ChatGPT itself.The tasks are unconstrained and syntax-controlled paraphrasing and sentence summarization.Source: §1 · p. 2

2 Paraphrastic Proximity

The section argues that off-the-shelf GPT2 can exhibit paraphrastic proximity when its decoding space is constrained by informative context. Its experiments find more semantically equivalent generations with longer context and lower temperature, with a diversity trade-off at very low temperatures.

2 Paraphrastic Proximity

  • The authors argue that informative context narrows the decoding search space toward proximal paraphrase subspaces, enabling multiple paraphrastic generations from off-the-shelf GPT2.Source: §2 · p. 2
  • Because the full generation distribution is intractable, the study approximates it by sampling many completions under contextual constraints.Source: §2 · pp. 2–3
  • Longer context makes GPT2 generations more likely to be semantically equivalent, while average pair-wise Self-BLEU remains around 321.The experiment samples 100 completions for each of 1,000 XSUM contexts and identifies equivalence through bidirectional entailment.Source: §2 · p. 3, Figure 3 · p. 3
  • Lower decoding temperature sharply increases the ratio of semantically equivalent pairs, but temperatures that are too low make generations nearly identical and undesirable as paraphrases.Source: §2 · p. 3

3 Impossible Distillation

Impossible Distillation turns off-the-shelf teacher generations into filtered paraphrase data, a fine-tuned student, and a self-distilled task model. Its evaluation-based pipeline can also be adapted to sentence summarization by redefining the filters.

3.1 Pair Generation

  • Sample 1–5-sentence contexts, generate 100 next-sentence samples per context, then enumerate their combinations as candidate source–paraphrase pairs.The authors report that LM-generated contexts suffice for a diverse, domain-specific data pool without external data.Source: §3.1 · p. 3

3.2 Filtering with Critics

  • Bidirectional NLI checks meaning, ROUGE-L and tree edit distance enforce surface and syntactic difference, and duplicate filtering promotes corpus diversity.ROUGE filtering promotes lexical divergence, while tree edit distance helps prevent synonym swaps from gaming token overlap.Source: §3.2 · p. 4

3.3 Distilling Student Model

  • The student is first fine-tuned on filtered teacher pairs, then on critic-filtered self-generated paraphrases, yielding an end-stage model and reusable dataset.The authors report that this self-distillation improves task-model performance in their ablation study.Source: §3.3 · p. 4

3.4 Endowing Controllability

  • Parsing each paraphrase yields a constituency-tree-conditioned training triple, allowing the student to generate a paraphrase from its source and a target tree.The controllable model is trained by fine-tuning the student on the source sentence and tree.Source: §3.4 · pp. 4–5

3.5 DIMPLE and Impossible-T5

  • GPT2-XL and BioGPT teachers (1.5B) with a T5-large student (770M) yield Impossible-T5 and a 4M-pair DIMPLE2 corpus.The Table 1 caption reports that DIMPLE has better diversity than other datasets, including one constructed by prompting ChatGPT.Source: §3.5 · p. 5, Table 1 · p. 5

3.6 Sentence Summarization

  • For sentence summarization, the evaluation-based pipeline is adapted by redefining critics; a length filter can enforce outputs shorter than inputs.The paper contrasts paraphrasing, which preserves all source information, with summarization, which can omit tangential details.Source: §3.6 · p. 5

4 Experiments

Across paraphrase and summarization benchmarks, Impossible Distillation produces a diverse paraphrase dataset and a compact model that compares favorably with larger-data baselines. The experiments also show that dataset diversity and both critic filtering and self-distillation matter to performance.

4.1 Dataset Evaluation

  • DIMPLE outperforms the compared paraphrase datasets on semantic fidelity, lexical diversity, and syntactic diversity, including ParaBank1, which is more than 13 times larger.The dataset is generated synthetically from 1.5B LMs.Source: §4.1 · pp. 5–6

4.2 Unconstrained Paraphrase Generation

  • Up to 10% relative improvement over same-size models trained on order-of-magnitude larger ParaBank datasets; Impossible-T5 is also competitive with 175B LLMs across general and domain-specific benchmarks.On ParaSCI, it even outperforms ChatGPT.Source: §4.2 · p. 6, Table 2 · p. 6

4.3 Syntactically Controlled Paraphrase Generation

  • Impossible-T5 outperforms baselines on syntactically controlled paraphrase generation across all reported metrics except ROUGE-L.The table caption reports gains in both paraphrase quality and controllability.Source: §4.3 · p. 7, Table 4 · p. 7

4.4 Sentence Summarization

  • Redefining the composition of filters lets Impossible Distillation generalize to sentence summarization, while repurposing a paraphrase corpus yields suboptimal performance.The evaluation reports BERTScore-F1 and ROUGE-L and includes qualitative assessments of 100 summaries per model by six crowd-source workers.Source: §4.4 · p. 7, Table 5 · p. 7

4.5 Ablation Study

  • Removing self-distillation lowers BERT-iBLEU by 3.1, while critic-filtered ChatGPT-Para reaches 64.1 on MRPC but remains below Impossible-T5.The authors identify teacher-data diversity as crucial to fully eliciting student capability; the best configuration combines critic filtering and self-distillation.Source: §4.5 · pp. 7–8, Table 6 · p. 7

5 Related Works

Prior unsupervised paraphrasing and summarization approaches use task-specific surrogate tasks or large language models, while language-model data distillation often assumes a capable teacher and manually constructed prompts. Impossible Distillation instead leverages paraphrastic knowledge intrinsic to an off-the-shelf language model.

Unsupervised Paraphrasing and Summarization

  • Unsupervised paraphrasing and summarization have used back-translation (Wieting and Gimpel, 2018; Hu et al., 2019b) and autoencoding, but these surrogates offer weak supervision and can require engineered perturbations or auxiliary constraints.Other methods may require teacher-model retraining.Source: §5 · pp. 8–9

Task-solving with Language Model

  • Large-scale pretraining can give language models complex-task knowledge, while instruction tuning and human feedback can elicit capabilities but often require curated annotated data.The paper proposes leveraging intrinsic paraphrastic knowledge rather than human annotation.Source: §5 · p. 9

Data Generation with Language Model

  • Language-model data distillation transfers teacher knowledge to students through textual datasets, but related approaches typically assume a strong teacher and manually constructed prompts.Impossible Distillation generalizes data generation to an off-the-shelf setup without dependence on the teacher's target-task capability.Source: §5 · p. 9

6 Conclusion

Impossible Distillation distills paraphrase datasets and models from small, low-quality LMs, using paraphrastic proximity and critic-guided distillation. Its conclusion emphasizes performance and controllability across tasks and domains, as well as the quality of the resulting dataset.

6 Conclusion

  • The framework produces high-quality paraphrase datasets and models from small, low-quality LMs, without training on human-authored paraphrases.The authors report gains in performance and controllability across domains and tasks.Source: §6 · p. 9
  • DIMPLE has higher fidelity and diversity than paraphrase datasets up to an order of magnitude larger.Source: §6 · p. 9
  • The authors present Impossible Distillation as a way to rediscover and amplify under-explored capabilities of off-the-shelf language models.Source: §6 · p. 9

A Implementation Details

Implementation uses GPT2-XL and BioGPT-large teachers, task-specific critic filters, and a graph-based diversity filter. The reported unsupervised-paraphrase comparison finds Impossible-T5 ahead of all baselines on both benchmarks and metrics.

A.1 Pair Generation

  • GPT2-XL and BioGPT-large, each with 1.5B parameters, generate paraphrases using nucleus sampling with top_p = 0.9 and temp = 1.0.GPT2 uses a simple prefix for general-domain generation; BioGPT generates without a prefix.Source: §A.1 · p. 14

A.2 Critic Models for Paraphrase Generation

  • Paraphrase critics retain pairs with bidirectional entailment scores of at least 0.75, using ROUGE and TED thresholds of 0.75 and 12 for dissimilarity.Source: §A.2 · p. 14
  • The diversity filter connects duplicate pairs in a graph and keeps the pair with the highest summed bidirectional entailment score per component.With DFS, connected-component search takes O(|P| + |E|) time.Source: §A.2 · p. 14
  • Impossible-T5 outperforms all unsupervised baselines on Quora and MSCOCO in both reported metrics.Source: Table 8 · p. 14

B Comparison with Unsupervised Paraphrase Generation Methods

Impossible-T5 is compared with unsupervised paraphrase-generation methods on Quora and MSCOCO. It consistently outperforms the baselines across both benchmarks and both reported metrics.

B Comparison with Unsupervised Paraphrase Generation Methods

C.1 Critic Models for Summarization

For sentence summarization, Impossible Distillation redefines its generation filters and produces DIMSUM and a specialized Impossible-T5 model. The filter changes account for summaries that may omit details while remaining supported by the source.

C.1 Critic Models for Summarization

  • The generation pipeline adapts to sentence summarization by redefining its task filters.Source: §C.1 · p. 14
  • The summarization filters require unidirectional entailment, since summaries may omit less important source details, and discard pairs whose compression ratio exceeds a threshold.The semantic threshold is 0.75 and the compression-ratio threshold is 0.8; a diversity filter removes duplicate source-summary pairs.Source: §C.1 · p. 15

C.2 DIMSUM and Impossible-T5

  • After self-distillation, DIMSUM contains 1.5M sentence-summary pairs across news and biomedical domains, alongside the specialized Impossible-T5 model.Source: §C.2 · p. 15

D Human Evaluation Details

The human evaluation used MTurk annotators to rate paraphrase quality on fluency, faithfulness, and dissimilarity.

D Human Evaluation Details

  • Paraphrases were rated by six distinct MTurk evaluators, with Hit Rate over 99%, under IRB approval.Workers received $15 per hour.Source: D Human Evaluation Details · p. 15
  • Annotators used a 3-point Likert scale to assess fluency, faithfulness, and dissimilarity.Faithfulness concerned preserving the original content without hallucination; dissimilarity concerned being sufficiently different from the original.Source: D Human Evaluation Details · p. 15
  • Figure 4 shows the MTurk interface used for paraphrase evaluation.Source: D Human Evaluation Details · p. 15

E Limitations and Future Work

The authors identify task scope, filter availability, and possible risk transfer as constraints, while outlining potential extensions and intervention options.

E Limitations and Future Work

  • Experiments are limited to sentential paraphrasing and summarization; translation is proposed as a future extension using multilingual LMs and cross-lingual filters.Source: E Limitations and Future Work · p. 15
  • Fixed filters may not be accessible across a wider range of tasks, constraining where the current filtering approach can be used.The filters remain frozen throughout the distillation pipeline.Source: E Limitations and Future Work · p. 15
  • For harder tasks such as commonsense reasoning, defining which pairs qualify as good examples is non-trivial; co-evolving the task and filter models is proposed.Source: E Limitations and Future Work · p. 16
  • Distillation may amplify undesirable language-model properties; bias and toxicity could be inherited in more open-ended settings.The paper focuses on conditional generation, where outputs are closely bound to inputs.Source: E Limitations and Future Work · p. 16
  • The textual dataset can be interpreted and evaluated, allowing users to intervene and selectively filter amplified knowledge.The authors suggest combining this transparency with automatic bias detection and reduction techniques.Source: E Limitations and Future Work · p. 16
Loading 2305.16635v4…