PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learning

Jiahui Gao, Renjie Pi, Yong Lin, Hang Xu, Jiacheng Ye, Zhiyong Wu, Weizhong Zhang, Xiaodan Liang, Zhenguo Li, Lingpeng Kong

arXiv:2205.12679v2cs.CL

TL;DR

PLM-generated training data can contain low-quality samples, while prior remedies use heuristic filtering or costly human correction. SUNGEN learns data-quality weights through noise-robust bilevel reweighting using synthetic data without human annotation. Across eight text-classification tasks, SUNGEN-LSTM shows a 9.8% relative improvement over ZEROGEN on average accuracy.

  • Problem

    PLM-generated data can include low-quality samples that undermine task-model performance, while heuristic filtering and human correction require manual tuning or substantial cost.

    Source: §1 · pp. 1–2

  • Method

    SUNGEN uses bilevel reweighting with a noise-robust outer loss to learn sample weights from synthetic data without human annotation.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    9.8% relative improvement over ZEROGEN on average across all tasks for SUNGEN-LSTM, evaluated on eight text-classification datasets.

    Source: §1 · p. 2, §5.2 · p. 7

  • Takeaways & Limitations

    SUNGEN focuses on improving synthetic-data quality and can be jointly applied with other data-generation pipelines.

    Source: §7 · p. 9

    The theoretical analysis assumes a weighting function exists that maps the synthetic data distribution to the clean data distribution.

    Source: §4 · p. 5

Abstract

from arXiv · show

There is a rising interest in further exploring the zero-shot learning potential of large pre-trained language models (PLMs). A new paradigm called data-generation-based zero-shot learning has achieved impressive success. In this paradigm, the synthesized data from the PLM acts as the carrier of knowledge, which is used to train a task-specific model with orders of magnitude fewer parameters than the PLM, achieving both higher performance and efficiency than prompt-based zero-shot learning methods on PLMs. The main hurdle of this approach is that the synthesized data from PLM usually contains a significant portion of low-quality samples. Fitting on such data will greatly hamper the performance of the task-specific model, making it unreliable for deployment. Previous methods remedy this issue mainly by filtering synthetic data using heuristic metrics(e.g., output confidence), or refining the data with the help of a human expert, which comes with excessive manual tuning or expensive costs. In this paper, we propose a novel noise-robust re-weighting framework SunGen to automatically construct high-quality data for zero-shot classification problems. Our framework features the ability to learn the sample weights indicating data quality without requiring any human annotation. We theoretically and empirically verify the ability of our method to help construct good-quality synthetic datasets. Notably, SunGen-LSTM yields a 9.8% relative improvement than the baseline on average accuracy across eight different established text classification tasks.

1 INTRODUCTION

Data-generation-based zero-shot learning trains small task models on PLM-generated data, but low-quality samples can undermine their performance. SUNGEN learns sample weights without human annotation, and the paper reports theoretical and empirical support for its approach across eight text-classification datasets.

1 INTRODUCTION

  • ZEROGEN's test accuracy can decline as training accuracy rises, indicating overfitting to noisy PLM-generated data.The reported noisy samples include corrupted labels and task-irrelevant examples.Source: §1 · p. 2, Figure 1 · p. 2
  • Human-active correction can address noisy synthetic examples, but it incurs considerable cost and may be impractical.Heuristic sample weighting also requires task-specific knowledge and excessive manual tuning.Source: §1 · p. 2
  • Prior bilevel reweighting methods depend on a clean validation set, a major limitation in zero-shot settings.SUNGEN instead seeks an outer objective that learns weights using only noisy synthetic data.Source: §1 · p. 2
  • SUNGEN uses bilevel reweighting with a noise-robust outer objective to learn sample weights without human annotation.The inner loop trains on weighted synthetic data; the outer loop uses a noise-robust loss to guide the weights.Source: §1 · p. 2
  • Across eight text-classification datasets, SUNGEN outperforms the current baseline by large margins.The introduction also reports theoretical justification and empirical verification of reliable noise-free dataset recovery using synthetic data only.Source: §1 · p. 2

2 BACKGROUND

Prompt-based zero-shot learning asks a PLM to classify prompted examples, while data-generation-based methods use PLM-produced examples to train a smaller task model. The latter can improve efficiency, but synthetic data may be noisy and its weights need automatic optimization.

2.1 PROMPT-BASED ZERO-SHOT LEARNING

  • Prompt-based prediction formats a query with a manually designed prompt and selects the label whose words receive the highest PLM probability.The approach can be difficult to use to fully leverage task-specific PLM knowledge and still requires inference on a cumbersome PLM.Source: §2.1 · pp. 2–3

2.2 EFFICIENT ZERO-SHOT LEARNING VIA DATA GENERATION

  • Data-generation-based zero-shot learning samples a pseudo label, prompts a PLM to generate an input, and pairs them into a synthetic training set.A tiny task model, such as a one-layer Bi-LSTM, is then trained on that set for more efficient training and inference.Source: §2.2 · p. 3
  • ZEROGEN performance can rapidly decline after several epochs, consistent with overfitting to noisy samples.The proposed response is to optimize sample weights automatically rather than rely on heuristic weighting that requires task-specific knowledge.Source: §2.2 · p. 3

3 METHOD

SUNGEN learns sample weights through bilevel optimization, using weighted training of a task model and a noise-robust outer loss on synthetic validation data. Its theoretical claims rely on stated conditions, including that the clean distribution can be represented by reweighting the synthetic distribution.

3 METHOD

  • SUNGEN's inner loop trains a task model with weighted cross-entropy on synthetic training data, while its outer loop uses a noise-robust loss on synthetic validation data to update sample weights.The two procedures alternate, and the outer gradient is calculated via truncated backpropagation.Source: §3 · p. 4
  • SUNGEN avoids the clean validation set required by previous bilevel reweighting approaches.Its outer objective evaluates the trained task model on a synthetic validation set.Source: §3 · pp. 3–4
  • The noise-robust loss has the same global minimizer on clean and noisy data when most training samples are correctly labelled.This property supports optimizing sample weights using a noisy validation dataset; the method uses reversed cross-entropy as one such loss.Source: §3 · p. 5
  • SUNGEN's continuous weights can select subsets at arbitrary budgets, with Bernoulli sampling based on normalized weights.The text states that training on selected high-weight data can perform on par with weighted training over the full dataset while being more efficient.Source: §3 · p. 5

4 THEORETICAL ANALYSIS

The analysis shows that, under its stated assumptions, SUNGEN can recover optimal sample weights and model parameters using only noisy synthetic data. With finite samples and an approximate bilevel solution, its test performance is close to that of an oracle model.

4 THEORETICAL ANALYSIS

  • The analysis assumes a robust loss whose noisy-data minimizer matches the clean-data minimizer under mild conditions, and a unique robust-loss optimum on clean data.The optimization argument uses that loss to find the optimal model parameters from synthetic data and then optimize the corresponding weights.Source: §4 · pp. 5–6
  • Assumption 1 posits that a weighting function maps the synthetic data distribution to the clean distribution, which the paper motivates by synthetic data’s wide coverage.The paper states that a suitable weight may recover the clean distribution by reweighting synthetic data.Source: §4 · p. 5
  • Under Assumptions 1 and 2 and Property 1, SUNGEN can uniquely recover the optimal sample weights and resulting model parameters using only synthetic noisy data.Assumption 1 requires the clean distribution to be expressible as a weighting of the synthetic distribution.Source: §4 · pp. 5–6
  • With finite samples, an approximate bilevel solution has test performance close to the oracle model.The result is stated for a solution satisfying the theorem’s condition.Source: §4 · p. 6

5 EXPERIMENTS

Experiments evaluate SunGen on eight text-classification tasks against zero-shot and denoising baselines, then analyze its data selection. Results report gains over ZeroGen and examine data quality, subset size, and learned weights.

5.1 SETUP

  • On eight text-classification tasks, experiments compare SunGen with prompting and ZeroGen using GPT2-XL generation and lightweight Bi-LSTM or DistilBERT-base classifiers.The bilevel procedure is iterated 50 times per task.Source: §5.1 · pp. 6–7

5.2 EXPERIMENTAL RESULTS

  • SunGen-LSTM improves relative to ZeroGen by 9.8% on average across all tasks, with gains over ZeroGen across the tasks.The comparison uses a 200k synthetic dataset for both methods.Source: §5.2 · p. 7, Table 1 · p. 7
  • Training on SunGen-selected data avoids ZeroGen’s observed overfitting, and test performance continues to improve with longer training.The paper reports that SunGen significantly surpasses ZeroGen in this setting.Source: §5.2 · p. 7
  • SunGen achieves similar performance to bilevel reweighting that uses gold data for its outer objective, while optimizing weights with a robust objective on noisy synthetic data.This comparison supports using synthetic data rather than a clean validation set to supervise weight optimization.Source: §5.2 · p. 7
  • SunGen outperforms the compared denoising baselines in the zero-shot setting.The comparison includes confidence-based filtering, Co-Teaching, and Meta-Weight-Net.Source: §5.2 · pp. 7–8, Table 3 · p. 7

5.3 ABLATION STUDY AND ANALYSIS

  • A 2% IMDb subset selected by SunGen performs better than the original 1,000k dataset; unweighted training on the full set drops IMDb accuracy from 86.56 to 78.29.The passage attributes the decline to erroneous data despite its small proportion.Source: §5.3 · p. 8, Table 4 · p. 8
  • SunGen-selected data is more diverse than ZeroGen data, although average correctness is slightly lower on Amazon and Yelp and the highest-weight samples are slightly less correct than average.The authors suggest these samples may be harder but more informative than redundant or simple examples.Source: §5.3 · p. 8, Table 5 · p. 8
  • IMDb samples given low weights are mostly wrongly labeled, while some are task-irrelevant; high-weight examples include complex transitional sentences.The histogram shows weights gradually separating during optimization.Source: §5.3 · p. 8, Figure 3 · p. 8, Table 6 · p. 9

6 RELATED WORK

Related work covers PLM-based zero-shot learning through prompting or generated training data, alongside methods for handling noisy labels and samples. The paper positions its approach as combining noise-handling strategies without a clean validation set.

Zero-shot Learning via PLM

  • Prompt-based zero-shot methods use task-dependent prompts, while generation-based approaches train a task model on PLM-synthesized data, including tiny models trained from scratch.Some prior generation-based methods instead use task-related human data to instruct or fine-tune the generative PLM.Source: §6 · p. 9

Noise Robust Learning

  • Noise-handling methods include heuristic resampling, reweighting, and label correction, as well as training with noise-robust losses.The passage notes thresholds or clean validation sets for heuristic approaches and tuning difficulty for robust-loss methods; SunGen is presented as an end-to-end alternative without a clean validation set.Source: §6 · p. 9

7 CONCLUSION

The paper concludes that SUNGEN constructs clean synthetic datasets without human annotation or intervention and can be applied to other data-generation pipelines. Appendix experiments and proofs examine robust-loss optimization, the relationship between CE and RCE optima, and noise tolerance.

7 CONCLUSION

  • SUNGEN aims to construct clean synthetic datasets without human annotation or intervention and can be applied to other data-generation pipelines.The paper frames this as a way to select high-quality data automatically.Source: §7 · p. 9

A.1.1 OPTIMIZATION DIFFICULTY OF NOISE-ROBUST LOSS.

  • Robust-loss optimization is difficult because robust losses lack CE’s implicit weighting of harder samples, and their wide flat regions can produce vanishing gradients away from an optimum.The appendix identifies both the absent implicit weighting and flat loss-surface regions as contributors to optimization difficulty.Source: §A.1.1 · pp. 14–15

A.1 EMPIRICAL VERIFICATION FOR ASSUMPTION 2 AND 3

  • Training with RCE struggles to reduce either RCE or CE loss, while CE training reduces both; the experiments attribute the degradation to optimization difficulty rather than a lack of good solutions.The appendix also reports that CE and RCE reach a plateau at the same time when training with CE.Source: §A.1 · p. 14, §A.1.1 · p. 15

A.1.2 THE OPTIMAL SOLUTIONS OF CE AND RCE ARE CLOSE.

  • The CE optimum is also near an RCE optimum, as indicated by the loss surfaces plotted around the CE solution.The visualization varies model parameters along two random directions; its contour compares CE and RCE losses.Source: §A.1.2 · p. 15, Figure 4 · p. 15

A.2 PROOF FOR PROPERTY 1

  • The appendix revisits robust-loss noise tolerance for uniform, simple non-uniform, and class-dependent label noise.It presents these cases for completeness, building on prior proofs of the robust-loss property.Source: §A.2 · p. 16

A.3 PROOF FOR THEOREM 1

  • Theorem 1’s proof establishes existence and uniqueness of the clean-data optimum in the framework and connects the optimal weighting function to that model solution.The proof describes this mapping as being achieved by the inner loop, then uses it to motivate the outer-loop objective.Source: §A.3 · pp. 17–18

A.4 PROOF FOR THEOREM 2

This block covers the proof discussion for Theorem 2, a relaxed-assumption extension of Theorem 1, and an ablation of outer objectives.

A.4 PROOF FOR THEOREM 2

  • The Theorem 2 proof explanation derives its bound by applying a uniform inequality over the weight-function class, using an ϵ-approximate solution, and taking an infimum over weights.The passage identifies these as the successive justifications for the displayed inequalities.Source: §A.4 · p. 18

B RELAXING ASSUMPTION 2

  • Under Assumption 1, Theorem 3 states that some optimal weighting function recovers θ*, and gives a closeness result under relaxed optimality and additional convexity and differentiability assumptions.The relaxed condition is that θ* achieves ϵ-optimal robust loss on clean data.Source: B RELAXING ASSUMPTION 2 · p. 19

C ABLATION STUDY OF OUTER OBJECTIVES

  • SUNGEN with RCE and MAE as outer objectives surpasses baseline methods, while using CE yields marginal improvement or worse performance than ZEROGEN.The authors attribute CE’s weaker result to its synthetic-data outer loss providing inaccurate guidance for updating sample weights.Source: C ABLATION STUDY OF OUTER OBJECTIVES · p. 19

D SUNGEN VS. OTHER NOISE-ROBUST TRAINING STRATEGIES

The appendix reports comparisons with other noise-robust strategies and baseline methods, then describes the evaluated task set and its evaluation scope.

D SUNGEN VS. OTHER NOISE-ROBUST TRAINING STRATEGIES

E EXPERIMENTS ON OTHER CLASSIFICATION TASKS

  • SunGen consistently surpasses baseline methods on the GLUE NLI tasks RTE and QNLI and paraphrase task MRPC; more challenging tasks requiring expert knowledge or reasoning remain future work.The experiments use DistilBERT as the task model on 20k synthetic examples.Source: Table 9 · p. 20, E EXPERIMENTS ON OTHER CLASSIFICATION… · p. 20

F.1 DATASET

  • Evaluation spans eight text-classification tasks covering sentiment, reviews, subjectivity, news topics, and ontology classification, with accuracy reported on the test sets.DBpedia uses a random 5,000-example test sample; the other tasks use their full test sets.Source: §F.1 · p. 20

F.2 FULL IMPLEMENTATION DETAILS

The implementation uses GPT2-XL for generation and lightweight task models, with a bi-level optimization procedure; the appendix also specifies prompts and truncated meta-gradient backpropagation.

F.2 FULL IMPLEMENTATION DETAILS

  • The implementation uses GPT2-XL with nucleus sampling at p = 0.9, Adam for sample-weight optimization, and 1-layer Bi-LSTM or DistilBERT-base as task models.The bilevel procedure runs for 50 iterations per task; task-model accuracy is averaged over three random-seed runs.Source: §F.2 · pp. 20–21

F.3 PROMPTS

  • For IMDb, SST-2, and Rotten Tomatoes, the authors use Ye et al.’s best prompts; for other tasks, they manually design five prompts and report the best prompting result.The same prompt or a minor revision is used for ZEROGEN and SUNGEN data generation.Source: §F.3 · p. 21

G BI-LEVEL VS. ONE-LEVEL OPTIMIZATION

H TRUNCATED BACK-PROPAGATION FOR META GRADIENT.

  • For computational efficiency, the method uses one-step truncated backpropagation rather than unrolling all T optimization steps.The passage identifies this as the procedure for calculating the meta-gradient.Source: H TRUNCATED BACK-PROPAGATION FOR META… · p. 21

I COMPARISON WITH OTHER NOISE-ROBUST LEARNING METHODS

SunGen is presented as a noise-robust reweighting framework that learns sample weights without in-domain labels or extensive manual tuning. In IMDb examples, it removes mostly mislabeled data and selects correct, challenging sentences.

I COMPARISON WITH OTHER NOISE-ROBUST LEARNING METHODS

  • SunGen learns sample weights end-to-end without excessive manual tuning or task-specific knowledge, and does not require in-domain labeled data.Its inner and outer objectives use the same synthetic training set; Table 11 compares it with one-level ℓrce on 1,000k data using LSTM as TAM.Source: I COMPARISON WITH OTHER NOISE-ROBUST… · p. 22, Table 11 · p. 22
  • SunGen uses a noise-robust loss to learn sample weights, aiming to remove low-quality data without hurting model performance.This differs from using the loss directly to train the model.Source: I COMPARISON WITH OTHER NOISE-ROBUST… · p. 22

J MORE RELATED WORK

  • Coreset selection favors informative, often high-loss samples, which can overlap with harmful examples under noise; the SST-2 histogram also shows loss values do not separate clean from noisy data.The histogram uses 30% uniform label noise.Source: J MORE RELATED WORK · p. 23, Figure 6 · p. 23

K SELECTED AND REMOVED DATA

  • IMDb accuracy rose from 78.29 to 86.56 after removing erroneous data, which included mostly noisy labels and a small portion of task-unrelated text.SunGen-selected examples were well-written transitional complex sentences.Source: K SELECTED AND REMOVED DATA · p. 23

L WHY LOSS-VALUE-BASED DE-NOISE METHODS DOES NOT WORK IN OUR SITUATION?

Loss-value-based selection may fail here because correctly labeled and mislabeled examples can have clustered loss values, and noisy data need not always have higher loss.

L WHY LOSS-VALUE-BASED DE-NOISE METHODS DOES NOT WORK IN OUR SITUATION?

  • On SST-2 with 30% manually introduced label noise, clean and mislabeled examples had clustered loss values, so loss-based subset selection was not applicable in this setting.The authors conjecture that noisy examples do not have higher losses in every task.Source: L WHY LOSS-VALUE-BASED DE-NOISE… · p. 23
Loading 2205.12679v2…