PaperlayerView exact revision on arXiv

Source-linked AI summary

Geometric Filtering of LLM-Generated Samples for Few-Shot Text Classification

Benjamín Schindler, Gonzalo A. Ruz

arXiv:2608.13866v1cs.LGcs.CL

TL;DR

Few-shot text classification suffers from scarce labeled data, while LLM-generated samples vary in geometric consistency. This paper filters candidates by embedding-space distance and weights them softly, achieving +2.61pp over SMOTE across 6,700+ configurations and generalizing to NER without modification.

  • Problem

    LLM augmentation lacks geometric awareness, so generated samples may be class-consistent, peripheral, or cross-class, complicating reliable few-shot training.

  • Method

    The framework filters LLM-generated samples by Euclidean distance to real class examples and converts filter scores into continuous training weights.

  • Results

    +2.61pp over SMOTE across 6,700+ configurations, with statistically significant gains, while NER improves +9.26pp without filter modification.

  • Takeaways & Limitations

    The simplest distance-based filter consistently outperforms complex multi-criteria alternatives across the evaluated settings.

  • Takeaways & Limitations

    The evaluation is English-only, so multilingual transfer remains unknown.

Abstract

from arXiv · show

Large language models (LLMs) can generate synthetic training data for text classification, but the quality of generated samples is heterogeneous: some fall in correct class regions of the embedding space while others land in peripheral or cross-class zones. We propose a geometric filtering framework that evaluates each LLM-generated sample by its Euclidean distance to real class examples in a sentence embedding space, selecting only geometrically consistent candidates. A soft weighting mechanism transforms filter scores into sample weights for classifier training. Evaluated across 13 datasets, 5 classifiers, 10 augmentation methods, and over 6,700 configurations, our method achieves +2.61 percentage points (pp) over SMOTE ($p<0.0001$, Cohen's $d=0.95$, 88.9% win rate). The approach generalizes to named entity recognition (+9.26pp, 100% win rate) without filter modification, and is robust across 5 LLMs from 4 providers. A key finding is that the simplest distance-based filter consistently outperforms complex multi-criteria alternatives.

I. INTRODUCTION

The introduction frames few-shot text classification as a data-scarcity problem caused by limited labeled examples and identifies a gap between linguistic validity and geometric consistency in augmentation. It proposes post-generation geometric filtering with Euclidean distances and soft sample weighting to retain and prioritize LLM-generated samples consistent with real data.

  • Problem: Few-shot classification often has only 10 to 50 labeled examples per class, making robust decision boundaries difficult, especially in multi-class settings.The introduction highlights hate speech detection, news classification, and spam filtering as practical examples.
  • Augmentation gap: Classical oversampling provides geometric guidance but not linguistically coherent text, whereas LLM generation produces plausible text without geometric awareness.The introduction contrasts SMOTE and ADASYN with LLM-based generation.
  • Motivation: LLM-generated samples vary in embedding-space placement, with some matching target-class regions and others lying in peripheral or cross-class regions.Indiscriminate inclusion of such heterogeneous samples can degrade training quality.
  • Approach: The proposed post-generation system uses Euclidean distance in embedding space to select LLM-generated samples consistent with the real data distribution.The framework combines linguistically valid generation with geometric filtering and is decoupled from the generative model.
  • Approach: A soft weighting mechanism converts geometric filter scores into continuous classifier-training weights rather than relying only on binary accept/reject decisions.The contribution extends filtering from selection to graded sample influence during training.

II. RELATED WORK … III. METHODOLOGY

Prior work spans classical oversampling, LLM-based augmentation, and hybrid methods, but existing approaches lack quantitative post-generation validation in embedding space. The methodology addresses this identified gap.

  • A. Classical Oversampling: SMOTE interpolates between nearest neighbors, ADASYN adapts sampling density, and EDA uses surface-level text perturbations.These methods include synonym substitution, random insertion, and random deletion.
  • A. Classical Oversampling: Classical methods can lack linguistic coherence in embedding space or produce only superficial variations.The passage characterizes these as limitations despite effectiveness in certain settings.
  • B. LLM-Based Augmentation: +12.7% accuracy and +13pp macro F1 were reported for GPT-3-generated reviews in sentiment analysis.These results come from prior work on synthetic review generation.
  • B. LLM-Based Augmentation: F1 reached 0.76 versus 0.60 for SMOTE when fine-tuned LLMs were used for multiclass imbalanced text.The comparison reports fine-tuned LLMs outperforming SMOTE.
  • B. LLM-Based Augmentation: LLM-based augmentation includes AugGPT across multiple tasks, while reported benefits diminish above 20 examples per class.Evaluation protocols also significantly affect conclusions.
  • C. Hybrid and Quality-Aware Approaches: Hybrid methods include LLMOverTab, SMOTExT, SentiGEN, and AugmenToxic, combining generation with interpolation, validation, or other augmentation strategies.The examples use SMOTE, LLM decoding, T5, genetic algorithms, XLNet validation, or LLM augmentation for toxicity detection.
  • III. METHODOLOGY: Existing hybrid approaches do not implement quantitative post-generation validation in embedding space.The paper identifies this absence as the gap addressed by its methodology.

A. Problem Setup · B. LLM Generation · C. Geometric Filters

The study evaluates few-shot text classification and NER across diverse datasets using normalized sentence embeddings and LLM-generated candidates. It compares increasingly complex geometric filters, finding that Euclidean-distance filtering is the strongest simple alternative because cosine similarity is redundant and LOF can degenerate in low-shot settings.

  • A. Problem Setup: 13 datasets span nine textual domains, including seven classification datasets, three scalability datasets, and three NER corpora.Few-shot scenarios use 10, 25, or 50 examples per class while retaining full test sets.
  • A. Problem Setup: All texts use L2-normalized 768-dimensional all-mpnet-base-v2 embeddings, with 90.5% cross-model agreement in an ablation of three additional embedding models.
  • B. LLM Generation: Gemini 3 Flash is the primary generator, with robustness tested across five models from four providers.GPT-5-mini, Claude 4.5 Haiku, Kimi K2.5, and GLM-5 provide the additional models.
  • B. LLM Generation: For each class, real examples serve as in-context demonstrations and three surplus candidates are generated per desired sample without fine-tuning.Responses are cached using MD5 hashing for reproducibility.
  • C. Geometric Filters: Five filtering strategies range from random selection without filtering to LOF, cascade, and combined filters of increasing complexity.The combined filter requires simultaneous LOF and cosine-similarity threshold passage and is included as a case study of over-filtering.
  • C. Geometric Filters: Level-1 cascade filtering, using Euclidean distance only, outperforms all multi-criteria alternatives.The full cascade can additionally use cosine similarity, KNN purity, and centroid distance, combining scores by geometric mean.
  • C. Geometric Filters: For L2-normalized vectors, Euclidean distance is strictly monotonic with cosine similarity, while LOF requires 20 neighbors despite only nine being available at 10-shot.Thus cosine similarity adds no discriminative information, and LOF produces degenerate density estimates in that setting.

D. Soft Weighting

Geometric filter scores are converted into continuous sample weights rather than binary accept/reject decisions, with temperature scaling emphasizing the highest-scoring candidates. Real samples receive weight 1.0, while weighted synthetic samples contribute proportionally to classifier training.

  • D. Soft Weighting: The weighting procedure computes cascade level-1 scores, selects the top-N candidates, min-max normalizes scores to [0, 1], and applies temperature scaling.These four steps transform geometric filter scores into continuous sample weights.
  • D. Soft Weighting: T = 0.5 sharpens the weight distribution toward the highest-scoring candidates through an exponent of 1/T = 2.With wmin = 0, the max operator remains only as a safeguard against zero-weight samples.
  • D. Soft Weighting: Real samples always receive weight 1.0, while synthetic weights are passed as sample_weight so geometrically consistent samples contribute more to training.The weighting mechanism replaces binary filtering with graded contributions during classifier training.

E. Baselines and Classifiers … A. Main Comparison

Across 1,890 main-comparison configurations, soft geometric weighting improves macro F1 over SMOTE, while binary filtering also significantly improves performance and modern augmentation baselines do not surpass SMOTE.

  • E. Baselines and Classifiers: Eight augmentation baselines, plus binary filtering and soft weighting, form the 10 compared methods, with SMOTE as the reference.The baselines include random oversampling, EDA, back-translation, no augmentation, embedding mixup, T5 paraphrasing, and BERT contextual augmentation.
  • E. Baselines and Classifiers: Five classifiers are evaluated: logistic regression, linear SVC, Ridge, random forest, and MLP.Random forest uses 100 trees, and MLP uses 100 hidden units.
  • F. Statistical Protocol: 1,890 runs compare 10 methods using three linear classifiers and three seeds, while a second 3,675-run protocol covers five classifiers, five seeds, and seven methods.The primary metric is macro F1 on the full test set, with paired t-tests and Bonferroni correction used for significance assessment.
  • A. Main Comparison: 1,890 configurations comprise 21 dataset configurations × 3 classifiers × 3 seeds × 10 methods in the main comparison.Table I compares the 10 augmentation methods against SMOTE.
  • A. Main Comparison: +0.22pp separates soft weighting from binary filtering, while binary filtering achieves +1.75pp over no augmentation.The soft-weighting versus binary-filtering difference has p = 0.053; binary filtering versus no augmentation has p < 0.0001 and d = 0.63.
  • A. Main Comparison: +2.61pp over SMOTE is achieved by soft weighting, with d = 0.95 and 88.9% win rate; binary filtering reaches +2.39pp and d = 0.82.All modern baselines, T5 paraphrasing, BERT contextual augmentation, and embedding mixup fail to outperform SMOTE.

B. Effect of Training Set Size · C. Classifier Analysis

Geometric filtering is most valuable when training data are scarce, where it produces large gains and clusters selected samples around real class examples. Benefits vary by classifier, with linear models gaining most and random forests least.

  • B. Effect of Training Set Size: d = 1.48 at 10-shot, with absolute macro F1 improving from 67.23% (SMOTE) to 72.12%.The effect is described as very large when each class has only 10 examples.
  • B. Effect of Training Set Size: 95.2% win rate at 25-shot indicates near-universal improvement from filtered synthetic samples.The reported win rate is for the 25-shot setting.
  • B. Effect of Training Set Size: At 50-shot, benefits diminish because classifiers already have sufficient data for reasonable decision boundaries.The diminishing-return pattern is attributed to increasing adequacy of the real training data.
  • B. Effect of Training Set Size: Filtered samples cluster tightly around real examples of each class, unlike many unfiltered candidates in cross-class or peripheral regions.The comparison comes from a t-SNE projection of a 6-class emotion dataset in the 10-shot setting.
  • C. Classifier Analysis: +2.98pp is the Linear SVC gain from geometric filtering, with d = 1.02 and over 92% win rates.Linear SVC is among the classifiers benefiting most from geometric filtering.
  • C. Classifier Analysis: +2.74pp is the Ridge gain, with d = 1.04 and over 92% win rates.Ridge also shows a large effect size under geometric filtering.
  • C. Classifier Analysis: The MLP shows a medium-to-large effect (d = 0.74), while random forest shows the weakest effect (d = 0.35).Random forest remains significant but exhibits only a small effect compared with the other four classifiers.

D. Cross-Task Generalization: NER

The geometric filter transfers to named entity recognition without modification beyond redefining the class as the dominant entity type per sentence. Cascade level-1 delivers a +9.26pp gain over the unaugmented baseline, while the results support task-agnostic geometric filtering.

  • D. Cross-Task Generalization: NER: No filter modification is required for NER; only “class” changes to the dominant entity type per sentence.This adaptation extends the same geometric filtering framework across tasks.
  • D. Cross-Task Generalization: NER: +9.26pp over the unaugmented baseline is achieved by cascade level-1, with pBonf = 0.002, d = 2.05, and a 100% win rate across 9 configurations.It records the best result on 5 of the 9 configurations.
  • D. Cross-Task Generalization: NER: +9.28pp is achieved by Relaxed LOF, which is statistically indistinguishable from cascade level-1 and ties the two simplest filters at the top.The passage reports d = 1.91 for Relaxed LOF.
  • D. Cross-Task Generalization: NER: +8.40pp comes from unfiltered generation, while geometric filtering adds a further +0.86pp.This indicates that most of the NER gain is attributable to the LLM-generated data itself.
  • D. Cross-Task Generalization: NER: Distance-based filtering captures task-agnostic geometric properties of the embedding space, while one filter fails to reach significance with p = 0.29, d = 0.38.The passage characterizes this result as reinforcing the over-filtering conclusion.
  • D. Cross-Task Generalization: NER: The NER results mirror the diminishing-returns pattern observed for text classification.This pattern is presented in the discussion of Fig. 2.

E. Why Simple Filters Outperform Complex Ones · V. DISCUSSION · A. LLM Robustness

Simple geometric filters outperform complex combinations because redundant similarity criteria, restrictive intersections, and unstable density estimates can discard useful diversity or degrade in few-shot settings. The framework’s gains remain statistically significant across five generative models, with no significant LLM effect.

  • E. Why Simple Filters Outperform Complex Ones: Euclidean distance and cosine similarity add no independent signal for L2-normalized vectors, while cosine becomes quasiuniform in 768 dimensions.Euclidean distance is a monotonic transformation of cosine similarity, so adding cosine provides no new discriminative information.
  • E. Why Simple Filters Outperform Complex Ones: Restrictive criterion intersections eliminate diverse candidates, retaining quasi-duplicates instead of covering underrepresented regions.The passage identifies diversity coverage as the augmentation value that combined filtering destroys.
  • E. Why Simple Filters Outperform Complex Ones: p = 0.0003 (raw), pBonf = 0.002, d = 2.05, and 100% win rate versus baseline for the level-1 cascade.These results are reported after evaluating 7 filters.
  • V. DISCUSSION: With only 9 real examples per class at 10-shot, LOF density estimates degenerate, whereas centroid distance remains stable with N = 10 points.The passage links this stability to evidence that representation quality matters more than selection-algorithm complexity.
  • A. LLM Robustness: Evaluation across 5 generative models shows that all achieve statistically significant improvements after Bonferroni correction for 5 comparisons.The models are Gemini 3 Flash, GPT-5-mini, Claude 4.5 Haiku, Kimi K2.5, and GLM-5.
  • A. LLM Robustness: +2.31pp for Gemini versus +1.49pp for Kimi yields a 0.82pp spread between the best and worst models.The reported spread indicates relatively consistent improvements across the evaluated generators.
  • A. LLM Robustness: χ2 = 8.15, p = 0.086 indicates no significant LLM effect, while W = 0.59 indicates substantial inter-model agreement.The passage concludes that geometric filtering is decoupled from the generative model.

B. Per-Class Analysis … E. Geometric Space Analysis

Geometric filtering benefits difficult classes most, performs best at moderate class counts, and reaches its strongest curriculum threshold when roughly half of candidates are included. Embedding-space analysis confirms that filtering compacts within-class structure while improving class separation.

  • B. Per-Class Analysis: +10.44pp average gains occur for classes with baseline F1 below 30%, versus +0.55pp for classes above 80%.In emotion at 10-shot, anger improves by +16.9pp, while sadness gains +1.0pp.
  • B. Per-Class Analysis: +16.9pp improves anger in emotion at 10-shot, from 27.8% to 44.6%, while sadness gains only +1.0pp.The benefit concentration is strongest where classifiers have the largest difficulty.
  • C. Effect of Number of Classes: +5.42pp and a 100% win rate occur at 6 classes, while gains turn negative at 77 classes and neutral at 150 classes.The class-count correlation is ρ = −0.260 (p = 0.003).
  • D. Curriculum Learning: +2.74pp versus SMOTE and an 86.2% win rate result when approximately 50% of candidates are included.This surpasses including all candidates (+2.22pp) by +0.52pp, indicating that roughly half introduce noise.
  • D. Curriculum Learning: +0.52pp separates the approximately 50% inclusion threshold from including all candidates, which achieves +2.22pp versus SMOTE.The curriculum experiment provides practical guidance to generate a 3× surplus and r
  • E. Geometric Space Analysis: 0.822 versus 0.850 unfiltered intra-class distance and 0.616 versus 0.571 inter-class distance show more compact and better-separated filtered clusters.The silhouette score rises from 0.075 to 0.085 (p = 0.029), with filtering winning in 71% of configurations.

F. Limitations · VI. CONCLUSION

The framework filters and softly weights geometrically consistent LLM-generated samples, improving few-shot classification across broad evaluations while generalizing to NER. Its benefits are limited by language, data-volume, and class-count conditions, motivating broader validation and extensions.

  • F. Limitations: The evaluation is English-only, so multilingual transfer remains unknown.
  • F. Limitations: +0.76pp at 50-shot versus +4.89pp at 10-shot, with the 50-shot difference nonsignificant under the 5-classifier protocol (p = 0.16).
  • F. Limitations: The benefit turns negative at 77 classes (−0.99pp) and neutral at 150 classes (+0.01pp).
  • VI. CONCLUSION: The framework generates surplus LLM candidates, embeds them, retains geometrically consistent samples, and softly weights training contributions by filter score.
  • VI. CONCLUSION: +2.61pp over SMOTE across 6,700+ configurations, with p < 0.0001, Cohen’s d = 0.95, and an 88.9% win rate.The evaluation covered 13 datasets, 5 classifiers, and 10 augmentation methods.
  • VI. CONCLUSION: +9.26pp on NER with a 100% win rate, without filter modification, while remaining robust across 5 LLMs from 4 providers.The largest benefits reached +10.44pp for classes with baseline F1 < 30%.
  • VI. CONCLUSION: The simplest distance-based filter consistently outperforms complex multi-criteria alternatives.The paper attributes this to metric correlation, diversity loss from restrictive intersections, and density-estimator instability in few-shot settings.
  • VI. CONCLUSION: Future work targets multilingual evaluation, locally deployed open-source LLMs, joint Bayesian hyperparameter optimization, question answering, and abstractive summarization.
Loading 2608.13866v1…