PaperlayerView exact revision on arXiv

Source-linked AI summary

Exact Degeneracy Under Balanced k-Shot Sampling:Consequences for Small-Sample Discriminant Analysis on LLM Embeddings

Lingxiao Qu

arXiv:2609.09860v1cs.LG

TL;DR

Few-shot text classification with frozen LLM embeddings lies in the small-sample regime where KLPCDA was designed to help, but balanced k-shot sampling creates an exact degeneracy in several variants. The paper derives and repairs the repairable cases, evaluates them across datasets and embedding models, and finds that a properly cross-validated logistic probe remains stronger on most datasets while the decoder-model gap is largely recoverable with more support examples.

  • Problem

    The paper asks whether small-sample discriminant-analysis guidance from other feature spaces transfers to few-shot text classification with frozen LLM embeddings.

  • Method

    The paper derives the balanced-sampling degeneracy of seven KLPCDA variants, adds an in-formula tie-break where possible, and evaluates the framework across four datasets, three embedding sizes, and competing baselines.

  • Results

    A properly cross-validated linear probe beats every KLPCDA variant on three of four datasets, while three geometric metrics fail to explain the weaker decoder-based embedder.

  • Takeaways & Limitations

    Balanced k-shot sampling can make KLPCDA selection arbitrary or void, and the observed high-dimensional embedding disadvantage is substantially recoverable with more labeled examples.

  • Takeaways & Limitations

    In-context learning is evaluated only on TREC and AG News because exhaustive demonstrations are impractical for datasets with 77–150 classes.

Abstract

from arXiv · show

Balanced k-shot sampling draws exactly k labeled examples per class. We show that it induces an exact, provable degeneracy in a family of small-sample discriminant estimators. Under balanced sampling, the within-class scatter operator of Kernelized Linear Principal Component Discriminant Analysis (KLPCDA) is not merely rank-deficient but exactly a scaled orthogonal projector. We derive the consequences in closed form: two of KLPCDA's seven variants have every signal eigenvalue exactly equal, so their eigenvector selection criterion is provably indifferent rather than ill-conditioned, and a third has a provably void objective. This follows from the estimators' construction, not any dataset; we confirm it on frozen sentence embeddings and, separately, on residual-stream activations from a decoder-only generative model. An in-formula tie-break repairs the two repairable variants, with recovery gated by class count: the residual subspace constraint costs 5x more on few-class than many-class datasets (p=0.000001). We then evaluate the repaired framework on few-shot text classification on frozen LLM embeddings (n much smaller than d, up to 4096), across four datasets, three embedding sizes, and three trained baselines (SetFit, LoRA, in-context learning). A properly cross-validated logistic-regression probe still beats every KLPCDA variant on three of four datasets, at every embedding size; guidance carried from pixel, vibration-signal, and gene-expression data does not directly generalize to this feature space. Three independent geometric separability metrics fail to explain why one high-dimensional decoder-based embedding model underperforms smaller bidirectional encoders, ruling out anisotropy; the gap is substantially an estimation-efficiency effect, not a permanent ceiling, closing by more than 80% when the support set grows from k<=10 to k=30-50 (p=0.00195, both many-class datasets).

1 Introduction

This paper tests whether small-sample discriminant-analysis guidance transfers to few-shot classification with frozen LLM embeddings, while diagnosing estimator failures induced by balanced sampling.

  • Motivation: Frozen embeddings routinely satisfy n ≪ d, with dimensions from 384 to 4096 and realistic budgets of k ≤10.With n = kL, few-shot text classification falls in the rank-deficient regime targeted by KLPCDA.
  • Contributions: Balanced k-shot sampling creates an estimator-level degeneracy in three of seven KLPCDA variants, independent of any dataset or embedding space.The study derives this failure mode in closed form and verifies it on frozen sentence embeddings and decoder-model residual activations.
  • Contributions: A properly cross-validated linear probe still beats every KLPCDA variant on three of four datasets.The evaluation compares seven KLPCDA variants with linear and nearest-centroid probes plus SetFit, LoRA, and in-context learning.
  • Contributions: The paper provides quantified practitioner guidance on class count, layer or pooling choice, in-context-learning regimes, and recoverability with more labels.These guidelines are presented as mechanism-backed consequences of the evaluation.
  • Contributions: Three independent geometric separability metrics fail to explain why one embedding model underperforms others in this regime.The negative result rules out several natural explanations rather than leaving the comparison untested.

2 Related Work

Related work frames KLPCDA as a response to singular within-class scatter in small-sample discriminant analysis and situates the study among frozen embeddings, adaptation baselines, and anisotropy-based explanations.

  • Small-sample discriminant analysis: Small-sample discriminant analysis addresses n < d, where Fisher LDA’s within-class scatter is singular or ill-conditioned.Kernelized extensions combine total, between-class, and within-class scatter in multiple variants.
  • Frozen sentence embeddings: Frozen sentence embeddings span small bidirectional encoders, larger bidirectional encoders, and decoder-only language models repurposed as embedders.The paper evaluates this architectural range in few-shot classification.
  • Few-shot baselines: SetFit, LoRA, and in-context learning adapt an encoder or language model directly to the k-shot task.They represent contrastive fine-tuning, low-rank adaptation, and prompt-based prediction, respectively.
  • Embedding anisotropy: The paper tests anisotropy and related separability measures as predictors of downstream few-shot accuracy across different encoder architectures.It reports that anisotropy does not explain the observed performance differences.

3 Method

KLPCDA combines kernelized total, between-class, and within-class scatter operators, but balanced sampling makes the within-class operator an exactly scaled projector and degenerates several variants.

  • 3.2 An Exact Eigenvalue Degeneracy Under Balanced k-Shot Sampling: Under balanced k-shot sampling, W has eigenvalues 0 with multiplicity L and 1/L with multiplicity n − L.This exact projector structure was verified analytically and on Banking77 embeddings, where the eigenvalues were {0, 1/77}.
  • 3.2 An Exact Eigenvalue Degeneracy Under Balanced k-Shot Sampling: B lies entirely in the class-indicator subspace that W projects away, and is zero on W’s remaining range.The empirical residual for ΠBΠ was at most 2.8 × 10^-36.
  • 3.2 An Exact Eigenvalue Degeneracy Under Balanced k-Shot Sampling: No.5 has n − L identical signal eigenvalues, so its criterion cannot select a unique eigenbasis.On real data, the eigenvalues were 77.000000000 ± 1.2 × 10^-10; the returned basis can therefore depend on eigensolver arbitrariness.
  • 3.2 An Exact Eigenvalue Degeneracy Under Balanced k-Shot Sampling: No.3’s operator is exactly zero because its pseudo-inverse cannot recover between-class signal from W’s null space, making its objective void.Observed eigenvalues around 10^-12 to 10^-14 are floating-point noise around the exact zero.
  • 3.3 A Tie-Break Fix, and a Direction the Formula Does Not Predict: An in-formula secondary-term tie-break repairs No.1 and No.5, but the empirically effective No.5 ranking reverses the literal variance-maximization direction.Ascending variance performed better than descending variance across k ∈ {2, 3, 5, 10} on two datasets, while No.3 and No.7 remain unfixed.

4 Experimental Setup

The study evaluates frozen embeddings and KLPCDA against probe, centroid, adapted-model, and prompting baselines across four datasets, three embedding sizes, and balanced few-shot supports.

  • Datasets: The benchmarks range from 4 to 150 classes: Banking77, CLINC150, TREC, and AG News.This design varies class count by nearly two orders of magnitude.
  • Embeddings: The frozen embeddings are MiniLM at d = 384, BGE-large at d = 1024, and E5-Mistral-7B-Instruct at d = 4096.No embedding model is fine-tuned; only support-set labels are used downstream.
  • Sampling and evaluation: For k ∈ {2, 3, 5, 10}, the study draws 10 stratified supports per dataset and evaluates methods on the full official test split.Results are summarized by mean and standard deviation, with paired Wilcoxon tests for matched comparisons.
  • Methods compared: The comparison includes seven KLPCDA variants, cross-validated logistic regression, nearest centroid, SetFit, LoRA, and in-context learning.The logistic probe selects C using only the k-shot support set.

5 Results

Across four datasets and multiple embedding sizes, a properly cross-validated linear probe generally outperforms KLPCDA, while repairs, class count, model configuration, and adaptation regime shape the remaining differences.

  • 5.1 Headline: KLPCDA Does Not Yet Beat a Fair Linear Probe, With One Exception: A tuned linear probe beats every KLPCDA variant on Banking77, CLINC150, and AG News at every k ≥3; TREC is the exception, where No.6 leads at d=384.The TREC advantage is directionally consistent but not individually significant at any single k.
  • 5.2 Embedding Dimension: A Real Effect With Dataset-Dependent Generalization: At d=4096, every method loses absolute accuracy and the probe–KLPCDA gap widens on both many-class datasets, reversing the narrowing observed from d=384 to d=1024.The reversal is specific to the E5-Mistral-7B-Instruct model rather than embedding dimension alone.
  • 5.3 The Tie-Break Fix and Its Class-Count Dependence: The tie-break fix makes No.1 and No.5 competitive with No.2/No.4/No.6 on many-class datasets, but its structural constraint costs 0.0379 versus 0.0072 accuracy on few- versus many-class datasets, a 5× difference.The pooled difference is significant at p=0.000001 across 48 paired comparisons.
  • 5.4 Layer, Pooling, and the Margin Metric’s Within-Model Validity: Within MiniLM, accuracy improves monotonically with layer depth and mean pooling beats CLS and max pooling, while the within/across-class margin tracks accuracy across layers but not architectures.The margin therefore supports within-model configuration comparisons, not direct cross-architecture ranking.
  • 5.5 Trained Baselines: SetFit, LoRA, and In-Context Learning: SetFit beats all KLPCDA variants across datasets, while in-context learning leads SetFit and LoRA at k=2–3 on TREC and AG News before fine-tuned methods catch up by k=10.The ICL comparison covers only TREC and AG News.
  • 5.5 Trained Baselines: SetFit, LoRA, and In-Context Learning: KLPCDA and training-free baselines run in well under 0.1 seconds on TREC, whereas ICL takes 48.9 ± 40.0 seconds and SetFit and LoRA require several seconds of GPU fine-tuning.KLPCDA’s low-k competitiveness with SetFit and LoRA comes at roughly two to three orders of magnitude lower wall-clock cost.

6 Guidelines: Which Variant, When

The paper turns KLPCDA guidance into scoped practitioner rules for class count, within-encoder layer selection, low-shot baselines, and support-set expansion, while documenting important boundaries.

  • Guideline 1: Prefer No.2, No.4, or No.6 over No.1 or No.5 for tasks with single-digit to low-tens class counts; the choice matters less as classes reach dozens to hundreds.This was tested at L ∈ {4, 6, 77, 150}, with no dataset covering 15–30 classes or exceeding 150.
  • Guideline 4: More labeled examples shrink the BGE-large–E5-Mistral accuracy gap by more than 80% on both many-class datasets, with every paired comparison at p=0.00195.Banking77 reaches k=30 and CLINC150 k=50; the mechanism remains unresolved.
  • Guideline 2: A within-class-versus-across-class cosine margin predicts relative accuracy within an encoder but fails to rank different encoder architectures.The cross-architecture limitation was observed in comparisons involving MiniLM, BGE-large, and E5-Mistral.
  • Guideline 3: In-context learning is preferred at k=2–3, but at k=10 at least one of SetFit or LoRA catches up to or overtakes it on every tested dataset.The ICL comparison covers only TREC and AG News.
  • Scope boundaries: The study does not establish a general n/d rule, a resolution for E5-Mistral’s architectural underperformance, or a universal SetFit-versus-LoRA choice.These boundaries are explicitly retained as unresolved or rejected generalizations.

7 Discussion and Limitations

The discussion finds that geometric separability metrics do not explain E5-Mistral’s underperformance, while larger support sets substantially reduce the gap; several theoretical and experimental scope limits remain.

  • 7.1 A Negative Result We Consider Important: Three geometric metrics fail to explain E5-Mistral’s lower accuracy: anisotropy is inconsistent, within/across-class margins mis-rank architectures, and whitened spectra rank it most separable.The whitened-spectrum reversal persists across two sample-size regimes and broad ridge sweeps.
  • 7.1 A Negative Result We Consider Important: The tuned-probe gap to BGE-large shrinks from 0.19–0.22 at k=2 to 0.03–0.04 at the largest tested k, a reduction exceeding 80%.All paired comparisons across both datasets and both methods reach p=0.00195.
  • 7.2 An Unresolved Direction in the Tie-Break Fix: The tie-break direction that wins empirically contradicts No.5’s literal maximize formula, and its theoretical explanation remains unresolved.The paper presents the Fisher-LDA-flavored explanation as an open hypothesis rather than a derived mechanism.
  • 7.3 Scope Limitations: The study evaluates ICL only on TREC and AG News, leaving high-class-count datasets without a directly comparable ICL baseline.High-class-count ICL would require a different candidate-label design and was not attempted.
  • 7.3 Scope Limitations: No.3 and No.7 remain unreliable because they lack an effective in-formula secondary term for tie-breaking.For No.3, the additional between-class term is void in the relevant subspace; No.7 has no second term.
  • 7.3 Scope Limitations: Other scope limits include using only linear kernels, ten random draws per condition, one GPU machine, and several compute-constrained dataset and sampling choices.The authors report these constraints rather than presenting the study as exhaustive.

8 Conclusion

Guidance developed for other domains mostly does not transfer to few-shot text classification on frozen LLM embeddings: a properly tuned linear probe beats every tested KLPCDA variant on three of four datasets. The study nevertheless identifies an exact KLPCDA failure mode, repairs two affected variants, and finds that the remaining cross-architecture gap closes substantially with larger support sets.

  • Benchmark conclusion: A properly tuned linear probe beats every KLPCDA variant on three of four datasets, while transferred discriminant-analysis guidance mostly fails in frozen LLM-embedding text classification.The evaluation also compares four datasets, three embedding sizes, and trained or adapted baselines.
  • Methodological contribution: Balanced k-shot sampling causes an exact KLPCDA degeneracy affecting three of seven variants, with a closed-form fix for the two repairable cases.The failure mode is structural rather than dataset-specific and concerns arbitrary eigenvector selection or a void objective.
  • Scaling conclusion: The cross-architecture accuracy gap closes by more than 80% when the support set grows beyond k≤10, indicating substantial estimation-efficiency effects rather than a permanent ceiling.This conclusion is supported by the study’s larger-support-set analysis.
  • Resources and scope: The authors release the estimator, evaluation harness, and complete experimental log, including both confirmed findings and dead ends.They also report that KLPCDA variants underperform nearest-centroid scoring for the tested CLINC150 held-out out-of-distribution class.
Loading 2609.09860v1…