PaperlayerView exact revision on arXiv

Source-linked AI summary

LE-GDA: Layer Ensemble Gaussian Discriminant Analysis for Training-Free Few-Shot CLIP Adaptation

Pavodi Maniamfu, Vogel Kiketa, Zezina Kuzulu Kuetu, Jakin Mukendi Kabongo, David Mayoya Kutangila, Pescie Honore Kinkani Mampuya, L'Or Wolo Sieka

arXiv:2610.34901v1cs.CVcs.AIcs.LG

TL;DR

Training-free few-shot CLIP adaptation typically uses only the final projected image representation, while combining intermediate representations is complicated by differences in score scale. LE-GDA fits and standardizes discriminants across representations, then selects and fuses them with the zero-shot classifier; it improves on matched GDA-CLIP across evaluated backbones and shot counts.

  • Problem

    Training-free few-shot CLIP adaptation typically constructs classifiers from the final projected image representation, leaving intermediate representations unused and making their differently scaled scores difficult to combine fairly.

    Source: §1 · p. 2

  • Method

    LE-GDA fits regularized shared-covariance discriminants to intermediate and final frozen CLIP representations, standardizes their scores per query, and validation-selects a small ensemble for zero-shot fusion without gradient-based optimization.

    Source: §1 · p. 2

  • Results

    79.54% versus 78.01% mean accuracy at 16 shots, and LE-GDA improves on matched GDA-CLIP for every evaluated backbone and shot count.

    Source: §5.1 · p. 6, §7 · p. 9

  • Takeaways & Limitations

    Intermediate CLIP representations contain useful information beyond the final embedding for training-free few-shot adaptation.

    Source: §7 · p. 9

    The primary results characterize validation-assisted adaptation; transferring selected quantities leave-one-dataset-out does not significantly improve the validation-free baseline.

    Source: §6 · p. 8

Abstract

from arXiv · show

Training-free few-shot adaptation of vision-language models typically constructs a classifier from CLIP's final projected image representation, leaving intermediate representations unused. Combining them is not straightforward. Class scores from different representations differ in spread by orders of magnitude for the same query. Direct averaging allocates influence by numerical scale rather than predictive information. We propose Layer-Ensemble Gaussian Discriminant Analysis (LE-GDA), a method that fits regularized shared-covariance discriminants independently to intermediate and final representations extracted in one frozen encoder pass. LE-GDA standardizes each discriminant's class scores per query, selects a small representation multiset on labeled validation data, and combines it with the zero-shot classifier without gradient-based optimization. Across 11 datasets, four CLIP backbones, five shot counts, and three independently sampled K-shot training sets, LE-GDA achieves the highest mean accuracy of all evaluated methods on every backbone, raising the 16-shot mean from 78.01% to 79.54% over a matched GDA-CLIP (final-feature) baseline and improving that baseline for every backbone-shot combination. At one shot, LE-GDA retains a 1.20-point mean gain over the baseline even though covariance estimation is unavailable and per-query standardization is neutral. A control with richer per-class structure inside a single representation does not reproduce the gain, indicating that the benefit comes from combining representations. These results show that intermediate CLIP representations carry information beyond the final embedding and can improve training-free few-shot adaptation. Code and evaluation resources are available at https://github.com/pavodi-NM/le-gda.

1. Introduction

Training-free few-shot adaptation builds classifiers from frozen CLIP features, but prior approaches typically use only the final projected embedding. LE-GDA combines discriminants from intermediate and final representations, standardizing their scores before validation-based selection and fusion.

1. Introduction

  • The evaluation uses validation-assisted adaptation, in which a held-out labeled split selects fusion and regularization hyperparameters.Interpreting gains requires every method to use the same validation budget.Source: §1 · p. 1
  • Final-feature methods leave intermediate CLIP representations unused, although the final projected embedding is often the strongest individual representation.GDA-CLIP and cache- or adapter-based methods generally operate on the final projected image embedding.Source: §1 · p. 2
  • Direct score averaging can let numerical scale, rather than predictive information, determine each representation’s influence.Class-score spread varies by more than three orders of magnitude across intermediate representations of a single backbone.Source: §1 · p. 2
  • LE-GDA fits shared-covariance discriminants independently to intermediate and final representations, then standardizes scores and selects a small ensemble on validation data for zero-shot fusion.The representations come from one frozen encoder pass, and the method uses no back-propagation.Source: §1 · p. 2
  • The study evaluates LE-GDA against matched GDA-CLIP across 11 datasets, four backbones, and five shot counts, with controls to isolate the source of gains.The supplied contribution passage also notes that at one shot, neither covariance estimation nor standardized fusion applies.Source: §1 · p. 2

2. Related work

Prior few-shot CLIP adaptation includes gradient-trained task-specific methods and training-free methods that construct predictors from frozen features. The paper studies whether intermediate representations can contribute to training-free adaptation, which prior approaches generally based on the final embedding.

2. Related work

  • Prompt learning, feature adapters, residual classifiers, and learned prototypes obtain task-specific parameters through gradient-based training.Examples include CoOp, CLIP-Adapter, TaskRes, and ClusterAdapter.Source: §2 · p. 2
  • Training-free approaches use frozen features, but cache, prototype, and Gaussian-process methods generally operate on the final CLIP image embedding.APE selects informative channels within that final representation.Source: §2 · p. 2
  • To the authors’ knowledge, training-free CLIP few-shot adaptation had operated exclusively on the final projected embedding.Prior uses of intermediate CLIP layers for few-shot classification employ trained side adapters and learned multiscale fusion.Source: §2 · pp. 2–3

3. Layer-ensemble Gaussian discriminant analysis

LE-GDA extracts multiple frozen CLIP representations, fits a regularized shared-covariance discriminant in each, standardizes their class scores per query, and uses validation data to select and fuse a small representation ensemble.

3.1. Problem formulation and feature spaces

  • A single frozen encoder pass supplies normalized intermediate and final candidate features, with candidate layers drawn from ViT blocks or ResNet stages.The ViT candidates include selected class-token outputs and the feature before projection; ResNet candidates use pooled residual-stage features.Source: §3.1 · p. 3

3.2. Gaussian discriminant in each feature space

  • Each candidate feature space receives its own shared-covariance Gaussian discriminant, using class means and a regularized covariance estimated from the few-shot training set.For one shot, within-class residuals cannot estimate covariance, so the method sets covariance to the identity and reduces the classifier to nearest-class-mean.Source: §3.2 · p. 3

3.3. Per-query score standardization

  • Per-query standardization makes discriminant scores from different feature spaces commensurate while preserving class ordering within each space.The transformation is invariant to additive shifts and, above its standard-deviation floor, positive rescaling.Source: §3.3 · pp. 3–4

3.4. Feature-space selection and fusion

  • Greedy forward selection with replacement chooses up to four feature spaces using validation accuracy, then fuses their standardized scores with zero-shot logits using a validation-selected coefficient.Repeating a feature space provides discrete weighting; ridge strength is selected independently for each space, and no parameters are updated by gradient descent.Source: §3.2 · p. 3, §3.4 · p. 4

4. Experimental setup

The evaluation compares matched methods on 11 few-shot CLIP datasets, four backbones, and five shot counts using shared training and evaluation conditions. LE-GDA extracts candidate representations in one frozen encoder pass, fits discriminants per representation, and selects the ensemble using validation data.

4. Experimental setup

  • The study uses 11 datasets and evaluates RN50, RN101, ViT-B/32, and ViT-B/16 at K ∈ {1, 2, 4, 8, 16}, averaging three sampled training sets per shot count.Ten datasets use standard CoOp test splits; ImageNet uses a separately partitioned evaluation set.Source: §4 · p. 4
  • ImageNet results use a class-balanced 40,000-image evaluation subset, so they are not directly comparable with published results on all 50,000 validation images.The official validation set is split into 10 model-selection and 40 evaluation images per class.Source: §4 · p. 4
  • All methods requiring model selection use the same labeled validation images, disjoint from few-shot training and test data.The validation partition remains unchanged across shot counts.Source: §4 · p. 5

(b) LE-GDA

  • LE-GDA extracts candidate features in one frozen encoder pass, fits a shared-covariance discriminant per space, and validation-selects ridge coefficients and a representation multiset.Greedy selection uses replacement, allows up to four selections, and retains the best validation prefix.Source: (b) LE-GDA · p. 5
  • The matched GDA-CLIP baseline and other baselines use the same sampled training sets, validation splits, cached features, text classifier, and evaluation splits as LE-GDA.The baseline selects trace-scaled ridge and fusion coefficients on the same validation grids.Source: (b) LE-GDA · p. 5

5. Results

Across the evaluated datasets and backbones, LE-GDA improves on matched GDA-CLIP (final-feature), while the results also examine score standardization, representation diversity, and transfer of selected configurations. Its one-representation and richer-structure controls do not reproduce the multi-representation gain.

5.1. Comparison at 16 shots

  • 79.54% versus 78.01% mean accuracy: LE-GDA exceeds matched GDA-CLIP (final-feature) at 16 shots and improves it in 36 of 44 dataset–backbone settings.LE-GDA also exceeds the strongest trained cache baseline, 77.94%, without gradient-based optimization.Source: §5.1 · p. 6

5.2. Performance across shot counts

  • 1.20, 1.03, 1.00, 1.26, and 1.52 points: mean gains over GDA-CLIP (final-feature) at K = 1, 2, 4, 8, and 16; every backbone–shot combination is positive.At one shot, covariance estimation is unavailable and standardization is neutral, yet the gain remains 1.20 points.Source: §5.2 · p. 6

5.3. Score standardization and representation selection

  • 0.373 median accuracy points: per-query standardization improves uniform averaging at K = 16, with 98 positive, 31 negative, and three tied settings.Validation selection after standardization tracks standalone accuracy across all four backbones, unlike uniform averaging on the ViT backbones.Source: §5.3 · p. 6

5.4. A larger search over one representation

  • 0 versus 0.829 points: the matched search control over ridge variants of one representation has zero median gain, while LE-GDA improves the same baseline by 0.829 points.The control is not an exact nesting of GDA-CLIP because each member uses a prescribed ridge coefficient.Source: §5.4 · p. 7

5.5. Richer per-class structure within one representation

  • 0.10, 0.38, 1.10, and 2.10 points: mean accuracy falls relative to m = 1 as fixed mixture components per class increase to m = 2, 4, 8, and 16.Validation selection of m recovers only +0.020 points; the reported gain is attributed to combining representations rather than richer per-class structure within one.Source: §5.5 · p. 7

5.6. Transfer of the selected configuration

  • +0.611 ± 2.771 points: transferred configurations yield 28 wins, but two-sided tests provide no strong evidence of a difference; excluding EuroSAT, the lead is 0.09 points.At K = 16, full validation access improves LE-GDA by 2.270 ± 2.276 points; the fusion coefficient contributes 1.046 points of that difference.Source: §5.6 · pp. 7–8

6. Limitations

The reported evidence is bounded by validation-assisted settings, limited analyses, and requirements on available model features. Some comparisons and transfer results do not establish broader performance claims.

6. Limitations

  • LE-GDA underperforms both gradient-trained cache adapters on every backbone on ImageNet, and the cache comparisons do not establish best attainable performance for every cache formulation.Among training-free methods, it has the highest ImageNet accuracy averaged over backbones.Source: §6 · p. 8
  • Dispersion heterogeneity is associated with accuracy gains on ResNets but not ViTs, and the selection analysis covers only K = 16.On ViT-B/16, the 16-shot accuracy difference is 0.05 points despite a large change in selected representations.Source: §6 · p. 8
  • The primary results use additional labeled validation images, while leave-one-dataset-out transfer does not significantly improve the validation-free baseline.The validation images are outside the K-shot training set.Source: §6 · p. 8
  • LE-GDA requires intermediate activations and stores three to five times more classifier values than GDA-CLIP; distribution shift and base-to-new generalization remain future work.Its storage is 19–31% of a sparse 16-shot Tip-Adapter cache.Source: §6 · p. 8

7. Conclusion

LE-GDA combines discriminants fitted to intermediate and final CLIP features, with per-query score standardization and validation-selected fusion. It outperforms the matched final-feature baseline across evaluated backbones and shot counts, including one shot.

7. Conclusion

  • LE-GDA outperforms the matched GDA-CLIP (final-feature) baseline for every evaluated backbone and shot count.At one shot, neither covariance estimation nor standardized fusion explains the gain.Source: §7 · p. 9
  • LE-GDA fits regularized Gaussian discriminants to intermediate and final CLIP features, standardizes class scores per query, and fuses a validation-selected subset with the zero-shot classifier.Source: §7 · p. 9
  • Intermediate CLIP representations contain useful information left unused by final-feature methods, making layer choice, score normalization, and matched validation access consequential design decisions.Source: §7 · p. 9

Supplementary Material

The supplement provides complete K = 16 dataset–backbone comparisons and clarifies the scope of the matched baseline and evaluation results.

Supplementary Material

  • The matched GDA-CLIP baseline uses the same data splits, features, and model-selection protocol as LE-GDA; published GDA-CLIP refers to the original method or results.Source: A. Complete dataset–backbone results · p. 11
  • Table S1 compares LE-GDA only with its matched baseline at K = 16, with entries averaged over three independently sampled training sets.Tables S2–S12 provide the same settings for all eight methods.Source: A. Complete dataset–backbone results · p. 11
  • Across 44 dataset–backbone comparisons, LE-GDA improves on matched GDA-CLIP in 36 cases; the largest reported gains are 6.20 and 7.83 points with RN101.The largest negative difference is 0.40 points, supporting a broad improvement rather than a result driven by one dataset or architecture.Source: A. Complete dataset–backbone results · p. 11

A.1. Per-method comparisons by dataset and backbone

The supplementary comparisons report per-dataset results and implementation checks, with distinct scopes for the matched LE-GDA comparison and comparisons to published results.

A.1. Per-method comparisons by dataset and backbone

  • Tables S2–S12 report per-method results separately by dataset, while Table S1 compares LE-GDA with its matched baseline across datasets and backbones at K = 16.Table S1 averages results over three independently sampled training sets.Source: §A.1 · p. 11, Table S1 · p. 12
  • Table S13 averages results over 11 datasets and distinguishes published values from matched results; its delta is matched minus published.Source: Table S13 · p. 16
  • Published-versus-matched reimplementation deviations in Table S13 are at most 0.65 points; Proto-Adapter agreement with published RN50 means supports the reimplementation.Tip-Adapter and Tip-Adapter-F use released code; Proto-Adapter is our implementation.Source: §A.1 · p. 16, Table S13 · p. 16
  • For RN50, excluding ImageNet to remove the split mismatch yields 78.94 for LE-GDA versus 78.00 for published GPCache, a +0.94-point difference.Source: §A.1 · p. 16

B. Error overlap among selected feature spaces

Selected feature spaces make different errors from the matched final-feature search control, but the analysis does not establish error decorrelation as the source of LE-GDA’s accuracy gain. A numerical check also confirms the implementation of the GDA-CLIP estimator on one test setting.

B. Error overlap among selected feature spaces

  • H is 0.0352±0.0265 higher for selected feature spaces than the matched search control across 44 settings, but lower error overlap does not establish the gain’s mechanism.The difference is positive in 41 of 44 settings (p = 1.6×10−9); pooled correlation between −O and gain is ρ = 0.221 (p = 0.173).Source: B. Error overlap among selected… · pp. 16–17, Table S15 · p. 16

C. Numerical fidelity to the published GDA-CLIP estimator

  • On DTD with ViT-B/16 at K = 16, the trace-scaled ridge estimator matched the published closed-form scores up to a positive factor and preserved every test prediction.This is an implementation check, not a reproduction of published aggregate accuracies.Source: C. Numerical fidelity to the published… · p. 17

D. Gaussian-mixture ablation protocol

The supplementary analyses test within-class mixtures, score scaling, and alternative fusion rules. They find that the gain is associated with combining feature spaces and that per-query standardization better aligns selection with standalone accuracy than uniform averaging.

D. Gaussian-mixture ablation protocol

  • Adding within-class mixture components reduced accuracy by as much as 2.10 points at m = 16, while validation-selected m recovered only 0.020 points over m = 1.The m = 1 estimator is identical to LE-GDA; covariance was held fixed as component count changed.Source: D. Gaussian-mixture ablation protocol · p. 17, Table S16 · p. 17

E. Score scaling in the non-standardized control

  • The non-standardized control divides each space’s scores by a validation-derived scale; absolute value and ε guards prevent class-order reversal and unbounded amplification near zero.LE-GDA’s standardized fusion is invariant to positive rescaling apart from its fixed standard-deviation floor.Source: E. Score scaling in the… · p. 17

F. Representation selection under the two fusion rules

  • Across 132 settings, selection rates track standalone accuracy on all four backbones under per-query standardization, but not under uniform averaging.The spaces dropped by uniform averaging had lower score dispersion than those retained (median rank difference −0.333, p = 1.6 × 10−10).Source: F. Representation selection under the… · p. 17

F.1. Fusion rules and controlled selection comparisons

  • LE-GDA beat its matched baseline in 109 of 131 non-tied comparisons; per-query standardization was the only tested fusion alternative with a positive median difference from uniform averaging.Other tested alternatives reduced median accuracy; controlled comparisons show uniform averaging could not reliably separate predictive quality from residual score scale.Source: §F.1 · p. 18, Table S17 · p. 18, Table S18 · p. 18

F.2. Per-dataset selection under per-query standardization

Per-dataset selection patterns differ between ResNets and ViTs, while transfer analyses show modest validation-free gains and stronger performance when configuration selection uses target-dataset validation. LE-GDA’s additional classifier storage remains below that of an example-level cache.

F.2. Per-dataset selection under per-query standardization

  • On ResNets, standardized selection follows standalone accuracy; on ViTs, it spreads across near-equivalent spaces, with block 11 and pre-projection features within 1.5 points of the final embedding on every dataset.The figures show per-dataset selection counts and standalone top-1 differences relative to the final embedding at K = 16.Source: §F.2 · p. 18, Figure S1 · p. 19, Figure S2 · p. 19

G. Configuration transfer across datasets

  • The transfer experiment derives the representation multiset, ridge coefficients, and fusion coefficient from the other ten datasets, then applies them unchanged to the held-out dataset.The configurations separately restore validation selection for individual choices to measure their contributions.Source: G. Configuration transfer across… · p. 21

G.1. Complete configuration results

  • At K = 16, the validation-free difference between Configurations 3 and 1 ranged from 0.09 to 1.19 points across backbones, while Configuration 4 consistently outperformed Configuration 3.The primary validation-free comparison was not significant; target-dataset validation primarily improves configuration selection rather than the closed-form classifier.Source: §G.1 · p. 21, Table S22 · p. 22

G.2. Per-dataset results at 16 shots

  • The four negative transfer datasets had relatively strong zero-shot text classifiers, and the transferred configuration assigned excessive weight to the visual classifier.Seven datasets were positive and four negative across all four backbones.Source: §G.2 · p. 21, Table S23 · p. 23

G.3. Validation-curve sensitivity of the fusion coefficient

  • Local flatness around a dataset’s selected fusion coefficient does not bound the loss from transferring the complete configuration with a different representation multiset.Transferred α accounted for 46% of the 2.270-point Configuration 4–3 difference at K = 16 and 60% of the 2.850-point difference at K = 4.Source: §G.3 · p. 21

G.4. Checks for validation leakage and model-selection nesting

  • Validation-free evaluation replaced validation tensors with an object that raises an exception on every access and reproduced the reported standardized results exactly.Positive rescaling of each feature-space score vector left Configuration 3 accuracy unchanged.Source: §G.4 · p. 21

G.5. Ablation with uniform averaging

  • Transferring the representation multiset had no significant difference under uniform averaging at K = 16, but cost 0.941 points under LE-GDA’s per-query standardization.Source: §G.5 · p. 21

H. Model storage

  • LE-GDA stores 2,051–3,074 floating-point values per class: three to five times the final-feature discriminant’s storage, but only 19–31% of Tip-Adapter’s cache.It requires one frozen CLIP forward pass and stores a median of three distinct feature spaces.Source: H. Model storage · p. 23, Table S24 · p. 23

Related papers

Loading 2610.34901v1…