PaperlayerView exact revision on arXiv

Source-linked AI summary

LAMP: Layerwise Adaptive Mixing of Predictors for Training-Free Few-Shot Vision-Language Adaptation

Zhipeng Ye, Feng Jiang, Qiufeng Wang, Hao Li

arXiv:2610.22182v1cs.CV

TL;DR

Few-shot adaptation often uses only an encoder’s final global descriptor, leaving intermediate-layer information underused despite differences in dimensions and task-specific usefulness. LAMP combines global and regional kernel predictions in class-score space, with LAMP-F adding text-anchored prototype adaptation. Across the reported benchmarks, LAMP outperforms the paired global-feature-only baseline and reaches state-of-the-art mean accuracy among compared published training-free RN50 methods at 4, 8, and 16 shots.

  • Problem

    Few-shot adaptation often relies on final global descriptors, while intermediate-layer features offer complementary information whose dimensions and usefulness vary across tasks.

    Source: §1 · p. 2

  • Method

    LAMP combines independently fitted global and support-calibrated regional kernel predictors in class-score space, while LAMP-F adds text-anchored prototype adaptation with cached prediction updates.

    Source: §1 · p. 3

  • Results

    LAMP achieves state-of-the-art mean accuracy among compared published training-free RN50 methods at 4, 8, and 16 shots, reaching 78.73% at 16 shots.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    LAMP combines features of different dimensions without learning a shared projection, providing a way to use complementary frozen-encoder features for few-shot prediction.

    Source: §6 · p. 22

    Using multiple layers increases storage and kernel computation costs; on ImageNet at 16 shots, batch-size-one latency rises from 14.22 to 64.83 ms on ViT-B/16 and from 14.81 to 65.86 ms on RN50.

    Source: §5.2 · p. 22

Abstract

from arXiv · show

Few-shot adaptation often relies on an encoder's final global representation, leaving complementary information in intermediate layers underused. Combining these features is challenging because their dimensions can differ and their usefulness varies across tasks. We propose LAMP, Layerwise Adaptive Mixing of Predictors, a training-free framework that combines global and regional information from multiple encoder layers. A single forward pass per view supplies all descriptors. Global and support-calibrated regional kernel predictors independently fit the same label residuals. Labeled validation data select their mixture in class-score space, preserving native feature dimensions without learned projections. LAMP-F further adapts text-anchored class prototypes, using a cached response to update the fixed-kernel residual predictions exactly. Experiments cover eleven datasets and ViT and ResNet backbones. LAMP achieves state-of-the-art mean accuracy among the compared published training-free ResNet-50 methods at 4, 8, and 16 shots, reaching 78.73% at 16 shots. Averaged across five support budgets, LAMP improves over a paired global-feature-only baseline by 1.14 and 1.59 percentage points on ViT-B/16 and ResNet-50; LAMP-F raises these gains to 1.96 and 2.54 points. LAMP-F reaches 16-shot accuracies of 83.57% and 79.10%, respectively. Controlled experiments identify multi-layer integration as the main source of improvement and examine pooling, calibration, validation labels, and computational cost. These findings show that combining layer-specific predictions can exploit complementary features from frozen encoders to improve few-shot classification.

1. Introduction

Few-shot adaptation commonly relies on final global descriptors, although intermediate layers contain complementary visual information and differ in dimension and task-specific usefulness. LAMP combines global and regional predictions in class-score space, and its reported evaluations show gains across ViT and ResNet backbones.

1. Introduction

  • Many adaptation methods use final global descriptors, while intermediate token grids and convolutional maps retain visual information at different abstraction levels.Integrating layers is challenging because their dimensions differ and the most useful layers can vary by task.Source: §1 · p. 2
  • LAMP independently fits global and support-calibrated regional kernel predictors to the same label residual, then uses labeled validation data to select their mixture.A single encoder pass per view supplies descriptors from three layers, and score-level fusion preserves native feature dimensions without gradient optimization or learned projections.Source: §1 · pp. 2–3
  • LAMP achieves state-of-the-art mean accuracy among compared published training-free RN50 methods at 4, 8, and 16 shots, reaching 78.73% at 16 shots.Across five support budgets, LAMP improves over the paired global-feature-only baseline by 1.14 percentage points on ViT-B/16 and 1.59 on RN50; LAMP-F raises those gains to 1.96 and 2.54 points, respectively.Source: §1 · pp. 2–3
  • Evaluations with RN101 and ViT-B/32 and transfer to ImageNet-V2 and ImageNet-Sketch assess generalization across architectures and image distributions.The reported evaluation spans eleven datasets, five support budgets, and ViT and ResNet backbones.Source: §1 · pp. 2–3
  • LAMP-F adapts text-anchored class prototypes while retaining the selected kernels and mixture weights, using a cached response to update predictions exactly.The cached response accounts for changes to both initial scores and residual corrections as prototypes change.Source: §1 · p. 3

2. Related work

Prior adaptation work changes class representations or visual features through optimization, or adjusts frozen features using global or local information. LAMP instead combines regional predictors in class-score space, retaining each layer’s native feature space.

2.1. Class-level semantic adaptation

  • Prompt, prototype, and text-anchored methods adapt class representations through optimization, whereas LAMP keeps class text vectors fixed and corrects prediction errors using support-image visual similarities.LAMP-F combines text anchoring with global and regional residual predictors and caches prototype-update responses for initial query scores and support residuals.Source: §2.1 · p. 3

2.2. Instance-level global adaptation

  • Global instance-level methods either optimize visual representations or adjust frozen features; LAMP adds intermediate-layer features and combines their predictions while preserving each layer’s original feature space.Its branches fit the same support-label residual, and labeled validation data determine their relative contributions.Source: §2.2 · pp. 3–4

2.3. Patch- and region-level adaptation

  • SSP uses subspace projection for cross-modal alignment, while LAMP fits regional predictors to the same support-label residual and combines their class scores across layers.A single encoder pass per view supplies regional descriptors; comparisons with simpler summaries test the value of retaining regional descriptor sets as well as using multiple layers.Source: §2.3 · p. 4

3. Method

LAMP combines global and intermediate-layer regional information by fitting separate kernel predictors to CLIP support-label residuals and fusing their class-score corrections. LAMP-F additionally adapts text-derived class prototypes while keeping the visual kernels and mixture weights fixed.

3.1. Residual prediction around the CLIP prior

  • LAMP starts from CLIP class scores and fits a regularized global kernel predictor to transfer support-label residuals to queries.The support residual is the difference between one-hot labels and CLIP scores; query-support similarities transfer the fitted correction.Source: §3 · p. 4, §3.1 · p. 6

3.2. Extracting native-layer regional descriptors

  • A single encoder pass yields regional descriptors from three selected layers, using a 2 × 2 pooling grid while retaining each layer’s native channel width.For ViT, token directions are normalized before regional averaging; for RN50, GeM pooling retains activation strength.Source: §3.2 · pp. 6–7

3.3. Calibrating regional kernels

  • Regional kernels average similarities across all region pairs, then center and normalize the resulting embeddings relative to the support bank.Using a common support reference calibrates queries independently of other queries in the batch; the global kernel remains uncentered.Source: §3.3 · pp. 7–8

3.4. Fitting and selecting the residual mixture

  • Each branch independently fits the same residual target with its own kernel, and validation selects the global and regional mixture contributions.The regional weights sum to one, while ρ balances their combined correction against the global correction; independently fitted predictions are reused across candidate mixtures.Source: §3.4 · pp. 8–9

3.5. Prototype adaptation with fixed kernels

  • LAMP-F optimizes text-derived class prototypes using support labels and a confidence-weighted anchor to the original text vectors.The anchor weight is each class’s mean correct-class softmax confidence across original views; higher weights retain prototypes more strongly near their text vectors.Source: §3.5 · p. 9
  • A cached response exactly updates both the CLIP prior and fitted corrections as prototypes change, while visual kernels and mixture weights remain fixed.After caching, evaluating a new prototype matrix requires only matrix products with the prototype-direction change.Source: §3.5 · pp. 9–10

4. Experiments

The experiments compare LAMP and LAMP-F across eleven datasets, five support budgets, and ViT and ResNet backbones. LAMP leads the compared training-free RN50 methods at 4, 8, and 16 shots, while results also examine backbone comparisons, dataset-level performance, and evaluation boundaries.

4.1. Evaluation protocol

  • The main benchmark uses ViT-B/16 and RN50 across eleven datasets, with 1, 2, 4, 8, and 16 support images per class and three support-sampling seeds.Ten datasets have distinct validation and test lists, but ImageNet uses its official validation partition for both parameter selection and benchmark scoring.Source: §4.1 · pp. 10–11

4.2. Comparison with state-of-the-art methods

  • Published comparison coverage differs by backbone and reported results, with ViT-B/16 comparisons including trained methods and RN50 comparisons using published training-free methods.Paired controls use the same support images, augmented views, prompts, and query images.Source: §4.1 · p. 10, §4.2 · p. 11
  • LAMP achieves state-of-the-art mean accuracy among compared training-free RN50 methods at 4, 8, and 16 shots, reaching 78.73% at 16 shots and exceeding the best published training-free result by 1.98 points.The RN50 comparison reports mean top-1 accuracy over eleven datasets.Source: §4.2 · p. 11, Table 2 · p. 12
  • On ViT-B/16, LAMP exceeds CLIP-LoRA by 0.35 and 0.30 points at 8 and 16 shots, respectively, without gradient-based training.LAMP-F raises mean accuracy to 80.94% and 83.57%; the comparison includes published methods that require training.Source: §4.2 · p. 11, Table 1 · p. 12
  • On RN50, LAMP exceeds the strongest published method by 0.77, 1.18, and 1.98 points at 4, 8, and 16 shots, respectively.LAMP-F’s reported means are 65.91, 68.83, 72.65, 75.50, and 79.10% across the five support budgets.Source: §4.2 · p. 11, Table 2 · p. 12
  • At 16 shots on ViT-B/16, LAMP-F ranks highest among listed methods on ImageNet, EuroSAT, Food101, Flowers102, and DTD.Its gains over the best published scores on those datasets are 0.66, 1.58, 0.14, 0.90, and 2.74 points, respectively; other methods lead on Aircraft, Cars, UCF101, SUN397, Caltech101, and Pets.Source: §4.2 · pp. 11, 13, Table 3 · p. 13

4.3. Paired gains over the global baseline

LAMP and LAMP-F improve on a paired global-feature baseline across support budgets and backbones. Dataset-level results show that multi-layer integration and prototype adaptation benefit different tasks.

4.3. Paired gains over the global baseline

  • At 16-shot, LAMP reaches 83.30% with ViT-B/16 and 78.73% with RN50; LAMP-F reaches 83.57% and 79.10%, respectively.The paired global baseline reaches 82.30% and 76.62% on those backbones.Source: §4.3 · p. 13
  • Across five support budgets, LAMP gains 1.14 points with ViT and 1.59 with RN50, while LAMP-F gains 1.96 and 2.54 points.The paired global-feature-only comparison averages equally across datasets and support budgets.Source: §4.3 · p. 13, Table 4 · p. 14

4.4. Dataset-wise performance gains

  • At 4-shot, native-layer integration raises mean accuracy by 1.446 points with ViT and 1.568 with RN50 across eleven datasets.The largest gains are on EuroSAT: 5.860 points with ViT and 6.695 with RN50; gains occur on all eleven datasets with ViT and ten with RN50.Source: §4.4 · pp. 13–14, Table 5 · p. 14
  • At 4-shot, prototype adaptation gains most on SUN397, while native-layer integration is particularly effective on satellite imagery and, with ViT, texture recognition.SUN397 gains are 5.620 points with ViT and 6.715 with RN50; the two adaptation strategies benefit different tasks.Source: §4.4 · p. 14

4.5. Layer depth, pooling, and calibration

Controlled comparisons test whether LAMP's gains come from information across encoder depths, spatial pooling, calibration, or layer weighting. They find that multi-depth information is the main source of improvement, with calibration especially beneficial for RN50.

4.5. Layer depth, pooling, and calibration

  • Native-region kernels outperform MULTIBAND by 1.286 points with ViT and 1.635 with RN50 across ten non-ImageNet datasets.MULTIBAND changes bandwidths while retaining the final global descriptor; family-adjusted 98.75% intervals are positive for both backbones.Source: §4.5 · p. 15
  • WHOLE exceeds FINAL by 1.193 points with ViT and 0.905 with RN50, showing that combining depths helps even when each map is pooled into one descriptor.The pointwise 95% intervals are [0.648, 1.806] and [0.499, 1.384], respectively.Source: §4.5 · p. 15
  • Removing support centering and diagonal normalization reduces RN50 mean accuracy by 0.627 points, while the ViT calibration change is less consistent.On EuroSAT, RN50 accuracy falls from 82.802% to 80.811%; the ViT change is +0.215 [−0.075, +0.514].Source: §4.5 · p. 16
  • Equal layer weights reduce mean accuracy by a further 0.331 points with ViT and 0.180 with RN50, with adjusted intervals including zero.Even after all three simplifications, S3 exceeds the paired global baseline by 2.179 and 1.117 points.Source: §4.5 · p. 16

4.6. Performance gains of LAMP-F over LAMP

LAMP-F usually improves on LAMP, with gains concentrated in particular tasks and support regimes. Additional experiments find that LAMP's gains persist across new support draws, while validation-label budgets affect model selection.

4.6. Performance gains of LAMP-F over LAMP

  • LAMP-F improves the three-seed mean in 53 of 55 settings with ViT and 52 of 55 with RN50 across eleven datasets and five support budgets.Mean gains decrease from 1.097 and 1.463 points at 4-shot to 0.274 and 0.366 at 16-shot for ViT and RN50, respectively.Source: §4.6 · p. 16
  • Against an independently adapted global predictor, LAMP-F improves mean accuracy by 1.840 points with ViT and 1.949 with RN50 at 4-shot.The comparison covers ten non-ImageNet datasets; family-adjusted intervals are [1.451, 2.283] and [1.483, 2.417].Source: §4.6 · p. 16

4.7. Support-draw robustness and validation-label budget

  • With the full validation split, LAMP gains 2.494 and 2.252 points on DTD and 5.253 and 6.738 points on EuroSAT, for ViT and RN50.All four adjusted confidence intervals lie above zero across ten new support draws.Source: §4.7 · p. 17, Figure 2 · p. 17
  • On EuroSAT, one validation image per class selects the global-only candidate in 19 of 20 combinations; four images yield gains of 3.267 points with ViT and 4.857 with RN50.The complete parameter search is repeated for each validation budget.Source: §4.7 · pp. 16–17

4.8. Additional backbones and frozen-target transfer

LAMP improves on published ProKeR scores across four CLIP backbones on ImageNet and remains beneficial when frozen models transfer to two shifted datasets. Its selected mixture strengths and layer contributions vary with task, while tested regional bandwidth ratios yield similar mean performance.

4.8. Additional backbones and frozen-target transfer

  • On ImageNet at 16-shot, LAMP exceeds ProKeR by 1.28, 1.13, 1.00, and 0.85 points for RN50, RN101, ViT-B/32, and ViT-B/16, respectively.LAMP has the highest accuracy among the methods listed for each backbone.Source: §4.8 · p. 17, Table 7 · p. 18
  • With RN101, LAMP improves mean accuracy across ten non-ImageNet datasets at every support budget by 1.196–2.333 points.Across all eleven datasets and five budgets, five of 55 changes are negative; on ImageNet at 16-shot, ViT-B/32 exceeds its paired baseline by 0.465 points.Source: §4.8 · p. 18, Table 8 · p. 18
  • On frozen ImageNet-trained models, LAMP improves accuracy in all twelve seed, target-dataset, and backbone combinations across ImageNet-V2 and ImageNet-Sketch.All four aggregate-gain intervals lie above zero; ViT accuracy is 66.297% on ImageNet-V2 and 49.910% on ImageNet-Sketch.Source: §4.8 · p. 18, Table 9 · p. 19

4.9. Hyperparameter behavior and layer contributions

  • Mean validation performance peaks at ρ = 0.5 for ViT-B/16 and ρ = 0.75 for RN50, supporting a mixture of global and regional corrections.The gains over the global baseline are 1.437 and 1.678 points; at ρ = 1, they are 0.470 and 1.394 points.Source: §4.9 · p. 19, Figure 3 · p. 19
  • Across r values of 0.5, 1, and 2, mean validation gains range from 1.575 to 1.604 points with ViT and 1.746 to 1.754 with RN50.Selecting the mixture separately at each ratio yields similar mean performance across these similarity scales.Source: §4.9 · pp. 19–20, Figure 3 · p. 19
  • Selected mixtures favor the global branch on Pets but assign larger regional contributions on DTD and EuroSAT; preferred depths also vary by task.Flowers102 places its largest ViT regional weight on the late layer, while DTD favors the middle layer.Source: §4.9 · p. 20, Figure 4 · p. 20

4.10. Computational cost

The section quantifies LAMP's inference, throughput, and memory costs and identifies compact regional descriptors as a storage–accuracy option.

4.10. Computational cost

  • At 16-shot ImageNet, LAMP raises batch-one latency from 14.22 to 64.83 ms (ViT-B/16) and 14.81 to 65.86 ms (RN50).Batch-128 throughput changes from 113.38 to 107.89 images/s for ViT-B/16 and 118.64 to 112.28 images/s for RN50.Source: §4.10 · p. 20, Table 10 · p. 21
  • At batch 128, peak memory rises from 2.71 to 7.64 GiB (ViT) and 3.02 to 11.55 GiB (RN50); LAMP-F reaches 8.56 and 13.39 GiB.Table 10 measures throughput and peak allocated GPU memory at batch 128 on one RTX 4090.Source: §4.10 · p. 20, Table 10 · p. 21
  • On the DTD 4-shot diagnostic bank, CENTROID matches REGION accuracy on both backbones with one quarter of the native descriptor storage.Source: §4.10 · p. 21

5. Discussion

The discussion attributes adaptation gains to combining features across layers, describes task-dependent calibration and prototype adaptation, and notes their computational costs.

5.1. Implications for training-free adaptation

  • Matched-search-budget controls show that multi-layer descriptors outperform a final-global-feature ensemble, while pooling several maps outperforms pooling only the deepest map.Regional centroids and whole-map descriptors retain most of the accuracy gain, offering compact options for limited-storage or strict-latency applications.Source: §5.1 · p. 22
  • Calibration is particularly effective for RN50 on EuroSAT, layer weights vary across datasets and backbones, and LAMP-F gains most with few support images.Caching prototype responses makes their evaluation inexpensive relative to constructing regional predictors.Source: §5.1 · p. 22

5.2. Limitations

  • Using multiple layers increases storage and kernel computation costs; dense fitting requires O(N^2) storage and O(N^3) factorization per branch.On ImageNet at 16 shots, batch-one latency rises from 14.22 to 64.83 ms on ViT-B/16 and from 14.81 to 65.86 ms on RN50; LAMP-F also stores prototype responses.Source: §5.2 · p. 22

6. Conclusion

The conclusion presents LAMP as training-free multi-layer prediction mixing for frozen vision–language encoders, with strong RN50 results and an exact prototype-update extension.

6. Conclusion

  • LAMP achieves state-of-the-art mean accuracy among compared training-free RN50 methods at 4, 8, and 16 shots, while matching or exceeding representative trained methods at higher support budgets.Source: §6 · p. 22
  • LAMP-F adds text-anchored prototype adaptation and uses a cached response to update predictions exactly.Source: §6 · p. 22
  • LAMP combines global and regional predictions while retaining native feature dimensions, avoiding a learned shared projection.The conclusion describes separating branch fitting, score fusion, and prototype updates as a foundation for studying task-specific layer selection and more scalable residual prediction.Source: §6 · p. 22

Declaration of generative AI and AI-assisted technologies

The authors disclose using ChatGPT/Codex during manuscript preparation and state that they reviewed the material and take responsibility for the article.

Declaration of generative AI and AI-assisted technologies

Related papers

Loading 2610.22182v1…