PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Prompt Search

Yuanhan Zhang, Kaiyang Zhou, Ziwei Liu

arXiv:2206.04673v2cs.CVcs.AIcs.LG

TL;DR

Large vision models make transfer learning difficult because full fine-tuning can overfit and increases computational costs, while prompt-module choices vary across datasets. NOAH uses neural architecture search to select prompt-module designs for each dataset and outperforms individual modules across transfer, few-shot, and domain-generalization evaluations.

  • Problem

    Large vision models make full fine-tuning prone to overfitting and costly, while existing prompt modules require difficult dataset-specific design choices.

  • Method

    NOAH incorporates Adapter, LoRA, and VPT into Transformer blocks and uses one-shot neural architecture search to learn dataset-suited prompt designs.

  • Results

    NOAH outperforms individual prompt modules on 10 of 19 VTAB-1k datasets and is highly competitive on the remainder, with superiority also shown in few-shot learning and domain generalization.

  • Takeaways & Limitations

    Search-based prompt design is better than hand-engineering for the evaluated downstream transfer, few-shot, and domain-generalization settings.

  • Takeaways & Limitations

    NOAH requires additional supernet training and more labeled images to realize its full few-shot advantage.

Abstract

from arXiv · show

The size of vision models has grown exponentially over the last few years, especially after the emergence of Vision Transformer. This has motivated the development of parameter-efficient tuning methods, such as learning adapter layers or visual prompt tokens, which allow a tiny portion of model parameters to be trained whereas the vast majority obtained from pre-training are frozen. However, designing a proper tuning method is non-trivial: one might need to try out a lengthy list of design choices, not to mention that each downstream dataset often requires custom designs. In this paper, we view the existing parameter-efficient tuning methods as "prompt modules" and propose Neural prOmpt seArcH (NOAH), a novel approach that learns, for large vision models, the optimal design of prompt modules through a neural architecture search algorithm, specifically for each downstream dataset. By conducting extensive experiments on over 20 vision datasets, we demonstrate that NOAH (i) is superior to individual prompt modules, (ii) has a good few-shot learning ability, and (iii) is domain-generalizable. The code and models are available at https://github.com/Davidzhangyuanhan/NOAH.

1 Introduction

Large vision models make full fine-tuning costly and prone to overfitting, while existing parameter-efficient modules require dataset-specific design choices. NOAH addresses this by searching over prompt-module architectures and reports stronger transfer performance across diverse datasets.

  • Large vision models increase overfitting risk during transfer learning while also raising compute and storage costs.
  • Adapter, LoRA, and VPT reduce tuning cost by training tiny inserted modules while freezing most pre-trained model parameters.Adapter uses bottleneck networks, LoRA uses low-rank residual layers, and VPT adds learnable input tokens.
  • None of the three prompt modules performs consistently across VTAB-1k datasets, so selecting the best method requires extensive dataset-specific evaluation.VPT leads on SmallNORB/azimuth but performs substantially worse on SmallNORB/elevation and Clevr/count.
  • Prompt-module performance is also sensitive to design parameters such as Adapter feature dimension and VPT token length.The optimal VPT token length varies from 1 to 200 across datasets.
  • NOAH treats existing tuning methods as prompt modules and uses neural architecture search to learn dataset-specific designs for large vision Transformers.Its search space subsumes Adapter, LoRA, and VPT within each Transformer block, with module parameters selected by one-shot NAS.
  • On VTAB-1k, NOAH significantly outperforms individual prompt modules on 10 of 19 datasets and remains highly competitive on the others.The experiments also report superiority in few-shot learning and domain generalization.

2 Neural Prompt Search

The section reviews Transformer-based vision models and three parameter-efficient prompt modules, then presents NOAH, which searches their designs for each dataset. Its search combines a module-rich space, one-shot supernet training, and evolutionary architecture selection.

  • 2.1 Background: Vision Transformer alternates multihead self-attention and multi-layer perceptron blocks for processing token sequences.The input sequence has token length N and embedding dimension D; self-attention projects tokens into queries, keys, and values.
  • 2.1 Background: Multihead self-attention computes value-weighted representations from query-key relationships, with d serving as the scaling factor.The projections use Wq, Wk, and Wv to produce queries, keys, and values before attention aggregation.
  • 2.1 Background: Adapter inserts a bottleneck network using down-sampling, nonlinear activation, and up-sampling to transform normalized MLP outputs.Its down-sampled dimension is r, with Wdown and Wup defining the two learned projections.
  • 2.1 Background: LoRA updates query and key projections indirectly through rank-decomposed changes rather than directly optimizing the full projection matrices.The rank-decomposition parameters use a down-projection dimension r and modify the projections for a given input.
  • 2.1 Background: VPT prepends m learnable D-dimensional tokens to a Transformer block’s input sequence and applies prompting across multiple layers.The method treats these learnable tokens as additions to the input alongside the class token and image patch embeddings.
  • 2.2 Prompt Search Algorithm: NOAH treats Adapter, LoRA, and VPT as prompt modules and learns the dataset-specific design that best suits each downstream task through one-shot NAS.Its supernet is a ViT-like model with 12 Transformer blocks, and AutoFormer performs the prompt-module search.
  • 2.2 Prompt Search Algorithm: The search space places VPT at block inputs, LoRA beside query and key projections, and Adapter after normalized MLP outputs.Each module searches embedding dimensions and depths, with depths selected from {3, 6, 9, 12}; VPT’s embedding dimension is token length, while Adapter and LoRA use r.
  • 2.2 Prompt Search Algorithm: Supernet training randomly samples prompt-module depths and embedding dimensions while learning only prompt parameters, after which evolutionary search selects a subnet under a parameter limit.The evolutionary process uses high-performing architectures as parents and generates new candidates through crossover and mutation.

3 Experiments

Across VTAB-1k, few-shot learning, domain generalization, and subnet analyses, NOAH generally outperforms individual prompt modules while revealing dataset-dependent, complementary architectures.

  • VTAB-1k: On VTAB-1k group averages, NOAH performs best in Natural and Structured tasks, while matching LoRA closely in the Specialized group.NOAH is reported at 84.9 versus LoRA at 84.6 on average in the Specialized group.
  • VTAB-1k: NOAH achieves the best overall VTAB-1k performance, averaging 1% higher than the strongest individual prompt module, LoRA.Its advantage is reported across the 19-dataset benchmark, although performance is closer among Specialized tasks.
  • Few-shot learning: With 16 shots, NOAH leads the few-shot benchmarks by around 2% over the runner-up, whereas at 1 or 2 shots it performs similarly to LoRA and Adapter.VPT largely lags in the low-data regime, and no individual module performs consistently across all five datasets.
  • Domain generalization: On four domain-shifted ImageNet variants, NOAH improves over individual modules by 6.8%, 4.8%, 5%, and 5.2%, respectively.The source-domain gap on ImageNet is only about 1%, while robustness gains appear on ImageNet-V2, Sketch, A, and R.
  • Further analysis: NOAH subnets place Adapter and LoRA mainly in deep layers, while VPT spans nearly all depths with group-dependent embedding dimensions.The co-existence of all three modules, especially in deep layers, indicates complementarity that is difficult to obtain through manual design.
  • Further analysis: The ImageNet-found subnet transfers to VTAB-1k with an overall gap below 3%, and gaps below 1% in the Natural group.Transferability is better when source and target datasets are visually closer.

4 Related Work

Related work develops parameter-efficient tuning and prompt-learning methods for large models, while unified frameworks and NAS provide broader design-search perspectives.

  • Parameter-Efficient Tuning: Parameter-efficient tuning inserts a small trainable module into a frozen pre-trained model, including Adapter, LoRA, and VPT.These methods optimize task-specific losses while adjusting only the inserted module parameters.
  • Prompt Learning: Prompt learning converts prompt tokens into learnable vectors and inspired VPT’s learnable pixels for vision Transformers.
  • Unified Tuning: Unified frameworks connect or jointly activate multiple parameter-efficient tuning modules instead of treating each method independently.Adapter and prompt learning have been unified through modification vectors, while UNIPELT uses gating functions to selectively activate modules.
  • Neural Architecture Search: NAS searches over architectures, hyperparameters, or module arrangements using algorithms ranging from reinforcement learning to weight-sharing and differentiable optimization.AutoFormer is a relevant one-shot NAS method for Transformers that shares weights among sampled subnets.

5 Discussion, Limitation and Future Work

The discussion emphasizes that prompt-module design varies across datasets and that NOAH shows promise for transfer, low-data settings, and domain shift, while incurring training cost and needing more labels to realize its advantages.

  • Discussion: Hand-designing an optimal prompt module is extremely challenging for any specific downstream dataset.The paper frames neural prompt search as a NAS-based solution to this design problem.
  • Discussion: NOAH demonstrates potential for downstream transfer learning, low-data regimes, and robustness to domain shift.
  • Discussion: NOAH’s ultimate subnet exhibits different architectural patterns for the three prompt modules across datasets of different natures.The authors relate these patterns to feature progression from low-level primitives in bottom layers to high-level abstractions in top layers.
  • Limitation: NOAH requires additional supernet training, which increases development cost.
  • Limitation: NOAH’s advantages become clearer with more labeled images, so more labels are needed to unleash its full practical power.

A.1 Datasets

The appendix introduces the datasets used, including VTAB-1k, a few-shot benchmark, and a research-only designation for one dataset.

  • Datasets: Table 4 introduces the datasets used in the study.
  • Datasets: The VTAB-1k block covers benchmark datasets, while the second block lists five fine-grained datasets used in the CoOp few-shot learning benchmark.“Research only” indicates that a dataset is intended solely for research purposes.

A.2 Training Details

Training details specify image preprocessing, augmentation, prompt dimensions, optimizer settings, and shared few-shot/domain-generalization configurations.

  • Augmentation: VTAB-1k images are resized to 224 × 224 and normalized with ImageNet means and standard deviation.Few-shot and domain-generalization experiments additionally use color jitter and RandAugmentation.
  • Hyperparameters: Adapter and LoRA use embedding dimension 8, while VPT prompt length follows its original paper and is set to 8 for few-shot and domain-generalization experiments.
  • Hyperparameters: Training uses AdamW with cosine scheduling, weight decay 1e−3, 10 warm-up epochs, and batch size 64.
  • Hyperparameters: Table 5 lists the remaining hyperparameters, including settings for few-shot learning and domain generalization.
Loading 2206.04673v2…