PaperlayerView exact revision on arXiv

Source-linked AI summary

Low-Rank Prompt Learning for Vision-Language Models with Fixed-Token Bases

Tanvir Muntakim Tonoy, Sajjad Ghiasvand, Mahnoosh Alizadeh, Ramtin Pedarsani

arXiv:2609.09462v1cs.CV

TL;DR

The paper asks whether dense CoOp prompts are over-parameterized in few-shot adaptation. It factorizes prompts, evaluates fixed token bases, and finds that optimizing embedding-side coefficients over a fixed basis matches trainable factorization and dense CoOp while using fewer parameters.

  • Problem

    Few-shot CoOp uses a dense prompt with md free parameters despite only a handful of labeled examples per class, raising whether this parameterization is over-parameterized.

  • Method

    The paper factorizes the prompt as P = BA, compares trainable and fixed token-side bases, and analyzes factor asymmetry and local update-space dimensions.

  • Results

    Across seven few-shot benchmarks and two CLIP backbones, low-rank prompts match or improve dense CoOp, while fixed Gaussian, orthogonal, SVD-derived, or random B stays on par with fully trainable factorization.

  • Takeaways & Limitations

    Embedding-side coefficients carry prompt adaptation, so CLIP prompt learning can optimize A over a fixed low-dimensional token basis, reducing trainable parameters to rd.

  • Takeaways & Limitations

    The theoretical asymmetry analysis relies on a linear prompt-space surrogate that approximates local adaptation by least squares.

Abstract

from arXiv · show

Prompt learning adapts CLIP to downstream recognition by replacing hand-written templates with learned continuous context vectors, which in Context Optimization (CoOp) form a dense prompt matrix $\mathbf{P}\in\mathbb{R}^{m\times d}$ trained from only a few examples per class. We study whether this matrix is over-parameterized by factorizing it as $\mathbf{P}=\mathbf{B}\mathbf{A}$, which cuts the trainable prompt parameters from $md$ to $r(m+d)$, and to $rd$ once the token-side factor $\mathbf{B}$ is fixed. Across seven few-shot benchmarks and two CLIP backbones, low-rank prompts match or improve dense CoOp at far fewer parameters, with the clearest gains on low-shot base-to-new generalization. We then find that the token-side factor need not be learned at all: fixing $\mathbf{B}$ to a Gaussian, orthogonal, SVD-derived, or even random basis and training only the embedding-side factor $\mathbf{A}$ stays on par with the fully trainable factorization, and a source-trained $\mathbf{B}$ offers no advantage over a random one. A prompt-factor asymmetry and a local update-space dimension gap show why fixing $\mathbf{B}$ is far less restrictive than fixing $\mathbf{A}$, and a smoothness-only guarantee certifies that optimizing $\mathbf{A}$ over a fixed $\mathbf{B}$ converges. In the CLIP prompt setting, the embedding-side coefficients carry the adaptation while the token basis can simply be fixed.

1 Introduction

The introduction asks whether dense CoOp prompts are over-parameterized in few-shot learning and proposes low-rank factorization to reduce trainable parameters. Experiments and analyses indicate that the token-side basis can be fixed while embedding-side coefficients carry adaptation.

  • Motivation: CoOp uses a dense prompt matrix with md parameters despite only a handful of labeled examples per class.The standard configuration has 16 × 512 = 8,192 prompt parameters.
  • Low-rank formulation: P = BA reduces trainable parameters from md to r(m + d), or to rd when B is fixed.B mixes latent components across context positions, while A places them in the CLIP text-embedding space.
  • Factor geometry: Factorized prompts show a nearly shared token-side basis across datasets, shots, and ranks, while A and BA vary substantially.The diagnostic motivates testing whether B must be learned.
  • Empirical findings: Across seven few-shot benchmarks, low-rank prompts match or improve dense CoOp with the largest gains in low-shot base-to-new generalization.The result is reported across two CLIP backbones in the broader experimental summary.
  • Empirical findings: Gaussian, orthogonal, SVD-derived, and random fixed bases match fully trainable factorization, while source-trained B offers no advantage over random B.These fixed-basis and transfer controls support training only A.
  • Theory: A prompt-factor asymmetry, update-space dimension gap, and smoothness-only guarantee explain why fixing B remains effective.The contribution combines geometric analysis with a convergence guarantee for optimizing A over fixed B.

2 Related Work

Related work adapts vision-language models through prompts or low-rank updates while generally keeping pretrained backbones fixed. Existing low-rank approaches span weight-space and prompt-space methods, commonly training both factors.

  • Prompt Learning: Prompt learning adapts pretrained vision-language models without modifying the backbone through textual, visual, or multimodal prompts.Textual methods optimize continuous vectors in the language branch, visual methods add learnable visual tokens, and multimodal methods prompt both branches.
  • Low-Rank Adaptation: Weight-space low-rank methods such as CLIP-LoRA and Block-LoRA adapt frozen-backbone projections, whereas prompt-space methods impose low-rank structure on prompts.The related work distinguishes adapting pretrained weights from leaving the backbone untouched.
  • Positioning: The paper's approach belongs to prompt-space adaptation, motivated by low-rank parameter efficiency while retaining a frozen CLIP backbone.This positions factorized prompt learning alongside, but distinct from, backbone adapter methods.

3 Preliminaries

CLIP classifies images by comparing frozen image features with text features produced from class prompts. CoOp replaces fixed template tokens with learnable shared context vectors, making the prompt matrix the sole adapted object.

  • CLIP: CLIP encodes an image into a visual feature and a class description into a text feature using frozen pretrained encoders.The image is split into patches, while the text sequence contains special tokens, template words, and a class name.
  • CLIP: Zero-shot prediction scores each image against class descriptions using cosine similarity and a temperature parameter.The score compares the visual feature with each class text feature.
  • Prompt Learning with CoOp: CoOp replaces the fixed template words with m learnable continuous context vectors optimized from a few labeled examples.The context vectors are shared across classes and inserted alongside the class name and special tokens.
  • Prompt Learning with CoOp: Only the prompt matrix P is trained by minimizing few-shot classification loss while the CLIP encoders remain frozen.The prompt matrix is therefore the sole object of adaptation and is factorized in the next section.

4 Low-Rank Prompt Learning

The paper factorizes continuous CLIP prompts into token-side and embedding-side factors, then studies which factor must be learned. Theory shows that fixing the token basis preserves more prompt capacity and admits more local update directions, while smoothness guarantees convergence when optimizing the remaining factor.

  • 4.1 Factorized Prompt Parameterization: The prompt is factorized as P = BA, reducing trainable parameters from md to r(m + d), or rd when B is fixed.B mixes latent components across context positions, while A places them in the CLIP text-embedding space.
  • 4.1 Factorized Prompt Parameterization: The study compares joint factor training, fixed-B training of A, and fixed-A training of B to test whether the factors are interchangeable.Fixed-B experiments use Gaussian, scaled-orthogonal, SVD-derived, and previously learned same-task bases; fixed-basis initialization projects a sampled dense prompt onto the selected basis.
  • 4.2 Prompt-Factor Asymmetry: Under the linear prompt-space surrogate, fixing B projects in the m-dimensional token-position space, whereas fixing A projects in the d-dimensional embedding space.Because d is typically much larger than m, fixing the embedding-side factor imposes the more severe restriction.
  • 4.2 Prompt-Factor Asymmetry: For m = 16 and d = 512, a random fixed token basis removes r/16 of ideal prompt energy in expectation, compared with r/512 for a fixed embedding-side factor.The expected residual ratio is (1 − r/16)/(1 − r/512) for ranks r ≤ 16.
  • 4.3 Local Update-Space Dimension Gap: Locally, fixing B and training A provides rd update directions, while fixing A and training B provides mr directions, yielding a d/m = 32 dimension ratio.The classification objective is locally approximated by projecting an ideal update onto parameterization-admitted subspaces under a curvature metric.
  • 4.3 Local Update-Space Dimension Gap: The asymmetry and dimension gap jointly indicate that A preserves more ideal-prompt content and supplies more adaptation directions, while B can be fixed.When the curvature metric becomes Euclidean, the local analysis recovers the Frobenius surrogate exactly.
  • 4.4 Convergence Under a Fixed Factor: With B fixed, gradient descent on A converges to a first-order stationary point under directional smoothness and lower boundedness, without convexity or strong convexity.The full-gradient method uses step sizes 0 < η ≤ 1/L_A and has an O(1/T) rate at η = 1/L_A; an analogous stochastic guarantee holds for unbiased bounded-variance gradients.
  • 4.4 Convergence Under a Fixed Factor: The fixed basis scale controls optimization conditioning through L_A ≤ L||B||_2^2, making an orthonormal B the cleanest case.This motivates norm-matching the fixed bases in the experiments.

5 Experimental Setup

The experiments evaluate factorized prompts across seven diverse classification datasets, two CLIP backbones, and standard few-shot protocols. Tables compare generalization, fixed-basis variants, and parameter-matched dense baselines under controlled training conditions.

  • Datasets and backbones: Evaluation covers seven datasets spanning object, scene, texture, and action recognition, using CLIP ViT-B/16 and RN50 backbones.The datasets are Caltech101, DTD, FGVC Aircraft, Food101, Oxford Flowers, Oxford Pets, and UCF101.
  • Base-to-new generalization: Table 1 compares rank-4 factorized CoOp with dense CoOp for seen, unseen, and harmonic-mean base-to-new performance across seven datasets and three shot settings.Means use three seeds, with per-seed standard deviations shown for individual rows and bold marking the better mean in each column.
  • Training details: Training uses frozen CLIP encoders, a shared 16-token context, and 1-, 4-, and 16-shot settings with three random seeds.All variants share the optimizer, schedule, augmentation, batch size, and data splits of dense CoOp.
  • Fixed token-side basis: Table 2 reports mean accuracy across seven datasets for Gaussian, orthogonal, SVD-derived, and learned-then-frozen B bases across ranks and shots.Each fixed-B variant trains only A and is compared with the fully trainable factorization.
  • Few-shot accuracy and parameter efficiency: Table 3 compares factorized prompts with Dense-16 and parameter-matched Dense-4 using mean accuracy across seven datasets and three seeds.Header counts are trainable prompt parameters, and Δ denotes factorized minus Dense-4.

6 Experimental Results

Across few-shot benchmarks, low-rank prompts improve or match dense CoOp generalization and accuracy with fewer parameters. Fixed token-side bases remain competitive, while factor-geometry analyses show that B is much more stable than A or BA across task changes.

  • Generalization to New Classes: +3.58, +4.25, and +1.74 H points at 1, 4, and 16 shots on ViT-B/16; RN50 gains were +4.21, +3.01, and +2.05.Rank-4 factorized prompts improved H in all 21 dataset–shot cells on ViT-B/16 and 17 of 21 on RN50.
  • Few-Shot Classification Accuracy: +1.93 and +1.99 mean-accuracy points at 1 shot versus parameter-matched Dense-4 on ViT-B/16 and RN50, respectively.At 4 and 16 shots, the methods were statistically indistinguishable; the factorized prompt used roughly a quarter of Dense-16’s parameters.
  • Fixing the Token-Side Basis: 18 of 24 rank–shot–backbone settings favored a non-trainable token basis over the fully trainable factorization.Gaussian, orthogonal, SVD-derived, and learned-then-frozen bases generally stayed within a few tenths of a point of the trainable reference.
  • Transfer Controls: 89% versus 78% average accuracy when training A versus B in transfer controls, a gap that held on every target except Food101.Source-initialized and random B differed by at most about 0.3 points per pair, while training both factors averaged 89.01%, identical to freeze-B/train-A.
  • Factor Geometry: Table 5 compares median subspace overlap and mean principal angle for B, A, and BA across controlled changes in seed, dataset, shot count, and rank.Higher overlap and lower angle indicate greater alignment.
  • Factor Geometry: 0.86 to 0.89 token-basis overlap versus 0.17 to 0.35 embedding-factor and prompt overlap across datasets, shots, and ranks.The corresponding principal angles were near 27–30° for B and 70–80° for A and BA.

7 Conclusion

The paper concludes that low-rank CoOp prompts can freeze the token-side basis and learn only embedding-side coefficients. Geometry and local update-space analyses explain why this is effective, while the scope remains limited to CoOp-style text-only prompts and CLIP backbones.

  • Conclusion: Low-rank prompts match or improve dense CoOp with fewer parameters, and fixed token bases can perform without loss relative to learned bases.The supported conclusion covers random fixed bases and the broader fixed-basis setting.
  • Prompt-factor asymmetry: Fixing B restricts prompts through token-position projections, whereas fixing A restricts them through CLIP embedding-space projections.Because d is typically much larger than m, the embedding-space restriction is more severe.
  • Prompt-factor asymmetry: Fixing B and training A preserves a larger expected fraction of ideal prompt updates than fixing A and training B.In the CLIP setting, m = 16 and d = 512, so the two one-sided choices have different projection dimensions.
  • Scope of analysis: The analysis uses a linear prompt-space surrogate and a local quadratic approximation, so its geometric explanation abstracts from the nonlinear CLIP text encoder.The surrogate isolates parameterization geometry rather than reproducing the full CoOp objective.
  • Local update spaces: With B fixed, the local update space has dimension rd, while fixing A yields a smaller space by a factor of 32 in the CLIP setting.The comparison is between feasible subspaces under the local prompt objective.

A.3 Smoothness-Only Optimization Guarantees

The paper gives convergence guarantees for optimizing A with B fixed under directional smoothness assumptions. These guarantees cover full-gradient and stochastic-gradient descent without requiring convexity or strong convexity.

  • Guarantees: Fixing B and optimizing A converges to a first-order stationary point under directional smoothness.The guarantee does not require convexity or strong convexity.
  • Setup: The restricted objective is defined as ϕ_B(A) = F(BA) for a fixed token-side factor B.The optimization is performed over the embedding-side factor A.
  • Full-gradient descent: Full-gradient descent requires smoothness and lower boundedness of the restricted objective.The theorem applies under Assumptions 7 and 8.
  • Stochastic optimization: Stochastic-gradient descent additionally assumes unbiased gradients with bounded variance and uses a step size bounded by 1/L_A.The result is stated under Assumptions 7, 8, and 9.
  • Rate: Choosing η = Θ(T^-1/2) yields an O(T^-1/2) convergence rate.The step size must also maintain η ≤ 1/L_A.

B.1 Datasets and evaluation splits

Evaluation covers seven few-shot classification benchmarks, multiple shot counts, and three seeds, with fixed implementation choices for class partitions and prompt parameterization. The audit could not reconstruct exact historical image assignments from preserved results alone.

  • Datasets: The evaluation uses Caltech101, DTD, FGVC Aircraft, Food101, Oxford Flowers102, Oxford-IIIT Pets, and UCF101.UCF101 is evaluated as static-image classification using extracted video midframes.
  • Few-shot protocol: Few-shot experiments use K ∈ {1, 4, 16} examples per class and seeds {1, 2, 3}.Validation samples use min(K, 4) examples per class.
  • Evaluation splits: Base-to-new evaluation assigns the first ceil(C/2) sorted class identifiers to base and the remainder to new, consistently across seeds.Few-shot sampling precedes class subsampling.
  • Audit limitation: The audit could not reconstruct exact historical image assignments because original split files, few-shot caches, and image manifests were not recovered.Fallback split procedures are used when CoOp-style split files are unavailable.
  • Model and prompts: The experiments use RN50 and ViT-B/16 checkpoints with prompt-token width d = 512, while dense CoOp uses m = 4 and m = 16.Factorized prompts use m = 16.
  • Parameterization: At m = 16, d = 512, and r = 4, dense, joint-factorized, fixed-B, and fixed-A prompts train 8192, 2112, 2048, and 64 parameters, respectively.The remaining CLIP parameters are frozen.

B.3 Training and initialization

The experiments use fixed CLIP backbones, few-shot training, and multiple prompt-factor initialization and geometry analyses. Reported results aggregate three seeds, but recovered records limit reproducibility and interpretation of factor changes, numerical rank, and computational savings.

  • Training configuration: Training uses SGD for 200 epochs with cosine decay, one warm-up epoch, cross-entropy loss, batch size 32, and final-step checkpoint selection.The preserved launch paths use an initial learning rate of 0.002, momentum 0.9, weight decay 5 × 10^-4, and FP16 defaults.
  • Data processing: Images are augmented with random resized crops, horizontal flips, CLIP normalization, and 224 × 224 evaluation using resize, center crop, and bicubic interpolation.
  • Initialization: Joint factorization initializes from a balanced truncated SVD, while fixed-basis variants initialize the trainable factor by least-squares projection onto Gaussian, orthogonal, SVD-derived, or learned bases.Transfer experiments instead replace the frozen factor and retain the target initialization of the trainable factor.
  • Aggregation: Results report means and sample standard deviations across three seeds, with macro summaries averaging datasets within each seed before aggregating seed-level averages.Some fixed-basis and base-to-new exports instead pool 21 dataset-seed or run-level observations, so their standard deviations have a different interpretation.
  • Interpretation limits: Factor comparisons are descriptive endpoint comparisons and do not establish how much B changes during training or whether archived low-precision products retain exact rank-4 structure.Numerical rank diagnostics can be altered by rounding residuals, and no inferential significance or equivalence tests were recovered.
  • Evaluation scope: The factorized prompt sweeps cover seven datasets, 1/4/16 shots, ranks 1/2/4/8, and RN50 and ViT-B/16 backbones, while Table 7 reports best-rank and dense-CoOp deltas.

C.2 Factor-transfer results

Factor-transfer analyses compare source-frozen and random-frozen factors across transfer modes, while basis ablations evaluate whether fixed token-side factors preserve performance. The preserved records support numerical summaries but not exact replay of every historical run.

  • Factor transfer: Transfer experiments compare freezing A and training B versus freezing B and training A, with each frozen factor either source-trained or randomly drawn.The comparisons use ViT-B/16 rank-4 prompts and report mean accuracy with sample standard deviation over three seeds.
  • Base-to-new generalization: Base-to-new tables evaluate seen, unseen, and harmonic-mean accuracy at 1, 4, and 16 shots for dense CoOp, trainable factorization, and fixed orthogonal-B variants.
  • Reproducibility: The preserved project includes numerical records and documentation, but complete historical logs and trained factor checkpoints were not recovered, preventing exact replay of every training run.

D.3 Potential impacts and limitations of use

The paper frames lower trainable prompt state as a potential adaptation benefit but limits the implications to the evaluated CoOp-style classification setting. It does not establish broader efficiency, fairness, privacy, or deployment claims.

  • Potential impact: Reducing trainable prompt parameters can reduce adaptation state associated with each task, but the study does not establish lower training energy, total GPU memory, or inference latency.
  • Scope boundaries: The findings concern CoOp-style text prompts on evaluated classification benchmarks and do not establish the same behavior for deeper multimodal prompting systems or other pretrained model families.The adapted models also inherit limitations from pretrained CLIP and the benchmark data.
Loading 2609.09462v1…