What Does Prompt Learning Change? -A Natural-Language Concept Analysis of Vision-Language Models

Method

• PromptSpLiCE represents initial and learned prompt embeddings as sparse combinations of concepts from one fixed natural-language dictionary.The shared coordinate system makes fitted coefficient profiles directly comparable before and after learning.Source: §1 · p. 1

Results

• 1.6 of the initial top-10 concepts remain in the top 10 after CoOp learning, on average across 11 image-classification datasets.The result indicates substantial reorganization of natural-language concept profiles.Source: §1 · p. 1
• Larger concept-profile changes tend to accompany larger accuracy gains across datasets.The reported relationship is an association, not a stated causal effect.Source: §1 · p. 1
• Qualitative results show that recognizable class concepts can persist even while less intuitive terms rise in rank.The analysis therefore captures both persistence and re-ranking within learned profiles.Source: §1 · p. 1

Geometric analysis

• A local gradient analysis gives geometric intuition for why image-aligned concept directions distinct from the current prompt can have greater loss sensitivity.This extends the profile comparison with an embedding-level explanation of directional sensitivity.Source: §1 · pp. 1–2

Prompt Learning

• CoOp freezes pretrained model parameters and learns text-prompt token embeddings to maximize image similarity for the ground-truth class.Its learnable embeddings start from a manually designed prompt such as “A photo of a {class}.”Source: §2.1 · p. 2
• Most prompt-learning methods prioritize classification performance and leave semantic changes in task-adapted continuous prompts difficult to observe directly.Interpretability-aware methods instead incorporate human-readable attributes or concepts during training or inference.Source: §2.1 · p. 2

Post-hoc concept decomposition

• Post-hoc concept decomposition explains high-dimensional CLIP embeddings using human-interpretable concepts while keeping the pretrained model fixed.Sparse autoencoders reconstruct embeddings with learned basis vectors interpreted from common activation properties.Source: §2.2 · p. 2

Related analyses

• PatchSAE compares visual-feature activations before and after MaPLe adaptation, whereas this analysis studies final text embeddings after CoOp using a fixed natural-language dictionary.The cited comparison identifies the analyses as complementary rather than interchangeable.Source: §2.2 · p. 2

SpLiCE

• SpLiCE reconstructs CLIP embeddings as sparse nonnegative combinations of natural-language concept embeddings, and PromptSpLiCE compares coefficients before and after prompt optimization.The shared decomposition procedure shifts the analysis to prompt-level semantic changes.Source: §2.2 · p. 2

3.1 Concept Decomposition of Prompt Embeddings

• PromptSpLiCE applies Sparse Linear Concept Embeddings to represent prompt embeddings as sparse combinations of a fixed natural-language dictionary.Using one dictionary before and after learning enables visualization and comparison of fitted coefficient profiles.Source: §3.1 · p. 2, §3 · p. 2
• The dictionary contains M natural-language concepts encoded by CLIP, with their embeddings arranged as dictionary columns.The construction provides the concept coordinate system used for decomposition.Source: §3.1 · p. 2
• Centering and normalization mitigate CLIP embedding anisotropy so a shared common direction does not obscure differences among concepts.The preprocessing removes the dominant common component before fitting concept coefficients.Source: §3.1 · p. 2
• Nonnegative sparse coefficients are fitted post hoc by trading reconstruction error against sparsity with an ℓ1 penalty.Because the dictionary is overcomplete and correlated, different coefficient supports may yield similar reconstructions.Source: §3.1 · p. 3
• Direction-only reconstruction retains the embedding direction while fixing residual scale to one, introducing an approximation relative to exact inversion.Exact inversion would require both the direction and scale; the scale-aware expression is provided separately.Source: §3.1 · p. 3

3.2 Local Sensitivity in Concept Coordinates

• The local sensitivity analysis treats concept coefficients as local coordinates for a cosine-similarity classification objective while holding image and other class embeddings fixed.The setup derives a hypothetical embedding-level sensitivity in concept coordinates.Source: §3.2 · p. 3
• The prediction-error factor scales sensitivity, while temperature and embedding norm rescale updates without changing their qualitative dependence on concept direction.Orthogonal projection determines directional alignment with the image embedding; near-parallel dictionary directions have small projected components and sensitivity.Source: §3.2 · p. 4
• A concept can change substantially when its component orthogonal to the prompt aligns with the image’s orthogonal component, producing high local loss sensitivity.Figure 1 contrasts this case with nearly parallel dictionary directions, whose changes are small.Source: Figure 1 · p. 3, §3.2 · p. 4

Experimental Setup

• PromptSpLiCE evaluates post-hoc natural-language decompositions of prompt embeddings before and after learning across 11 image-classification datasets.The datasets span object, fine-grained, scene, action, texture, and satellite-image classification.Source: §4 · p. 4, §4.1 · p. 4

Embedding Reconstruction Fidelity

• At λ = 0.01, learned prompts use approximately 450 nonzero coefficients and achieve a cosine similarity of 0.98 to their full reconstructions.The reconstruction is sparse relative to the 10,000-term dictionary, but 450 terms are not a concise explanation; top-10 displays are illustrative subsets.Source: §4.2 · p. 4

Qualitative Analysis

• EuroSAT’s “Annual Crop Land” profile shifts from farmland and crops toward defend and baseman after prompt learning.The StanfordCars example retains class-related terms such as fiat and convertible while also elevating less intuitive terms.Source: §4.2 · p. 5, §4.3.2 · p. 5

Quantitative Analysis

• The rank-transition analysis groups terms as High→High, High→Low, or Low→High using fixed dictionary, solver, and regularization settings.The cutoffs are top 10 before and after learning, below rank 100 after learning, or below rank 100 before learning, respectively.Source: §4.3 · p. 5, §4.3.1 · p. 5
• 1.6 initial top-10 terms remain in the top 10 on average across 11 datasets; 6.8 fall below rank 100 and 6.8 rise into the top 10.High→High terms are few on most datasets, while High→Low and Low→High transitions are common.Source: §4.3.1 · p. 5

Exploratory Association with Accuracy Gain

• The dataset-level correlation between mean JS divergence and CoOp accuracy gain is r = 0.64 (nominal p = 0.035, n = 11).The paper treats this as an exploratory association and notes sensitivity to the small sample and prominent EuroSAT endpoint.Source: §4.4 · p. 6

• PromptSpLiCE places class-conditioned text embeddings before and after learning in a shared coordinate system defined by a fixed natural-language dictionary.This post-hoc analysis makes continuous prompt changes readable through named concepts without modifying prompt learning.Source: §5 · pp. 6–7
• Larger fitted concept-profile changes tended to accompany larger classification-accuracy gains across the 11 datasets.The reported association is exploratory.Source: §5 · pp. 6–7, Figure 7 · p. 7

• The preprocessing separates the centered prompt embedding into a direction and a scale.The formulation writes the embedding as its mean plus a scaled centered direction.Source: §A.1 · p. 8
• The reconstruction pipeline defines a decomposition residual and then forms an unnormalized reconstruction before final unit normalization.The supplied passage sequence describes these intermediate reconstruction stages.Source: §A.1 · p. 9
• The direction-only reconstruction sets ρ_z = 1 and therefore does not generally exactly invert preprocessing.Exact inversion would require retaining both the direction and the scale.Source: §A.1 · p. 9

• The derivation differentiates each concept-coordinate term and combines the resulting expressions into the gradient with respect to the full concept-coordinate vector.The calculation proceeds through derivatives with respect to s_i and normalized embeddings z_i.Source: §A.2 · p. 9
• The local calculation assumes each w_i affects only s_i, whereas the actual method optimizes prompt parameters that may be shared across classes.The general parameter gradient uses the text-embedding Jacobian J_i.Source: §A.2 · p. 9
• The resulting expression is class-wise embedding-level sensitivity, not the total derivative through the Lasso solution and changing active support.At nonnegativity boundaries, the derivative is local and feasible coefficient perturbations are one-sided.Source: §A.2 · p. 9

B Additional Qualitative Results

• Figures 8–10 cover the remaining seven datasets using before-versus-after coefficient scatter plots and High→High, High→Low, and Low→High transition colors.Green, red, and orange denote the three transition types; gray marks remaining dictionary terms.Source: B Additional Qualitative Results · p. 9, Figure 8 · p. 10, Figure 9 · p. 10
• Category-related terms can remain highly ranked while other related and less intuitive dictionary terms undergo different rank transitions.Examples include DTD’s “lined” class and FGVCAircraft’s “Cessna 172” class.Source: B Additional Qualitative Results · p. 9
• Substantial coefficient-rank turnover can coexist with retention of highly diagnostic terms.This pattern appears in the UCF101 “Bench Press” and Caltech101 “watch” examples.Source: B Additional Qualitative Results · pp. 9–10
• Across the qualitative cases, at least one intuitive term often remains High→High, while Low→High frequently includes labels with no clear human relation to the class.The observed pattern is consistent with Table 1.Source: B Additional Qualitative Results · p. 11
• High full-embedding reconstruction fidelity alone does not establish stability or semantic fidelity of displayed top-ranked terms.The qualitative transitions motivate distinguishing reconstruction quality from interpretability of ranked concepts.Source: B Additional Qualitative Results · p. 11