PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Prompt Inference Mitigates Rare-Class Suppression in Imbalanced VLM Finetuning

Emanuel Ben Baruch, Sivan Biham, Alon Shoshan, Yochai Zvik, Avi Ben-Cohen

arXiv:2610.21470v1cs.CV

TL;DR

Imbalanced VLM finetuning can suppress rare-class recall below zero-shot performance, and the standard matched descriptive prompt may preserve that suppression. The paper tests cross-prompt inference and finds that a different descriptive prompt recovers rare-class recall, with TuCo analysis linking the effect to reduced reliance on finetuning-induced computation.

  • Problem

    Under class-imbalanced VLM finetuning, rare-class recall can fall below zero-shot performance, motivating study of how to recover rare-class knowledge.

    Source: §1 · p. 1

  • Method

    The paper evaluates finetuned VLMs with a different descriptive inference prompt and uses TuCo to analyze and reduce the finetuning-induced computational contribution.

    Source: §3.3 · p. 4, §6 · p. 9

  • Results

    Across six datasets and six VLMs, cross-prompt inference recovers rare-class recall, with varying frequent-class trade-offs, while mismatched prompts rely less on finetuning-induced computation.

    Source: §1 · pp. 1–2, §6 · p. 10

  • Takeaways & Limitations

    Rare-class knowledge suppressed by imbalanced finetuning can remain accessible under a different descriptive inference prompt, especially when finetuning substantially suppresses rare-class recall.

    Source: §1 · p. 2, §6 · p. 9

    Experiments use controlled 20-class subsets, so extending the approach to hundreds or thousands of classes may be constrained by practical prompt context lengths.

    Source: §5 · p. 9

Abstract

from arXiv · show

When a vision-language model is finetuned on class-imbalanced data, rare-class recall can fall well below the model's own zero-shot performance. In this regime, we uncover a surprising matched-prompt penalty whereby the standard choice of reusing the descriptive finetuning prompt at inference can yield lower rare-class recall than using a different descriptive prompt. We call this inference-time strategy cross-prompt inference. We evaluate diverse VLMs across six datasets and find that cross-prompt inference consistently recovers rare-class recall, with varying trade-offs in frequent-class recall. The effect reflects a train-eval prompt interaction rather than prompt quality alone, and does not arise with non-descriptive prompt changes. It is strongest when imbalance and visual confusability produce substantial rare-class suppression, and is weak or absent when such suppression does not occur. Using Tuning Contribution (TuCo), which decomposes a model's computation into pretrained and finetuning parts, we find that cross-prompt inference reduces reliance on finetuning-induced computation and shifts the model toward its pretrained component. Reducing that finetuning contribution also substantially reduces the cross-prompt advantage. These results show that rare-class knowledge suppressed by imbalanced finetuning can remain accessible under a different inference prompt.

1 INTRODUCTION

Imbalanced finetuning can suppress rare-class knowledge, and reusing its descriptive prompt at inference can preserve that suppression. The paper identifies cross-prompt inference as a way to recover rare-class recall and examines its prompt interaction, computational basis, and conditions.

1 INTRODUCTION

  • A different descriptive inference prompt can recover rare-class recall that the matched finetuning prompt suppresses, with varying trade-offs in frequent-class recall.The paper calls the effect the matched-prompt penalty and the inference-time strategy cross-prompt inference.Source: §1 · p. 1, Figure 1 · p. 2
  • The penalty reflects interaction between training and evaluation prompts, not prompt quality alone; changing between non-descriptive prompts does not recover rare-class recall.Descriptive content is necessary for the reported recovery.Source: §1 · p. 1
  • TuCo analysis finds that mismatched descriptive prompts rely less on finetuning-induced computation, and reducing that contribution also substantially reduces the cross-prompt advantage.The results support a shift toward pretrained computation and recovery of rare-class knowledge suppressed by finetuning.Source: §1 · p. 2
  • Cross-prompt recovery is strongest as imbalance and visual similarity increase, is weak or absent when rare classes remain intact, and complements oversampling across six datasets and six VLMs.The paper identifies the effect as most relevant where finetuning substantially degrades rare-class performance.Source: §1 · p. 2

2 RELATED WORK

Related work addresses long-tailed recognition, descriptive prompts for VLM classification, and forgetting during VLM finetuning. This paper distinguishes its approach by changing only the inference prompt and studying rare-class suppression under imbalance.

Long-tailed recognition

  • Long-tailed methods use resampling, reweighting, or inference-time prior correction, while recent VLM work modifies training or output logits; this work changes only the inference prompt.Prior VLM studies address adaptation, tail-class harm from heavy finetuning, pretraining label bias, data rebalancing, and rare-concept underrepresentation.Source: §2 · p. 2

Class descriptions for VLM classification

  • Description-based VLM classification uses generated class descriptions or contrastive distinctions, while prompt adaptation learns prompts in training or optimizes them at test time on an unchanged model.Prior work also reports that prompt wording and format can affect accuracy.Source: §2 · pp. 2–3

Forgetting in VLM fine-tuning

  • Prior VLM forgetting work links degraded pretrained abilities to behavioral shifts or prompt changes, whereas this setting keeps task and label space fixed and concentrates degradation on rare classes under imbalance.Kotha et al. (2024) report capability recovery by moving prompts away from the finetuning distribution, mainly through translation into other languages.Source: §2 · p. 3

Prompt mismatch

  • Although prompt mismatch can harm performance and format consistency is often valued, class imbalance makes the matched descriptive prompt suboptimal for rare classes in this setting.An alternative descriptive prompt recovers part of the suppressed rare-class performance.Source: §2 · p. 3

3 PROBLEM SETTING AND CROSS-PROMPT INFERENCE

The paper formulates imbalanced VLM classification and evaluates a finetuned model with a different descriptive prompt. It quantifies matched-prompt effects and uses TuCo to analyze and intervene on the finetuning contribution.

3 PROBLEM SETTING AND CROSS-PROMPT INFERENCE

  • The setup finetunes a VLM for image classification on an imbalanced training distribution, with frequent classes having abundant examples and rare classes few.The imbalance ratio is ρ = max_c n_c / min_c n_c, with ρ ≫ 1.Source: §3 · p. 3
  • For each image and finetuning prompt, the model generates the target class name by minimizing autoregressive cross-entropy over response tokens.The resulting parameters are denoted θ_Pt.Source: §3 · p. 3

3.1 CROSS-PROMPT INFERENCE

  • The conventional inference choice reuses the finetuning prompt, but under severe imbalance it can suppress rare classes; cross-prompt inference instead evaluates with a different descriptive prompt.The proposed rationale is that the alternative prompt may weaken prompt-specific bias and allow greater reliance on pretrained class knowledge.Source: §3.1 · p. 3
  • Cross-prompt inference changes only the inference prompt, not finetuned parameters or decoding; description-free prompt changes do not yield the same rare-class recovery.This distinguishes the effect from prompt mismatch alone.Source: §3.1 · p. 4

3.2 QUANTIFYING THE MATCHED-PROMPT PENALTY

  • The prompt-interaction matrix places matched prompts on its diagonal and cross-prompt evaluations off diagonal, with average and best mismatch gains capturing typical and largest recovery.Both gains are reported averaged over training prompts.Source: §3.2 · p. 4

3.3 FINETUNING CONTRIBUTION ANALYSIS WITH TUCO

  • TuCo decomposes each layer’s residual-stream update into pretrained and extra finetuning updates, and scores the relative magnitude of accumulated finetuning updates at the final prompt token.The decomposition enables analysis of prompt-dependent computation, including changes that cancel by the final layer.Source: §3.3 · p. 4
  • The analysis rescales the TuCo finetuning component by λ: λ = 1 reproduces finetuned computation, while λ = 0 removes that contribution.The intervention tests whether the cross-prompt advantage shrinks as the finetuning contribution decreases.Source: §3.3 · p. 4

4 EXPERIMENTS

Across six datasets and six VLMs, imbalanced finetuning suppresses rare-class performance, while evaluating with a different descriptive prompt recovers rare-class recall. The experiments characterize when this recovery occurs, its relationship to finetuning computation, and how it can be used alongside oversampling and prompt ensembling.

4.1 EXPERIMENTAL SETTING

  • The experiments used six datasets split into ten frequent and ten rare classes, six VLMs, and seven prompts spanning descriptive styles and a plain prompt.Each finetuned model was evaluated under all seven prompts, producing a 7×7 train–evaluation matrix.Source: §4.1 · p. 5

4.2 RARE-CLASS SUPPRESSION

  • Across fine-grained datasets, frequent-class accuracy saturated near 100% within the first epoch while rare-class accuracy fell and remained suppressed, with deeper drops as rare-class samples decreased.The pattern was also reported for the remaining datasets.Source: §4.2 · p. 5, Figure 2 · p. 5

4.3 MATCHED-PROMPT PENALTY AND CROSS-PROMPT INFERENCE

  • Rare-recall gains from averaged mismatched prompts were positive in all 24 dataset–model cells, with further gains from the best mismatch and a dataset-varying frequent-class cost.Cross-prompt inference also complements oversampling; on oversampled models, averaged-mismatch rare-recall gains ranged from +0.9 to +9.5 pp and best-mismatch gains from +5.8 to +22.9 pp.Source: §4.3 · pp. 6–7, Figure 3 · p. 6

4.4 RARE-CLASS DEGRADATION: IMBALANCE AND CLASS SIMILARITY

  • Rare-class recall declined as imbalance increased and visual similarity between rare and frequent classes rose; cross-prompt gains grew with suppression, reaching +31.4 pp on Food-101 at 750:1 and +20.2 pp on CUB-200’s most similar partition.When rare recall remained high, matched and mismatched evaluation largely coincided.Source: §4.4 · pp. 7–8

4.5 TUCO ANALYSIS RESULTS

  • Removing finetuning computation raised matched rare-class recall from 67.9% to 89.2% on Food-101 and from 34.3% to 77.7% on FGVC-Aircraft; the mismatch advantage was absent at λ = 0.TuCo was lower under mismatch in all sixteen finetuned models, and comparably so for frequent classes, indicating a global shift in finetuning influence.Source: §4.5 · p. 8

4.6 NON-DESCRIPTIVE PROMPTS

  • Switching between two non-descriptive prompts did not improve rare recall, while the matched descriptive pairing was the weakest descriptive setting and switching to another descriptive prompt improved recall.The reported results suggest that the mismatch matters most, rather than which side carries descriptions.Source: §4.6 · p. 8

4.7 LEVERAGING CROSS-PROMPT INFERENCE

  • Averaging class scores across evaluation prompts improved rare-class recall on all four datasets relative to matched-prompt inference, at roughly half the average frequent-class cost of individual mismatched prompts.Including the matched prompt produced smaller rare-class gains while better preserving frequent-class recall; combining cross-prompt inference with oversampling achieved the highest overall recall among evaluated configurations on all four datasets.Source: §4.7 · p. 9

5 LIMITATION

Experiments use controlled 20-class subsets to keep per-class descriptions practical. The authors identify scaling to hundreds or thousands of classes, and the presence of rare-class suppression, as boundaries on cross-prompt inference.

5 LIMITATION

  • Experiments use controlled 20-class subsets so per-class descriptions remain practical in the VLM prompt.Extending the approach to hundreds or thousands of classes may be constrained by practical context lengths for class descriptions.Source: §5 · p. 9
  • Cross-prompt inference may provide little benefit or even hurt when finetuning does not suppress rare-class performance.The authors identify its greatest value as settings where imbalanced finetuning suppresses knowledge already present in the pretrained model.Source: §5 · p. 9
  • The paper identifies settings with hundreds or thousands of classes as future work because descriptions may exceed practical context lengths.Source: §5 · p. 9

6 CONCLUSION

The conclusion identifies cross-prompt inference as a way to recover rare-class recall suppressed by imbalanced VLM finetuning. Its benefit is greatest when suppression is substantial, and TuCo analysis supports a shift toward pretrained computation.

6 CONCLUSION

  • A different descriptive inference prompt can recover suppressed rare-class recall without retraining when the matched finetuning prompt is suboptimal.The benefit varies with imbalance ratio and visual similarity between frequent and rare classes, and is largest where finetuning substantially suppresses rare-class recall.Source: §6 · p. 9
  • The authors interpret cross-prompt inference as drawing more on pretrained class knowledge.Source: §6 · p. 9
  • TuCo shows lower relative contribution from finetuning-induced computation under mismatched prompts, and removing that component removes the cross-prompt advantage.An extensive study across six datasets and six VLMs supports these findings.Source: §6 · p. 10

AI USE STATEMENT

The authors used generative AI tools to assist with paper refinement, figure creation, and software code. They also used a large language model to draft per-class descriptions, which the authors verified against dataset documentation.

AI USE STATEMENT

  • Generative AI tools assisted with refining and polishing parts of the paper, figure creation, and software code.Source: AI USE STATEMENT · p. 10
  • A large language model drafted Appendix J per-class descriptions from class names and dataset taxonomies, and the authors checked them against dataset documentation.Source: AI USE STATEMENT · p. 10
  • The authors validated and tested AI-assisted code, reviewed and edited AI-assisted text and figures, and take responsibility for the final content.Source: AI USE STATEMENT · p. 10

A IMPLEMENTATION DETAILS

The implementation uses a fixed configuration across datasets, prompts, seeds, imbalance ratios, and backbones, without hyperparameter search. Classification compares candidate class-name log-probabilities and selects the argmax.

A IMPLEMENTATION DETAILS

  • The dataset-configuration table identifies nf and nr as images per frequent and rare class, with nf set to the largest count supported by the source pool.The supplied table text also lists dataset, ratio, and train/test fields.Source: Table 4 · p. 14, A IMPLEMENTATION DETAILS · p. 14
  • For each image, classification compares log-probabilities for candidate class names and takes the argmax, making predictions deterministic.The method does not use free-form generation for classification.Source: A IMPLEMENTATION DETAILS · p. 14
  • One configuration is used across every dataset, prompt, seed, imbalance ratio, and backbone, with no hyperparameter search.The prompt bank is fixed before runs, and descriptions are authored once per dataset, class, and style.Source: A IMPLEMENTATION DETAILS · p. 14

B FINETUNING CONTRIBUTION ANALYSIS: DETAILS

The analysis decomposes finetuned computation into pretrained and finetuning contributions, then tests how those contributions relate to prompt mismatch and rare-class recall. Across sixteen finetuned models, TuCo is lower under mismatched prompts in every model, while the matched prompt has the highest TuCo in thirteen.

B FINETUNING CONTRIBUTION ANALYSIS: DETAILS

  • Each finetuned layer is decomposed into pretrained and finetuning parts whose sum reproduces the finetuned computation exactly.Scaling the finetuning part by λ interpolates between the components; the analysis extends λ to 0 to remove that component, though this internal measure need not track output-level metrics.Source: B FINETUNING CONTRIBUTION ANALYSIS:… · p. 14
  • TuCo is lower under mismatch in all sixteen models, and the matched prompt has the highest TuCo in thirteen.Table 5 reports mean TuCo for each model under its training prompt and averaged over the six remaining prompts.Source: Table 5 · p. 15, B FINETUNING CONTRIBUTION ANALYSIS:… · p. 15

C RECALL BREAKDOWN ACROSS ALL MODELS

  • Table 6 reports rare, frequent, and overall recall breakdowns across models, with delta columns measured relative to matched prompts.The recall measures average per-class recall over all 20 classes or the rare or frequent subset; Table 1 summarizes Qwen3-VL-2B.Source: Table 6 · p. 16, C RECALL BREAKDOWN ACROSS ALL MODELS · p. 16

B FINETUNING CONTRIBUTION ANALYSIS: DETAILS

  • Figure 7 tracks frequent- and rare-class accuracy over finetuning epochs across rare-class sample counts, with rare accuracy falling below zero-shot levels more as samples decrease.The figure identifies epoch 0 as the zero-shot model and includes the nr = 0 setting, which trains on frequent classes only.Source: Figure 7 · p. 16

D RARE-CLASS SUPPRESSION

The suppression pattern holds on CUB-200 and Stanford Dogs, and the appendix describes prompt-selection and LoRA tests. Selecting a mismatched prompt with as few as five labeled rare-class examples gives the highest overall recall, while the matched-prompt penalty also persists under LoRA.

D RARE-CLASS SUPPRESSION

  • On CUB-200 and Stanford Dogs, the rare-class accuracy drop deepens as the rare-class training count nr decreases.Figure 7 repeats the suppression experiment with the same frequent/rare class assignment as the earlier experiment.Source: D RARE-CLASS SUPPRESSION · p. 16

E SELECTION METHODS FOR CROSS-PROMPT INFERENCE

  • Selecting a mismatched prompt with as few as five labeled images per rare class gives the highest overall recall; ensembling is comparable but requires multiple forward passes.Prompt selection uses a minimal held-out validation set; the reported results average 200 random splits.Source: Table 7 · p. 17, E SELECTION METHODS FOR CROSS-PROMPT… · p. 17
  • Mismatched-prompt recall can be averaged across prompts, ensembled per-class before prediction, or obtained by selecting one prompt using validation data.The selection protocol uses k labeled rare-class images for selection and evaluates on the remaining images.Source: E SELECTION METHODS FOR CROSS-PROMPT… · p. 17

F LORA VERSUS FULL FINETUNING

  • The LoRA comparison uses rank 16 adapters with scaling 32 on attention and feedforward projections, keeping data, prompts, epochs, and seeds identical to full finetuning.The comparison changes only the adaptation method.Source: F LORA VERSUS FULL FINETUNING · p. 17
  • The matched-prompt penalty persists under LoRA with comparable magnitude, so it is not specific to full-parameter finetuning.On CUB-200, LoRA has higher matched-prompt recall than full finetuning, while its cross-prompt gain remains comparable and is slightly larger.Source: F LORA VERSUS FULL FINETUNING · p. 17

G RARE-CLASS DEGRADATION: IMBALANCE AND CLASS SIMILARITY IMPLEMENTATION DETAILS

The implementation details fix a shared full-finetuning protocol and construct frequent–rare class partitions to vary imbalance and visual similarity. Similarity is controlled by permuting those partitions while keeping the selected classes constant.

G RARE-CLASS DEGRADATION: IMBALANCE AND CLASS SIMILARITY IMPLEMENTATION DETAILS

  • All runs use Qwen3-VL-2B with full language-model finetuning for six epochs, while the vision encoder and multimodal projector remain frozen.The experiments use a fixed bank of seven prompt styles and select 20 classes per dataset.Source: G RARE-CLASS DEGRADATION: IMBALANCE… · p. 17
  • For each dataset, ten visually similar class pairs are split into ten frequent and ten rare classes, fixed across imbalance ratios so only rare-class training counts vary.Pairs are selected using SigLIP2 similarity, and no class is reused.Source: G RARE-CLASS DEGRADATION: IMBALANCE… · p. 18
  • To vary similarity, the study creates six frequent–rare partitions at a fixed dataset-specific imbalance ratio and measures each rare class against its most similar frequent class using SigLIP2 cosine similarity.The same 20 classes are used across partitions, which are permuted to obtain different similarity levels.Source: G RARE-CLASS DEGRADATION: IMBALANCE… · p. 18, Table 9 · p. 18

H DESCRIPTIVE PROMPTS AT FINETUNING AND EVALUATION

This section examines descriptive prompts at finetuning and evaluation, reporting how evaluation-prompt choice and description placement relate to rare-class recall across four datasets, alongside seed variability. It also specifies the prompt grid for ImageNet-20 and PlantNet-20, including descriptions that highlight distinguishing traits and contrast visually similar classes.

H DESCRIPTIVE PROMPTS AT FINETUNING AND EVALUATION

  • The prompt-placement analysis compares five configurations formed by using plain prompts (P) or descriptive prompts (D) at finetuning and evaluation.The reported rare-recall values are averaged over prompts and seeds.Source: H DESCRIPTIVE PROMPTS AT FINETUNING… · p. 18
  • Food-101 and Stanford Dogs do better with descriptions only during finetuning, whereas CUB-200 and FGVC-Aircraft do better with descriptions only at evaluation.The penalty follows from using the same prompt at both ends, rather than from where descriptions are placed.Source: H DESCRIPTIVE PROMPTS AT FINETUNING… · p. 18
  • Switching to a different descriptive evaluation prompt improves rare recall on all four datasets, so matched descriptive prompting is never best.The comparison reports rare-class recall averaged over prompts and seeds.Source: H DESCRIPTIVE PROMPTS AT FINETUNING… · p. 18, Table 10 · p. 19

I SEED VARIABILITY

  • The seed-variability analysis reports rare-class recall, gains over the matched prompt, and standard errors across seeds.Table 11's caption describes the values in brackets as gains over the matched prompt and their standard errors.Source: I SEED VARIABILITY · p. 19, Table 11 · p. 19

J PROMPT GRID SPECIFICATION

  • The prompt specification uses 20 classes per dataset—10 frequent and 10 rare—and seven prompts: one plain prompt plus six descriptive prompts.The descriptive prompts combine appearance, contrastive, or attribute styles with descriptions for all classes or rare classes only.Source: J PROMPT GRID SPECIFICATION · p. 19
  • The plain prompt lists class names only, while each descriptive prompt adds class descriptions in one of three styles and applies them to all classes or only rare classes.The section provides per-class prompt specifications for reproducibility.Source: J PROMPT GRID SPECIFICATION · p. 19
  • Table 16 orders ImageNet-20 descriptions as ten similarity-matched pairs, frequent class first; detailed-all covers the whole column, while detailed-rare covers only rare rows.The table provides per-class descriptions for all three styles.Source: Table 16 · p. 24
  • ImageNet-20 examples describe animal classes through distinguishing traits such as salamanders’ spot patterns, spiders’ abdomen markings, and shorebirds’ bill shapes.Some descriptions also contrast the trait directly with that of the paired class.Source: Table 16 · p. 24, J PROMPT GRID SPECIFICATION · p. 25
  • Other ImageNet-20 examples distinguish paired reptiles and insects by features including snout shape, snake head shape, antenna length, and resting wing position.The prompts state these characteristics both as descriptions and as explicit differences from the paired class.Source: Table 16 · p. 24, J PROMPT GRID SPECIFICATION · pp. 25–26
  • Table 17 gives PlantNet-20 per-class descriptions for all three styles, again ordering ten similarity-matched pairs with the frequent class first.Detailed-all supplies the whole column, whereas detailed-rare supplies only rare rows.Source: Table 17 · p. 27
  • PlantNet-20 examples distinguish paired plants through visible traits such as flower petal shape, clover head size, and succulent leaf color.The descriptions contrast broader versus narrow petals, fuller versus tiny flower heads, and yellow-green versus copper-orange leaves.Source: Table 17 · p. 27, J PROMPT GRID SPECIFICATION · p. 28
  • Further PlantNet-20 descriptions differentiate paired rock plants and ferns by petal notching and leaflet length, spacing, and shape.The examples contrast notched with unnotched petals and short, close-set leaflets with long, widely spaced toothed leaflets.Source: Table 17 · p. 27, J PROMPT GRID SPECIFICATION · p. 28

Related papers

Loading 2610.21470v1…