PaperlayerView exact revision on arXiv

Source-linked AI summary

Your Model Already Knows Don't Teach It, Learn to Ask It: Soft Prompting for Few-Shot Adaptation of Vision-Language Models

Gautam Rajendrakumar Gare, Siyi Li, Hewei Wang, Cesar Daniel Hernandez, Wei Zhao, Wolfgang M. Pauli, John Galeotti, Deva Ramanan

arXiv:2609.11310v1cs.CVcs.AIcs.LGeess.IVstat.ML

TL;DR

Few-shot VLM adaptation for specialized domains must work with ten annotated images without sacrificing the pretrained model’s general capabilities. The paper learns a few continuous prompt tokens in a frozen backbone, placing them at the cross-modal boundary and initializing them from the space token. This matches the best LORA detection result with far fewer parameters and no forgetting, while supporting transfer, verbalization, and extension to robot policies.

  • Problem

    Specialized-domain VLM detection requires adaptation from only ten annotated images, while existing discrete prompt optimization and LoRA provide incomplete alternatives for balancing adaptation and retained general capabilities.

  • Method

    SOFTPROMPT optimizes one to three continuous tokens at the cross-modal boundary of a frozen VLM, using space-token initialization and training only those tokens.

  • Results

    14.2 mAP matches the best LORA configuration on Roboflow20-VL with over 20,000× fewer trainable parameters and zero forgetting, while the approach also transfers, verbalizes, and helps a frozen robot policy.

  • Takeaways & Limitations

    Prompt placement and initialization make soft prompting competitive for multimodal adaptation while retaining the portability and editability of language prompts.

  • Takeaways & Limitations

    Generalization is not uniform across VLM families: on Gemma-4-12B, holdout selection can reverse on test, including a 2.9 mAP loss versus baseline on defect detection.

Abstract

from arXiv · show

We address few-shot object detection with vision-language models (VLMs) in out-of-domain settings such as aerial, industrial, and medical imagery, using only ten annotated images for supervision. Existing adaptation methods are discrete prompt optimization and LoRA fine-tuning. We revisit a third option: soft prompting, where a small number of continuous prompt tokens are optimized while the pretrained backbone remains frozen. We identify two key design choices. First, placing prompt tokens at the cross-modal boundary between visual and text tokens outperforms other placements (10.0 vs. 8.4 mAP). Second, initializing prompts from the empty space token outperforms semantic and random initialization. With these choices, one to three learned tokens (7,168 parameters on average) match the best LoRA configuration on Roboflow20-VL (14.2 mAP, 10-shot) while training over 20,000x fewer parameters. Soft prompting remains harder to optimize, exhibiting higher variance across random seeds. Unlike LoRA, however, it causes no forgetting: the LoRA rank matching our accuracy reduces NaturalBench VQA accuracy by 35% relative, rising to 56% at the largest rank, whereas soft prompting leaves pretrained performance unchanged. The learned tokens behave like prompts rather than weights. They transfer to a newer model without retraining (+0.8 mAP on Qwen3.5-9B) and can be verbalized into readable prompts competitive with prompt-search methods (matching DetPO and outperforming GEPA). The approach also extends beyond detection. On RoboCasa manipulation tasks, the frozen $π_{0.5}$ vision-language-action policy benefits from soft prompting, matching the LoRA baseline on two of three tasks when tokens are placed at the gradient bottleneck. These results suggest modern VLMs already encode much of what is needed for specialized domains; the challenge is learning how to ask.

1 INTRODUCTION

Few-shot adaptation of VLMs for specialized domains must balance accuracy against scarce annotations and damage to general-purpose capabilities. SOFTPROMPT addresses this by learning a few continuous tokens in a frozen backbone, with cross-modal placement and meaning-preserving initialization yielding competitive accuracy, efficiency, transfer, and broader applicability.

  • Motivation: Specialized-domain VLM detection remains difficult because common-object competence does not reliably transfer to aerial, industrial, and medical imagery under ten-image supervision.The paper frames the central practical question as how to spend very small annotation budgets.
  • Adaptation costs: 35% relative NaturalBench VQA forgetting at accuracy-matching LORA rank r=64 contrasts with zero forgetting from SOFTPROMPT.LORA forgetting rises to 56% at r=128, whereas frozen-backbone prompting leaves pretrained performance unchanged.
  • Design choices: 10.0 average mAP from cross-modal placement beats 8.4 for the NLP-standard prefix transplant and 6.7 for the worst ordering.Space-token initialization also beats every semantic and random initialization tested.
  • SOFTPROMPT: 14.2 mAP matches the best LORA configuration on Roboflow20-VL using 7,168 trainable parameters on average and over 20,000× fewer parameters.The backbone remains frozen, while the matching LORA rank uses 174.6M parameters.
  • Beyond detection: SOFTPROMPT tokens transfer to a newer model, can be verbalized into editable prompts, and extend to frozen vision-language-action policies.On RoboCasa, placement at the gradient bottleneck matches or beats LORA on two tasks, while a prefix-only prompt fails.

2 RELATED WORK

Prior adaptation methods either search discrete prompts with a frozen model or update parameters through PEFT. SOFTPROMPT belongs to the frozen-model prompt family but learns continuous input tokens by gradient descent rather than discrete search.

  • VLM detection: Specialist open-vocabulary detectors remain stronger on raw mAP, but they are single-task models rather than generalist VLM adaptations.Grounding DINO and OWL-ViT serve as specialist reference points.
  • Prompt tuning and PEFT: Soft prompting trains continuous input tokens while keeping the model frozen, unlike deep prompt methods that re-inject prompts throughout transformer layers.The paper positions its method as shallow prompt tuning, distinct from VPT-Deep and P-Tuning v2.
  • Hard prompt optimisation: Hard prompt methods search discrete vocabulary sequences, whereas SOFTPROMPT optimizes continuous prompt embeddings with numerical gradient descent.GEPA, TextGrad, AutoPrompt, and DetPO represent related discrete or language-mediated alternatives.
  • Prompt distillation and forgetting: SOFT2HARD differs from prompt distillation by asking the adapted model to verbalize its own soft tokens without student training.The paper also treats forgetting analysis as measurement using NaturalBench and RefCOCO probes, not as a new mitigation.

3 METHOD

SOFTPROMPT inserts a small set of learnable tokens between visual and textual embeddings, trains only those tokens against the frozen model’s detection loss, and initializes them to preserve the original prompt meaning. The method also supports prompt verbalization into reusable hard prompts.

  • 3.1 SOFT TOKENS AT THE CROSS-MODAL BOUNDARY: The multimodal sequence is formed as [I; S; T], with image embeddings I, soft tokens S, and text embeddings T arranged in the chosen ordering.The six candidates are prefixes, suffixes, and the two cross-modal boundaries IST and TSI.
  • 3.1 SOFT TOKENS AT THE CROSS-MODAL BOUNDARY: IST places learnable soft tokens between image patches and text tokens, allowing them to attend to visual context and influence the text side.The embedding matrix is untouched; the model gains new soft-token rows without modifying existing embeddings.
  • 3.2 TRAINING AND WHAT THE GRADIENT REACHES: Training minimizes the standard next-token detection loss on the ten-shot support set while keeping the backbone parameters frozen.Only the input-side soft-token parameters receive updates, so gradients traverse the frozen network to a single trainable sink.
  • 3.2 TRAINING AND WHAT THE GRADIENT REACHES: L is selected per domain, with Roboflow configurations using L ∈ {1, 2, 3} and 7,168 trainable parameters on average for Qwen3-VL-8B.The corresponding range is 4,096–12,288 trainable parameters.
  • 3.3 INITIALISATION: Space-token initialization starts the adapted model at unchanged prompt meaning, letting optimization move away from the frozen baseline only as required by the loss.This applies a residual-learning principle to prompt initialization.
  • 3.4 SOFT2HARD: SOFT2HARD generates descriptions, summarizes them into a detection instruction, and evaluates that instruction without soft tokens.A matched base-source recipe isolates the contribution of the trained soft prompt through the soft−base gap.

4 EXPERIMENTS

Across 10-shot out-of-domain detection, SOFTPROMPT matches the best LORA accuracy with far fewer parameters and avoids the cross-task forgetting observed at higher-rank LORA settings. Its benefits extend to another benchmark and VLM family, but specialist detectors remain stronger on raw single-task mAP, medical adaptation fails, and Gemma-4 selection is unreliable.

  • 4.2 MAIN RESULTS ON ROBOFLOW20-VL: 14.2 all-domain mAP matches LORA r=64 while SOFTPROMPT trains 24,000× fewer parameters and exceeds every other evaluated LORA rank.The comparison uses 7,168 prompt parameters on average and 174.6M parameters for LORA r=64.
  • 4.3 WHAT ADAPTATION COSTS THE GENERALIST: 35% relative NaturalBench VQA degradation at accuracy-matching LORA r=64 rises to 56% at r=128, whereas SOFTPROMPT incurs no forgetting.LORA detection accuracy peaks at r=64 before declining at r=128, while the frozen backbone preserves pretrained performance under soft prompting.
  • 4.2 MAIN RESULTS ON ROBOFLOW20-VL: 1.2 mAP is the maximum improvement from discrete prompt optimisation over hand-written prompts, while continuous gradient-based prompting reaches 14.2 mAP.GEPA reaches 11.6 mAP and DetPO 12.5 versus 11.3 for class names with instructions.
  • 4.2 MAIN RESULTS ON ROBOFLOW20-VL: 35.7 mAP from a fine-tuned Grounding DINO specialist remains 2.5× higher than generalist methods when raw single-task detection accuracy is the only goal.The specialist does not provide the multi-task generalist capability that motivates the comparison.
  • 4.2 MAIN RESULTS ON ROBOFLOW20-VL: ≤1.0 mAP on Medical is achieved by every generalist adapter, while the fine-tuned specialist reaches 35.3 mAP.The passage characterizes this as a boundary where ten-shot adaptation does not recover the missing knowledge.
  • 4.5 GENERALISING TO A DIFFERENT VLM FAMILY: GEMMA-4-12B: +0.8 mAP is the mean Gemma-4 test gain across eight above-floor datasets, but the +11.0 mAP holdout margin overstates realized performance by roughly 14×.The selected configuration wins on five of eight datasets, while three datasets show a test loss after a holdout win.

5 ABLATION STUDY

The ablations show that soft-prompt performance depends strongly on injection position, prompt length, and initialization. Cross-modal placement and meaning-preserving space-token initialization are strongest overall, while prompt length remains dataset- and position-dependent.

  • 5.1 INJECTION POSITION: 12.4 average mAP: IST, placing tokens at the cross-modal boundary, wins 6 of 8 above-floor datasets and beats the frozen baseline on all 8.Neither soft-first prefix ordering wins any above-floor dataset, and the ordering spread reaches approximately 6× on lacrosse.
  • 5.1 INJECTION POSITION: 6.9–12.4 average mAP: injection position produces large variation, with text-first TSI helping text-dominant domains but failing on image-dependent lacrosse.TSI wins single-class aerial and document-heavy all-elements, while lacrosse scores 4.1 versus IST’s 24.0.
  • 5.2 PROMPT LENGTH: 5.9 mAP at L=8 versus a 27.6 frozen baseline: an oversized prompt can be harmful, so L=0 should remain in every length sweep.Optimal length is dataset-dependent, and length interacts with position: aerial IST decays while TSI reaches 12.1 at L=8.
  • 5.3 INITIALISATION: Validation loss does not track detection mAP across the ablations, so model selection should use the task metric rather than language-model loss.Mean-embedding initialization achieves the lowest validation loss on every dataset but wins or ties on mAP for only two.
  • 5.3 INITIALISATION: A single space-token initialization wins 5 of 8 above-floor datasets, averages 12.4 mAP, and beats the frozen baseline on all 8.Semantic initializations are less reliable, including task-text’s 6.6 versus 24.1 for space on lacrosse.

6 PROMPT DISTILLATION: WHAT DO THE TOKENS ENCODE?

The learned tokens behave like contextual prompts: they can encode missing visual class information or general detection instructions, be verbalized into competitive hard prompts, and transfer partially across model versions. Their benefits are real but uneven across datasets and embedding spaces.

  • 6.1 TWO REGIMES: CLASS SURROGATES VS. GENERAL INSTRUCTIONS: Soft tokens act as visual class surrogates when the class name is withheld, but encode general detection instructions when class names are present.Nearest visual patches retrieve the target class in the probe configuration; with class names present, the tokens no longer align with object patches.
  • 6.1 TWO REGIMES: CLASS SURROGATES VS. GENERAL INSTRUCTIONS: Nearest text tokens or visual patches retain most of the adaptation when substituted for the trained tokens, indicating that their neighbors are functionally meaningful.The substitution result is reported on the ablation pool.
  • 6.2 VERBALISING THE PROMPT: SOFT2HARD: Prompt meaning is contextual: describing a learned token alone is ineffective, whereas preserving its training query context makes it reliably describable.This behavior also appears in the synthetic-arithmetic analysis.
  • 6.2 VERBALISING THE PROMPT: SOFT2HARD: 12.5 all-domain mAP: SOFT2HARD matches DetPO and exceeds GEPA after verbalizing soft-prompt-conditioned descriptions into a hard prompt.The describe-summarize recipe contributes 12.2 mAP, while the soft prompt adds 0.3 on average, concentrated in weak-backbone categories.
  • 6.2 VERBALISING THE PROMPT: SOFT2HARD: Soft-prompt-conditioned descriptions name finer distinguishing object features than the base model, supplying the descriptions that SOFT2HARD summarizes.The richer descriptions are presented as the source for the distilled prompt’s additional information.
  • 6.2 VERBALISING THE PROMPT: SOFT2HARD: The soft prompt’s average +0.3 mAP gain is uneven: it helps 10 of 20 Roboflow20-VL datasets by a mean of +1.9 mAP and hurts the other 10.The gain is concentrated where the frozen backbone is weakest, including Aerial (+4.1) and Documents (+2.6).
  • 6.3 TRANSFER ACROSS MODEL VERSIONS: +0.8 all-domain mAP: prompts trained on Qwen3-VL-8B improve the Qwen3.5-9B zero-shot baseline without projection or retraining, but lose 4.0 mAP relative to origin-model performance.Transfer beats baseline on 13 of 20 datasets and loses on 7, showing a genuine split rather than a uniform small effect.
  • 6.3 TRANSFER ACROSS MODEL VERSIONS: Raw embedding transfer beats the distilled hard prompt on 18 of 20 datasets by a mean of 2.3 mAP, showing that verbalization is a lossy transfer channel.The two exceptions are near-floor swaps below 4 mAP.

7 BEYOND DETECTION: PROMPTING A ROBOT POLICY

Soft prompting extends to a frozen vision-language-action policy when tokens reach the action expert’s gradient path. It improves two weak RoboCasa tasks but can harm an already competent task, making base competence an important boundary.

  • 7 BEYOND DETECTION: PROMPTING A ROBOT POLICY: 1.7% success: a prefix-only prompt fails on PickPlaceCounterToCabinet because the prefix output is discarded and lies outside the action loss’s direct path.Adding tokens to the action-expert input moves them into the gradient path and reaches 23.3% success.
  • 7 BEYOND DETECTION: PROMPTING A ROBOT POLICY: 30.0% to 21.7%: the same full-strength dual prompt harms OpenDrawer, a task the frozen base already partly solves, while LoRA variants improve it to 66.7–78.3%.The prompt still fits the demonstrations, indicating a learned global input edit rather than simple training failure.
  • 7 BEYOND DETECTION: PROMPTING A ROBOT POLICY: Three tasks with 20 trials per seed and a 10–30% seed spread on PickPlace limit this experiment to a mechanism result rather than a benchmark claim.The early-stop point was selected on a seed-0 sweep before the three-seed rerun.
  • 7 BEYOND DETECTION: PROMPTING A ROBOT POLICY: 23.3% and 31.7% success: dual prompts reaching the action expert lift two RoboCasa tasks from a 5.0% zero-shot floor, matching or beating coverage-matched LoRA.On PickPlaceCounterToCabinet, the dual prompt matches LoRA at 23.3%; on TurnOnSinkFaucet, it exceeds LoRA at 31.7% versus 21.7%.

8 DISCUSSION

Ten-shot adaptation can identify what to attend to but cannot compensate for missing domain knowledge. Soft prompting is therefore most useful as a lightweight diagnostic and adaptation step, with heavier adaptation or specialist models needed when performance remains at the floor.

  • Ten images provide tens of labelled boxes—enough to identify attention targets but not to estimate millions of weight updates.
  • r=64 is the accuracy peak for LORA, while r=128 declines, indicating capacity overshoot in the ten-shot regime.
  • ≤1.0 mAP across prompt- and weight-space adapters versus 35.3 for a fine-tuned specialist shows that ten-shot adaptation cannot conjure absent medical knowledge.The paper recommends treating a short soft-prompt sweep that fails to lift performance as evidence of a knowledge deficit.
  • A practical workflow is to sweep injection position and token length, use space-token initialization, and select models by task metric rather than loss.For VLA policies, the prompt should reach both the VLM prefix and action-expert input.
  • The study reports 20–28% seed spread, treats deltas below about 1 mAP as ties, and scopes its claims to tested shot counts and robotics tasks.

9 CONCLUSION

The paper concludes that carefully placed, semantically neutral soft prompts can match the best LORA accuracy with far fewer trainable parameters while preserving the frozen model’s capabilities. The same placement principle extends to a frozen robot policy, but the work remains bounded by its tested settings and risks shared with object detection.

  • One to three learned tokens match the best LORA on Roboflow20-VL with over 20,000× fewer trainable parameters and zero forgetting.The accuracy-matching LORA rank reduces NaturalBench VQA ability by 35%.
  • Soft prompts transfer across model versions and can be verbalized into editable prompts that match dedicated prompt search.
  • On a frozen robot policy, prompting succeeds when the tokens reach the module that produces the action.
  • The work aims to lower adaptation’s annotation and compute barriers while avoiding silent degradation of other deployed capabilities.Distilled prompts are readable text, providing a modest transparency benefit over opaque weight updates.
  • The paper identifies no method-specific risks beyond general object-detection risks, including surveillance applications.

C PER-DATASET CONFIGURATIONS

Per-dataset Roboflow20-VL configurations favor a fixed cross-modal injection position on most datasets, while text-dominant and single-class domains more often favor an alternative ordering. Selected prompts remain short, averaging 1.75 tokens.

  • Selected prompt lengths are L ∈ {1, 2, 3}, with mean 1.75 and 7,168 trainable parameters on average.
  • IST wins 14 of 20 datasets, while TSI wins the other 6, concentrated on single-class and text-dominant domains.

D GEMMA-4 GENERALISATION: SEARCH METHODOLOGY AND FULL RESULTS

The Gemma-4-12B experiment shows that the soft-prompt recipe generalizes beyond Qwen3-VL, but its per-domain search is less reliable on the unrelated backbone. The broader deployment analysis also emphasizes standard serving, compact checkpoints, and careful comparison boundaries.

  • Gemma-4 generalisation: On unrelated Gemma-4-12B-it, the recipe beats baseline on 5 of 8 above-floor datasets, but its per-domain search does not transfer holdout reliability as well as on Qwen3-VL.
  • Search reliability: The search-selected configuration is worse than the untuned default on three datasets, showing that holdout gains do not always predict test gains.
  • Serving: A trained soft prompt can be exported as new vocabulary rows and served by an unmodified high-throughput engine.
  • Storage: At r=64, the measured storage ratio is 23,041×, slightly below the raw parameter-count ratio because fixed serialization overhead matters for tiny prompt files.
  • Comparison boundaries: DetPO’s 17.5 mAP with VQA-score reranking uses one VLM call per class and is therefore not cost-comparable to SOFTPROMPT’s single-call multi-class protocol.

H CROSS-TASK GROUNDING: FULL REFCOCO RESULTS

The paper compares soft-prompt transfer with distilled descriptions across Roboflow20-VL super-categories and reports category-dependent gains and losses when moving from Qwen3-VL-8B to Qwen3.5-9B.

  • Low-rank LORA does not measurably forget grounding, while the cross-task comparison includes RefCOCO acc@0.50 changes relative to the frozen backbone.
  • Cross-model transfer helps most on Sports (+5.3) and Documents (+3.3), but hurts on Industrial (−2.9).
  • The transferred prompt is evaluated from Qwen3-VL-8B to Qwen3.5-9B without retraining or projection.

J DISTILLATION VS. DIRECT TRANSFER: PER-DATASET RESULTS

Direct embedding transfer generally outperforms distilled descriptions, while nearest-token substitution preserves useful signal more reliably in text space than in visual-patch space.

  • Every dataset with a meaningful gap favours direct transfer, with gains reaching +9.9 on lacrosse-object-detection.
  • Token substitution: Text substitution retains roughly 33% of the trained prompt’s aggregate gain across the six datasets where it remains above baseline.
  • Direct embedding transfer beats its distilled description on 18 of 20 datasets.
  • Token substitution: Nearest-text-token substitution stays above the frozen baseline on 6 of 8 above-floor datasets, whereas nearest-patch substitution is weaker and collapses generation on three datasets.
  • Contextual verbalisation: In a non-visual companion task, isolated soft tokens are not self-describing, but preserving their training context enables natural-language descriptions that are often exactly correct.
  • Robustness: The seed-noise analysis reports a 28% mean coefficient of variation across eight fully collected above-floor datasets, so sub-1 mAP deltas should be treated as ties.

M.1 THE SPREAD IS TRAINING NOISE, NOT EVALUATION NOISE

Evaluation-seed variation is small, while soft-prompt training shows substantial initialization-dependent variability that exceeds the corresponding LORA spread on most datasets.

  • Fixed-checkpoint evaluation across three generation seeds changes test mAP by at most 0.51 points, versus 12–43% training-seed CVs.
  • The dominant variability comes from different soft-prompt initializations reaching different local optima rather than from test-time measurement noise.
  • SOFTPROMPT’s training-seed spread averages 7.9× LORA’s at the accuracy-matching rank across eight above-floor datasets, with LORA more stable on six.
  • Qualitative detections compare the frozen baseline, DetPO, LoRA, and SoftPrompt across seven Roboflow20-VL domains.
Loading 2609.11310v1…