Source-linked AI summary
Training-Free VLM Personalization via Calibrated Residual Decoding
Jiaao Yu, Yujian Ma, Xianming Hu, Pengran Wang, Ang Li
TL;DR
Direct personalized prompting may mix genuine user evidence with generic model priors, making positive-profile confidence difficult to interpret. The paper introduces training-free calibrated residual decoding across positive, counterfactual, and empty profiles, with entropy-based residual weighting. Experiments report improved personalized multimodal understanding without fine-tuning, including consistent gains on identity-sensitive visual tasks.
Problem
Positive-profile prompting can mix personalized evidence with generic priors, so the model’s response alone does not reliably reveal whether personalization supports a high-scoring answer.
Method
The method anchors decoding on E+, estimates personalization residuals by comparing E+, E−, and E0, and modulates residual strength using normalized entropy.
Results
The method improves personalized multimodal understanding without fine-tuning, with consistent gains across text-profile, image-reference, and open-ended personalized QA settings.
Takeaways & Limitations
Entropy calibration reduces the influence of uncertain residual signals, while reliable residuals strengthen profile-specific personalization.
Abstract
from arXiv · showhide
Vision-language models can be personalized in a training-free manner by directly providing user profiles, preferences, or visual references at inference time, without updating model parameters. However, direct personalized prompting does not guarantee that the model will reliably exploit such evidence. The predictive distribution under the positive user profile often mixes two sources: personalized signals genuinely supported by the current profile, and the model's generic visual or linguistic priors. As a result, from the positive-profile response alone, it is difficult to determine whether a high-confidence answer is supported by the user profile or merely reflects the model's default preference. To address this problem, we propose a training-free calibrated residual decoding framework. Given the same image and question, we construct three evidence conditions: a positive profile , a counterfactual profile , and an empty profile . Our method keeps the prediction under as the anchored base, and explicitly estimates the marginal contribution of personalization from score differences across the three conditions. We further introduce normalized-entropy-based uncertainty calibration, allowing the strength of personalized enhancement to adapt to the reliability of the residual signal. Experiments on MMPB, YoLLaVA, and MyVLM show that the proposed method improves personalized multimodal understanding without fine-tuning, with consistent gains on identity-sensitive visual personalization tasks. Additional analysis shows that entropy calibration stabilizes residual decoding when the contrastive personalization signal is uncertain.
Introduction
Personalized VLMs must distinguish user-supported evidence from generic model priors, because positive-profile prompting alone cannot reliably separate them. The paper proposes training-free calibrated residual decoding using positive, counterfactual, and empty profiles, with entropy-based control of personalization strength.
- Personalized multimodal understanding aims to make responses reflect user preferences, identity cues, and visual experiences for the same image and question.
- Existing personalized VLM approaches commonly use additional training, user memory modules, or curated personalized data to adapt models.
- Positive-profile prompting mixes profile-supported signals with generic visual or linguistic priors, so its scores and confidence cannot reliably identify trusted personalization.
- The framework compares positive E+, counterfactual E−, and empty E0 evidence conditions to estimate personalized residuals while anchoring decoding on the positive-profile prediction.
- Entropy-based uncertainty calibration adaptively controls personalized enhancement according to residual reliability, and the framework is evaluated across multiple personalized multimodal benchmarks without fine-tuning.
Related Work
Prior personalized VLM work mainly learns or maintains user-specific evidence, whereas this paper studies how a frozen VLM should use already-provided evidence during decoding. Its framework compares three evidence conditions to estimate and calibrate personalization during inference.
- MyVLM and Yo’LLaVA learn personalized concept representations or latent visual tokens from user-provided examples, while other approaches address personalization through related adaptation mechanisms.
- The paper is complementary to prior methods because it studies decoding with already-provided evidence rather than how personalized evidence is learned, retrieved, or maintained.
- Inference-time adaptation includes prompting, retrieval, decoding-time control, and contrastive decoding for frozen language and multimodal models.
- The proposed decoding anchors on the positive-profile prediction and uses counterfactual and empty-profile conditions to estimate and calibrate personalization's marginal contribution.
Method
The method estimates profile-specific personalization by comparing positive, counterfactual, and empty-profile evidence while retaining the positive profile as the decoding anchor. An entropy-derived confidence coefficient adapts residual strength, with implementations for multiple-choice and open-ended decoding.
- Motivating Observation: 24.09% of the combined subset contains positive-prompting errors, and 84.26% of those errors are prior-aligned with the empty-profile prediction.This motivates comparing positive, empty, and counterfactual profiles to isolate personalized evidence.
- Personalized Residual from Reference Conditions: For each candidate, the method compares positive-profile support with counterfactual and empty-profile support to identify profile-specific signals.The counterfactual tests support from an alternative user, while the empty profile tests support from the non-personalized prior.
- Personalized Residual from Reference Conditions: The personalized residual adjusts, rather than replaces, the positive-profile prediction, keeping E+ as the base evidence.E− and E0 provide contrastive measurements of profile-specific support, with α and β controlling their relative strengths.
- Entropy-Calibrated Residual Decoding: The residual is converted into a candidate distribution, and normalized entropy produces γ ∈[0, 1] to calibrate residual strength.Concentrated residuals yield high confidence, whereas flat residuals yield low confidence.
- Entropy-Calibrated Residual Decoding: The final score combines the anchored positive-profile score with the reliability-weighted residual enhancement.Uncertain residuals receive less influence, reducing the risk of over-amplifying noisy contrasts.
- Instantiations: For multiple-choice QA, the method selects the option with maximum calibrated score; for open-ended QA, it applies the same procedure token by token.Open-ended decoding restricts entropy computation to the top-K positive-profile tokens, with K = 24 in the experiments.
Experiments
Experiments across three personalized multimodal benchmarks show that counterfactual references and entropy calibration improve training-free personalization, especially for identity-sensitive visual tasks and preference-oriented subtasks.
- Evaluation: The evaluation covers MMPB for text-profile preference and identity reasoning, plus YoLLaVA and MyVLM for image-reference personalized visual recognition.Macro accuracy is reported on MMPB, while YoLLaVA and MyVLM use weighted accuracy.
- Main results: Direct E+ prompting improves over no-profile decoding but remains incomplete because its scores mix profile-supported evidence with generic model priors.On Pref. Aware, E+ prompting reaches only 32.85, indicating weak separation between personalized and prior-driven responses.
- Reference ablations: E− Contrast consistently outperforms E0 Contrast across nearly all model–benchmark pairs, including 95.90 versus 89.23 on YoLLaVA with Qwen2.5-VL-3B.The reported difference is +6.67, supporting counterfactual profiles as a stronger reference than the absence of personalization.
- Main results: Entropy-calibrated decoding outperforms E+ prompting by +3.59 to +35.90 across YoLLaVA models and reaches 99% on two MyVLM models.On MMPB, it closely tracks or matches E− Contrast while conservatively modulating diffuse residuals.
- Fine-grained results: On MMPB preference subtasks, Pref. Aware improves from 32.85 to 43.00 (+10.15), while Pref. Incon. improves from 73.90 to 90.00 (+16.10).Recognition-oriented subtasks show smaller but consistent gains.
- Component analysis: The method combines positive-profile anchoring with contrastive residual evidence and uncertainty-aware modulation, while ablations assess reference conditions and entropy calibration.Token-level confidence is computed over the top-K=24 tokens for open-ended decoding; default α, β, and T are all 1.25.
- Open-ended generation: In open-ended personalized QA, token-level decoding achieves a 63.0% overall win rate and raises the average score from 3.54 to 4.70.Image-profile QA yields a 75.0% win rate and improves from 3.37 to 4.92, while text-profile QA yields 45.0% and improves from 3.80 to 4.38.
- Open-ended generation: Qualitative analysis shows selective token modulation: identity-specific tokens receive larger shifts, while low-confidence punctuation tokens are adjusted weakly.The method can also move the first-token decision from an incorrect E+ response toward a profile-supported answer.
Conclusion
The paper presents a training-free calibrated residual decoding framework that separates personalized evidence from generic model priors and adapts residual strength using entropy-based confidence. Across text-profile, image-reference, and open-ended personalized QA settings, it reports consistent gains without fine-tuning, while requiring additional inference cost.
- The framework anchors decoding on positive-profile predictions and estimates personalized evidence using counterfactual and empty-profile references.Entropy-based confidence calibrates the strength of the residual signal.
- Experiments across text-profile, image-reference, and open-ended personalized QA settings show consistent gains without model fine-tuning.
- Evaluating multiple evidence conditions increases inference cost, especially during token-level open-ended generation.Suggested future directions include caching, candidate pruning, and selective residual decoding at key decision steps.