PaperlayerView exact revision on arXiv

Source-linked AI summary

Contextualized Visual Personalization in Vision-Language Models

Yeongtak Oh, Sangwon Yu, Junsung Park, Han Cheol Moon, Jisoo Mok, Sungroh Yoon

arXiv:2602.03454v3cs.CV

TL;DR

Existing VLMs struggle to use users’ accumulated visual-textual experiences when interpreting new images, motivating contextualized visual personalization. CoViP addresses this gap with personalized captioning, RL-based post-training, caption-augmented generation, and diagnostics that test genuine visual-context use. Experiments report improved captioning and holistic gains across downstream personalization tasks, supporting CoViP as a foundational stage for robust and generalizable personalization.

  • Problem

    Existing VLMs often fail to personalize visual understanding from user-specific context, especially rich experience-level histories combining past images and textual information.

  • Method

    CoViP formulates contextualized visual personalization as personalized image captioning and combines RL-based post-training with caption-augmented generation and diagnostic evaluations.

  • Results

    CoViP improves personalized image captioning and consistently improves personalization across the benchmark and diagnostic downstream tasks.

  • Takeaways & Limitations

    CoViP serves as a foundational stage for contextualized visual personalization, enhancing the robustness and generalizability of existing VLMs.

  • Takeaways & Limitations

    The benchmark relies on synthetic dialogues and generated images that may contain factual inconsistencies or visual artifacts.

Abstract

from arXiv · show

Despite recent progress in vision-language models (VLMs), existing approaches often fail to generate personalized responses based on the user's specific experiences, as they lack the ability to associate visual inputs with a user's accumulated visual-textual context. We newly formalize this challenge as contextualized visual personalization, which requires the visual recognition and textual retrieval of personalized visual experiences by VLMs when interpreting new images. To address this issue, we propose CoViP, a unified framework that treats personalized image captioning as a core task for contextualized visual personalization and improves this capability through reinforcement-learning-based post-training and caption-augmented generation. We further introduce diagnostic evaluations that explicitly rule out textual shortcut solutions and verify whether VLMs truly leverage visual context. Extensive experiments demonstrate that existing open-source and proprietary VLMs exhibit substantial limitations, while CoViP not only improves personalized image captioning but also yields holistic gains across downstream personalization tasks. These results highlight CoViP as a crucial stage for enabling robust and generalizable contextualized visual personalization.

1. Introduction

The paper formalizes contextualized visual personalization as using a user’s multimodal visual history to interpret new images, and proposes CoViP to learn this process through personalized captioning. Its diagnostics and experiments show improved personalization across tasks, with retrieval identified as a key bottleneck.

  • Existing VLMs often recognize visible attributes but fail to connect new images with user-specific experiences such as a previously mentioned person.
  • Contextualized visual personalization uses a history combining previously observed images with associated personal textual information to interpret new visual input.
  • Personalized image captioning serves as CoViP’s proxy task for learning the shared process of grounding visual inputs in user-specific context.
  • CoViP combines a personalized image-captioning benchmark, reinforcement-learning-based post-training, and caption-augmented generation.
  • Diagnostic tasks test whether models recognize and retrieve personalized visual context rather than exploit textual shortcuts, spanning reactive to proactive personalization.
  • CoViP consistently improves personalized captioning and downstream personalization, while its gains are driven primarily by better retrieval rather than recognition.

2. Previous Works on VLM Personalization

Prior VLM personalization methods mainly address explicit concepts or limited settings, whereas this paper studies implicit personalization grounded in interleaved multimodal experience. It positions CoViP as a post-training approach for deeper contextual reasoning and personalized generation.

  • Early methods rely on off-the-shelf VLMs and external databases to handle explicitly defined concepts.
  • Recent methods add retrieval, personalized generation, multi-concept captioning, or recognition of repeated objects across query images and context.
  • The paper excludes methods lacking long in-context personalization support or relying on external-VLM-controlled memory from direct comparisons.
  • Unlike prior work focused on surface attributes, this study targets implicit personalization in interleaved multimodal contexts requiring visual experience and contextual reasoning.

3. CoViP: Contextualized Visual Personalization via Image Captioning

CoViP formalizes contextualized visual personalization as jointly reasoning over new images and user-specific interaction histories, using personalized image captioning to learn this shared process. It introduces a benchmark, RL-based post-training, caption-based evaluation, and Caption-Augmented Generation.

  • Problem Definition: Contextualized visual personalization requires generating responses by jointly reasoning over a query image and user-specific visual-textual interaction history.The context contains accumulated user experiences, and the response should reflect them rather than remain generic.
  • Unified Framework: CoViP models this capability through personalized image captioning, which externalizes a latent personalization state shared across downstream tasks.The framework focuses on learning the shared contextual visual encoder rather than separately optimizing every downstream output.
  • Evaluation Protocol: CapEval-QAs probes whether generated captions include relevant contextual information while excluding irrelevant details.Positive Accuracy measures capturing relevant information, whereas Negative Accuracy measures avoiding irrelevant contextual details.
  • Training and Inference: The framework adds RL-based post-training for faithful captions conditioned on the query image, context, and captioning prompt, plus Caption-Augmented Generation at inference.The captioning policy is optimized using an expected verifiable reward.

4. Diagnostic Evaluation of General Personalization Capability

The diagnostic evaluation tests whether models recognize people visually and retrieve the correct personalized experiences from contextual history. Its tasks cover temporal retrieval, fine-grained action recall, and proactive responses triggered by implicit instructions.

  • Evaluation Design: The diagnostic suite evaluates visual grounding and personalized experience retrieval while explicitly disallowing textual shortcut behaviors.It is designed to assess personalized reasoning grounded in visual identity.
  • Last-Seen Detection: Last-Seen Detection requires identifying a person in a query image and retrieving the most recent encounter from multiple contextual interactions.The task requires temporal reasoning over relevant entries rather than partial matches or surface-level textual cues.
  • Last-Action Recall: Last-Action Recall extends person and encounter retrieval by requiring the model to recover the fine-grained action described in the latest interaction.It combines temporal localization with extraction of what the user was doing.
  • Instruction-Triggered Recall: Instruction-Triggered Recall evaluates proactive personalization by requiring a model to recall and apply a past notification instruction when the person reappears.The keyword must be incorporated even though the current user turn does not explicitly request it.

5. Experiments

Experiments evaluate CoViP against proprietary, open-source, and post-training baselines on personalized captioning and downstream diagnostic tasks. CoViP consistently improves personalization, while CAG provides gains that depend on caption quality and task difficulty.

  • Benchmark evaluation: CoViP shows superior performance on the personalized image captioning benchmark relative to the base VLM.Table 2 reports CapEval-QAs performance and defines Δ as the gain relative to the base VLM.
  • Benchmark evaluation: Existing open-source and proprietary VLMs have limited ability to generate context-grounded captions, while Gemini-3.0-Pro is strongest among proprietary models.Existing post-training personalization baselines provide limited improvements on the benchmark.
  • Downstream tasks: CoViP consistently delivers stable and notable improvements over the base VLM across downstream ITR, LSD, and LAR diagnostic tasks.Gemini-3.0-Pro and GPT-5 lead different individual tasks, whereas RePIC shows marginal or no improvement across tasks.
  • Downstream tasks: CoViP improves personalization across all downstream tasks, indicating that personalized captioning post-training strengthens context-grounded caption generation.The reported gains extend beyond the captioning benchmark to the proposed diagnostic tasks.
  • Caption-augmented generation: CAG yields consistent gains with CoViP, but generic captions can reduce ITR recall below 20% because they miss triggering keywords.CoViP’s granular contextual captions enable it to benefit from CAG, unlike generic captions.
  • Diagnostic generalization: RAP and RePIC show limited generalization on diagnostic tasks compared with CoViP.These baselines also show limited improvements on the personalized captioning benchmark.

6. Discussions

The discussions identify retrieval, rather than recognition, as CoViP’s main source of improvement and examine evaluation fidelity and generalization. They also show that caption quality preserves the ability to describe new images while improving contextual grounding.

  • Why captioning precedes downstream tasks: Proprietary VLMs and CAG show unstable, task-dependent downstream behavior when personalized image captioning is unreliable.The discussion motivates personalized-captioning post-training before downstream inference for more robust diagnostic improvements.
  • Recognition and retrieval: CoViP’s gains arise primarily from improved retrieval and integration of implicit personal cues, not improved visual recognition.Recognition F1 increases moderately, while MCQA accuracy improves by a substantially larger margin at comparable F1 levels.
  • Human evaluation: CapEval-QAs-preferred captions obtain higher win-or-tie rates in human evaluation across both evaluation criteria.This supports alignment between the QA-based metric and human judgment.
  • Human evaluation: CoViP has a clear contextual-grounding preference advantage over all baselines while describing new query images on par with Qwen3-VL-8B.The results do not show a significant gap in new-image description quality relative to other baselines.
  • Multi-image generalization: CoViP consistently improves over the base model across evaluated multi-image benchmarks for visual grounding and cross-image identity matching.These experiments assess generalization beyond the proposed benchmark.

7. Conclusions and Limitations

The paper concludes that CoViP combines a contextualized visual personalization paradigm, personalized-captioning post-training, and CAG to improve robustness and generalizability. Its benchmark remains limited by reliance on synthetic dialogues and generated images.

  • Conclusions: CoViP defines contextualized visual personalization as grounding visual understanding in user-specific past visual experiences to guide personalized outputs.The framework uses personalized image captioning post-training and caption-augmented generation.
  • Conclusions: Extended diagnostic studies indicate that CoViP enhances the robustness and generalizability of VLM personalization.The conclusion characterizes CoViP as a foundational stage for contextualized visual personalization.
  • Limitations: The benchmark relies on synthetic dialogues and generated images that may contain factual inconsistencies or visual artifacts.Model-based quality filtering was applied, but additional human verification could improve dialogue factuality and image fidelity.
  • Future directions: Future benchmarks should incorporate omnimodal and real-world user signals such as shopping histories, voice recordings, and long-term interaction logs.These signals are proposed to support more practical evaluation across diverse use cases.

Impact Statement

CoViP targets personalization grounded in visual experiences and historical context, using MCQA-based evaluation and RL post-training to assess and improve caption-level contextual grounding. Its design emphasizes selective retrieval, abstention, and sequence-level optimization.

  • Framework motivation: CoViP targets responses grounded in user-specific visual experiences and historical context through a personalized image captioning framework.The framework is intended to support contextualized personalization in VLMs.
  • Evaluation protocol: MCQA evaluates whether generated captions provide evidence for contextual inferences while reducing sensitivity to surface-level lexical variation.Questions probe temporal cues, personalized descriptions, and user-specific experiences.
  • Evaluation algorithm: The caption-based probing algorithm samples personalized captions, judges MCQA answers, and reports positive accuracy, negative accuracy, and verifiable caption rewards.The procedure uses a personalization model and judge LLM over query images, interleaved context, and concept-indexed QA banks.
  • Evaluation protocol: Each dialogue produces MCQA items with one context-grounded answer, two plausible distractors, and a “cannot be determined” option.The construction tests contextual use, unsupported detail generation, and appropriate abstention.
  • Evaluation metrics: Positive Accuracy measures relevant contextual retrieval, while Negative Accuracy measures avoidance of irrelevant or misleading content.Together they assess selective recall and selective omission of contextual information.
  • Post-training optimization: GSPO applies sequence-level clipped importance weighting aligned with whole-caption rewards, improving stability for long-form generation.Length normalization keeps importance weights comparable across caption lengths.
  • Post-training optimization: Degeneration filtering penalizes repetitive or overly long generations to prevent reward hacking.The filter uses sentence-, 5-gram-, and chunk-level duplication ratios.

D. Experimental Configurations

The experiments use Qwen3-VL-based deterministic generation and diagnostic tasks requiring visual identity recognition, contextual retrieval, and temporal reasoning. The benchmark combines constructed visual contexts with filtered synthetic query images and real-image evaluation components.

  • Experimental setup: Qwen3-VL-30B-A3B-Instruct-FP8 generates dialogues, while Qwen3-30B-A3B-Instruct-2507-FP8 generates MCQA pairs and judges VR accuracy.Decoding temperature is fixed at 0.0 across experiments.
  • Diagnostic tasks: The diagnostic suite requires recognizing the query person, retrieving associated dialogues, and identifying the most recent interaction.This design prevents solutions based only on text matching or partial context retrieval.
  • Diagnostic tasks: LSD asks where the person was last seen, using word-level F1 between the response and ground-truth location.The model must identify the person, retrieve all associated dialogues, compare timestamps, and extract the latest location.
  • Diagnostic tasks: LAR tests recall of a fine-grained personal action from the most recent dialogue, evaluated by an LLM judge for semantic agreement.It extends LSD from factual location recall to episode-level user-state retrieval.
  • Diagnostic tasks: ITR tests proactive personalization by requiring a specified trigger keyword in response to a generic query, measured by trigger success rate.The keyword is SKS, and the user does not explicitly mention it at inference time.
  • Data construction: Benchmark construction combines human, object, and pet images from existing datasets or royalty-free web sources, alongside synthetic query-image generation and quality filtering.Images with four concepts have an approximately 45% prompt-faithfulness failure rate, motivating stricter filtering for three- and four-concept samples.

E. Further Limitations

The benchmark deliberately simplifies real-world contextualized visual personalization by relying primarily on synthetic data. Its limitations include restricted real-world data access, possible biases, noisy memory conditions, and limited open-ended visual robustness.

  • Data limitations: Real-world evaluation data remain difficult to incorporate because privacy concerns and unreliable human annotations constrain current datasets and benchmarks.The authors identify medical logs as one example of privacy-sensitive data.
  • Data limitations: Synthetic data are used to isolate the core capability in a controlled setting because real-world contexts require heterogeneous-memory retrieval and multi-hop reasoning.The authors describe this as a deliberate simplification of the problem setting.
  • Data limitations: The benchmark relies primarily on synthetic data, leaving alignment with real-world data distributions as a target for future work.The paper presents the formalization as a foundation for subsequent research.
  • Textual robustness: Real-world in-context memories may be less structured, conflicting, noisier, or more ambiguous than the benchmark contexts.The authors conjecture that CoViP may fail under such hard conditions, while noting that the extent is difficult to quantify.
  • Visual robustness: The benchmark covers relatively fine-grained perception within context/query setups but does not extend to open-ended real-world visual robustness.Visually challenging cases and imperfect perception examples are included in supplementary evaluations.

F. Additional Analysis

Additional analyses examine optimization stability, metric interpretation, and transfer beyond personalized captioning. They show that GSPO with degeneration filtering is stable, while CoViP’s gains extend to broader multi-image visual retrieval and grounding.

  • Training dynamics: GSPO achieves the best performance across all tested concepts among GRPO, DrGRPO, and GSPO, motivating its use as the default on-policy RL algorithm.The comparison is reported in Table S.2.
  • Training dynamics: Degeneration filtering stabilizes GSPO training after naive GSPO causes accumulated rewards to decline during later training stages.The filtering is based on n-gram and chunking criteria.
  • Training dynamics: GRPO converges more slowly, DrGRPO tends to diverge early, and GSPO provides faster convergence with substantially higher rewards and stable post-training.The distinction is associated with KL-divergence regularization and optimization stability.
  • Prompt design: Randomly alternating prompt roles causes completion length to diverge, whereas fixed captioning prompts preserve instruction-following behavior.The authors associate random role switching with indiscriminate incorporation of relevant and irrelevant dialogues.
  • Metric interpretation: CoViP’s higher Acc+ and slightly lower Acc− reflect stronger fine-grained contextual retrieval, not weaker reasoning ability.The baseline often answers “cannot determine,” which counts as correct under Acc− and can create false positives.
  • Transfer analysis: CoViP improves long-context retrieval and multi-image grounding benchmarks, suggesting broader gains in multi-image visual perception beyond personalization.The reported benchmarks include MM-NIAH, MMNeedle, MuirBench, and MMIU.

G. Additional Results

Further experiments analyze benchmark transfer, reward design, prompt generalization, and caption preservation. CoViP’s reinforcement-learning post-training improves downstream generalization while retaining detailed captioning quality and hallucination performance.

  • Benchmark transfer: CoViP shows comparable or superior performance on RePIC without relying on the VRs introduced by that benchmark.The reproduced Qwen3-VL baseline was not extensively hyperparameter-optimized and may therefore be slightly weaker than the original report.
  • Reward ablations: Using only visual or only caption rewards weakens performance, showing that personalized captioning benefits from joint visual and retrieval supervision.Visual supervision alone can fall below the Qwen3-VL-8B baseline, while retrieval-only signals are also consistently weaker.
  • Reward ablations: Replacing binary object-consistency reward with set-based F1 visual reward consistently improves positive accuracy through denser multi-concept supervision.The proposed reward balances precision and recall in discriminative visual grounding rather than directly optimizing localization.
  • Reward ablations: Using all four positive concepts during post-training yields the best performance among the tested concept counts.This result is reported in the VR ablation analysis.
  • Training stability: N-gram and chunking-based degeneration filtering improves positive accuracy, especially for multi-concept settings, by steering optimization toward meaningful contextualized generations.The filtering is presented as an alternative to naively applying GSPO.
  • Prompt generalization: CoViP generalizes across prompt-role changes, while removing detailed task descriptions causes a significant performance drop.The captioning user prompt encourages context-grounded responses without dependence on one fixed template.
  • Caption quality: CoViP preserves general captioning quality: detailed captioning remains comparable apart from a minor BLEU drop, and hallucination metrics show no notable degradation.MMHal remains on par with baseline, while POPE metrics and CHAIR scores remain largely consistent.
  • RL versus SFT: SFT with distilled high-quality captions achieves comparable benchmark performance but fails to generalize to downstream tasks, supporting RL as essential for generalizability.The authors note that these SFT results were not exhaustively tuned.

H. Specifications on Evaluation Settings

The evaluation specifications describe a preference-based human study comparing method and baseline captions for context groundedness and new-image description quality, alongside benchmark tables and prompt templates for evaluation and task construction.

  • Evaluation results: The supplementary specifications include CapEval-QAs performance and downstream-task recall tables.These are presented as benchmark and downstream evaluation summaries.
  • Human evaluation: 21 participants produced 276 judgments by comparing method-generated and baseline VLM captions on context groundedness and new image description quality.Each participant completed approximately 10–15 evaluation tasks.
  • Human evaluation: The human-evaluation materials use representative multi-concept samples with interleaved image–text contexts to assess contextual integration and new-image description.The evaluation is preference-based and considers both context groundedness and new image description quality.
  • Prompt specifications: Prompt templates cover benchmark quality filtering, image captioning, dialogue and MCQA construction, answer extraction, judging, diagnostic tasks, and LAR evaluation.The listed materials include prompts for training and inference, benchmark construction, diagnostic downstream tasks, and LAR samples and evaluation.
  • Prompt specifications: Additional prompt visualizations specify judge-LLM answer extraction, diagnostic dialogue and evaluation, current-action candidates for LAR samples, and LAR judging.These materials are documented in Tables S.17–S.21.
Loading 2602.03454v3…