PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Grounding First, Multimodal In-context Learning Follows

Minhyuk Seo, Minjae Lee, Chaeeun Lee, Wei Lin, Muhammad Jehanzeb Mirza, Tinne Tuytelaars, Jonghyun Choi

arXiv:2610.30592v1cs.CVcs.AI

TL;DR

Multimodal ICL remains challenging because models may over-rely on textual cues rather than visual evidence in demonstrations. ProCoRe trains visual grounding with proposition-level contrastive rewards on image pairs, without ICL-formatted examples. It improves performance across ICL benchmarks and introduces PICL to evaluate grounded use of personalized demonstrations.

  • Problem

    Multimodal ICL is challenging because MLLMs may over-rely on textual cues instead of binding demonstration images to outputs and transferring that association to queries.

    Source: §1 · p. 2

  • Method

    ProCoRe generates captions for contrastive image pairs, decomposes them into propositions, and uses contrastive reinforcement rewards to align propositions with corresponding images and discourage mismatches.

    Source: §1 · pp. 2–3

  • Results

    ProCoRe improves performance across ICL benchmarks beyond ICL-data-dependent methods’ training data, alongside stronger target-region attention, reduced hallucinations, and less forgetting on standard VQA.

    Source: §6 · p. 10

  • Takeaways & Limitations

    PICL uses personalized concept names to test whether MLLMs leverage multimodal demonstrations rather than rely on pre-trained semantic priors.

    Source: §1 · p. 3

    ProCoRe relies on curated contrastive benchmarks, which may limit scalability and diversity, while PICL evaluates captioning rather than more complex reasoning tasks.

    Source: §A.18 · p. 30

Abstract

from arXiv · show

LLMs have shown strong in-context learning (ICL) capability, but extending it to Multimodal LLMs (MLLMs) remains challenging. Prior multimodal ICL methods often rely on ICL-specific datasets for additional training or task vector extraction, which can improve performance on the same ICL benchmark used for adaptation. However, they do not generalize well to other benchmarks and often induce forgetting in previously well-solved tasks. For this reason, we identify 'visual grounding' as the key bottleneck in multimodal ICL; MLLMs often fail to attend to task-relevant visual evidence in demonstrations, instead over-relying on textual cues and producing hallucinated outputs. To address this, we propose ProCoRe to enhance visual grounding through contrastive reinforcement learning on multimodal contrastive data for improving multimodal ICL. ProCoRe generates captions for contrastive image pairs, decomposes them into propositions, and optimizes contrastive rewards that encourage alignment with corresponding images while discouraging mismatched ones. Despite never seeing ICL-formatted training examples, ProCoRe improves average multimodal ICL classification accuracy by 5.6% and captioning ROUGE-L by 13.8% over the off-the-shelf Qwen-VL-3-8B model, outperforming ICL-data-dependent state of the arts across benchmarks. Finally, we introduce PICL, a personalized multimodal ICL captioning benchmark that tests whether MLLMs genuinely understand visual demonstrations rather than relying on pre-trained semantic priors.

1 Introduction

Multimodal ICL is difficult because models may rely on textual cues instead of grounding demonstrations in task-relevant visual evidence. The paper identifies weak visual grounding as a bottleneck, proposes ProCoRe to address it without ICL-formatted training data, and introduces PICL to test grounded use of personalized demonstrations.

1 Introduction

  • Multimodal ICL requires binding demonstration images to textual outputs and transferring those visual–linguistic associations to the query.Models often over-rely on textual cues and mimic demonstration text rather than fully using visual information.Source: §1 · p. 2
  • The paper frames weak visual grounding as a root issue beyond methods that adapt models to ICL-formatted data.Prior retrieval and ordering methods are sensitive to prompt design and sample order, while ICL-data-dependent methods can degrade on other benchmarks.Source: §1 · p. 2
  • Weak visual grounding contributes to hallucinations, especially for personalized identifiers that cannot be reliably inferred from pre-trained priors.The paper contrasts personalized references such as a dog named “John” with generic concepts such as “dog.”Source: §1 · p. 2
  • Prior ICL data-dependent methods generalize poorly across benchmarks and can cause forgetting, while ProCoRe outperforms them without ICL data.The Figure 2 caption contrasts performance on matching versus mismatched training and test ICL datasets.Source: §1 · p. 2, Figure 2 · p. 3
  • ProCoRe strengthens visual grounding through proposition-level contrastive rewards on image pairs, without relying on ICL datasets.It generates captions, decomposes them into propositions, and rewards alignment with corresponding images while discouraging alignment with mismatched images.Source: §1 · pp. 2–3
  • PICL tests whether models use multimodal demonstrations by requiring captions to identify personalized concept names rather than common concepts.This design aims to distinguish genuine use of demonstrations from reliance on pre-trained semantic priors.Source: §1 · p. 3

2 Related Work

Prior multimodal ICL work improves demonstrations or adapts models using ICL-formatted data, but these approaches face sensitivity, generalization, or forgetting concerns. ProCoRe instead targets visual grounding with contrastive rewards.

Multimodal In-context Learning

  • Retrieval- and ordering-based methods select demonstrations similar to the query, but multimodal ICL remains sensitive to prompt design, sample order, and cross-modal similarity estimates.These methods can underperform random selection.Source: §2 · p. 3
  • ICL training-based methods can improve performance on their adaptation datasets but often generalize poorly to unseen ICL benchmarks and may forget previously well-performing tasks.Task-vector methods also show limited generalization to unseen benchmarks and may lose fine-grained visual information needed for complex tasks.Source: §2 · pp. 3–4

Visual Grounding and Hallucination in MLLMs

  • Hallucination-mitigation methods may require large-scale instruction data and add parameter or computational overhead, while directly amplifying visual-token norms can increase irrelevant attention or distort representations.ProCoRe avoids modifying token norms and instead uses contrastive rewards to promote discriminative visual cues.Source: §2 · p. 4

3 Approach

ProCoRe targets weak visual grounding in multimodal ICL by training on visually contrastive image pairs rather than ICL-formatted data. Its proposition-level rewards encourage captions to capture both image-specific distinctions and shared visual details.

3 Approach

  • ProCoRe improves multimodal ICL by strengthening grounding in task-relevant visual evidence rather than relying on ICL-specific training data.Its contrastive rewards are designed to encourage fine-grained visual discrimination.Source: §3 · p. 4

3.1 Multimodal Contrastive Training Pairs

  • Training pairs contain two visually contrastive image-caption pairs, and the approach is compatible with multimodal contrastive datasets.This work uses InpaintCOCO and EQBEN.Source: §3.1 · p. 4

3.2 Contrastive Reward Function

  • Sentence-level image-text rewards favor captions matching their corresponding images over mismatched images, while text-text rewards favor matching paired captions.The combined reward is optimized with GRPO.Source: §3.2 · p. 5
  • Sentence-level contrastive rewards may encourage shortcut captions that state only the distinguishing object and omit shared scene details.Shared content can increase similarity to both images and thereby lower the contrastive reward.Source: §3.2 · p. 5
  • ProCoRe decomposes captions into propositions unique to each caption and propositions shared by both, then applies contrastive rewards to image-specific propositions.The decomposition uses GPT-4o-mini, and proposition-level text reward uses entailment scores.Source: §3.2 · pp. 5–6, Figure 3 · p. 5
  • A separate shared-content reward encourages propositions shared by the captions to align with both images and both ground-truth captions.This design avoids penalizing content that is true of both images.Source: §3.2 · p. 6

4 PICL Benchmark

PICL evaluates whether models use visual demonstrations to learn personalized identifiers that cannot be inferred from a query image alone. It is built from visual storytelling datasets and tests increasingly complex relationships across demonstrations.

4 PICL Benchmark

  • PICL addresses a shortcut in standard benchmarks, where familiar identifiers may let models answer from pre-trained priors without using demonstrations.Its personalized identifiers require grounding the identifier in demonstrations and transferring that association to the query.Source: §4 · p. 6
  • PICL uses PororoSV, FlintstonesSV, and StoryStream frames, replacing original caption identifiers with randomly sampled personalized names.The construction includes 1,000 training samples and 100 test samples per dataset, yielding 3,300 samples per shot setting.Source: §4 · p. 6
  • Higher-shot settings require reasoning over increasingly complex relationships among demonstrations, so performance need not improve as shots increase.Table 1 compares multimodal ICL captioning benchmarks by dataset and shot.Source: §4 · p. 6, Table 1 · p. 7

5 Experiments

The experiments compare ProCoRe with multimodal ICL and hallucination-mitigation baselines across classification, captioning, and standard VQA, using Qwen-VL-3-8B as the main model. ProCoRe performs strongly across ICL tasks while improving visual grounding and avoiding the forgetting seen with ICL-training baselines.

5.1 Experimental Setups

  • The primary comparisons use Qwen-VL-3-8B, with accuracy for classification, ROUGE-L for captioning, TAS for grounding, and OpenCHAIR for hallucinations.The evaluation includes TrueMICL and ISEKAI classification, plus MS COCO and PICL captioning; additional evaluation spans other MLLMs and model sizes.Source: §5.1 · p. 7, §5.2 · p. 9

5.2 Quantitative Analysis

  • ProCoRe significantly outperforms baselines on multimodal ICL classification across shot settings and on MS COCO and PICL captioning, despite using no ICL-formatted training data.The classification results also show stronger generalization than ICL data-dependent baselines.Source: §5.2 · pp. 7, 9, Table 2 · p. 8
  • ProCoRe achieves the highest target attention score and lower OpenCHAIR scores than baselines, accompanying stronger task-relevant grounding and fewer hallucinations.Hallucination-mitigation baselines show only marginal gains and often underperform the off-the-shelf model.Source: §5.2 · pp. 8–9, Figure 6 · p. 8, Figure 7 · p. 9
  • ProCoRe improves standard VQA performance, while ICL-training baselines often degrade on previously well-solved tasks.The cited examples are IconQA and TextVQA; the passage attributes degradation to forgetting.Source: Figure 5 · p. 8, §5.2 · p. 9
  • Ablations indicate that ProCoRe’s gains come from proposition-level contrastive rewards, not contrastive-pair training alone, and exceed sentence-level reward results.Contrastive-pair supervised fine-tuning produces only marginal gains.Source: §5.2 · p. 9, Table 3 · p. 9

6 Conclusions

The conclusion argues that stronger visual grounding can elicit multimodal ICL without explicit ICL training. ProCoRe uses proposition-level rewards on visually contrastive pairs and is reported to improve ICL performance, grounding, hallucination, and standard VQA outcomes.

6 Conclusions

  • ProCoRe improves multimodal ICL performance and generalizes beyond ICL-data-dependent methods’ training data, alongside stronger target-region attention, reduced hallucinations, and less forgetting on standard VQA.The paper frames these outcomes as support for stronger visual grounding eliciting multimodal ICL without explicit ICL training.Source: §6 · p. 10

A.1 Additional Analyses on Multimodal ICL Captioning Tasks

The additional analyses assess personalized identifier accuracy and validate its automatic metric against human judgments. They also note that ROUGE-L can remain high when a personalized identifier is wrong, because generic visual description content still contributes to the score.

A.1 Additional Analyses on Multimodal ICL Captioning Tasks

  • PICL can yield high zero-shot ROUGE-L even when a personalized identifier is incorrect, because generic visual descriptions can still score well.The authors retain ROUGE-L because it evaluates overall visual-description quality as well as identifier use.Source: §A.1 · p. 16
  • The identifier metric checks whether predicted captions correctly assign and use character names from the ground-truth caption.It treats wrong or swapped names, missing characters, and unsupported extra names as incorrect.Source: §A.1 · p. 16
  • ProCoRe achieves the highest personalized identifier accuracy across all shots, alongside its ROUGE-L gains.The authors interpret this as better grounding of personalized identifiers and more accurate descriptions of target entities and other visual details.Source: §A.1 · p. 16, Table 5 · p. 17
  • The GPT-4o-mini identifier-accuracy metric shows high agreement with majority human judgments across agreement, Cohen’s κ, precision, recall, and F1.The study compares the metric and humans on 30 sampled query examples judged by 50 annotators, using captions without query images.Source: §A.1 · pp. 16–17

A.2 Details of Experimental Setups

The setup defines measures for grounding and hallucination, specifies reward weights and training choices, and presents a reward-weight analysis. The provided table caption identifies a human-agreement study for personalized identifier accuracy.

A.2 Details of Experimental Setups

  • Table 4 is a human-agreement study on personalized identifier accuracy.The accompanying study compares GPT-4o-mini-based metric judgments with majority human judgments.Source: Table 4 · p. 17
  • Target attention score averages attention from generated tokens to image tokens in the associated target region across attention heads; higher TAS indicates stronger region-token grounding.Target regions are obtained from SAM3 segmentation maps.Source: §A.2 · p. 17
  • The reward-weight analysis reports strong performance at moderate ratios such as λI/λT = 2, with λs = 4, while overly weighting either visual or text reward weakens performance.The analysis evaluates MS COCO captioning ICL and uses InpaintCOCO for contrastive RL training.Source: Figure 8 · p. 18
  • OpenCHAIR measures the proportion of hallucinated words in predicted captions by checking support in the ground-truth caption, accounting for synonyms through LLMs.Lower OpenCHAIR scores indicate fewer object hallucinations.Source: §A.2 · p. 18
  • ProCoRe uses λI = 2, λT = 1, and λs = 4, with batch size 16 and LoRA rank 128 for trainable baselines.ProCoRe training uses 1,000 samples each from InpaintCOCO and EQBEN; reported results are single runs.Source: §A.2 · p. 18

A.3 Hyperparameter Analysis

The analysis varies relative reward weights, finding that overly text-heavy or image-heavy weighting hurts performance and selecting a moderate image-favoring ratio.

A.3 Hyperparameter Analysis

  • The authors vary λI and λT relative to fixed λs because reward-term ratios, rather than absolute coefficients, are the focus.Source: §A.3 · p. 18
  • When λT > λI, performance falls, while λI = 4 and λT = 1 also reduces performance by underusing ground-truth details.Source: §A.3 · p. 18
  • Across experiments, they choose λI = 2, λT = 1, and λs = 4, balancing visual grounding against overemphasis.Source: §A.3 · p. 18

A.4 Additional Details and Analysis of Proposition Decomposition

Proposition decomposition runs online during reinforcement-learning training using generated and ground-truth captions. It adds modest overhead, and smaller open-source decomposers achieve comparable performance.

A.4 Additional Details and Analysis of Proposition Decomposition

  • At each GRPO step, the MLLM generates captions for a contrastive image pair, which GPT-4o-mini decomposes alongside the corresponding ground-truth captions.Source: §A.4 · p. 19
  • The text-only API calls cost approximately 4 × 10^-4 USD per training sample and increase wall-clock training time by 5.32%.Calls can run in parallel with GRPO rollout generation using multiple threads.Source: §A.4 · p. 19
  • Qwen3-1.7B and LLaMA3.2-3B achieve comparable performance to GPT-4o-mini as proposition decomposers on PICL and TrueMICL.The authors attribute gains to proposition decomposition rather than GPT-4o-mini's capacity.Source: §A.4 · p. 19, Table 6 · p. 19, Table 7 · p. 19

A.5 Detailed ablation study of ProCoRe’s contrastive reward

Together, the supplemental analyses examine ProCoRe’s reward components and multimodal ICL gains across benchmarks and model scales, alongside PICL’s visual-grounding evaluation and comparisons with ICL-data-dependent adaptation. They also report robustness across random seeds, grounding of personalized identifiers, scalability, evaluation, and bias-related limitations.

A.5 Detailed ablation study of ProCoRe’s contrastive reward

  • Combining image and text contrastive rewards improves multimodal ICL more than either reward alone; shared-content rewards help, while sentence-level supervision underperforms proposition-level decomposition.The ablations suggest that shared-content rewards encourage attention to attributes common to both images, while proposition-level supervision provides finer-grained contrastive guidance.Source: §A.5 · p. 19

A.6 Additional Experiments on Multimodal ICL Classification Tasks

  • Without ICL data, ProCoRe matches or exceeds ICL-data-dependent baselines on their training benchmarks and significantly outperforms them on unseen classification benchmarks.Those baselines can perform well on benchmarks used for training or task-vector extraction but degrade substantially on others.Source: §A.6 · p. 20

A.7 Quantitative Analysis across Diverse MLLMs

  • ProCoRe consistently outperforms baselines with LLaVA-OneVision-1.5-8B and InternVL3.5-8B, demonstrating applicability across these MLLMs.The reported comparisons are in Tables 11 and 12.Source: §A.7 · p. 20, Table 11 · p. 21, Table 12 · p. 22

A.10 Details of Multimodal Contrastive Training Pairs

  • Across InpaintCOCO, VisMin, Winoground, and EQBEN, ProCoRe’s performance is consistent across contrastive training benchmarks; the mixed set samples 500 pairs each from EQBEN and InpaintCOCO.EQBEN varies object behavior and count, while InpaintCOCO focuses on object identity, attributes, and relations.Source: §A.10 · pp. 23–24

A.11 Quantitative Analysis across Model Sizes

  • ProCoRe consistently improves over baselines with Qwen-VL-3-4B and InternVL-3.5-14B, extending its reported effectiveness beyond 8B models.The authors use InternVL-3.5-14B because Qwen-VL-3 is available only up to 8B.Source: §A.11 · p. 25, Table 13 · p. 25, Table 14 · p. 25

A.12 Extended Related Work

  • Unlike prior approaches that rely on demonstration selection or ICL-data-dependent adaptation, ProCoRe targets poor visual grounding without relying on ICL datasets.The related-work discussion notes that ICL-training and task-vector methods can generalize poorly to unseen benchmarks, and ICL-data training can cause forgetting on previously well-performing tasks.Source: §A.12 · p. 26

A.13 Additional Examples of PICL; A.14.1 Details of Benchmark Construction

  • PICL tests whether models use visual evidence across demonstrations: its personalized identifiers require linking characters across frames, while construction criteria guard against text-only shortcuts and ambiguous examples.The benchmark uses visual storytelling datasets, rewrites frame captions with character reference images, and selects demonstrations so query characters appear in examples and identities can be distinguished.Source: §A.13 · p. 27, §A.14.1 · pp. 28–29

A.15 Robustness across Random Seeds

  • ProCoRe significantly outperforms baselines across three random seeds, with gains substantially larger than observed standard deviations; deterministic ordering baselines show no variance.RICES and MMICES return identical results across seeds because their demonstrations and ordering are fixed by their ranking criteria.Source: §A.15 · p. 30

A.17 Qualitative Analysis of Visual Grounding

  • With personalized identifiers unseen during training, ProCoRe attends to relevant query-image regions and predicts the identifier, while ICL baselines fall back to generic or incorrect names.For generic concepts present in demonstrations, both baselines and ProCoRe attend to relevant visual regions; Figure 12 illustrates the personalized-identifier contrast.Source: §A.17 · p. 30, Figure 12 · p. 31

A.18 Limitations and Future Work

  • Curated contrastive benchmarks may limit scalability and visual-attribute diversity; generative data such as GenOL [48] could help, and PICL evaluation could extend beyond captioning to reasoning.ProCoRe shows consistent gains across datasets, but the authors call for evaluation on more complex multimodal tasks to better reflect real-world settings.Source: §A.18 · p. 30

A.19 Impact Statements

  • Post-training with contrastive rewards may amplify model bias and ethical misalignment inherited from training data, although addressing these risks is not the work’s primary focus.The authors state their commitment to taking necessary measures to address these risks.Source: §A.19 · p. 32

Related papers

Loading 2610.30592v1…