Source-linked AI summary
Beyond Surface Imitation: Contrastive Modeling for Reasoning Path Alignment in Multimodal In-Context Learning
Mingbo Yang, Wenqiang Wang, Zhaolu Kang, Peng Chen, Yannan Chen, Sunshang Wang, Yan Xiao
TL;DR
Existing multimodal ICL can encourage surface imitation instead of aligning responses with reasoning paths grounded in the input’s multimodal evidence. COMIL uses contrastive demonstrations, response-conditioned retrieval, and a lightweight alignment controller to guide refinement, and experiments report consistent gains, especially on reasoning-intensive tasks and VQA.
Problem
Existing multimodal ICL often relies on surface imitation rather than grounding responses in query-relevant multimodal evidence, especially in complex tasks.
Method
COMIL contrasts suboptimal and better responses under the same input with a reasoning path, retrieves demonstrations conditioned on the current response, and uses a lightweight controller to guide refinement.
Results
COMIL consistently improves performance across multimodal tasks, with particularly clear gains on reasoning-intensive tasks; it reaches 81.9% accuracy on VQAv2 with Qwen3.5-9B.
Takeaways & Limitations
The framework moves multimodal ICL beyond final-output imitation by promoting reasoning path alignment during inference.
Takeaways & Limitations
COMIL adds inference cost through iterative refinement and is not the lowest-cost ICL method.
Abstract
from arXiv · showhide
In-context learning (ICL) is widely used in multimodal large language models (MLLMs) and achieves strong performance across a wide range of multimodal tasks. However, existing multimodal ICL methods often rely on surface level imitation of in-context demonstrations, making it difficult for MLLMs to align their responses with the reasoning path required by the given multimodal input. This limitation becomes more pronounced in complex multimodal tasks, thereby restricting further improvements in MLLM performance. To address this issue, we propose a new multimodal ICL framework that combines contrastive demonstration modeling with the self-refinement capability of MLLMs. Specifically, our framework reformulates each demonstration by explicitly contrasting a suboptimal response with a better response under the same input, together with a reasoning path that reveals how the response should be refined. This contrastive formulation makes the reasoning path toward the desired response more explicit and guides the MLLM beyond superficial imitation. Furthermore, because effective refinement depends on the current response, we introduce a response-conditioned retrieval mechanism to select demonstrations whose reasoning paths are more relevant to the current response. In addition, we use a lightweight alignment controller to predict response quality and determine whether further refinement is needed. Experiments on three types of multimodal tasks show that the proposed framework consistently improves MLLM performance, with particularly notable gains on visual question answering (VQA).
1 Introduction
Existing multimodal ICL can imitate superficial patterns instead of grounding responses in relevant visual evidence. COMIL addresses this by contrasting response quality and refinement paths, retrieving demonstrations conditioned on the current response, and controlling iterative refinement.
- Multimodal ICL adapts MLLMs to new tasks at inference time without parameter updates.
- Standard demonstrations often align final outputs while encouraging surface imitation and incorrect correlations from visually similar examples.This can produce correct output formats without fine-grained visual grounding.
- COMIL reformulates each demonstration as a suboptimal response, a better response, and a reasoning path explaining how relevant visual evidence supports refinement.The contrastive structure makes both the desired response and the refinement process explicit.
- Response-conditioned retrieval selects demonstrations using similarity between their suboptimal responses and the MLLM’s current response, beyond input similarity alone.The retrieved demonstrations therefore provide more relevant refinement guidance for the current response.
- 81.9% accuracy on VQAv2 is achieved by Qwen3.5-9B with COMIL, the best result among compared methods.The framework also improves Gemma-3-27B to 0.587 CIDEr on Flickr30k and remains sample efficient with limited retrieved demonstrations.
- COMIL combines contrastive demonstrations with a lightweight alignment controller to guide refinement toward the desired reasoning path.
2 Related Work
Multimodal ICL adapts MLLMs through multimodal demonstrations, but existing improvements largely focus on demonstration inputs and organization. For complex tasks, this leaves a need for methods that guide the reasoning path rather than only improving target-output imitation.
- Multimodal ICL adapts MLLMs at inference time using demonstrations containing images, text, and corresponding responses.
- Prior multimodal ICL methods improve retrieval, demonstration organization, prompt design, and instruction construction.These approaches show that demonstration quality can substantially affect downstream performance.
- Existing methods still emphasize better demonstrations or alignment with demonstrated target outputs rather than the reasoning path to the correct result.
- Complex multimodal tasks motivate guidance that helps MLLMs follow evidence-grounded reasoning instead of superficially imitating demonstrated responses.
3 Method
COMIL aligns multimodal reasoning paths by contrasting suboptimal and better responses, retrieving demonstrations conditioned on the current response, and controlling iterative refinement with predicted quality.
- Contrastive Demonstration Modeling: COMIL represents each demonstration with an input, suboptimal response, better response, and generated reasoning path describing the refinement.The path identifies deficiencies, relevant multimodal evidence, and how to revise the response.
- Contrastive Demonstration Modeling: The contrastive formulation provides explicit guidance about both the preferred response and how to refine toward it.The reasoning path is generated guidance rather than a direct observation of the MLLM’s internal reasoning process.
- Contrastive Demonstration Modeling: COMIL constructs its retrieval dataset by generating an initial response for each training input, pairing it with the ground-truth response and a generated refinement path.These tuples form the dataset used for subsequent retrieval and refinement.
- Response-Conditioned Retrieval and Refinement: At the initial step, retrieval uses multimodal-input relevance; after a response exists, it combines input relevance with similarity between the current and suboptimal responses.Response similarity is used as a practical proxy for refinement relevance, although it does not guarantee the same error mode.
- Response-Conditioned Retrieval and Refinement: Retrieved demonstrations guide iterative response revision, but refinement is not guaranteed to improve monotonically at every step.The framework therefore uses retrieval and refinement together to address the current response state.
- Lightweight Alignment Control: The lightweight controller predicts response quality from the input and intermediate response, using the task metric as supervision because latent reasoning-path alignment is unobservable.It estimates q_t = P(x,y_t) rather than directly measuring the latent alignment function Q.
- Lightweight Alignment Control: Refinement stops when the predicted quality reaches τ; otherwise, it continues until the maximum budget T, with final selection based on predicted quality.The threshold is set to the evaluation metric of the top 25% of training responses.
4 Experiment
COMIL is evaluated across diverse multimodal tasks, models, datasets, and baselines, with component ablations testing its contrastive demonstrations, response-conditioned retrieval, and alignment control. Results consistently favor the complete framework and indicate that its gains arise from the combination of these components.
- Experimental Setup: COMIL is evaluated on image classification, image captioning, and visual question answering using four open-source MLLMs and four datasets.The metrics are Accuracy, CIDEr, and VQA Accuracy, respectively.
- Main Results: Across 16 MLLM–dataset settings, COMIL ranks first in 13 and second in three.It achieves the best Flickr30k performance under all four MLLMs and leads three of four settings on both CIFAR-10 and OKVQA.
- Main Results: On VQAv2, COMIL reaches 81.9% accuracy with Qwen3.5-9B, exceeding TACO by 1.0 percentage point.This is the reported best result among the compared methods for that setting.
- Ablation Study: Adding components of the contrastive demonstration tuple improves performance consistently across datasets and target MLLMs.The ablation compares positive-only demonstrations, suboptimal responses, and reasoning paths.
- Ablation Study: Replacing response-conditioned retrieval with random retrieval consistently degrades performance, showing that retrieval relevance to the current response matters.The retrieved reasoning paths are intended to guide refinement toward the desired reasoning path.
- Ablation Study: Removing alignment control causes a smaller but consistent performance drop, while the complete combination outperforms simple self-refinement baselines.The controller contributes to final performance without being the primary source of gains.
5 Discussion
Discussion analyses reasoning-path alignment directly, retrieval behavior, retrieval-set size, closed-source generalization, and computational cost. COMIL generally improves alignment and performance, while response similarity remains a proxy and iterative refinement adds inference cost.
- Reasoning-Path Alignment: COMIL achieves the highest reasoning-path alignment score in five of six settings.The evaluation scores evidence mention, correctness, task relevance, and response–evidence consistency, and also reports Spearman correlation with task metrics.
- Case Study: Response-conditioned retrieval transfers correction patterns across different objects, but response similarity does not guarantee the same error mode.A counting error involving water bottles is corrected using a retrieved example involving glasses.
- Retrieval Dataset Size: Increasing the retrieval dataset from 20 to 500 examples generally improves performance across Flickr30k, VQAv2, and OKVQA, with diminishing gains at larger scales.The analysis covers three target MLLMs.
- Closed-Source MLLMs: COMIL achieves the best CIFAR10 performance on both Claude Sonnet 4.6 and GPT-4o.This supports evaluation beyond the open-source MLLM settings.
- Cost Considerations: COMIL adds iterative-refinement cost but reduces average latency and token consumption by 51.6% and 82.7%, respectively, over SC-CoT.It is also 29.2% faster than Self-Refine, while the target MLLM remains frozen.
6 Conclusion
COMIL addresses surface-level imitation by aligning multimodal reasoning paths through contrastive demonstrations, response-conditioned retrieval, and lightweight control. Experiments and reasoning-path analyses report consistent improvements across multimodal tasks and model settings.
- Conclusion: COMIL reformulates demonstrations as contrastive tuples that describe refinement from a suboptimal response to a better response.It combines this formulation with response-conditioned retrieval and a lightweight alignment controller.
- Conclusion: The framework consistently improves multimodal-task performance, with particularly clear gains on reasoning-intensive tasks.Further reasoning-path analyses and closed-source results support its effectiveness across model settings.