Through the Lens of Contrast: Self-Improving Visual Reasoning in VLMs

1 INTRODUCTION

• The paper identifies visual hallucinations in reasoning paths as the obstacle to directly adapting language-based self-improving methods to VLMs.It notes that text-focused approaches may privilege textual priors over visual evidence.Source: §1 · p. 1
• Hints plus contrast prevent new errors and rectify original hallucinations, whereas hints alone do not achieve both outcomes.The comparison uses ground-truth answers as hints and adds visual contrasting in the combined setting.Source: §1 · p. 2, Figure 1 · p. 2
• VC-STaR generates a coarse rationale, analyzes a curated contrastive VQA pair, then uses an LLM to rethink and refine the rationale.Contrastive pairs are curated within individual datasets using image and question similarity; the framework is intended to adapt across VQA tasks.Source: §1 · p. 2
• VisCoR-55K is a visual contrastive reasoning dataset of faithful rationales generated from contrastive VQA pairs curated within individual datasets for reasoning, math, chart, and OCR tasks.Finetuning with VisCoR-55K enhances VLMs’ visual reasoning capability.Source: §1 · p. 2
• VC-STaR outperforms existing self-improving baselines and models trained on recently proposed reasoning datasets across challenging benchmarks.The named benchmarks include MMVP, HallusionBench, MathVista, MathVision, and MMStar.Source: §1 · p. 2

Reasoning in Language

• Language self-improvement can refine reasoning data, use ground-truth answers as hints, or rely on internal feedback, with fewer external constraints than other approaches.Other approaches discussed include reward models, macro reasoning actions, and reinforcement learning, each with noted dependencies or adaptation limits.Source: §2 · p. 3

Reasoning in Vision

• Visual reasoning methods have used scene graphs, macro actions, or bounding boxes, but can be constrained by data structure or produce stereotyped reasoning paths.The text describes self-improvement for visual reasoning as underexplored because existing text-centric methods do not readily rectify visual hallucinations.Source: §2 · p. 3

Power of Contrasting

• Contrastive learning and cross-image comparisons have been studied across unimodal and multimodal settings, and VLMs have robust multi-image comprehension and comparison capabilities.Prior work also used contrasting to create instruction-tuning data, while its role in visual reasoning remained an open question.Source: §2 · pp. 3–4
• VC-STaR repurposes VLMs’ comparative ability to suppress visual hallucinations and bootstrap visual reasoning.The paper presents this as a new perspective on contrasting in reasoning.Source: §2 · p. 4

3 VISUAL CONTRASTIVE SELF-TAUGHT REASONER (VC-STAR)

• VC-STaR enriches VQA samples with generated rationales, then refines them by contrasting each target sample with a counterpart sharing a synonymous question and similar image context.The curated pairs support contrastive analysis and rationale refinement across diverse VQA tasks.Source: §3 · p. 4

3.1 CONTRASTIVE VQA PAIR CURATION

• Contrastive pairs use synonymous questions, visually similar images, and reasoning-dependent questions to encourage fine-grained comparison.The curation pipeline collects diverse VQA samples, searches for counterparts by multimodal similarity, and retains median-difficulty pairs.Source: §3.1 · pp. 4–5, Figure 3 · p. 5

3.2 CONTRASTING AND RETHINKING

• The framework generates a coarse rationale, analyzes a contrastive VQA pair, then uses an LLM to rethink and refine the rationale.The rethinking prompt asks the LLM to rectify visual hallucinations in the coarse rationale according to visual information from the contrastive analysis.Source: §3.2 · pp. 5–6
• VC-STaR turns refined rationales into VisCoR-55K, a 55K-sample visual reasoning dataset used to improve VLM reasoning through supervised finetuning.Rationales are filtered with text-matching post-processing to remove samples containing incorrect reasoning patterns.Source: §3 · p. 4, §3.2 · p. 6

4.2 MAIN RESULTS

• VC-STaR gains 2.4% on average over Qwen2.5VL-7B, including 5.7% on MMVP and 3.2% on the Hallusion Benchmark.Qualitative comparisons show rationales grounded in visual evidence, including on visually complex patterns.Source: Table 1 · p. 7, §4.2 · p. 7, Figure 5 · p. 8
• Existing self-improving methods improve hallucination benchmarks at the expense of math and general capabilities, while VC-STaR achieves consistent gains.The compared methods are STaR, Verifier, and Feedback, each generating rationales on VisCoR-55K for finetuning.Source: §4.2 · p. 7
• Models trained on caption-derived rationales improve notably, but trail VC-STaR; Virgo’s purely textual rationales are ineffective and LLaVA-CoT shows limited improvement.The authors attribute the gap to the advantage of visually native training over textual captions.Source: §4.2 · p. 8

4.3 ANALYSIS

• Alternative HQ-Edit and DOCCI pair strategies are effective but data-dependent, while VisCoR-55K’s broader task coverage yields more balanced performance.The authors link the alternatives’ data dependence to the biased distributions and limited curation scope of those datasets.Source: §4.3 · p. 9
• VC-STaR produces consistent, significant gains on Qwen2.5VL-3B and InternVL2.5-8B, indicating effectiveness across different base models.Each model was used to generate a visual reasoning dataset from VisCoR contrastive pairs before finetuning.Source: §4.3 · p. 9
• Adding easy samples in two batches of 20K each reduces performance, so the authors exclude them to avoid potential overthinking on straightforward problems.The experiment starts from VisCoR-55K and incrementally adds the easy samples.Source: §4.3 · p. 9
• On GQA, negative contrastive counterparts are more effective than positive ones, and combining both yields the optimal total gain.The authors attribute negative pairs’ stronger effect to stronger semantic contrast and retain both pair types.Source: §4.3 · pp. 9–10

• Finetuning on VisCoR-55K delivers consistent gains across six benchmarks, significantly surpassing self-improving baselines and models trained on state-of-the-art visual reasoning datasets.Source: §5 · p. 10
• VC-STaR refines hallucinatory reasoning through analysis of curated contrastive VQA pairs, producing the VisCoR-55K dataset.Source: §5 · p. 10
• The authors conclude that contrast can mitigate visual hallucination and enhance visual reasoning.Source: §5 · p. 10

• The authors connect learning by comparison to identifying new identities through comparison with prototype concepts.Source: §A.1 · p. 16
• They relate analogical reasoning to mapping shared relationships between domains, describing this as fine-grained contrasting.Source: §A.1 · p. 16
• In VC-STaR, contrasting offers an opportunity to learn visual concepts from prototypes, while rethinking reinforces structure-mapping by generating reasoning paths through contrast.Source: §A.1 · p. 16

• A dataset-by-dataset greedy search stops at the first valid contrastive match, avoiding a costly O(n^2) search across the full data pool.Source: §A.2 · p. 16
• 240k raw contrastive VQA pairs are filtered to 86k challenging pairs and then to the final VisCoR-55K dataset after rationale generation and text-matching quality control.Figure 7 charts pair counts by dataset and category, the ratio of median samples, and categorical breakdowns.Source: §A.2 · p. 17, Figure 7 · p. 17
• Difficulty-based sampling retains median samples, whose proportions vary across datasets, yielding 86k challenging contrastive pairs from the initial pool.Source: §A.2 · p. 17
• Each of the 86k samples receives a rationale from the contrasting-and-rethinking pipeline, followed by text-matching filtering for unexpected or erroneous reasoning patterns.Source: §A.2 · p. 17

• The thinking prompt asks the model to analyze the question, focus on relevant image content, and conclude with visual evidence for the answer.Source: §A.3 · p. 17
• The contrasting prompt compares two images for synonymous questions, summarizing shared patterns when answers match and identifying visual differences when they differ.Source: §A.3 · pp. 17–18
• The rethinking prompt rewrites the coarse rationale using contrastive analysis, applying a reasoning schema for matching answers or backward chaining for differing answers.Source: §A.3 · p. 18

A.4 ADDITIONAL QUALITATIVE RESULTS

• Figure 8 illustrates examples of rationales generated by VC-STaR in VisCoR-55K.The passage does not specify the rationale contents.Source: §A.4 · p. 18, Figure 8 · p. 19
• A model finetuned with VisCoR-55K on Qwen2.5VL-7B is tested on customized visual question answering cases.The passage describes the cases as customized but gives no further details.Source: §A.4 · p. 18
• Figure 9 presents an additional qualitative comparison based on those tests.The supplied passage does not state the comparison's outcome.Source: §A.4 · p. 18, Figure 9 · p. 19