PaperlayerView exact revision on arXiv

Source-linked AI summary

MagiC: Evaluating Multimodal Cognition Toward Grounded Visual Reasoning

Chengfei Wu, Ronald Seoh, Bingxuan Li, Liqiang Zhang, Fengrong Han, Dan Goldwasser

arXiv:2507.07297v2cs.CVcs.CL

TL;DR

MagiC addresses the limited evidence available on whether vision-language models genuinely reason over relevant visual content rather than superficial patterns or dataset biases. It builds a benchmark with human-curated and weakly supervised examples to assess answers, reasoning, grounding, and correction; its analyses find that better region focus tends to accompany better task performance, though region-focus errors remain.

  • Problem

    Existing benchmarks often emphasize final answers and provide limited insight into reasoning quality or whether intermediate steps align with relevant visual regions.

    Source: §1 · p. 1

  • Method

    MagiC combines weakly supervised and human-curated examples to evaluate answer correctness, stepwise reasoning, visual grounding, and revision of faulty reasoning.

    Source: §1 · pp. 1–2

  • Results

    Models that focus better on relevant regions tend to achieve better task performance, yet current state-of-the-art models show location, spatial-relation, and region-description failures.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The analyses identify region-focus failures even among current state-of-the-art models, including incorrect locations, wrong spatial relations, and inaccurate region descriptions.

    Source: §6 · p. 9

    MagiC is limited to grounded visual reasoning over natural scenes and does not cover OCR, document, chart, mathematical-diagram, or medical-image reasoning.

    Source: Limitation · p. 9

Abstract

from arXiv · show

Recent advances in large vision-language models have led to impressive performance in visual question answering and multimodal reasoning. However, it remains unclear whether these models genuinely perform grounded visual reasoning or rely on superficial patterns and dataset biases. In this work, we introduce MagiC, a comprehensive benchmark designed to evaluate grounded multimodal cognition, assessing not only answer accuracy but also the quality of step-by-step reasoning and its alignment with relevant visual evidence. Our benchmark includes approximately 5,500 weakly supervised QA examples generated from strong model outputs and 900 human-curated examples with fine-grained annotations, including answers, rationales, and bounding box groundings. We evaluate 15 vision-language models ranging from 7B to 70B parameters across four dimensions: final answer correctness, reasoning validity, grounding fidelity, and self-correction ability. MagiC further includes diagnostic settings to probe model robustness under adversarial visual cues and assess their capacity for introspective error correction. We introduce new metrics such as MagiScore and StepSense, and provide comprehensive analyses that reveal key limitations and opportunities in current approaches to grounded visual reasoning.

1 Introduction

MagiC addresses the gap between answer-focused evaluation and assessing whether multimodal models reason over relevant visual evidence. It introduces a benchmark that evaluates answers, reasoning, grounding, and error correction, and finds that precise region focus tends to accompany better answers.

1 Introduction

  • Existing benchmarks often emphasize end-task performance and provide limited insight into reasoning quality or whether intermediate steps align with relevant image regions.Prior evaluations of chain-of-thought, rationale generation, and grounding consistency are described as fragmented and lacking unified protocols and comprehensive grounding diagnostics.Source: §1 · p. 1, Figure 1 · p. 2
  • MagiC evaluates grounded visual reasoning by assessing answer correctness, coherent step-by-step reasoning, alignment to visual evidence, and revision of faulty reasoning.Its diagnostic settings probe responses to misleading or irrelevant visual cues and whether models can detect and revise reasoning errors.Source: §1 · pp. 1–2
  • The benchmark combines 5,534 weakly supervised instances with 896 human-curated instances annotated with answers, rationales, and bounding-box groundings.The weakly supervised examples are derived from high-performing model outputs, while the human-curated set includes a held-out test set.Source: §1 · p. 2
  • MagiC evaluates 15 open-source and proprietary models from 7B to 70B+ parameters across answer correctness, reasoning validity, grounding quality, and self-correction.MagiScore measures overlap between predicted and reference boxes; a human evaluation of four models informs StepSense, which captures reasoning coherence and factual consistency.Source: §1 · p. 2
  • Models that focus better on relevant regions tend to achieve better task performance, while current state-of-the-art systems still exhibit several region-focus failures.The reported failures include incorrect object locations, wrong spatial relations, and inaccurate region descriptions.Source: §1 · p. 2

2 Related Work

Prior benchmarks broaden visual reasoning evaluation but generally emphasize end-task performance, while grounding research often focuses on activations or specific tasks. MagiC targets explicit reasoning-to-image alignment, and the paper reports a region-reference fine-tuning result as an example of a method it can measure.

2.1 Benchmarks for Large Vision-Language Models

  • Most visual-reasoning benchmarks still focus on end-task performance, while explicit reasoning-quality evaluation remains challenging and current methods leave room for improvement.The surveyed benchmarks include MMBench, MMMU, SEED-Bench, and MM-Vet.Source: §2.1 · p. 3

2.2 Grounding in Vision Tasks

  • MagiC examines whether explicit textual reasoning remains connected to visual reality, extending grounding evaluation beyond encoder activations and task-specific settings.The paper argues that fluent explanations and answers may be incorrect or irrelevant if reasoning loses its connection to visual evidence.Source: §2.2 · p. 3

2.3 Grounded Visual Reasoning Methods

  • Region-reference fine-tuning lifted MagiScore macro-F1 from 43.2 to 82.2, illustrating a use case for measuring grounding-improvement methods.The cited methods include Visual-RFT, ViGoR, UniVG-R1, and VGR.Source: §2.3 · p. 3

3 The MagiC Benchmark

MagiC uses a human-annotated test set of reasoning steps and corrections to evaluate grounded visual reasoning over natural scenes. Its construction combines question-relevant and adversarial regions, model-generated reasoning, expert correction, and a separate weakly supervised dataset.

3 The MagiC Benchmark

  • MagiC's human-curated benchmark contains 698 test question-and-reasoning pairs spanning 41 subtypes grouped into five question types.The types are Verify, Query, Choose, Logical, and Compare; the full collection contains 896 annotated questions, with 198 reserved for development.Source: §3 · p. 3

3.1 Task-Input Construction

  • Task-input construction combines human-attended hotspot regions and GQA objects as relevant boxes, then adds three non-overlapping regions irrelevant to the current question as adversarial boxes.Human saliency maps inform construction; the resulting relevant and adversarial boxes form the candidate set and relevance labels.Source: §3.1 · p. 4

3.2 LVLM Response Collection

  • Four LVLMs generated step-by-step responses from image-question inputs, which were filtered, segmented into reasoning steps, and sampled for annotation.Invalidly formatted responses and repetitive reasoning loops were removed before sentence-level segmentation.Source: §3.2 · p. 4

3.3 Reasoning Correction Collection

  • Expert annotators assessed final answers and individual reasoning steps, reaching Cohen’s Kappa of 0.72, and supplied complete rewrites for incorrect steps.The dataset records original steps, correctness labels, and corresponding corrections; the full annotation includes 8,403 step labels and 2,700 natural-language corrections.Source: §3 · p. 3, §3.3 · pp. 4–5

3.4 Weakly Supervised Dataset

  • The weakly supervised split uses GQA object boxes without human saliency, excludes images from the primary annotated set, and collects responses without adversarial regions.Its responses were generated from image, question, and answer inputs to produce cleaner reasoning data.Source: §3.4 · p. 5

4 Task Overview

The section defines three evaluation tasks, operationalizing region focus and answer accuracy, then testing whether models can correct injected reasoning errors.

4 Task Overview

  • The benchmark samples model reasoning responses for three evaluation tasks and pairs them with ground-truth answers, relevant boxes, and human-corrected reasoning chains.The inputs include short- and full-form answers and model-generated reasoning steps.Source: §4 · p. 5

4.1 Region Focus

  • MagiScore compares the regions selected across a model’s reasoning steps with human-labeled relevant boxes using micro- or macro-averaged precision, recall, and F1.It measures explicit evidence selection, not internal attention or step-level faithfulness.Source: §4.1 · p. 5

4.2 LLM-assisted Final Answer Accuracy

  • An LLM judge evaluates predicted answers against references, with short-answer accuracy allowing core semantic coverage and full-answer accuracy requiring comprehensive alignment.The approach addresses exact match’s inability to recognize semantic equivalents and its susceptibility to matching a ground-truth term appearing randomly in a longer response.Source: §4.2 · pp. 5–6

4.3 Self-Correction

  • Self-correction is tested by giving a model reasoning up to an annotated error, then checking whether its continuation matches the human correction.Self-Heal is the fraction of annotated incorrect subsequences for which the model corrects the injected error.Source: §4.3 · p. 6

5 Experiment and Results

Across the evaluated models, stronger focus on relevant visual regions tends to accompany higher answer accuracy, while scaling is associated with gains in region focus, reasoning, and self-correction. The experiments also expose recurring errors in region selection and interpretation.

5.2 Main Results

  • Across 15 models, higher MagiScore tends to accompany better answer performance, and the section reports that accuracy climbs almost linearly with alignment to relevant regions.The pattern is reported for both small- and large-scale models.Source: §5 · p. 6, §5.2 · p. 7

5.1 Experimental Settings

  • The evaluation covers 12 open-source general-purpose models, QvQ, and two proprietary models, with reasoning and answers assessed on a test set containing relevant and adversarial boxes.QvQ is evaluated zero-shot; standardized in-context demonstrations leave MagiScore nearly unchanged but reduce its answer accuracy.Source: §5.1 · p. 6, Table 1 · p. 7

5.2 Main Results

  • Medium-sized models generally score higher than smaller models on region attention and answer performance, with further gains for large models; the Qwen2.5-VL sizes follow this trend.Figure 4 presents the MagiScore scaling curve.Source: §5.2 · p. 7, Figure 4 · p. 8
  • QvQ performs comparably to Qwen2.5-VL 72B without bounding-box demonstrations, but its low attention precision reflects exhaustive consideration of boxes.The authors present test-time scaling as a potential route to better visual grounding, while noting that other comparable models remain to be tested.Source: §5.2 · p. 7
  • In human-annotated evaluation, models with higher sentence-level reasoning accuracy almost always have higher final-answer accuracy; Qwen2.5-VL 7B leads both metrics (61.4% vs. 70.0%).The passage says InternVL 8B trails on both metrics.Source: Table 3 · p. 8
  • Each Qwen2.5-VL size increase brings roughly a six-point boost in correction accuracy, and LLaVA-OneVision 7B and Gemma 3 12B lag Qwen2.5-VL 7B despite similar short-answer performance.The authors report that stronger models tend to have better self-correction capabilities.Source: §5.2 · p. 8

5.3 Error Analysis

  • Models often inspect every box, including irrelevant ones, despite instructions to select relevant regions; this pattern is especially common for QvQ.The authors note that exhaustive inspection can be inefficient and may increase hallucination risk.Source: §5.3 · p. 9
  • Failures also include assigning a recognized detail to the wrong box, reversing spatial relations, and misdescribing objects within a selected region.Figure 5 shows examples of failed reasoning with adversarial boxes labeled in green.Source: Figure 5 · p. 8, §5.3 · p. 9

6 Conclusion

MagiC evaluates 15 LVLMs across answer correctness, reasoning validity, grounding fidelity, and self-correction; models focusing better on relevant regions tend to perform better, yet current SOTA models still show region-focus failures.

6 Conclusion

  • MagiC evaluates 15 LVLMs from 7B to 70B+ parameters across answer correctness, reasoning validity, grounding fidelity, and self-correction.Source: §6 · p. 9
  • Models that focus better on relevant regions tend to achieve better task performance, and this focusing ability improves with increased model size.Source: §6 · p. 9
  • Current SOTA LVLMs all show region-focus failures, including incorrect object locations, wrong spatial relations, and inaccurate region descriptions.Source: §6 · p. 9

Limitation

  • MagiC covers grounded visual reasoning over natural scenes, not OCR, document, chart, mathematical-diagram, or medical-image reasoning.Source: Limitation · p. 9
  • Answer accuracy and Self-Heal rely on LLM-based semantic-equivalence judgments, while StepSense is fully human-annotated.Source: Limitation · p. 9
  • Public GQA text could at most inflate final-answer accuracy, and even the best models show room to improve in answer accuracy and process scores.Source: Limitation · p. 9

A Data Statistics

The appendix points readers to dataset statistics and a MagiScore calculation example.

A Data Statistics

B MagiScore Calculation Example

C Data Examples

The appendix points to four additional data examples.

C Data Examples

D Experimental Details

The experimental appendix describes the evaluated models, inference setup, hardware, LLM-assisted evaluation, and task-input algorithm reference.

D Experimental Details

  • The evaluation includes 13 open and 2 proprietary LVLMs, uses each model’s default sampling temperature, and runs on A100 or H100 GPUs.Source: D Experimental Details · p. 13
  • LLM-assisted evaluation uses Qwen2.5-72B-Instruct-AWQ with greedy decoding on one Nvidia A100-SXM-80GB GPU.Source: D Experimental Details · p. 13

E Details on Task-Input Processing

F Cross-Judge Agreement and Human Validation

The section reports strong agreement among automated judges and gains from training Qwen2.5-VL-7B on weakly supervised data.

F Cross-Judge Agreement and Human Validation

  • 94.2% mean pairwise judge agreement, κ values of 0.79–0.95, and ranking correlations of ρ = 0.86–0.95; the Qwen2.5-72B judge matched 91.5% of 698 human labels (κ = 0.83).All three judges identified the same top two and bottom two models; human annotation matched the judge on 100% of decisions in a randomly sampled subset.Source: F Cross-Judge Agreement and Human… · p. 13

G Model Trained with Weakly Supervised Split

H Capability Comparison with Prior Benchmarks

Table 7 compares MagiC with prior benchmarks across data characteristics, annotation types, and evaluation protocols. It identifies MagiC as the only benchmark combining all seven capabilities.

H Capability Comparison with Prior Benchmarks

  • Examples span fully correct reasoning and answers, partially incorrect reasoning with a correct answer, and incorrect reasoning with an incorrect answer.In the latter case, red boxes without correction indicate an annotator manually steered the reasoning.Source: Figure 8 · p. 14, Figure 9 · p. 15, Figure 10 · p. 15
  • Table 7 identifies MagiC as the only benchmark combining all seven capabilities.The comparison covers data characteristics, annotation types, and evaluation protocols.Source: H Capability Comparison with Prior… · p. 13, Table 7 · p. 20
  • Visual CoT uses one evidence box per question, scores grounding with box detection at IoU ≥0.5, lacks human step-level labels on model reasoning, and has no self-correction protocol.Its reasoning steps are largely GPT-generated.Source: Table 7 · p. 20

Related papers

Loading 2507.07297v2…