PaperlayerView exact revision on arXiv

Source-linked AI summary

Think Visually, Reason Textually: Vision-Language Synergy in ARC

Beichen Zhang, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, Jiaqi Wang

arXiv:2511.15703v2cs.CVcs.AIcs.CL

TL;DR

Prior ARC-AGI approaches represent input-output grids as text, losing visual intuition and 2D structural information useful for recognizing spatial patterns. VLSR assigns visual representations to rule summarization and text to rule application, while MSSC uses visual checks of text-generated predictions to guide self-correction. The combined approach yields up to a 4.33% improvement over text-only baselines.

  • Problem

    Prior ARC-AGI approaches represent input-output grids as text, losing visual intuition and 2D structural information useful for recognizing spatial patterns.

    Source: §1 · p. 1

  • Method

    VLSR assigns visual representations to rule summarization and text to rule application, while MSSC uses visual checks of text-generated predictions to guide self-correction.

    Source: §1 · pp. 2–3

  • Results

    The combined approach yields up to a 4.33% improvement over text-only baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings support strategically combining vision and text according to their complementary strengths at different reasoning stages.

    Source: §1 · p. 2

    Naively rendering grids as images degrades performance versus text-only baselines because precise element-wise operations are difficult.

    Source: §1 · p. 2

Abstract

from arXiv · show

Abstract reasoning from minimal examples remains a core unsolved problem for frontier foundation models such as GPT-5 and Grok 4. These models still fail to infer structured transformation rules from a handful of examples, which is a key hallmark of human intelligence. The Abstraction and Reasoning Corpus for Artificial General Intelligence (ARC-AGI) provides a rigorous testbed for this capability, demanding conceptual rule induction and transfer to novel tasks. Most existing methods treat ARC-AGI as a purely textual reasoning task, overlooking the fact that humans rely heavily on visual abstraction when solving such puzzles. However, our pilot experiments reveal a paradox: naively rendering ARC-AGI grids as images degrades performance due to imprecise rule execution. This leads to our central hypothesis that vision and language possess complementary strengths across distinct reasoning stages: vision supports global pattern abstraction and verification, whereas language specializes in symbolic rule formulation and precise execution. Building on this insight, we introduce two synergistic strategies: (1) Vision-Language Synergy Reasoning (VLSR), which decomposes ARC-AGI into modality-aligned subtasks; and (2) Modality-Switch Self-Correction (MSSC), which leverages vision to verify text-based reasoning for intrinsic error correction. Extensive experiments demonstrate that our approach yields up to a 4.33\% improvement over text-only baselines across diverse flagship models and multiple ARC-AGI tasks. Our findings suggest that unifying visual abstraction with linguistic reasoning is a crucial step toward achieving generalizable, human-like intelligence in future foundation models. Source code is released at https://github.com/InternLM/ARC-VL.

1. Introduction

ARC-AGI asks systems to infer transformation rules from minimal examples, but text-only representations omit useful 2D structure. The paper proposes assigning visual and textual reasoning to complementary stages and combining them for inference and self-correction.

1. Introduction

  • Text-only ARC-AGI representations preserve precise element values but lose visual intuition and 2D structure; naive image rendering instead struggles with exact cell operations.The paper gives a 20 × 20 grid and the cell at (5, 7) as an example of image-based imprecision.Source: §1 · pp. 1–2
  • Vision improves rule summarization by 3.0%, while using vision for rule application causes a 20.5% performance drop in o4-mini.The analysis attributes the difference to holistic perception of 2D structure versus imprecise element-wise manipulation.Source: §1 · p. 2
  • VLSR visualizes example grids for rule summarization, then uses textual representations for precise element-wise rule application.The authors attribute its improvements to decomposing the task and matching each subtask to a modality's strengths.Source: §1 · p. 2
  • MSSC uses vision to verify a text-generated prediction against example patterns, then provides feedback for another round of textual inference if needed.It is presented as intrinsic self-correction without external information or ground truth.Source: §1 · p. 2
  • Up to 4.33% average improvement over text-only baselines across GPT-4o, Gemini-2.5-Pro, o4-mini, Qwen3-VL, and ARC-AGI, BARC, and Re-ARC.On the official ARC-AGI evaluation set, the reported improvements are up to 7.25% for Gemini-2.5-Pro and 4.5% for o4-mini.Source: §1 · pp. 2–3
  • The study identifies four modality trade-offs: holistic versus independent processing, 2D structure preservation, encoding efficiency, and element-wise precision.It presents VLSR and MSSC as training-free strategies for using these complementary strengths across the reasoning pipeline.Source: §1 · p. 3
  • Separate models for visual rule summarization and textual rule application improve same-data text-only fine-tuning by 3.5%, with Qwen3-8B surpassing GPT-4o.This contribution extends the vision-language synergy principle to training.Source: §1 · p. 3

2. Related Work

Prior ARC-AGI work includes synthetic-task training, test-time training, prompting with predefined rules, and memory-based guidance. This paper identifies the neglect of image representations as a gap and situates its approach among visual reasoning methods.

2. Related Work

  • Some ARC-AGI methods fine-tune on synthetic tasks made by permuting and combining rules [3] [8] [10] [15] [16] [17] [19] [20] [23] [37], while test-time training uses demonstrations for additional fine-tuning.Source: §2 · p. 3
  • Other strategies prompt models with predefined transformation rules [18] [25] [26] [30] or use memories of prior problems [2] [9] [14] [24] [28] [29]; these approaches treat ARC-AGI as text-only.Source: §2 · p. 3
  • Visual Sketchpad [11] supports geometry problem-solving with diagrams, while ViLaSR [35] visualizes intermediate spatial hypotheses for tasks such as maze solving.Source: §2 · p. 3

3. Method

The method assigns visual reasoning to global rule summarization and textual reasoning to precise rule application, then uses visual verification to guide intrinsic self-correction. Its design follows an empirical comparison showing complementary modality strengths across ARC-AGI subtasks.

3. Method

  • The method’s central premise is that visual thinking supplies global perception and 2D structure, whereas textual thinking supports precise element-wise processing.VLSR switches modalities between summarizing exemplar rules and applying them to test inputs.Source: §3 · p. 3

3.1. Problem Setup and Notation

  • ARC-AGI tasks require inferring a transformation rule from example matrix pairs and applying it to a test matrix; matrices can be represented as nested-list text or color-coded grids.The section identifies loss of 2D structure in text and the combination of summarization and application as limitations of prior approaches.Source: §3.1 · p. 4

3.2.1. Quantitative Comparison

  • Visual rule summarization improved accuracy by 3.2% on average, while visual rule application reduced accuracy by 15.0% on average.For Gemini-2.5, visual summarization yielded 40.75% versus 37.25%; visual application fell from 40.75% to 23.75%.Source: Table 1 · p. 4, §3.2.1 · p. 5

3.2. Comparative Analysis of Vision and Text

  • The analysis separates ARC-AGI into rule summarization and rule application, varying modality in each phase while holding other factors constant.For application comparisons, the study uses a rule derived from the visual modality and includes the standard text-only approach as a baseline.Source: §3.2 · p. 4, §3.2.1 · pp. 4–5

3.2.2. Qualitative Analysis: Understanding the “Why”

  • Vision’s holistic perception preserves 2D spatial relations and efficiently represents large matrices, while text enables precise element-wise access and manipulation.Visual reasoning can falter on cell localization during rule application, whereas text-derived rules may miss inter-row or diagonal regularities.Source: §3.2.2 · p. 5

3.3. Vision-Language Synergy Reasoning

  • VLSR summarizes rules from visualized examples, applies them to textual test matrices, and matches each subtask to its modality-specific strengths.The two phases use the same base model, with input modality and prompting strategy differing.Source: §3.3 · p. 6

3.4. Modality-Switch Self-Correction

  • MSSC re-encodes a text-generated output as an image for consistency checking, then uses inconsistency feedback to trigger another textual inference pass.The process repeats until consistency is achieved or the limit of Nmax = 3 is reached.Source: §3.4 · p. 6

4. Experiments

Experiments test the approach across models and ARC benchmarks, finding gains from modality-aligned reasoning and self-correction, with an extension to fine-tuning. Analysis compares iterative correction strategies and illustrates how visual reasoning can identify spatial patterns missed by text-only reasoning.

4.1. Experimental setup

  • Evaluation uses Pass@1 accuracy at temperature 0.7 across the official 400-task ARC-AGI set and 100-task samples from Re-ARC and BARC.The evaluated models include open-source Qwen3-VL-235B-A22B-Instruct and closed-source GPT-4o, Gemini-2.5-pro-thinking-8192, and o4-mini.Source: §4.1 · p. 6

4.2. Main Results

  • Combining VLSR and MSSC improves text-only reasoning by up to 6.25% for GPT-4o and 7.25% for Gemini-2.5-Pro on ARC-AGI.Across models, VLSR improves baseline reasoning by an average of 3.02%, and MSSC adds 1.82% on average.Source: §4.2 · p. 7, Table 2 · p. 7
  • On GPT-4o, vision-language synergy uses global 2D perception during rule summarization to identify the correct spatial pattern, while text-only reasoning processes elements without spatial context and reaches an incorrect rule.Source: Figure 4 · p. 7
  • The proposed method achieves the highest accuracy on all three training-free comparison benchmarks, beating ArcMemo-PS by 1.5% on ARC-AGI.The comparison includes ARC-AGI-400, an ARC-AGI-100 subset, and Re-ARC; the method also outperforms memory-augmented text-only strategies.Source: §4.2 · p. 7, Table 3 · p. 7, §4.1 · p. 7

4.3. Analysis

  • Across three rounds, MSSC improves monotonically while text-only self-correction stagnates or degrades.For GPT-4o, MSSC rises from 8.25 to 12.0; text-only self-correction gains only 0.5 points overall and falls to 8.0 in Round 2. The authors attribute the difference to visual verification revealing spatial inconsistencies overlooked by text reasoning.Source: §4.3 · pp. 7–8, Table 4 · p. 7

4.4. Extension to Model Fine-tuning

  • Vision-language synergy fine-tuning reaches 13.25% on ARC-AGI, 3.5% above text-only fine-tuning and 5.0% above GPT-4o.The models specialize in visual rule summarization and textual rule application using approximately 200k synthetic training tasks; similar trends hold on BARC-100 and Re-ARC.Source: §4.4 · p. 8, Table 5 · p. 8

5. Conclusion

The paper combines visual and textual reasoning for abstract reasoning tasks, reporting gains over text-only approaches and extending the strategy to fine-tuning. Its conclusion is that this approach improves performance in the evaluated settings.

5. Conclusion

  • The approach combines visual 2D intuitiveness with text-based precision and improves performance by an average of 4.3% over text-only reasoning across multiple tasks and base models.The paper also reports outperforming text-centric training-free strategies.Source: §5 · p. 8
  • Vision-language synergy fine-tuning improves over text-only fine-tuning by 3.5% and outperforms several larger open- and closed-source models.This extends the inference strategy to model fine-tuning.Source: §5 · p. 9

A. Prompts

The prompts separate textual rule application from visual rule summarization and visual consistency verification. They also allow a proposed rule to be checked against examples before application.

A. Prompts

  • The text-only prompt asks the model to infer a matrix-changing rule from input-output matrices and apply it to a new input.The examples use nested-list matrix representations.Source: A. Prompts · p. 11
  • The vision-centric prompt presents example matrix pairs as images and asks for a summarized grid-changing rule.The prompt supplies image examples rather than textual matrix lists.Source: A. Prompts · p. 11
  • The text-centric application prompt asks the model to apply a rule to new matrix input, while a possible rule may first be checked against the examples.If the supplied rule is incorrect, the prompt directs the model to summarize a new rule and apply it.Source: A. Prompts · p. 11
  • The vision-centric verification prompt asks whether a new input-output image pair follows the same transformation rule as the examples.It requests a True-or-False judgment.Source: A. Prompts · p. 11

B. Matrix-to-Image Visualization

The visualization converts matrix values into distinct colors arranged as a 2D grid. White dividing lines separate the colored cells to make element values and block structure clearer.

B. Matrix-to-Image Visualization

C. Qualitative Examples

Across GPT-4o, Gemini-2.5-Pro-thinking-8192, and o4-mini, visual thinking uses 2D structure, a global perspective, and long-range perception; using visual information during global rule summarization enables more effective rule extraction.

C. Qualitative Examples

Loading 2511.15703v2…