PaperlayerView exact revision on arXiv

Source-linked AI summary

VGRP-Bench: Visual Grid Reasoning Puzzle Benchmark for Large Vision-Language Models

Yufan Ren, Konstantinos Tertikas, Shalini Maiti, Junlin Han, Tong Zhang, Sabine Süsstrunk, Filippos Kokkinos

arXiv:2503.23064v2cs.CV

TL;DR

LVLMs face challenges in visual puzzles that combine perception and multi-step reasoning, an area not systematically covered by existing benchmarks. VGRP-Bench introduces 20 diverse puzzles and evaluates models and two fine-tuning strategies; models struggle to solve puzzles, and fine-tuning gains generalize poorly to unseen ones.

  • Problem

    Visual grid puzzles demand perception, rule comprehension, and logical reasoning, yet existing benchmarks provide limited dedicated and systematic evaluation of LVLMs on reasoning puzzles.

    Source: §1 · p. 2

  • Method

    VGRP-Bench provides 20 customizable visual puzzles across difficulty levels and evaluates LVLMs, including with solution-based and synthetic-reasoning supervised fine-tuning.

    Source: §1 · p. 2, §3.4 · p. 5

  • Results

    LVLMs struggle to solve VGRP-Bench puzzles, while both fine-tuning strategies improve performance on trained puzzles but generalize poorly to unseen ones.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The authors hope VGRP-Bench will inspire research and advance LVLM studies for complex, real-world tasks.

    Source: §6 · p. 9

    High fine-tuning costs limit the SFT experiments to smaller 11B models.

    Source: §5 · p. 8

Abstract

from arXiv · show

Large Vision-Language Models (LVLMs) struggle with puzzles, which require precise perception, rule comprehension, and logical reasoning. Assessing and enhancing their performance in this domain is crucial, as it reflects their ability to engage in structured reasoning - an essential skill for real-world problem-solving. However, existing benchmarks primarily evaluate pre-trained models without additional training or fine-tuning, often lack a dedicated focus on reasoning, and fail to establish a systematic evaluation framework. To address these limitations, we introduce VGRP-Bench, a Visual Grid Reasoning Puzzle Benchmark featuring 20 diverse puzzles. VGRP-Bench spans multiple difficulty levels, and includes extensive experiments not only on existing chat LVLMs (e.g., GPT-4o), but also on reasoning LVLMs (e.g., Gemini-Thinking). Our results reveal that even the state-of-the-art LVLMs struggle with these puzzles, highlighting fundamental limitations in their puzzle-solving capabilities. Most importantly, through systematic experiments, we identify and analyze key factors influencing LVLMs' puzzle-solving performance, including the number of clues, grid size, and rule complexity. Furthermore, we explore two Supervised Fine-Tuning (SFT) strategies that can be used in post-training: SFT on solutions (S-SFT) and SFT on synthetic reasoning processes (R-SFT). While both methods significantly improve performance on trained puzzles, they exhibit limited generalization to unseen ones. We will release VGRP-Bench to facilitate further research on LVLMs for complex, real-world problem-solving. Project page: https://yufan-ren.com/subpage/VGRP-Bench/.

1. Introduction

Grid puzzles combine visual perception with multi-step reasoning, but existing LVLM benchmarks underuse them. VGRP-Bench introduces 20 customizable puzzles and evaluates off-the-shelf models and post-training strategies.

1. Introduction

  • Existing research underuses grid puzzles as LVLM benchmarks, despite their structured deduction and constraint-satisfaction demands being relevant to real-world tasks.The paper names robotic path planning, logistics scheduling, and embodied-AI control as examples.Source: §1 · p. 2
  • VGRP-Bench offers 20 diverse, customizable grid puzzles across difficulty levels to evaluate LVLM perception, rule adherence, and puzzle solving.Difficulty depends on grid size, required reasoning steps, and decision-space size; text versions help separate reasoning from perception.Source: §1 · p. 2
  • The authors evaluate state-of-the-art LVLMs, including reasoning models, and report that most struggle even on easy puzzles.GPT-4o solves a simple 4 × 4 Sudoku at below 30% in the text-only version.Source: §1 · p. 2
  • The paper explores solution-based S-SFT and reasoning-trajectory R-SFT, finding significant easy-level improvement while models still struggle at medium and hard levels.R-SFT trains on solver-recorded stepwise trajectories; the authors also examine generalization to puzzles not used for fine-tuning.Source: §1 · pp. 2–3

2. Related Works

Prior benchmarks cover general language, multimodal perception, and game interaction, while VGRP-Bench focuses on visual reasoning puzzles that combine perception, decision-making, and multi-step reasoning.

2.1. General LLM/LVLM Benchmarks

  • General LLM and LVLM benchmarks span language, coding, mathematics, and multimodal perception, but VGRP-Bench targets reasoning puzzles requiring perception, decision-making, and multi-step reasoning.The related-work discussion names SuperGLUE, MMLU, BigBench, MMBench, EMMA, SEED-Bench, and BLINK among prior benchmarks.Source: §2.1 · p. 3

2.2. LLM/LVLM Game Benchmarks

  • Visual-game benchmarks include agent interaction, puzzle environments, and scraped puzzles; VGRP-Bench distinguishes itself with customizable generators and evaluation from inference through post-training.A concurrent scraped-puzzle benchmark contains 949 instances.Source: §2.2 · p. 3
  • VGRP-Bench defines puzzle primitives and pairs them with sample questions to establish a unified evaluation framework for reasoning, rule-following, and perception.Source: Figure 3 · p. 4

3. VGRP-Bench: The Benchmark

VGRP-Bench organizes customizable visual reasoning puzzles around shared primitives, evaluates models at multiple levels, and explores solution- and reasoning-based fine-tuning.

3.1. Grid-Like Visual Reasoning Puzzles

  • Puzzle selection emphasizes multi-step decisions and rule validation, diverse visual clues and interactions, and coverage of a structured puzzle taxonomy.Sudoku uses numerical repetition constraints, Trees-and-Tents requires matching, and Thermometers uses physical-world filling rules.Source: §3.1 · p. 4
  • The benchmark represents puzzles using variables, states, constraints, and conditions: assignments, current values, validity rules, and preset clues.A 4 × 4 Sudoku has 16 variables, each taking a value from 1 to 4.Source: §3.1 · p. 4

3.2. Evaluation Protocol

  • Evaluation measures holistic board perception and puzzle-solving success, with finer-grained assessments at the cell and step levels.The evaluation spans multiple capabilities and difficulty levels.Source: §3.2 · pp. 4–5

3.3. Puzzle Rule/Capability Taxonomy

  • The taxonomy includes counting, arithmetic, matching, numerical comparison, procedural order, and connecting components; puzzles may require multiple capabilities.Killer-Sudoku, Kakuro, Kakurasu, and Renzoku involve addition and subtraction; Trees-and-Tents requires bijective matching.Source: §3.3 · p. 5, Figure 4 · p. 5

3.4. Post-Training Techniques

  • S-SFT trains on final solutions, whereas R-SFT trains on synthetic trajectories of intermediate puzzle states generated with depth-first search and process of elimination.The search procedure explores systematically and backtracks upon failure states; random starts can branch excessively on a 4 × 4 Sudoku with 12 missing values.Source: §3.4 · p. 5

4. Experiments

The experiments find that off-the-shelf LVLMs struggle with grid-puzzle solving, with performance varying by difficulty, clues, and input format. They also compare solution- and reasoning-supervised fine-tuning, which improves performance on evaluated puzzles.

4.1. Implementation Details

  • The evaluation covers closed- and open-source chat and reasoning LVLMs, using five runs of 20 instances and reporting mean correctness and standard deviation.Post-training uses Llama 3.2 Vision Instruct, with no overlap between training and test puzzles by input or solution.Source: §4.1 · pp. 6–7

4.2. Off-the-Shelf LVLMs Evaluation

  • All evaluated LVLMs achieve puzzle-solving success rates below 80%.Closed-source models generally outperform open-source models, and larger models tend to perform better.Source: §4.2 · p. 7
  • Claude leads step-level rule following, while LlaVA performs worst; Sudoku has the highest accuracy among the four evaluated puzzles.For cell-level perception, most closed-source models except Claude score below 50%, while Qwen2.5-72B leads the open-source models.Source: §4.2 · p. 7, Figure 7 · p. 7, Figure 6 · p. 7
  • Accuracy declines as grids grow and more reasoning steps are required; medium Thermometers yield below 5% perception accuracy, and hard puzzles worsen further.More clues correspond to higher puzzle-solving rates and perception accuracy; text versions have higher solving rates than vision-based versions, though they remain challenging.Source: Figure 8 · p. 7, §4.2 · p. 8, Figure 10 · p. 8
  • Off-the-shelf chat models often mislocalize grid values, confuse clue roles, and repeat responses.For example, they may mistake a Killer Sudoku cage clue for a board number.Source: §4.2 · p. 8

4.3. Post-Training Evaluation

  • Both S-SFT and R-SFT improve the pretrained Llama 3.2 model, but which strategy performs better varies across puzzles.R-SFT is stronger on some puzzles, while S-SFT is stronger on others; the authors hypothesize that R-SFT's greater supervision may be offset by compounding errors in long reasoning trajectories.Source: §4.3 · p. 8, Figure 11 · p. 8

5. Limitations and Future Work

Fine-tuning experiments are limited to smaller 11B models because fine-tuning larger models has high computational cost.

5. Limitations and Future Work

  • High fine-tuning costs limit the SFT experiments to smaller 11B models.The authors cite 70B-parameter models as an example of the larger models whose fine-tuning is costly.Source: §5 · p. 8

6. Conclusion

The conclusion presents VGRP-Bench and reports that off-the-shelf LVLMs struggle with puzzle solving. Fine-tuning improves performance on trained puzzles but generalizes poorly to unseen ones.

6. Conclusion

  • VGRP-Bench evaluates visual grid puzzles across varied settings, difficulty levels, and rules; off-the-shelf LVLMs struggle, while post-training improves trained-puzzle performance but generalizes poorly to unseen puzzles.The authors hope the benchmark will inspire research on LVLMs for complex, real-world tasks.Source: §6 · p. 9

7. Versions of Models Used

The section lists the specific model versions used in the experiments.

7. Versions of Models Used

  • The model list records version identifiers where applicable, including GPT-4o (gpt-4o-2024-08-06), Claude 3.5 (claude-3-5-sonnet-20241022), and QVQ-72B (Qwen/QVQ-72B-Preview).It also lists identifiers for Gemini, Qwen, Llama, and LlaVA models.Source: §7 · p. 14

8. Additional Evaluation Details

The evaluation pipeline addresses noncompliant LVLM outputs so that puzzle-solving capability can be assessed despite formatting and extraction problems.

8. Additional Evaluation Details

  • Models often produce malformed JSON, inconsistent empty-cell symbols, or final answers that are difficult to extract from chain-of-thought responses.These issues can make responses unsuitable for parsing even when the evaluation focus is puzzle solving.Source: §8 · p. 14
  • The authors use json-repair to fix broken JSON, including escape-character and quote inconsistencies.Source: §8 · p. 14
  • GPT-4o reformats outputs into a unified format, allowing noncompliant responses to be assessed rather than rejected outright.Source: §8 · p. 14

9. Additional Implementation details

The implementation details cover model deployment, puzzle dataset construction, fine-tuning, and clue-based difficulty settings.

9. Additional Implementation details

  • Closed-source LVLMs use official APIs and open-source models run locally with vLLM; closed-source models do not construct solvers dynamically.They rely solely on vision-language modeling capabilities to solve puzzles.Source: §9 · p. 14
  • The dataset contains 100,000 puzzles per type, split without duplicated conditions or solutions across sets, but solutions are not required to be unique.The authors say enforcing uniqueness would reduce possible conditions and increase generation overhead.Source: §9 · p. 14
  • LLaMa 3.2 Vision Instruct is fine-tuned for five epochs on one node with eight A100 GPUs, taking approximately 24 hours.Training uses batch size four per GPU and masks the input question to supervise answer predictions.Source: §9 · p. 14
  • For reveal-based games such as Sudoku, clue ranges decrease across easy, medium, and hard settings.The stated ranges are 25%–75%, 25%–50%, and 15%–40% of cells, respectively.Source: §9 · p. 14

10. Image Augmentations

The image-augmentation approach combines geometric variation with texture integration, and the authors report fewer artifacts than conventional strategies.

10. Image Augmentations

  • The augmentation approach uses randomized affine transforms for geometric variation and Poisson blending for texture integration.Source: §10 · p. 14
  • The authors report realistic transformations with significantly fewer artifacts than conventional augmentation strategies.They describe the approach as bridging the domain gap while preserving pretrained visual understanding.Source: §10 · pp. 14–15, Table 2 · p. 15

11. Common Mistakes of LVLMs in Puzzle Solving

The reported mistakes span both visual perception and puzzle-solving behavior, including misreading layouts or components and failing to maintain or check solution state.

11.1. Common Mistakes in LVLMs’ Perception

  • LVLMs misread grid layouts, confuse component roles such as Killer-Sudoku cage sums with cell values, and misunderstand visual components in Thermometers and Battle-Ships.In Sudoku, performance improves with more hints; the authors suggest difficulty detecting gridlines may contribute to layout errors.Source: §11.1 · p. 15

11.2. Common Mistakes in Pre-Trained LVLMs’ Puzzle-Solving

  • Puzzle-solving mistakes include sequential solving without prioritizing constrained cells, forgetting prior actions, failing to detect errors or backtrack, and repetitive generation.Examples include accepting a row with duplicated numbers and reasoning until the context limit is exhausted.Source: §11.2 · p. 15

12. Generalization Capability of SFT Models

SFT models trained on one puzzle perform much worse on other puzzles, but transfer remains higher between puzzles with similar rules. Both S-SFT and R-SFT show this pattern.

12. Generalization Capability of SFT Models

  • Performance on other puzzles is significantly lower than on the training puzzle, with transfer depending on how different the puzzles’ rules are.The comparison uses Sudoku, Odd-Even Sudoku, Renzoku, and Aquarium, which have increasingly different rules.Source: §12 · p. 16, Figure 12 · p. 16
  • Sudoku-trained models retain high puzzle-solving rates on similar-rule Odd-Even Sudoku, but achieve zero success on Aquarium.The passage reports the same generalization pattern for both S-SFT and R-SFT.Source: §12 · p. 16
  • Both S-SFT and R-SFT show similar generalization behavior across the four puzzles.The passage notes that recent work has also discussed limitations of SFT generalization [11] [65].Source: §12 · p. 16

13. Reinforcement Learning with Text Input

Text-input results show near-zero performance on most medium puzzles and zero accuracy on hard puzzles. The evaluation also examines difficulty, clue count, SFT performance, and training objectives.

13. Reinforcement Learning with Text Input

  • GRPO optimizes groups of trajectories with a KL penalty, normalizes advantages across rollouts, and combines solution-success and structured-format rewards.The success and format rewards are balanced by separate weights.Source: §13 · p. 16
  • Sample screenshots cover easy, medium, and hard puzzles; some games lack hard levels because their rules constrain grid sizes, as in Vanilla Sudoku.The next Vanilla Sudoku grid after 9 × 9 is 16 × 16, which the passage says is too large.Source: Table 3 · p. 17, Table 4 · p. 18, Table 5 · p. 19
  • Query templates specify indexing conventions and distinguish undefined cells marked “*” from deliberately empty cells marked “e.”Source: Table 6 · p. 20, Table 7 · p. 21
  • At medium level, no LVLM solves any puzzle above 10%; hard-level perception and solving accuracy are lower than medium, while closed-source models outperform open-source models at easy level.The easy-level comparison is reported for evaluation without chain-of-thought prompting.Source: Figure 13 · p. 24, Figure 14 · p. 25, Figure 15 · p. 26
  • The evaluation covers holistic perception and solution success, cell-level perception accuracy, and step-level rule understanding, with action validity based only on immediate rule violations.Source: Figure 18 · p. 29, Figure 19 · p. 30, Figure 20 · p. 31
  • On easy-level evaluations, more clues generally correspond to higher perception and puzzle-solving accuracy, including cell-level perception accuracy.Source: Figure 24 · p. 35, Figure 25 · p. 36
  • On medium text-input puzzles, almost all LVLMs are near zero on most puzzles, but Claude-3.5 solves Field-Explore at 93%; hard-level text-input accuracies are all zero.The authors interpret Field-Explore’s local-only rules as making it easier than other medium-level puzzles.Source: Figure 27 · p. 38
  • On easy puzzles, both S-SFT and R-SFT improve perception and puzzle solving, perform comparably, and have different puzzle-specific advantages.S-SFT does better on some puzzles, including Nonogram and Trees-and-Tents; R-SFT does better on Binairo and Kakuro.Source: Figure 28 · p. 39

Sudoku - Easy - W/ CoT - Vision Input

The easy Sudoku examples with vision input and chain-of-thought show failures in both reading the board and completing it under Sudoku constraints. Models variously misread grid dimensions, misperceived clues, or returned incorrect solutions.

Sudoku - Easy - W/ CoT - Vision Input

Sudoku - Easy - W/ CoT - Text Input

In these easy Sudoku text-input examples, outputs range from answers marked correct to incorrect completions, including malformed boards and repetitive reasoning.

Sudoku - Easy - W/ CoT - Text Input

  • QVQ-72B and GPT-4o-Mini answers are marked correct, while several other sampled model answers are marked incorrect.The incorrect examples include Claude-3.5, Gemini-1.5-Pro, Gemini-1.5-Flash, and Qwen models.Source: Sudoku - Easy - W/ CoT - Text Input · pp. 54–59
  • Claude-3.5 produces repetitive reasoning and cannot stop, and its final Sudoku answer is marked incorrect.The output’s remark identifies repetitive reasoning and inability to stop.Source: Sudoku - Easy - W/ CoT - Text Input · p. 55
  • Gemini-1.5-Pro gives a Sudoku answer marked incorrect despite laying out row, column, and block deductions.Its output proposes a completed grid, but the filtered answer check is false.Source: Sudoku - Easy - W/ CoT - Text Input · p. 56
  • Gemini-2.0-Flash and Qwen2.5-72B outputs are also marked incorrect on the example puzzle.The Flash output has an incorrect answer; Qwen2.5-72B's answer is marked false.Source: Sudoku - Easy - W/ CoT - Text Input · pp. 57, 59
  • One CoT text-input response returns an invalid board size, and its answer is marked incorrect.The filtered output reports “Remarks: invalid board size.”Source: Sudoku - Easy - W/ CoT - Text Input · p. 63

Battle-Ships - Easy - W/ CoT - Vision Input

The sampled easy Battle-Ships vision-input outputs show perception and answer errors, including misread clues and incorrect ship placements.

Battle-Ships - Easy - W/ CoT - Vision Input

  • QVQ-72B, Gemini-2.0-Flash, and Gemini-1.5-Flash outputs are marked incorrect on the Battle-Ships example.QVQ-72B and Gemini-2.0-Flash have incorrect perception and answers; Gemini-1.5-Flash also has both checks false.Source: Battle-Ships - Easy - W/ CoT - Vision… · pp. 65, 67, 68
  • Qwen2.5-7B and Qwen2-7B outputs are marked incorrect, with the former misreading the initial grid and the latter failing perception.The filtered output for Qwen2-7B marks perception and answer false; Qwen2.5-7B's perception is also marked false.Source: Battle-Ships - Easy - W/ CoT - Vision… · pp. 70–71
  • LLaVA-Qwen2-7B and LLaVA-Mistral-7B outputs are marked incorrect after proposing ship placements that conflict with the task clues.The displayed answers fail the answer check, and LLaVA-Mistral-7B also has incorrect perception.Source: Battle-Ships - Easy - W/ CoT - Vision… · pp. 73–74

Field-Explore - Medium - W/ CoT - Vision Input

The qualitative examples show that models often misread medium Field-Explore boards and fail to solve them, even when they provide step-by-step reasoning. Some outputs also repeat the input board as the answer or make unsupported deductions.

Field-Explore - Medium - W/ CoT - Vision Input

Gemini-1.5-Flash

  • A GPT-4o-Mini response reproduces its initial board as the solution, while its perception and answer are marked incorrect.This illustrates failure to make progress beyond the presented state.Source: Gemini-1.5-Flash · p. 82
  • Other outputs misread the board itself, including a response that interprets a numbered grid as sequentially numbered cells.The corresponding perception and answer are both marked incorrect.Source: Gemini-1.5-Flash · p. 91

Llama 3.2 Instruction Vision after Reasoning-SFT (Successful Example)

The appendix examples contrast a reasoning-SFT output that completes a puzzle with an example that perceives the board correctly but gives a wrong answer. A text-input reinforcement-learning example shows accuracy reward lagging behind perception and format rewards, then reaching 1.00 in a later output.

Llama 3.2 Instruction Vision after Reasoning-SFT (Successful Example)

Llama 3.2 Instruction Vision after Reasoning-SFT (Failure Example)

  • The failure example gets perception correct but answer incorrect because step 2 wrongly rules out 4 for cell (1,2).The remarks say step 1 was correct and effective, but identify step 2 as the error.Source: Llama 3.2 Instruction Vision after… · p. 93

Reinforcement Learning Training Process with Text Input

  • During text-input reinforcement learning, perception and format rewards increase first, while accuracy reward rises slowly afterward.The remarks distinguish puzzle solving from producing a format and copying perception from the prompt.Source: Reinforcement Learning Training… · p. 94
  • Before RL, the text-input example earns format reward 1.00 but accuracy and perception rewards 0.00.Its generated answer does not match the correct solution.Source: Reinforcement Learning Training… · p. 95
  • After RL, the text-input example earns 1.00 for accuracy, format, and perception rewards.The output gives a completed 4×4 grid.Source: Reinforcement Learning Training… · p. 96
Loading 2503.23064v2…