PaperlayerView exact revision on arXiv

Source-linked AI summary

Game2Real-Bench: Do Toy Games Reveal and Transfer to Real-World Visual Reasoning?

Qingan Liu, Juntong Feng, Yue Zhu, Yujie Cheng, Xinzhe Han, Shuai Hao, Yunzhi Zhuge, Haiwen Diao, Huchuan Lu

arXiv:2610.18834v1cs.CV

TL;DR

Whether toy games reflect real-world visual reasoning and support transfer remains unclear. Game2Real-Bench aligns game and real-world tasks by shared reasoning requirements and evaluates 25 VLMs. Performance is strongly correlated across settings, while one-shot transfer varies by capability.

  • Problem

    It remains unclear whether abilities measured in toy games correspond to real-world visual reasoning and whether game examples support transfer across models and capabilities.

    Source: §1 · pp. 1–2

  • Method

    Game2Real-Bench links four capability groups across 1,000 verified game samples and 400 real-world samples selected by shared reasoning requirements.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    Pearson and Spearman correlations of 0.935 indicate strong game–real performance correspondence, while one-shot transfer benefits constraint reasoning for several models and lowers fine-grained perception for all four tested models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Toy games are informative probes of real-world visual reasoning, but their usefulness as demonstrations depends on capability and model.

    Source: §1 · p. 2

    The benchmark’s 18 toy-game tasks and 400 real-world samples do not span the full range of visual reasoning in natural environments.

    Source: §5 · p. 9

Abstract

from arXiv · show

Visual reasoning is an important capability for vision-language models (VLMs) to understand visual evidence, spatial structure, constraints, and state changes. Existing evaluations mainly rely on real-world data or procedurally generated tasks. Real-world data preserve natural complexity, while procedural games provide controllable generation and verifiable solutions. However, it remains unclear whether abilities measured in toy games correspond to real-world visual reasoning and whether game examples can support transfer. We introduce Game2Real-Bench, a benchmark that studies these connections across four capability groups, including fine-grained perception, spatial reasoning, constraint reasoning, and sequential planning. We construct 1,000 procedurally generated game samples with executable verification and curate 400 real-world samples aligned by shared reasoning requirements rather than visual similarity. We evaluate 25 representative VLMs and observe strong game-to-real consistency, with Pearson and Spearman correlations of 0.935. We further study one-shot game-to-real transfer on four models. Constraint reasoning improves for several models, while fine-grained perception declines with game demonstrations. We also construct Game2Real-Dataset with nearly 24K verified game samples across 18 tasks and 65 QA families. Our benchmark, dataset, and code are available at https://github.com/QingAnLiu/Game2Real-Bench.

1 INTRODUCTION

Game2Real-Bench asks whether toy games reflect real-world visual reasoning and whether solved game examples transfer. It links game and real-world tasks by shared reasoning requirements, evaluates VLMs, and finds strong cross-setting correspondence with selective transfer effects.

1 INTRODUCTION

  • Existing evaluations and procedural-training studies leave open how game-based assessment relates to real-world reasoning across models and capabilities.The paper frames shared evaluation by reasoning requirements as a way to examine this relationship and the usefulness of game examples.Source: §1 · pp. 1–2
  • The benchmark contains 1,000 verified toy-game samples and 400 real-world samples aligned by shared reasoning requirements across four capabilities.The four groups are fine-grained perception, spatial reasoning, constraint reasoning, and sequential planning.Source: §1 · p. 2
  • Pearson and Spearman correlations of 0.935 show strong game–real performance correspondence across 25 VLMs, while one-shot transfer effects vary by capability.Constraint reasoning improves for several models, while fine-grained perception declines for all four models tested.Source: §1 · p. 2
  • Game2Real-Dataset adds nearly 24K programmatically verified samples across 18 toy-game tasks and 65 QA families.The dataset includes executable generators and task-specific verifiers for scalable visual-reasoning research.Source: §1 · p. 2

2 RELATED WORK

Prior work uses games and procedural environments for multimodal reasoning evaluation and training, but usually assesses real-world relevance indirectly. Game2Real-Bench links game and real-world tasks by shared reasoning requirements to study correspondence and transfer.

2.1 GAMES AS TESTBEDS FOR VISUAL REASONING

  • Game2Real-Bench makes game-to-real correspondence explicit by organizing generated game tasks and real-world samples around shared reasoning requirements.This supports analysis of cross-setting performance correspondence and transfer at the capability level.Source: §2.1 · p. 2
  • Prior interactive game benchmarks assess perception, planning, memory, and decision making, while other work uses games and procedural tasks for reasoning supervision and transfer studies.The cited examples include BALROG, LMGame-Bench, VideoGameBench, Game-World, Game-RL, Play to Generalize, and VBVR-Pro.Source: §2.1 · p. 2

2.2 BENCHMARKS FOR VISUAL REASONING

  • Visual-reasoning benchmarks span general-purpose, vision-centric, structured, and synthetic or procedural settings, with different emphases on knowledge, perception, spatial structure, and constraints.Prior work includes real-world benchmarks, foundational visual-skill evaluations, structured reasoning tasks, and synthetic scenes such as CLEVR.Source: §2.2 · p. 3
  • Whether capabilities reflected by procedurally generated games correspond to real-world visual reasoning remains insufficiently understood.Game2Real-Bench combines both settings under a shared capability framework to examine performance consistency and game-to-real transfer.Source: §2.2 · p. 3

3 GAME2REAL-BENCH

Game2Real-Bench is built by designing capability-aligned toy tasks, generating and verifying game samples, and curating real-world samples with corresponding reasoning demands. Its benchmark combines both settings, alongside a larger verified game dataset.

3 GAME2REAL-BENCH

  • The three-stage construction designs 18 toy-game tasks and 65 QA families, generates verified samples, and curates 400 real-world samples aligned to the games’ reasoning operations.The resulting benchmark has 1,400 samples, including 1,000 independently generated game samples.Source: §3 · p. 3

3.1 TASK DESIGN

  • Tasks are organized around four capabilities, with each task assigned to the group best reflecting its primary demand despite possible overlap.The groups cover fine-grained perception, spatial reasoning, constraint reasoning, and sequential planning; QA families specify reasoning targets, formats, task notation, and output conventions.Source: §3.1 · pp. 3–4

3.2 PROCEDURAL GENERATION

  • Procedural generation targets visual grounding, diversity, and verifiability, using task-specific generators and deterministic solvers or verifiers.Generators create observations, questions, and answers from sampled states; verification checks solutions against executable rules, including uniqueness or optimality where required.Source: §3.2 · pp. 4–5

3.3 REAL-WORLD CURATION

  • Real-world samples are selected or constructed to match the toy tasks’ reasoning requirements, then checked for consistency and manually reviewed.Ambiguous samples and those with insufficient visual evidence are removed.Source: §3.3 · p. 5

3.4 BENCHMARK STATISTICS

  • Game2Real-Dataset contains 23,893 verified samples across 18 tasks and 65 QA families, with open-ended and single-choice items and 36,441 images.The evaluation benchmark contains 1,400 samples: 1,000 toy-game and 400 real-world samples.Source: §3.4 · p. 5

4 EXPERIMENTS

The experiments evaluate 25 VLMs across toy-game and real-world tasks, then examine capability-level performance, game-to-real consistency, one-shot transfer, thinking, and recurring failure patterns. Game and real-world scores are strongly correlated, but transfer effects vary by capability and model.

4.1 SETUP

  • The zero-shot evaluation covers 25 VLMs, using shared instructions, inputs, and questions, with deterministic game evaluators and reported capability and overall scores.The evaluated open-weight models span 2B–35B parameters; proprietary models use their official APIs.Source: §4.1 · p. 5, Table 1 · p. 6

4.2 MAIN RESULTS

  • 69.8% is GPT-5.6-Sol’s highest overall score; Gemini-3.1-Pro-Preview scores 59.6%, Claude-Opus-4.8 53.7%, and 16 of 22 open-weight models score below 30%.Qwen3.8-27B is the highest-scoring open-weight model at 52.6%.Source: §4.2 · p. 6, Table 1 · p. 6
  • Toy-game tasks are generally harder and separate models more clearly than real-world tasks, where performance ranges are more compressed.The best proprietary/open-weight gaps on toy games are 16.3, 39.0, and 20.4 points for fine-grained perception, constraint reasoning, and sequential planning, respectively.Source: §4.2 · pp. 6–7
  • Sequential planning is the strongest capability for 15 models on toy games and 17 on real-world tasks, while other capabilities’ relative difficulty shifts between settings.Fine-grained perception is lowest on toy games but among the stronger real-world capabilities; constraint reasoning remains difficult, especially for stronger models.Source: §4.2 · p. 7

4.3 FINE-GRAINED ANALYSIS

  • One-shot game demonstrations raise constraint reasoning by 4.89 points on average but lower fine-grained perception by 4.92 points for all four models.Overall accuracy decreases by 1.34 points on average; Qwen3.5-9B gains 1.49 points overall, while the 2B and 4B models decrease.Source: §4.3 · pp. 7–8, Figure 3 · p. 7
  • Across 25 models, game and real-world scores correlate at r = 0.935 (Pearson) and ρ = 0.935 (Spearman).Their model-level performance patterns and relative rankings are largely preserved across the two settings.Source: §4.3 · p. 8, Figure 4 · p. 8
  • Thinking improves all four tested models’ scores on the 1,400-sample benchmark, with gains of 6.2, 6.0, 11.1, and 41.5 points for Qwen3.5-2B, Step3-VL-10B, Gemma4-31B-it, and GPT-5.6-Sol, respectively.The larger gains may reflect stronger models’ greater ability to use extra reasoning computation.Source: §4.3 · p. 8, Figure 5 · p. 8
  • Failure cases span incorrect spatial reconstruction, loss of relations during deduction, and missed state transitions, including a missed Mancala capture.Examples are shown for Block Counting, Hexakai, and Mancala.Source: §4.3 · pp. 8–9, Figure 6 · p. 9

5 LIMITATIONS

Game2Real-Bench covers four capability groups and connects toy-game tasks with corresponding real-world visual reasoning settings, but its scope and transfer study remain limited.

5 LIMITATIONS

  • The benchmark covers 18 toy-game tasks and 400 real-world samples, not the full range of visual reasoning in natural environments.Future extensions could add task types and richer visual settings while preserving game–real alignment.Source: §5 · p. 9
  • Transfer is studied only through one-shot in-context learning, while the nearly 24K-sample dataset could support multi-shot prompting, fine-tuning, and reinforcement-learning studies.Source: §5 · p. 9
  • Model-level performance correlation does not establish that toy games and real-world tasks use the same internal reasoning processes.Source: §5 · p. 9

6 CONCLUSION

Game2Real-Bench pairs controlled toy games with real-world visual reasoning tasks to examine performance correspondence and transfer. Across models, performance is strongly correlated, while one-shot transfer varies by capability.

6 CONCLUSION

  • Across 25 VLMs, game and real-world performance is strongly correlated, while game tasks are generally harder and distinguish models more clearly.Capability-level results also show different difficulty profiles across settings despite shared reasoning requirements.Source: §6 · p. 9
  • One-shot game-to-real transfer benefits constraint reasoning for several models, but fine-grained perception consistently declines.Source: §6 · p. 9
  • The study also examines thinking mode and analyzes representative failure cases.Source: §6 · p. 9
  • Game2Real-Bench and Game2Real-Dataset provide a verifiable evaluation framework and scalable data resource for studying game-to-real transfer.Source: §6 · p. 9

A DETAILED BENCHMARK SPECIFICATION

The benchmark organizes toy-game reasoning tasks into capability groups and QA families, with an independently generated evaluation set and a larger verified dataset.

A.1 TASK TAXONOMY AND QA FORMULATION

  • The 18 tasks are grouped by primary demand into fine-grained perception, spatial reasoning, constraint reasoning, and sequential planning, though solving them can involve multiple capabilities.Source: §A.1 · p. 13
  • A task defines rules and solution criteria, while each QA family specifies a reasoning target and output format for samples with visual observations, questions, and verified answers.Instructions also specify notation, coordinate systems, and answer conventions.Source: §A.1 · p. 13
  • Game2Real-Dataset contains 23,893 verified samples across 65 QA families; the evaluation set contains 1,000 samples across all 18 tasks.Evaluation samples comprise 600 open-ended and 400 single-choice questions.Source: §A.1 · p. 13, Table 2 · p. 13

A.2 TASK DEFINITIONS AND REPRESENTATIVE EXAMPLES

The appendix illustrates the benchmark’s task range through representative questions and describes how procedural generation produces, renders, and verifies game instances.

A.2 TASK DEFINITIONS AND REPRESENTATIVE EXAMPLES

  • Representative examples retain reference answers while shortening questions for readability; visual inputs are omitted and sample IDs point to complete records.Source: §A.2 · p. 14

A.2.1 FINE-GRAINED PERCEPTION

  • Fine-grained perception examples test connector matching, counting occluded animals, contour correspondence, color remapping, and cube-net sticker reading.Source: §A.2.1 · p. 14, §A.2.2 · p. 14

A.2.2 SPATIAL REASONING

  • Spatial reasoning examples require inferring a missing orthographic view, counting cube piles with hidden supports, and tracking cube faces through rolls.Source: §A.2.2 · pp. 14–15

A.2.3 CONSTRAINT REASONING

  • Constraint tasks include completing a hexagonal grid, satisfying shaded-run and fleet clues, placing tetrominoes, and planning pipe rotations under locking rules.Source: §A.2.3 · p. 15

A.2.4 SEQUENTIAL PLANNING

  • Sequential-planning examples require optimal mirror configurations, shortest routes through keys and moving bodies, game-state execution, and vehicle moves.Source: §A.2.4 · pp. 15–16

A.3 QA-FAMILY OVERVIEW

  • The 65 QA families vary query scope and output, covering local attributes, aggregate counts, complete configurations, intermediate states, and action sequences.Some families support both open-ended and single-choice variants.Source: §A.3 · p. 17, Table 3 · p. 17

B BENCHMARK CONSTRUCTION DETAILS

  • Construction details use Rolling Cube to illustrate generator and verifier tracking, and an IKEA assembly example to show aligned instructions and video frames forming a multi-image question.Source: B BENCHMARK CONSTRUCTION DETAILS · p. 18

B.1 PROCEDURAL GENERATION

  • Rolling Cube generation samples face patterns and valid paths, renders the path and initial faces as separate inputs, then selects checkpoints and constructs distractors that vary face state or orientation.Duplicate distractor grids are removed before options are selected and shuffled.Source: §B.1 · p. 18

B.2 TASK-SPECIFIC VERIFICATION

The appendix describes verification and evaluation procedures for game tasks, real-world alignment, model inputs, and answer scoring. It also details how real-world samples and game demonstrations are curated and presented.

B.2 TASK-SPECIFIC VERIFICATION

  • Rolling Cube verification replays the ordered moves from initial face grids, recomputes the Top grid, and accepts only a uniquely valid option matching the exported answer.The checker compares every position in each parsed 3×3 candidate grid; candidate A alone matches the recomputed arrangement in the example.Source: §B.2 · pp. 18–19, Table 4 · p. 19

B.3 REAL-WORLD CURATION AND GAME-TO-REAL ALIGNMENT

  • Real-world curation aligns samples by shared reasoning requirements, not one-to-one task matches, and primary task annotations cover 14 of the 18 toy tasks.The 400 selected samples undergo manual review; the full-suite correlation is not an estimate of individual task-link predictive value.Source: §B.3 · pp. 19, 21

C EVALUATION PROTOCOL; C.1 MODEL AND INFERENCE CONFIGURATIONS; C.2 PROMPTING AND INPUT FORMATTING

  • Evaluation uses shared prompts and model-specific interfaces, with paired Qwen zero-shot and one-shot runs differing only in demonstration context.One-shot demonstrations contain a game question and its verified final answer, while the real-world target remains unchanged.Source: §C.1 · p. 22, C EVALUATION PROTOCOL · p. 22, §C.2 · p. 23

C.2 PROMPTING AND INPUT FORMATTING

  • Prompts distinguish reference images from answer-option images and preserve task-specific answer formats, while withholding reference answers and evaluator metadata.For example, candidates supplied as separate images are mapped to option letters; IKEA frame selections remain JSON arrays.Source: §C.2 · pp. 22–23

C.3 TOY-GAME ANSWER NORMALIZATION AND SCORING

  • Toy-game normalization follows each sample’s schema and does not repair omitted values or incorrect solutions.Evaluators check complete requested fields, preserve meaningful action order, and replay tasks with executable rules; alternative valid solutions receive full credit when permitted.Source: §C.3 · p. 23

C.4 REAL-WORLD EVALUATION PROTOCOLS

Real-world predictions are extracted and scored under task-specific rules, then aggregated across samples and capabilities. The appendix also specifies the analyses that use these scoring procedures.

C.4 REAL-WORLD EVALUATION PROTOCOLS

  • The 400 real-world samples are scored with binary correctness except for SpatiaLQA, which contributes a normalized graph score.Responses are first converted to each sample’s required output type without access to reference answers, then passed to the appropriate evaluator.Source: Table 8 · p. 24, §C.4 · p. 24
  • Real-world evaluation applies distinct exact-match or structured criteria to choices, counts, frame selections, rankings, routes, and SpatiaLQA plans.SpatiaLQA uses rule-based step matching at threshold 0.72 and averages content and precondition F1; this replaces the official GPT-4o correspondence stage.Source: §C.4 · pp. 24–25
  • The joint benchmark score averages all 1,400 samples, weighting toy-game and real-world sets by 1,000/1,400 and 400/1,400, respectively.Each real-world capability has 100 samples, so its overall score is the unweighted mean of the four capability scores.Source: §C.4 · p. 25

D DETAILS OF ANALYSIS EXPERIMENTS

  • The analysis appendix retains the benchmark partitions and scoring procedures while detailing correlation, one-shot transfer, and thinking-mode experiments.It describes data correspondence, demonstration selection and prompting, and inference controls.Source: D DETAILS OF ANALYSIS EXPERIMENTS · p. 26

D.1 CROSS-SETTING CORRELATION

The cross-setting analysis pairs each model’s scores on the full toy-game and real-world sets, then measures their score and rank association. The reported coefficients describe model-level correspondence, not shared internal reasoning.

D.1 CROSS-SETTING CORRELATION

  • Each observation pairs one model’s score on all 18 toy tasks with its score on the full real-world set, including 400 samples.The toy-game aggregate uses sample-count weighting, whereas an unweighted mean of its four capability scores would produce a different statistic.Source: §D.1 · pp. 26–27
  • r = 0.9345988 and ρ = 0.9346154, each rounded to 0.935, quantify Pearson score correlation and Spearman rank correlation across the 25 models.The fitted line in Figure 4 visualizes the trend but is not part of either correlation calculation.Source: §D.1 · p. 27
  • The correlation sample size is 25 models, and the coefficients do not establish that models use identical internal reasoning processes across settings.The analysis joins the two evaluation scores by model identity.Source: §D.1 · p. 27

D.2 ROBUSTNESS OF MODEL-LEVEL PERFORMANCE CORRESPONDENCE

Robustness analyses test whether model-level correspondence changes under alternative aggregation, model coverage, and real-world task composition. The association remains strong across these checks, although its magnitude varies with task composition.

D.2 ROBUSTNESS OF MODEL-LEVEL PERFORMANCE CORRESPONDENCE

  • r = 0.935 under equal toy-game capability weighting and r = 0.960 for the 22 open-weight models under that aggregation.The first estimate’s 95% confidence interval is [0.903, 0.974].Source: §D.2 · p. 28
  • r = 0.964 with equal source weighting; excluding SpatiaLQA yields r = 0.946 with sample averaging and r = 0.933 with equal-capability averaging.The correspondence remains strong when source weights change and when continuous graph scores are excluded.Source: §D.2 · p. 28, Table 11 · p. 28
  • For non-single-choice samples, r = 0.857 with sample averaging and r = 0.766 with capability averaging, as task composition and weights differ.The 164 samples include 100 constraint-reasoning questions, while perception and planning samples come only from GQA and ReasonMap, respectively.Source: §D.2 · p. 28, Table 11 · p. 28
  • Across leave-one-source-out analyses, Pearson correlations range from 0.940 to 0.966 with sample averaging and 0.928 to 0.966 with capability averaging.The analysis retains the same 22 models while removing each of 11 real-world sources in turn.Source: §D.2 · p. 28, Table 12 · p. 29
  • A joint bootstrap over 22 models and 400 real-world samples gives an interval of [0.881, 0.981], while toy-game capability scores remain fixed.The question-resampling interval does not include variation from resampling independently generated toy-game questions.Source: §D.2 · p. 29
  • Together, the checks support stable model-level correspondence across aggregation rules, model coverage, and source removals, but its magnitude varies with retained task composition.The one-shot experiment separately measures score changes after randomly assigned solved game demonstrations are added to unchanged real-world targets.Source: §D.2 · p. 29

D.3 ONE-SHOT GAME-TO-REAL IN-CONTEXT LEARNING

The one-shot study tests whether verified game examples transfer to real-world questions, varying demonstration assignments while holding targets and model configurations fixed. Constraint reasoning benefits for three models, but fine-grained perception and average overall scores remain below zero-shot baselines; a separate comparison reports main and non-thinking configurations.

D.3 ONE-SHOT GAME-TO-REAL IN-CONTEXT LEARNING

  • Three randomized assignments select one verified game example per real-world target from a fixed pool of 168 demonstrations, with a balanced overall capability mix.Each seed is reused across the four models, and each evaluation uses the same 400 real-world targets; a demonstration may be reused across targets.Source: §D.3 · pp. 29–30
  • Each one-shot prompt presents a solved game example in a separate conversation turn before the unchanged real-world target prompt.The demonstration includes its verified final answer but no intermediate solution trace; target reference answers and evaluator metadata are not supplied to the model.Source: §D.3 · pp. 29–30, Figure 7 · p. 30
  • The reported change compares the three-assignment mean with the same model’s zero-shot score by real-world target capability, and includes the full prompting context.Overall scores average the four equally sized capabilities; the analysis does not separate game content from formatting cues or added context.Source: §D.3 · p. 30
  • Constraint reasoning improves for the 2B, 9B, and 27B models but declines for the 4B model; averaged fine-grained perception and Overall-400 remain below zero-shot baselines.Qwen3.5-9B gains 14.56–16.64 points in constraint reasoning; four-model average gains are 3.76, 5.27, and 5.65 points for seeds 11, 23, and 37.Source: §D.3 · p. 31, Table 13 · p. 31

D.4 THINKING-MODE CONFIGURATION

  • Main and non-thinking configurations are compared zero-shot on the same 1,400 benchmark samples, with gains reflecting complete inference settings.The comparison covers Qwen3.5-2B, Step3-VL-10B, Gemma4-31B-it, and GPT-5.6-Sol.Source: §D.4 · pp. 31–32, Table 15 · p. 32

Related papers

Loading 2610.18834v1…