PaperlayerView exact revision on arXiv

Source-linked AI summary

SPHINX: A Synthetic Environment for Visual Perception and Reasoning

Md Tanvirul Alam, Saksham Aggarwal, Justin Yang Chae, Nidhi Rastogi

arXiv:2511.20814v2cs.CVcs.AIcs.LG

TL;DR

LVLMs remain challenged by visual reasoning, and existing benchmarks do not systematically evaluate core perceptual primitives. SPHINX generates controlled tasks with verifiable answers, and the authors report that RLVR improves performance on SPHINX and external visual-reasoning benchmarks.

  • Problem

    LVLMs must jointly parse images and integrate them with language, while existing benchmarks do not systematically test core perceptual primitives such as symmetry detection and mental rotation.

    Source: §1 · pp. 1–2

  • Method

    SPHINX procedurally generates visual reasoning tasks with verifiable ground truth, and the authors apply RLVR using a separate SPHINX-derived training set.

    Source: §1 · p. 2

  • Results

    RLVR yields consistent gains on in-distribution tasks, improved generalization to held-out task families, and measurable improvements on external benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    SPHINX provides a synthetic environment for evaluating visual perception and reasoning, while RLVR shows promise for improving LVLM performance and generalization.

    Source: §8 · p. 8

    Even after RLVR training, in-distribution accuracy remains below 50%, and sparse correct traces can make target solutions effectively unreachable during training.

    Source: §7 · p. 8

Abstract

from arXiv · show

We present Sphinx, a synthetic environment for visual perception and reasoning that targets core cognitive primitives. Sphinx procedurally generates puzzles using motifs, tiles, charts, icons, and geometric primitives, each paired with verifiable ground-truth solutions, enabling both precise evaluation and large-scale dataset construction. The benchmark covers 25 task types spanning symmetry detection, geometric transformations, spatial reasoning, chart interpretation, and sequence prediction. Evaluating recent large vision-language models (LVLMs) shows that even state-of-the-art GPT-5 attains only 51.1% accuracy, well below human performance. Finally, we demonstrate that reinforcement learning with verifiable rewards (RLVR) substantially improves model accuracy on these tasks and yields gains on external visual reasoning benchmarks, highlighting its promise for advancing multimodal reasoning.

1. Introduction

LVLMs face a more demanding visual-reasoning problem than text-only systems, while existing benchmarks do not systematically test core perceptual primitives. SPHINX addresses this gap with procedurally generated tasks and verifiable answers, and the authors report that RLVR improves performance across SPHINX and external benchmarks.

1. Introduction

  • LVLMs must jointly parse visual inputs and integrate them with language, while existing benchmarks emphasize perception over reasoning.The introduction contrasts this challenge with text-based systems and notes that models can reduce visual inputs to text and rely on language reasoning.Source: §1 · pp. 1–2
  • SPHINX targets under-evaluated primitives such as symmetry detection, mental rotation, and structured pattern matching with controlled tasks and unambiguous ground-truth solutions.The framework also supports scalable dataset generation for reinforcement learning.Source: §1 · p. 2
  • SPHINX comprises 25 tasks in five broad categories and supports comparisons between human and LVLM performance.The authors describe it as the largest-scale synthetic environment designed for these tasks, to their knowledge.Source: §1 · p. 2
  • RLVR on a separate SPHINX-derived training set yields consistent in-distribution gains, better generalization to held-out task families, and measurable improvements on external benchmarks.The introduction presents these results as a contribution of the work.Source: §1 · p. 2

2. SPHINX Design

SPHINX separates visual appearance, spatial layout, and reasoning rules so they can be flexibly combined, varied, and evaluated with verifiable ground truth. Its motifs, tilings, and task families support controlled variation across visual reasoning problems.

2. SPHINX Design

  • SPHINX separates appearance, spatial layout, and reasoning rules into composable modules that can be varied independently.The framework generates visual reasoning data with verifiable ground truth.Source: §2 · p. 2, §2.1 · p. 2

2.1. Design Principles

  • Deterministic checkers certify rule satisfaction and a single correct answer, while weighted sampling and difficulty knobs control task mixtures and complexity.Each sample also exports an image, prompt, answer, distractors when applicable, and analysis-ready metadata.Source: §2.1 · p. 2

2.2. Building Blocks

  • Parameterized motifs and canonical geometric primitives provide varied visual elements, while tilings specify cell layouts, adjacency, and coloring regimes.The described tilings include square, triangular, hexagonal, rhombille, and circle variants.Source: §2.2 · p. 2

2.3. Tasks

  • Geometric and counting tasks cover spatial relations, shape measures, chart proportions, Venn regions, composite figures, and lines on tilings.Examples include positional counting, shape sorting, pie-chart sorting, chart comparison, and counting tile-line intersections.Source: §2.3 · p. 3
  • Symmetry and pattern tasks test mirror symmetry, symmetry completion, and odd-one-out patterns in frieze and wallpaper groups.Sequence and transformation tasks ask models to identify transformation results, infer transformations between tiles, and predict missing sequence elements.Source: §2.3 · pp. 3–4
  • Topological and graph tasks cover region geometry, connected components, shortest paths, missing tiles, and composition on tilings or grids.The design groups these among 25 tasks across five broad families.Source: §2.3 · pp. 2, 4

3. Benchmark

Across the benchmark, GPT-5 has the highest reported overall LVLM accuracy, but remains below human accuracy. The evaluation covers proprietary and open-source models on 2,500 questions spanning 25 tasks.

3. Benchmark

  • The benchmark contains 2,500 questions, with 100 instances per task, and evaluates six proprietary and 13 open-source vision-language models.The open-source models range from 3B to 235B parameters.Source: §3 · p. 4
  • 51.1% average accuracy is GPT-5’s best overall result, yet it falls 24.7% below human accuracy.GPT-5 leads in four reasoning categories, while Claude4.5-Sonnet has the best counting score at 40.2%.Source: §3 · p. 4, Table 1 · p. 5
  • 41.3% accuracy makes Qwen3-VL-235B the strongest open-source model in the evaluation, above Claude4.5-Opus and smaller open-weight baselines.GPT-5 Mini’s overall accuracy is 4.0% below GPT-5’s.Source: §3 · p. 4

4. Analysis

The analysis finds visual-reasoning tasks challenging for humans and models alike, identifies task-specific differences between GPT-5 and GPT-5 Mini, and shows that accuracy falls as task complexity rises.

4.1. Human Evaluation

  • Without pencil-and-paper aids, participants also performed less well on mental-arithmetic and grid-enumeration tasks, so their scores on counting-heavy tasks may underestimate their ability.Tiles Composition had low accuracy and the longest average response time, indicating cognitive load rather than inattention.Source: §4.1 · p. 5
  • Participants’ prior familiarity with visual-reasoning puzzles strongly correlates with accuracy, and tasks involving mental rotation, multi-step spatial reasoning, or fine-grained symmetry classification challenge even humans.Frieze Group Odd-One-Out accuracy was 48.4%; the correct odd-one-out had a different symmetry class despite another option looking more visually distinct.Source: §4.1 · p. 5

4.2. GPT-5 vs. Humans

  • GPT-5’s largest deficits are on tile-based tasks and Transform Similarity Identify, while it matches or exceeds humans on Shape Counting, chart tasks, and Tiles Composition.Tiles Composition is also difficult for participants, who must mentally rotate and assemble pieces without trivial distractors.Source: §4.1 · p. 5, §4.2 · p. 6

4.3. GPT-5 vs. GPT-5 Mini

  • GPT-5 performs best relative to GPT-5 Mini on tasks with explicit procedures and shows a stronger ability to identify global image structure, while both have low-level visual-perception weaknesses.The comparison covers the five tasks with the largest advantage for each model.Source: §4.3 · p. 6

4.4. Task Complexity vs. Model Accuracy

  • Accuracy decreases consistently from Easy to Hard instances across all evaluated models, including the RLVR-trained model, on the 17 tasks with a well-defined complexity scale.The study excludes tasks whose difficulty depends strongly on motif geometry and arrangement; tile-based tasks can be made harder by increasing maximum tile size.Source: §4.4 · p. 6

5. Reinforcement Learning

The paper trains four open-source LVLMs with RLVR on SPHINX and evaluates transfer to held-out tasks and external benchmarks; performance improves broadly, though gains vary by task and benchmark.

Data Split

  • Training uses 1,600 selected samples per in-distribution task from 100,000 synthetic samples, while five of the 25 tasks are withheld for OOD testing.Selection maximizes minimum semantic similarity to same-task evaluation samples.Source: §5 · p. 6

Model Training

  • The four Qwen2.5-VL and Qwen3-VL base models are fine-tuned with GRPO, using binary correctness and an auxiliary formatting reward verified against ground truth.The reward combines formatting and correctness terms with λ = 0.1; correctness is checked using mathruler.Source: §5 · pp. 6–7

SPHINX test results

  • All four RLVR-trained models gain substantially on SPHINX’s in-distribution tasks, and gains also transfer to held-out tasks with greater variation across task families.Table 2 reports accuracy on both IID and OOD subsets.Source: §5 · p. 7, Table 2 · p. 7
  • Tiles Recoloring is the most challenging held-out task, with two models performing worse than their base versions; Positional Count shows the largest improvements.The authors hypothesize that transfer to Positional Count reflects structural similarity to the trained Stack Count task and shared spatial-relation reasoning.Source: §5 · pp. 6–7

External benchmarks

  • Across 32 model–dataset combinations on eight external benchmarks, RLVR improves accuracy in 26 cases, declines in four, and is unchanged in two.Average accuracy gains are +2.8% for Qwen3-VL-4B and +2.4% for Qwen3-VL-8B; the authors could not test larger Qwen3 models due to resource constraints.Source: §5 · p. 7, Table 3 · p. 7

Response Length

  • RLVR-trained models consistently produce shorter average responses than their base counterparts on SPHINX and external benchmarks.The authors note that the 2048-token maximum response length during training may partly drive this trend.Source: §5 · p. 8, Figure 11 · p. 8

6. Related works

Related work spans cognitive-test and fixed visual-reasoning benchmarks, procedural synthetic environments, and RL for visual reasoning; SPHINX is positioned as a diverse generator–verifier environment within these lines.

Visual reasoning benchmarks

  • Cognitive tests and fixed datasets probe abstraction, analogy, and visual reasoning, but are limited in size and visual and structural diversity.Examples discussed include Raven’s Progressive Matrices, ARC, Bongard Problems, MM-IQ, MARVEL, and MATH-Vision.Source: §6 · p. 8

Synthetic and procedural benchmarks

  • Procedural benchmarks enable controlled variation, and SPHINX extends this work with diverse procedurally generated tasks paired with deterministic verifiers for reproducible evaluation.The cited examples include CVR, A-I-RAVEN, IconQA, Reasoning Gym, Enigmata, and UniBench.Source: §6 · p. 8

Reinforcement learning for visual reasoning

  • Prior visual-reasoning RL work reports gains or explores grounded reasoning, data mixing, self-reflection, and curricula; SPHINX supplies deterministic task verifiers suited to RLVR.The discussion names Reason-RFT, Visual-RFT, and Jigsaw-R1 among works reporting consistent gains.Source: §6 · p. 8

7. Limitations & Future Work

The study’s scope and RLVR training setup constrain how broadly its gains can be interpreted, and the authors identify task diversity, learning signals, and evaluation shortcuts as areas for future work.

Scope

  • The study focuses on a subset of task families, so observed gains may not fully translate to broader multimodal benchmarks.The authors propose expanding task diversity and including more visually complex or noisy inputs to better approximate real-world conditions.Source: §7 · p. 8

RLVR limitations

  • Even after training, in-distribution accuracy remains below 50%, and scarce correct traces can make target solutions effectively unreachable under RLVR.The authors suggest combining RLVR with supervised fine-tuning on reasoning traces from stronger teacher models to improve learning-signal quality and density.Source: §7 · p. 8

Future directions

  • The authors identify curriculum strategies that incorporate task difficulty and less guessable multiple-choice formats as potential directions for improving generalization and testing genuine reasoning.They note that multiple-choice shortcuts may otherwise affect whether improvements reflect genuine reasoning.Source: §7 · p. 8

8. Conclusion

SPHINX provides a 25-task synthetic environment for visual perception and reasoning. State-of-the-art LVLMs still struggle across its tasks, while reinforcement learning with verifiable rewards improves performance and generalization to external visual reasoning benchmarks.

8. Conclusion

  • Across 25 tasks, state-of-the-art LVLMs still struggle, while reinforcement learning with verifiable rewards yields meaningful improvements and enhances generalization to external visual reasoning benchmarks.Future work will expand SPHINX with additional task families and more advanced RL paradigms; the framework is open source for broader adoption and community-driven extensions.Source: §8 · p. 8

A. Implementation Summary

This section summarizes SPHINX’s implementation for generating rendered puzzles and structured records with answers and provenance from parameterized motif and tiling registries and task classes. It covers tasks in geometric and chart reasoning, counting, symmetry and pattern recognition, transformations, sequences, graph queries, missing-tile completion, and piece assembly.

A.1. Overview

  • Task classes sample parameters and registered motifs or tilings to generate rendered puzzles with answers, distractors, and structured metadata for dataset creation and model training or evaluation.Prompt templates add language variation, while configurable task-family weights enable controlled category sampling and recorded provenance supports verifiable rewards.Source: §A.1 · p. 12

A.2. Motif Library

  • The motif library provides 25 parameterized visual building blocks whose geometry, appearance, and layout can be varied to construct puzzles, scenes, and transformations.The listed families range from basic shapes and lines to grid-based forms, icons, and repeating patterns.Source: §A.2 · pp. 12–13

A.3. Tilings

  • Five tiling generators provide polygonal cells, coordinates, and adjacency information, enabling uniform connectivity and geometry queries across regular and irregular layouts.Regular square, triangular, hexagonal, and rhombille tilings expose Wythoffian colorings; circle packing uses vertex-touch adjacency.Source: §A.3 · pp. 14–15

B.1. Geometric Reasoning

  • Geometric and chart tasks cover spatial-relation counting, metric-based shape ordering, counting within overlapping sheets, ranking pie slices, and matching proportions across chart types.The chart-comparison task uses a pie or bar chart as the reference and options of the opposite chart type, with distractors filtered for distinctness.Source: §B.1.1 · p. 15, §B.1.2 · p. 15, §B.1.3 · p. 15

B.2. Counting

  • Counting tasks include labeled-region inclusion/exclusion, exact sub-shape counts, polyline edge lengths and intersections on tilings, and cell differences between related boards.Sub-shape counts use generator-matched combinatorial routines, while tile-line intersections count shared vertices, including endpoints.Source: §B.2.1 · p. 16, §B.2.2 · p. 16, §B.2.3 · p. 16

B.3. Symmetry & Pattern Recognition

  • Symmetry and pattern-recognition tasks ask models to classify mirror symmetry, complete a symmetry-constrained grid, or find the odd strip or patch among examples from symmetry groups.The odd-one-out designs cover six Conway frieze groups and seventeen IUC wallpaper groups.Source: §B.3.1 · p. 16, §B.3.2 · p. 16, §B.3.3 · pp. 17, 19

B.4. Sequence & Transformation Reasoning

  • Transformation tasks require selecting a transformed image, inferring the rule between two tiles, or identifying a similar or dissimilar option under Euclidean similarity.Transform-pair inference verifies uniqueness against the full rule set and can make “none of the above” correct when the true transform is withheld.Source: §B.4.1 · p. 19, §B.4.2 · p. 19, §B.4.3 · p. 19
  • Sequence tasks test masked-panel prediction for constant-step rotations and count-based increases or decreases in the number of shapes.Rotation sequences use steps of 30°, 45°, 60°, or 90° in either direction; count sequences use increments or decrements of 0, 1, 2, or 3.Source: §B.4.4 · p. 19, §B.4.5 · p. 19

B.4.5. Sequence Arithmetic

  • The sequence tasks ask models to identify a mirror-symmetry axis or choose the result of a specified transformation.Source: §B.4.5 · pp. 20–21

B.4.6. Sequence Multi-Column Arithmetic

  • Multi-column arithmetic tasks hide the fifth step of independent column progressions and offer four continuation candidates.They use 2–6 columns, with distractors changing exactly one column per wrong option.Source: §B.4.6 · pp. 21–22

B.5.1. Tiles Geometry

  • Tiling-geometry queries measure region area, perimeter, holes, area difference, or the perimeter of a two-region union.The tiling size sets complexity, and answers are integers.Source: §B.5.1 · p. 22

B.5.2. Tiles Connected Component

  • Tiling-component tasks ask for the largest or smallest component size, or the number of components, under a specified adjacency rule.They use edge adjacency or, for circular tilings, point-touch adjacency; extreme queries are constrained to have unique answers.Source: §B.5.2 · p. 22

B.5.3. Tiles Shortest Path

  • Tiling shortest-path tasks require the minimum edge-step distance between cells, or -1 when obstacles make the destination unreachable.Generated cases are 90% reachable and 10% unreachable, with BFS used to verify paths or enforce unreachable cases.Source: §B.5.3 · p. 22

B.5.4. Missing Tiles

  • Missing-tiling tasks restore either the exact colors of blanked cells or a missing shape, allowing the tiling’s dihedral symmetries.The balanced color and shape variants span four tilings, and complexity is measured by tiling size.Source: §B.5.4 · p. 22

B.5.5. Tiles Composition

  • Tile-composition tasks either decompose a connected region into a bag of pieces or assemble a bag into its connected target.They vary piece count, tiling family, and uniform versus per-cell coloring; distractors alter piece shapes or offer equal-area but incorrect unions.Source: §B.5.5 · p. 22

C. Human Evaluation

The human evaluation collected responses to assigned SPHINX problems and recorded correctness and timing. GPT-5 exceeded human accuracy on shape counting and, by a small margin, Tiles Composition, but its errors reveal inconsistent visual extraction and reasoning.

C. Human Evaluation

  • The evaluation collected 32 completed responses; participants completed 25 assigned problems, or 10 if they chose the shorter version.Recorded measures included correctness, per-question time, overall completion time, and assigned task types.Source: C. Human Evaluation · pp. 22, 24

D.1. Shape Counting

  • GPT-5 exceeded human performance on shape counting, 76% versus 55.2%, its strongest task-level advantage over participants.Participants counted mentally without notes; arithmetic slips and overlooked configurations may help explain their lower accuracy.Source: §D.1 · p. 25
  • Shape-counting errors often began with inaccurate visual extraction, after which otherwise coherent reasoning followed the mistaken count.Examples included miscounting rows or parallel lines; the resulting answer could be wrong even when the subsequent reasoning was suitable.Source: §D.1 · p. 25

D.2. Tiles Composition

  • GPT-5 exceeded human accuracy on Tiles Composition by about 2%, despite the task being non-trivial for humans.Thirty of 100 API responses contained no visible reasoning.Source: §D.2 · p. 25
  • On Tiles Composition, correct answers sometimes accompanied imperfect reasoning, while incorrect cases included unsupported counts and potentially post-hoc justifications.Overall, success depended heavily on extracting accurate low-level visual features, which remained inconsistent.Source: §D.2 · p. 25

D.3. Tiles Line Length

GPT-5’s Tiles Line Length errors reflect unreliable visual extraction, while its failures on recoloring and symmetry tasks reveal difficulty accurately comparing colored cells and analyzing global symmetry. On Wallpaper Groups, performance is stronger, but the model still struggles to reconcile local transformations with global structure.

D.3. Tiles Line Length

  • 14% accuracy versus human accuracy of 86.4% makes Tiles Line Length GPT-5’s worst relative-to-human task, with errors reflecting unreliable visual extraction.Examples include confusing colored segments, missing segments, and misreading the grid or path structure.Source: §D.3 · p. 25

D.4. Tiles Recoloring

  • GPT-5’s recoloring errors arise during color-specific cell counting, even when it proposes valid strategies for comparing the two boards.Examples include misjudging a region’s alignment, transcription errors, overcounting cells, and missing newly appearing cells.Source: §D.4 · p. 29

D.5. Frieze Groups

  • With human accuracy at 48.4% and GPT-5 at 26%, Frieze Groups is difficult even for people and leaves GPT-5 near four-choice random guessing.Examples show the model confusing reflection and rotation patterns in one-dimensional sequences.Source: §D.5 · p. 29

D.6. Wallpaper Groups

  • GPT-5 reaches 48% on Wallpaper Groups, below human accuracy of 86.4%; it often attends to local neighborhoods without reconciling them with global symmetry.The paper suggests the more apparent global symmetry in two-dimensional layouts may help explain higher human accuracy than on Frieze Groups.Source: §D.6 · p. 29

E. GPT-5 vs GPT-5 Mini

GPT-5 and GPT-5 Mini show task-dependent strengths: GPT-5 is generally better at global symmetry and transformation judgments, while both models struggle with visual counting. GPT-5 Mini leads on Sequence Arithmetic, and both models make errors tracking multi-column progressions.

E. GPT-5 vs GPT-5 Mini

  • GPT-5 outperforms GPT-5 Mini by 36% on Stack Count, 17% on Transform Result Identify, and 12% on Transform Similarity Identify.The paper associates GPT-5’s advantage on these transformation tasks with more reliable reasoning about global shape and structure.Source: E. GPT-5 vs GPT-5 Mini · pp. 29, 33
  • GPT-5 Mini outperforms GPT-5 by 18% on Sequence Arithmetic, where GPT-5 makes motif-counting errors.GPT-5 Mini is more reliable at counting motifs in the examples discussed.Source: E. GPT-5 vs GPT-5 Mini · p. 33
  • GPT-5 outperforms GPT-5 Mini by 12% on Sequence Multi-Column Arithmetic, but both struggle to track column-wise progressions and count motifs.Even correct answers can accompany subtle reasoning errors and partial observations.Source: E. GPT-5 vs GPT-5 Mini · p. 33
  • Across these tasks, both models struggle with counting, while GPT-5 is generally better at global symmetry and transformation reasoning; success depends on visual extraction demands.The paper notes that GPT-5’s strong Shape Counting performance may reflect tasks requiring substantial mental arithmetic but less visual extraction.Source: E. GPT-5 vs GPT-5 Mini · p. 33

F. Discussion of RLVR trained models

The discussion examines RLVR training dynamics and task-level response patterns, alongside the prompt, evaluation procedure, and task-accuracy tables. Some tasks show high accuracy or improvements, while several tile and symmetry tasks remain difficult even after training.

F. Discussion of RLVR trained models

  • Reward curves suggest the four models are not fully saturated; Qwen3-VL response lengths initially decrease and then rise, while Qwen2.5-VL lengths stay closer to base-model levels.The paper identifies additional training as a possible direction.Source: F. Discussion of RLVR trained models · p. 33
  • Pie Chart, Shape Sorting, and Venn Diagram are among the shortest high-accuracy tasks; RLVR improves Shape Sorting and Venn Diagram, possibly where test data resemble training data.The proposed explanation is that these tasks lack Motifs or Tiles, limiting visual variety.Source: F. Discussion of RLVR trained models · p. 39
  • Tiles Shortest Path stays below 20% accuracy for Qwen2.5-VL, with only slightly better Qwen3-VL performance; several long-response tasks remain below 40%.The below-40% tasks include Tiles Composition, Symmetry Fill, Transform Similarity Identify, and Frieze Groups; three are included in training.Source: F. Discussion of RLVR trained models · p. 39

G.1. Prompt

  • The prompt asks models to reason step by step and place the final answer inside a boxed format.Source: §G.1 · p. 39

G.2. Evaluation Protocol

  • Evaluation first infers answer type, applies heuristic extraction and strict type-consistent scoring, then uses GPT-5-mini to extract an answer if needed.The fallback extractor uses temperature 0 and a short few-shot prompt.Source: §G.2 · p. 39

H. Task Accuracy

Loading 2511.20814v2…