Source-linked AI summary
MathSticks: A Benchmark for Visual Symbolic Compositional Reasoning with Matchstick Puzzles
Yuheng Ji, Huajie Tan, Cheng Chi, Yijie Xu, Yuting Zhao, Enshen Zhou, Huaihai Lyu, Pengwei Wang, Zhongyuan Wang, Shanghang Zhang, Xiaolong Zheng
TL;DR
Existing vision and reasoning tasks often isolate perception, symbolic manipulation, or arithmetic rather than testing them together. MATHSTICKS benchmarks that combination with constrained matchstick edits in text-prompted and pure-visual settings. Evaluations reveal persistent model limitations, while human participants exceed 90% accuracy.
Problem
Existing benchmarks provide limited coverage of solvable constrained edits, fine-grained task settings, and protocols isolating pure-visual reasoning.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
Abstract
from arXiv · showhide
We introduce \textsc{MathSticks}, a benchmark for Visual Symbolic Compositional Reasoning (VSCR), which unifies visual perception, symbolic manipulation, and arithmetic consistency. Each task presents an incorrect matchstick equation that must be corrected by moving one or two sticks under strict conservation rules. The benchmark includes both text-guided and purely visual settings, systematically covering digit scale, move complexity, solution multiplicity, and operator variation, with 1.4M generated instances and a curated test set. Evaluations of 14 vision--language models reveal substantial limitations: closed-source models succeed only on simple cases, open-source models fail in the visual regime, while humans exceed 90\% accuracy. These findings establish \textsc{MathSticks} as a rigorous testbed for advancing compositional reasoning across vision and symbols. Our code and dataset are publicly available at https://github.com/Yuheng2000/MathSticks.
1 Introduction
The paper frames Visual Symbolic Compositional Reasoning as a unified challenge spanning visual perception, symbolic manipulation, and arithmetic consistency. It introduces MATHSTICKS, a matchstick benchmark, and reports that current models lag on harder puzzles and pure-visual tasks while humans exceed 90% accuracy.
1 Introduction
- Existing vision and reasoning tasks often isolate perception, reasoning, or arithmetic, while benchmarks lack constrained solvable edits, fine-grained difficulty control, and pure-visual evaluation.VSCR links visual recognition, constrained transformations, and arithmetic verification.Source: §1 · p. 1
- MATHSTICKS requires correcting an incorrect matchstick equation by moving one or two sticks, with text-prompted and pure-visual regimes across controlled puzzle dimensions.Its pipeline generates 1.4M instances and releases a balanced 400-item evaluation set.Source: §1 · p. 1
- Humans exceed 90% accuracy, while closed-source models falter on multi-move and operator-flip cases and open-source models score near zero in pure-visual evaluation.The evaluation covers 14 representative vision-language models.Source: §1 · p. 2
2 MathSticks
MATHSTICKS formulates each puzzle as an invalid seven-segment arithmetic equation that must be repaired through legal stick relocations. Its generation pipeline exhaustively enumerates and validates solvable edits, then renders them with structured diagnostic labels.
Task Definition
- Each invalid equation x = [a, ⊕, b, =, c], with ⊕ in {+, −}, must be corrected using one or two relocated sticks so that a ⊕ b = c.The task combines visual recognition, constrained symbolic edits, and arithmetic verification.Source: §2 · p. 2
- Predictions use indexed matchstick positions and canonical Move(source, destination) notation, enabling machine-parsable, unambiguous verification against ground truth.Two-stick solutions are represented by two move operations.Source: §2 · p. 2
Categorization
- Instances are annotated by digit scale, move complexity, solution multiplicity, and operator flipping, yielding seven diagnostic categories.Digit scale ranges from single- to two-digit operands; edits require one or two moves.Source: §2 · p. 2
Dataset Construction
- The two-stage pipeline exhaustively enumerates legal one- and two-stick edits, filters for arithmetic validity, deduplicates solutions, and deterministically renders symbolic states as images.It produces about 1.4M solvable instances with structured metadata.Source: §2 · p. 2
Dataset Statistics
- Of 1,411,388 validated instances, 79.07% are Level 4 and 82.01% require two-stick edits; the released 400-item test set is balanced across four levels.Unique corrections account for 43.12% of instances, while 56.88% have multiple valid corrections.Source: §2 · p. 2
3 Experiments
Across text-prompted and pure-visual regimes, proprietary models outperform open-source systems, but a large gap to human performance remains. The experiments also identify perception, edit-planning, arithmetic-verification, operator-handling, and output-format failures, with error patterns linked to benchmark difficulty dimensions.
3.1 Experimental Setup
- The setup evaluates closed- and open-source vision-language models across text-prompted and pure-visual regimes, four difficulty levels, and additional diagnostic dimensions.Accuracy is the primary metric; three adults independently completed the pure-visual benchmark as a human reference.Source: §3.1 · p. 3
3.2 Performance Analysis
- Closed-source models substantially outperform open-source systems, whose accuracies remain near chance across both regimes; reasoning-enhanced variants also outperform their standard counterparts.For example, o3 reaches 60.00% versus GPT-4o at 0.00%, and Gemini-2.5-Pro reaches 45.25% versus Gemini-2.5-Flash at 26.50%.Source: §3.2 · pp. 3–4
- o3 achieves the highest average accuracy across regimes at 49.25%, followed by Gemini-2.5-Pro at 33.88%.The paper attributes o3's performance to its “thinking with image” approach combining visual parsing and multi-step symbolic reasoning.Source: §3.2 · p. 4
- Providing equation text improves accuracy for every model; o3 rises from 38.50% without text to 60.00% with text.Without text, models must also recognize digits and operators, making perception a bottleneck.Source: §3.2 · p. 4
- Human participants achieved 91.72% average accuracy, substantially exceeding the best model’s 49.25%; this indicates the benchmark is readily solvable by humans while current VLMs retain limitations integrating perception and reasoning.Source: §3.2 · p. 4
3.3 Error Analysis
- Perception errors include confusing visually similar digits or missing distinguishing segments, while edit-planning errors propose nonexistent segments or implicitly add sticks.These failures occur within the perception and legal-edit stages of the benchmark pipeline.Source: §3.3 · p. 4
- Other failures include visually valid but arithmetically false edits, mishandled operator flips, and free-form answers that prevent automatic evaluation.The observed error patterns align with task dimensions: planning errors with two-stick puzzles, operator errors with flip cases, and perception errors with pure-visual inputs.Source: §3.3 · p. 4
4 Conclusion
MATHSTICKS targets visual symbolic compositional reasoning with matchstick arithmetic, combining constrained edits and arithmetic verification. Its conclusion presents the benchmark as a diagnostic testbed, while the appendix outlines supporting representation, construction, statistics, experiments, and case studies.
4 Conclusion
- Experiments with 14 VLMs show that strong closed-source systems struggle with multi-step edits and operator flips, open-source models collapse in pure-visual evaluation, and humans exceed 90% accuracy.The conclusion presents these results as a substantial capability gap and describes MATHSTICKS as a testbed for future progress in perception–symbol reasoning.Source: §4 · p. 5
Appendix
- The appendix supplements the main text with related literature, matchstick representation, dataset construction and statistics, extended experiments, evaluation prompts, and qualitative case studies.Its sections include reproducibility-oriented prompts and examples of model successes and failure patterns.Source: Appendix · p. 10
A Related Work
- MATHSTICKS operationalizes VSCR through matchstick equations, requiring visual recognition, constrained symbolic transformations, and arithmetic correctness.Tasks begin with an invalid equation and require relocating one or two sticks without insertion or deletion.Source: A Related Work · pp. 10–11
B Visual Representation of Matchsticks
- Each digit uses seven-segment indexing, with slot-prefixed labels for individual sticks; the operator has a dedicated editable vertical segment, while the equality sign is not edited.The indexed representation supports canonical, parsable move outputs independent of rendering details.Source: B Visual Representation of Matchsticks · p. 11
C Dataset Construction Details
Dataset construction separates symbolic search from visual rendering to cover candidate equations systematically and keep rendered stimuli aligned with their symbolic transformations.
C Dataset Construction Details
- A two-stage pipeline combines systematic symbolic enumeration and solution mining with deterministic rendering from manually designed segment templates.The symbolic stage searches valid one- and two-stick moves; the rendering stage assembles equation images from templates.Source: C Dataset Construction Details · p. 12
- The construction design targets completeness in the symbolic search space and fidelity in the visual representation.It is intended to provide large-scale coverage with precise alignment between symbolic transformations and visual stimuli.Source: C Dataset Construction Details · p. 12
- Visual stimuli are assembled deterministically from manually designed segment templates.This rendering stage follows symbolic enumeration and solution mining.Source: C Dataset Construction Details · p. 12
C.1 Symbolic Enumeration
Symbolic enumeration encodes each equation as a discrete state, explores stick-conserving edits, and filters reachable outcomes for arithmetic validity. The resulting solvable instances receive diagnostic labels and can be rendered from indexed templates.
C.1 Symbolic Enumeration
- Each equation is encoded as a seven-slot state, with blanks allowed in tens positions, and arithmetic validity is checked after converting the slots into operands and result.The operator determines whether the check is addition or subtraction.Source: §C.1 · p. 12
- Lookup tables encode legal one-stick moves and two-stick composite edits, including operator flips, under stick conservation.The two-stick edits include within-digit changes and cross-digit transfers.Source: §C.1 · p. 12
- The operator slot uses an editable vertical segment: its presence denotes plus and its absence denotes minus, while the equality sign is excluded from edits.The horizontal operator bar remains fixed.Source: §C.1 · p. 12
- The procedure enumerates one- and two-stick reachable states, keeps arithmetically valid outcomes, and removes two-stick solutions also found among one-stick solutions.This deduplicates across move types.Source: §C.1 · p. 12
- Each surviving transformation is labeled by digit scale, move complexity, solution multiplicity, and whether the operator flips.These labels support diagnostic categorization.Source: §C.1 · p. 13
C.2 Visual Rendering
Visual rendering uses manually created, segment-indexed templates and deterministic slot-wise assembly to map symbolic equations and edits to consistent images.
C.2 Visual Rendering
- The template library contains seven-segment digits and an operator slot, with unique indices corresponding one-to-one to visible sticks.The assets use fixed canvas size, baseline alignment, and uniform spacing.Source: §C.2 · p. 13
- Rendering retrieves templates for the equation's slots and concatenates them horizontally, so a symbolic move corresponds to a unique visible stick relocation.Template coordinates and segment bounding boxes are stored for re-rendering without manual intervention.Source: §C.2 · p. 13
- The dataset-distribution figure divides its view between difficulty-level proportions and breakdowns by move complexity, solution multiplicity, and operator flipping.Table 3 reports counts and percentages for these dimensions.Source: Figure 4 · p. 14, Table 3 · p. 14
D Detailed Dataset Statistics
The dataset contains 1,411,388 solvable instances, distributed across difficulty levels, move complexity, solution multiplicity, and operator-flip requirements. Its composition is heavily weighted toward Level 4 and two-stick solutions, while more than half of instances admit multiple corrections.
Dataset overview
- 1,411,388 solvable instances span difficulty levels, move complexity, solution multiplicity, and operator-flip requirements.Figure 4 visualizes these dimensions, and Table 3 reports their counts and percentages.Source: D Detailed Dataset Statistics · p. 14
Difficulty levels
- 79.07% of instances are Level 4, compared with 0.11% at Level 1, 1.31% at Level 2, and 19.51% at Level 3.The passage attributes this skew to the larger search spaces of more complex puzzles.Source: D Detailed Dataset Statistics · p. 14
Move complexity
- 82.01% require two-stick transformations, 4.18% have one-stick solutions, and 13.81% permit either move count.The distribution emphasizes composite rather than local single-edit corrections.Source: D Detailed Dataset Statistics · p. 14
Solution multiplicity
- 56.88% of instances admit multiple valid edits, while 43.12% have a unique correction.The passage notes that multiple-solution cases challenge autoregressive models to converge on one valid output.Source: D Detailed Dataset Statistics · p. 14
Operator flipping
- Operator-flip tasks require changing + to − or − to +, adding abstract symbolic edits to digit-morphology changes.Source: D Detailed Dataset Statistics · p. 14
E.1 Fine-Grained Analysis
Performance varies with reasoning complexity and input regime: multi-stick, ambiguous, and operator-flip cases are harder, and pure-visual inputs add a parsing challenge. Open-source models remain near zero across categories, including at 70B+ scale.
Move complexity
- Two-stick puzzles are consistently harder than one-stick puzzles, while mixed cases often score higher because an easier one-stick solution may exist.The passage notes that accuracy often falls below 20% for weaker closed-source models on two-stick puzzles.Source: §E.1 · p. 15
Solution multiplicity
- Multiple-solution problems reduce GPT-o3 and Gemini 2.5 Pro accuracy by 20–30 points relative to single-solution cases.The passage links this difficulty to ambiguity among structurally distinct transformations.Source: §E.1 · p. 15
Operator flipping
- Operator-flip performance is consistently below no-flip performance, and open-source baselines almost entirely fail these puzzles.The passage describes this as a bottleneck for moving beyond digit-level edits to abstract symbolic transformations.Source: §E.1 · p. 15
Cross-regime comparison
- Gemini 2.5 Pro falls from 51.7% on one-move text-prompted puzzles to 25.8% in the pure-visual setting.The passage attributes the added gap to OCR and structural parsing; GPT-o3 has relatively smaller gaps, indicating greater robustness to visual noise and layout variability.Source: §E.1 · pp. 15–16
Open-source models
- All tested open-source models score close to zero across categories, and even variants with 70B+ parameters remain at chance level.The passage suggests the failure reflects missing targeted supervision and inductive biases rather than insufficient capacity.Source: §E.1 · p. 16
E.2 Human Evaluation
In a small human study, participants achieved over 90% accuracy on the benchmark, with roughly one minute per problem. The paper also specifies prompt regimes and presents case studies that illustrate model errors across difficulty levels.
E.2 Human Evaluation
- Humans averaged above 90% accuracy while spending roughly one minute per problem, confirming that the benchmark is reliably solvable by people yet challenging for current vision–language models.Three adult participants solved the pure-visual evaluation set; because they transcribed the visually unambiguous digits and operators without error, the same results apply to text-prompted inputs.Source: §E.2 · pp. 16–17
F Prompts
- The text-prompted regime supplies the symbolic equation string, whereas the pure-visual regime supplies only the rendered matchstick image and requires OCR and structural parsing.Source: Figure 5 · p. 17, Figure 6 · p. 17, F Prompts · p. 18
G Case Studies
- Sixteen case studies sample two puzzles per difficulty level across both input regimes and compare predictions from representative models.The analysis highlights incorrect operator flips, partial edits that leave digits illegible, and failures to converge when multiple solutions exist.Source: G Case Studies · p. 18