SPaRC: A Spatial Pathfinding Reasoning Challenge

• Existing benchmarks can saturate as models achieve superhuman scores, motivating abstract-reasoning datasets with minimal priors.Source: §1 · p. 1
• Spatial grid tasks often omit the combined planning and logic skills needed for navigation, manipulation, scene understanding, and augmented reality.Source: §1 · p. 1
• SPaRC combines pathfinding with arithmetic and geometric rules, requiring multi-step constraint solving on 2D grid puzzles.Source: §1 · p. 1
• 98.0% human accuracy versus 15.8% for o4-mini shows SPaRC challenges reasoning models; on difficulty-5 puzzles, the figures are 94.5% and 1.1%, respectively.Source: Figure 2 · p. 2, §1 · p. 2
• The evaluation includes six human annotators, three instruction-tuned models, and four reasoning models, with analyses of path failures, reasoning mistakes, and multiple attempts.Source: §1 · p. 2
• Prompt and few-shot ablations have modest effects, while screenshot-based multimodal prompting does not improve o4-mini performance over text.Source: §1 · p. 2

• Long-horizon benchmarks such as MATH, AIME, BBH, and MUSR challenge multi-step problem solving, but rely on data priors, knowledge recall, or pattern matching.Source: §2 · p. 2
• ARC-AGI tests abstract pattern recognition but not step-by-step planning or discrete-rule following; VisualPuzzles uses multiple choice rather than constructed solutions.Source: §2 · pp. 2–3
• SpatialEval separates short mazes from counting or relation questions, while PPNL focuses on obstacle avoidance and EnigmaEval does not focus on pathfinding.Source: §2 · p. 3
• SPaRC combines long-term path planning with counting, segregation, and color or shape logic, using complex interacting rules and allowing multiple correct solutions.Source: §2 · p. 3

3 Dataset

• SPaRC targets spatial navigation, rule understanding, constraint satisfaction, and multi-step planning, with mechanics inspired by The Witness.Source: §3 · p. 3

3.1 Puzzle Rules

• Each puzzle uses a grid of rule cells and asks for a continuous, non-crossing path from start to end that fulfills cell conditions.Source: §3.1 · p. 3
• Rules require collecting dots, avoiding gaps, counting touched edges, separating colors, pairing stars, and fitting polyomino shapes within regions.Source: §3.1 · p. 3
• Ylops subtract their shape and area from polys in the same region, and exact cancellation removes the constraint.Source: §3.1 · p. 3

3.2 Dataset Creation

• The dataset contains 500 training and 500 testing puzzles; generation adjusts rule density when puzzles are unsolvable or have too many solutions, and only 25 puzzles contain ylops because they require polys and are hardest for humans.Source: §3.2 · p. 4
• Difficulty combines rule count, rule-cell count, density, and estimated interactions in a weighted score normalized to levels 1–5.Source: §3.2 · p. 4

4.1 Main Results

• The top reasoning model, o4-mini, achieved 15.8% accuracy (79/500 puzzles), versus 1.6% for GPT-4.1; o4-mini fell from 47.7% at difficulty 1 to 1.2% at level 4.At difficulty 4, humans achieved 94.4% accuracy.Source: §4.1 · p. 5
• Gaps yielded superior performance across all models, while polys and ylops were among the most challenging rules; on polys, QwQ and Gemma exceeded their average performance by 13.2 and 12.8 points, respectively.Source: §4.1 · p. 5

4.2 Path Errors and Reasoning Mistakes

• More than 50% of puzzles failed because models generated invalid paths, most often by crossing rule cells; in 48 valid R1 paths, logical fallacies were the most common reasoning error.The 48-puzzle analysis also found grid/index misinterpretation in 26 cases and careless shortcutting in 23; these findings concern selected single-rule puzzles where R1 produced valid paths but missed rules.Source: §4.2 · pp. 6–7
• For o4-mini, pass@8 raised accuracy from 15.8% to 35.0%, but additional attempts improved difficulty 4–5 puzzles by only 2.3 points.Pass@k counts a puzzle as solved if any of k generated solutions is correct, so this is an upper-bound setting rather than a practical test-time selection method.Source: §4.2 · p. 7, Figure 7 · p. 7

4.3 Ablations

• One-shot prompting had the highest overall o4-mini accuracy at 15.8%, while screenshots reduced accuracy from 12.6% with text to 5.6%.Two-shot prompting outperformed one-shot only at difficulty 1, then reversed at higher levels; the visual comparison used zero-shot text because screenshots lacked an example solution.Source: §4.3 · p. 8
• Emphasizing path rules raised o4-mini accuracy from 15.8% to 21.0%, but did not improve its difficulty-5 result of one solved puzzle out of 89.Rule Cell Crossing declined from 51.2% to 29.0%, and Intersecting Line from 31.2% to 21.2%.Source: §4.3 · p. 8

4.4 Human Evaluation

• Human average solve time rose from 10.7 seconds at difficulty 1 to 131.5 seconds at difficulty 5, while reasoning-model token counts did not rise with difficulty.QwQ token counts changed from 14,433 to 13,114 across difficulties, whereas Qwen 2.5 counts increased from 790 to 1,161.Source: Table 2 · p. 8, §4.4 · p. 9

5 Conclusion

• Humans achieved 98% accuracy, while o4-mini reached 16% overall and only 1.1% on level-5 puzzles.Source: §5 · p. 9
• Generating up to eight attempts improved accuracy up to 30% for difficulty 1 and 2% for difficulty 5; reasoning models increased token use ∼5% on harder puzzles.Humans consistently solved puzzles across levels, including 95% at level 5; ablations of visual representation, prompting, and few-shot examples showed only mild variation.Source: §5 · p. 9
• The authors conclude that enhanced prompting and computational sampling offer only partial improvements, and that fundamental advances in reasoning are still needed for human-level results.Source: §5 · p. 9

Limitations

• The evaluation can produce false negatives from formatting failures; trace analysis is limited, and the dataset does not exhaustively cover interactions among its seven rule types.Formatting failures are rare despite prompt reminders; OpenAI reasoning traces are redacted, and broader rule combinations are left for future work.Source: Limitations · pp. 9–10
• Poly-set performance spikes are subset-specific: when polys fill the grid, edge-following can provide a shortcut, especially for smaller models.Source: Limitations · p. 10

A Models & Hardware

• Appendix A lists experimental models, hardware, and approximate token counts, including Gemma 3 27B, QwQ 32B, Qwen 2.5 72B, and R1 70B.It also lists GPT-4.1, o3-mini, and o4-mini; both OpenAI reasoning models used medium reasoning effort.Source: A Models & Hardware · p. 12, Table 3 · p. 12

B Grid Indexing

• Figure 10 annotates the puzzle grid with coordinates, matching the indexing used in the prompts.Source: Figure 10 · p. 12, B Grid Indexing · p. 12

C Licenses and Code Acknowledgments

• The source code is provided under BSD 3-Clause, while the study datasets use CC-BY-4.0.Source: C Licenses and Code Acknowledgments · p. 12

D Difficulty Metric Calculation

• The metric is intended to capture multiple aspects of puzzle complexity in a single difficulty rating.Its calculation combines component scores through a weighted sum before normalization.Source: D Difficulty Metric Calculation · p. 12, §D.1 · pp. 12–13

D.1 Combination and Normalization

• The difficulty score combines five components, with grid size weighted most heavily, then normalizes and clamps the result to the 0–5 range.The weighted sum is standardized using an empirically derived normal distribution, converted through its CDF, and scaled to [0, 5].Source: §D.1 · p. 13

D.2 Component Scores

• The five components measure distinct rule types, interactions between rules, grid area, rule density, and absolute rule count.The interaction score applies only when multiple rule types are present and depends on potential interactions and rule density.Source: §D.2 · p. 13

E.1 Reasoning Fine-Tuned Models

• ReasonFlux-F1 32B performs comparably to another same-size reasoning-tuned model (8.6% vs. 6.0%), with gains remaining small relative to human performance.Figure 11 compares accuracy across difficulty levels for Qwen 3 32B and ReasonFlux variants.Source: §E.1 · pp. 13–14, Figure 11 · p. 14

E.2 Puzzle Grid Representations

• The array format achieves the highest overall accuracy (21.0%), slightly above the two markdown formats (19.0% and 20.8%), which show no consistent improvement.The tested formats are a plain markdown table and a table with row and column coordinates; Figure 12 compares them across difficulty levels.Source: §E.2 · p. 14, Figure 12 · p. 14

F Tokens by Puzzle Difficulty

• Table 4 reports average tokens produced by the different models, broken down by puzzle difficulty.Figure 12 separately compares accuracy across difficulty levels for the grid representations.Source: F Tokens by Puzzle Difficulty · p. 14, Table 4 · p. 14

G Rule Visualizations

• Figures 13–19 show unsolved grids alongside valid solution paths; the Dots rule requires the path to pass through all dots, while Gaps prohibit crossing marked edges.Source: G Rule Visualizations · p. 14, Figure 13 · p. 15, Figure 14 · p. 15
• The Stones example separates differently colored stones into distinct regions, while each star-containing region must include exactly one other rule of the same color.Source: Figure 15 · p. 15, Figure 16 · p. 16
• The Triangles rule requires the path to touch as many grid edges as the adjacent cell’s triangle count.Source: Figure 17 · p. 16
• Poly rules require a region to contain the depicted shape; the Polys–Ylops example combines shape inclusion with subtraction constraints.Multiple poly shapes in one region can be combined.Source: Figure 18 · p. 16, Figure 19 · p. 17

H Additional Dataset Statistics

• Table 5 reports statistics for all SPaRC splits, with difficulty and rule statistics based on the test set used for evaluation.Source: Table 5 · p. 17, H Additional Dataset Statistics · p. 17

I Prompting

• Listings 1–5 provide the prompts and few-shot examples used for the Section 4 experiments.Source: I Prompting · p. 17

I.1 Default Prompt

• The default prompt requests step-by-step reasoning and a complete start-to-end coordinate path, with intermediate nodes included.Its instructions specify grid coordinates and constraints such as mandatory dots and prohibited gaps.Source: §I.1 · pp. 17–19

I.2 Alternative Prompt

• The alternative prompt restates path and region rules, including triangle edge counts and polyshape inclusion or subtraction, and requests a coordinate-list solution.Source: §I.2 · p. 20

I.3 Vision Prompt

• The vision prompt asks models to analyze a puzzle image and provide step-by-step reasoning followed by a complete coordinate path.It also supplies a text description of the puzzle and specifies path constraints.Source: §I.3 · pp. 20–21

I.4 Few-Shot Example

• Few-shot examples walk through paths satisfying triangle, polyshape, stone, and star constraints; another example combines positive and negative polyshapes.Source: §I.4 · pp. 21–22

J Full Tabular Main Results

• Tables 6–8 provide the detailed and complete results for the experiments in Section 4.1.Source: J Full Tabular Main Results · p. 22

J.1 Difficulty per Level

• 15.8% is the highest overall accuracy, achieved by o4-mini; performance generally decreases sharply as difficulty rises.Table 6 reports overall accuracy and results by difficulty level 1–5 for reasoning and instruction models.Source: Table 6 · p. 23

J.2 Path Metrics

• Table 7 organizes path-violation rates by Instruction and Reasoning model categories and by specific violation types.The caption does not provide the table's cell values.Source: Table 7 · p. 23
• The table covers generated solutions for SPaRC puzzles.No particular model or violation rate can be identified from the caption alone.Source: Table 7 · p. 23
• Its columns report percentages for specific path violations.Source: Table 7 · p. 23

J.3 Rule Specific Analysis

• Table 8 reports accuracy (%) for LLMs on the Full Set and puzzle splits.The caption does not supply cell values or identify a highest-performing model.Source: Table 8 · p. 23
• The listed single-rule splits are Gaps, Dots, Stones, Stars, Tri, and Polys.Source: Table 8 · p. 23
• The combined-rule splits include St-S, P-Y, G-D-T, and D-S-P.The caption expands these as Stones & Stars; Polys & Ylops; Gaps & Dots & Triangles; and Dots & Stars & Polys.Source: Table 8 · p. 23

K Details on Path Errors

• Figure 20 pairs an empty puzzle with its solution and depicts four kinds of path-generation errors.The examples illustrate the puzzle structure and fundamental path violations discussed in Section 4.Source: K Details on Path Errors · p. 23, Figure 20 · p. 24

L Details on Reasoning Mistakes

• Figures 21–24 present four examples of common reasoning mistakes by DeepSeek R1 70B variants.One example shows a model committing to reasoning steps without thoroughly checking them, resulting in careless mistakes.Source: L Details on Reasoning Mistakes · p. 25, Figure 21 · p. 25
• In a stars puzzle, DeepSeek R1 70B misses that one region can contain two stars of each color and attempts impossible loops around stars.The model recognizes that it is stuck and offers a later attempt at a valid solution.Source: Figure 22 · p. 26
• Other examples show confusion about grid dimensions and x/y indexing, coordinate orientation, and paths crossing rule cells.The resulting paths are incorrect; the stones example explicitly fails to satisfy the puzzle rules.Source: Figure 23 · p. 27, Figure 24 · p. 28

M.1 Vision Mode

• 12.6% overall accuracy for standard o4-mini Zero-Shot exceeds 5.6% for o4-mini Vision.Vision mode generally shows higher rates of structural path errors and underperforms at nearly all difficulty levels.Source: Figure 25 · p. 29

M.2 Alternative Prompt

• 21.0% overall accuracy with the alternative prompt exceeds 15.8% with o4-mini’s standard prompt.The alternative prompt also yields higher accuracy across all difficulty levels and generally reduces structural path-violation rates.Source: Figure 26 · p. 29

M.3 Few-Shot

• 15.8% overall accuracy with one-shot prompting is slightly above two-shot at 14.8% and zero-shot at 12.6%.The error profiles differ little across strategies, and accuracy sharply declines as difficulty rises.Source: Figure 27 · p. 30

N.1 Annotators

• The annotator group comprised two Ph.D. students and four research assistants, all male and aged 22–27.The assistants were two bachelor’s and two master’s students; all received at least Germany’s minimum wage through their research jobs.Source: §N.1 · p. 31

N.2 Annotation Instructions

• The annotation guide describes six 50-puzzle test sets covering all combinations of rules and grid sizes, and asks annotators to solve puzzles quickly.Annotators could reveal a solution or skip after 5–10 minutes without a solution, then return annotations including solve time, attempts, and path.Source: §N.2 · p. 31
• Valid paths connect Start to End continuously without reusing cells; they must avoid gaps, pass through dots, and separate differently colored stones into regions.Each star must pair with exactly one same-color element in its region; triangles require the specified number of touched edges.Source: §N.2 · p. 31
• Polyshapes constrain a region’s shape, while negative polyshapes must be used and may overlap or extend beyond the line-defined area.The guide also specifies that triangles require the line to touch the edge count shown by the triangle.Source: §N.2 · p. 31