PaperlayerView exact revision on arXiv

Source-linked AI summary

JigShape: Evaluating Visual-Geometric Reasoning in VLMs through Jigsaw Puzzles

Shawn Li, Wei Yang, Jike Zhong, Jiate Li, Jiawei Yang, You Qin, Ryan Rossi, Franck Dernoncourt, Roger Zimmermann, Yue Wang, Zhengzhong Tu, Vicente Ordonez, Mohit Bansal, Yue Zhao

arXiv:2607.27670v2cs.CVcs.AI

TL;DR

Existing jigsaw benchmarks can have ambiguous labels and coarse grids, limiting their ability to test visual-geometric reasoning. JigShape uses interlocking tab-and-blank pieces across dense grids to provide unambiguous evaluation, revealing largely absent zero-shot reasoning and a scaling cliff as puzzles grow.

  • Problem

    Existing rectangular-cut benchmarks can have ambiguous labels in repeated textures and typically use coarse grids, limiting evaluation of spatial reasoning at scale.

  • Method

    JigShape benchmarks joint visual-geometric reasoning using tab-and-blank interlocking pieces across 95K+ instances and grids from 4×4 to 16×16.

  • Results

    All models collapse toward chance on larger grids, although GPT-5.5 exceeds random baseline on 4×4 and fine-tuned models surpass 97% there.

  • Takeaways & Limitations

    Scalable geometric reasoning remains an open challenge for vision-language models.

  • Takeaways & Limitations

    Existing rectangular-cut benchmarks may conflate reasoning failures with label arbitrariness in visually indistinguishable regions.

Abstract

from arXiv · show

Jigsaw puzzle solving requires jointly reasoning about visual content and geometric constraints, yet existing benchmarks use rectangular cuts that create ambiguous ground truth in texture-repeated regions. We introduce \textit{\ours{}}, a benchmark with tab-and-blank interlocking pieces where geometric constraints provide strong local compatibility requirements that, combined with visual content, yield unambiguous ground truth. Across 95K instances at four grid densities (4$\times$4 to 16$\times$16), we find that \textbf{zero-shot VLMs largely lack geometric reasoning}: only one of five frontier models (GPT-5.5) exceeds random baseline on 4$\times$4 puzzles, while all others perform at chance level. While supervised fine-tuning achieves $>$97\% on 4$\times$4, \textbf{all models collapse on larger grids}: GPT-5.5 drops from 70\% to near-random on 8$\times$8, and even fine-tuned models fall below 5\% on 12$\times$12. This ``scaling cliff'' suggests current architectures cannot maintain consistent constraint satisfaction as the number of pieces increases. \ours{} establishes scalable geometric reasoning as an open challenge for vision-language models.

1 INTRODUCTION

JigShape frames jigsaw solving as joint visual-geometric reasoning and addresses ambiguity and limited difficulty in rectangular-cut benchmarks with interlocking pieces and dense grids. Its evaluation finds that zero-shot VLMs largely lack geometric reasoning and that performance collapses as puzzle size increases despite strong fine-tuned results on 4×4 puzzles.

  • Motivation: Jigsaw solving combines local visual cues with global tiling constraints, requiring precise spatial localization because each piece has one correct position.This makes jigsaw puzzles a probe of spatial reasoning beyond tasks where partial semantic understanding suffices.
  • Limitations of Existing Benchmarks: Rectangular cuts create visually interchangeable patches in uniform-texture regions, making single-label evaluation ill-posed.A perceptually reasonable permutation may be marked wrong when repeated regions, such as sky, are indistinguishable.
  • Limitations of Existing Benchmarks: 2×2–5×5 benchmarks provide coarse difficulty, whereas 3×3 has 9! ≈363K permutations and 8×8 has 64! ≈10^89 permutations.Dense grids are needed to stress-test spatial reasoning under combinatorial complexity.
  • JigShape: JigShape uses tab-and-blank interlocking pieces whose geometric constraints, combined with distinct visual content, yield unambiguous ground truth.The benchmark includes 95K+ instances across 4×4, 8×8, 12×12, and 16×16 grids, plus a shape-versus-no-shape ablation.
  • Key Findings: 69.65% vs. 6.25%: only GPT-5.5 exceeds random baseline on 4×4 puzzles among five frontier models and six configurations; supervised fine-tuning exceeds 97%.Reasoning-enabled Claude Opus 4.8 and Grok-4.2 fail entirely, while fine-tuning Qwen3-VL-8B and Gemma3-12B demonstrates learnability.

2 RELATED WORK

Prior work studies spatial reasoning and jigsaw solving through spatial-relation benchmarks, cognitive tasks, self-supervised training, and computational puzzle methods. Existing jigsaw benchmarks rely on rectangular cuts that can produce ambiguous labels, whereas JigShape adds geometric shape constraints and dense grids for unambiguous evaluation.

  • Spatial Reasoning in VLMs: Spatial benchmarks show that large VLMs struggle with basic positional reasoning and 3D depth and distance estimation.VSR evaluates binary spatial relations, while SpatialBench extends evaluation to 3D spatial understanding.
  • Jigsaw Puzzle Benchmarks for VLMs: 77% overall accuracy was achieved by the best model on Jigsaw-Puzzles, but open-ended order generation reached only 30%.Jigsaw-Puzzles uses 2×2 and 3×3 grids across five cognitive tasks and 1,100 real-world images.
  • Jigsaw Puzzle Benchmarks for VLMs: JigShape introduces tab-and-blank shape constraints combined with visual content to ensure unambiguous ground truth, with dense grids up to 16×16.The benchmark also includes a controlled shape-versus-no-shape ablation.
  • Jigsaw Puzzle Benchmarks for VLMs: Existing jigsaw benchmarks use rectangular cuts, creating ambiguous ground truth when images contain repeated textures and potentially conflating reasoning failures with label arbitrariness.Perceptually equivalent arrangements may exist even though only one is marked correct.
  • Self-Supervised Representation Learning: Jigsaw solving has been used as a self-supervised pretext task, a source of improved domain generalization and Vision Transformer robustness, and a training signal for large VLMs.Visual Jigsaw uses jigsaw solving as reinforcement-learning post-training.
  • Computational Jigsaw Puzzle Solving: Computational jigsaw methods include pairwise color-gradient and texture compatibility, genetic algorithms for large puzzles, loop constraints for global consistency, and deep reinforcement-learning approaches.SD2RL introduces the JPwLEG dataset and a Siamese DQN for puzzles with eroded gaps.

3 BENCHMARK

JigShape evaluates whether VLMs can assign shuffled, labeled jigsaw pieces to grid positions using complementary tab-and-blank geometry and visual content. Its benchmark spans four grid densities, uses 95,468 leakage-controlled instances, and includes a controlled 4×4 rectangular-cut ablation with ambiguous larger-grid ground truth.

  • Task definition: The task is to map each labeled piece ID in a shuffled N × N layout to its predicted (row, col) position in the solved puzzle.Each instance is defined by a source image, grid, edge-type assignment, and random permutation of piece IDs to grid positions.
  • Geometric constraints: Complementary tab-blank edges impose local compatibility, while flat edges identify boundary positions and distinguish corners, edge pieces, and interior pieces.The number of flat edges is 2 for corners, 1 for edge pieces, and 0 for interior pieces.
  • Grid configurations: 4×4, 8×8, 12×12, and 16×16 configurations contain 16, 64, 144, and 256 pieces, respectively, spanning entry-level through extreme combinatorial difficulty.The corresponding search spaces are ∼2 × 1013, ∼1089, ∼10249, and ∼10507 permutations.
  • Dataset construction: 95,468 total instances are generated from 23,742 unique source images across four grid configurations, with source-image splits preventing train-test leakage.Each source image generates one puzzle instance per grid configuration, and the same image does not appear in both training and evaluation sets, even at different grid sizes.
  • Ablation: 500 paired 4×4 ablation instances compare tab-and-blank pieces with rectangular cuts while holding source image and piece permutation fixed.The ablation is restricted to 4×4 because larger rectangular-cut grids can have ambiguous ground truth in repeated-texture regions.

4 EVALUATION PROTOCOL

JigShape evaluation measures piece-level correctness, exact puzzle recovery, local adjacency coherence, and geometric edge compatibility against grid-specific random baselines. Main and auxiliary results use 250 evaluation samples per grid, with frontier VLMs omitted at 16×16 because of poor 12×12 performance.

  • Metrics: Four metrics assess Piece Accuracy, Exact Match, Adjacency Accuracy, and Shape Compatibility, covering placement correctness, complete recovery, local coherence, and geometric compatibility.Exact Match counts instances where all pieces are correct; Shape Compatibility checks whether predicted adjacencies pair tabs with blanks.
  • Evaluation coverage: 250 eval samples per grid are used for all model evaluations in the main results.Table 4 reports Piece Accuracy and Exact Match, while Table 5 reports Adjacency Accuracy and Shape Compatibility.

5 EXPERIMENTS

Experiments show that zero-shot VLMs generally fail to use geometric constraints, with GPT-5.5 the sole exception on 4×4 puzzles. Supervised fine-tuning makes 4×4 puzzles highly learnable but produces a sharp scaling cliff and heavy reliance on shape cues.

  • Zero-shot results: All other zero-shot models achieve near-random 4×4 PA of 6.62–11.97% with 0% EM, while SC remains near the 30.8% random baseline.Their SC values range from 32.9–37.6%, indicating little use of tab-and-blank geometry.
  • Zero-shot results: A 5 percentage point gain from reasoning raises Grok-4.2-reasoning to 11.97% versus 6.90% PA, but the benefit vanishes on larger grids.The result suggests extended reasoning helps process visual-spatial information without enabling reliable geometric constraint satisfaction.
  • Scaling: 4.37% PA on 8×8 and 0.67% PA on 12×12 show GPT-5.5’s 4×4 advantage disappears as grid density increases.On larger grids, SC values of 39.4–42.5% match grid-specific random baselines of 39.0% and 42.3%.
  • Scaling: >97% PA and ∼90% EM on 4×4 confirm supervised learning, but SFT models fall to 27–34% PA on 8×8, 3–5% on 12×12, and 0.35–0.40% on 16×16.Gemma3-12B outperforms Qwen3-VL-8B on 8×8, reaching 33.58% versus 27.34% PA.
  • Shape ablation: Removing shape constraints drops GPT-5.5 from 69.65% to 23.90% PA and SFT models from 97.28–97.82% to 10.00–13.78% PA.The residual SFT accuracies only marginally exceed the 6.25% random baseline, revealing predominant reliance on geometric cues.
  • Zero-shot results: Only GPT-5.5 exceeds random baseline in zero-shot evaluation, reaching 69.65% PA and 26.40% EM on 4×4 puzzles.It also achieves 54.22% AA and 71.33% SC, versus 20.0% and 30.8% random baselines.

6 CONCLUSION

JigShape frames jigsaw solving as joint visual-geometric reasoning, using tab-and-blank constraints and distinct visual content to provide unambiguous evaluation across 95K+ instances and four grid densities. Results show zero-shot geometric reasoning is largely unsolved, models face a scaling cliff, and supervised models primarily exploit geometric cues, leaving scalable geometric reasoning open.

  • Benchmark contribution: JigShape converts jigsaw solving into well-posed joint visual-geometric reasoning through tab-and-blank constraints and distinct visual content.The benchmark spans four grid densities from 4×4 to 16×16 with 95K+ instances.
  • Key findings: 69.65% PA and 26.40% EM: only GPT-5.5 exceeds random baseline on 4×4, while all other frontier models fail.The evaluation identifies zero-shot geometric reasoning as largely unsolved.
  • Key findings: >97% on 4×4: SFT models collapse to near-random performance on 8×8 and larger grids, establishing a scaling cliff.This collapse occurs despite strong performance on the smallest grid.
  • Key findings: 97.28% to 10.00% PA: Qwen3-VL-8B drops sharply without shape constraints, barely exceeding the 6.25% random baseline.The result indicates reliance on geometric cues as the primary signal, with limited integration of visual content.
  • Future directions: Scalable geometric reasoning remains an open challenge requiring curriculum learning, joint visual-geometric training objectives, and compositional constraint-satisfaction architectures.These directions are proposed as future work.

7 ETHICS STATEMENT & REPRODUCIBILITY STATEMENT · A GENERATION PIPELINE DETAILS · B PROMPT TEMPLATES

The paper supports reproducibility through public-data sourcing and released generation and evaluation resources. Its deterministic puzzle pipelines and explicit shape/no-shape prompts standardize both geometric and visual-only evaluation conditions.

  • 7 ETHICS STATEMENT & REPRODUCIBILITY STATEMENT: Publicly available datasets supply all source images, and the complete generation pipeline, evaluation code, and configuration are released.Sources include DIV2K and DIV8K under academic licenses and Unsplash under a permissive license.
  • A GENERATION PIPELINE DETAILS: Shape-version generation resizes images to dimensions divisible by N, samples internal tab-blank polarities, and keeps boundary edges flat.Tabs are semicircular, tab regions carry neighboring pixels, and the pipeline is deterministic given a seed.
  • A GENERATION PIPELINE DETAILS: The shape pipeline randomly permutes piece IDs from 0 to N^2−1 before arranging and numerically labeling pieces by ID.This separates labels from their correct spatial positions while preserving a reproducible arrangement under a fixed seed.
  • A GENERATION PIPELINE DETAILS: The 4×4 no-shape ablation preserves source images, random seed, piece permutations, and grid layout while replacing interlocking boundaries with straight cuts.Thus, tab-blank boundaries are the only difference between the shape and no-shape versions.
  • B PROMPT TEMPLATES: All models receive identical system-and-user prompt structures, with separate templates for shape-constrained and no-shape conditions.Prompts specify task context, random ID assignment, grid size, and output format.
  • B PROMPT TEMPLATES: Shape-constrained prompts require matching visual content with tab-and-blank edge geometry and identifying corner and edge pieces from flat-edge counts.They explicitly warn that randomly assigned labels do not determine positions or piece types.
  • B PROMPT TEMPLATES: Shape prompts direct models to identify corners, identify edge pieces, and match pieces using both visual content and edge shapes.The prescribed output is a JSON mapping from piece_id to 0-indexed [row, col] coordinates.
  • B PROMPT TEMPLATES: No-shape prompts remove interlocking geometry and instruct models to reconstruct the arrangement using visual continuity and matching edge content alone.They retain the warning against inferring positions from randomly assigned labels.

C EXPERIMENT DETAILS

The experiments compare five frontier VLMs across six zero-shot configurations with two fine-tuned models trained on mixed-grid data. All models are evaluated on 1,000 instances, with invalid JSON responses receiving zero accuracy on every metric.

  • Frontier VLMs (Zero-Shot): Five frontier VLMs are evaluated in six zero-shot configurations using identical prompts and high-resolution image input with 128K+ token context windows.The evaluated models are GPT-5.5, GPT-5.4-mini, Grok-4.2, Llama-4-Maverick-17B-128E-Instruct-FP8, and Claude Opus 4.8.
  • Fine-Tuned Models: 2 models, Qwen3-VL-8B and Gemma3-12B, are fine-tuned for 1 epoch on a 6.5K mixed-grid training set with weighted sampling.Training uses learning rate 1 × 10−5, effective batch size 64, BF16 mixed precision, and Flash Attention 2.
  • Evaluation Protocol: 1,000 evaluation instances comprise 250 instances per grid size, and responses with invalid JSON receive zero accuracy on all metrics.Model outputs are parsed to extract piece ID to position mappings.

D EVALUATION METRICS · E ADDITIONAL RESULTS

The evaluation defines four metrics that separately capture per-piece correctness, whole-puzzle success, local adjacency coherence, and geometric edge compatibility. Random baselines are grid-dependent for adjacency and shape compatibility, with shape compatibility requiring correction for boundary effects and sampling dependence.

  • D EVALUATION METRICS: Piece Accuracy (PA) measures the fraction of pieces placed in their correct positions, without considering spatial relationships.
  • D EVALUATION METRICS: Exact Match (EM) is 1 only when all N^2 pieces occupy their correct positions, equivalent to PA = 100%.
  • D EVALUATION METRICS: Adjacency Accuracy (AA) measures the fraction of correctly predicted horizontally or vertically adjacent piece pairs.For an N × N grid, the ground-truth adjacency set contains 2N(N −1) pairs.
  • D EVALUATION METRICS: AA captures local coherence, so models can group pieces correctly even when their global positions are wrong.
  • D EVALUATION METRICS: 20.0% (4×4), 5.6% (8×8), 2.6% (12×12), and 1.5% (16×16) are AA baselines under a uniformly random permutation.
  • D EVALUATION METRICS: Shape Compatibility (SC) measures the fraction of adjacent predicted placements whose facing internal edges are geometrically complementary.Internal edges are randomly assigned tab or blank, while boundary edges are flat.
  • D EVALUATION METRICS: 30.8% (4×4), 39.0% (8×8), 42.3% (12×12), and 44.1% (16×16) are SC random-placement baselines, confirmed by Monte Carlo simulation over 2×10^6 trials.The baselines account for boundary pieces being incompatible when placed at internal positions; the without-replacement correction exceeds the simple approximation.

E.1 WEIGHTED SFT TRAINING · E.2 CLASSICAL SOLVER BASELINE

Weighted SFT that emphasizes harder grids worsens performance across all evaluated settings, indicating that the scaling cliff is architectural rather than caused by training-data distribution. A classical edge-constraint solver reaches 100% accuracy on every tested grid and confirms JigShape’s solutions are unique.

  • E.1 WEIGHTED SFT TRAINING: Inverse-accuracy weighting upweights 12×12 and 16×16 samples while downweighting 4×4 samples during Qwen3-VL-8B fine-tuning.The model is trained for 3 epochs on 20K mixed-grid samples, with weights applied to the loss function.
  • E.1 WEIGHTED SFT TRAINING: Weighted SFT performs significantly worse than baseline SFT across all grid sizes and metrics.Table 8 compares Piece Accuracy (PA), Exact Match (EM), Adjacency Accuracy (AA), and Shape Compatibility (SC).
  • E.1 WEIGHTED SFT TRAINING: Reweighting harder grids does not alleviate the scaling cliff, suggesting a fundamental architectural limitation rather than a data-distribution issue.The underlying architecture remains unable to solve larger puzzles regardless of how training signal is distributed.
  • E.2 CLASSICAL SOLVER BASELINE: The classical solver verifies uniqueness using exact edge signatures, with each piece defined by four tab, blank, or flat edge types.This constraint-based solver is designed to verify that every puzzle instance has a unique solution.
  • E.2 CLASSICAL SOLVER BASELINE: Adjacent pieces must have complementary edges, while flat edges are restricted to puzzle boundaries and determine corner and border placement.A tab fits a blank; corners have two flat edges, and border pieces have one.
  • E.2 CLASSICAL SOLVER BASELINE: Backtracking with constraint propagation precomputes boundary-compatible candidates, fills positions row by row, checks neighboring compatibility, and backtracks when necessary.Corner positions require exactly two flat edges before candidate placement.
  • E.2 CLASSICAL SOLVER BASELINE: 100% accuracy was achieved across 4×4, 8×8, 12×12, and 16×16 grids on all 500 test instances per size.These results confirm that every JigShape puzzle has a unique solution.

E.3 AUXILIARY HINTS DO NOT HELP

Auxiliary hints do not overcome VLMs’ spatial reasoning limitations on larger jigsaw puzzles. Oracle positions leave GPT-5.5 swapping exact locations, while visual-similarity hints reduce performance by over 10%.

  • Partial Ground Truth: 25% PA: GPT-5.5 solves only one-quarter of the remaining pieces when 60 of 64 positions are revealed.The model places the four pieces within the correct 2×2 region but swaps their exact positions.
  • Visual Similarity Hints: Over 10%: visual similarity hints hurt performance rather than improving puzzle solving.The hints cluster pieces using LAB-space color histograms and communicate groupings through natural language.
  • Visual Similarity Hints: The image already provides visual content and geometric cues, so textual hints can create information overload.Models must integrate dense visual features, geometric constraints, and auxiliary guidance simultaneously.
  • Conclusion: Oracle positions and similarity groupings cannot compensate for a fundamental architectural gap in processing spatially dense, geometrically complex inputs.The experiments indicate that the bottleneck is hint integration rather than hint quality.

F DATASET EXAMPLES

The dataset examples illustrate JigShape puzzles across grid sizes, pairing shuffled layouts with source images as ground-truth solutions. Models must map randomly assigned piece IDs to their correct grid locations.

  • Dataset examples: JigShape examples span different grid sizes, showing shuffled puzzle inputs alongside original source images as ground-truth solutions.Each row presents the shuffled layout on the left and the original source image on the right.
  • Dataset examples: Models determine the correct mapping from each randomly assigned piece ID to its grid location.Piece IDs are displayed on the pieces and assigned randomly.
Loading 2607.27670v2…