Source-linked AI summary
ShapeCodeBench: A Renewable Benchmark for Perception-to-Program Reconstruction of Synthetic Shape Scenes
Shivam Kumar
TL;DR
ShapeCodeBench addresses the need for renewable, controlled evaluation of models that reconstruct executable drawing programs from raster images. It combines a restricted four-primitive DSL, seeded scene generation, deterministic rendering, and automatic raster metrics on a reproducible 150-sample split. Multimodal models recover much foreground structure but achieve low exact match, while the benchmark remains unsaturated; zero-shot evaluation and the absence of a human baseline limit interpretation.
Problem
Existing image-to-code evaluations provide limited combinations of deterministic scoring, controlled generation, and cheap renewable splits for measuring perception-to-program reconstruction.
Method
ShapeCodeBench uses a four-primitive DSL, seeded difficulty-controlled scenes, deterministic re-rendering, and five automatic raster metrics on reproducible eval_v1 data.
Results
GPT-5.5/extra_high reaches mean foreground IoU 0.87, while best multimodal exact match is 0.027 and the classical heuristic reaches easy-tier exact match 0.26.
Takeaways & Limitations
ShapeCodeBench provides a renewable, auditable evaluation resource for distinguishing foreground perception from structured code emission.
Takeaways & Limitations
The study evaluates zero-shot systems without chain-of-thought or few-shot prompting and reports no human performance baseline.
Abstract
from arXiv · showhide
We introduce ShapeCodeBench, a synthetic benchmark for perception-to-program reconstruction: given a rendered raster image, a model must emit an executable drawing program that a deterministic evaluator re-renders and compares with the target. The v1 DSL has four primitives on a 512 x 512 black-on-white canvas, but every instance is generated from a seeded RNG, so fresh held-out sets can be created to reduce exact-instance contamination. We release a frozen eval_v1 split with 150 samples across easy, medium, and hard tiers, scored by exact match, pixel accuracy, foreground IoU, parse success, and execution success. We evaluate an empty-program floor, a classical computer-vision heuristic, Claude Opus 4.7 at high and max effort, and GPT-5.5 at medium and extra_high reasoning effort. The heuristic is competitive on easy scenes but collapses when overlaps fuse components; the strongest multimodal configuration preserves much of the foreground structure but still misses exact match because of small parameter errors. Best overall exact match remains low, so ShapeCodeBench is far from saturated. The benchmark code, frozen dataset, run artifacts, and paper sources are released to support independent replication and extension.
1 Introduction
ShapeCodeBench targets perception-to-program reconstruction with a controlled, deterministic, renewable benchmark. Its initial evaluation shows substantial multimodal progress on foreground structure but low exact reconstruction, leaving the task unsaturated.
- Motivation: ShapeCodeBench combines deterministic rendering, render-based scoring, controlled generation, and renewable held-out evaluation to support objective regression feedback.Fresh seeds reduce exact-instance contamination, although models may still learn the generator distribution.
- Benchmark: The benchmark uses a four-primitive drawing DSL on a fixed 512×512 black-on-white canvas with seeded difficulty controls and deterministic raster comparison.Controls include shape count, size, stroke width, overlap, and canvas clipping.
- Benchmark: eval_v1 contains 150 deterministically seeded samples, with published per-sample raster hashes for reproducible evaluation.The split spans three difficulty tiers.
- Evaluation: The evaluation includes empty-program, classical-CV, Claude Opus 4.7, and GPT-5.5 baselines, with recorded configurations and artifacts for auditability.The runner stores prompts, configurations, outputs, normalized predictions, metrics, and per-sample artifacts.
- Results: 0.87 mean foreground IoU is achieved by GPT-5.5/extra_high, while best multimodal exact match is 0.027 and easy-tier heuristic exact match is 0.26.The heuristic is competitive on easy scenes, whereas multimodal systems preserve foreground structure without consistently recovering exact parameters.
2 Related Work
ShapeCodeBench extends visual program induction and synthetic diagnostic benchmarking into a renewable, controlled image-to-code evaluation. Its design prioritizes reproducibility and failure attribution over realism or broad task coverage.
- Positioning: ShapeCodeBench sits at the intersection of visual program induction, synthetic diagnostic benchmarks, and multimodal image-to-code evaluation.It is positioned as complementary to existing benchmark families rather than a replacement.
- Visual program induction: Prior visual program-induction systems establish that executable programs can be recovered from images, but they are not benchmark-first evaluations of modern multimodal models.The prior work spans constructive geometry, scene DSLs, perspective scenes, repeated 3D structure, parametric primitives, and reusable program libraries.
- Benchmark design: ShapeCodeBench adopts explicit variation axes and seed-based regeneration from synthetic diagnostic benchmarks to expose reasoning failure modes.This approach favors factorized control over a large noisy dataset.
- Renewability: Unlike natural-data benchmarks, renewable evaluation mints controlled instances from fresh seeds and scores them through deterministic rendering.This follows the round-trip evaluation philosophy associated with Image2Struct and related renewable benchmarks.
- Closest predecessors: Compared with TurtleBench, ShapeCodeBench uses a tiny shape-primitive DSL, fresh-seed renewability, and deterministic raster scoring to isolate perception-plus-emission.Compared with broader image-to-code benchmarks, it removes confounders such as OCR, library conventions, rendering variability, and external assets.
- Training relevance: The benchmark is currently an evaluation resource rather than a training system, although its execution feedback motivates future use as a verifiable training environment.The paper explicitly does not train or fine-tune models on the task.
3 Benchmark Design
ShapeCodeBench defines a restricted executable DSL, deterministic scene generation and rendering, and raster-based evaluation across five metrics. Seeded tier constraints provide controlled difficulty while preserving reproducibility.
- The ShapeCodeBench DSL: A ShapeCodeBench program is a sequence of one-line top-level calls using exactly four shape primitives.The primitives are filled_circle, circle, filled_square, and square.
- The ShapeCodeBench DSL: The parser accepts only whitelisted top-level calls with integer keyword arguments and rejects unsupported Python constructs and invalid parameter ranges.Canonical serialization fixes one call per line, keyword order, and whitespace.
- Renderer: The renderer produces a 512×512 8-bit grayscale image with black shapes on a white background using Pillow drawing primitives.Program order is preserved, but the binary palette makes scenes order-invariant because later shapes cannot erase foreground pixels.
- Scene generator: Seeded rejection sampling controls shape count, extent, stroke width, canvas clipping, and bounding-box overlap, with tiers optionally requiring overlap.These constraints define the benchmark’s controlled difficulty structure.
- Dataset: eval_v1 uses seeds 0−49 per tier for 150 total samples and publishes SHA-256 hashes alongside PNG, metadata, and ground-truth program information.Metadata records split, difficulty, canvas size, shape inventory, and render configuration.
- Evaluation: The evaluator parses a predicted program, renders it, and compares target and prediction rasters using exact match, pixel accuracy, foreground IoU, parse success, and execution success.Parse or execution failures set all similarity metrics to 0 and are recorded by failure type.
4 Experiments
The experiments compare two non-LLM baselines with four multimodal-model configurations under a common evaluation and reporting pipeline. Results are aggregated across eval_v1 and decomposed by difficulty, revealing distinct performance patterns across systems.
- Systems: Six systems are evaluated: an empty-program baseline, a classical-CV heuristic, and four configurations of Claude Opus 4.7 and GPT-5.5.The multimodal configurations vary by model and reasoning-effort setting.
- Baselines: The empty-program baseline always emits an empty string, causing every sample to fail parsing.
- Baselines: The classical-CV heuristic thresholds images, labels connected components, classifies primitive geometry and fill state, and estimates stroke widths from component measurements.
- Protocol: All LLM configurations use the same zero-shot prompt and shared output normalizer, without chain-of-thought prompting or few-shot examples.The normalizer extracts likely DSL code while preserving raw responses when parsing fails.
- Metrics and reporting: The evaluation reports aggregated metrics across all 150 samples, with Table 2 defining exact match, pixel accuracy, foreground IoU, parse success, and confidence intervals.Figure 2 decomposes exact match by difficulty, while Figure 3 decomposes all four scored metrics.
- Results: Exact match collapses on hard scenes for every system, whereas foreground IoU degrades more gradually; the heuristic is competitive on easy scenes but is outclassed on hard scenes.LLMs can enumerate and place overlapping shapes that classical connected components cannot individuate.
5 Analysis
The analysis shows structured failures and tier-dependent performance: exact match worsens with difficulty, while foreground overlap degrades more gradually. The heuristic excels on easy scenes but loses its advantage when overlap fuses components, whereas multimodal models preserve structure yet miss precise parameters.
- Error taxonomy: All 150 Empty-Program samples receive the empty_program parse error, establishing the intended evaluation floor.An empty DSL program is a parse failure by construction, not a pathology.
- Error taxonomy: LLM parse failures are small but include out_of_range and invalid_stroke violations, showing that format understanding does not ensure valid parameter ranges.These errors involve coordinates or extents outside valid ranges and stroke widths exceeding documented limits.
- Error taxonomy: The Heuristic-CV baseline has zero parse failures because it emits only constructible programs, with errors instead appearing as low foreground IoU.Its bottom-up construction avoids parser violations but cannot guarantee accurate scene reconstruction.
- Qualitative wins and losses: Qualitative losses for the best exact-match multimodal configuration cluster around small parameter errors, missed occluded shapes, and hollow-versus-filled confusion.The figure compares target images, re-rendered predictions, and foreground-XOR differences.
- Difficulty validity: Exact-match rate falls monotonically from easy to hard for every system, while foreground IoU follows the same ordering with shallower degradation.This pattern supports the validity of the benchmark’s difficulty tiers as performance gradients rather than arbitrary labels.
- Heuristic vs. LLM gap: The heuristic is competitive on easy scenes because separated, unclipped components support direct connected-component matching and approximate stroke estimation.Hollow-versus-filled classification also rarely errs on easy scenes.
- Heuristic vs. LLM gap: On medium and hard scenes, overlap or clipping merges components and sharply harms heuristic IoU, while multimodal models retain spatial structure but miss exact parameters.Multimodal predictions often contain the right number of shapes in roughly the right positions but cannot parameterize them precisely under occlusion.
6 Limitations and Future Work
ShapeCodeBench v1 deliberately limits palette, primitives, prompting, model coverage, and training use, while keeping evaluation reproducible. Future work targets richer scenes, broader evaluation, human calibration, contamination-resistant refreshes, and training-oriented extensions.
- Scope boundaries: V1 uses a monochrome palette, so draw order is not first-class; multiple colors or an explicit clear primitive would enable sharper compositionality tests.Later shapes can add foreground pixels but cannot erase or overwrite earlier ones.
- Scope boundaries: The DSL currently supports only filled and hollow circles and squares, leaving rectangles, lines, polygons, and curves for future work.Expanding primitives would stress different kinds of visual reasoning and may move saturation further out.
- Evaluation design: Evaluation is zero-shot, without chain-of-thought prompting or few-shot examples, which may change model ordering on the reasoning-heavy hard tier.These prompting strategies remain natural experimental knobs.
- Reproducibility: Reported model scores cannot be guaranteed bit-exactly reproducible because hosted multimodal inference may vary across repeated requests.The frozen images, parser, renderer, and scorer are deterministic, while each run records configuration and invocation settings.
- Evaluation design: The paper reports no human performance baseline, leaving the benchmark’s practical difficulty uncalibrated against human performance.The authors plan an informal human baseline on a small sample in a subsequent revision.
- Evaluation design: Model coverage is limited to Claude Opus 4.7 and GPT-5.5 across two reasoning-effort tiers each.Other frontier multimodal systems can be added through the ModelAdapter Protocol.
- Evaluation design: Fresh seeds can reduce exact-instance contamination, but public current seeds require regenerated eval_vN splits for contamination-free evaluation.The full 150-sample split regenerates in under a second.
- Future work: The evaluator is an offline benchmark rather than a differentiable pretraining loss or complete training pipeline.Future training would require supervised pairs, render-based rewards, learned critics, or differentiable rendering, with held-out evaluation seeds reserved for clean testing.
A Reproducibility Details
The appendix documents the operational paths used for the reported multimodal sweeps and standardizes their outputs through shared normalization and artifact recording. GPT-5.5 runs use Codex, while Claude Opus 4.7 runs use Claude Code, with retries and long per-sample timeouts.
- OpenAI Codex invocation: GPT-5.5 runs use the OpenAI Codex CLI with image input, model selection, temporary workdir, and configurable medium or extra_high reasoning effort.The command writes the final response to an output path for normalization.
- OpenAI Codex invocation: The Codex sweeps use two retries with exponential backoff and 1800-second medium or 2400-second extra_high per-sample timeouts.These settings define the operational tolerance for the reported runs.
- Claude invocation: Claude Opus 4.7 runs use the Claude Code CLI with the 1M-context model, high or max effort, text output, and no session persistence.The target image is exposed through the prompt and sample-directory access.
- Shared execution workflow: Both multimodal paths receive the same zero-shot prompt and pass through the same prediction normalizer before parsing and rendering.Run artifacts store raw and normalized responses, latency, adapter metadata, evaluation results, and aggregate summaries.