Source-linked AI summary
ScratchEval : A Multimodal Evaluation Framework for LLMs in Block-Based Programming
Yuan Si, Simeng Han, Daming Li, Hanyuan Shi, Jialu Zhang
TL;DR
LLMs perform strongly on textual programming but remain unreliable on Scratch, where concurrency, shared state, and multimodal assets complicate semantics and repair. ScratchEval introduces a 100-project executable benchmark with controlled bugs and VM-level evaluation, and uses it to study model repair behavior, including recurring near-miss patches that fail under whole-program execution. The benchmark offers a reproducible foundation for evaluating and improving LLMs on block-based programming.
Problem
Scratch lacks rigorous executable benchmarks that assess LLM understanding, repair quality, semantic fidelity, and explanation correctness beyond coarse pass/fail outcomes.
Method
ScratchEval pairs 100 curated Scratch projects with controlled reversible bugs, gold fixes, executable tests, and a three-layer VM-based protocol for correctness, repair quality, and explanation faithfulness.
Results
ScratchEval reveals recurring near-miss repairs that appear locally plausible but fail under VM execution because they violate whole-program event-driven semantics.
Takeaways & Limitations
The benchmark establishes a closed-loop, execution-based foundation for studying and improving LLM understanding, debugging, and repair on block-based programs.
Takeaways & Limitations
Finite synthesized test suites are objective and reproducible but incomplete, so passing tests does not guarantee semantic correctness on untested behaviors.
Abstract
from arXiv · showhide
LLMs have achieved strong performance on text-based programming tasks, yet they remain unreliable for block-based languages such as Scratch. Scratch programs exhibit deeply nested, non-linear structures, event-driven concurrency across multiple sprites, and tight coupling between code and multimedia assets, properties that differ fundamentally from textual code. As a result, LLMs often misinterpret Scratch semantics and generate large, invasive edits that are syntactically valid but semantically incorrect when repairing buggy programs. We introduce ScratchEval, the first executable benchmark designed to evaluate LLM-based repair for Scratch programs, covering program understanding, debugging, analysis, and repair. The benchmark contains 100 curated Scratch projects from the public repository, selected for structural and semantic complexity. Each project is paired with executable test suites, bug descriptions with corresponding fixes, block-level edit constraints defining minimal semantically correct repairs, and required multimedia assets. The benchmark is constructed through a human-in-the-loop pipeline combining automated project mining with expert validation of trigger-outcome semantics and representative bug patterns, with emphasis on event ordering, concurrency, and state management. To enable rigorous and reproducible evaluation, we propose a three-layer executable protocol measuring functional correctness via VM-level execution, repair quality using block-level edit distance and behavioral trajectory comparisons, and explanation quality via structured rubrics assessing alignment between model reasoning and generated patches. Using ScratchEval, we study domain-specific fine-tuning, training data effectiveness, and model generalization to unseen bug types. ScratchEval provides a reproducible foundation for evaluating and post-training LLMs on block-based programming tasks.
1 Introduction
ScratchEval addresses the lack of rigorous, executable benchmarks for evaluating LLMs on block-based programming. It combines a curated Scratch benchmark with VM-based assessment of correctness, repair quality, and explanation grounding.
- Block-based languages remain less understood because their structure, execution model, and representation differ fundamentally from textual languages.
- ScratchEval targets event-driven concurrency, state management, and cross-sprite interactions that frequently cause semantically incorrect or over-invasive LLM repairs.
- The benchmark contains 100 complex Scratch projects, each paired with executable tests, bug descriptions, gold fixes, minimal-edit constraints, and required multimedia assets.
- Its three-layer protocol measures functional correctness, repair minimality and semantic fidelity, and explanation faithfulness under Scratch’s event-driven execution model.
- The empirical study examines domain-specific fine-tuning, training-data effectiveness, and generalization to unseen bug types.
- ScratchEval contributes a VM-executable methodology, a controlled benchmark, and a systematic study of LLM understanding, diagnosis, and repair behavior.
2 Background and Motivating Examples
Scratch programs combine nested visual blocks, asynchronous scripts, shared state, and multimedia assets, making their behavior difficult to analyze with text-oriented methods. Motivating examples show that correct repairs are often small and localized, whereas LLM patches can disrupt synchronization or add unnecessary logic.
- Scratch behavior emerges from independent event-driven scripts distributed across sprites and coupled to graphics, costumes, and sounds.
- Existing evaluations often use ad hoc examples or manual inspection without ground-truth fixes and reliable test oracles.
- Concurrency: Replacing broadcast-and-wait with broadcast can create a race condition, although restoring the synchronization requires only one local change.
- State initialization: A missing score initialization causes values to accumulate across runs, while the minimal repair is a single inserted or relocated initialization block.
- Clone management: Clone leaks can produce incorrect behavior and are typically fixed by adding a single delete this clone block.
- Visibility toggle: Faulty visibility toggles can be repaired locally, but LLMs may add auxiliary counters or arithmetic state that increases complexity and failure modes.
- These examples motivate evaluation that emphasizes precise, minimal repairs because LLM-generated fixes may be overly complex or nonfunctional.
3 System Design
ScratchEval constructs executable benchmark instances from curated Scratch projects through controlled bug injection, oracle synthesis, and structured model-facing artifacts. Its evaluation applies machine-readable patches in the Scratch VM and measures both success and behavioral similarity to minimal gold repairs.
- The pipeline produces project snapshots with assets, one reversible buggy edit, its inverse gold fix, executable tests, and evaluation metadata.
- Five stages cover project collection, expert curation, bug synthesis and annotation, test-oracle synthesis, and evaluation integration.
- The design exercises Scratch-specific concurrency, non-linear control flow, and code–asset coupling through reproducible VM-level evaluation.
- Project selection: Projects are mined using interaction and complexity criteria, then manually reviewed to select 100 diverse, semantically nontrivial projects.
- Reference semantics: Reference semantics record project goals, event hooks, involved sprites, and intended observable outcomes without revealing fix-specific ground truth.
- Bug synthesis: Bug Forge applies one localized, reversible JSON-IR transformation per project and records paired forward and inverse patches for reproducible repair evaluation.
- Repair evaluation: Repair patches are structured JSON lists of atomic block edits identified by sprite and block identifiers, making application unambiguous.
- Repair evaluation: Patch quality combines symmetric-difference edit distance with normalized VM-trace drift against the gold project.
4 Evaluation
ScratchEval evaluates LLM understanding and repair of Scratch programs using deterministic, VM-based experiments and structured metrics. Results show stronger global project understanding than bug diagnosis or repair, while Qwen LoRA tuning improves understanding and patch quality more than repair success.
- Evaluation setup: Models were evaluated with deterministic zero-shot decoding, while Qwen additionally used two-fold held-out LoRA adaptation.Inputs included buggy .sb3 projects and original multimedia assets; outputs were evaluated locally through the Scratch VM pipeline.
- Global project understanding: Global understanding reached 81% for Gemini, 79% for ChatGPT, and 59% for Qwen; Qwen LoRA tuning increased it from 59% to 65%.The paired McNemar test reported n01 = 6, n10 = 0, p = 3.13 × 10^-2, and N = 100 for Qwen’s change.
- Bug understanding: Bug understanding remained difficult: baseline U-Acc was 57% for ChatGPT, 66% for Gemini, and 40% for Qwen, with Qwen LoRA reaching 42%.U-Acc requires jointly identifying a bug’s trigger, mechanism, and outcome; tuned explanations improved grounding modestly but often still deviated from the true mechanism.
- Trigger identification: Baseline trigger F1 was 0.38 for Qwen, 0.52 for ChatGPT, and 0.55 for Gemini; Qwen LoRA raised trigger detection from 0.38 to 0.45.Even the best zero-shot trigger score remained 0.55, reflecting difficulty with event interdependencies.
- Mechanism classification: Baseline mechanism accuracy was 62% for ChatGPT, 65% for Gemini, and 50% for Qwen; Qwen LoRA increased Qwen’s score from 50% to 58%.Across models, approximately 35–50% of mechanism instances remained misclassified, especially in concurrency-intensive scenarios.
- Repair success and quality: Zero-shot repair success was 41% for Gemini, 32% for ChatGPT, and 23% for Qwen, while patches averaged about 4 extra block edits and a 0.30 drift score.Qwen LoRA raised success from 23% to 26% without statistical significance, but reduced average edit distance from about 4 to 2 blocks and drift from 0.30 to 0.15.
5 Discussion
ScratchEval shows that locally plausible Scratch repairs often fail because they violate whole-program temporal or binding constraints enforced by the VM. These near-misses motivate semantic validation and favoring minimal, representation-aware edits over local heuristics.
- Failure patterns: Near-miss patches look plausible within individual scripts but fail under VM execution because they violate whole-program semantic commitments.The recurring failures are not usually malformed edits; they arise from Scratch’s event-driven runtime and serialized .sb3 representation.
- Failure patterns: Temporal constraints make synchronization primitives such as broadcast and wait difficult to preserve through local reasoning.Replacing broadcast and wait with broadcast lets later blocks run before receivers update shared state, while fixed delays and polling alter scheduling.
- Failure patterns: Binding constraints require edits to preserve stable variable and sprite identifiers rather than merely matching displayed labels.A correct repair must update the intended .sb3-level identifier and scope consistently across reads and writes.
- Benchmark design: ScratchEval’s curated projects prioritize semantic density and executable rigor over raw scale, supporting multiple evaluations per instance.Each project is manually curated, paired with VM-executable tests, and annotated with reversible minimal repairs for understanding, diagnosis, repair, minimality, and explanation grounding.
- Implications: Repair systems should pair patch generation with lightweight validators for synchronization and .sb3 binding contracts.Near-miss patches make divergences between local plausibility and executable correctness measurable under VM oracles and minimality constraints.
6 Threats to Validity
ScratchEval improves evaluation rigor through controlled execution and rerun stabilization, but residual nondeterminism, incomplete tests, metric limitations, judge bias, and limited scenario coverage remain.
- Internal validity: Residual variability from timing and event interleavings may leave a small number of borderline cases despite fixed seeds and rerun stabilization.The evaluation pipeline is also dependent on correct parsing, patch application, and VM execution.
- Construct validity: VM-executed test suites provide objective, reproducible oracles but cannot guarantee semantic correctness for untested behaviors.Minimality, behavioral drift, and differential traces mitigate this incompleteness, but regressions outside evaluated scenarios may remain.
- Construct validity: Block-level edit distance, behavioral metrics, and rubric-guided LLM judging approximate repair and explanation quality without capturing every subjective criterion.Some preference or calibration bias may persist in the explanation judge.
- External validity: ScratchEval does not cover all possible Scratch scenarios and is intended as an extensible foundation for future projects, bug patterns, and evaluation refinements.Its rigor and reproducibility come from curated instances, controlled bug injection, and executable VM-level evaluation.
7 Related Work
Related work supplies extensive benchmarks and tooling for text-based programming and Scratch analysis, but ScratchEval targets the missing combination of block-based programs, executable repair evaluation, and LLM support.
- Code and repair benchmarks: Established benchmarks such as HumanEval, CodeXGLUE, SWE-bench, and Defects4J primarily evaluate text-based programming tasks.These resources support code generation, understanding, software engineering, or repair with executable tests, but do not include graphical or block-based programs.
- Positioning: ScratchEval fills the gap with a curated, executable benchmark specifically designed for LLM-based repair of block-based Scratch programs.Its contribution combines block-based artifacts with executable tests and repair-focused evaluation.
- Scratch analysis and tooling: Prior Scratch work includes verification, testing, debugging, evolutionary repair, static analysis, and automated testing infrastructure.Examples include Bastet, NuzzleBug, RePurr, Hairball, Dr. Scratch, LitterBox, and Whisker.
- LLM support for Scratch: LLM systems for Scratch have mainly supported creativity, ideation, planning, storyboarding, and project structuring rather than executable repair benchmarking.This contrasts with LLM debugging and feedback work in text-based programming.
8 Conclusion
ScratchEval is an executable benchmark for systematic evaluation of LLM understanding and repair on Scratch. Its controlled bugs, VM tests, minimality measures, and explanation assessment support reproducible study of grounding and over-editing.
- Benchmark: ScratchEval contains 100 semantically complex projects with controlled reversible bugs, minimal gold fixes, and VM-executed interaction tests.The benchmark uses rerun-stabilized oracle assertions to support closed-loop evaluation.
- Evaluation: The benchmark evaluates functional repair success, patch minimality and semantic fidelity, and explanation faithfulness to trigger–mechanism–outcome semantics.These dimensions support execution-based assessment of LLM behavior on block-based, event-driven debugging and repair.
- Conclusion: ScratchEval highlights persistent challenges in grounding fixes and avoiding over-editing.The authors intend its public release and harness to support comparisons and future progress on LLMs for block-based programming.