PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating ChatGPT and GPT-4 for Visual Programming

Adish Singla

arXiv:2308.02522v1cs.LGcs.AIcs.CY

TL;DR

Prior work assessed generative models in text-based Python programming, leaving their capabilities in visual programming uncertain. This paper evaluates ChatGPT and GPT-4 on visual-programming tasks and finds that they perform poorly, struggling to combine spatial, logical, and programming skills.

  • Problem

    Prior studies of generative models in programming education focused on Python, leaving open whether their capabilities extend to visual programming used in K–8 education.

    Source: §1 · p. 1

  • Method

    The paper evaluates ChatGPT and GPT-4 on ten HoCMaze and Karel reference tasks using expert-based annotations.

    Source: §1 · p. 2

  • Results

    Generative models perform poorly in visual programming and struggle to combine spatial, logical, and programming skills, despite GPT-4’s improvement over ChatGPT.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The observed difficulty centers on combining spatial, logical, and programming skills in visual programming.

    Source: §6 · p. 9

    The study covers a small set of basic tasks from two domains and assesses only one generated output per instance.

    Source: §6 · p. 9

Abstract

from arXiv · show

Generative AI and large language models have the potential to drastically improve the landscape of computing education by automatically generating personalized feedback and content. Recent works have studied the capabilities of these models for different programming education scenarios; however, these works considered only text-based programming, in particular, Python programming. Consequently, they leave open the question of how well these models would perform in visual programming domains popularly used for K-8 programming education. The main research question we study is: Do state-of-the-art generative models show advanced capabilities in visual programming on par with their capabilities in text-based Python programming? In our work, we evaluate two models, ChatGPT (based on GPT-3.5) and GPT-4, in visual programming domains for various scenarios and assess performance using expert-based annotations. In particular, we base our evaluation using reference tasks from the domains of Hour of Code: Maze Challenge by Code-dot-org and Karel. Our results show that these models perform poorly and struggle to combine spatial, logical, and programming skills crucial for visual programming. These results also provide exciting directions for future work on developing techniques to improve the performance of generative models in visual programming.

1 Introduction

Generative models could support computing education by generating personalized programming feedback and content, but their visual-programming capabilities remain untested by prior work focused on Python. This study evaluates ChatGPT and GPT-4 in visual programming and finds that GPT-4 improves on ChatGPT but still performs poorly.

1 Introduction

  • Generative models may enhance introductory computing education by producing personalized feedback and content for learners.The paper focuses on visual programming domains used in K–8 education, including HoCMaze and Karel.Source: §1 · p. 1
  • Prior programming-education studies examined text-based Python programming, leaving open whether generative models perform similarly in visual programming.Earlier work covered scenarios including program repair, hint generation, pair programming, and task synthesis.Source: §1 · p. 1
  • GPT-4 substantially improves on ChatGPT, yet remains poor at combining spatial, logical, and programming skills in visual programming.The evaluation uses expert-based annotations on ten reference tasks from HoCMaze and Karel.Source: §1 · p. 2

2 Evaluation Setup

The evaluation covers three visual-programming scenarios across ten reference tasks in HoCMaze and Karel. It characterizes solution complexity and evaluates model outputs through repeated queries, selected outputs, and expert annotations.

2 Evaluation Setup

  • The evaluation considers execution tracing, solution synthesis, and task synthesis as distinct visual-programming capabilities.These scenarios respectively concern analyzing code behavior, generating code for a task, and generating tasks that exercise concepts.Source: §2 · p. 2
  • Ten reference tasks come from HoCMaze and Karel, with illustrative tasks using grid-based worlds and block-limited solutions.The HoCMaze example uses an 8×8 grid and a five-block minimal solution; the Karel example transforms paired 10×10 grids using an eight-block minimal solution.Source: §2 · p. 2, Figure 2 · p. 3, Figure 3 · p. 3
  • Solution-code complexity is described by block count, abstract-syntax-tree depth, and the nesting structure of programming constructs.The paper refers to for a more formal specification of the task and code spaces.Source: §2 · p. 3
  • For each method and scenario, evaluation uses ten instances corresponding to the reference tasks, with multiple queries and a manually selected output for each instance.The selected output is the one judged best on scenario-specific metrics.Source: §2 · pp. 3–4
  • One human expert annotates generated outputs against scenario-specific metrics, and results are averaged across the ten instances.Performance is assessed with expert-based annotation because metric assessment is challenging to fully automate.Source: §2 · p. 4

3 Execution Trace Scenario

In execution tracing, GPT-4 outperforms ChatGPT on overall performance but still struggles on elementary-level tasks. The scenario tests whether models can trace code behavior on visual tasks, including correct state transitions and condition sensing.

3 Execution Trace Scenario

  • The task and solution code are inputs, and the requested output is an execution trace listing avatar positions during code execution.The HoCMaze prompt is presented in Figure 4; the generated trace is manually extracted for evaluation.Source: §3 · pp. 4–5
  • 60.0 Overall for GPT-4 versus 10.0 for ChatGPT shows a substantial execution-trace improvement, especially in transitions and sensing.The results are aggregated across ten instances and reported as percentages.Source: Figure 5 · p. 5, §3 · p. 5
  • TraceCorrect tests full trace agreement, while TracePartialCorrect allows a trace that could become correct after a few modifications.TransitionsCorrect checks position changes against actions, SensingCorrect checks branch choices against Boolean conditions, and Overall requires TraceCorrect, TransitionsCorrect, and SensingCorrect all to be 1.Source: §3 · p. 5
  • GPT-4 achieves only 60.0 overall performance on these elementary-level visual-programming tasks, and Figure 6 illustrates its struggles generating an execution trace for HoCMaze:18.Source: §3 · p. 5

4 Solution Synthesis Scenario

The solution-synthesis scenario tests whether models can generate minimal-sized code for a given visual programming task, a capability relevant to completing students’ partial programs or providing next-step hints. GPT-4 improves on ChatGPT in Overall performance but still struggles to produce compact, task-specific solutions.

Solution synthesis scenario

  • The scenario targets completing students’ partial programs or giving next-step hints.Source: §4 · p. 6

Prompt and output generation

  • The input is a task, and the desired output is a minimal-sized solution code, manually extracted from the model’s response for evaluation.Source: §4 · p. 6

Output quality and performance metrics

  • Binary quality attributes assess syntax, task solving, similarity to a minimal solution, and whether generated code’s size and depth stay within the minimal solution’s limits.Overall requires correct syntax, either task solving or code similarity, and acceptable size and depth.Source: §4 · pp. 6–7

Results

  • 40.0 Overall for GPT-4 versus 10.0 for ChatGPT; GPT-4 improves on minimal-solution similarity, while ChatGPT does better on CodeSolvesTask but often produces generic code that ignores the task.GPT-4’s Overall performance is still only 40.0.Source: Figure 8 · p. 7, §4 · p. 7
  • GPT-4’s HoCMaze:18 code solves the task but uses 9 blocks and depth 4, compared with 5 blocks and depth 3 for the minimal solution.Source: Figure 9 · p. 7

5 Task Synthesis Scenario

The task-synthesis scenario asks models to create a visual programming task for a supplied solution code. Both models perform poorly, and GPT-4’s generated task is often not solvable by that code, even for low-complexity examples.

Task synthesis scenario

  • The scenario is motivated by providing new practice tasks to students in visual programming domains.Source: §5 · p. 7

Prompt and output generation

  • The input is solution code, and the desired output is a task that the code would solve; the model’s response is manually extracted for evaluation.Source: §5 · pp. 7–8

Output quality and performance metrics

  • Binary attributes assess task layout, whether the input code or edited input code solves the task, and whether any code can solve it; Overall requires correct layout and solvability by the input or edited input code.Source: §5 · p. 8

Results

  • GPT-4 achieves 20.0 Overall versus 10.0 for ChatGPT, yet the generated task is not solvable by the input code.GPT-4 is slightly better on LayoutCorrect and TaskSolvable, but both models perform poorly.Source: §5 · pp. 8–9, Figure 11 · p. 9
  • GPT-4 struggles to generate tasks even for elementary-level, low-complexity solution code; an example task it generates cannot be solved by the input code.Source: §5 · p. 9, Figure 12 · p. 9

6 Concluding Discussions

The authors conclude that generative models perform poorly in visual programming and struggle to combine the spatial, logical, and programming skills it requires. They identify limited task coverage and single-output assessment as study limitations, alongside directions for future work.

Conclusion

  • The models struggle to combine spatial, logical, and programming skills crucial for visual programming, where their performance is poor compared with their advanced text-based Python capabilities.Source: §6 · p. 9

Limitations

  • The study covers only a small set of basic tasks from two visual programming domains and assesses one generated output per instance.The authors suggest larger, more complex task sets and multiple outputs per instance to account for model stochasticity.Source: §6 · p. 9

Future work

  • Future directions include novel visual programming benchmarks, evaluation of alternate models including open-source variants, and techniques such as symbolic methods, automated prompting, or fine-tuning.Source: §6 · p. 9

Appendix

The appendix provides prompts for all three evaluation scenarios in the Karel domain: execution trace, solution synthesis, and task synthesis.

Appendix

Loading 2308.02522v1…