PaperlayerView exact revision on arXiv

Source-linked AI summary

BigCodeArena: Unveiling More Reliable Human Preferences in Code Generation via Execution

Terry Yue Zhuo, Xiaolong Jin, Hange Liu, Juyong Jiang, Tianyang Liu, Chen Gong, Bhupesh Bishnoi, Vaisakhi Mishra, Marek Suppa, Noah Ziems, Saiteja Utpala, Ming Xu, Guangyu Song, Kaixin Li, Yuhan Cao, Bo Liu, Zheng Liu, Sabina Abdurakhmanova, Wenhao Yu, Mengzhao Jia, Jihan Yao, Kenneth Hamilton, Kumar Shridhar, Minh Chien Vu, Dingmin Wang, Jiawei Liu, Zijian Wang, Qian Liu, Binyuan Hui, Meg Risdal, Ahsen Khaliq, Atin Sood, Zhenchang Xing, Wasi Uddin Ahmad, John Grundy, David Lo, Banghua Zhu, Xiaoning Du, Torsten Scholak, Leandro von Werra

arXiv:2510.08697v2cs.SEcs.AIcs.CL

TL;DR

Judging generated code from static text is difficult because understanding its behavior and correctness can require expertise and execution. BIGCODEARENA collects human preferences through interactive code execution and supports two evaluation benchmarks. The authors report that execution results improve most LLM judges’ reliability, and GPT-5 leads the evaluated frontier models in code generation.

  • Problem

    Static code snippets can be difficult to judge because understanding code semantics and runtime behavior is mentally demanding and often requires expertise.

    Source: §1 · p. 2

  • Method

    BIGCODEARENA collects human preferences by enabling execution and interactive testing of generated code, and the paper releases BIGCODEREWARD and AUTOCODEARENA.

    Source: §1 · pp. 2–3

  • Results

    Most LLM judges become substantially more reliable when execution results are available, and GPT-5 leads code generation among the evaluated frontier LLMs.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors present execution-based human evaluation and its associated benchmarks as tools for examining code-generation quality and alignment with human preferences.

    Source: §1 · p. 3, §7 · p. 10

    Current reward models for code generation remain short of human-level perception and reasoning.

    Source: §7 · p. 11

Abstract

from arXiv · show

Crowdsourced model evaluation platforms, such as Chatbot Arena, enable real-time evaluation from human perspectives to assess the quality of model responses. In the coding domain, manually examining the quality of LLM-generated content is extremely challenging, as it requires understanding long chunks of raw code and deliberately simulating code execution. To this end, we introduce BigCodeArena, an open human evaluation platform for code generation backed by a comprehensive and on-the-fly execution environment. Built on top of Chatbot Arena, BigCodeArena enables the execution of LLM-generated code and allows humans to interact with the execution process and outcomes. We collected over 14,000 raw code-centric conversation sessions across 10 widely used LLMs, spanning 10 languages and 8 types of execution environments. Among these conversations, we identified more than 4,700 multi-turn samples with pairwise human preferences. Further analysis uncovers underexplored preferences of LLMs in fine-grained domains characterized by tasks, languages, and frameworks. To systematically examine code understanding and generation capabilities of frontier LLMs, we curated two benchmarks based on the collected data, namely BigCodeReward and AutoCodeArena. For BigCodeReward, we post-processed the 4,700 conversations and evaluated the consistency between reward models and human preferences. The evaluation shows that most LLMs have superior performance in judging coding preferences when the execution results are available. Inspired by these findings, we propose AutoCodeArena, an automatic Elo rating benchmark designed to assess the coding quality of LLMs without human involvement. We find that proprietary LLMs like GPT-5, Claude-Sonnet-4, and Claude-Opus-4 still lead in code generation performance among recent emerging models.

1 INTRODUCTION

Code is difficult to evaluate from static snippets because judging behavior and correctness can require expertise and execution. BIGCODEARENA adds interactive execution to human preference evaluation and supports analyses and benchmarks of code-generation preferences.

1 INTRODUCTION

  • Execution feedback helps people judge code quality more reliably than static inspection, which can make runtime behavior and correctness difficult to assess.The platform enables on-the-fly compilation and execution, interactive debugging through code editing, and direct UI interaction.Source: §1 · p. 2
  • The platform collected more than 14K crowdsourced conversation sessions on code-generation tasks, spanning 10 languages, including Python, Golang, and JavaScript, and 8 execution environments, including PyGame, React, and Mermaid.The collected conversations included a subset of 4.7K multi-turn pairwise conversations.Source: §1 · pp. 2–3
  • BIGCODEARENA data reveals differences in model strengths across languages and framework preferences, alongside varied coding scenarios such as web design and scientific computing.The identified scenarios also include game development, diagram creation, creative coding, and problem solving.Source: §1 · p. 3
  • BIGCODEREWARD evaluates reward-model alignment with human judgments, while AUTOCODEARENA automates evaluation; most LLM judges become more reliable with execution results, and GPT-5 leads code generation among the evaluated frontier models.Claude-Sonnet-4 and Claude-Opus-4 are tied in second place.Source: §1 · p. 3
  • The paper releases two preference-focused benchmarks and reports experiments on their utility and recent LLMs.The benchmarks assess alignment with human judgments and automate output evaluation via LLM-as-a-Judge.Source: §1 · p. 3

2 BIGCODEARENA

BIGCODEARENA addresses the difficulty of judging code from static text by enabling execution and user-driven testing. Its platform combines head-to-head model comparisons with interactive outputs and sampling procedures intended to balance exposure and reduce speed-related bias.

2.1 MOTIVATION

  • Static code comparisons can obscure whether generated code works or fulfills the prompt, so BIGCODEARENA extends human evaluation with code execution and user-driven testing.The paper describes BIGCODEARENA as the first fully open-source human evaluation platform for LLM-generated code via execution.Source: §2.1 · p. 4, §2 · p. 4

2.2 SYSTEM DESIGN

  • BIGCODEARENA presents two anonymized model responses with extracted-code execution results, which users can test, explore, and edit to judge behavior and robustness.Results may appear as interactive applications or web pages, or as static outputs such as text and images.Source: §2.2 · p. 4
  • A lightweight frontend and secure modular backend support code editing, dependency configuration, sandboxed execution, and rendering of results.The platform also supports one-sided chat for testing model-specific features.Source: §2.2 · p. 4
  • Model pairs are sampled with equal weights for established models and temporarily higher weights for new entrants, while both completed outputs appear simultaneously to limit response-speed bias.This sampling strategy is intended to provide fair exposure and more stable preference signals over time.Source: §2.2 · p. 4

3 DEPLOYMENT

BIGCODEARENA gathered crowdsourced and volunteer evaluations across a broad set of coding models, languages, and execution environments. The deployment also categorized prompts into recurring programming topics through manual inspection.

3 DEPLOYMENT

  • The study recruited 15 volunteer experts, required multi-turn conversations, and collected overall preference votes plus optional subcategory judgments and rationales.Participants were not compensated but received free access to state-of-the-art models; logged data included inputs, sandbox environments, execution results, and interactions.Source: §3 · p. 5
  • Over five months, the deployment collected more than 14,123 conversations from over 500 unique IP addresses using 10 frontier LLMs selected in February 2025.Model-generation settings defaulted to temperature 0.7 and top-p 0.95, though users could adjust them.Source: §3 · p. 5
  • The platform supports 10 programming languages and 8 execution environments, balancing widely used languages with frameworks for interactive and UI-oriented applications.Python and interpreter-based workflows are emphasized, alongside web and game frameworks.Source: §3 · pp. 5–6
  • After automatic topic clustering failed to yield clear boundaries, four authors manually inspected a random half of the prompts and identified six recurring topics.These were Web Design, Game Development, Diagram Creation, Creative Coding, Scientific Computing, and Problem Solving.Source: §3 · p. 6

4 MODEL RANKING

The leaderboard uses preference judgments from conversations with executed code and compares models under progressively controlled runtime and language settings. The analysis reports both overall rankings and variation in win rates across languages and execution environments.

4 MODEL RANKING

  • The leaderboard filters to 4,731 conversations with at least two turns and code execution, with each evaluated model receiving at least 700 votes.Relative strengths are estimated with the Bradley-Terry model and ranked by median bootstrap ratings using 100 resamples for 95% confidence intervals.Source: §4 · p. 6
  • o3-mini and o1-mini form the top tier across evaluation settings, Claude-3.5-Sonnet follows closely under language matching, and Qwen2.5 models and Llama-3.3-70B lag behind.The settings are All Data, Environment Matched, and Language Matched; GPT-4o weakens slightly under language matching.Source: §4 · pp. 6–7, Figure 3 · p. 6

4.1 DETAILED ANALYSIS

  • o3-mini and o1-mini have dominant win rates in Python, Java, and C++, while Gemini-2.0-Pro has the highest win rate in Rust.Qwen2.5 variants perform inconsistently, with weaknesses in Rust and Go.Source: §4.1 · p. 7, Figure 4 · p. 7
  • o3-mini performs consistently strongly across React, Streamlit, Gradio, Core Web, and PyGame, while Qwen2.5 models struggle in interactive and visualization-oriented environments.Claude-3.5-Sonnet and Gemini-2.0-Flash are also stable, but have lower win rates in more complex environments such as Vue and Mermaid.Source: §4.1 · p. 7, Figure 4 · p. 7

5 AUTOMATIC LLM EVALUATION WITH BIGCODEARENA

The section introduces BIGCODEREWARD and AUTOCODEARENA as complementary tools for evaluating reward models and automating code-generation judgments. Results indicate that execution feedback generally improves reward-model accuracy, while recent proprietary models lead AUTOCODEARENA rankings.

5.1 BIGCODEREWARD: EVALUATING REWARD MODELING FOR PRACTICAL CODING

  • BIGCODEREWARD evaluates how closely code reward models align with human judgments, using preference votes from collected conversations.It targets practical coding scenarios that existing general-domain reward-model evaluations and code benchmarks may not capture comprehensively.Source: §5 · p. 7, §5.1 · p. 8
  • Most reward models improve in accuracy with execution outputs, but InternVL3-78B drops from 57.3% to 46.8%.Gains are largest in Diagram Creation and Game Development, and smaller in Problem Solving; proprietary models score highest, while Qwen2.5-VL-72B Instruct is competitive among open models.Source: Table 2 · p. 8, §5.1 · p. 8

5.2 AUTOCODEARENA: AUTOMATING THE JUDGEMENT OF CODE GENERATION

  • GPT-5 leads AUTOCODEARENA, with Claude-Opus-4 and Claude-Sonnet-4 also strong; leading open models narrow the gap with mid-tier proprietary models.GPT-4.1 is shown as the 50% win-rate baseline, and Claude-3.7-Sonnet is excluded from rankings because it serves as judge.Source: Figure 5 · p. 9, §5.2 · p. 9
  • AUTOCODEARENA uses strong LLMs to compare model outputs against a baseline, then applies Bradley–Terry scoring and bootstrapping for win-rate confidence intervals.Higher win rates generally correspond to greater human preference.Source: §5.2 · p. 9
  • AUTOCODEARENA selects 600 prompts proportionally from six programming topics to reflect the collected conversations, and executes code locally across languages and frameworks.The prompts are selected from 4.7K multi-turn conversations; most do not specify a programming language.Source: §5.2 · p. 9

6 RELATED WORK

Related work traces the development of code-capable LLMs and benchmarks, alongside human-preference platforms. The paper distinguishes its aim of dynamic, execution-based evaluation across programming scenarios from prior approaches.

Training LLMs on Code

  • Code-specific models such as CodeGen, StarCoder, and InCoder advanced code generation, while later unified text-and-code training improved instruction following and narrowed the gap with general-purpose models.The passage notes that earlier code-specialized models had limited natural-language understanding and struggled with complex instructions.Source: §6 · pp. 9–10

Benchmarking Code Generation Quality

  • HumanEval and MBPP test algorithmic code against test cases, while DS-1000, APIBench, and BigCodeBench emphasize library use; newer work also benchmarks multimodal outputs.Examples include webpage, plot, and SVG generation.Source: §6 · p. 10
  • The paper targets dynamic, interactive evaluation across programming scenarios by incorporating execution results, unlike benchmarks focused on text or narrower output types.Prior work includes benchmarks for algorithmic problems, library use, webpages, plots, and SVGs.Source: §6 · p. 10

Judging LLMs via Human Preference

  • Chatbot Arena uses pairwise human preferences for model rankings, but human scoring studies are unscalable and can vary with evaluation criteria.The passage also identifies code-focused platforms that have limited application scope or do not disclose details.Source: §6 · p. 10

7 CONCLUSION

The paper presents BIGCODEARENA as an open platform for collecting human preferences on generated code through execution and interactive testing. Its benchmarks examine reward-model alignment and automated judgments; the authors report GPT-5 leading overall and Claude models performing strongly.

Conclusion

  • BIGCODEARENA integrates real-time execution and interactive testing to support human judgments of code correctness, functionality, and intent alignment.The authors report that execution-based evaluation reveals issues overlooked by static comparisons.Source: §7 · p. 10
  • The paper releases BIGCODEREWARD for measuring alignment with human preferences and AUTOCODEARENA for automating judgments with LLM-as-a-Judge.The reported results identify GPT-5 as the overall leader, with Claude models also performing strongly.Source: §7 · p. 10

Future Work

  • BIGCODEARENA currently supports only a limited set of languages and frameworks, and the authors identify expanding that diversity as future work.They also propose live benchmarks with refreshed prompts and judge agents that interact with web applications beyond initial screenshots.Source: §7 · p. 10
  • Code reward models remain an open challenge because current systems fall short of human-level perception and reasoning.The authors state that improved reward models could support more capable and aligned code LLMs.Source: §7 · p. 11

ETHICS AND REPRODUCIBILITY STATEMENT

The statement describes the study’s intended benefits and its privacy, security, and sustainability considerations. It notes protective measures while acknowledging risks and limits that remain.

ETHICS AND REPRODUCIBILITY STATEMENT

  • The platform sends user inputs to model API providers, and removing personally identifiable information during dataset preparation cannot guarantee complete elimination of sensitive content.Generated code runs in a controlled, one-time remote sandbox, but the authors cannot fully rule out malicious or harmful code generation.Source: ETHICS AND REPRODUCIBILITY STATEMENT · p. 11
  • BIGCODEARENA relies on inference endpoints rather than hosting LLMs locally, reducing its infrastructure’s direct energy footprint, though emissions are difficult to estimate precisely.The paper documents the specific inference endpoints in Appendix G.Source: ETHICS AND REPRODUCIBILITY STATEMENT · p. 11
  • The authors frame the released platform and benchmarks as tools for transparent, human-centered evaluation, while emphasizing continued vigilance around cybersecurity and privacy.They also call for continued investigation of trade-offs among evaluation scale, environmental sustainability, and accessibility.Source: ETHICS AND REPRODUCIBILITY STATEMENT · p. 11

B DATACARD

This section documents BigCodeArena’s dataset rationale and limitations and describes the platform’s design for practical human judgments of coding quality. It covers the comparison interface, execution pipeline and sandbox infrastructure, model-sampling strategy, code-generation constraints, and system-prompt output requirements.

B DATACARD

  • The datacard reports that 15 BigCode community volunteers communicate in mainstream English (en-US), although English is not their first language and prompts may contain inaccurate expressions.The authors do not report annotator nationalities because they know little about the public annotators.Source: B DATACARD · p. 20

C.1.1 FOR WHAT PURPOSE WAS THE DATASET CREATED?

  • BIGCODEARENA targets practical coding tasks and highlights human understanding of coding matters and execution feedback as key to judging code quality.The dataset is intended to help stakeholders understand LLM abilities and limitations relevant to deployment.Source: §C.1.1 · p. 22

C DATA SHEET

  • The dataset is managed by the BigCode community, has been released under the BigCode OpenRAIL-M license, and may be updated with added tasks or corrections alongside results.The data sheet says it will not be distributed to third parties on behalf of the creating entity.Source: §C.3.1 · p. 22, §C.3.3 · p. 22, §C.3.4 · p. 22

D.2 SYSTEM DESIGN

  • The interface enables anonymized pairwise comparison of code outputs, optional execution and in-place edits, and votes, ties, or abstentions.It includes a shared prompt panel, dual response panels, execution controls, and conversation history for multi-turn interactions.Source: Figure 6 · p. 23
  • The platform supports 10 programming languages and 8 execution environments, with sandboxed code runs and structured logs, errors, and runtime results.Its pipeline identifies runtimes from code blocks, manages dependencies, and isolates executions with time and memory limits.Source: §D.2 · pp. 23–24, Figure 7 · p. 24

D.3 MODEL SAMPLING STRATEGY

  • BIGCODEARENA synchronizes display until both models have generated and executed their responses, while balancing pair sampling and temporarily upweighting newly added models.This is intended to reduce latency-based preference bias and help late entrants collect comparable votes.Source: §D.3 · p. 24

D.5 SYSTEM PROMPT DESIGN

  • The system prompt directs models to produce complete, self-contained, executable code for supported frameworks and environments, generally in a single file.It also provides framework-specific constraints, including direct browser execution for web code and supported interactive libraries for Python tasks.Source: §D.5 · pp. 25–27
  • The system prompt asks for example test cases, stdout or stderr output, and code formatted in a language-labeled Markdown block.The prompt fragment also says code should not be long-running.Source: §D.5 · p. 27

D.6 SANDBOX INFRASTRUCTURE FOR BIGCODEARENA

  • E2B-managed, preconfigured sandbox instances support execution across languages and frameworks, with fresh instances intended to make results reproducible and comparisons fair.A centralized manager handles instance creation, lifecycle, and resources; web-framework projects use dedicated configurations, while the code interpreter supports C, C++, Java, Go, and Rust.Source: §D.6 · p. 28
  • The infrastructure pre-installs 101 top PyPI packages and uses uv or npm for dependencies, while monitoring background applications and serving web apps through an E2B nginx proxy.Python packages use --system installation; npm uses conflict-tolerant flags including --legacy-peer-deps and --prefer-offline.Source: §D.6 · p. 28

D.7 CASE STUDIES ON UI INTERACTIONS

The UI case study follows users from validating an event form’s basic functionality to evaluating its visual appeal and usability. The broader data analysis examines conversation characteristics and user activities.

D.7 CASE STUDIES ON UI INTERACTIONS

  • The first interaction tests whether event fields accept inputs and the basic event-creation flow works, prioritizing functionality over aesthetics.The user submits a simple test event.Source: §D.7 · p. 29
  • The second interaction shifts to visual appeal and usability, with the user requesting richer styling and checking that the event appears in the upcoming-events list.The paper describes this as a transition from functional validation to user-experience evaluation.Source: §D.7 · p. 29

E DATA ANALYSIS

  • The data analysis examines the 14K collected conversations from the perspectives of conversation characteristics and user activities.Source: E DATA ANALYSIS · p. 29

E.1 CONVERSATION CHARACTERISTICS ANALYSIS

Conversation patterns are predominantly short, Python- and interpreter-centered, while logged UI use is usually brief. Across evaluation settings and programming topics, leading models remain strong, though performance varies by context.

E.1 CONVERSATION CHARACTERISTICS ANALYSIS

  • Most conversations cluster between 3,000 and 20,000 characters, with a long-tailed, log-normal-like character-count distribution.Source: Figure 10 · p. 30, §E.1 · p. 30
  • 76.1% of conversations contain exactly two turns, and 87.2% conclude within two to three turns; the mean is 4.12 messages.The paper reports that most users tend to request add-on features in later turns.Source: §E.1 · p. 30, Figure 9 · p. 30
  • Python dominates language use with more than 4,000 conversations, while Interpreter sessions are most prevalent and React is the most common framework.JavaScript/TypeScript accounts for 3,359 conversations; React accounts for 2,729.Source: §E.1 · p. 31, Figure 11 · p. 31
  • Automatic topic modeling did not yield clear boundaries, so four authors manually inspected a random 50% sample of prompts to identify recurring programming topics.Source: §E.1 · p. 31

E.2 UNDERSTANDING USER INTERACTION WITH EXECUTION OUTCOMES

  • The interaction framework captures rendered UI state and records timestamped screen-resize, mouse, scroll, and keyboard actions using relative coordinates.These logs support sequential trajectories and high-fidelity replay.Source: §E.2 · pp. 31–32, Table 3 · p. 32
  • 72.0% of 5,557 recorded UI sessions last 30 seconds or less, while Core Web sessions average 59.6 seconds and have 18.8 keyboard events per session.Vue and Gradio sessions are shorter on average, at 29.9 and 26.1 seconds respectively.Source: §E.2 · p. 32, Table 4 · p. 32

F.1 OVERALL ANALYSIS

  • o3-mini and o1-mini lead across all three evaluation settings, while rankings show modest context sensitivity for some models and Qwen2.5 and Llama-3.3-70B generally underperform.The settings are All Data, Environment Matched, and Language Matched; Claude-3.5-Sonnet narrows the gap in the language-matched setting.Source: §F.1 · pp. 32–33

F.2 ANALYSIS OF PROGRAMMING TOPICS

  • Top models including o3-mini, o1-mini, and o1 outperform others across topics, while other models show uneven strengths and aggregate Elo can obscure topic-specific differences.Web-related tasks account for 36.1% of prompts, followed by problem solving at 22.8%.Source: §F.2 · p. 33, Figure 12 · p. 33

F.3 COMPARISONS TO PREVIOUS EVALUATIONS

BIGCODEARENA’s rankings align most with conversational arena evaluations, and expert relabeling shows substantial agreement with crowdsourced preferences. The paper also lists reproducibility artifacts.

F.3 COMPARISONS TO PREVIOUS EVALUATIONS

  • BIGCODEARENA aligns most with Chatbot Arena; its strongest reported correlation is with Chatbot Arena-Coding (ρ = 0.68), while BigCodeBench has weaker alignment (ρ = 0.43).Core Web rankings correlate with WebDev Arena at ρ = 0.68, compared with ρ = 0.50 for BIGCODEARENA overall.Source: §F.3 · pp. 33–34, Figure 13 · p. 34

F.4 VALIDATION OF VOTE QUALITY

  • Two expert raters agreed with original preferences at 80.4% and 86.0%, with corresponding Kappa coefficients of 0.61 and 0.72.The experts’ inter-annotator agreement was 83.2% (Kappa 0.67); disagreements often involved differing interpretations of interactive execution feedback.Source: §F.4 · p. 34

G ARTIFACTS

H.1 EXPERIMENT DETAILS

This appendix section details experimental materials and procedures, including reward-model judging prompts, output handling, evaluation metrics, and a macro-F1 table. It also provides coding-domain exemplars and AutoCodeArena prompt specifications and local execution-sandbox design.

H.1 EXPERIMENT DETAILS

  • When a reward model returns invalid JSON, GPT-4.1-mini reconstructs it into the required winner-and-reasoning JSON format.The expected winner is A, B, or TIE, and reasoning is a single paragraph; the prompt also requests valid escaping and JSON only.Source: §H.1 · p. 36

H.2 JUDGEMENT PROMPT (WITH OUTPUT)

  • With outputs, the judge compares implementations, execution results, and screenshots, prioritizing accurate implementation of all requested features.It also considers efficiency, explanations, readability, maintainability, correctness, and UI/UX, and returns a winner and reasoning in JSON.Source: §H.2 · pp. 36–37

H.3 JUDGEMENT PROMPT (WITHOUT OUTPUT)

  • Without outputs, the judge sees only the code and assesses code quality, structure, theoretical correctness, and how completely the request is addressed.The prompt retains accurate feature implementation as its primary criterion and uses the same A/B/Tie winner choices.Source: §H.2 · p. 36, §H.3 · pp. 37–38

H.4 METRIC

  • Evaluation uses accuracy and macro F1 over A, B, and Tie labels, with accuracy measuring exact agreement and macro F1 averaging class-level F1 scores.The labels indicate whether A or B is preferred, or whether the candidates are equally good.Source: §H.4 · p. 38

H.5 EXPERIMENT RESULTS

  • Table 6 reports reward-model macro F1 by task category with and without execution outputs, but the supplied passage does not show the scores.The caption defines “–” as without execution outputs and “+” as with them; it does not support identifying a winner or reporting values.Source: Table 6 · p. 39

H.6.1 WEB DESIGN; H.6.2 GAME DEVELOPMENT

  • The web-design and game-development examples show competing fitness-studio websites and React light-sequencer implementations, including a typed version with saved-pattern handling.The website snippets include class-schedule and booking features; the game snippets show light-grid controls and pattern functions.Source: §H.6.1 · p. 40, §H.6.2 · p. 41

H.6.3 CREATIVE CODING; H.6.4 DIAGRAM CREATION

  • The creative-coding and diagram examples contrast frosted-glass blur simulators with investment-decision flowcharts.The snippets show an image-upload blur interface and a flowchart branching on risk, expected return, duration, and market conditions.Source: §H.6.3 · p. 42, §H.6.4 · p. 43

H.6.5 SCIENTIFIC COMPUTING

  • The scientific-computing examples implement diffusion-limited aggregation simulations with adjustable grid size, particle count, and stickiness.One snippet wraps the simulation in a Gradio interface that displays the generated cluster.Source: §H.6.5 · p. 44
  • The scientific-computing example implements a diffusion-limited aggregation simulator and exposes grid size and particle count through Gradio sliders.The interface displays the resulting cluster visualization.Source: §H.6.5 · p. 44

H.6.6 PROBLEM SOLVING

  • The problem-solving examples use A* search to find grid paths and provide Gradio interfaces for selecting scenarios or entering grids and endpoints.The snippets include predefined scenarios and an example grid with start and end coordinates.Source: §H.6.6 · p. 45

I.1 CLASSIFICATION PROMPT

  • The classification prompt directs a model to assign each programming instruction to one dominant category and return the result in JSON.It specifies six categories, including scientific computing and algorithmic programming.Source: §I.1 · p. 46

I.2 GENERATION PROMPT

  • The generation prompt is unchanged from the prompt shown in Section D.5.Source: §I.2 · p. 46

I.3 JUDGEMENT SYSTEM PROMPT

  • The judging prompt tells evaluators to compare execution results and screenshots first, while discounting environment-caused failures and setup complexity.When execution cannot establish quality, judges should inspect the code; dependencies should not affect preference.Source: §I.3 · p. 47

I.4 CUSTOMIZED SANDBOX

  • AutoCodeArena uses a local Docker-backed sandbox with hardened ephemeral containers and deterministic capture of files, plots, and web-app screenshots.The pipeline replaces E2B remote sandboxing with locally managed execution and artifact discovery.Source: §I.4 · p. 48

I.5 MORE RESULTS

The appendix extends AutoCodeArena performance figures across web, game, creative-coding, diagram, scientific-computing, and problem-solving tasks, then presents query-based model-comparison case studies.

I.5 MORE RESULTS

  • AutoCodeArena figures report model performance for web design and game development.Both use GPT-4.1 as the baseline and Claude-3.7-Sonnet as judge, excluding the judge from rankings to avoid potential self-judgment bias.Source: Figure 20 · p. 49, Figure 21 · p. 49

I.5.3 CREATIVE CODING

  • Creative coding, diagram creation, and scientific computing each have AutoCodeArena performance figures.Each figure uses GPT-4.1 as baseline and Claude-3.7-Sonnet as judge, excluding the judge from rankings to avoid potential self-judgment bias.Source: Figure 22 · p. 50, Figure 23 · p. 50, Figure 24 · p. 50

I.5.6 PROBLEM SOLVING

  • AutoCodeArena also reports model performance for problem-solving tasks.The figure uses GPT-4.1 as baseline and Claude-3.7-Sonnet as judge, excluding the judge from rankings to avoid potential self-judgment bias.Source: Figure 25 · p. 51

I.6 CASE STUDIES

  • The cited requests describe an interactive human anatomy diagram with hover information, system filtering, and search, alongside a fluid-flow simulation visualizing particles in liquid or gas using JS.Source: §I.6 · p. 52
  • A further model-comparison table covers queries 9&10.The case-study tables use query pairs 1&2, 3&4, 5&6, 7&8, and 9&10.Source: Table 7 · p. 52, Table 8 · p. 53, Table 9 · p. 54
Loading 2510.08697v2…