PaperlayerView exact revision on arXiv

Source-linked AI summary

Agents in the Sandbox: End-to-End Crash Bug Reproduction for Minecraft

Eray Yapağcı, Yavuz Alp Sencer Öztürk, Eray Tüzün

arXiv:2503.20036v2cs.SEcs.AI

TL;DR

BugCraft targets the largely unaddressed problem of reproducing Minecraft crash bugs from imperfect user reports. It synthesizes structured steps with LLMs and Wiki knowledge, then executes them through a vision-based agent and custom macro API; on BugCraft-Bench, it reproduced 34.9% of crashes with GPT-4.1 and outperformed computer-use baselines.

  • Problem

    Crash-bug reproduction in evolving games such as Minecraft is manual, time-consuming, and difficult, while complex game environments remain under-addressed by automated bug-reproduction research.

    Source: §I · p. 1

  • Method

    BugCraft uses a Step Synthesizer with LLMs, Minecraft Wiki retrieval, and structured steps, followed by a vision-based Action Model with a custom macro API.

    Source: §I · p. 2, 2) Step Synthesizer · p. 4, 3) Action Model · p. 5

  • Results

    34.9% success rate with GPT-4.1 exceeded OpenAI CUA’s 25.5% and UI-TARS-1.5-7B’s 0%, improving from 30.2% with GPT-4o.

    Source: D. Comparative Evaluation with Baselines · p. 9

  • Takeaways & Limitations

    BugCraft-Bench provides 86 curated Minecraft crash reports for standardized evaluation of automated reproduction.

    Source: §I · p. 2

    Current limitations include Action Model loops and poor decisions, plus Step Synthesizer errors in command accuracy and detail completeness.

    Source: §VII · pp. 11–12

Abstract

from arXiv · show

Reproducing game bugs, particularly crash bugs in continuously evolving games like Minecraft, is a notoriously manual, time-consuming, and challenging process to automate; insights from a key decision maker from Minecraft we interviewed confirm this, highlighting that a substantial portion of crash reports necessitate manual scenario reconstruction. Despite the success of LLM-driven bug reproduction in other software domains, games, with their complex interactive environments, remain largely unaddressed. This paper introduces BugCraft, a novel end-to-end framework designed to automate the reproduction of crash bugs in Minecraft directly from user-submitted bug reports, addressing the critical gap in automated game bug reproduction. BugCraft employs a two-stage approach: first, a Step Synthesizer leverages LLMs and Minecraft Wiki knowledge to transform bug reports into high-quality, structured steps to reproduce (S2R). Second, an Action Model, powered by a vision-based LLM agent and a custom macro API, executes these S2R steps within Minecraft to trigger the reported crash. To facilitate evaluation, we introduce BugCraft-Bench, a curated dataset of Minecraft crash bug reports. On BugCraft-Bench, our framework end-to-end reproduced 34.9% of crash bugs with GPT-4.1, outperforming baseline computer-use models by 37%. BugCraft demonstrates the feasibility of automated reproduction of crash bugs in complex game environments using LLMs, opening promising avenues for game testing and development. Finally, we make our code open at https://bugcraft2025.github.io

I. INTRODUCTION

BugCraft addresses the lack of end-to-end automated crash reproduction for complex games by converting unstructured Minecraft bug reports into executable reproduction plans and actions.

I. INTRODUCTION

  • Crash bugs are severe, recurrent, and difficult to fix, while Minecraft teams still manually reconstruct scenarios for many crash reports.Low-quality reports often omit steps to reproduce and expected outcomes, prolonging bug lifecycles.Source: §I · p. 1
  • Existing automated bug-reproduction work has focused on other software domains, leaving open-ended game environments such as Minecraft largely unaddressed.Prior GUI-based approaches are limited in their ability to handle complex game interaction.Source: §I · pp. 1–2
  • BugCraft is presented as the first end-to-end framework for reproducing crash bugs in a complex 3D game, spanning unstructured reports, scripts, and reproduction logs.The framework is designed to transform raw reports into reliable crash-reproduction artifacts.Source: §I · p. 2
  • The Step Synthesizer combines user comments and domain knowledge through retrieval, game-state analysis, and Step Clusters to produce flexible reproduction plans.The contribution targets ambiguous reports by structuring actionable steps for later execution.Source: §I · p. 2
  • BugCraft-Bench contains 86 manually confirmed and reproducible crash reports, and BugCraft achieves a 34.9% success rate while outperforming existing approaches.The benchmark and implementation are released to support reproducible evaluation.Source: §I · p. 2

A. Bug Reproduction in Games

The paper situates BugCraft within manual game testing, LLM-based bug reproduction, and open-ended game agents, then introduces a two-phase framework and custom Minecraft dataset.

A. Bug Reproduction in Games

  • Game testing remains manual and time-consuming, and crash bugs can recur after updates because testing time and programmable game-state coverage are limited.These constraints make reproducing known failures distinct from discovering new bugs.Source: A. Bug Reproduction in Games · p. 2

B. Automated Bug Reproduction Using LLMs

C. LLM Agents

  • LLM agents use tools and in-context learning to perceive, decide, and act in open-ended environments, including Minecraft.This prior capability motivates applying agentic interaction to game bug reproduction.Source: C. LLM Agents · p. 2

III. METHODOLOGY

  • BugCraft operates in two phases: creating steps to reproduce and executing them through an agentic framework for complex interactive environments.The methodology covers dataset construction, framework components, implementation, and evaluation.Source: §III · pp. 2–3

A. Data Curation

  • Because no suitable Minecraft bug-report dataset was available, the authors constructed one and added Minecraft Wiki information to enrich reproduction.The dataset and knowledge base support quantitative evaluation and in-game reasoning.Source: A. Data Curation · p. 3

1) BugCraft-Bench Dataset Curation:

BugCraft-Bench is curated from Minecraft crash reports, reconstructed at challenging early states, and supplemented with Wiki knowledge for retrieval and reproduction planning.

1) BugCraft-Bench Dataset Curation:

  • The final BugCraft-Bench contains 86 Minecraft Java Edition crash reports collected from Mojang’s official tracker with associated comments, attachments, and links.Reports were filtered for single-player crashes and manually checked for reproducibility.Source: 1) BugCraft-Bench Dataset Curation · p. 3
  • The dataset prioritizes developer-verified reports labeled Confirmed or Community Consensus as reliable ground-truth instances.These labels reflect verification by Mojang Studios.Source: 1) BugCraft-Bench Dataset Curation · p. 3
  • Curation excluded multiplayer, cross-version, externally dependent, non-game, and substantially revised reports, then removed irreproducible cases from a sampled pool.The selection covered recent reports across 70 game versions before producing the final benchmark.Source: 1) BugCraft-Bench Dataset Curation · p. 3

2) Minecraft Wiki Dataset:

  • The Minecraft Wiki serves as a retrieval knowledge base, with non-redirect pages stored separately so the framework does not rely solely on LLM pretraining.Relevant pages are selected through entity extraction, fuzzy title matching, and Step LLM ranking.Source: 2) Minecraft Wiki Dataset · p. 3

B. BugCraft Framework

  • BugCraft transforms raw reports into executable scripts through a Step Synthesizer and an Action Model that generate steps and perform game actions.The framework combines planning with execution rather than treating report interpretation as a standalone task.Source: B. BugCraft Framework · pp. 3–4

1) Preprocessing:

  • Preprocessing reconstructs the report before later Mojira or trusted-user edits, retaining the challenging initial wording and removing later comments.A back-tracker parses the changelog to recover this earlier state.Source: 1) Preprocessing · p. 4

2) Step Synthesizer:

BugCraft separates reproduction into synthesizing structured S2R plans and executing them through a vision-based Action Model in Minecraft.

2) Step Synthesizer:

  • The Step Synthesizer transforms ambiguous bug reports into annotated S2R plans through knowledge augmentation, generation, refinement, and clustering.It retrieves relevant Minecraft Wiki information, generates initial steps, critiques inconsistencies, and organizes the result into readable clusters.Source: 2) Step Synthesizer · pp. 4–5
  • Knowledge augmentation uses entity matching and LLM-selected Minecraft Wiki pages to supply reproduction-relevant external information.The system analyzes retrieved pages individually and appends relevant reasoning trajectories to later generation calls.Source: 2) Step Synthesizer · p. 4
  • Command-based substitutions, such as /setblock, reduce difficult spatial-navigation requirements and produce clearer reproduction instructions.The synthesizer replaces challenging physical actions with Minecraft commands when appropriate.Source: 2) Step Synthesizer · p. 4
  • Step clustering organizes finalized instructions into typically 2–4 titled groups and reorders misplaced steps for downstream execution.For MC-276621, clustering incorporated the initial report and a later comment while adding commands that simplified replication.Source: 2) Step Synthesizer · p. 5

3) Action Model:

  • The Action Model converts refined S2R into executable scripts using a custom macro API that supports GUI, menu, and in-game interactions.The API includes keyboard and mouse input, text entry, Minecraft commands, and coordinate-based clicking, while click-place bridges OmniParser bounding boxes to clicks.Source: 3) Action Model · p. 5
  • The Action Model repeatedly observes screenshots, generates and verifies actions, evaluates feedback, and advances clusters until a crash or iteration limit occurs.OmniParser supplies structured UI annotations, while redundant completion checks and action verification guard against ineffective actions and loops; logged actions can reproduce non-random bugs.Source: 3) Action Model · p. 6

4) Implementation Details:

The implementation uses bounded LLM context and adapted baseline deployments to run BugCraft and its comparisons.

4) Implementation Details:

  • BugCraft retains the last 25 Action LLM iterations and stops after 30 iterations, representing commands verbally to improve interpretability.The framework is implemented in Python with LangChain for LLM calls.Source: 4) Implementation Details · p. 6
  • UI-TARS-1.5-7B and OpenAI-CUA are adapted from OSWorld implementations and run with 30 iterations on the initial bug reports.UI-TARS-1.5-7B is hosted with VLLM on RunPod, while OpenAI-CUA follows OpenAI’s official usage example.Source: 4) Implementation Details · pp. 6–7

C. Evaluation Setup

The evaluation asks whether BugCraft synthesizes correct plans, reproduces crashes end to end, and outperforms general GUI and computer-use agents.

C. Evaluation Setup

  • BugCraft is evaluated through three questions covering Step Synthesizer accuracy, end-to-end crash reproduction, and comparison with existing agents.The questions address both component effectiveness and comparative performance in real-world usage.Source: C. Evaluation Setup · p. 7
  • BugCraft-Bench supports the first two questions, while UI-TARS-1.5-7B and OpenAI’s Computer Use Agent provide comparison baselines for the third.The same dataset is used for the comparative evaluation.Source: C. Evaluation Setup · p. 7

1) Evaluation of the Step Synthesizer:

The evaluation manually assesses synthesized step clusters and measures end-to-end Action Model reproduction on reports judged reproducible.

1) Evaluation of the Step Synthesizer:

  • Step clusters are manually labeled True, Faulty, or Irreproducible by attempting reproduction and consulting reports or external resources when needed.Faulty plans contain missing, incorrect, or inconsistent instructions, whereas irreproducibility reflects external causes such as outdated versions or hardware dependence.Source: 1) Evaluation of the Step Synthesizer · p. 7
  • Inter-rater reliability was assessed with percentage agreement and Cohen’s Kappa after jointly calibrating categories and independently labeling 100 reports.Faulty clusters were additionally categorized by wrong commands, missing steps, or logic errors.Source: 1) Evaluation of the Step Synthesizer · p. 7

2) Evaluation of the Action Model:

4) Failure Analysis and Inter-Rater Agreement:

BugCraft’s failure analysis distinguishes errors arising from the Step Synthesizer, agent decisions, framework constraints, and internal model failures. The evaluation used controlled GPT-4o experiments and structured failure categories to analyze end-to-end reproduction.

4) Failure Analysis and Inter-Rater Agreement:

  • Failure analysis was based on recurring unsuccessful-attempt patterns and collaborative rubric refinement, including edge cases and common agent mistakes.The resulting categories were intended to provide a structured account of why reproduction attempts failed.Source: 4) Failure Analysis and Inter-Rater… · p. 7

A. On Step Synthesizer Success

  • Failure analysis separates successful reproduction, agent failure, framework incapability, and internal action-model errors.Agent failures include loops and poor decisions, while framework failures involve constraints such as tight timing or inaccessible menu elements.Source: A. On Step Synthesizer Success · pp. 7–8

B. On Step Synthesizer Failure

  • Faulty Step Synthesizer plans can still lead to recovery, whereas simultaneous failures produce total failure or an action-model error.The recovery category captures cases where the Action Model reproduces the bug despite a faulty plan.Source: B. On Step Synthesizer Failure · p. 8

IV. RESULTS

  • The results section evaluates BugCraft end-to-end and benchmarks baseline models using a GPT-4o-based experimental setup with deterministic temperature-0 inference.The setup used GPT-4o for both components and also tested GPT-4.1 with otherwise unchanged hyperparameters.Source: §IV · p. 8, A. Experimental Setup · p. 8

B. Step Synthesizer Analysis

The Step Synthesizer produced mostly reproducible plans, while the Action Model achieved substantial end-to-end reproduction and could sometimes recover from faulty plans. Labeling reliability was assessed through inter-rater agreement.

B. Step Synthesizer Analysis

  • 66.28% of Step Synthesizer clusters were True, while 33.72% were Faulty; faulty plans most often involved Wrong Command, Missing Step, or Logic Error.The reported faulty-plan causes were 48.28%, 34.49%, and 31.03%, respectively.Source: B. Step Synthesizer Analysis · p. 8, Table I · p. 8
  • 14 reports were classified as Irreproducible because of hardware issues or errors in the original reports rather than synthesizer faults.Source: B. Step Synthesizer Analysis · p. 8

1) Inter-Rater Agreement for Step Synthesizer:

C. Action Model Analysis

  • 30.2% of GPT-4o attempts reproduced bugs end-to-end, including 38.60% with correct step clusters and 13.79% recovery with faulty clusters.The Action Model reproduced 26 of 86 reports overall, 22 of 57 with successful synthesis, and 4 of 29 with faulty synthesis.Source: C. Action Model Analysis · pp. 8–9, Table II · p. 9
  • With correct plans, failures were primarily attributed to Agent Incapability, while 82.76% of failures after faulty plans were attributed to incorrect steps.Agent Incapability included Poor Decision Making and Stuck in Loop; Framework Incapability accounted for 7.02% of failures with correct plans.Source: C. Action Model Analysis · p. 9, Table II · p. 9

D. Comparative Evaluation with Baselines

BugCraft was evaluated against competitive baselines and analyzed across planning and end-to-end reproduction, revealing gains from GPT-4.1 but persistent planning, agent, and framework limitations.

D. Comparative Evaluation with Baselines

A. Addressing RQ1

  • 66.28% of GPT-4o-generated step clusters were accurate, while faulty plans most often involved wrong commands, missing steps, or logic errors.The reported error shares were 48.28% for Wrong Command, 34.49% for Missing Step, and 31.03% for Logic Error.Source: A. Addressing RQ1 · p. 9
  • The Step Synthesizer’s reliability is constrained by evolving Minecraft versions, outdated command knowledge, changing item-tag schemas, and hallucinated commands.The discussion recommends version-specific information, command verification, logical consistency checks, and dynamic game-environment feedback, while noting the difficulty of supporting many game versions.Source: A. Addressing RQ1 · pp. 9–10

B. Addressing RQ2

  • 30.2% of crash-inducing reports were reproduced end-to-end with GPT-4o, but the Action Model failed in a majority of cases even with accurate plans.Agent incapability included Poor Decision Making at 29.82% and Stuck in Loops at 19.30%, while framework incapability contributed 7.02% of failures when correct steps were available.Source: B. Addressing RQ2 · p. 10
  • Improving agent game understanding, interaction control, feedback, error detection, and recovery is identified as essential for stronger reproduction performance.Suggested directions include symbolic or hybrid approaches and a more robust game interaction API.Source: B. Addressing RQ2 · p. 10

C. Addressing RQ3

The analysis attributes BugCraft’s advantage to game-specific planning, knowledge retrieval, flexible execution, and unified interaction APIs, while emphasizing efficiency and a concrete baseline failure case.

1) Isolation of Framework Contributions:

  • 36.9% success, a 32% relative improvement over CUA, is attributed to game-specific RAG, flexible step clustering, and unified game interaction APIs.These components address domain knowledge and spatial-reasoning challenges that generic approaches miss.Source: 1) Isolation of Framework Contributions · p. 10

2) Performance Justification and Efficiency Analysis:

  • BugCraft completed reproductions in 10 minutes versus a human 3.41-day turnaround and reduced cost per successful reproduction to $3.44 versus $35.25.The analysis also highlights parallel processing across instances for high-throughput bug triage.Source: 2) Performance Justification and… · p. 10

3) Technical Differentiation from Prior Work:

  • BugCraft combines intelligent bug analysis, game-knowledge RAG, and a unified macro API for both 2D menus and 3D gameplay.These innovations target spatial reasoning, complex state dependencies, and version-specific mechanics beyond 2D GUI automation.Source: 3) Technical Differentiation from… · p. 10
  • In the MC-161902 case, OpenAI CUA obtained a water bucket but failed to aim and place water on a valid surface.The case centers on correctly placing water to trigger the crash.Source: 3) Technical Differentiation from… · p. 10, Figure 8 · p. 11
  • BugCraft bypassed the difficult physical placement by spawning the water block directly through a command, after which its Action Model reproduced the bug.This case illustrates how Step Synthesizer planning can avoid visually difficult actions.Source: 3) Technical Differentiation from… · p. 11

4) Qualitative Analysis of the Models:

BugCraft’s model analysis reports complementary agent behavior and then qualifies the findings through construct, internal, and external validity constraints, while identifying concrete directions for improvement.

4) Qualitative Analysis of the Models:

  • 34.9% of bugs were reproduced with GPT-4.1 versus 30.2% with GPT-4o, while complementary successes yielded 41.9% oracle coverage across 86 reports.GPT-4.1 reproduced 30 of 86 bugs, GPT-4o reproduced 26, and 20 successes overlapped.Source: 4) Qualitative Analysis of the Models · p. 11

A. Construct Validity

  • The Minecraft Wiki and Minecraft’s popularity may inflate effectiveness relative to games with less documentation or training data, although external knowledge can replace the Wiki.The evaluation used initial, unedited Mojira reports to reduce concern that models had seen corrected versions during training.Source: A. Construct Validity · p. 11
  • Using the smaller UI-Tars-1.5-7B model and unavailable internal prompts may have produced suboptimal baseline results.The authors followed the official implementation but note that small models are sensitive to prompt changes.Source: A. Construct Validity · p. 11

B. Internal Validity

  • Temperature zero improved consistency, but limited repeated runs prevented fully quantifying environmental and agent stochasticity, and few manual evaluators may introduce bias.Source: B. Internal Validity · p. 11

C. External Validity

  • The evaluation covers only Minecraft: Java Edition crash bugs, so direct generalization to other bug types or games remains uncertain.The dataset spans more than 70 Minecraft versions and over six years, but public Mojira reports may not represent all player issues.Source: C. External Validity · p. 11

VII. CONCLUSION

  • The conclusion identifies action-model loops and poor decisions, plus Step Synthesizer inaccuracies and incomplete detail, especially for newer Minecraft versions.Source: §VII · p. 11
  • Future work targets stronger game understanding, developer-supported APIs, verification for non-crash bugs, broader generalization studies, and support for multiplayer and file-loading bugs.The roadmap prioritizes improvements to both the Action Model and Step Synthesizer.Source: §VII · p. 12
Loading 2503.20036v2…