PaperlayerView exact revision on arXiv

Source-linked AI summary

Integrating Crash Report Mining and LLMs for Bug Localization and Repair: An Industrial Report

Marcos Medeiros, Uirá Kulesza, Christoph Treude, Daniel Lucena, Rafael Gomes, Roberta Coelho, Eiji Adachi, Rodrigo Bonifacio

arXiv:2609.04483v1cs.SE

TL;DR

Crash-report bugs are costly to analyze at industrial scale, and evidence about LLM support for this task remains limited. This study combines crash mining with LLM-based localization and repair, finding that its best configuration localized 71% and repaired 52% of 38 crash bugs.

  • Problem

    Crash reports accumulate at scale, making manual localization costly, while evidence on LLM support for crash-report bugs in industrial settings remains limited and structured context is needed under code-access constraints.

    Source: §I · p. 1

  • Method

    The study integrates stack-trace clustering and suspicious file and method ranking with LLM patch suggestions, evaluating the approach retrospectively against developer fixes.

    Source: §I · p. 1, §VII · p. 10

  • Results

    71% localization and 52% repair were achieved on 38 crash bugs using prompts composed of suspicious-file source code and stack-trace samples.

    Source: §VII · p. 10

  • Takeaways & Limitations

    The results indicate that structured crash-mining context can support LLM-based localization and repair in industrial maintenance workflows.

    Source: §VII · p. 10

    The study covers two Java enterprise systems from one company and uses retrospective historical reports and fixes, so results may differ in other domains or real-time debugging.

    Source: §V · p. 10

Abstract

from arXiv · show

Analyzing crash-report bugs in large-scale industrial software systems requires substantial maintenance effort, particularly in production environments where developers must handle large volumes of crash reports and source code artifacts to localize and fix their root causes. While recent studies have shown that Large Language Models (LLMs) can assist with maintenance tasks, little is known about their effectiveness in supporting developers in analyzing crash-report bugs and repairing bugs associated with groups of crash reports in industrial settings. To address this gap, we investigate whether integrating crash report mining techniques---specifically stack trace clustering and suspicious file and method ranking---with LLMs can support crash localization and repair in production environments. We conduct a retrospective evaluation of five LLMs under four prompt configurations. After that, we chose the best model to run on 38 crash bugs collected from two large Java enterprise systems. We further analyze the structural characteristics and explanatory patterns of LLM-generated responses and assess localization and repair effectiveness through manual validation. Our results show that the best-performing configuration localizes up to 71% and correctly repairs 52% of crash bugs on the full dataset. These findings provide empirical evidence that combining crash report mining with LLM-based repair can effectively support debugging activities in industrial maintenance workflows.

I. INTRODUCTION

The paper addresses the difficulty of localizing crash bugs as reports accumulate and examines how structured crash-mining context can guide LLMs. Its workflow groups stack traces, ranks suspicious artifacts, and supplies prioritized context for LLM-based localization and repair.

I. INTRODUCTION

  • Crash-report analysis becomes costly as report volumes grow, while industrial LLM use is constrained by the cost, confidentiality, and context-window limits of unrestricted code access.The paper identifies a need for structured, prioritized context that narrows the search space while retaining relevance to the failure.Source: §I · p. 1

II. BUG LOCALIZATION AND FIXING APPROACH

  • The study integrates crash mining with LLMs because industrial evidence remains scarce and prior work had not systematically evaluated this combination with ranked suspicious files and methods.The workflow combines crash report clustering, suspicious file and method ranking, and LLM patch suggestions; clustering and ranking procedures draw on prior literature,.Source: §I · p. 1, §II · p. 2

I. INTRODUCTION

  • The introduction reports that LLMs often suggest plausible fixes, but structural heterogeneity prevents automated patch integration.The evaluation compares localization and repair against developer corrective commits used as ground truth.Source: §I · pp. 1–2
  • The study also evaluates how stack traces, stack messages, and ranking information affect LLM localization and repair.Its validation compares the ability to locate and fix crash root causes against developer corrective commits.Source: §I · pp. 1–2

A. Crash Report Grouping

  • Crash reports are grouped by stack-trace similarity, consolidating related failures to reduce root-cause analysis effort.The grouping scheme begins with identical traces and progressively includes equivalent signatures, structural containment, and traces whose top-frame files match.Source: A. Crash Report Grouping · p. 2
  • The final grouping level merges traces with different signaler methods when their qualified top-frame file is the same.The authors treat such crashes as manifestations of the same underlying fault, citing prior validation,.Source: A. Crash Report Grouping · p. 2

B. Suspicious file and method ranking

The ranking stage scores files in a crash group using proximity to the crash point, specificity to that group, and recurrence within it. It then ranks methods in top files by their frequency in the group’s stack traces.

B. Suspicious file and method ranking

  • Each file’s crash-group score combines inverse average distance to the crash point, inverse bucket frequency, and file frequency.The score is computed multiplicatively, and files are ranked in descending score order.Source: B. Suspicious file and method ranking · p. 2
  • The multiplicative score favors files close to the crash point, specific to the crash group, and recurrent within that failure context.Inverse bucket frequency penalizes files frequently associated with different faults, while file frequency rewards repeated appearances within the group.Source: B. Suspicious file and method ranking · p. 2
  • Methods within the top-ranked files are suggested according to how frequently they appear in the group’s stack traces.Source: B. Suspicious file and method ranking · p. 2

C. Bug fixing

The bug-fixing workflow gives an LLM ranked suspicious-file code and crash context to produce patch suggestions. The study evaluates response characteristics and tests how contextual information affects localization and repair.

C. Bug fixing

  • The workflow submits top-ranked file source code and a representative stack trace to an LLM, then attaches its patch suggestion to the corresponding crash report for developer triage and resolution.Source: C. Bug fixing · p. 2
  • Prompts use five components—role instructions, suspicious source files, crash information, ranking information, and a repair request—and follow a zero-shot strategy.The authors cite input-size and context-window limits, avoiding overfitting to handcrafted examples, and realistic industrial deployment as reasons for zero-shot prompting.Source: C. Bug fixing · p. 3

III. EMPIRICAL STUDY

  • The empirical study asks about the structural and explanatory characteristics of LLM responses, including code completeness, granularity, and explanatory behavior.Source: §III · p. 3
  • The ablation varies exception granularity and ranking information to assess their marginal contributions to localization and repair, first on a subset and then on the full dataset.The study also examines which clustering-derived information contributes to bug fixes generated by LLMs.Source: §III · p. 3
  • The empirical-study procedures cover the target systems and industrial context, selected LLMs and configurations, prompt combinations, and evaluation with human validation.Source: §III · p. 3

A. Target Systems

The evaluation took place in two Java Enterprise systems maintained by a public institution. The paper describes their scale and the criteria and models used for the LLM evaluation.

A. Target Systems

  • The two systems together receive over one million daily requests and comprise about 2.1 million lines of code across more than 12,000 classes.They have operated for over 16 years and are used and customized by more than 30 institutions.Source: A. Target Systems · p. 3
  • SIGAA generates about 9,200 crash reports weekly, while SIGRH generates about 290.Source: A. Target Systems · p. 3

B. Target LLMs

  • LLMs were selected for widespread adoption, REST API availability, cloud execution support, and context-window size; the listed models are Claude 3.5 Sonnet, Gemini 1.5 Pro, Mistral Large, and Llama 3 405B Instruct.The study fixed temperature at 1.0 and otherwise retained provider defaults unless a parameter was mandatory.Source: B. Target LLMs · pp. 3–4

C. Prompt combinations

The prompt configurations vary exception detail and whether ranked suspicious artifacts accompany the top-ranked files’ source code. The evaluation measures response-level correctness and bug-level consistency against developer fixes.

C. Prompt combinations

  • All prompts include code for the top three ranked suspicious files; the study varies exception detail and inclusion of the ranking information.The restricted context reflects practical limits on codebase sharing, costs, rate limits, and LLM context windows.Source: C. Prompt combinations · p. 4
  • The four configurations combine source code of suspicious files with full stack traces or stack messages, each with or without ranking information.Stack messages retain exception types and messages, including nested causes, but omit the full method call stack.Source: C. Prompt combinations · p. 4
  • The configurations support ablation analysis of the marginal contribution of ranking information and exception granularity to localization and repair.Source: C. Prompt combinations · p. 4

D. Evaluation metrics

  • Correctness was manually judged against developers’ corrective commits, accepting semantically valid fixes even when they differed syntactically.Limited test infrastructure, restricted source access, rebuilding costs, and the need to assess semantic equivalence made pass@k and BLEU unsuitable.Source: D. Evaluation metrics · p. 4
  • Localization@response measures correct identification of the developer-modified artifact, while Repair@response measures responses producing an equivalent or valid alternative patch.Both are proportions over all responses; their indicators are 1 for correct localization or repair and 0 otherwise.Source: D. Evaluation metrics · p. 4
  • Per-bug assessment evaluates consistency across multiple responses, counting a bug as localized or repaired when the majority of responses are correct.The per-bug metrics average binary success indicators across all bugs.Source: D. Evaluation metrics · pp. 4–5

E. Study Procedures

The study combines an 18-month production deployment of crash mining with a retrospective LLM evaluation using artifacts from real maintenance. LLM responses were compared with developer fixes through independent review.

Industrial phase

  • For 18 months, the team ran the crash mining and bug management process weekly in production, grouping reports from the previous seven days.Source: E. Study Procedures · p. 5
  • Development teams used group impact information to prioritize crashes, then received tasks containing the top five suspicious files and methods plus contextual information.Teams analyzed the group identifier, occurrence dates, report counts, affected URIs, impacted users, and stack-trace classes.Source: E. Study Procedures · p. 5
  • The deployment dataset captured buggy code, suspicious-file and method rankings, frequent stack traces, and patched code for each crash group.Source: E. Study Procedures · p. 5

Retrospective LLM evaluation

  • Retrospective prompts included at most three suspicious-file source codes and the most frequent stack trace, with four configurations applied consistently across LLMs and attempts.The context restriction reflected practical context-window and rate-limit constraints.Source: E. Study Procedures · p. 5
  • Each prompt received five zero-shot requests in fresh sessions; five LLMs were first assessed on a subset, then the best-performing model was run across the full dataset.Source: E. Study Procedures · p. 5
  • Two reviewers independently assessed localization and semantic repair against developer solutions, resolving disagreements through discussion or a third reviewer.Localization required the same code snippet, or for NullPointerException the same variable; repair could be equivalent to or a valid alternative to the developer’s fix.Source: E. Study Procedures · p. 5

IV. RESULTS AND DISCUSSION

The results section covers dataset construction, LLM response characteristics, and the information supplied by clustering and ranking. It also examines consistency of the best-performing LLM across the full dataset.

IV. RESULTS AND DISCUSSION

  • The results section reports dataset construction and analyzes structural and explanatory characteristics of LLM responses.Source: §IV · p. 5
  • It also reports information from clustering and ranking that assists the LLM in locating and fixing crash bugs.Source: §IV · p. 5
  • The section examines whether the best-performing LLM remains consistent when evaluated across the full dataset.Source: §IV · p. 6

A. Dataset Building

The dataset was built from crash groups handled by development teams over an 18-month deployment, with issues selected when code changes and corrective commits could be identified.

A. Dataset Building

  • Of 131 correction issues opened on crash groups, teams analyzed 86; 62 produced code changes and 50 had Java commits.The other 24 analyzed issues closed without crash reproduction.Source: A. Dataset Building · p. 6
  • The selected dataset contains 38 resolved issues: 10 from SIGAA and 28 from SIGRH, after sampling and exclusions for analysis or context-limit reasons.The resulting issues were all code-related, with no environmental factors involved.Source: A. Dataset Building · p. 6
  • For each issue, the dataset includes the top three suspicious files and top two frequent stack traces; the developer-modified file ranked in the top three in all 38 cases.It ranked first in 36 cases, second in one, and third in one.Source: A. Dataset Building · p. 6

B. Analysis of LLM Responses and Effectiveness

The section analyzes LLM response patterns and localization and repair effectiveness, comparing models on the eight-issue SMALLDS before evaluating Claude-3.5-sonnet on all 38 FULLDS bugs.

B. Analysis of LLM Responses and Effectiveness

  • At least two authors independently assessed each patch; agreement was substantial for localization (κ=0.767) and moderate for repair status (κ=0.500).Disagreements were resolved by consensus.Source: B. Analysis of LLM Responses and… · p. 6
  • Prompt violations, omitted documentation, response diversity, and LLM nondeterminism challenge automated patch extraction, so the authors manually analyzed suggestions.Manual assessment treated a repair as plausible when it changed the developer-modified method and would prevent the observed exception, even if the implementation differed.Source: B. Analysis of LLM Responses and… · pp. 6–7
  • A single fragment from one class was the most frequent response type, and response structure varied with both prompt combination and model.Stack-trace prompts most often produced Fragment Single Class responses; prompts with SCoSF and stack message more often produced Full Multiple Classes.Source: B. Analysis of LLM Responses and… · p. 7
  • 43% of responses suppressed documentation; explanations appeared outside code in 45.5% and within generated code in 37.73%.Claude suppressed documentation in 68.13% of responses, while Gemini suppressed none but had the highest truncation rate, 48.75%.Source: B. Analysis of LLM Responses and… · p. 7
  • 65.75% of SMALLDS responses referenced a developer-patched method; ST prompts reached 76%, rising to 81% with RoSFM.By model, Claude led at 78%, followed by Llama at 75% and GPT at 72%.Source: B. Analysis of LLM Responses and… · p. 7
  • On SMALLDS, Claude-3.5-sonnet reached 82.5% per-response localization with SCoSF+ST+RoSFM and 52.5% repair with SCoSF+SM+RoSFM.At least three of five responses localized 87.5% of bugs with the former prompt and repaired 50.0% with the latter.Source: B. Analysis of LLM Responses and… · p. 8, Table III · p. 8, Table IV · p. 9
  • 71.05% localization and 52.63% repair: SCoSF+ST was Claude-3.5-sonnet’s best FULLDS combination.At bug level, SCoSF+ST also had the highest correct-fix rate (51.58%) and lowest unsuccessful-fix rate (48.42%).Source: B. Analysis of LLM Responses and… · pp. 8–9, Table VI · p. 9
  • Adding RoSFM improved SCoSF+SM per-response localization from 44.21% to 64.21% and repair from 28.42% to 47.37%, but slightly reduced performance with ST.At bug level, SM localization and repair also rose from 44.74% to 63.16% and 28.95% to 47.37%, respectively.Source: B. Analysis of LLM Responses and… · p. 9

V. THREATS TO VALIDITY

The study’s validity is bounded by uncertainty in commit-derived ground truth, subjective manual assessment, stochastic LLM outputs, and a retrospective evaluation of two Java enterprise systems. The authors describe checks to mitigate some of these risks, but note that results may not generalize to other settings.

Internal Validity

  • Commits from issue and Git records may be incomplete, imprecise, partial, or include unrelated changes, complicating identification of the true ground-truth fix.The authors manually inspected commits and issue discussions when identifying ground-truth artifacts.Source: §V · p. 9

Construct Validity

  • Manual inspection rather than execution or test-based validation may introduce subjectivity when judging semantic equivalence between generated patches and developer fixes.Two reviewers assessed responses, with a third reviewer arbitrating when necessary.Source: §V · p. 9

Conclusion Validity

  • LLM randomness may still affect results despite five independent requests per prompt and a majority-response success criterion for each bug.Source: §V · p. 9

External Validity

  • Results may not generalize beyond two Java enterprise systems from one company or to real-time debugging, specialized models, or future model versions.Source: §V · p. 10

VI. RELATED WORK

Prior studies examine LLM-based crash repair, general bug fixing, and crash-report enrichment. This study differs in its industrial setting, contextual inputs, and evaluation criteria, with reported localization and repair outcomes varying across comparisons.

LLM-based crash repair

  • 71% localization and 52% repair in this study compare with 76.7% and 33.3% for Du et al.’s interactive IntDiagSolver approach [26],.Du et al. used small Stack Overflow snippets; this study used zero-shot prompts with richer context and larger real-world codebases.Source: §VI · p. 10

Automatic bug fixing

  • 52% repair accuracy on industrial Java crash bugs was comparable to Sobania et al.’s 47.5% on QuixBugs, despite this study’s stricter success criterion.This study required at least 3 correct fixes out of 5 responses and used a single interaction per response; Sobania et al. counted success if one of four responses succeeded.Source: §VI · p. 10

Crash-report enrichment

  • Claude-3.5-sonnet achieved 71% localization and 52% repair here, versus up to 58% and 41% for Fahim et al.’s agentic strategy [44].Fahim et al. enriched open-source issue reports, while this study used raw production crash reports without textual descriptions and supplied clustered, ranked context.Source: §VI · p. 10

VII. CONCLUSION

The study retrospectively evaluates crash localization and repair with LLMs on real-world bugs from large Java Enterprise systems, examining response characteristics and prompt/model performance. On the full dataset, the best-performing configuration achieves 71% localization and 52% repair.

VII. CONCLUSION

  • The retrospective evaluation examined real-world bugs in large-scale web-based systems implemented with Java Enterprise technologies.Source: §VII · p. 10
  • Across 800 responses for 8 crash bugs, a single fragment from one class was the most frequent response type, with response characteristics varying by prompt and LLM.Source: §VII · p. 10
  • Claude-3.5-sonnet localized better with suspicious-file code, stack-trace samples, and artifact rankings, while its bug-fixing performance was better with stack messages than traces.Source: §VII · p. 10
  • 71% localization and 52% repair were achieved on 38 crash bugs using prompts with suspicious-file source code and stack-trace samples.The authors report these full-dataset results for the best-performing LLM.Source: §VII · p. 10
Loading 2609.04483v1…