PaperlayerView exact revision on arXiv

Source-linked AI summary

ChatDBG: Augmenting Debugging with Large Language Models

Kyla H. Levin, Nicolas van Kempen, Emery D. Berger, Stephen N. Freund

arXiv:2403.16354v5cs.SEcs.AIcs.LGcs.PL

TL;DR

Programmers must reason through complex code and execution state to diagnose bugs, a challenging and time-consuming task. ChatDBG extends standard debuggers with conversational LLM assistance that can inspect and control program execution. Its evaluation reports diagnosis and fixes across Python and C/C++ examples, while the Python suite’s results may not represent experienced programmers’ code.

  • Problem

    Debugging remains difficult and time-consuming because programmers must reason about behavior, test hypotheses, and inspect potentially large amounts of code and execution information.

    Source: §1 · p. 2

  • Method

    ChatDBG integrates LLM dialogue with standard debuggers, letting the model inspect runtime state and issue debugger commands before returning control to the programmer.

    Source: §1 · p. 2

  • Results

    One or two queries diagnose and fix defects in 85% of student Python labs and notebooks, and ChatDBG also identifies causes and fixes for real-world C/C++ bugs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The evaluation indicates that debugging dialogue with ChatDBG can significantly assist in identifying root causes and developing correct fixes.

    Source: §7 · p. 20

    The Python evaluation uses unpublished student labs that may not represent experienced programmers’ code, while the C/C++ suite is mostly memory errors and may be affected by training-data leakage.

    Source: §5.3 · p. 19

Abstract

from arXiv · show

Debugging is a critical but challenging task for programmers. This paper proposes ChatDBG, an AI-powered debugging assistant. ChatDBG integrates large language models (LLMs) to significantly enhance the capabilities and user-friendliness of conventional debuggers. ChatDBG lets programmers engage in a collaborative dialogue with the debugger, allowing them to pose complex questions about program state, perform root cause analysis for crashes or assertion failures, and explore open-ended queries like "why is x null?". To handle these queries, ChatDBG grants the LLM autonomy to "take the wheel": it can act as an independent agent capable of querying and controlling the debugger to navigate through stacks and inspect program state. It then reports its findings and yields back control to the programmer. By leveraging the real-world knowledge embedded in LLMs, ChatDBG can diagnose issues identifiable only through the use of domain-specific reasoning. Our ChatDBG prototype integrates with standard debuggers including LLDB and GDB for native code and Pdb for Python. Our evaluation across a diverse set of code, including C/C++ code with known bugs and a suite of Python code including standalone scripts and Jupyter notebooks, demonstrates that ChatDBG can successfully analyze root causes, explain bugs, and generate accurate fixes for a wide range of real-world errors. For the Python programs, a single query led to an actionable bug fix 67% of the time; one additional follow-up query increased the success rate to 85%. ChatDBG has seen rapid uptake; it has already been downloaded more than 75,000 times.

1 Introduction

Debugging remains challenging because programmers must form and test hypotheses while navigating code, execution, and large amounts of state. ChatDBG adds conversational, agentic LLM assistance to standard debuggers, and its evaluation reports successful diagnosis and fixes across Python and C/C++ examples.

1 Introduction

  • Programmers still face difficult, time-consuming reasoning: they must test hypotheses and sift through code, execution histories, data, and stack frames.These demands can span lengthy executions, large program states, and multiple threads.Source: §1 · pp. 1–2
  • ChatDBG draws on LLM real-world knowledge to reason about issues requiring logical and domain-specific insight unavailable from the program alone.The paper illustrates this with statistical reasoning in a debugging example.Source: §1 · p. 2
  • ChatDBG lets an LLM take control of the debugger to inspect state, run commands, and answer programmers’ high-level debugging questions.After responding, the LLM yields control back to the programmer, who can continue with commands or further dialogue.Source: §1 · p. 2
  • One or two queries diagnose and fix defects in 85% of student Python labs and notebooks, typically for under $0.20.The prototype integrates with GDB, LLDB, and Pdb, and also identifies causes and proposes fixes for real-world C/C++ bugs.Source: §1 · p. 2
  • The authors present ChatDBG as an assistant for novices and experienced programmers, adding conversational reasoning to debugging sessions.They describe the prototype and evaluate its advantages over existing debugger functionality.Source: §1 · p. 2

2 Overview

The overview follows a marble-resampling program through an assertion failure and a debugging dialogue. ChatDBG identifies a faulty statistic, then uses statistical knowledge to explain a subsequent failure caused by too few trials.

2 Overview

  • The example resamples a marble-color array and expects the mean proportion of blue marbles to approach the original sample’s 0.7 when enough trials are run.The user begins debugging by inspecting num_trials and asking why stats has five elements; ChatDBG can issue debugger commands while constructing its answer.Source: §2 · p. 3
  • ChatDBG identifies and corrects the faulty proportion_blue statistic; after that fix, the assertion still fails because five trials produce high variance.The LLM identifies trial count as the issue without seeing information about trial size or variance in the program state, code, or documentation encountered during the chat.Source: §2 · pp. 3, 5, Figure 3 · p. 5

3 Related Work

Prior work established core interactive-debugging features and explored trace-based questions, slicing, fault localization, and automated repair. ChatDBG distinguishes itself through LLM access to runtime state and debugger control, while concurrent systems use different context and workflow assumptions.

3 Related Work

  • Earlier debuggers introduced breakpoints, single-stepping, stack navigation, conditional breakpoints, and watchpoints over several decades.Source: §3 · p. 6
  • Whyline answers selected queries using trace-based timelines, whereas ChatDBG accepts arbitrary queries through dialogue; trace use limits Whyline to relatively short-lived executions.Source: §3 · p. 6
  • Program slicing produces code relevant to an error, and ChatDBG uses backward slicing to gather code across Jupyter cells.Source: §3 · p. 6
  • ChatDBG gives an LLM runtime state and control of the debugger, complementing fault-localization approaches that rely on static features or source code alone.The paper also describes combining ChatDBG with techniques such as notebook slicing and AddressSanitizer for native-code memory errors.Source: §3 · pp. 6–7
  • ChatDBG requests best-effort code fixes but leaves code changes to the programmer, unlike repair systems that generate patches automatically.Source: §3 · p. 7

3.1 Concurrent Work

  • ChatDBG’s diagnosis rate is nearly twice that of the Enriched Stack configuration, and targeted questions and follow-ups generally improve its effectiveness.Source: §3.1 · p. 7
  • AutoCodeRover and SWE-agent rely on issue descriptions and source code, while ChatDBG additionally uses runtime information; their SWE-bench evaluation is not applicable to ChatDBG.Source: §3.1 · p. 7
  • RepairAgent and AutoSD require a failing test case and precise bug location, whereas ChatDBG supports developer collaboration without those inputs.Source: §3.1 · p. 7

4 Implementation

ChatDBG integrates with existing debuggers and routes natural-language debugging questions through an LLM that can issue debugger commands. It builds prompts from the current debugging context, including enriched stack traces, and returns control to the programmer after the response.

4.1 Using ChatDBG: Preliminaries

  • ChatDBG extends Pdb and LLDB, with a subset of features ported to GDB and WinDBG; installation requires an OpenAI API key.Python setup uses the chatdbg package and can add an IPython extension; LLDB’s plug-in is installed with a shell command.Source: §4.1 · pp. 7–8

4.2 Debugging a Target Program

  • Python targets need no special preparation, while C/C++ targets require an unstripped executable built with -g to provide DWARF debugging information.Rust targets may require a ChatDBG dependency and an annotated main function so error messages reach ChatDBG through a log file.Source: §4.2 · p. 8

4.3 ChatDBG Architecture Overview

  • ChatDBG sends standard commands to the debugger and natural-language queries to the LLM, which can request debugger commands before returning prose and control to the user.The initial prompt combines instructions, stack and error information, program inputs, command history, and the user’s query; later prompts include only new history and text.Source: §4.3 · p. 8

4.4 Initial Prompts and Enriched Stack Traces

  • Initial prompts include debugging instructions, stack and error context, available inputs, command history, and the user’s query.For assertion failures, the LLM is told to assume the assertion is valid; responses end with a fix or suggestions for further debugging.Source: §4.4 · pp. 9, 11
  • Enriched stack traces show per-frame variable types and values with at least 10 source lines, omit library frames, and abbreviate large or deeply nested values.In Python, value conversion is bounded to balance useful context against prompt size; the LLM can request elided details, while LLDB decodes stack values using DWARF types and dereferences pointers.Source: §4.4 · pp. 9, 11
  • The take-the-wheel instructions let the LLM inspect variables, navigate frames, retrieve code or documentation, and continue investigating until it identifies a root cause or recommends next steps.The prompt permits repeated function calls and requires a final Recommendation with either a fix or one to three debugging suggestions.Source: §4.4 · p. 10

4.5 Taking the Wheel

ChatDBG lets the LLM use function calls to interact with the underlying debugger while constructing an answer. This approach exposes session-specific runtime state that is difficult to supply through retrieval alone.

4.5 Taking the Wheel

  • ChatDBG uses function calls to let the LLM query the debugger, execute code, or obtain source while composing a response.For example, a debug(command) call runs a debugger command and returns its output; commands and outputs are shown to the user.Source: §4.5 · p. 11
  • LLMs can use existing debugger knowledge for common operations, but ChatDBG’s function calls expose runtime state that is distinct to each debugging session and not easily vectorized for RAG.The paper characterizes this access to runtime state through agentic reasoning as a key technical innovation.Source: §4.5 · p. 11
  • Table 2 presents debugger command extensions that help the LLM navigate and understand the target’s code.The extensions are made available to the LLM through its function-call API.Source: Table 2 · p. 12

4.6 Navigating the Code

Because the LLM may lack knowledge of user-defined code and third-party libraries, ChatDBG adds debugger commands for retrieving source and documentation. These commands are exposed through function calls.

4.6 Navigating the Code

  • ChatDBG adds code-navigation commands because the LLM may not know user-defined or third-party functions, and including all relevant source in the initial prompt would not scale.The extensions include Python’s info command for documentation and user-defined source, and LLDB’s code and definition commands for source around locations or symbol definitions.Source: §4.6 · p. 12

4.7 Slices for Interactive Python

ChatDBG uses backward slices to help debug notebook code spread across separately evaluated cells. It also sanitizes LLM-generated commands and leaves proposed fixes for the user to assess.

4.7 Slices for Interactive Python

  • Backward slicing collects code from earlier notebook cells that could contribute to a variable used in the current cell.The slice command returns the relevant cell source, with ipyflow computing the slices.Source: §4.7 · pp. 12–13

4.8 Security and Risks

  • Python commands are restricted to a user-configurable function whitelist, while native-code commands that call functions are rejected.The paper cites harder code-provenance tracking and more permissive languages as reasons for the stricter native-code rule.Source: §4.8 · p. 13
  • Because LLM diagnoses and fixes may be misleading, ChatDBG presents suggestions to users rather than applying them directly to target code.Users can vet the responses and decide whether to follow the suggested changes.Source: §4.8 · p. 13

5 Evaluation

The evaluation examines ChatDBG's bug diagnosis and repair in student-written Python programs and large C/C++ programs. Its Python experiments compare progressively enabled debugger features across scripts and notebooks with varied real-world errors.

5 Evaluation

  • The evaluation tests bug diagnosis and repair in student-written Python programs and large C/C++ programs, addressing three research questions about effectiveness and feature contributions.The Python cases support systematic assessment through well-defined expected behavior; the C/C++ cases examine unmanaged code and unusual crash-triggering corner cases.Source: §5 · p. 13

5.1 Python

  • The Python collection contains eight command-line scripts and fourteen Jupyter notebooks drawn from two introductory courses.The programs include common human-introduced mistakes, semantic errors and crashes, and are unpublished with objective correctness criteria.Source: §5.1 · pp. 13–14, Table 3 · p. 14
  • The Python ablation compares five configurations, progressively adding richer stacks, debugger control, targeted questions, and a follow-up dialog.Each program ran ten times; targeted questions were designed to be neutral and failure-specific, and the dialog used a consistent follow-up prompt.Source: §5.1 · pp. 14–15, Table 4 · p. 15
  • The final +Dialog configuration adds a second query to +Targeted Question, while the experiments used gpt-4-1106-preview.A targeted-question trial averaged about 10,000 tokens, $0.12 USD, and 25 seconds; extended-dialog steps incurred comparable costs.Source: §5.1 · p. 15

RQ1: Is ChatDBG effective at diagnosing and fixing bugs in Python?

ChatDBG's Python success rate rose as questions became more specific and the dialog gained a follow-up. Success meant providing both an accurate explanation and an actionable fix.

RQ1: Is ChatDBG effective at diagnosing and fixing bugs in Python?

  • The Default Stack baseline approximates pasting a stack trace and basic error information into an LLM chat and requesting a fix.It provides standard stack traces and does not let the LLM take control of the debugger.Source: RQ1: Is ChatDBG effective at… · p. 15
  • With all features enabled, ChatDBG succeeded in well over half of trials, and Figure 7 shows success rates by program and configuration.The figure's vertical lines indicate the mean.Source: RQ1: Is ChatDBG effective at… · p. 16, Figure 7 · p. 16
  • 85% of trials succeeded with an additional dialog step, compared with 67% using targeted questions and 57% with the simple question “why?”.A success required an accurate error explanation and an actionable fix, assessed against criteria set before examining responses.Source: RQ1: Is ChatDBG effective at… · pp. 15–16

RQ2: Which components of ChatDBG contribute to its effectiveness?

ChatDBG’s debugging features contribute differently across crashes and semantic bugs, while follow-up dialogue and user context improve effectiveness. In C/C++ trials it often diagnosed crashes, though root-cause fixes and the study’s evaluation scope have limits.

RQ2: Which components of ChatDBG contribute to its effectiveness?

  • Enriched stacks help when crashes expose diagnostic information, but +Take the Wheel improves success on both crashing and semantic bugs; notebook slicing is critical.Without the slice command, success rates rarely improved when the LLM took the wheel.Source: RQ2: Which components of ChatDBG… · p. 16
  • Modest expected-behavior details and a content-free follow-up dialogue improve success, suggesting response-length limits may constrain single-step reasoning.The authors expect context and continued dialogue to matter more for complex programs.Source: RQ2: Which components of ChatDBG… · p. 17
  • ChatDBG’s responses also draw on background knowledge, including Python idioms, the h-index, and statistical techniques.Source: RQ2: Which components of ChatDBG… · p. 17
  • Failures arose when the LLM missed program behavior; longer prompts could degrade results, and greater distance between a bug’s root cause and its observable effect made diagnosis harder.Some suggestions were not actionable or could introduce bugs; subsequent user feedback could mitigate these problems.Source: RQ2: Which components of ChatDBG… · p. 17

5.2 C and C++

  • Suggested C/C++ fixes ranged from buffer-safe replacements that silently truncate input to appropriate recursion limits or input validation.The authors note that truncation needs validation or other measures for a robust fix.Source: §5.2 · p. 18
  • 36% of suggestions fixed the C/C++ root cause, while an additional 55% fixed the proximate cause.ChatDBG diagnosed and explained the crash in virtually all trials.Source: §5.2 · p. 18, Figure 8 · p. 19

5.3 Threats to Validity

  • Validity is bounded by student-lab representativeness, mostly memory-error C/C++ cases and possible training leakage, prompt sensitivity and stochasticity, and manual fix judgments.Each test program was run at least ten times; the authors used criteria decided in advance to reduce subjectivity in manual evaluation.Source: §5.3 · p. 19

6 Future Work

The authors propose combining ChatDBG with techniques that focus debugging on suspicious code or failure-inducing inputs, and extending it to explore program state over time. Such integrations may require training the LLM to use the added techniques.

6 Future Work

  • Fault localization could direct the LLM toward suspicious files, functions, or source lines, potentially improving effectiveness and efficiency.Source: §6 · p. 19
  • Delta debugging could limit the input given to the LLM and provide failure-inducing events as guidance.Source: §6 · p. 19
  • Time-travel debugging could let ChatDBG answer queries unavailable from current state alone, but the LLM may need fine-tuning or additional training to use such techniques.Source: §6 · pp. 19–20

7 Conclusion

The paper presents ChatDBG as an AI-based debugging assistant and reports that debugging dialogue can significantly assist root-cause identification and the development of correct fixes.

7 Conclusion

  • The paper presents ChatDBG as the first AI-based debugging assistant.Source: §7 · p. 20
  • The evaluation shows that debugging dialogue with ChatDBG can significantly assist in identifying error root causes.Source: §7 · p. 20
  • The evaluation also shows that debugging dialogue can significantly assist in developing correct fixes.Source: §7 · p. 20
Loading 2403.16354v5…