Source-linked AI summary
Do Coding Agents Understand Why Systems Crash---and Why They Don't?
Chenxi Huang, Haonan Wang, Jinghe Zhang, Aleksandr Nogikh, Franjo Ivančić, Petros Maniatis, Eugene Wu, Kostis Kaffes, Junfeng Yang, Baishakhi Ray
TL;DR
Evaluating whether coding agents understand execution in complex systems codebases is difficult, and a correct crash prediction alone may reflect recognition of suspicious code rather than reasoning. This paper pairs source trees with opposite crash outcomes and evaluates four agents on SYSRB. Agents frequently miss crashes, struggle to discriminate outcomes even after finding relevant code, and can appear correct under single-tree evaluation.
Problem
How well coding agents reason about execution in complex, reliability-critical system codebases remains unclear, and a correct crash prediction alone does not establish understanding of its cause.
Source: §1 · p. 2Method
The paper compares agent predictions on execution-validated source-tree pairs that differ by one code change and have opposite crash outcomes for the same input.
Source: §3.1 · p. 4, §3.4 · p. 5Results
Agents miss 27.5–76.8% of crashing trees and predict crashes for only 2.8–10.9% of non-crashing trees; finding relevant code does not ensure discrimination.
Source: §1 · p. 2Takeaways & Limitations
Paired evaluation reduces bias from prediction tendencies shared across crashing and non-crashing trees, helping measure whether agents distinguish execution outcomes.
Source: §5 · p. 8The evaluation restricts agents to local source navigation and limited builtin software, unlike practitioners who may use debuggers, git history, and online documents.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Coding agents are increasingly capable of fixing real-world bugs, yet they still struggle with large systems codebases such as the Linux kernel. To understand why, we test how agents reason about execution in these codebases through crash reasoning: predicting from source code whether a given input crashes the system. A correct prediction alone, however, does not establish this ability, since it may come from merely spotting suspicious code. In this paper, we propose source interventions: we pair source trees that differ by a single commit, so that the same input crashes only one of them, and test whether agents predict crashes more often on the crashing tree. Experiments with four coding agents on SysRB, our dataset of 934 such pairs from Linux and OSS-Fuzz projects, yield three main findings. First, agents are biased toward predicting no crash: they rarely raise false alarms but miss many real crashes, and even the best agent misses over a quarter of them. Second, locating the code that determines the crash outcome is not enough: agents often misjudge its effect, especially when the crash occurs far from that code. Third, testing only on crashing code is misleading: agents that merely notice suspicious code predict crashes on both source trees, which would look like correct reasoning on the crashing tree alone. Together, these findings indicate that current coding agents cannot yet reliably reason about execution in complex systems codebases, and that paired evaluation is necessary to measure this ability.
1 Introduction
Reliable software engineering requires agents to understand how inputs and code changes affect execution, but their execution reasoning in complex systems codebases remains unclear. The paper evaluates crash reasoning with paired source interventions and reports that agents often miss crashes, misjudge code changes, and can appear correct by noticing suspicious code alone.
1 Introduction
- Execution reasoning remains unclear in complex, reliability-critical systems, and successful crash repair alone cannot establish that an agent understands the crash mechanism.A manual study found that 52% of 79 kAgent patches altered or removed intended functionality.Source: §1 · p. 1
- Crash reasoning tests whether an agent can predict from source code, build configuration, and input whether a program crashes, and where and why.This provides an execution-reasoning evaluation without reconstructing every step of execution.Source: §1 · pp. 1–2
- Source interventions compare predictions on two source trees differing by one code change, with the same input verified to crash on only one tree.The paired design measures outcome discrimination and examines whether agents find and analyze both the intervention and crash sites.Source: §1 · p. 2
- Agents miss 27.5–76.8% of crashing trees, while predicting crashes for only 2.8–10.9% of non-crashing trees.Across four agents, finding the intervention site does not ensure discrimination; Claude Opus 5's discriminability falls from about 65% to 38% when the crash is in a different subsystem.Source: §1 · p. 2
- SYSRB evaluates four agents on 934 execution-validated pairs from Linux and OSS-Fuzz, addressing a need for system-level reasoning evaluation.The contributions include paired evaluation, trajectory-level diagnostics, and findings about missed crashes, outcome discrimination, and cross-subsystem failures.Source: §1 · p. 2
- Single-tree evaluation can reward predictions based on suspicious code patterns shared across crashing and non-crashing trees.Such predictions may appear correct when evaluated only on the crashing tree, even though they do not change after the crash is resolved.Source: §1 · p. 2
2 Related Work
Prior crash-resolution benchmarks assess whether agents can suppress crashes, but suppression does not itself demonstrate understanding of the crash mechanism. Existing code-reasoning benchmarks use isolated functions or recorded execution states that do not readily transfer to system codebases.
2 Related Work
- Crash suppression does not by itself demonstrate understanding, while function-level and repository-level benchmarks rely on isolated functions or recorded states that do not readily transfer to system codebases.This paper evaluates crash reasoning directly through source interventions.Source: §2 · p. 3
3 Analyzing Agentic Crash Reasoning
This section defines crash reasoning through source-tree pairs with opposite outcomes for the same reproducer, then evaluates agents’ outcome discrimination, reasoning traces, and performance on SYSRB.
3.1 Crash Reasoning with Source Intervention
- Source intervention pairs trees that differ by one commit while holding the reproducer and execution environment fixed, retaining pairs with opposite crash outcomes.Agents predict whether the program crashes from source code without receiving the crash report, compiling the source, or running the reproducer.Source: §3.1 · pp. 3–4
3.2 Policy-Level Evaluation
- Crash-outcome discriminability averages each instance’s difference in crash-prediction probability between its crashing and non-crashing trees, equivalent to true-positive rate minus false-positive rate.A score of 1 represents perfect discrimination; equal crash-prediction likelihood on both trees contributes zero in expectation.Source: §3.2 · p. 4
3.3 Trajectory-Level Evaluation
- Trajectory evaluation checks whether agents observe and reason about the intervention and crash sites, and separately scores predicted crash types and stack traces against reference reports.Site profiles distinguish whether relevant code was shown from whether the agent analyzed its execution or semantics.Source: §3.3 · pp. 4–5, Table 1 · p. 5
3.4 Dataset Curation
- SYSRB contains 934 execution-validated pairs: 545 Linux kernel crashes and 389 OSS-Fuzz crashes, each pairing crashing and non-crashing trees around a single commit.Linux pairs use fix or cause-bisected commits; OSS-Fuzz pairs use fix commits, and both datasets retain stable outcomes on repeated execution.Source: §3.4 · p. 5
4 Evaluation
The evaluation tests four coding agents on SYSRB using discriminability, crash-site prediction, and trajectory-level analyses. Agents often miss crashes, and paired source-tree comparisons reveal limits that single-tree predictions can hide.
4.1 Experiment Setup
- The evaluation samples eight trajectories per instance and source tree for four agents, measures policy-level discriminability with Youden’s J, and checks trajectory annotations against manual labels.Manual review of 160 annotations found 93.1% agreement with the labels.Source: §4.1 · p. 6
4.2 Overall Crash-Reasoning Performance
- Discriminability varies by agent and crash type; all four agents discriminate better on OSS-Fuzz than on Linux, while Opus on Linux reaches 39.5% for hung-task and 40.5% for deadlock reports.Source: §4.2 · p. 6
- False-negative rates range from 27.5% to 76.8%, while false-positive rates range from 2.8% to 10.9%; even the best agent misses more than a quarter of crashes.Opus has a 72.5% true-positive rate and a 4.8% false-positive rate, compared with Gemini’s 49.0% and 2.8%.Source: §4.2 · p. 6
4.3 Discriminability and the Intervention Site
- Finding the intervention site does not ensure discrimination: Opus observes it in 89.0% of trajectories and has 75.4% conditional discriminability, while GLM and Gemini observe it more often than they discriminate.Measured discriminability without observing the intervention site is near zero; the passage attributes the remainder to possible side channels and sampling noise.Source: §4.3 · p. 7
- For Opus, Linux discriminability falls from about 65% when intervention and crash sites are in the same function or file to 38% when they are in different subsystems.In the far group, Opus observes both sites in 25.2% of trajectories and only the intervention site in 44.7%.Source: §4.3 · p. 7, Figure 5 · p. 8, Figure 6 · p. 8
4.4 Source Interventions Reveal Bias
- Without observing the intervention site, agents usually predict no crash, and suspicious crash-site cues can raise crash verdicts on both trees, yielding little paired discriminability.For Opus, crash-verdict rates for the suspicious-site profile are 42.6% on crashing trees and 39.2% on non-crashing trees, versus 6.0% and 5.1% for the comparison profile; shared tendencies cancel in paired J.Source: §4.4 · p. 8
5 Conclusion
The evaluation finds that agents miss many real crashes, often misjudge crash outcomes even after locating the intervention site, and perform worse when crashes occur far from that site. Paired evaluation reduces the misleading bias of single-source-tree tests.
5 Conclusion
- Agents miss 27.5–76.8% of real crashes, and even Claude Opus 5 falls 24.6 pp. short of perfect discriminability after observing the intervention site.Finding the site is not enough to ensure correct outcome discrimination.Source: §5 · p. 8
- Discriminability drops sharply when the crash occurs far from the intervention site.Source: §5 · p. 8
- Single-source-tree evaluations can reward misleading crash predictions, while paired evaluation reduces this bias.Source: §5 · p. 8
Limitations
The evaluation is limited by its restricted debugging environment, cost, and intervention sources; public commits may also contaminate agent verdicts.
Limitations
- Agents use only local source navigation and limited builtin software, unlike practitioners who may use debuggers, git history, and online documents.Source: Limitations · p. 9
- Soundly estimating crash-verdict frequency requires usually at least eight runs per source tree, and long trajectories make the evaluation budget-constrained.Source: Limitations · p. 9
- Interventions come from commits in Linux and open-source system libraries, and public commits may have appeared in training data and affect verdicts.Source: Limitations · p. 9
Ethical Considerations
SYSRB uses publicly available crashes that were fixed and disclosed, and the study does not involve human subjects or personal data.
Ethical Considerations
- SYSRB crashes come from public syzbot reports and OSS-Fuzz crashes collected after they were fixed and disclosed.Source: Ethical Considerations · p. 9
- The benchmark contains no previously undisclosed vulnerabilities, and the task does not involve exploit development or attacks on deployed systems.Source: Ethical Considerations · p. 9
- The study uses open-source projects under their original licenses and involves no human subjects or personal data.Source: Ethical Considerations · p. 9
A.1 Model Serving and Sampling Parameters
All agents use the same harness, prompt, source-tree images, and limits. The coding CLIs do not send sampling parameters, so decoding uses checkpoint or provider defaults.
A.1 Model Serving and Sampling Parameters
- All agents share a harness, prompt, source-tree images, and limits of k = 8 samples per instance and source tree, two hours, and no network.Source: §A.1 · p. 10
- Because coding CLIs send no sampling parameters, each model decodes with its checkpoint’s or provider’s defaults.Source: §A.1 · p. 10
- Claude Opus 5 trajectories rejected by the cybersecurity classifier are treated as invalid and excluded from statistics and accounting.Source: §A.1 · p. 10
B Crash Report
Crash type and subsystem labels are assigned by fixed rules rather than a model. Predicted crash frames are compared with reference frames using normalized labels and path-based subsystem assignments.
B Crash Report
- Reference crash types come from an ordered regular-expression table, while agent labels are normalized to the same closed vocabulary and compared for equality.Labels outside the vocabulary are mismatches; classes not offered to the agent map to other.Source: B Crash Report · pp. 10–11
- Subsystem labels are fixed by source path: Linux uses leading directory components, while OSS-Fuzz uses the top-level source directory.For example, mm/huge_memory.c maps to mm.Source: B Crash Report · p. 11
- The same-subsystem stack score compares predicted and reference crash-frame subsystems after removing entry, sanitizer, and reporting frames.Compiler suffixes are stripped before comparison.Source: B Crash Report · p. 11
C J Decomposition
The decomposition separates crash-verdict discrimination into runs that observe the intervention site and runs that do not. It also accounts for unequal gate rates across crashing and non-crashing trees.
C J Decomposition
- The exact identity decomposes overall TPR and FPR into gated and blind rates, weighted separately by intervention-site observation rates on crashing and non-crashing trees.Each run is counted once, and runs with an undetermined gate are excluded from the decomposition population.Source: C J Decomposition · p. 11
- Conditional discriminabilities are J1 = TPR1 − FPR1 for gated runs and ϵ = TPR0 − FPR0 for blind runs.The gated and blind contributions to J′ are not generally multiples of J1 and ϵ because the two trees have separate gate weights.Source: C J Decomposition · pp. 11–12
- Table 9 reports crash-verdict rates conditioned on the four site-profile patterns FFTT and FFFF.The table covers rates conditioned on those two combinations of intervention-site and crash-site indicators.Source: Table 9 · p. 12
- The correction κ is omitted by Table 4 and vanishes when gate rates are equal across trees or gated and blind runs have equal overall crash-verdict rates.Its magnitude is bounded by |κ| ≤ |δ|.Source: C J Decomposition · p. 12