Source-linked AI summary
ExplainBench: Evaluating Code Explanations from Agents
Zhiyuan Pan, Sungmin Kang, Imam Nur Bani Yusuf, Abhik Roychoudhury
TL;DR
Agent-generated explanations lack a benchmark for trustworthy evaluation even as manual review becomes costly. ExplainBench evaluates explanations through LLM-answerable questions about intended behavior and patch effects, and finds explanation quality can differ from coding efficacy. An audit agent using additional testing improved explanations across evaluated agents.
Problem
The field lacks benchmarks for trustworthy agent explanations, although manual inspection of increasingly large agent changes is costly and explanations are used to understand implementations.
Method
ExplainBench asks an LLM multiple-choice questions about patch intent and effect, while ExplanationAuditAgent runs differential tests to audit and refine explanations.
Results
Explanation quality was independent from agent efficacy, and ExplanationAuditAgent improved explanations for all evaluated agents.
Takeaways & Limitations
Agent explanations require separate evaluation from patch efficacy, and automated auditing can make them more trustworthy.
Takeaways & Limitations
The benchmark is limited to open-source SWE-bench Verified projects and captures only selected explanation-quality dimensions, not readability or usefulness.
Abstract
from arXiv · showhide
Large Language Model (LLM) agents have seen rapid adoption in software engineering. As agents take a greater role in the actual generation of code, they are making larger changes, spanning tens to hundreds of lines. This makes manual review of agent results increasingly infeasible, leading developers to turn to explanations to understand enacted changes. Despite this, there are no benchmarks that evaluate the trustworthiness of agent-generated explanations. To bridge this gap, we propose ExplainBench, a benchmark to automatically evaluate explanations from coding agents. ExplainBench is based on the intuition that informative explanations should enable an LLM to correctly answer questions, allowing quantitative comparison of explanation quality between agents. With this observation, we construct a suite of questions that evaluates whether explanations accurately describe (1) the intended behavior of buggy code and (2) the effect of applying the agent patch itself. Experiments first reveal that explanation quality is a distinct axis of agent evaluation: ExplainBench ranks agents differently from the widely-used SWE-bench Verified benchmark. A deeper breakdown of explanation quality in agents shows frequent problems in explanations, such that explanations often claim that a patch is correct when it is not. Based on this insight, we implement and evaluate an explanation audit agent which runs additional tests to validate and refine explanations. This agent improved the explanations of all evaluated agents, demonstrating agent explanations can be automatically made more trustworthy.
1 Introduction
ExplainBench addresses the missing evaluation of agent explanation trustworthiness by using LLM-answerable questions to compare explanations. Experiments show explanation quality can diverge from coding efficacy, while auditing improves explanations.
- Agent explanations matter because growing code-generation scale makes manual inspection costly, increasing the need for accurate summaries.
- The field lacks benchmarks for explanation trustworthiness, hindering reliable comparison and systematic improvement of agent explanations.
- ExplainBench scores explanations by testing whether an LLM correctly answers multiple-choice questions about the bug and patch.
- Five-agent evaluation found explanation quality independent from agent efficacy; trae-agent had the highest SWE-bench Verified resolution rate but the second-lowest explanation score.
- ExplanationAuditAgent uses differential testing to audit explanation claims and generated higher-quality refined explanations for every evaluated agent.
- ExplainBench targets explanation correctness rather than developer satisfaction or readability, although its modular code can support additional criteria.
2 Related Work
Prior work established coding-agent benchmarks and demand for explanations, but did not provide a benchmark for explanation quality. ExplainBench fills that gap and treats explanation quality as distinct from issue-resolution efficacy.
- Software-engineering agents use LLM scaffolding to retrieve information and autonomously perform tasks such as issue resolution, testing, and patch generation.
- SWE-bench Verified is a prominent benchmark built from 500 human-verified real-world software issues, alongside multilingual, time-ordered, and related variants.
- ExplainBench is presented as the first benchmark to automatically evaluate agent explanation quality and compares it with SWE-bench Verified issue-resolution rate.
- Developer surveys identify explanations as highly sought-after artifacts for trusting automated program-repair tools and reasoning about their outputs.
- LLM-based systems increasingly generate explanations with patches, but the field lacks a framework for evaluating those explanations.
3 Motivation
Agent summaries can make ineffective patches appear successful, motivating an automated benchmark based on questions with verifiable answers. The benchmark treats explanation quality as answerability about bugs and patches.
- Developers often read an agent’s natural-language summary before inspecting its patch, making misleading claims especially consequential.
- The motivating Lingxi example claims the duplicate-code check was disabled when min-similarity-lines=0 and that the fix resolved the issue completely.
- The example patch had no impact on the bug because its modified method was never covered by bug-reproducing tests.
- ExplainBench addresses this problem by asking verifiable questions and checking whether an LLM can answer them from the natural-language explanation.
4 ExplainBench Framework
ExplainBench gathers behavioral evidence from developer tests and patches, constructs questions about intent and patch effect, and evaluates explanations through LLM answers. Its four question types cover end-to-end and function-level views.
- The framework gathers behavioral information from developer tests, developer patches, and agent patches before constructing and scoring explanation questions.
- Multiple-choice questions use validated expressions rather than requiring explanations to contain complex values or generate code expressions directly.
- ExplainBench evaluates whether explanations describe both intended program behavior and the actual effect of the agent patch.
- Intent uses the developer patch as an oracle for intended behavior, whereas effect is derived from the agent patch’s actual code change.
- The benchmark combines intent versus effect with end-to-end versus local scope, yielding four question categories.
4.4 Question Types
ExplainBench uses four question types to test whether explanations communicate intended behavior and patch effects at end-to-end and function-local levels. An LLM answers multiple-choice questions from each explanation, and the explanation score is the proportion answered correctly.
- Question dimensions: Four question types cross intent versus effect with end-to-end versus local program behavior.Intent questions assess intended behavior, while effect questions assess behavior after applying the agent patch.
- End-to-end questions: End-to-end intent questions test whether explanations convey intended behavior rather than buggy behavior using masked bug-reproducing property-based tests.The LLM selects the expected behavior expression that completes the masked test.
- End-to-end questions: End-to-end effect questions test whether explanations describe how the agent patch changes test outcomes before and after patch application.These questions focus on whether a reported fix can be trusted through observed test behavior.
- Local questions: Local intent questions assess whether explanations identify a faulty function and state the function-level specification it violates.The LLM reasons from the influenced function, bug line, inputs, and execution timing.
- Local questions: Local effect questions assess whether explanations describe the patch’s actual effect on program state at function level.The question provides the pre-patch function, affected line, and inputs.
- Evaluation: The explanation score is the proportion of questions an LLM answers correctly, with scores also available for individual question types.Answers must match the ground-truth letter or letters for the question.
5 ExplainBench Construction
ExplainBench is constructed on SWE-bench Verified by extracting agent explanations, generating grounded multiple-choice questions, and comparing buggy and patched executions. Local questions use differential execution traces to identify the first observable behavioral divergence.
- Benchmark inputs: ExplainBench derives multiple-choice questions from SWE-bench Verified instances, grounding intent in developer patches and effect in agent patches.The benchmark pairs each bug instance with an extracted agent explanation.
- Explanation extraction: 9 of the top 10 SWE-bench Verified agents generate natural-language explanations in their final tool-call step.These explanations are extracted directly from public agent trajectories without modification.
- End-to-end construction: End-to-end intent questions use masked property-based tests whose expected behavior expression is manually identified and paired with generated distractors.The LLM must select the expression that reconstructs the bug-reproducing test.
- End-to-end construction: End-to-end effect questions are generated per agent submission because each agent patch can produce a different behavior.Construction records whether the test passes or fails before and after the patch, including failure lines and exceptions.
- Local construction: Local questions compare buggy and patched execution traces to find the earliest difference in variable state or control flow.This delta behavior marks the beginning of subsequent behavioral differences and supplies the basis for local answer choices.
- Local construction: The tracing algorithm preserves call context, compares aligned events and outcomes, and returns no delta when observable behavior remains equivalent.It records control flow and variable values at line-level granularity using an instrumented Python execution harness.
- Candidate selection: Candidate expressions are generated from the diverging functions, statement, and complete variable state before distractors are selected.The selection process favors distractors similar to the correct candidate, reducing trivial questions.
6 Auditing Code Explanations from Agents
ExplanationAuditAgent improves agent explanations by testing their claims and inspecting relevant call graphs. It compares collected evidence with the original explanation, revising contradictions or appending confirmations.
- Audit operation: ExplanationAuditAgent accepts an agent patch, explanation, and issue description, then generates tests to check the explanation’s claims.Its DiffExecution tool runs each generated test before and after the patch and reports outcomes such as return codes or stdout.
- Audit operation: The auditor can inspect the call chain leading to a target function and use shell, file, and patch-management tools during validation.These tools support differential testing and investigation of how the patch affects execution.
- Explanation refinement: When evidence contradicts an explanation, ExplanationAuditAgent revises it and explicitly describes the conflicting evidence.When evidence agrees, it appends confirmation describing the validation steps and why they support the claim.
7 Experimental Setup
The experiments compare explanation quality, agent efficacy, explanation failure modes, and auditing improvements across selected open-scaffold agents. ExplainBench is built from a filtered subset of SWE-bench Verified and evaluated with a standardized LLM question-answering prompt.
- Research questions: The experiments ask how agents score on ExplainBench, what explanation problems they exhibit, and how ExplanationAuditAgent improves their explanations.These questions respectively target explanation quality, explanation failure modes, and auditing effectiveness.
- Agent selection: Agents are selected from the top 20 SWE-bench Verified submissions using adoption or publication evidence and publicly available complete trajectories.The selection also requires coverage of at least 90% of bug instances.
- Prompt design: ExplainBench applies one question-answering prompt template across question types, filling context with generated tests for end-to-end questions and local execution information for local questions.Local context includes the pre-patch function, inputs, and the divergence line.
- Benchmark construction: 297 bug instances remain after excluding SWE-bench Verified cases with failing developer patches or impractical tracing requirements.The excluded cases include fragile program states and excessive trace, runtime, or resource demands.
- Benchmark construction: The authors report no statistically significant differences between ExplainBench and SWE-bench in project composition, human-rated difficulty, or patch size after exclusions.This comparison addresses whether the filtering changed the benchmark’s broad composition.
- Evaluation implementation: GPT-5.2 generates bug-reproducing tests and candidate expressions, while GPT-5-mini answers the benchmark questions.The weaker question-answering model is intended to rely more on the supplied explanation.
8 Results
ExplainBench reveals that explanation quality differs from patch efficacy, with failures involving missing information, local-intent errors, and overconfidence. ExplanationAuditAgent improves explanations by testing patches and adding behavioral details and caveats.
- OpenHands ranks fourth in SWE-bench efficacy but first in explanation quality, while trae-agent ranks first in efficacy but fourth in explanation quality.
- Failure analysis: End-to-end intent failures are dominated by uninformative explanations, while misalignment remains low.
- Failure analysis: Local-intent explanations show substantially more misalignment because agents inaccurately infer precise function-level behavior.
- Failure analysis: 79.30% of incorrect patches are predicted to pass the bug-reproducing test, demonstrating frequent overconfidence about patch correctness.
- Case study: The case study shows ExplainBench detects explanations that omit intended behavior and incorrectly describe irrelevant patch changes.The benchmark identifies both insufficient end-to-end intent explanations and inaccurate patch-effect claims.
- Auditing explanations: ExplanationAuditAgent improves all evaluated agents’ explanation scores by running tests and incorporating expected behavior, unchanged code, edge cases, and examples.In 60 sampled explanations, additions covered before-and-after behavior in 46%, unchanged code in 26%, edge cases in 5%, and example values in 17%.
- Auditing explanations: The audit costs $0.05 per explanation on average and can add caveats that accurately distinguish intended behavior from patch effects.In the case study, testing exposed that a pathlib.Path equal to app_dir would still evade detection and avoid the expected error.
9 Implications
The paper recommends structuring agent workflows and prompts to require informative explanations. Its examples connect architectural enforcement and explicit content requirements with better explanation scores.
- Structurally encourage explanations: Agents should structurally require explanations, such as through a finish tool that mandates a clear summary of actions and results.
- Describe explanation desiderata: System prompts should specify explanation content, with refact directing agents to summarize the suspected root cause in concise bullet points.
10 Threats to Validity
The evaluation is bounded by its project selection, stochastic scoring, and partial coverage of explanation quality. The authors retain the design for scalable and reproducible assessment.
- External validity: The benchmark and findings are limited to open-source SWE-bench Verified projects with regression test suites and reproducible failures.The framework can nevertheless be extended beyond SWE-bench.
- Internal validity: Explanation scores include stochastic variation, so experiments are repeated five times and aggregated results report low standard errors.
- Construct validity: Explanation score captures behavioral alignment but not readability, usefulness, or other qualitative dimensions of explanation quality.This scope enables easily scalable and reproducible evaluation.
11 Conclusion
ExplainBench evaluates whether agent explanations support correct answers about bugs and patches, revealing that explanation quality can diverge from patch efficacy. ExplanationAuditAgent then uses testing to calibrate explanations and improves scores across agents.
- ExplainBench automatically evaluates explanations by testing whether they support correct answers about buggy behavior and agent patches.
- Explanation quality and patch-generation efficacy do not always correspond, supporting independent evaluation of explanations.
- ExplanationAuditAgent runs tests to calibrate overconfident explanations and improved agent scores by 10.9% on average.