PaperlayerView exact revision on arXiv

Source-linked AI summary

Retrofitting Code Using LLMs to Support Exceptional Behavior

Linghan Zhong, Jiyang Zhang, Jayanth Srinivasa, Junyi Jessy Li, Milos Gligoric

arXiv:2609.10397v1cs.SEcs.CL

TL;DR

Manually adding ERC is laborious, and existing LLM approaches do not directly implement developer-specified exceptional behaviors from tests. EXCODER combines static and dynamic program analysis with LLMs for ERC retrofitting, improving benchmark performance over baseline while remaining imperfect because test oracles and scope constrain semantic correctness.

  • Problem

    Retrofitting ERC is laborious, while prior LLM approaches mainly avoid unhandled exceptions rather than implementing developer-specified exceptional behaviors from EBTs.

  • Method

    EXCODER combines static and dynamic program analysis to provide LLMs with contextual information for retrofitting ERC into existing methods.

  • Results

    EXCODER with Qwen 2.5 Coder 32b achieved 85.92% pass@1, 86.18% pass@5, and 86.51% pass@10, improving over baseline by 12.56, 12.82, and 13.15 percentage points, respectively.

  • Takeaways & Limitations

    EXCODER provides an effective, though imperfect, way to retrofit ERC with LLMs using test-driven development.

  • Takeaways & Limitations

    The benchmark approximates naturally missing ERC, and pass@k measures test-oracle success rather than semantic equivalence or practical usefulness.

Abstract

from arXiv · show

Exception Related Code (ERC), which includes throw statements, conditions (if statements) that guard those throw statements, and try/catch blocks, is an essential component of software systems, allowing developers to detect and handle exceptional states that deviate from the expected program behavior. However, manually writing ERC across large codebases is tedious. We propose a novel task: retrofitting existing code with ERC. Namely, given code (without ERC) and Exceptional Behavior Tests (EBTs) (e.g., check if method throws InvalidArgumentException if null is given as the value to the argument) we aim to automatically generate missing ERC, such that the given tests pass. We design and implement Exception Coder (EXCODER) that performs context engineering to help Large Language Models (LLMs) tackle this task. EXCODER integrates static and dynamic program analysis with LLMs by providing the extracted contextual information to the LLMs. To evaluate EXCODER, we build a benchmark constructed from GitHub Java repositories, where we systematically remove ERC in 304 methods from 75 projects. Our results demonstrate that EXCODER provides an effective, though imperfect, solution to this problem in automated code generation, offering developers the first way to implement ERC following test-driven development. When combined with Qwen 2.5 Coder 32b, EXCODER achieves pass@1, 5, and 10 rates of 85.92% (12.56 percentage points over baseline), 86.18% (12.82 p.p. over baseline), and 86.51% (13.15 p.p. over baseline), respectively, on developer-written test suites. Our manual inspection of the generated code further reveals limitations of EXCODER, pointing to directions for future work.

I. INTRODUCTION

The paper introduces ERC retrofitting from Exceptional Behavior Tests and presents EXCODER, which supplies static and dynamic context to LLMs. On a Java benchmark, EXCODER consistently outperforms a test-and-method-only baseline, though generated code remains imperfect.

  • Motivation: Writing ERC is laborious because developers must identify exceptional conditions and the control-flow branches where exceptions should be signaled.ERC includes throw statements, try/catch blocks, and exceptional condition checks.
  • Problem and approach: EXCODER retrofits existing methods with exception-related code from Exceptional Behavior Tests, including checks and throw statements that implement specified exceptional behavior.The task follows test-driven development: developers express intended exceptional behavior through executable tests rather than natural-language conditions.
  • Evaluation: The benchmark is derived from Java repositories and evaluates whether generated ERC passes the associated exceptional-behavior and existing non-exceptional tests.The study compares EXCODER against a baseline containing only the target method and EBTs, and evaluates five models with different sizes and architectures.
  • Results: 85.92% pass@1, 86.18% pass@5, and 86.51% pass@10 were achieved by EXCODER with Qwen 2.5 Coder 32b, gains of 12.56, 12.82, and 13.15 percentage points over baseline on developer-written test suites.EXCODER consistently outperformed the baseline across all reported metrics for the selected models.
  • Limitations: Manual inspection of generated code reveals limitations of EXCODER and motivates future work.The reported results describe EXCODER as effective but imperfect for automated ERC generation.
  • Approach: EXCODER combines static and dynamic program analysis to provide LLMs with context for generating ERC at appropriate source-code locations.The context includes repository information and runtime information collected from executing EBTs.

II. TASK DEFINITION

The task is to add ERC to a target method using EBTs while preserving its original functionality. EXCODER focuses on collecting repository and execution context that helps an LLM generate the updated method.

  • Task definition: Given a target method and EBTs, the system must produce an updated method that passes all exceptional-behavior tests while preserving existing non-exceptional behavior.ERC comprises throw statements and the checks or try/catch structures that determine when exceptions are thrown.
  • Assumptions: The task assumes desired exceptions originate directly from throw statements within the target method and that at least one EBT is defined on it.Such cases account for 49.7% of methods with EBTs in the collected data.
  • Example: An example requires adding a check for prior initialization and throwing FaxException when the object is initialized twice.The EBT specifies the exceptional input behavior, while the generated method adds the corresponding condition and throw statement.
  • EXCODER scope: EXCODER’s workflow collects useful context from the target method, EBTs, and their repository to guide LLM generation.The workflow is illustrated as an overview of context engineering for ERC retrofitting.

III. EXCODER

EXCODER context-engineers prompts for ERC retrofitting by combining static information about code and exceptions with dynamic execution information from EBTs. This context is supplied to an LLM alongside the target method and tests.

  • Workflow: EXCODER extracts exception constructors and available symbols statically, then captures thrown exceptions, line coverage, and non-EBT behavior dynamically.These context elements are integrated into a structured prompt for the target method and its EBTs.
  • Workflow: The resulting prompt helps the LLM reason about appropriate ERC and identify source locations where it can insert the generated code.EXCODER uses both repository-derived and execution-derived information rather than only the target method and tests.

A. Static Analysis

EXCODER’s analysis supplies static construction and symbol information together with dynamic execution evidence, then annotates the method in a structured prompt for LLM generation.

  • A. Static Analysis: Static analysis provides exception-constructor signatures so the LLM can construct the exception type required by an EBT.For repository-external exceptions, EXCODER inspects dependency class files to recover constructor parameter types and order.
  • B. Dynamic Analysis: Dynamic analysis uses thrown-exception information and EBT line coverage to narrow ERC locations to reachable code and expose program states at branch points.This helps the LLM infer exceptional conditions while preserving normal execution paths.
  • C. LLM Generation: EXCODER combines static and dynamic context with the method and EBTs into a structured prompt, including inline annotations for thrown exceptions and covered lines.The annotations mark where exceptions occur and which EBT executed each covered line.

IV. DATASET

The dataset is built from GitHub Java projects by identifying methods with exceptional behavior tests and systematically removing exception-related code. The removal rules cover try/catch, if/else, switch, and standalone throw statements, while retaining cases that may no longer compile.

  • Data collection: The dataset comprises Java methods from GitHub projects selected for Maven compatibility, successful compilation, and permissive licensing.Methods were selected when at least one exceptional behavior test was defined for them.
  • ERC removal: ERC is removed using structure-specific rules for try/catch, if/else, switch, and standalone throw statements.The rules delete or flatten constructs depending on where throw statements occur.
  • ERC removal: Switch removal deletes throwing cases and removes or preserves the switch depending on how many cases remain.A single remaining case is moved outward, whereas multiple remaining cases preserve the switch.
  • Dataset assumptions: The removal process leaves 34 methods with missing-return errors and 4 with unreported-exception errors, and these data points are retained.These compilation failures arise because removing ERC can leave branches without returns or checked exceptions without handling.

C. Tool-Generated Tests

The evaluation augments developer-written exceptional behavior tests with tests generated by Randoop and EvoSuite. The dataset records broad exception coverage and ERC structure statistics while defining multiple evaluation questions.

  • Tool-generated tests: Randoop and EvoSuite generate additional tests against ground-truth implementations to assess semantic equivalence of generated ERC.Only generated tests that trigger an exception inside a target method are retained and matched using runtime stack traces.
  • Dataset statistics: The full dataset contains 518 methods, 934 exceptional behavior tests, and 100 exception types across 118 eligible projects.These statistics include both validation and evaluation sets.
  • Dataset statistics: The dataset contains 558 throw statements, including 49 in try-catch blocks, 364 in if statements, 12 in switch statements, and 133 elsewhere.Projects are split between validation and evaluation sets, with validation used to guide design decisions.
  • Research questions: The evaluation asks how EXCODER performs across models, how prompt components contribute, how method complexity affects performance, and how self-repair changes results.These questions define four evaluation dimensions.

A. Baselines

EXCODER is compared with a baseline that supplies only the target method and exceptional behavior tests, while evaluation uses multiple models and standardized sampled-output metrics. Additional ablation and self-repair variants provide comparison points.

  • Baseline: The baseline prompt contains only the target method and its exceptional behavior tests, excluding context from static and dynamic analysis.Ablation variants and the self-repair baseline provide additional comparisons.
  • Comparison design: The study compares EXCODER with a baseline prompt and evaluates how context components and self-repair affect ERC generation.The experiment reports results across five models and component-specific variants.
  • Evaluation metrics: The @k metrics estimate the probability that at least one of k generated samples satisfies a success criterion, using n = 10 samples and k ≤10.Reported criteria include compilation, exceptional behavior tests, and user tests with or without tool-generated tests.
  • Models: The evaluation uses five models spanning open- and closed-source systems, coding-specialized and general-purpose models, and parameter sizes from 7b to 32b.The models include Llama3.1 8b, Phi4 14b, Qwen 2.5 Coder 7b, Qwen 2.5 Coder 32b, and GPT-5 Mini.

A. RQ1: Effectiveness Across Models

EXCODER consistently improves ERC retrofitting across five models and metrics, with especially large gains for Qwen 2.5 Coder 32b. Its context engineering expands the set of successfully solved tasks, although added context can occasionally introduce noise.

  • Effectiveness across models: EXCODER consistently outperforms the baseline across all five models and all metrics.For Qwen 2.5 Coder 32b, gains include 9.87 percentage points on compiled@5 and 13.15 p.p. on pass@5 (EBTs).
  • Effectiveness across models: EXCODER solves 48 target methods that the baseline fails to solve while missing 8 methods solved by the baseline.The comparison uses pass@10 (All User) with Qwen 2.5 Coder 32b.
  • Limitations: Additional context can hurt straightforward tasks when noise makes relevant information harder for the model to prioritize.These rare failures motivate methods for prioritizing the most relevant context.
  • Ablation study: EXCODER outperforms every single-context ablation variant on every metric in the Qwen 2.5 Coder 32b study.The ablations evaluate individual context components added to the baseline prompt.
  • Ablation study: Available symbols and exception constructors yield large correctness gains because they directly specify usable code elements and required exception signatures.These components are easier for the model to use than behavior-oriented context.
  • Ablation study: Line coverage, thrown exceptions, and non-EBTs provide smaller gains that appear mainly at pass@5 or pass@10.Using these signals requires reasoning about program states and control flow.

C. RQ3: Effect of Target Method Complexity

EXCODER matches or outperforms the baseline across target-method complexity bins, while iterative repair improves both prompts but remains more effective with EXCODER’s contextual information.

  • C. RQ3: Effect of Target Method Complexity: Methods longer than 14 lines and methods with cyclomatic complexity at least five are uncommon, comprising 19.7% and 14.1% of targets, respectively.These groups are reported in open-ended 15+ and 5+ bins because their lengths and complexities vary substantially.
  • C. RQ3: Effect of Target Method Complexity: EXCODER matches or outperforms Base in every method-length and cyclomatic-complexity bin.The analysis uses pass@5 (All User) on Qwen 2.5 Coder 32b, with lines of code and cyclomatic complexity as complexity proxies.
  • C. RQ3: Effect of Target Method Complexity: EXCODER’s advantage is largest for methods of 6 to 14 lines and for methods with cyclomatic complexity two.The authors associate this pattern with simpler exceptional checks and the additional information available in longer methods.
  • D. RQ4: Combination with Self-Repair: 96.22%: After four repair rounds, Iter-EXCODER reaches pass@5 (All User), versus 84.47% for Iter-Base.Most tasks are solved in the first repair iteration, with diminishing returns afterward.
  • D. RQ4: Combination with Self-Repair: Self-repair alone cannot substitute for EXCODER’s contextual information when the underlying issue requires missing context rather than surface-level correction.After four rounds, Iter-Base still trails Iter-EXCODER by 11.75 percentage points on pass@5 (All User).

B. Equivalence to Ground Truth

EXCODER produces more functionally equivalent ERC than the baseline, but passing tests do not ensure equivalence because many false positives arise from insufficiently specific exceptional-condition tests.

  • B. Equivalence to Ground Truth: 65.13%: EXCODER’s generated ERC is functionally equivalent to ground truth for 198 of 304 target methods, compared with 54.61% for the baseline.The equivalence labels compare generated ERC with the original ground-truth implementation.
  • B. Equivalence to Ground Truth: False positives are classified as Too Lenient, Too Strict, Destroyed Code, or Wrong Handling according to differences between generated and ground-truth throwing behavior.The categories distinguish input-set mismatches, rewritten target code, and differing throw locations or side effects.
  • B. Equivalence to Ground Truth: 63.08% of EXCODER’s false positives are too lenient, meaning they reject fewer inputs than the ground-truth ERC.The baseline share is 64.91%, and the false-positive rates differ by only 0.8 percentage points.
  • B. Equivalence to Ground Truth: An EBT can pass while generated ERC deviates from ground truth when it omits inputs distinguishing a weaker exceptional condition.The authors identify EBT coverage as a key limitation for communicating the developer’s intended condition.
  • B. Equivalence to Ground Truth: In the VideoPost example, EXCODER checks that embed is non-null and non-empty, whereas ground truth rejects setData whenever embed is non-null.Consequently, an empty embed string is incorrectly accepted by the generated ERC.

VIII. LIMITATIONS

EXCODER’s evaluation has important scope and validity boundaries: its benchmark approximates missing ERC, pass@k measures test passing rather than semantic usefulness, and its technical scope is limited to exceptions originating within target methods.

  • The benchmark removes developer-written ERC, approximating but not reproducing naturally missing ERC.
  • pass@k measures whether sampled candidates pass a test oracle, not whether they are semantically equivalent or useful in practice.
  • EvoSuite and Randoop strengthen test oracles beyond user-written tests but may miss relevant cases and do not fully establish developer intent.
  • EXCODER targets exceptions originating from throw statements within the target method, leaving propagated or wrapped call-chain exceptions for future work.
  • The dataset contains only compilable Java projects using Maven, although EXCODER’s context-engineering design is language-agnostic.

IX. RELATED WORK

Prior work addresses exception handling, fault localization, TDD, and LLM code generation, whereas EXCODER focuses on retrofitting ERC from exceptional behavior tests and improves LLM performance across model sizes and architectures.

  • Machine learning for generating exception-related code: Existing work includes exception prediction and handling, such as ThEx, FuzzyCatch, Neurex, and LLM-based prompt chaining.
  • Exception-related code generation: Unlike prior approaches that avoid unhandled exceptions, EXCODER makes methods throw the correct exception under specified exceptional conditions.
  • Fault localization: Fault-localization research informs ERC-location analysis, while EXCODER also uses test coverage as dynamic-analysis context.
  • Test-driven development: TDD-based systems use tests to validate or improve generated code, but EXCODER specifically leverages exceptional behavior tests rather than general functional requirements.
  • Contributions: EXCODER is presented as the first approach to retrofit ERC into target methods using exceptional behavior tests and LLMs.
  • Contributions: EXCODER extracts context from target methods and EBTs through static and dynamic analysis, consistently improving LLM performance across model sizes and architectures.
Loading 2609.10397v1…