PaperlayerView exact revision on arXiv

Source-linked AI summary

ASAD: Adaptive Software Agents for Debugging

Yacine Majdoub, Eya Ben Charrada, Haifa Touati

arXiv:2610.00629v1cs.SE

TL;DR

Fixed multi-agent debugging architectures do not adapt agent counts, roles, and interactions to bugs of different complexity. ASAD analyzes faulty code to configure a coordinator-led repair process, and across three benchmarks it reports higher repair performance than prompting and static multi-agent baselines with lower average agent use.

  • Problem

    Many multi-agent debugging systems use predefined agent numbers, roles, and communication patterns regardless of bug complexity.

    Source: §I · p. 1

  • Method

    ASAD analyzes faulty code and dynamically configures agent roles, team size, and collaboration, using single-pass repair for simple bugs and specialized teams for complex ones.

    Source: §I · p. 1

  • Results

    Across DebugBench, Defects4J, and CodeFlaws, ASAD improves fix accuracy 12–20% over chain-of-thought prompting and 4–9% over static multi-agent systems, with 32% lower average agent usage.

    Source: §X · p. 14

  • Takeaways & Limitations

    A single repair agent resolves 73% of straightforward bugs, while more complex cases trigger teams of specialized agents.

    Source: §X · p. 14

    The authors identify threats to validity in their empirical findings.

    Source: §VII · p. 13

Abstract

from arXiv · show

The integration of Large Language Models (LLMs) into multi-agent systems has shown great potential for automated debugging. Yet nearly all current frameworks rely on rigid, predefined architectures: the number of agents, their roles, and their interaction patterns are fixed before any analysis of the bug occurs. This one-size-fits-all approach is fundamentally mismatched to the heterogeneous nature of software defects. Simple bugs waste resources on unnecessary coordination, while complex ones suffer from insufficient or poorly aligned expertise. This paper introduces ASAD, an adaptive agentic system for debugging that configures its team according to the nature and complexity of each bug. ASAD initiates the debugging process by analyzing the faulty code and dynamically determines the number of agents to deploy, the specialized roles they should have, and the collaboration strategy they should follow. A central coordinator orchestrates this process through iterative planning, reflection, and execution; applying fast single-pass repairs for simple issues while assembling purpose-built teams to tackle more complex failures. We evaluate ASAD on three established benchmarks: Defects4J, DebugBench, and CodeFlaws, using multiple LLMs, such as DeepSeek-V3, Qwen-3 and GPT-5. ASAD consistently improves bug-fix rates by 12--20% over chain-of-thought(CoT) prompting and consistently outperforms static multi-agent systems by 4--9% in fix precision while reducing average agent usage by 32%. Crucially, our system dynamically adjusts the number and roles of agents: it resolves simple bugs with minimal coordination and scales agent involvement only for more complex cases.

I. INTRODUCTION

ASAD addresses the mismatch between fixed multi-agent debugging teams and bugs of varying complexity by configuring its agents to the faulty code. The paper reports improved repair performance and lower agent use across three benchmarks.

I. INTRODUCTION

  • Existing multi-agent debugging systems commonly fix agent counts, roles, and communication patterns in advance, regardless of bug complexity.The paper identifies both unnecessary coordination for simple bugs and insufficient or misaligned expertise for complex ones.Source: §I · p. 1
  • ASAD analyzes faulty code and dynamically configures agent number, roles, and collaboration, using a single-pass repair for simple bugs and specialized teams for complex ones.Its coordinator combines planning, iterative reflection, and multi-agent collaboration.Source: §I · pp. 1–2
  • 12–20% higher bug-fix rates than chain-of-thought prompting, 4–9% higher fix precision than static multi-agent systems, and 32% lower average agent usage.The evaluation covers DebugBench, Defects4J, and CodeFlaws using six LLMs.Source: §I · p. 1
  • A single agent handles 73% of syntax and logic errors, while ASAD scales up to four specialized agents for complex bugs.The cited introduction names concurrency bugs and API misuse as complex cases.Source: §I · p. 1
  • This work extends an earlier feasibility study with an expanded system, evaluation across multiple benchmarks and LLM families, and deeper analysis of system behavior.The earlier work introduced adaptive multi-agent orchestration for code debugging.Source: §I · p. 2

1) Reasoning Techniques :

The section introduces reasoning techniques that support multi-step debugging, from explicit reasoning and self-evaluation to planning, tool use, and collaboration among specialized agents.

1) Reasoning Techniques :

  • The section surveys Chain-of-Thought, Self-Consistency, Iterative Reflection, Planning, Tool Use, and Multi-Agent collaboration as reasoning techniques for LLMs.These techniques are presented as supporting explicit reasoning, reliability, self-improvement, strategic decisions, grounding, and distributed problem-solving, respectively.Source: 1) Reasoning Techniques · p. 2

2) Chain-of-Thought:

  • Chain-of-Thought prompting asks an LLM to reason step by step, breaking debugging into manageable subproblems such as identifying errors and tracing variables.The passage says this can improve accuracy and make the reasoning traceable.Source: 2) Chain-of-Thought · p. 2

3) Self-Consistency:

  • Self-Consistency generates multiple reasoning paths for one input and selects a common or best-supported conclusion, such as one that passes all test cases.The passage characterizes this as improving reliability without additional training data.Source: 3) Self-Consistency · p. 2

4) Iterative Reflection:

  • Iterative Reflection has an LLM assess and revise a draft solution for correctness, efficiency, and completeness, with the cycle repeatable across attempts.Another LLM may provide external critique of logical inconsistencies or style issues.Source: 4) Iterative Reflection · p. 2

5) Planning:

  • Planning lets an LLM design an action sequence before execution, including task decomposition, plan selection, tool-aided planning, refinement, and memory use.The passage lists five planning strategies, including reflection and refinement based on failed attempts.Source: 5) Planning · pp. 2–3

6) Tool Use:

  • Tool Use lets LLMs consult external systems such as code executors, web search, documentation APIs, and retrieval systems for code examples or library documentation.The passage presents external access as a way to ground reasoning in information not contained in the model.Source: 6) Tool Use · p. 3

7) Multi-Agent Collaboration:

  • Multi-Agent Collaboration distributes reasoning across specialized agents that share findings and resolve conflicts, while debugging naturally involves hypothesis generation and iterative validation.The passage describes the agents as mirroring human software teams.Source: 7) Multi-Agent Collaboration · p. 3

8) Relevance of Reasoning Techniques for Debugging:

The section relates structured reasoning to debugging’s uncertain, iterative work and frames agenticness through goal and environmental complexity, adaptability, and independent execution.

8) Relevance of Reasoning Techniques for Debugging:

  • Debugging involves hypothesis formation, validation under partial observability, and iterative refinement, making stepwise reasoning and self-consistency relevant to its ambiguity and multi-stage demands.The passages also connect reflection, planning, tool grounding, and multi-agent reasoning to debugging activities.Source: 8) Relevance of Reasoning Techniques… · p. 3

B. Agenticness in LLM-Based Systems

2) Goal complexity:

  • Goal complexity concerns the structure and abstraction of objectives, including whether an agent must decompose vague or underspecified instructions into interdependent subgoals.The passage links this to hierarchical reasoning and planning.Source: 2) Goal complexity · p. 3

3) Environmental complexity:

  • Environmental complexity concerns operating amid dynamic codebases, partial information, ambiguous inputs, or concurrent agents while maintaining goal-directed behavior.The passage emphasizes responding to uncertainty and shifting constraints.Source: 3) Environmental complexity · p. 3

4) Adaptability:

The section presents ASAD as a response to underdeveloped adaptability and independent execution in debugging systems, with a coordinator that assesses task complexity and configures agent involvement.

4) Adaptability:

  • Adaptability is the ability to adjust strategy in response to new information, task variation, or performance feedback.The passage says such adjustment can occur on the fly.Source: 4) Adaptability · p. 3

5) Independent execution:

  • Independent execution means pursuing goals without step-by-step human intervention, including diagnosing a bug, proposing and validating fixes, and iterating.The passage describes monitoring progress and revising plans when obstacles arise.Source: 5) Independent execution · p. 4

6) Agenticness of Debugging Systems:

  • Current debugging systems are described as weak in adaptability and independent execution, with fixed workflows imposing overhead on simple bugs and potentially mismatched specialists on complex ones.The passage also notes that static multi-agent designs can drift from the core objective during extended interactions.Source: 6) Agenticness of Debugging Systems · p. 4

A. Overview of the system

B. Main Agent / Team Leader

  • The main agent directs specialized-agent creation and prioritization, integrates their feedback, validates results, and adjusts the strategy toward a solution.For complex tasks it plans and coordinates subtasks; for simple ones it attempts a direct repair, followed by a final validation.Source: B. Main Agent / Team Leader · p. 4, A. Overview of the system · p. 4

1) Code Analysis:

  • Code analysis combines five complexity dimensions: error severity, source-location difficulty, program-understanding difficulty, system-level operations, and inter-function dependencies.The resulting evidence helps the main agent choose a single-pass repair for low-complexity cases or a multi-step plan with specialized agents for complex cases.Source: 1) Code Analysis · pp. 4–5

2) Agents Profiling and Prioritization:

ASAD tailors debugging coordination to task complexity, sequencing specialized agents under coordinator review while using a single-pass repair for simple bugs. Bounded retries and final validation allow the coordinator to revise the strategy when a repair remains incorrect.

2) Agents Profiling and Prioritization:

  • Generated agent profiles specify each agent’s role, objective, and modification constraints, while dependency-aware ordering places structural fixes before semantic reasoning.Agents execute sequentially, and a step can run only after prerequisite responsibilities are completed.Source: 2) Agents Profiling and Prioritization · p. 5

3) Iterative Task Review:

  • The main agent reviews each agent’s report and requests refinement or repetition when problems remain, up to three attempts.It proceeds to the next agent when the report is satisfactory and no critical issues remain.Source: 3) Iterative Task Review · p. 5

4) Final Validation:

  • Final validation returns corrected code when the issue is resolved; otherwise, the failure log starts a new iteration with a revised strategy, up to five iterations.The revised strategy may change agent configuration, prompts, or priorities.Source: 4) Final Validation · pp. 5–6

5) Simple Fix:

  • For a simple bug, the main agent attempts a direct single-pass repair and sends its output to validation without deploying additional agents.This avoids unnecessary agent deployment and coordination overhead.Source: 5) Simple Fix · p. 6

C. Agents Setup & Workflow Organization

  • Execution is bounded to three refinement attempts per agent and five full system iterations, limits motivated by observed diminishing returns.The passage reports that further refinement rarely improves repair quality and that repairs after five cycles yield marginal improvements.Source: C. Agents Setup & Workflow Organization · p. 6

D. Specialized Agents / Team Members

ASAD’s specialized agents work sequentially under main-agent review, and the experimental setup frames evaluation around performance, adaptation, efficiency, and robustness. The benchmarks cover multiple programming languages and bug types, including DebugBench’s 4,253 instances across four bug categories.

D. Specialized Agents / Team Members

  • Specialized agents execute sequentially under main-agent supervision, which reviews their reports before proceeding or requesting refinement.This review-driven protocol is intended to allow corrections at each step and help prevent error propagation.Source: D. Specialized Agents / Team Members · p. 6

A. Research Questions

B. Benchmarks

  • The three benchmarks collectively cover Python, Java, C, and C++ and diverse bug categories.They are DebugBench, Defects4J, and CodeFlaws.Source: B. Benchmarks · p. 6

1) DebugBench:

  • DebugBench contains 4,253 instances covering syntax, reference, logic, and multiple bugs across C++, Java, and Python.It is designed for zero-shot assessment and constructed from correct LeetCode community code with bugs injected using GPT-4.Source: 1) DebugBench · p. 6

2) Defects4J:

The evaluation spans benchmarks with different languages and data characteristics and measures repair success, adaptation, and computational overhead. Bug-fix-rate reporting distinguishes test-passing plausible fixes from correct fixes that are functionally equivalent to the developer solution when reference information is available.

2) Defects4J:

  • Defects4J 3.0.1 contains 854 reproducible Java bugs from 17 open-source projects, each paired with buggy and fixed versions and accompanied by test suites.Its patches are typically small and localized, averaging about four lines in one file.Source: 2) Defects4J · pp. 6–7

3) CodeFlaws:

  • CodeFlaws contains 3,902 real-world C bugs from programming contest submissions, classified into 39 syntactic difference-based defect classes.It includes buggy and corrected versions, test suites, and metadata.Source: 3) CodeFlaws · p. 7

C. Evaluation Metrics

  • Evaluation covers bug-fix rate, adaptability, and computational overhead, including time and resource use.The primary bug-fix metric counts successful repairs; on DebugBench and CodeFlaws, generated code must pass the complete test suite.Source: C. Evaluation Metrics · p. 7, 1) Bug Fix Rate · p. 7

1) Bug Fix Rate:

  • A plausible fix compiles and passes available regression tests, while a correct fix additionally matches the developer patch’s behavior or satisfies the benchmark’s official protocol.Reporting both distinguishes test-suite consistency from semantic equivalence with the intended solution.Source: 1) Bug Fix Rate · p. 7

2) Adaptability:

  • Adaptability is assessed by tracking agent counts and iterations across bug types to see how strategy varies with task complexity.The analysis examines trends in agent usage across different types of bugs.Source: 2) Adaptability · p. 7

3) Computational Overhead:

  • Computational overhead is assessed against fixed multi-agent and single-shot approaches using inference time, token consumption, iterations, and agent resources.The measures include resources used by both the main and specialized agents.Source: 3) Computational Overhead · p. 7

D. Baselines

The setup compares ASAD with prompting methods, static repair systems, and fixed-configuration ablations, and independently characterizes bug complexity for routing analysis. The implementation uses a modular Python framework; the results section introduces findings for four research questions and a table scoped to bug-fix rates.

D. Baselines

  • Baselines include Zero-shot, CoT, four static agentic repair systems, and three ASAD ablations that use minimal, fixed-three-agent, or maximum-agent configurations.The comparisons are intended to isolate adaptive orchestration from prompting strategies and underlying model capability.Source: D. Baselines · p. 7

E. Bug Complexity Characterization

  • Bug complexity is characterized offline using changed-line count and Halstead Effort in the ground-truth-fix region, with DebugBench’s easy, medium, and hard ranges applied to the other benchmarks.The main agent does not receive the resulting labels or information derived from the ground-truth fixes; it independently routes using debugging-time information.Source: E. Bug Complexity Characterization · p. 7

F. Implementation

  • ASAD is implemented as a modular Python framework with a central controller and agent modules using structured, role-based prompts and standardized outputs.The task-analysis prompt asks the main agent to classify complexity, locate bugs, and produce a step-by-step repair plan using buggy code and a static-analysis report.Source: F. Implementation · p. 8

V. RESULTS AND ANALYSIS

  • The results section reports findings for four research questions, and Table I is scoped to bug-fix rates across benchmarks and languages versus Zero-shot and CoT prompting using four LLMs.The caption names Llama-3, DeepSeek-V3, Mistral, and Qwen-3; no table values are supplied in these passages.Source: §V · p. 8, Table I · p. 8

A. Impact on Bug-Fixing Performance

Across the evaluated benchmarks, ASAD improves bug-fix rates over prompting and produces more correct Defects4J repairs than the compared static agentic systems. Its resource use rises with bug complexity rather than remaining fixed.

1) Performance across benchmarks:

  • ASAD raises DebugBench bug-fix rates to 71.6%–77.9%, versus 59.1%–63.8% with CoT; gains also appear on CodeFlaws and Defects4J.The paper reports stability across four LLM families.Source: 1) Performance across benchmarks · pp. 8–9

2) Comparison with agentic baselines:

B. Adaptive Resource Allocation and Computational Efficiency

C. Adaptive Agent Selection: Trade-off and Agent Contributions

The ablation results show a trade-off between repair performance and computational cost across fixed strategies. ASAD’s routing classifications track bug complexity, and the agent analysis distinguishes first successful fixes from later refinement and validation.

1) Ablation Study:

  • ASAD-Lite has the lowest overhead but weakest fix rates, while ASAD-Max reaches up to 73.4%–88.3% on DebugBench and 30.5%–34.3% on Defects4J at higher cost.ASAD-Max execution time reaches 128s and token usage reaches 9.6K.Source: 1) Ablation Study · p. 10, Table V · p. 11
  • ASAD beats ASAD-Max and ASAD-Fixed on CodeFlaws-Medium (78.3% vs. 76.4%) and Defects4J-High (29.1% vs. 26.4%), while cutting ASAD-Max execution time by 50%–80%.It also lowers token consumption by roughly 40%–70% compared with ASAD-Max.Source: 1) Ablation Study · p. 10, Table V · p. 11

2) Main Agent routing analysis:

  • Complex classifications increase monotonically with reference complexity across benchmarks; CodeFlaws contains no high-complexity bugs.Complex classifications are 5.9%, 27.6%, and 74.2% on DebugBench; 23.6%, 62.8%, and 91.6% on Defects4J; and 2.7% and 38.2% on CodeFlaws (low and medium only). Medium cases have no single expected binary class.Source: 2) Main Agent routing analysis · p. 10, Table VI · p. 11

3) Agent contribution analysis:

  • In 100 successfully repaired DebugBench cases using three agents, agent two first produced a passing patch in 54% of cases; agent three handled refinement or validation in 85%.Agent one accounted for 31% and agent three for 15% of first successful fixes.Source: 3) Agent contribution analysis · pp. 10–11, Table VII · p. 11

D. Performance Variability

Across six LLM backbones, ASAD achieves bug-fix rates of 68.9%–82.2%, while agent use and execution time vary by model. Repeated DeepSeek-V3 runs show limited variation, and the discussion attributes gains to complexity-aware resource allocation and iterative refinement.

D. Performance Variability

  • Across six models, bug-fix rates range from 68.9% to 82.2%; GPT-5 reaches 82.2%, while Qwen-3 averages 1.6 agents and Mistral 2.8.The table covers bug-fix rate, average agents, execution time, and token consumption across 500 DebugBench instances.Source: D. Performance Variability · p. 11, Table VIII · p. 12
  • Execution time ranges from 51.1 s to 68.2 s, and token consumption remains between 3.6K and 4.6K across models.Models activating more agents generally incur higher latency; Mistral has both the most agent use and longest execution time.Source: D. Performance Variability · p. 12
  • Across five DeepSeek-V3 runs on 100 DebugBench instances, bug-fix rates vary from 77% to 82% (σ = 1.6%), with similarly small resource fluctuations.The figure tracks bug-fix rate, activated agents, execution time, and token consumption across the runs.Source: D. Performance Variability · p. 12, Figure 3 · p. 12

VI. DISCUSSION

  • Up to 25 percentage points higher bug-fix rates on synthetic benchmarks, nearly tripled repair performance on Defects4J, and approximately 44% lower overhead than fixed multi-agent baselines.The discussion identifies complexity-aware coordination as a determinant of efficiency, correctness, and robustness.Source: §VI · p. 12

A. Interpretation of Results

ASAD’s adaptive coordination is associated with more efficient and accurate repairs than static configurations and standard prompting, while resource use scales with bug complexity. The authors also discuss benchmark contamination, non-deterministic generation, test-suite overfitting, and subjective complexity assessment as validity concerns.

A. Interpretation of Results

  • Agent activation, execution time, and token consumption rise monotonically with bug complexity; the second agent is the primary fixer in 54% of successful repairs.The first agent typically diagnoses or provides a partial fix, while the third predominantly refines and validates.Source: A. Interpretation of Results · p. 12
  • Stronger models require fewer agents for high fix rates, while less capable models benefit from increased orchestration; repeated runs show low variance (σ = 1.6%).The evaluation spans six models, and the reported variance is across repeated runs.Source: A. Interpretation of Results · pp. 12–13
  • Defects4J repair rates remain below synthetic-benchmark rates amid project-level context and semantic dependencies, although ASAD nearly triples conventional-prompting performance.The authors characterize complex real-world Java bugs as challenging for automated systems.Source: A. Interpretation of Results · p. 13

B. Comparison with Static and Baseline Approaches

  • ASAD matches or exceeds static fix rates while reducing execution time and token consumption by 44%–55%; standard prompting plateaus on challenging benchmarks.The authors also state that unconstrained generation can produce over-correction and hallucination.Source: B. Comparison with Static and Baseline… · p. 13

C. Data Contamination in Established Benchmarks

  • Defects4J and CodeFlaws contamination may threaten external validity; the maximum Defects4J fix rate reported is 32.5%, rather than near-perfect repair.The authors argue this imperfect rate suggests models actively reason through complex semantic dependencies rather than simply retrieving memorized solutions.Source: C. Data Contamination in Established… · p. 13

A. Internal Validity

  • Complexity allocation relies on the main agent’s judgment, formalized into five dimensions and checked against human judgments on 100 bugs.The authors also report five replications with σ = 1.6% and manual inspection of all 223 patches classified as correct to address other validity threats.Source: A. Internal Validity · p. 13

B. External Validity

The paper reviews debugging research on LLM reasoning and agentic systems, and concludes that ASAD’s benchmark results support adapting agent configuration to bug complexity. Its evaluation remains bounded by benchmarks that primarily cover single-file or unit-level defects, leaving repository-scale effectiveness uncertain.

B. External Validity

  • Benchmarks spanning different languages and bug types primarily cover single-file or unit-level defects, so effectiveness on large industrial and repository-scale systems remains uncertain.Consistent trends across the evaluated model families suggest the findings are not tied to a specific evaluated LLM, but newer models may behave differently.Source: B. External Validity · pp. 13–14

VIII. RELATED WORK

  • The related-work section surveys recent LLM reasoning techniques and agent-based approaches to debugging.Source: §VIII · p. 14

A. LLM Reasoning Techniques in Debugging

B. LLM-based Agentic Systems for Debugging

IX. FUTURE WORK

  • Future work targets repository-scale coordination, runtime-feedback validation, developer-tool integrations, and human-in-the-loop studies of trust, intervention timing, and efficiency.Source: §IX · p. 14

X. CONCLUSION

  • ASAD improves fix accuracy by 12–20% over chain-of-thought prompting and by 4–9% over static multi-agent systems, with 32% lower average agent usage.The authors conclude that agent configuration should depend on problem complexity rather than a fixed predefined choice.Source: §X · p. 14
Loading 2610.00629v1…