PaperlayerView exact revision on arXiv

Source-linked AI summary

Code Understanding is a Bottleneck for Coding Agents

Nishant Balepur, Kiran Tomlinson, Tobias Schnabel

arXiv:2610.10610v1cs.SEcs.AIcs.PL

TL;DR

Repository benchmarks jointly vary code and task types, making it difficult to identify which abilities drive agent errors. CABRA builds controlled coding tasks from call graphs and scales them across ability dimensions. Its results show that code understanding can predict task difficulty beyond edit volume, and that understanding-intensive tasks can reduce agent accuracy.

  • Problem

    Repository benchmarks jointly vary code and task types, limiting diagnosis of which abilities drive coding-agent errors.

    Source: §1 · p. 1

  • Method

    CABRA generates coding tasks from directed acyclic call graphs, transforms them into task types, and scales them across four ability axes.

    Source: §1 · p. 1

  • Results

    Across CABRA and SWE-bench Verified analyses, understanding-related tool use predicts difficulty better than edit counts, and agent accuracy falls on an understanding-intensive code-merging task.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The authors argue for controllable evaluations alongside repository benchmarks to diagnose agent errors and identify code understanding as an accuracy and efficiency bottleneck.

    Source: §1 · p. 2, §7 · p. 9

    Tasks once difficult for LLMs can be trivial for agents with tools, so discerning agent-hard tasks requires formalizing tool-calling work and what tools can execute.

    Source: §7 · p. 10

Abstract

from arXiv · show

Repository benchmarks (e.g., SWE-bench) for coding agents often assume that lines of code edited can predict task difficulty, but such datasets' poor control over code and task types makes it hard to know which abilities truly drive agent errors. We present CABRA: a Coding Ability Blueprint for Rigorous Agent evaluation. CABRA builds tasks from scratch as call graph transformations and scales difficulty via a task size parameter on four axes: function traversal, search, runtime resolution, and instruction following. We run eight LLMs and six coding agents on 6,840 CABRA tasks to show: 1) LLM accuracy falls as task size~grows, but agents stay near-perfect by offloading work to tools (e.g., grep); 2) Larger CABRA tasks elicit more tool calls for reading and analysis, while a separate study on SWE-bench Verified shows these tool call counts predict agents' accuracy better than lines of code edited, suggesting task difficulty for agents can lie in understanding code to edit, not just in making edits; 3) Extending CABRA to an intense understanding task where models analyze divergent logic across two classes backs this finding, as agent accuracy finally falls. More broadly, we argue for synthetic evaluations like CABRA to unmask LLM weaknesses trivialized by tools (e.g., needle-in-a-haystack) and abilities beyond just editing (e.g., understanding) that coding agents still find difficult, pairing SWE-bench-style tasks with controlled diagnosis.

1 INTRODUCTION: HERDING CODING AGENT ABILITIES WITH CABRA

Repository-derived benchmarks jointly vary code and task types, limiting diagnosis of why agents fail. CABRA controls task construction and size; its results point to code understanding, not edit volume alone, as a difficulty for coding agents.

1 INTRODUCTION: HERDING CODING AGENT ABILITIES WITH CABRA

  • Repository-augmented benchmarks jointly vary code and task types, supporting correlational analyses that can misidentify why agents fail.The passage contrasts this with an analysis finding low accuracy when agents read, analyze, and test codebases more.Source: §1 · p. 1
  • CABRA generates programs from directed acyclic call graphs, transforms them into five task types, and scales task size across traversal, search, runtime resolution, and instruction following.It separates task-type definition from task-size scaling to support controlled analyses.Source: §1 · p. 1
  • On larger CABRA tasks, LLM accuracy falls while agents sustain near-perfect scores with tools; on SWE-bench Verified, understanding-related tool-call counts predict accuracy better than line-edit counts.Agents often spend more tokens reading and analyzing code as task size grows, without always editing more.Source: §1 · p. 2
  • In a controlled code-merging task, agent accuracy falls as task size grows, with understanding-tool tokens ballooning while edits remain small.Follow-up analyses suggest failures involve logical decomposition, surface-level naming, and excising one-line differences.Source: §1 · p. 2
  • The paper argues for controllable tasks alongside repository benchmarks to attribute agent errors and identifies code understanding as an accuracy and efficiency bottleneck.Its contributions also include a code-merging task to track understanding and a taxonomy and classifier for agent tool calls.Source: §1 · p. 2

2 CABRA: TAKING CONTROL OF CODING AGENT EVALUATIONS

CABRA builds coding tasks from directed acyclic call graphs, then varies task size along four ability axes. Its synthetic construction trades downstream realism for simplicity and tight control over tested coding abilities.

2.1 TRANSFORMING DIRECTED ACYCLIC GRAPHS AS CODE EDITING TASKS

  • Programs are represented as directed acyclic call graphs, with functions as vertices and calls as directed edges; each function computes a random mathematical function of its input.The synthetic generation trades downstream realism for simplicity and tight control over tested abilities.Source: §2.1 · p. 3
  • Five refactoring tasks use an unedited Cinit graph and an edited Cedit graph, combined into the input codebase C.The tasks are grounded in a catalogue of refactoring types and are intended to support scaling analyses across more than one task type.Source: §2.1 · p. 3

2.2 SCALING TASK SIZE ACROSS COMPLEXITY AXES

  • CABRA scales base tasks along four axes: function traversal, function search, runtime resolution, and instruction following, each using task size n.The authors note that these axes do not cover every ability agents need and that CABRA can be extended to others.Source: §2.2 · pp. 3–4
  • Function traversal scales the number of functions to edit; function search scales functions not needing edits, runtime resolution lengthens a check, and instruction following adds distinct requests.Instruction following cannot be applied to Remove Dead Code because deletion is the only valid action.Source: §2.2 · pp. 3–4

3 EXPERIMENTAL SETUP

The experimental setup evaluates 6,840 controlled tasks across code types, task types, axes, and sizes. It compares language models without tools with coding agents that use a public harness and read-write code access.

3.1 TASKS

  • CABRA generates 6,840 tasks across three code types, five tasks (four for instruction following), four axes, and six sizes, with 20 tasks per configuration.The varied code and task types support assessment of task-size scaling per axis.Source: §3 · p. 4
  • Tasks sweep n ∈ {5, 10, 25, 50, 100, 200}, with different fixed graph sizes for traversal/search versus runtime-resolution/instruction-following configurations.The setup samples 20 graphs per configuration, varying calls, operations, runtime check inputs, and instruction-following rules.Source: §3.1 · p. 4
  • Scores are binary per task: a model earns 1.0 only if all black-box tests and task-specific checks pass against the ground-truth program.Each function is checked on 50 random inputs, with additional checks for dead-code removal, valid runtime branches, and instruction execution.Source: §3.1 · p. 4

3.2 A FLOCK OF LANGUAGE MODELS AND AGENTS

  • The comparison includes eight language models and six coding agents; models receive a problem statement, while agents use Copilot’s three-stage ReACT loop with bash tools and read-write folder access.The experiments use default parameters when unspecified; only one harness is used for the agents.Source: §3.2 · p. 4

4 AGENTS SPEND EFFORT UNDERSTANDING CODE, NOT JUST EDITING

CABRA results show that tools let agents retain accuracy on larger tasks that degrade LLMs, while traces point to code understanding—not simply editing—as a substantial part of agent effort and difficulty.

4.1 COMPARISON OF LLM AND CODING AGENT ACCURACY

  • As task size grows, LLM accuracy drops while agents stay mostly constant, using scripts, grep, or exec() for traversal, search, and runtime resolution; instruction following is the exception.Agents also degrade on instruction following, where natural-language rules must be mapped to edits.Source: §4.1 · p. 5
  • GPT-5.4 Mini in an agent harness surpasses tool-free GPT-5.5 and Grok on non-instruction-following tasks when n > 100.The comparison suggests that smaller LLMs can perform well in harnesses when they generate and compose tool calls effectively.Source: §4.1 · p. 5

4.2 TRACE ANALYSIS: AGENTS OFTEN SPEND TOOL CALLS ON UNDERSTANDING CODE

  • READ and ANALYZE calls and tokens scale with task size on three axes, while EDIT scales only for instruction following; READ and ANALYZE often precede the first edit.The authors interpret READ and ANALYZE as code-understanding actions: acquiring knowledge of program construction and reasoning about behavior to guide edits.Source: Figure 3 · p. 6, §4.2 · p. 6

4.3 UNDERSTANDING PERSISTS IN SWE-BENCH VERIFIED

  • Across human-rated difficulty levels, edits, reading, analysis, and testing all rise together, so SWE-bench alone cannot disentangle edit volume from understanding demands.The levels estimate the time developers think humans would need to complete tasks.Source: §4.3 · p. 7
  • Understanding-tool call count (READ + ANALYZE) is most strongly correlated with agent accuracy on SWE-bench Verified, exceeding EDIT-token use and lines edited, though all correlations are low.The association does not by itself disentangle whether errors stem from edit volume or from understanding how to edit.Source: §4.3 · p. 7

5 INTENSE UNDERSTANDING TASKS CAN MAKE CODING AGENTS BLEAT

The authors extend CABRA with a task that requires comparing two classes to extract shared logic while preserving behavioral differences. Scaling code size, transformations, and differences reveals where coding agents struggle to understand code.

5.1 A CODE UNDERSTANDING TASK: EXTRACTING LOGICALLY EQUIVALENT CODE

  • The task models each calculator class as a DAG of variable-defining lines, moving equivalent lines to a parent class and retaining divergent lines in child classes.For classes with ℓ lines and d behavioral differences, the classes start with ℓ lines each; edits scale modestly, but combinations of lines to compare can grow as Ω(ℓ^2) in the worst case.Source: §5.1 · pp. 7–8

5.2 UNDERSTANDING WHEN UNDERSTANDING IS CHALLENGING

  • The task varies class size, transformations to reason over, and the number of differing lines to probe understanding effort.To make equivalent code syntactically dissimilar, the authors rename variables, shuffle or decompose lines, distribute multiplication, and swap term order; agents receive one example and must preserve behavior while keeping as much code as possible in the parent class.Source: §5.2 · p. 8, §5.1 · p. 8
  • As lines of code per class increase, agent accuracy drops and UNDERSTAND tool use rises far more than EDIT tool use.The study samples 50 tasks at ℓ ∈ {5, 10, 25, 50, 100, 200}.Source: §5.2 · p. 8
  • Adding multi-line edits and distributive multiplication causes large accuracy drops for Gemini, while removing variable renaming boosts accuracy most.The authors suggest agents may rely on shallow naming to understand code.Source: §5.2 · p. 8
  • When agents must locate more differing lines, accuracy erodes; most failures arise from missing differences rather than breaking behavior during refactoring.Agents preserve behavior in over 90% of cases and often keep too many lines in child classes.Source: §5.2 · p. 8

6 RELATED WORK

Related work spans code-generation benchmarks, agent-failure analyses, and synthetic evaluations. The authors position CABRA as a controlled way to study coding-agent task complexity and identify code understanding as a difficulty axis.

Evaluating Code Models

  • Prior coding evaluations range from NL-to-code and algorithm tasks to issue resolution and app generation, with difficulty scaled through more edits, turns, or requirements.The authors identify understanding as a less studied axis for controlling difficulty.Source: §6 · p. 9

Agent Failures

  • Repository benchmarks and trace analyses vary in both codebase and task, making failure attribution difficult and correlational analyses potentially poor.The authors argue controlled CABRA-style tasks can probe agent errors and surface code-understanding weaknesses.Source: §6 · p. 9

Synthetic Tasks

  • Synthetic tasks define an ability and scale task size; prior examples include bAbI, Needle in a Haystack, LongCoT, RELIC, CruxEval, and Bug in the Code Stack.The authors distinguish CABRA from synthetic RL environments that use existing repositories or LLM-generated data, which impede full task control.Source: §6 · p. 9

6 RELATED WORK

  • Unlike prior work, the paper argues that tool use can make many LLM-hard tasks trivial for agents, and probes code understanding as a remaining weakness.Source: §6 · p. 9

7 CONCLUSION: RAISING THE BAA-R FOR CODING AGENT EVALUATIONS

The conclusion argues that edit count alone is a poor measure of coding-agent task difficulty, while code understanding is a key factor. It points to controlled evaluation and agent-understandable code as directions for further work.

7 CONCLUSION: RAISING THE BAA-R FOR CODING AGENT EVALUATIONS

  • Across CABRA’s initial task axes, editing effort scales insufficiently with task size, while understanding effort more often scales and better predicts SWE-bench Verified errors.The authors’ code-merging task also surfaces difficulty with multi-line decomposition and readability.Source: §7 · p. 9

Controlled Evaluation

  • CABRA-style controlled tasks can attribute agent errors; future work could study problem-solving and error recovery, then bring these weaknesses into more realistic SWE-bench-style evaluations.Source: §7 · p. 9

Agent-Understandable Code

  • Agents struggle to compare programs with renamed variables; future work could test whether indexes, memory, and readability practices help.The authors connect progress in understanding to maintainability and limiting context rot.Source: §7 · p. 9

A Theory of Agents

  • Determining what is agent-hard versus agent-easy will require formalizing work done to call tools and work tools can execute.Tasks that were once hard for LLMs may now be trivial with tools.Source: §7 · p. 10

AI USE STATEMENT

The authors report using generative AI primarily for code implementation and data visualization, with limited use in other research and writing tasks.

AI USE STATEMENT

  • Generative AI tools were used primarily for code implementation and data visualization, not for writing from scratch, figure generation, or high-level framing.Source: AI USE STATEMENT · p. 10
  • The authors report minimal AI assistance with revising, brainstorming, reviewing, and literature search.Source: AI USE STATEMENT · p. 10
  • Authors reviewed AI-assisted code with tests and visualization tools, and one author reviewed outputs before release; they take responsibility for the paper’s final content.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The paper argues that coding agents can struggle to understand code and calls for greater attention to evaluating this ability.

ETHICS STATEMENT

REPRODUCIBILITY STATEMENT

The reproducibility statement points to further model-setup details and identifies where the code, task data, results, and sandboxed Copilot traces are available.

REPRODUCIBILITY STATEMENT

A.1 DAG TASK SAMPLING PROCEDURE

The DAG-task sampling procedure sets graph height and levels, allocates vertices across levels, then samples incoming edges from earlier levels with a bias toward adjacent levels.

A.1 DAG TASK SAMPLING PROCEDURE

  • For N vertices, the procedure samples height h uniformly from 0.45 to 0.55 and sets the topological level count to round(N h).Source: §A.1 · p. 18
  • Vertices are assigned across levels using Poisson draws, with the mean equal to remaining vertices divided by remaining levels.Source: §A.1 · p. 18
  • Each non-root vertex samples an incoming edge from an earlier level using a geometric offset with parameter 0.9, favoring adjacent levels while allowing skips.Figure A.1 shows example DAGs, and Algorithm 1 summarizes the sampling procedure.Source: §A.1 · p. 18

A.2 EXAMPLE PROGRAMS

The examples cover Add Parameter programs and other task types, while the instruction-following task uses sampled rules whose number of unique instructions can be scaled.

A.2 EXAMPLE PROGRAMS

  • The examples include math, string, and array programs for Add Parameter, a Collatz-inspired check() function for runtime resolution, and two Calculator classes for code merging.The new line separating synonymous from divergent code is removed during evaluation.Source: §A.2 · p. 18
  • Instruction rules include adding, subtracting, multiplying, or dividing a number and assigning the result to a new variable.Source: §A.2 · p. 18
  • String rules add prefixes or suffixes, replace characters, or surround a value with characters to form a new variable.Source: §A.2 · p. 18
  • Other rules assert inequality or upper and lower bounds, print values with prefixes or suffixes, and add an exact comment.Source: §A.2 · p. 18
  • Sampling values for rand, prefix, and label scales the total number of unique instructions.Source: §A.2 · p. 18

A.3 MODEL IMPLEMENTATION DETAILS

This section documents prompt templates and code examples for call-graph transformations and related coding tasks, including instructions for edits, constraints, formatting, and tool-call classification. It also reports supplementary analyses and targeted tests of performance, tool use, code understanding, and classifier reliability.

A.3 MODEL IMPLEMENTATION DETAILS

  • Full codebase generation yields higher accuracy than diff-patch generation overall and for all tested models except GPT OSS 120B, so it is used in the remaining experiments.Models were retried until producing parseable output, typically within ten LLM tries; Copilot typically needed no more than three tries of up to ten minutes each.Source: §A.3 · p. 19, Figure A.2 · p. 25

A.4 CLASSIFIER IMPLEMENTATION

  • The tool-call classifier’s agreement with blinded author labels was 94% on initial CABRA tasks, 89% on SWE-Bench, and 94% on merge-codebases tasks.Summary and plan labels were rare and merged with other; invalid classifier labels were routed to a slower gpt-4.1 fallback.Source: §A.4 · p. 19

A.5 FURTHER CABRA ACCURACY ACROSS TASK TYPES AND MODELS

  • Across five task types and three program types, higher-complexity tasks tend to challenge LLMs while agent accuracy remains nearly perfect; controlling for GPT and Gemini models, LLM accuracy degrades more as N grows.Gemini-3.5 Flash and Gemini-3.1 Pro were run without tools on 15 programs (five math/string/array), at a total cost of $952 USD.Source: §A.5 · p. 20, Figure A.8 · p. 28

A.6 FURTHER TOOL CALL ANALYSIS

  • Function traversal and search elicit more ANALYZE tokens rather than EDIT, runtime resolution keeps tool-call tokens near-constant, and instruction following increases EDIT and often ANALYZE tokens.The three most common tool-call types are edit, bash, and view.Source: §A.6 · p. 20, Figure A.9 · p. 29

A.8 STUDYING THE IMPACT OF AGENT COLLABORATION ON OUR EVALUATIONS

  • In sandboxed runs, agent accuracy mostly degrades with task size only on the understanding-intensive merge-codebases task, while READ and ANALYZE use scale with task size.Filtering 11% of traces with possible cross-run access and disabling problematic tools in Docker did not change the conclusions.Source: §A.8 · pp. 21–22, Figure A.13 · p. 31, Figure A.14 · p. 31

A.9 EDITING TASKS WITHOUT MUCH UNDERSTANDING

  • As editing requirements grow, EDIT calls increase in both comment copying and variable renaming without large changes in UNDERSTAND calls, supporting distinct editing and understanding abilities.Accuracy stays consistent for comment copying, while LLM accuracy drops as variable-renaming n increases; EDIT call counts do not scale for any task, though smaller required edits can force more calls.Source: §A.9 · p. 23

A.10 ADDITIONAL RESULTS ON EXTRACTING EQUIVALENT CODE

  • Both LLMs and coding agents lose accuracy on larger merge-codebases tasks, and many LLMs never exceed 0.5 accuracy.Across ablations, 3/6 models struggle with multi-line differences, all models improve most when variables are not renamed, and agents struggle more as they must identify more one-line differences; accuracy rises when only one difference is possible.Source: §A.10 · p. 23

A.7 SWE-BENCH ANALYSIS WITH PER-MODEL BREAKDOWN

  • READ, SEARCH, and ANALYZE tool-call metrics predict SWE-Bench agent accuracy better than edit volume, beyond differences in agent baseline success.Per-model analyses also show agents do not only spend more tokens on EDIT operations.Source: §A.7 · pp. 20–21, Table A.4 · p. 29

A.11 PROMPTS

  • As comment-copying and variable-renaming tasks scale, accuracy stays stable; editing calls scale with task size, while reading and analysis calls grow modestly.This contrasts with instruction-following tasks, where analysis and reading demands scale more substantially.Source: Figure A.15 · p. 32, Figure A.16 · p. 32
  • The merge-codebase task injects differences into a variable-dependency DAG, and both LLMs and agents struggle to identify them.Most LLMs fail to exceed 0.5 accuracy, and agents struggle as the number of differences increases.Source: Figure A.17 · p. 33, Figure A.18 · p. 33, Figure A.19 · p. 34
  • Telling agents that the optimal extract-logic solution is exactly one line improves accuracy, indicating difficulty navigating the search space for minimal edits.The comparison is against the extract-logic task without that instruction.Source: Figure A.20 · p. 34
  • Gemini and Opus 4.7 struggle more with multi-line and distributive transformations, GPT-5 with swap order, and all models improve most when comments are not renamed.The results support the possibility that models over-rely on surface-level naming rather than behavior when understanding code.Source: Figure A.22 · p. 35, §A.11 · p. 35
  • The prompts cover removing functions unreachable from main(), recursively propagating a multiplier to terminal functions, and propagating a returned string up the call chain while preserving computations.They also specify constraints such as preserving function names and leaving main() unchanged.Source: §A.11 · pp. 36, 38
  • The cache prompt moves repeated slow() calls to the lowest common ancestor and passes the stored result downward; the DRY prompt computes a shared multiplier at the earliest shared downstream function.The DRY prompt explicitly forbids moving the computation to a caller or ancestor of the duplicated functions.Source: §A.11 · pp. 39–40
  • The merge-codebases prompt moves shared compute() logic into Calculate.compute(), leaving subclass-specific differences in helper methods while preserving behavior and class relationships.It allows multiple custom_compute_{idx}() helpers, but forbids unrelated classes, imports, or renaming the named classes or public methods.Source: §A.11 · pp. 41–42
  • The string-variation prompt propagates prefix through downstream calls, adds defaulted parameters, and prefixes results only at terminal functions, leaving upstream functions unchanged.It prohibits new functions, imports, or external variables and requires preserving function names and main().Source: §A.11 · pp. 42–44
  • The array-variation prompt recursively passes mult through downstream calls and applies multiplication only at terminal functions, without modifying upstream functions.As in the string variation, it requires preserving function names and main(), and disallows new functions, imports, or external variables.Source: §A.11 · pp. 44–45
  • The runtime-scaling prompt applies the mult-propagation transformation only to the branch that executes, preserving other branches unchanged.It otherwise specifies passing mult through downstream calls and multiplying only terminal computations.Source: §A.11 · p. 45
  • The parameter-addition prompt applies generated-parameter constraints in eligible functions, in a fixed order before the parameter is passed onward or used in a terminal computation.Constraints are excluded from main() and functions that do not take the named parameter; comments, assertions, prints, and temporary assignments have a specified order.Source: §A.11 · p. 47
  • The comment-copying prompt requires the same verbatim comment on a new line in every edited function.The supplied comment is a long, fixed word sequence.Source: §A.11 · p. 48
  • The variable-replacement prompt specifies five renames and requires consistent changes to assignments and uses without other code changes.The requested mappings are gg→bt, cn→gi, fv→g, d→ac, and dh→fi.Source: §A.11 · p. 48
  • The LLM-generation prompts request either a complete modified repository in a single solution block or changes in a unified-diff format.The full-response prompt also specifies a thought section before the solution; the diff prompt gives formatting instructions for hunks.Source: §A.11 · pp. 48–49
  • Separate prompts classify CABRA and SWE-Bench Verified tool calls into eight categories, including read, search, understand, edit, test, summary, plan, and other.The prompts distinguish reading or searching code from analysis beyond reading or searching, and instruct annotators to classify new files by their intent.Source: §A.11 · pp. 50–52
  • The numeric example composes helper functions into nested hypot aggregation and a min/hypot/subtraction expression.Source: §A.11 · p. 53
  • The snippets show helper functions that transform input strings through interleaving, alternating characters, character rotation, rotation by three, case swapping, and nested character-wise XOR operations.Source: §A.11 · p. 54
  • The NumPy array example uses maximum and minimum operations to combine helper outputs and applies elementwise arithmetic in a composed expression.Source: §A.11 · p. 55
  • A Collatz-inspired runtime function example defines an integer-checking function with a bounded step loop.Source: §A.11 · p. 55
  • The calculator example defines an abstract compute interface and two subclasses with separate compute methods and arithmetic expressions.Source: §A.11 · p. 56
  • A rename-variable example chains assignments through many short variable names before returning the final value.Source: §A.11 · p. 57
Loading 2610.10610v1…