PaperlayerView exact revision on arXiv

Source-linked AI summary

RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Code

Dhruv Gautam, Spandan Garg, Jinu Jang, Neel Sundaresan, Roshanak Zilouchian Moghaddam

arXiv:2503.07832v1cs.AIcs.CLcs.LGcs.SE

TL;DR

GitHub-based software-engineering evaluations can be noisy and skewed, while evidence about LM agents on multi-file refactoring remains limited. RefactorBench provides handcrafted multi-file tasks and the paper analyzes agent failures and tests state-aware interfaces. Agents struggle to compose simple actions, while state-aware conditioning improves RefactorBench performance.

  • Problem

    GitHub-based evaluations require filtration and validation that can skew task styles, while multi-file refactoring is described as a largely undocumented task for LM agents.

    Source: §1 · pp. 1–2

  • Method

    The authors introduce RefactorBench, evaluate LM agents and their failure modes, and modify a baseline agent to condition on representations of state.

    Source: §1 · p. 2, §6.1 · p. 8

  • Results

    43.9% relative increase over the instruction sets compared to baseline agents is reported for the state-update agent's overall RefactorBench performance.

    Source: §6.1 · p. 8

  • Takeaways & Limitations

    The authors conclude that diverse task evaluations are necessary for understanding and designing generalist language-agent systems.

    Source: §7 · p. 9

    The benchmark is limited to Python and highly used open-source repositories, and its intensive task construction leaves it with a limited number of instances.

    Source: §7.2 · p. 10

Abstract

from arXiv · show

Recent advances in language model (LM) agents and function calling have enabled autonomous, feedback-driven systems to solve problems across various digital domains. To better understand the unique limitations of LM agents, we introduce RefactorBench, a benchmark consisting of 100 large handcrafted multi-file refactoring tasks in popular open-source repositories. Solving tasks within RefactorBench requires thorough exploration of dependencies across multiple files and strong adherence to relevant instructions. Every task is defined by 3 natural language instructions of varying specificity and is mutually exclusive, allowing for the creation of longer combined tasks on the same repository. Baselines on RefactorBench reveal that current LM agents struggle with simple compositional tasks, solving only 22% of tasks with base instructions, in contrast to a human developer with short time constraints solving 87%. Through trajectory analysis, we identify various unique failure modes of LM agents, and further explore the failure mode of tracking past actions. By adapting a baseline agent to condition on representations of state, we achieve a 43.9% improvement in solving RefactorBench tasks. We further extend our state-aware approach to encompass entire digital environments and outline potential directions for future research. RefactorBench aims to support the study of LM agents by providing a set of real-world, multi-hop tasks within the realm of code.

1 INTRODUCTION

Existing software-engineering evaluations can be noisy and skewed by the filtering required for reliable GitHub-based benchmarks. RefactorBench targets multi-file refactoring with AST-based tests, and its evaluations expose compositional failures and improvements from state-aware interfaces.

1 INTRODUCTION

  • GitHub-based evaluations require strong filtration and validation, which the authors say skews task styles and motivates more diverse coding benchmarks.The paper presents this as a gap in evaluating LM agents on software-engineering tasks.Source: §1 · p. 1
  • RefactorBench evaluates multi-file refactoring in large codebases, using AST-based unit tests to check subtasks without requiring exact line matches.The benchmark also uses multiple instruction sets to study instruction-following with specified and unspecified objectives.Source: §1 · p. 2
  • Baseline agents solve at most 35% of tasks with the easiest instruction set, and state-update conditioning raises subtask completion rates by 71%.The authors identify tracking and reasoning about previous actions as a recurring failure mode.Source: §1 · p. 2
  • The paper contributes a multi-file refactoring benchmark, analyses of three failure modes, and state-aware interfaces that improve a modified baseline agent's reasoning.The benchmark tasks require reasoning based on previous actions.Source: §1 · p. 2

2 BACKGROUND

The background distinguishes RefactorBench from benchmarks of GitHub issues, function-level edits, and repository-level completion by focusing on multi-turn, multi-file refactoring. It frames stateful reasoning as decisions based on accumulated actions and observations in a dynamic, partially observable setting.

2.1 RELATED WORK

  • RefactorBench combines engineering tasks with multi-turn actions, using multi-file code dependencies to probe language-agent behaviors.Unlike function-level benchmarks, it evaluates broader actions than exact-line matching through generalist evaluations and AST testing.Source: §2.1 · p. 3
  • The paper focuses on concurrent reasoning and state-aware behavior, extending state-conditioning ideas from embodied control and small neural agents to real-world LM agents.This differs from work focused on long-term memory for learning skills after action sequences.Source: §2.1 · p. 3

2.2 DEFINITIONS

  • An LM agent receives an instruction, uses tools to take actions, and receives partial observations; the paper models its trajectory as actions and observations.The paper defines stateful reasoning as decisions based on accumulated actions and observations in a partially observable, dynamically changing state.Source: §2.2 · p. 3

3 REFACTORBENCH

RefactorBench comprises handcrafted, multi-file refactoring tasks in popular Python repositories, constructed with reference edits, AST tests, and instructions of varying specificity. Its design supports evaluation of task composition, instruction-following, and interpretable subtasks.

3 REFACTORBENCH

  • RefactorBench tasks require generating repository patches that implement specified refactoring rules.The paper describes an end-to-end construction process and highlights benchmark features.Source: §3 · p. 3

3.1 TASK CONSTRUCTION

  • The authors selected nine popular Python repositories with differing file structures and used Fowler et al. (2018) as a reference for diverse refactoring styles.They used an iterative LLM-assisted search for potential refactors, followed by manual filtering for multi-file changes.Source: §3.1 · p. 3
  • Experienced Python programmers handcrafted related edits for candidate refactors, while gpt-4o helped verify that each core refactor was tractable.The subsequent tests cover each verified core edit, parse file ASTs, and check broad code structure and syntax rather than exact matches.Source: §3.1 · p. 4
  • 100 multi-file tasks and corresponding tests were created across nine Python repositories, with success defined by passing all tests for a task.The paper reports a breakdown by repository and other task statistics.Source: §3.1 · p. 4

3.2 IMPORTANT FEATURES

  • All tasks edit 2 to 31 files, with 4 files edited on average in reference solutions, requiring reasoning across multiple files.The authors say this feature detracts from single-shot LLMs' ability to solve the tasks and requires feedback-based systems to reason across files.Source: §3.2 · p. 4
  • Three instruction sets vary in specificity: lazy instructions are unspecific, base instructions succinctly describe tasks, and descriptive instructions provide exhaustive guidance.The paper uses these variants to assess instruction-following and scale task difficulty; descriptive instructions serve as a theoretical upper bound on performance.Source: §3.2 · pp. 4–5
  • Each task has 2 to 27 subtests, averaging 6.51, designed to separate subtasks and make patch failures interpretable.The tractability checks target reasoning between function calls rather than open-ended generation of passing code changes.Source: §3.2 · p. 5

4 EXPERIMENTS

The experiments evaluate SWE-agent on RefactorBench, adjusting prompts to focus on refactoring and testing how instruction specificity affects performance. Baselines remain low relative to a time-limited human developer, and the authors flag that these baselines are outdated.

4 EXPERIMENTS

  • The authors note that these baselines are outdated compared with current SWE-agent implementations.They point readers to updated baselines in Section 7.3.Source: §4 · p. 5

4.1 PRELIMINARIES

  • The authors alter agent prompts to focus on refactoring, treating these prompt-adjusted baselines as an upper bound on performance.They report that task-specific prompting in existing systems steers agents toward GitHub bug fixes, such as creating reproduction scripts for renaming tasks.Source: §4.1 · p. 5

4.2 BASELINES

  • SWE-agent with gpt-4 solves 12%, 18%, and 27% of tasks with lazy, base, and descriptive instructions, respectively.The reported per-instance cost limit is $10.002; the table caption characterizes agents as sensitive to unspecified objectives and improved by file-location guidance.Source: §4.2 · p. 5, Table 2 · p. 6
  • SWE-agent solves no combined pseudotasks formed by joining descriptive prompts, despite solving the corresponding singular tasks.The authors sampled three solved instances from repositories with at least three solved tasks.Source: §4.2 · p. 6

5 ANALYSIS

Trajectory analysis identifies three recurring failure modes: difficulty locating and applying edits, interactions that require temporary errors, and context flooding that displaces task objectives. The authors classify about 58% of failed trajectories into these modes, with human reviewers agreeing with about 74% of classifications on held-out validation.

5 ANALYSIS

  • About 58% of failed trajectories match one of the three identified failure modes, and human reviewers agree with about 74% of classifications in held-out validation.The authors used GPT-4 with reference solution diffs and test outputs to classify unresolved trajectories after manual trajectory review.Source: §5 · p. 6
  • In about 44% of tests checking for a change, agents given descriptive instructions still fail to edit the target files; no baseline solves tasks requiring changes in more than six files.The authors hypothesize that the number of files acts as a proxy constraint on exploration and composing previous actions.Source: §5 · p. 6
  • 78.4% of trajectories error during a code-editing step, and some tasks require temporary errors before subsequent edits resolve them.The authors argue that automatically rejecting edits based on strict linting is impractical for scaling real-world agents.Source: §5 · p. 6
  • Context consumed by rejected commands or unexpected output can displace the initial objective and weaken trajectory structure.A common linting-error edit shows the model an average of about 1,466 tokens for code, error-handling prompts, and flagged errors.Source: §5 · p. 7

6 TOWARDS STATE-AWARE LANGUAGE AGENTS

This section investigates whether language agents lose track of state as actions accumulate and tests state-aware interfaces as a way to address that failure mode. It also extends state updates to digital environments and describes an initial example of concurrent edits.

6 TOWARDS STATE-AWARE LANGUAGE AGENTS

  • Failures in cumulative state construction linearly scale with the number of actions across 125 randomly initialized runs in a 15-category preference-tracking setup.The setup prompts a language model to update an increasingly long list of preference changes.Source: §6 · p. 7, Figure 4 · p. 7

6.1 STATE-AWARE INTERFACES

  • A state-aware code interface caches a representation of previous edits and supplies it before each function call as an externally computed state variable σ.The trajectory includes actions, observations, and state updates at each timestep.Source: §6.1 · p. 8
  • 43.9% relative increase over the instruction sets in overall RefactorBench performance was achieved by a state-aware SWE-agent, with an average 71% increase in subtask completion.The comparison is between the modified agent and baseline agents.Source: Figure 5 · p. 7, §6.1 · p. 8

6.2 STATE UPDATE POLICIES

  • A state update policy manages cumulative state information for an agent, and precomputation lets the language model skip reconstructing that state when generating its next function call.The policy is defined over the trajectory of actions, observations, and prior states.Source: §6.2 · p. 9
  • In an initial example, a state update policy propagated a concurrent user's function rename, but the paper did not evaluate conflicting objectives at larger scale.After receiving the update, the agent chose to view the edit for more context.Source: Figure 6 · p. 8, §6.2 · p. 9

7 DISCUSSION

The discussion presents RefactorBench as a way to study LM-agent failure modes and reports updated SWE-agent baselines. It also outlines future directions and limitations of the benchmark.

7 DISCUSSION

  • The authors present RefactorBench as isolating LM-agent failure modes and report that agents struggle to compose simple actions.They argue that diverse task evaluations are necessary for understanding and designing generalist language-agent systems.Source: §7 · p. 9

7.1 FUTURE DIRECTIONS

  • The authors identify state understanding over action sequences as a challenge and point to smarter context generalization, while noting the need for robust critics on open-ended tasks.They say repeated sampling is insufficient when real-world refactoring results are not immediately verifiable.Source: §7.1 · p. 9
  • The discussion flags inefficient code editing and full-file edit constraints, including token limits, inference speed, token cost, and context flooding, as interaction problems for future work.The authors suggest combining full-file rewrites with speculative decoding and trajectory truncation.Source: §7.1 · pp. 9–10

7.2 LIMITATIONS

  • RefactorBench is limited to Python and a small, costly set of popular repositories, and its results may reflect nondeterminism or overfitting to refactoring tasks.The authors recommend evaluating agents across multiple task styles.Source: §7.2 · p. 10

7.3 UPDATED BASELINES

  • 21% of base instances and 31% of descriptive instances were solved by the updated SWE-agent 1.0 with gpt-4o, at $1.69 per successful instance.The authors report that the speed and cost of task completion significantly decreased.Source: §7.3 · p. 10

A TASK CONSTRUCTION PROMPTS

The appendix provides task-construction prompts and examples, including an adapted SWE-agent prompt, a successful trajectory, and a task test file. The test file illustrates structural checks for gzip-related refactoring across several repository components.

A.1 PROMPT FOR LAZY INSTRUCTION

  • The lazy-instruction prompt asks GPT-4-turbo to turn the base task into a short, less-specific partial sentence without changing task behavior or mentioning tests.The prompt requests human-like prose and output of only the new instruction.Source: §A.1 · p. 17

A.2 PROMPT FOR DESCRIPTIVE INSTRUCTION

  • The descriptive-instruction prompt uses the base instruction and related test file to request a 2–3-sentence specification naming relevant filenames without prescribing implementation details.It asks the model to reason over AST tests, avoid mentioning tests in the output, and return only the new instruction.Source: §A.2 · p. 17

B.1 UPDATED SWE-AGENT SYSTEM PROMPT

  • The SWE-agent system prompt is adapted for refactoring by preventing bug-reproduction scripts and removing pytest and other testing functionality to reduce timeout issues in large codebases.The prompt also tells the agent to focus on completing the multi-file refactor rather than running repository tests.Source: §B.1 · pp. 18–19
  • The adapted prompt requires one command per turn, attention to indentation and file state, and searching for dependent imports when edits may affect them.It also advises agents to work in the task repository, inspect edits, and avoid oversized grep output.Source: §B.1 · pp. 18–19

C EXAMPLE CONDENSED SUCCESSFUL TRAJECTORY

  • A successful Flask trajectory renames the helper while preserving its public name through an alias, updates imports across source and test files, and submits the changes.The example identifies the task as renaming send_from_directory to send_from_directory_helper while retaining Flask usage as send_from_directory.Source: C EXAMPLE CONDENSED SUCCESSFUL… · p. 20

D EXAMPLE TEST FILE

  • The parameterize-gunzip test file checks for GunzipParams with data and max_size attributes, a gunzip(params: GunzipParams) -> bytes signature, and its use in SitemapSpider.The tests inspect Python ASTs in scrapy/utils/gz.py and scrapy/spiders/sitemap.py.Source: D EXAMPLE TEST FILE · pp. 21–22
  • The checks require SitemapSpider._get_sitemap_body to use gunzip with a GunzipParams argument.The test identifies the method and examines its gunzip call argument.Source: D EXAMPLE TEST FILE · p. 23
  • The sitemap module is checked for imports of GunzipParams, gunzip, and gzip_magic_number.The test scans the file's imports and asserts that each required name is present.Source: D EXAMPLE TEST FILE · p. 23
  • The gzip utility test file is checked for imports of gunzip and gzip_magic_number.The test scans imports in tests/test_utils_gz.py and asserts that both names are found.Source: D EXAMPLE TEST FILE · pp. 23–24
  • The gzip utility tests must call gunzip with a GunzipParams argument.The test walks the utility test file's syntax tree and checks the argument to gunzip.Source: D EXAMPLE TEST FILE · p. 24
  • The HTTP compression test file is checked for imports of GunzipParams and gunzip.The test scans imports in test_downloadermiddleware_httpcompression.py and asserts that both names are found.Source: D EXAMPLE TEST FILE · pp. 24–25
  • The HTTP compression middleware must call gunzip with a GunzipParams argument.The test inspects the middleware file's syntax tree and verifies the argument used in the gunzip call.Source: D EXAMPLE TEST FILE · p. 25

E EXAMPLE TEST OUTPUTS

The example test outputs show passing Django and Scrapy tests alongside failures in the Salt and FastAPI examples.

E EXAMPLE TEST OUTPUTS

F STATE RECONSTRUCTION EXPERIMENT

The experiment constructs synthetic preference-change trajectories and prompts GPT-4-turbo to reconstruct preferences from partial action histories. Its parser tolerates minor formatting errors but requires category separations.

F STATE RECONSTRUCTION EXPERIMENT

  • The experiment iteratively prompts with portions of JSON containing randomly initialized and changed preference lists.Generated data are written to complex_actions_states.json.Source: F STATE RECONSTRUCTION EXPERIMENT · pp. 28–29
  • Randomly generated preferences assign each product a value of Likes, Dislikes, or NA, and trajectories repeatedly change a randomly selected product preference.Each change is chosen to differ from the previous value.Source: F STATE RECONSTRUCTION EXPERIMENT · p. 28
  • The trajectory records actions and snapshots the preference state after each action, alongside the initial preferences.The generation code specifies 50 initial states, five trajectories per state, and 50 actions per trajectory.Source: F STATE RECONSTRUCTION EXPERIMENT · pp. 28–29
  • GPT-4-turbo is prompted across 125 random initializations using 0–50 generated actions; the parser allows minor format errors but requires category separations.Figure 7 gives an example prompt and expected output.Source: F STATE RECONSTRUCTION EXPERIMENT · p. 29, Figure 7 · p. 29

G SIMPLE SINGLE AGENT STATE-AWARE IMPLEMENTATION

The example state-aware interface appends a compact state summary to each prompt. It tracks prior edits and, in the paper’s results, incorporates related information from earlier observations.

G SIMPLE SINGLE AGENT STATE-AWARE IMPLEMENTATION

  • The state command tracks previous edit commands, while the results version also relays information from related prior observations.The displayed shell state includes working directory, open file, and recent edits.Source: G SIMPLE SINGLE AGENT STATE-AWARE… · p. 30
  • The shell implementation filters duplicate filename–line-number edit pairs before serializing recent edits as JSON.It builds the state object with working directory, open file, and recent edits.Source: G SIMPLE SINGLE AGENT STATE-AWARE… · p. 30
  • The state command output is appended at every step alongside the observation, current open file, and working directory.The prompt template labels the edit summary as Current State.Source: G SIMPLE SINGLE AGENT STATE-AWARE… · p. 30

H EXAMPLE STATE-AWARE OBJECTIVE UPDATE

A trajectory example shows the state-aware policy responding to an external repository edit while pursuing a local patch. A separate figure illustrates a failed function-call loop associated with context flooding and deprioritized objectives.

H EXAMPLE STATE-AWARE OBJECTIVE UPDATE

  • The trajectory shows the agent receiving an external edit to adapters.py alongside its own recent edits while working in test_utils.py.It then plans to update a function call and inspect the external edit.Source: H EXAMPLE STATE-AWARE OBJECTIVE UPDATE · p. 31

I CONTEXT FLOODING FIGURE

  • Figure 8 depicts a failed function-call loop as an example of context flooding and a deprioritized objective failure mode.Source: Figure 8 · p. 32

Related papers

Loading 2503.07832v1…