PaperlayerView exact revision on arXiv

Source-linked AI summary

SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering

John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, Ofir Press

arXiv:2405.15793v3cs.SEcs.AIcs.CLcs.HCcs.LG

TL;DR

Software-engineering tasks remain difficult for language-model agents using interfaces designed for humans or general-purpose tools. SWE-agent introduces an agent-computer interface tailored to LM capabilities, solving 12.47% of SWE-bench tasks versus 3.8% for the previous best non-interactive system.

  • Problem

    Whether interfaces tailored to language-model agents can improve performance on complex software-engineering tasks remains underexplored.

  • Method

    SWE-agent combines a language model with an agent-computer interface offering simple, guarded actions for viewing, searching, editing, and executing code.

  • Results

    12.47% of 2,294 SWE-bench tasks were solved, compared with 3.8% by the previous best non-interactive system.

  • Takeaways & Limitations

    Careful agent-computer interface design can substantially improve language-model agent performance without modifying the underlying model’s weights.

  • Takeaways & Limitations

    SWE-agent is evaluated exclusively on programmatic tasks such as software engineering and code generation, so transfer to other domains remains uncertain.

Abstract

from arXiv · show

Language model (LM) agents are increasingly being used to automate complicated tasks in digital environments. Just as humans benefit from powerful software applications, such as integrated development environments, for complex tasks like software engineering, we posit that LM agents represent a new category of end users with their own needs and abilities, and would benefit from specially-built interfaces to the software they use. We investigate how interface design affects the performance of language model agents. As a result of this exploration, we introduce SWE-agent: a system that facilitates LM agents to autonomously use computers to solve software engineering tasks. SWE-agent's custom agent-computer interface (ACI) significantly enhances an agent's ability to create and edit code files, navigate entire repositories, and execute tests and other programs. We evaluate SWE-agent on SWE-bench and HumanEvalFix, achieving state-of-the-art performance on both with a pass@1 rate of 12.5% and 87.7%, respectively, far exceeding the previous state-of-the-art achieved with non-interactive LMs. Finally, we provide insight on how the design of the ACI can impact agents' behavior and performance.

1 Introduction

SWE-agent introduces an agent-computer interface (ACI) that equips language-model agents with simple, guarded actions and concise feedback for software engineering tasks. Using GPT-4 Turbo, it solves 12.47% of SWE-bench tasks, outperforming the previous best non-interactive system’s 3.8% resolve rate.

  • Motivation: Direct interaction with a Linux shell can make LM agents unreliable because they struggle to edit file segments and receive feedback after invalid edits.These limitations motivate an abstraction layer between the LM agent and computer.
  • System: SWE-agent’s ACI improves agent performance by replacing Linux Shell’s granular action space with simple file-viewing, search, and editing actions.The interface uses guardrails to prevent common mistakes and provides agents with specific, concise feedback.
  • Results: 12.47% of 2,294 SWE-bench test tasks are solved by SWE-agent with GPT-4 Turbo, versus 3.8% by the previous best non-interactive retrieval-augmented system.The reported result establishes SWE-agent’s performance on challenging real-world software engineering problems.
  • Ablation: On 300 SWE-bench Lite instances, SWE-agent solves 10.7 percentage points more cases than a baseline agent using only the default Linux shell.The ablation evaluates how ACI design choices affect performance.
  • Contributions: The work introduces ACI as a design concept and open-sources SWE-agent for solving real-world software engineering tasks without modifying the underlying LM’s weights.The authors distinguish this focus from prior work separately studying tool use, prompting, and code execution in interactive settings.

2 The Agent-Computer Interface

The agent-computer interface (ACI) adapts digital-environment interactions to language models’ abilities and limitations, rather than assuming human-oriented interfaces will suffice. SWE-agent’s analyses and ablations show that ACI design principles recur across actions, feedback, and workflows and affect LM performance.

  • ACIs improve LM–computer interaction by accounting for limitations such as weak visual understanding, fixed memory and computation costs, and vulnerability to distracting context.Digital environments can be shaped through interfaces, but interfaces designed for humans or software may not match LM needs.
  • A well-designed ACI defines usable commands, communicates application state, and manages observation history so agents can act reliably without unnecessary context.The ACI formats prior commands and observations at each step to help agents understand state and track changes.
  • Effective ACI actions should be simple and easy to understand, while consolidating important operations into compact steps that enable meaningful progress.The design principles emphasize concise command documentation and minimizing multi-turn composition for operations such as navigation and editing.
  • Environment feedback should be informative but concise, reporting application state and recent action effects without unnecessary detail.For file editing, revised content provides useful information about the resulting environment state.
  • Guardrails reduce error propagation and accelerate recovery by detecting mistakes, such as syntax errors, so agents can recognize and correct them quickly.The paper specifically identifies automatic code syntax checking as an editing guardrail.
  • Analysis and ablation studies demonstrate that alternative ACIs affect LM performance and that these principles recur across actions, feedback, and workflows.The studies examine how interface choices shape agent behavior and performance across the system.

3 SWE-agent: Designing an ACI for Software Engineering

SWE-agent provides an agent-computer interface (ACI) that enables language models to search, navigate, edit, and execute code through coordinated commands and environment feedback. Its design combines repository tools, an integrated file viewer/editor, linting feedback, and context-management mechanisms.

  • ACI overview: SWE-agent’s ACI combines search/navigation, file viewing, file editing, and context management, with each step producing a thought, command, and execution feedback.The interface is built atop the Linux shell and follows a ReAct-style interaction loop [62].
  • Search and navigation: Search commands find_file, search_file, and search_dir summarize filename and string matches across files and directories.When a search exceeds the reporting limit, SWE-agent suggests writing a more specific query.
  • File editor: The edit command replaces a specified line range in the open file, after which the viewer automatically displays the updated content.Figure 3 shows the viewer and edit command integrated during a replacement of lines 404–407.
  • File viewer: The interactive file viewer shows at most 100 lines at once and supports scrolling or jumping to specific lines while displaying file-location metadata.Agents open files with open and navigate using scroll_down, scroll_up, and goto.
  • File editor: A code linter alerts agents to selected editing errors with surrounding snippets, discards invalid edits, and requests a corrected attempt.This feedback is designed to help agents notice format errors during file editing.
  • Context management: Context management uses prompts, documentation, demonstrations, error messages, and history processing to keep interactions concise and informative.Malformed generations trigger repeated error responses until a valid thought-and-action generation is received; older observations are collapsed while the last five remain expanded.

4 Experimental Setup

SWE-agent is evaluated primarily on SWE-bench, with supplementary testing on SWE-bench Lite and HumanEvalFix, using GPT-4 Turbo and Claude 3 Opus against non-interactive RAG and Shell-only baselines. Performance is measured by % Resolved or pass@1, alongside average inference cost under a per-instance budget, while configuration choices are tuned through qualitative analysis and hyperparameter sweeps.

  • Datasets: SWE-bench provides the primary evaluation, with 2,294 instances from 12 Python-package repositories; SWE-bench Lite contains 300 self-contained functional bug-fix instances for ablations and analysis.Main results use the full SWE-bench test set, whereas ablations and analysis use SWE-bench Lite unless otherwise specified.
  • Datasets: HumanEvalFix additionally tests SWE-agent’s basic code-editing abilities on a short-form code-debugging benchmark.HumanEvalFix is described as a supplementary benchmark alongside SWE-bench evaluations.
  • Models: All results, ablations, and analyses use GPT-4 Turbo and Claude 3 Opus, while additional models including Llama 3 and DeepSeek Coder performed subpar in the agent setting.The passage attributes this partly to limited context windows, noting Llama 3’s 8k context window and larger windows for the two primary models.
  • Baselines: SWE-agent is compared with non-interactive BM25-based RAG patch generation and a Shell-only baseline that resolves issues through Linux shell interaction.The RAG baseline retrieves relevant codebase files using the issue as the query, while Shell-only submits the final codebase state automatically.
  • Metrics: % Resolved or pass@1 measures the proportion of instances whose generated patches pass all tests, while $ Avg. Cost reports average API inference cost over successfully resolved instances under a $4 per-instance budget.Runs exceeding the budget submitted existing edits automatically.
  • Configuration search: The final ACI design was selected through qualitative analysis of hand-picked development examples, with remaining hyperparameters swept over window size, history processing, and decoding temperature.The hyperparameter sweep is reported in §B.1.

5 Results

SWE-agent with GPT-4 Turbo achieves the strongest reported performance on SWE-bench and HumanEvalFix. Results also show that ACI design affects localization, editing reliability, error recovery, and agent trajectories.

  • Main results: 88.3% pass@1 is achieved on HumanEvalFix, while average performance variance remains relatively low despite considerable per-instance variation.Figure 4 reports SWE-agent w/ GPT-4 Turbo pass@k performance across six SWE-bench Lite runs.
  • Main results: 12.47% (286/2,294) of full SWE-bench and 18.00% (54/300) of Lite instances are solved, outperforming RAG and Shell-only settings.SWE-agent is 8–13× more costly than RAG on Lite but improves resolved rate 6.7-fold, while increasing performance 64% relative to Shell-only with GPT-4 Turbo.
  • Agent behavior: Agents localize issues by starting with reproduction or broad directory-level searches, then narrowing to specific files and lines before iterative editing and execution.The most common early action sequence is create, edit, then python; later turns are dominated by edit-and-python loops with additional localization interspersed.
  • Agent behavior: 51.7% of 2,294 SWE-agent w/ GPT-4 Turbo trajectories contain at least one failed edit, and recovery becomes less likely as failures accumulate.Guardrails that prevent edits producing major errors are evaluated against No edit and edit w/o linting alternatives.
  • Agent behavior: Successful runs finish earlier and at lower cost than unsuccessful runs, while 52.0% of unresolved Lite instances involve incorrect or overly specific implementations.The failure categorization uses GPT-4o, whose judgments agree with authors’ labels on 87% of validation instances.

6 Related Work

Prior work evolved from simple code-generation benchmarks toward diverse, repository-level software-engineering evaluation and interactive language-agent settings. SWE-agent builds on this trajectory as, to the authors’ knowledge, the first language-agent system for end-to-end software engineering.

  • Code-generation benchmarks: Code-generation benchmarks measure natural-language-to-code synthesis, with later variants spanning programming languages, third-party libraries, code completion, and increased test coverage [32] [26].
  • Software-engineering evaluation: Repository-level software engineering provides a more diverse and challenging LM evaluation setting than simple code generation, requiring bug detection, cross-file reasoning, and other realistic subtasks [68].
  • Software-engineering evaluation: SWE-bench unifies program repair, bug localization, and testing across real GitHub issues from 12 repositories, using execution-based evaluation with human-written unit tests [55] [4] [58].
  • Interactive language agents: Interactive language agents have expanded across web navigation, computer control, and code generation, while coding agents support actions, tool construction, reasoning, offensive security, theorem proving, and clinical tasks [17, 42, 47, 54, 24, 33, 36, 41, 45, 61, 62, 71, 35, 53, 57, 16, 50, 63, 48, 59, 13, 51, 69, 39, 66, 67, 11, 37, 60, 44, 38, 43, 52].To the best of the authors’ knowledge, SWE-agent is the first work to explore language agents for end-to-end software engineering.

7 Discussion · Appendix · A SWE-agent Design

The paper introduces SWE-agent, an LM agent paired with a tailored agent-computer interface (ACI) for autonomously solving software engineering tasks. The discussion and appendix examine the value, design, implementation, analyses, and case studies of this ACI-centered system.

  • 7 Discussion: SWE-agent combines a language model with an ACI to autonomously solve software engineering tasks.The authors argue that ACIs tailored to language models can leverage their strengths and mitigate their weaknesses.
  • 7 Discussion: The discussion presents tailored ACI design as valuable for both practical agent performance and understanding language models and agents.The authors relate this potential synergy to the relationship between human-computer interaction and human capabilities.
  • Appendix: The appendix adds analyses, extended discussions of ACI design and model performance, and case studies of SWE-agent behavior on selected task instances.It also points readers to the project’s data, code, and leaderboard at swe-agent.com.
  • A SWE-agent Design: The design section discusses each SWE-agent component’s methodology, appearance, and implementation.It organizes the discussion around the interface components that enable fundamental software-engineering subtasks.
  • A SWE-agent Design: SWE-agent’s interface targets three core software-engineering subtasks: localization, editing, and testing.Localization identifies problematic files or lines, editing generates fixes, and testing reproduces issues or verifies fixes.
  • A SWE-agent Design: The system provides a file viewer, file editor, search and navigation system, and context management system to support those subtasks.These components are intended to help LM-based agents progress toward resolving issues in a codebase.
  • A SWE-agent Design: The appendix design discussion covers component breakdowns, technical design decisions, implementation challenges, and the configuration supporting the final system.These topics are organized into Sections A.1, A.2, and A.3.

A.1 ACI Design … B.3.1 Turns to Resolution

SWE-agent’s ACI combines specialized viewing, editing, search, navigation, configuration, and trajectory-management mechanisms to support language-model software engineering. Extended analyses characterize performance across hyperparameters, repositories, years, and solved-task trajectories, while identifying recovery limitations in later turns.

  • A.1 ACI Design: The File Viewer presents focused, line-numbered code regions and supports complementary navigation commands, helping agents inspect repositories without flooding context windows.It enables jumping around files and reduces repetitive parameter calculations compared with shell-only tools.
  • A.1 ACI Design: The File Editor grounds multi-line find-and-replace edits in viewer line numbers and reverts changes that introduce selected syntax or linting errors.This guardrail protects against cascading self-incurred errors but can force edits into a particular order.
  • A.1 ACI Design: Search and navigation provide directory- and file-level keyword localization while limiting results to 50 or fewer, trading concise observations against additional search calls.The module complements file viewing and editing, but capped results can require the model to issue another query.
  • A.2 Implementation: SWE-agent is organized into environment, agent, and logging modules, with trajectories and final patches produced for task analysis and SWE-bench evaluation.The environment adopts interactive coding infrastructure influenced by InterCode [59], while the agent renders the configurable ACI and supports closed/open models.
  • A.3 Configuration: The ACI is configured through prompt templates, command files, control flow, and environment variables in a single workflow that makes new interfaces easy to test.Configuration also specifies input/output parsing and per-turn history processing, while stateful environment variables and commands update interface state.
  • B.1 Hyperparameter Sweep: On 37 randomly sampled SWE-bench development instances, GPT-4 Turbo’s best sweep configuration achieved a 15.1% % Resolved rate with temperature 0.0, window length 100, and the last five observations.Claude 3 Opus had a three-way tie involving the aforementioned configuration.
  • B.2 Model Performance: SWE-agent solved a higher percentage of SWE-bench Lite problems across repositories than prior approaches and resolved problems in repositories that were previously nearly or completely unsolved.Across task years, there was no clear correlation between creation year and resolution rate; SWE-agent with GPT-4 led on 2021 instances, while RAG with GPT-4 and SWE-agent with Claude 3 Opus performed better on 2022 instances.
  • B.3.1 Turns to Resolution: 14.71 turns was the average for SWE-agent with GPT-4 on the full SWE-bench test set, with a 12-turn median and 75% of trajectories completed within 18 turns.On the Lite test split, SWE-agent with Claude 3 Opus averaged 12.71 turns; solved trajectories generally ended earlier and cost less, while agents struggled to recover from poor initial approaches later.

B.3.2 Walkthrough of Trajectory Phases

SWE-agent trajectories follow a recurring progression: early reproduction and localization, repeated edit–evaluation cycles with intermittent inspection, and submission concentrated in later turns. The trajectory length is bounded by a cost limit, with exit-cost termination appearing around Turns 30–40.

  • Initial reproduction, localization steps: Early trajectories are dominated by reproduction and localization, commonly beginning with the create, edit, python pattern or immediate file and line identification.Table 8 reports the create,edit,python pattern 156 times across the first three turns; Figures 16 and 17 show reproduction followed by localization dominating early operations.
  • Cycle of edit, then evaluate: From Turn 5 onward, agents generally alternate edits with python or pytest execution to test fixes and preserve existing behavior.Repeated execution feedback supports progressively better edits because an initial edit often fails to resolve the issue; Table 8 reflects this pattern from Turn 4 onward.
  • Cycle of edit, then evaluate: Localization remains interspersed in later turns as agents inspect additional regions or files, especially for multi-line or multi-file edits.Actions such as scroll_down, scroll_up, open, search_dir/file, and find_file remain steadily present from Turn 6 onward.
  • Concluding submission turns: Submissions occur consistently across turns but peak around Turn 10 and concentrate between Turns 10 and 20, declining thereafter.Most resolved task instances end with an intentional submit command, while later turns are used less effectively for submission.
  • Concluding submission turns: Exit-cost actions cluster sharply around Turns 30–40 because the imposed $4 cost limit usually exhausts the episode budget there.Trajectories with larger token-heavy observations terminate earlier, and increasing the per-task cost allowance would increase the maximum number of turns.

B.3.3 Breakdowns of Action Sequences … B.6 Patch Generations

The analyses characterize SWE-agent’s action patterns, failure modes, performance variability, and patch characteristics, showing that edit failures accumulate and that agent-generated patches are generally larger than human solutions. They also identify recurring multi-action problem-solving patterns, dominant unresolved-task failure categories, and relatively low average performance variance across runs.

  • B.3.3 Breakdowns of Action Sequences: 51.7% of 2,294 SWE-bench instances contain at least one failed edit, with a median of three failed edits per affected trajectory and up to 33.Failed edits are linting-error edit invocations; their rate is lower among resolved instances.
  • B.3.3 Breakdowns of Action Sequences: 57.2% of edits remain ultimately successful after one failed edit, down from 90.5% with no prior failed edit, and recovery worsens as failures accumulate.Across resolved instances, agents successfully recovered from 104 of 135 sequences containing at least one failed edit; successful recoveries followed 2.03 failed attempts on average versus 4.22 for unsuccessful recoveries.
  • B.3.3 Breakdowns of Action Sequences: Recurring action sequences correspond to higher-order operations such as issue reproduction, bug localization, and edit proposal or verification.Common patterns include create/find_file/search_dir at trajectory starts, submit or exit_cost at ends, repeated search or scrolling, and edit often following create, goto, pytest, or python.
  • B.4 Failure Modes: 52.0% of unresolved SWE-bench Lite instances fall into Incorrect or Overly Specific Implementation, while Failed Edit Recovery accounts for 23.4%.An automated categorization agreed with author labels on 87% of a 15-instance validation set.
  • B.5 Performance Variance and Pass@k Rate: SWE-agent’s average performance variance across six runs is relatively low, although which individual instances are resolved can change considerably; pass@k was also evaluated for k ∈{3, 6}.The evaluation uses pass@1 (% Resolved) unless otherwise stated, with six-run results summarized in Table 10.
  • B.6 Patch Generations: Patch analysis uses aggregated .patch files to compare model-generated code changes with original maintainer solutions using lines, hunks, and files as size measures.Statistics are reported as median/mean values across resolved and all instances, with outliers limited to the 90th percentile.
  • B.6 Patch Generations: Agent-generated patches contain more added or removed lines, hunks, and files than corresponding gold solutions for both resolved and all task instances.The increase is largely attributed to additional reproduction code; prior RAG baselines in Jimenez et al. typically produce smaller edits.
  • B.6 Patch Generations: Resolved-task patches are relatively smaller than the overall generated-patch distribution, while issues requiring multiple edits across a codebase remain challenging.This size trend is also consistent with RAG-based solutions.

B.7 HumanEvalFix Evaluation · B.8 Dataset Information

The HumanEvalFix evaluation targets code editing and debugging, using a 164-problem-per-language setup aligned with SWE-agent’s repository interaction workflow. The paper also documents SWE-bench and HumanEvalFix as permissively licensed evaluation datasets.

  • B.7 HumanEvalFix Evaluation: HumanEvalFix is chosen because code editing and debugging are difficult for language models and constitute a subtask of SWE-bench.Muennighoff et al. [32] report GPT-4 scores of 78.3% on HumanEval versus 47.8% on HumanEvalFix.
  • B.7 HumanEvalFix Evaluation: HumanEvalFix contains 164 problems per language, with SWE-agent editing buggy code and verifying fixes from an initialized single-file directory.Example tests are provided when available, and the SWE-bench configuration is reused with a language-specific demonstration.
  • B.7 HumanEvalFix Evaluation: The HumanEvalFix evaluation follows documentation from Muennighoff et al. [32] to define its task setup and procedure.The setup asks SWE-agent to edit code and verify its fixes.
  • B.7 HumanEvalFix Evaluation: Figure 25 shows the distribution of trajectory turn counts for solved HumanEvalFix task instances.The figure is presented as analogous to Figure 14.
  • B.8 Dataset Information: Both evaluation datasets are released under permissive software licenses that allow evaluation use and proprietary-system deployment.This licensing statement applies to SWE-bench and HumanEvalFix [32].
  • B.8 Dataset Information: Table 12 describes the SWE-bench and HumanEvalFix [32] datasets used to evaluate SWE-agent.The supplied passage identifies the table’s scope but does not provide its dataset-specific cell values.

B.9 Miscellaneous · C Prompts

SWE-agent localizes files more accurately than BM25, and its episode outcomes show that intentional submission is strongly associated with resolution. Its prompt workflow combines demonstrations, task-specific guidance, explicit execution feedback, context compression, and malformed-response recovery to support reliable interaction.

  • B.9 Miscellaneous: 59.05% F1 for SWE-agent w/ GPT-4 Turbo exceeds 45.47% for BM25 w/ Claude 3 Opus in identifying files to edit.F1 compares the agent’s predicted edited or removed files with the gold patch; the interactive setting also outperforms RAG baselines in file identification.
  • B.9 Miscellaneous: For resolved task instances, the large majority of SWE-agent episodes with either model ended through submit, indicating successful edits were often produced and verified within the turn budget.Across all task instances, GPT-4 Turbo mostly ended with submit, whereas slightly under half of Claude 3 Opus trajectories were submitted and a slight majority were auto-submitted at the cost limit.
  • B.9 Miscellaneous: 14.3% of full SWE-bench instances ending with submit were resolved, versus 3.1% ending with exit_cost for SWE-agent w/ GPT-4 Turbo.Table 13 reports four episode endings: Submit, Exit Cost (Submit), Exit Cost (No Submit), and Early Exit.
  • C Prompts: Templates improve problem solving by warning against common mistakes, demonstrating command usage, and giving task-specific tips about formatting, indentation, and execution pitfalls.Demonstrations slightly improve command understanding and reduce errant responses, while their primary intended role is teaching properly formatted commands rather than domain-specific strategies.
  • C Prompts: The prompt workflow begins with system, demonstration, and issue templates, then uses turn-specific prompts based on response formatting and whether actions produce standard output.Figure 26 depicts the ordering of these templates throughout a task episode.
  • C Prompts: The system template permanently defines the command-line environment, available commands, response format, one thought/action pair per turn, and proper indentation requirements.The instance template reiterates key constraints and introduces the issue statement and recommended approaches.
  • C Prompts: Successful actions receive their output as the next-step prompt, while silent actions are explicitly reported as successful to prevent unnecessary follow-up commands.Old observations are collapsed to reduce processed tokens and outdated or duplicate context while preserving interaction structure and order.
  • C Prompts: Malformed responses trigger corrective error messages, remove the malformed action-response pair when recovery succeeds, and terminate the episode after three consecutive malformed responses.Well-formatted responses thereby become additional in-context demonstrations that promote continued correct interaction.

D Qualitative Analysis · Analysis for psf/requests-2317

The qualitative analysis examines two successful and unsuccessful SWE-bench Lite instances using GPT-4 Turbo trajectories. One analyzed issue involved Python-version-specific mishandling of byte-valued HTTP methods in requests.

  • D Qualitative Analysis: The analysis selected two successful and unsuccessful SWE-bench Lite task instances.Both trajectories came from the default configuration using GPT-4 Turbo.
  • Analysis for psf/requests-2317: The analyzed task concerned resolving a defect in the requests library.
  • Analysis for psf/requests-2317: The method variable was a bytes object such as b’GET’.
  • Analysis for psf/requests-2317: builtin_str incorrectly converted the byte value into the literal string "b’GET’".
  • Analysis for psf/requests-2317: The malformed method string caused requests to raise a 404 Not Found response.
  • Analysis for psf/requests-2317: The issue appeared in Python 3.4 but not Python 2.6.The passage attributes this difference to version-specific handling of bytes.

Interaction Summary · Analysis for pylint-dev/pylint-5859

The agent fixed binary HTTP-method handling in requests, verified the change with unit tests and a successful request, but did not use the repository’s existing utility function. The related pylint issue concerns punctuation-only -notes tags being ignored despite being expected to produce W0511 warnings.

  • Interaction Summary: The agent located requests/sessions.py rather than the build artifact, searched for builtin_str, and identified the relevant conversion at line 428.This repository-navigation sequence narrowed the change to the production source file and the method-conversion call.
  • Interaction Summary: Unit tests pass, and a reproduction request using binary method b'GET' returns status code 200 without errors.The agent’s fix successfully handles binary method strings as valid HTTP methods.
  • Interaction Summary: Compared with the gold patch’s to_native_string utility, the agent’s equivalent fix resolves the issue but may be less maintainable or portable.The gold patch replaces builtin_str(method) with to_native_string(method), an existing utility in requests/utils.py#L655.
  • Interaction Summary: After validating the reproduction, the agent removed the temporary script and submitted the sessions.py modification.The temporary reproduce_issue.py script exercised requests.request(method, url) with method = b'GET'.
  • Interaction Summary: The agent decodes byte-valued methods with ASCII before applying builtin_str, preventing the binary prefix from becoming part of the HTTP method.The change adds an isinstance(method, bytes) check followed by method.decode('ascii').
  • Analysis for pylint-dev/pylint-5859: The pylint issue is that -notes="YES,???" recognizes YES but ignores the punctuation-only ??? tag, although it should report the corresponding W0511 fixme warning.The expected behavior is for punctuation-only note tags to be recognized normally.

Interaction Summary · Analysis for sympy/sympy-21614

The agent reproduced Pylint’s failure to report punctuation-only note tags, traced it to a word-boundary regular expression, and replaced that boundary so both standard and punctuation-only tags were detected. The targeted fix was verified by rerunning Pylint successfully.

  • Interaction Summary: Both “YES” and “???” tags were reported after the fix, resolving the punctuation-only note-tag detection issue.The rerun produced W0511 warnings for both tags.
  • Interaction Summary: The agent validated the change by creating a test.py fixture, running Pylint with the notes option, and removing the temporary file afterward.The fixture contained the YES and ??? comments used to reproduce and verify the issue.
  • Interaction Summary: The initial reproduction showed that Pylint reported the “YES” tag but ignored the punctuation-only “???” tag.The test file contained both note comments, and the first run emitted only the YES warning.
  • Interaction Summary: The agent located the relevant implementation in pylint/checkers/misc.py after searching repository matches for notes.The file contained the miscellaneous checker and the notes option’s regex construction.
  • Interaction Summary: The failure was traced to the regex matching configured note tags followed by a word boundary, which does not accommodate punctuation-only tags.The pattern is built from escaped configured notes and an optional notes-rgx expression.
  • Interaction Summary: The agent replaced the regex word boundary \b with (?=\W|$) so note tags could end before a non-word character or the string end.This change targets tags composed entirely of punctuation.

Interaction Summary · Analysis for django/django-14411

The agent reproduced the SymPy Derivative kind issue, then searched the repository to locate the relevant implementation. Its trajectory exposed inefficient search behavior and an incomplete investigation of the kind property.

  • Interaction Summary: The agent’s repository navigation was inefficient because it used speculative filename searches before trying the direct pattern “class Derivative.”The evaluation specifically identifies inefficient searches and recommends more commonsensical search patterns.
  • Analysis for django/django-14411: The evaluation indicates that fixing the issue required understanding that the updated code overrides an inherited property kind from sympy.basic.This explains why the agent’s investigation needed deeper understanding of the codebase’s inheritance structure.
  • Interaction Summary: The agent followed a reproduce–search–inspect workflow, beginning with the issue’s example code and then seeking the Derivative implementation.Its script constructed a MatrixSymbol, differentiated it, and printed the resulting kind before repository exploration.
  • Interaction Summary: The reproduction confirmed that Derivative(A, x).kind returned UndefinedKind instead of the expected MatrixKind(NumberKind).The agent created and ran a script using a MatrixSymbol to demonstrate the incorrect behavior.
  • Interaction Summary: After opening the large function.py file, the agent repeatedly scrolled without finding where Derivative’s kind property was implemented or overridden.The file contained 3389 lines, and multiple subsequent scrolls still failed to identify the relevant implementation.
  • Interaction Summary: The agent eventually located the Derivative class at line 1053 of sympy/core/function.py after several unsuccessful searches.It first searched for derivative.py, then searched broadly for Derivative, examined core.py, and found function.py.

Interaction Summary … E.2 Reproducibility

The agent traced the Django widget and attempted a custom field-level fix, but failed because the issue required overriding the Widget’s id_for_label property. SWE-agent also highlights security risks from executing generated code, malicious evaluation infrastructure, and real-world deployment.

  • Interaction Summary: The agent searched repository references, inspected helpers.py and forms.py, and traced ReadOnlyPasswordHashWidget through its template and UserChangeForm usage.The initial file search found no direct match, while a broader search located nine references across implementation and test files.
  • Interaction Summary: The agent submitted without testing because it assumed a Django server could not be run in the environment.Its proposed edit also introduced syntax errors during the interaction.
  • Interaction Summary: The attempted fix failed because the agent edited ReadOnlyPasswordHashField.label_tag instead of overriding the Widget’s id_for_label property.The agent correctly recognized that label rendering required customization, but submitted a solution that did not resolve the issue.
  • E.1 Ethics & Broader Impacts: SWE-agent identifies security risks from executing generated code locally, including unintended deletion of digital assets, and mitigates them with ephemeral containers for inference and evaluation.The risk arises because software-engineering tasks contain diverse issue descriptions and generated code may have harmful side effects.
  • E.1 Ethics & Broader Impacts: Unofficial evaluation datasets or infrastructure could inject malicious instructions, such as directing agents to build keylogging functionality and store it in an attacker-controlled location.The paper warns that community-built SWE-agent or benchmark harnesses could contain compromised task instances.
  • E.1 Ethics & Broader Impacts: Real-world deployment could place software-engineering agents in pipelines that produce malicious code, especially as SWE-agent’s strong SWE-bench performance signals increasingly capable future systems.The paper connects this concern to prior prototypes of agents performing offensive security measures.

E.3 Limitations & Future Work

The authors identify three directions for future work: expanding SWE-agent’s toolkit, automating ACI development, and testing whether ACI principles transfer beyond software engineering. These directions could improve task performance while broadening understanding of ACI principles.

  • Future Work: Adding tools such as web browsing, static analysis, fault localization, and fuzzing-based test generation could improve SWE-agent’s % Resolved performance.The final configuration uses a small toolkit, though it is described as highly effective.
  • Future Work: Automating parts of the manually designed ACI process could accelerate SWE-agent development and reveal broader principles for agentic software engineering.The current process relies on observations of recurring behavior within individual or multiple trajectories.
  • Future Work: Applying ACI principles beyond software engineering could improve digital-work performance and expand the set of principles across education, data analysis, and enterprise workflows.The paper frames SWE-agent as a software-engineering case study that can motivate ACI research in other domains.
Loading 2405.15793v3…