SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering

1 Introduction

• Inspired by human-interface research, the paper investigates whether interfaces tailored to LM agents can support complex software engineering.Earlier work demonstrated code generation with execution feedback, while programming agents typically used existing applications such as shells and Python interpreters.Source: §1 · p. 1
• SWE-agent’s ACI mediates computer interaction through simple file-viewing, searching, and editing actions, with guardrails and specific, concise feedback after every command.The Linux-shell environment lacks simple commands for editing small file segments and feedback for invalid edits, hampering reliable agent actions.Source: Figure 1 · p. 1, §1 · p. 2
• 12.47% of 2,294 SWE-bench test tasks were solved by GPT-4 Turbo SWE-agent, versus 3.8% for the previous best non-interactive, retrieval-augmented system.SWE-agent with Claude 3 Opus solves 10.5% of benchmark tasks, demonstrating portability beyond GPT-4 Turbo, for which the ACI was developed.Source: §1 · p. 2
• 10.7 percentage points more SWE-bench Lite instances were solved by SWE-agent than by the default Linux-shell agent in a 300-instance ACI ablation study.The study analyzes the contribution of ACI design choices.Source: §1 · p. 2
• The ACI framework unifies tool use, prompting, and code execution to improve agent performance without modifying LM weights; SWE-agent is evaluated and open-sourced.The contribution treats these interactive components as a unified interface-design problem rather than exploring them independently.Source: §1 · p. 2

2 The Agent-Computer Interface

• The agent-computer interface (ACI) treats LM agents as distinct computer users who benefit from interfaces tailored to their abilities and limitations.The motivation includes limited ability to operate visually rich GUIs and the memory and computation costs of all content. Distracting context can also harm performance.Source: §2 · pp. 2–3, Figure 2 · p. 2
• The ACI specifies available commands and communicates state through informative, concise feedback while managing history to avoid unnecessary context.It formats previous commands and observations together with high-level instructions into each LM input; feedback should explain current state and recent action effects.Source: §2 · p. 3
• With the LM fixed, ACI development combines manual inspection of agent behavior with grid search over configurations on a development set.Actions, documentation, and feedback are shaped around model abilities. Section 5 analyses and ablations examine how alternative ACIs affect performance and how these principles recur across workflows.Source: §2 · p. 3
• Actions should be simple and compact, using few options, concise documentation, and consolidated operations to support efficient, reliable agent use.Important operations such as navigation and editing should require few actions rather than multiple turns of composing small commands.Source: §2 · p. 3
• Guardrails such as automatic syntax checking help agents recognize and correct mistakes, limiting error propagation and accelerating recovery.The section identifies editing and searching as activities where agents can make mistakes and struggle to recover.Source: §2 · p. 3

Agent-computer interface

• SWE-agent generates thoughts and commands, incorporates execution feedback (ReAct; Yao et al.), and combines specialized ACI tools with access to Linux utilities.The interface comprises search/navigation, a file viewer, a file editor, and context management.Source: §3 · p. 3

Search and navigation

• 50 results per query is the search maximum; find_file, search_file, and search_dir summarize matches, withholding oversized results and requesting more specific queries.find_file searches repository filenames; search_file and search_dir locate strings within a file or files in a subdirectory.Source: §3 · pp. 3–4

File viewer

• 100 lines is the file viewer’s maximum window, with scrolling, direct line navigation, numbered lines, and file-location metadata supporting code localization.The open command loads a file; scroll_up, scroll_down, and goto move the window. Metadata includes the full path, total lines, and omitted-line counts.Source: §3 · p. 4

File editor

• The editor replaces specified line ranges, immediately displays updated content, and rejects invalid edits using integrated linting feedback.Edits specify a start line, end line, and replacement text. Selected lint errors appear with before/after snippets, and rejected edits prompt another attempt.Source: §3 · p. 4, Figure 3 · p. 4

Context management

• Context management combines command-use guidance, corrective error messages, and compressed observation history to keep interactions concise and informative.Instructions and demonstrations guide command use; malformed outputs trigger retries, after which past error messages except the first are omitted. Observations preceding the last 5 collapse into single lines, while successful commands producing no output receive an explicit confirmation.Source: §3 · pp. 3–4

Datasets

• 2,294 tasks from 12 Python repositories constitute SWE-bench, with the full test set used for main agent results.Ablations and analyses generally use SWE-bench Lite’s 300 self-contained functional bug fixes. HumanEvalFix [32] additionally tests basic code-editing abilities through short-form debugging.Source: §4 · pp. 4–5

Models

• Experiments use GPT-4 Turbo (gpt-4-1106-preview) and Claude 3 Opus (claude-3-opus-20240229).Their respective 128k- and 200k-token context windows accommodate prompts and several interaction turns. Additional models, including Llama 3 and DeepSeek Coder, performed poorly as agents; limited context, such as Llama 3’s 8k window, constrained many models.Source: §4 · p. 5

Baselines

• Baselines comprise non-interactive BM25-based RAG from Jimenez et al. and interactive Linux Shell-only adapted from Yang et al..RAG retrieves files using the issue as its query and directly generates a patch. Shell-only, like SWE-agent, derives its prediction from the codebase’s final state after interaction.Source: §4 · p. 5

Metrics

• % Resolved or pass@1 measures the proportion of instances where all tests pass after applying the generated patch.$ Avg. Cost reports average API inference cost over successfully resolved instances. Each instance has a $4 budget; exceeding it triggers automatic submission of existing edits.Source: §4 · p. 5

Configuration search

• The final ACI design emerged from qualitative behavioral analysis on hand-picked SWE-bench development examples, followed by hyperparameter sweeps.The sweeps covered window size, history processing, and decoding temperature.Source: §4 · p. 5

5 Results

• 12.47% resolved on full SWE-bench [20] and 18.00% on Lite make SWE-agent with GPT-4 Turbo the best-performing evaluated system overall.It solves 286/2,294 full-test instances and 54/300 Lite instances. On Lite, it costs 8–13x more than RAG but improves % Resolved 6.7-fold, with a 64% relative improvement over Shell-only using GPT-4 Turbo.Source: §5 · p. 5, Table 1 · p. 6
• 88.3% pass@1 on HumanEvalFix [32] accompanies relatively low average performance variance, despite considerable changes in which individual instances are resolved.Success rate is also uncorrelated with issue age in the analysis controlling for possible test pollution.Source: §5 · p. 5, Table 2 · p. 6

5.1 Analysis of ACI Design

• 12.0% performance with iterative search versus 15.7% with no search reflects agents exhaustively inspecting matches until their cost budget or context window is exhausted.Summarized search instead presents an exhaustive results list and guidance for refining underspecified queries. Ordinary shell navigation can waste actions, while grep or find can return excessive irrelevant output.Source: §5.1 · pp. 5–6, Figure 5 · p. 7
• 10.3% performance without the file editor and 15.0% without linting demonstrate the importance of efficient editing interfaces and error-prevention guardrails.These GPT-4 ablations test an editor that consolidates multiline changes with immediate feedback and rejects modifications introducing major errors. Viewer size also matters: displaying 30 lines yields 14.3%, while displaying the entire file yields 12.7%.Source: §5.1 · pp. 5–7

5.2 Analysis of Agent Behavior

• Agents begin with reproduction or localization, narrow searches from directories to specific lines, and predominantly alternate editing with execution from turn 5 onward.Reproduction commonly follows create, edit, and python actions. Later execution feedback prompts additional localization, while resolved tasks correlate with earlier submissions.Source: §5.2 · pp. 7–8
• 51.7% of SWE-agent with GPT-4 Turbo trajectories contain failed edits, and recovery becomes less likely as consecutive editing failures accumulate.Across 2,294 instances, 1,185 trajectories contain at least one failed edit. An editing attempt has a 90.5% chance of eventual success, falling to 57.2% after one failed edit.Source: §5.2 · p. 8
• $1.21 median cost and 12 steps characterize successful SWE-agent with GPT-4 runs, versus $2.52 mean cost and 21 steps for unsuccessful runs.The comparison excludes budget-exhausting runs; 93.0% of resolved instances finish before exhausting their budget, compared with 69.0% overall. The authors therefore suspect that larger budgets or token limits would not substantially improve performance.Source: §5.2 · p. 8
• 52.0% of unresolved Lite instances involve incorrect or overly specific implementations, while cascading failed edits account for another 23.4%.GPT-4o automatically classified 248 unresolved SWE-agent with GPT-4 Turbo trajectories into nine manually defined categories. Its judgments agreed with the authors on 87% of a hand-labeled validation set.Source: §5.2 · p. 8, Figure 8 · p. 8

6.1 Software Engineering Benchmarks

• 94.4% of HumanEval is solved by the top method, illustrating the emerging saturation of largely self-contained code generation benchmarks.Benchmark extensions cover programming languages, third-party libraries, completion tasks [32], test coverage [26], edit scope [8], and contamination robustness. Problems and solutions remain brief, with handwritten or fuzz-generated tests.Source: §6.1 · p. 9
• Repository-level editing challenges models to locate errant code, identify cross-file relationships, and understand codebase-specific symbols and conventions.Software engineering offers a challenging evaluation setting beyond simple code generation and costly human-created problems. Earlier benchmarks often isolated tasks from their surrounding codebases.Source: §6.1 · p. 9
• SWE-bench unifies program repair, bug localization [4], and testing within a task formulation reflecting practical software engineering.Instances are automatically collected from real GitHub issues across 12 repositories. Evaluation executes human-written unit tests.Source: §6.1 · p. 9

6.2 Language Models as Agents

• Interactive LM agents increasingly replace zero/few-shot generation across web navigation, computer control, and code generation.The passage attributes this shift to stronger models, harder benchmarks, and practical applications, citing web navigation, computer control, and code generation.Source: §6.2 · p. 9
• SWE-agent is, to the authors’ knowledge, the first work exploring language agents for end-to-end software engineering.Prior agents combine interaction with code for actions, tool construction, and reasoning. Applications include offensive security, theorem proving, and clinical tasks.Source: §6.2 · p. 9

7 Discussion

• SWE-agent demonstrates the value of interfaces tailored to leverage language models’ strengths and mitigate their weaknesses when autonomously solving software engineering tasks.The agent combines a language model with an agent-computer interface.Source: §7 · p. 9
• ACI research could illuminate differences between humans and LMs [12] [31], paralleling the synergy between HCI and psychology.The authors frame interaction design as systematic behavioral experimentation that could contribute to a comparative understanding of human and artificial intelligence.Source: §7 · p. 9

Appendix

• The appendix provides additional ACI and benchmark analyses, broader discussions, and detailed case studies of SWE-agent behavior on selected tasks.Data, code, and the leaderboard are available at swe-agent.com.Source: Appendix · p. 16

A SWE-agent Design

• The design appendix examines component methodology, appearance, implementation, technical challenges, and configuration for interface extensibility and customization.Section A.2 addresses technical decisions and challenges; Section A.3 explains configuration and support for altering the interface.Source: A SWE-agent Design · p. 16
• The interface supports three fundamental subtasks: localizing problematic files or lines, editing fixes, and testing issue reproduction or fix correctness.Testing can involve writing new scripts or modifying existing test files.Source: A SWE-agent Design · p. 16
• SWE-agent provides a file viewer, file editor, search/navigation system, and context management system to support efficient codebase issue resolution.Section A.1 breaks down these interface components.Source: A SWE-agent Design · p. 16

A.1 ACI Design

• The File Viewer shows numbered windows and relative position indicators, enabling fine-grained localization while complementary navigation commands reduce repetitive arithmetic.Agents can inspect relationships between entities within a file and move through its contents without repeatedly recalculating viewing ranges.Source: §A.1 · pp. 16–18
• The editor supports single-action multiline changes and reverts edits introducing lint errors, guarding against cascading mistakes while constraining edit order.Assuming initially syntactically sound code, rejected edits display the error, proposed content, and original content; accepted edits redisplay the affected region. Guardrails can require removing references before deleting an argument.Source: §A.1 · pp. 18–20
• 50 or fewer results keep search observations concise, while standardized directory- and file-level commands support localization but can require costly query reformulation.search_dir and find_file support directory-level searches, while search_file supports fine-grained localization within files.Source: §A.1 · p. 20

A.2 Implementation

• SWE-agent separates containerized execution, configurable agent interfaces, and logging into environment, agent, and logging modules.The environment adopts InterCode’s interactive coding formulation and Docker for reproducible, safe execution; the agent supports API-based and local models. Logging stores trajectories and final patches for SWE-bench evaluation.Source: §A.2 · pp. 20–21

A.3 Configuration

• Configuration files define prompts, commands, control flow, and environment variables, making interface refinement the main route to improved agent performance.Commands update shared environment state, configurable parsers enforce response formats, and history processors control message history each turn.Source: §A.3 · pp. 21–22
• Custom Python or Bash commands can share global state and use third-party libraries; per-turn context managers construct the interaction history supplied for each model response.Command implementations are unconstrained, require a signature and docstring, and expose results through standard output rather than return values.Source: §A.3 · pp. 21, 23, 24

B.1 Hyperparameter Sweep

• 15.1% Resolved was GPT-4 Turbo’s best configuration score in the 37-instance development sweep, using temperature 0.0, window length 100, and Last-5 history.Table 5 reports mean scores across five samples. Claude 3 Opus tied across 0.0/100/Last-5, 0.2/100/Last-5, and 0.2/200/Full; both models used 0.0/100/Last-5 for full and Lite test inference.Source: §B.1 · p. 25, Table 5 · p. 25

B.2 Model Performance

• SWE-agent outperforms prior approaches across SWE-bench Lite repositories, including repositories nearly or entirely unsolved by earlier retrieval-augmented baselines.Examples include matplotlib and sympy/sympy. A separate temporal analysis groups % Resolved by task year.Source: §B.2 · p. 25
• SWE-agent w/ GPT-4 solves its highest task percentage from 2021; RAG w/ GPT-4 and SWE-agent w/ Claude 3 Opus perform better on 2022 tasks.These comparisons concern task-instance creation years.Source: §B.2 · p. 26

B.3 Trajectory Analysis

• Trajectory characterizations focus on successfully resolved tasks, using SWE-agent w/ GPT-4 Turbo unless otherwise specified.Source: §B.3 · p. 26

B.3.1 Turns to Resolution

• 10 to 20 initial turns typically reveal the problem-solving approach; when it fails, agents struggle to recover using later turns.The authors propose improving the model, the ACI, or both to strengthen error recovery.Source: §B.3.1 · pp. 26–27
• 14.71 turns on average sufficed for successful SWE-agent w/ GPT-4 trajectories on full SWE-bench; the median was 12, with 75% completed within 18 turns.Successful SWE-agent w/ Claude 3 Opus trajectories on SWE-bench Lite averaged 12.71 turns (median 13), with 75% completed within 15 turns.Source: §B.3.1 · p. 26, Figure 14 · p. 27
• Resolved SWE-agent w/ GPT-4 Turbo trajectories have earlier means in step-count and cost distributions than unresolved trajectories.Resolved instances also have fewer trajectories with many steps or costs near the $4.00 maximum budget.Source: Figure 15 · p. 27

Initial reproduction, localization steps

• 156 occurrences of create,edit,python across the first three turns exemplify a common opening pattern in resolved SWE-agent trajectories using GPT-4.Agents create an empty Python file, add executable reproduction code, and run it. Early operations predominantly involve reproduction and localization, with some agents choosing localization first and refining searches as needed.Source: §B.3.2 · p. 27, Table 8 · p. 28, Figure 16 · p. 29
• Turns 1 and 4 show peaks in search_dir and search_file use, reflecting localization either immediately or after reproduction.The create command is much more frequent on the first turn than on any other turn.Source: §B.3.2 · p. 29, Figure 17 · p. 29

Cycle of edit, then evaluate

• Turns 5–31, except Turn 30, have edit as SWE-agent’s most frequent command, reflecting repeated cycles of editing and execution-based evaluation.Agents run reproduction scripts or existing tests to check fixes and preserve desirable behavior. Initial edits usually fail to resolve the issue, making execution feedback useful for subsequent edits.Source: §B.3.2 · pp. 29–30, Figure 18 · p. 30

Concluding submission turns

• Turns 10–20 concentrate submissions, while later turns reveal difficulty abandoning unsuccessful approaches in favor of further exploration.Most resolved instances end with intentional submission. Despite having 20+ turns remaining when failure becomes evident around Turn 20, agents tend to continue local editing rather than remedy accumulated errors or pivot.Source: Figure 17 · p. 29, Figure 18 · p. 30, §B.3.2 · p. 30

Cycle of edit, then evaluate

• Turn 6 onward retains localization operations that support inspecting long functions and making multi-line or multi-file edits.Agents navigate beyond the viewer’s visible lines or open other files. Execution can expose an error in a caller of an edited function, prompting additional navigation.Source: §B.3.2 · p. 30, Figure 18 · p. 30

Concluding submission turns

• $4 per-run cost limits produce exit_cost terminations around Turns 30–40, with token-heavy observations causing earlier termination.exit_cost automatically submits changes when the budget is exhausted, unlike intentional submit actions. Increasing the cost allowance would directly increase the maximum number of turns.Source: Figure 17 · p. 29, §B.3.2 · p. 30

B.3.3 Breakdowns of Action Sequences

• 51.7% of 2,294 task trajectories contain at least one lint-rejected edit, with a median of 3 failed edits among affected trajectories.The maximum was 33 failed edits per affected trajectory. Among 286 resolved instances, 113 contained failed edits, with a median of 2 and maximum of 26.Source: Figure 19 · p. 31, §B.3.3 · p. 31
• 90.5% eventual edit success falls to 57.2% after one failed edit, with recovery becoming less likely as consecutive failures accumulate.Recovery means failed edits immediately followed by a successful edit; across all instances, there were 810 successful versus 555 unsuccessful recoveries, averaging 2.2 versus 5.59 preceding failed attempts. Resolved trajectories recovered in 104 of 135 occurrences, averaging 2.03 preceding attempts for successful recoveries versus 4.22 for unsuccessful recoveries.Source: Figure 20 · p. 31, §B.3.3 · pp. 31–32
• Recurring action transitions reflect reproduction, localization, and edit–evaluation workflows, with edit the most likely successor to create, edit, goto, pytest, and python.The analysis normalizes next-action frequencies after the 15 most common sequences of each length from one through four; search and scrolling actions tend to repeat. After open, agents continue localization through search_file (0.35), scroll_down (0.18), or goto (0.09), or begin editing through edit (0.25).Source: §B.3.3 · p. 32
• Longer action sequences reveal reproduction-to-localization transitions, edit–test–cleanup–submission workflows, and cascading repetitions of editing or scrolling.After [create, edit, python], common next actions are edit (0.39), find_file (0.31), and search_dir (0.22); [find_file / search_dir, open, search_file] commonly leads to goto. The [edit, python, rm, submit] sequence commonly finishes trajectories, whereas four consecutive edit or scroll_down actions tend to continue repeating.Source: §B.3.3 · p. 32

B.4 Failure Modes

• 52.0% of unresolved instances involve incorrect or overly specific implementations, while 23.4% involve failed edit recovery.The automated analysis covered 248 unresolved SWE-bench Lite trajectories under the default configuration. LM-generated categories agreed with author labels on 87% of a 15-instance validation set.Source: §B.4 · pp. 32, 35

B.5 Performance Variance and Pass@k Rate

• Six runs of SWE-agent with GPT-4 on SWE-bench Lite show relatively low aggregate performance variance, despite considerable changes in which instances are resolved.Table 10 reports individual-run % Resolved and pass@k results; the additional evaluations examine k ∈ {3, 6}, beyond the default pass@1 reporting.Source: Table 10 · p. 35, §B.5 · p. 35

B.6 Patch Generations

• SWE-agent generates larger patches than corresponding human gold solutions, largely because of additional reproduction code.Both resolved and all-instance patches tend to contain more added lines, hunks, and files; Table 11 calculates statistics using values within the 90th percentile. Prior RAG baselines in Jimenez et al. typically produced smaller edits on average.Source: Table 11 · p. 36, §B.6 · p. 36
• Successfully resolved patches are relatively smaller than the overall patch distribution, while issues requiring multiple edits across a codebase remain challenging.The same size trend appears in RAG-based solutions.Source: §B.6 · p. 36

B.7 HumanEvalFix Evaluation

• 78.3% on HumanEval versus 47.8% on HumanEvalFix are GPT 4 scores reported by Muennighoff et al. [32], motivating evaluation of harder editing and debugging.Identifying and fixing bugs is also a major software-engineering activity and a subtask within SWE-bench.Source: §B.7 · p. 36
• Three evaluated languages see SWE-agent achieve the best HumanEvalFix benchmark performance.The supplied passage does not identify the languages or provide their scores.Source: §B.7 · p. 36
• 10 turns suffice for the large majority of solved HumanEvalFix instances.Figure 25 presents the distribution of trajectory lengths for solved instances.Source: §B.7 · p. 36, Figure 25 · p. 37
• 164 problems per language are evaluated by initializing SWE-agent with one buggy code file and available example tests, then requesting edits and verification.The configuration matches SWE-bench except for a language-specific demonstration, following the documentation in Muennighoff et al. [32].Source: §B.7 · p. 36
• HumanEvalFix isolates correct code editing without requiring bug localization or navigation through a large codebase.This narrows the evaluation relative to repository-level software-engineering tasks.Source: §B.7 · p. 36

B.8 Dataset Information

• SWE-bench and HumanEvalFix [32] have permissive software licenses allowing evaluation use and use in proprietary systems.Table 12 summarizes information about the two evaluation datasets.Source: §B.8 · p. 37, Table 12 · p. 37

B.9 Miscellaneous

• 59.05% file-localization F1 is achieved by SWE-agent with GPT-4 Turbo, versus 45.47% for BM25 with Claude 3 Opus in Jimenez et al..F1 compares the predicted set of edited or removed files against the gold patch.Source: §B.9 · p. 37
• GPT-4 Turbo episodes mostly end with explicit submission, whereas Claude 3 Opus episodes slightly more often end with automatic submission at the cost limit.Cost-limit endings submit existing edits; without edits, they are guaranteed unresolved. Early exits follow repeated malformed responses and also submit any existing edits.Source: §B.9 · p. 37
• 14.3% of explicit-submit episodes resolve on full SWE-bench with GPT-4 Turbo, compared with 3.1% of cost-limit endings.For both models, the large majority of resolved episodes end with explicit submission. Across all runs, only two episodes terminate because of repeated malformed responses.Source: Table 13 · p. 38, §B.9 · p. 38

C Prompts

• The prompt workflow begins with a persistent system message, an optional demonstration, and the issue, followed by prompts selected according to response validity and output.The system message documents commands and response format, emphasizes proper indentation and one thought/action pair per turn, and prohibits interactive session commands.Source: Figure 26 · p. 39, C Prompts · pp. 39–40
• Successful-task demonstrations slightly improve command usage and reduce errant responses, but their best representation—as separate turns or one explicitly labeled message—depends on the model.Separate turns can suggest prior successful experience but may confuse models by mixing task contexts. The authors consider demonstrations primarily formatting guidance; benefits for domain-specific problem solving remain uncertain.Source: C Prompts · pp. 40–41
• The instance template supplies the issue and manually developed tips that reduce recurring problem-solving mistakes identified through development-set trajectory inspection.It reiterates response-format, indentation, and noninteractive-shell constraints before prompting the first command. The authors find the tips effective but acknowledge that manually identifying failures and writing guidance does not scale.Source: C Prompts · p. 41
• Execution prompts explicitly acknowledge silent commands, while collapsed observations preserve history structure but replace old content to reduce tokens and outdated or duplicate information.Commands producing output return that output with a shell prompt; silent commands receive an explicit success acknowledgment to discourage unnecessary checks. Collapsed observations become one-line placeholders reporting the number of omitted lines.Source: C Prompts · p. 43
• 3 consecutive malformed responses terminate an episode; otherwise, format errors trigger correction requests, and subsequent valid responses enable removal of malformed exchanges.Removing malformed exchanges reduces unnecessary context and helps prevent further formatting errors. Valid responses provide continuing in-context demonstrations of correct interaction.Source: C Prompts · pp. 43–44

D Qualitative Analysis

• 2 successful and unsuccessful task instances from SWE-bench Lite were selected for full qualitative analysis.Source: D Qualitative Analysis · p. 45
• The analyzed trajectories were generated using the default configuration with GPT-4 Turbo.Source: D Qualitative Analysis · p. 45

Analysis for psf/requests-2317

• The builtin_str function incorrectly converted the requests method variable into a literal string such as "b’GET’".The task was to resolve this method-conversion problem in the requests library.Source: Analysis for psf/requests-2317 · p. 45
• The incorrect method string caused the requests library to return a 404 Not Found response.Source: Analysis for psf/requests-2317 · p. 45
• The issue appeared in Python 3.4 but not Python 2.6, indicating version-specific differences in bytes handling.Source: Analysis for psf/requests-2317 · p. 45
• The affected method variable was a bytes object, exemplified by b’GET’, rather than an ordinary string.Source: Analysis for psf/requests-2317 · p. 45

Interaction Summary

• The agent’s patch passed the instance’s unit tests but reimplemented an existing utility’s behavior, potentially reducing maintainability and portability.The gold patch instead uses to_native_string, defined in requests/utils.py, which behaves identically to the agent’s changes.Source: Interaction Summary · p. 45
• The agent fixed builtin_str’s incorrect binary-prefix conversion by decoding bytes-valued HTTP methods as ASCII before applying builtin_str.It selected requests/sessions.py rather than the build-directory copy and used search_file to locate the conversion at line 428.Source: Interaction Summary · pp. 45, 51
• 200 OK was returned when the reproduction script sent a request with a binary method string, and the script completed without errors.After verification, the agent deleted reproduce_issue.py and submitted the code changes.Source: Interaction Summary · pp. 45, 55
• The interaction’s patch decodes byte-valued methods as ASCII before applying builtin_str.Source: Interaction Summary · p. 56
• The gold patch replaces builtin_str(method) with to_native_string(method).Source: Interaction Summary · p. 56

Analysis for pylint-dev/pylint-5859

• Pylint fails to report fixme warnings (W0511) for punctuation-only note tags, although these tags should be recognized.With -notes="YES,???", pylint recognizes YES but ignores ???.Source: Analysis for pylint-dev/pylint-5859 · p. 57

Interaction Summary

• In EncodingChecker.open within pylint/checkers/misc.py, the agent replaced the regex word boundary with a lookahead accepting a non-word character or string end.The change targeted matching of punctuation-only note tags.Source: Interaction Summary · pp. 57, 68
• The fix passed the necessary tests and made Pylint correctly report both “YES” and punctuation-only “???” note tags.The agent reran Pylint on the reproduction file, then removed that temporary file and submitted the changes.Source: Interaction Summary · pp. 57–58
• The agent’s boundary pattern (\W|$) differed from the gold patch’s (:|\s|\Z), potentially causing unintended behavior beyond existing test coverage.Passing the tests therefore did not establish behavioral equivalence with the gold patch.Source: Interaction Summary · p. 57
• The agent reproduced the missing “???” warning, then located the relevant checker after initial search-command confusion.The initial run reported only “YES”; searching for “notes” identified pylint/checkers/misc.py among the matching files.Source: Interaction Summary · pp. 57, 59, 60
• The patch replaced \b with (?=\W|$) in both note-tag regex branches, allowing punctuation-only tags followed by a non-word character or string end.The displayed gold patch instead uses (?=(:|\s|\Z)), allowing a colon, whitespace, or string end after the tag.Source: Interaction Summary · pp. 70–71
• Pylint correctly detected both “YES” and “???” note tags after the fix, resolving the reproduced issue.Both tags produced W0511 fixme warnings.Source: Interaction Summary · p. 70

Analysis for sympy/sympy-21614

• The SymPy issue concerns Derivative.kind returning UndefinedKind when a derivative-specific type such as DerivativeKind is expected.This passage introduces the issue without reporting a fix or evaluation.Source: Analysis for sympy/sympy-21614 · p. 72

Interaction Summary

• The agent exhausted its budget scrolling through function.py after reproducing the issue, terminating without submitting a fix.It continued reading the Derivative class definition, which spans many hundreds of lines.Source: Interaction Summary · p. 72
• Searching for class Derivative located the definition only after unsuccessful speculative searches.The evaluation recommends using this direct search pattern earlier rather than guessing filenames.Source: Interaction Summary · p. 72
• The gold patch overrides the inherited kind property from sympy.basic, requiring an understanding of Derivative’s inheritance structure.The evaluation identifies more targeted, efficient searches as necessary to locate this problem.Source: Interaction Summary · p. 72
• The reproduction script confirmed that a Derivative of a MatrixSymbol returned UndefinedKind instead of the expected MatrixKind(NumberKind).The agent created reproduce_issue.py using the issue report’s reproduction code and ran it successfully.Source: Interaction Summary · pp. 72–74
• The displayed documentation explains that non-Symbol differentiation temporarily substitutes Symbols, treating functions independently and derivatives as dependent only on differentiation variables.Examples show x.diff(f(x)), g(x).diff(f(x)), and differentiation of f'(x) with respect to f(x) returning zero.Source: Interaction Summary · p. 83
• Replacing an undefined function with an expression involving additional differentiation variables can make substitution before differentiation disagree with substitution afterward.For f(x)*g(y), replacing f(x) with x*y before differentiating by x and y produces an extra g(y) term compared with substituting afterward.Source: Interaction Summary · pp. 83–84
• SWE-agent encountered diff and expand while scrolling through function.py but still had not located the Derivative class's kind implementation.The agent repeatedly inferred that related functions indicated proximity to the target and proposed continuing to scroll.Source: Interaction Summary · pp. 89–90
• The SymPy interaction repeatedly scrolled through function.py seeking Derivative.kind, then exited because it reached the cost limit.Repeated observations identified expansion functions and wrappers but continued proposing further scrolling to locate the Derivative implementation.Source: Interaction Summary · pp. 91–92
• The SymPy gold patch adds a kind property that returns self.args[0].kind.This is the reference patch, not an agent-produced fix.Source: Interaction Summary · p. 92

Analysis for django/django-14411

• ReadOnlyPasswordHashWidget renders text rather than an input, so its label should omit the for attribute that currently points to a non-labelable element.The issue occurs in the Django admin interface.Source: Analysis for django/django-14411 · p. 92

Interaction Summary

• After find_file returned no results, search_dir located widget references, leading the agent through helpers.py to the widget and field definitions in forms.py.helpers.py contained only a reference, while the widget’s rendering template lacked relevant information about the for attribute.Source: Interaction Summary · pp. 92–93
• The agent added a custom label_tag method to ReadOnlyPasswordHashField, attempting to suppress the label’s for attribute through field-level changes.It reasoned that Django’s form rendering added the attribute automatically and imported utilities from django.utils.html.Source: Interaction Summary · p. 93
• The agent submitted without testing because it assumed a Django server could not be run and tested in the environment.Source: Interaction Summary · p. 93
• The attempted fix failed because the agent misunderstood widget label handling; the correct solution required overriding the Widget class’s id_for_label property.Similar functionality was available elsewhere in the codebase.Source: Interaction Summary · p. 93
• The password widget template contained a div and summary entries but no label element, directing SWE-agent’s investigation toward rendering outside the template.The reported issue concerned a label’s for attribute pointing to a non-labelable element.Source: Interaction Summary · p. 105
• SWE-agent located ReadOnlyPasswordHashField in UserChangeForm and proposed overriding its label_tag method to omit the problematic for attribute.The agent hypothesized that Django’s form rendering automatically added the attribute, rather than the widget template.Source: Interaction Summary · pp. 109, 111
• The editor rejected SWE-agent’s proposed Django patch for undefined format_html and format_html_join, prompting an import-based retry.The rejected changes were not applied. SWE-agent then began a revised edit importing both utilities from django.utils.html; no outcome is shown here.Source: Interaction Summary · pp. 111–113
• For django__django-14411, SWE-agent submitted an untested label_tag override omitting the 'for' attribute; the gold patch instead adds id_for_label returning None.The agent stated that it could not run a Django server or render forms in that environment and proceeded to submission.Source: Interaction Summary · pp. 115–116

E.1 Ethics & Broader Impacts

• SWE-agent uses ephemeral Docker containers for inference and evaluation to limit harmful side effects from executing LM-generated code.Container isolation is weaker than virtualized hardware isolation and is considered sufficient only for code not deliberately engineered to exploit recent container-escape vulnerabilities.Source: §E.1 · p. 117
• Unofficial evaluation datasets or infrastructure could inject malicious code or instructions; the authors identify official repositories and channels to reduce this risk.They also encourage third parties to integrate improvements into the maintained codebase.Source: §E.1 · p. 117
• SWE-agent could support malicious-code pipelines; its open release can support research on behavioral constraints and experimentation by legal and policy experts.The authors interpret strong SWE-bench performance as suggesting that future systems will become increasingly capable of such misuse.Source: §E.1 · p. 117

E.2 Reproducibility

• The authors release source code, documentation, and inference/evaluation artifacts to support reproducing the paper’s results and extending SWE-agent.Resources include pipeline and interface implementations, trajectories, generated code, evaluation traces, and analysis notebooks; main and supplementary results can be fully rendered from the data. Text/video documentation, simple scripts, and an active help forum support reproduction.Source: §E.2 · p. 117

• SWE-agent’s small but effective toolkit could gain browsing, static/dynamic analysis, fault localization, and fuzzing-based test generation.Additional tools could exploit more signals from issue descriptions and codebases to improve % Resolved.Source: §E.3 · p. 117
• Automating currently manual ACI development could accelerate SWE-agent extensions and clarify design principles, although sustained iterative performance improvement remains challenging.Components were crafted from recurring behaviors observed within and across trajectories. Automation raises questions about how systems could scrutinize and iterate upon their own designs.Source: §E.3 · p. 118
• Transferring ACI beyond programmatic tasks remains an open question, with potential benefits for downstream performance and the development of additional ACI principles.Potential domains include education technology, data analysis, and enterprise workflows. Search interfaces could be tailored to agents and the specific downstream task.Source: §E.3 · p. 118