PaperlayerView exact revision on arXiv

Source-linked AI summary

Programming with Pixels: Can Computer-Use Agents do Software Engineering?

Pranjal Aggarwal, Sean Welleck

arXiv:2502.18525v2cs.SEcs.LG

TL;DR

It remains unclear whether general-purpose computer-use agents can automate sophisticated software engineering beyond the simple tasks in prior evaluations. This paper introduces PwP and PwP-Bench to evaluate that question across diverse IDE-based tasks, finding that visual-only agents underperform specialized coding agents while basic text APIs substantially improve performance. The analysis identifies visual grounding and limited use of IDE functionality as central weaknesses.

  • Problem

    It remains unclear whether general-purpose computer-use agents can match specialized agents on sophisticated software engineering tasks.

    Source: §1 · pp. 1–2

  • Method

    The paper introduces PwP, a VSCode-based computer-use environment, and PwP-Bench, a 15-task benchmark spanning software-engineering languages, modalities, and skills.

    Source: §1 · p. 2

  • Results

    22.9% average accuracy with pure visual interaction rose to 50.7% with file-editing and bash APIs, often approaching specialized-agent performance.

    Source: §1 · p. 2, §5.1 · p. 7

  • Takeaways & Limitations

    PwP establishes software engineering as a realistic benchmark for testing whether generalist computer-use agents can reach specialist-level performance.

    Source: §6 · p. 10

    Current computer-use agents exhibit significant visual-grounding limitations, including selecting incorrect interface elements even when trained for UI interaction.

    Source: §5.1 · p. 8

Abstract

from arXiv · show

Computer-use agents (CUAs) hold the promise of performing a wide variety of general tasks, but current evaluations have primarily focused on simple scenarios. It therefore remains unclear whether such generalist agents can automate more sophisticated and specialized work such as software engineering (SWE). To investigate this, we introduce $\texttt{Programming with Pixels}$ (PwP), the first comprehensive computer-use environment for software engineering, where agents visually control an IDE to perform diverse software engineering tasks. To enable holistic evaluation, we also introduce \texttt{PwP-Bench}, a benchmark of 15 existing and new software-engineering tasks spanning multiple modalities, programming languages, and skillsets. We perform an extensive evaluation of state-of-the-art open-weight and closed-weight CUAs and find that when interacting purely visually, they perform significantly worse than specialized coding agents. However, when the same CUAs are given direct access to just two APIs-file editing and bash operations-performance jumps, often reaching the levels of specialized agents despite having a task-agnostic design. Furthermore, when given access to additional IDE tools via text APIs, all models show further gains. Our analysis shows that current CUAs fall short mainly due to limited visual grounding and the inability to take full advantage of the rich environment, leaving clear room for future improvements.PwP establishes software engineering as a natural domain for benchmarking whether generalist computer-use agents can reach specialist-level performance on sophisticated tasks. Code and data released at https://programmingwithpixels.com

1 INTRODUCTION

Programming with Pixels asks whether general-purpose computer-use agents can automate sophisticated software engineering, beyond the simple tasks used in prior evaluations. It introduces an IDE environment and PwP-Bench to compare visual agents with specialized coding agents across diverse SWE tasks.

Motivation

  • PwP addresses whether general-purpose computer-use agents can match specialized agents on complex software engineering tasks.Prior evaluations mainly covered web navigation, document editing, and operating-system settings, leaving this question open.Source: §1 · pp. 1–2

Programming with Pixels

  • PwP provides a VSCode environment where agents use primitive keyboard and mouse actions while accessing IDE functionality through the visual interface.The environment is designed to support any software engineering task achievable in an IDE without language- or domain-specific modifications.Source: §1 · p. 2

PwP-Bench

  • PwP-Bench spans 15 software engineering tasks across multiple programming languages, modalities, and skills, including code generation, pull-request resolution, UI development, and data science.The benchmark combines 13 existing SWE tasks with 2 newly developed tasks.Source: §1 · p. 2, Figure 1 · p. 2

Evaluation

  • 22.9% average accuracy under pure visual interaction rose to 50.7% with file-editing and bash APIs, often approaching specialized-agent performance.The comparison highlights a large gap between visual-only interaction and access to two basic text APIs.Source: §1 · p. 2

Findings

  • The analysis identifies poor visual grounding and limited use of IDE functionality as primary weaknesses of current computer-use agents.The paper presents better visual grounding and stronger environment use as key future directions.Source: §1 · p. 2

2 RELATED WORK

Related work includes multimodal interface agents and specialized software-engineering agents. PwP differs by evaluating primitive-action computer-use agents and specialized agents on the same broad software-engineering benchmark.

Multimodal and Computer-Use Agents

  • Prior computer-use benchmarks mainly test simple browser, operating-system, and document tasks, leaving software engineering insufficiently evaluated.PwP-Bench is presented as the first benchmark specifically designed to test computer-use agents on software engineering.Source: §2 · p. 3

Software Engineering Agents

  • Specialized SWE agents use hand-engineered tools and scaffolding tailored to particular languages, tasks, or developer operations.Examples include language-specific parsers, task-specific modifications, IPython kernels, and custom browser views.Source: §2 · p. 3

Comparison

  • PwP enables direct comparison between visual computer-use agents and specialized agents while testing whether standard IDE interaction can replace specialized APIs.The platform supports both designs across diverse software-engineering tasks.Source: §2 · p. 3

3 PROGRAMMING W I T H PIXELS (PWP)

PwP is a general-purpose IDE environment that models software engineering as interaction with a partially observable VSCode-and-OS system. It combines unrestricted primitive actions, broad tool and modality support, programmatic verification, and extensible infrastructure.

PwP environment

  • PwP models the IDE and operating-system context as a POMDP with keyboard-and-mouse actions, screen-based observations, transitions, and task rewards.Rewards can be computed by running tests after an agent changes code.Source: §3 · p. 3

Expressive observation and action space

  • Agents can attempt any software engineering task achievable through an IDE’s visual interface rather than choosing from a predefined action set.They can navigate menus, move cursors, and press keys to perform multi-step actions.Source: §3 · pp. 3–4

Full Spectrum of Developer Tools

  • PwP exposes developer tools such as debuggers, linters, refactoring utilities, and code suggestions through the IDE interface.This creates a test of whether computer-use agents can exploit functionality that specialized agents access through APIs.Source: §3 · p. 4

Multimodality and language agnosticism

  • The same computer-use agent must handle multiple programming languages and modalities, including text, images, data files, and PDFs, without task-specific modifications.DOM trees and Set-of-Marks annotations support agents requiring visual-grounding assistance.Source: §3 · p. 4

Ease of verification

  • PwP separates visual interaction from programmatic evaluation by exposing IDE state, files, and processes for tests, compilation checks, and correctness verification.This makes task completion easier to verify and feedback easier to provide.Source: §3 · p. 4

Infrastructure and Future Adaptability

  • Sandboxed Docker deployment, checkpointing, and a Gymnasium interface support isolated, reproducible, extensible evaluation and training.New benchmarks can be added through configuration, and future IDE tools can be incorporated without architectural changes.Source: §3 · p. 4

4 PWP-BE N C H

PwP-Bench evaluates broad software engineering ability in a realistic IDE through diverse tasks, languages, modalities, and execution-based checks. Its design emphasizes multi-step tool use and includes both conventional and newly introduced SWE activities.

Benchmark Scope

  • PwP-Bench contains 15 diverse software engineering tasks spanning 14 programming languages and multiple modalities.Its goal is to test the depth and breadth of software engineering activities accessible through an IDE.Source: §4 · p. 4

Tasks

  • The benchmark includes 5,400 instances from 13 existing code-generation datasets and 2 newly created datasets.Tasks require substantial tooling interaction, multiple steps, and coverage across languages and modalities.Source: §4 · p. 4

Code Generation and Editing

  • Code generation and editing tasks cover completion, pull-request resolution, data science, and code editing while leveraging varied IDE functionality.Examples include debuggers, linters, IPython kernels, refactoring utilities, and repository search.Source: §4 · p. 5

Multimodal Code Synthesis

  • Multimodal code-synthesis tasks generate or edit code from images, charts, data files, or PDFs.The category includes UI development, chart-to-code generation, multimodal code editing, and data analysis.Source: §4 · p. 5

Domain-Specific Programming

  • Domain-specific tasks target ethical hacking and interactive theorem proving, requiring sustained interaction with IDE states, executables, images, or extensions.These tasks test software engineering in specialized fields rather than code generation alone.Source: §4 · p. 5

IDE-Specific and General SWE Tasks

  • New IDE Configuration and General-SWE tasks assess settings changes, profiling, refactoring, debugging, UI mockups, and code restoration.These additions target practical non-code activities often absent from conventional benchmarks.Source: §4 · p. 5

Benchmarking Design and Task Setup

  • Each task starts from an IDE state and instruction, then evaluates a target final state with execution-based criteria such as unit tests.PwP-Bench-Lite provides 300 instances while preserving task difficulty and distribution for faster experimentation.Source: §4 · p. 5

5 EVALUATING AGENTS IN PROGRAMMING WITH PIXELS

PwP evaluates three agent designs on diverse software-engineering tasks in a VSCode-based environment, comparing visual computer-use agents with API-augmented and specialized agents. Purely visual agents perform poorly because of visual-grounding and IDE-tool-use failures, while text APIs and assisted tooling substantially improve performance.

5 EVALUATING AGENTS IN PROGRAMMING WITH PIXELS

  • PwP evaluates primitive-action computer-use agents, file/bash-augmented agents, and the text-API specialized agent mini-sweagent under a common 20-step task limit.The benchmark uses 300 PwP-Bench-Lite instances and evaluates multiple vision-language and UI-trained models.Source: §5 · pp. 5–6

5.1 RESULTS AND ANALYSIS

  • Purely visual computer-use agents reach at most 22.9% average accuracy, substantially below mini-sweagent’s 48.8% specialized-agent result.The comparison covers four PwP-Bench categories, with task-wise results reported separately in Table 6.Source: Table 1 · p. 6, §5.1 · p. 7
  • 50.7% maximum average accuracy is reached when computer-use agents receive file-editing and bash APIs, approaching specialized-agent performance on some tasks.Purely visual agents reach at most 22.9%, whereas mini-sweagent achieves 48.8% accuracy; the API gains arise because agents can bypass unreliable visual editing and command execution.Source: §5.1 · p. 7
  • Agents can use basic IDE features such as navigation, editing, search, live preview, and visualization, but they rarely exploit advanced tools such as profilers and debuggers.General-SWE tasks can require only four to five appropriate IDE steps, yet agents achieve minimal performance, indicating substantial room for improvement in tool use and planning.Source: §5.1 · pp. 7–8, Figure 2 · p. 7, Figure 3 · p. 7
  • Visual-grounding failures include selecting incorrect UI elements, hallucinating screen contents, misunderstanding UI state, and failing to recover after misplaced clicks or typing.At least one visual-grounding error occurs in 20% of GPT-4o trajectories and 95% of Claude Sonnet-4.0 trajectories; Set-of-Marks does not resolve the problem.Source: §5.1 · p. 8, Figure 4 · p. 8, Figure 5 · p. 8
  • Agents choose interaction modes by task: they use file APIs for HumanEval, visual IDE tools for Design2Code and VSCode tasks, and few visual interactions for SWE-Bench-MultiModal.This flexibility explains why API access especially helps tasks requiring basic file editing, while visual tools remain important for design comparison and IDE configuration.Source: §5.1 · pp. 8–9, Figure 6 · p. 9
  • Claude’s pure computer-use performance nearly doubles from 10.5% to 22.9% over seven months, while the file/bash gap narrows from 35.0% to 27.8%.The authors report rapid progress but state that a substantial performance gap remains.Source: §5.1 · p. 9, Figure 7 · p. 9
  • Assisted IDE tools yield up to a 13.3% average gain, while explicit instructions raise Claude’s project-refactoring accuracy from 25% to 75%.These results suggest that current agents possess some latent tool-use ability but struggle to discover or operate complex IDE functionality reliably.Source: §5.1 · pp. 7, 9

6 CONCLUSION

PwP provides a unified, software-engineering-focused environment and benchmark for evaluating computer-use agents across diverse tasks. Its supporting infrastructure adds tool access, state management, and sandboxing features for systematic comparison.

6 CONCLUSION

  • PwP establishes software engineering as a dedicated domain for benchmarking computer-use agents, with a unified platform separating environment design from agent design.The environment supports standardized evaluation and incorporation of existing agents.Source: §6 · p. 9

A.2 COMPARISON WITH OTHER ENVIRONMENTS

  • The environment compares computer-use platforms by multimodal interaction, execution-based evaluation, specialization, and software-engineering specificity.These dimensions position PwP relative to general and specialized environments.Source: §A.2 · p. 14

A.1 TOOLS

  • PwP natively supports hand-engineered tools and adds streaming, Docker-based configuration, and checkpointing for reproducible and search-oriented evaluation.Checkpointing stores filesystem and complete process state.Source: §A.2 · p. 14, §A.1 · p. 14

A.2 COMPARISON WITH OTHER ENVIRONMENTS

  • PwP exposes interactive environment control through a Python API with keyboard and mouse actions, checkpointing, state management, and reward handling.Figure 9 illustrates the interaction pattern.Source: Figure 9 · p. 15

A.3 INFRASTRUCTURE AND IMPLEMENTATION

PwP runs agents in isolated, reproducible VSCode containers and exposes multiple channels for observing and controlling the IDE. Its implementation supports low-latency interaction, flexible resource configuration, and cross-platform testing.

A.3 INFRASTRUCTURE AND IMPLEMENTATION

  • Each PwP instance runs modified VSCode and a minimal operating system in an isolated Docker container with independent filesystems, processes, and checkpointable state.Checkpointing supports backtracking in search algorithms and reinforcement-learning training.Source: §A.3 · p. 15
  • PwP combines container lifecycle control, screen and video capture, DOM state, and the VSCode Extension API for high-level control and detailed IDE observations.These channels provide complementary environment and state access.Source: §A.3 · p. 15
  • Agents act through simulated keyboard and mouse events, while ImageMagick and ffmpeg provide static screenshots and streaming video.The controller supports code insertion, file opening, and menu navigation.Source: §A.3 · p. 15
  • A gymnasium-like Python API abstracts container management, observations, actions, and rewards while allowing customizable installations, resources, and display settings.The environment was tested on Ubuntu, macOS, and Windows.Source: §A.3 · pp. 15–16

B.1 TASKS

PwP-Bench unifies diverse software-engineering tasks and evaluates them with programmatic, task-specific metrics. Compared with prior agents and benchmarks, it emphasizes broad, multilingual, multimodal IDE interaction rather than narrow tool or task specialization.

B.1 TASKS

  • PwP-Bench-Lite contains 300 instances, formed by randomly sampling 20 instances from each task across the benchmark’s task categories.The benchmark materials identify tasks across all categories and report the languages used for each task.Source: §B.1 · p. 16

B.2 EVALUATION

  • Programmatic verifiers evaluate every task, and PwP-Bench reports macro-averaged original-dataset metrics rather than imposing a single universal measure.Metrics include accuracy and multi-axis scores for tasks such as testing, chart generation, design, and data science.Source: §B.2 · p. 17
  • PwP includes VSCode-state tasks and general SWE tasks covering profiling questions, refactoring, debugging, UI mockups, and code restoration.Evaluation uses IDE state, automated tests, CLIP scores, final answers, or restored repository state as appropriate.Source: §B.2 · p. 17

C.1 COMPARISON TO SOFTWARE ENGINEERING AGENTS

  • Unlike prior benchmarks confined to particular languages, modalities, or task types, PwP-Bench unifies multimodal and multilingual IDE-based evaluations.It is designed to test general-purpose agents across varied software-engineering activities.Source: §C.1 · p. 18
  • PwP agents are task- and language-agnostic, whereas existing SWE agents often rely on specialized parsers, kernels, search tools, linters, or browser views.In PwP, these capabilities are available within the IDE and agents must select how to use them.Source: §C.1 · pp. 18–19
  • PwP standardizes evaluation by separating the general-purpose environment from agent design and allowing existing SWE agents to be modeled within the framework.This supports comparison across current and future agents.Source: §C.1 · p. 19

C.2 COMPARISON TO GENERAL VISUAL AND COMPUTER-USE AGENTS

Existing visual and computer-use benchmarks target web, operating-system, or generic interfaces rather than the dense, tool-rich setting of software engineering. PwP addresses this gap with an expressive IDE environment and multimodal SWE tasks.

C.2 COMPARISON TO GENERAL VISUAL AND COMPUTER-USE AGENTS

  • Predefined-action visual agents often depend on accessibility trees or similar aids for grounding and can fail in dense IDE interfaces.PwP presents a more complex interface with many interactive elements.Source: §C.2 · p. 19
  • Primitive-action computer-use agents lack a software-engineering-specific evaluation environment, which PwP supplies through a unified expressive IDE platform.The platform challenges them with realistic and diverse SWE tasks.Source: §C.2 · p. 19
  • General expressive environments such as OSWorld support broad actions but are not designed for IDE-specific software-engineering challenges and may incur extra computational overhead.PwP is dedicated to the distinct demands of software engineering.Source: §C.2 · p. 19
  • PwP-Bench combines pull-request handling, debugging, and image-based code generation across multiple programming languages inside an IDE.Existing tool-based SWE agents can also be incorporated through exposed IDE APIs.Source: §C.2 · p. 19

D EXPERIMENTAL SETUP AND IMPLEMENTATION DETAILS

The experiments use multiple agent designs, including computer-use variants and a multimodal mini-sweagent, evaluated across PwP-Bench tasks. Main runs use 20 iterations, while longer-horizon analyses examine whether additional steps change performance trends.

D.1 AGENT DESIGN

  • The main experiments use 20 iterations because most tasks can be completed within that budget while larger budgets increase computational cost.Experiments with 250 steps on SWE-Bench-related datasets showed no difference in trends.Source: §D.1 · pp. 19–20

D.2 MINI-SWEAGENT

  • The mini-sweagent uses Claude-4.0 Sonnet and receives required task images through a multimodal prompt.Source: §D.2 · p. 20

E RESULTS

  • Table 6 reports comprehensive results for all agent designs across the 15 PwP-Bench datasets.Source: E RESULTS · p. 20

E.1 COMPARISON WITH BEST REPORTED SPECIALIZED SWE AGENTS

The comparison aggregates the highest publicly reported specialized-agent scores for each dataset using citations, leaderboards, and verified web searches. These specialized results generally exceed computer-use agents, though individual-dataset comparisons are limited by differing methods and evaluation samples.

E.1 COMPARISON WITH BEST REPORTED SPECIALIZED SWE AGENTS

  • The comparison takes the highest reported score found through citations, official leaderboards, and manually verified web-search results.Source: §E.1 · p. 20
  • Individual-dataset comparisons are not fully comparable because baselines use specialized methods and the evaluation samples only 20 examples per dataset.Reported scores may also change as newer results appear; the search was conducted on 22nd September 2025.Source: §E.1 · p. 20
  • Reported specialized baselines include 98.8% on HumanEval, 75.2% on SWE-Bench, 43% on SWE-Bench-Multilingual, 58% on ResQ, and 90.2% on Design2Code.Source: §E.1 · p. 20
  • 100% on Intercode-CTF was reached by a computer-use agent using Claude-3.5-Sonnet, exceeding the 72% specialized-agent score with statistical significance (p-value = 0.014).Source: §E.1 · p. 21
  • Specialized SWE agents generally achieve higher scores than computer-use agents, including those given file and bash APIs.The paper concludes that specialized agents still perform better overall, while scaffolding around computer-use agents may help.Source: §E.1 · p. 21, Table 6 · p. 21

F ADDITIONAL RESULTS

Additional analyses identify visual-grounding errors, tool-use failures, and long-horizon effects as important determinants of performance. More steps help, but access to tools and the ability to use them effectively remain central.

F ADDITIONAL RESULTS

  • 20% of Claude-Sonnet-4.0, 35% of Claude-3.5-Sonnet, and 95% of GPT-4o trajectories contained at least one visual-grounding error.Errors include incorrect clicks, misinterpreting the current state, or interacting with the wrong element.Source: F ADDITIONAL RESULTS · p. 21
  • Assisted agents averaged up to a 13.3% gain across GPT-4o, Gemini-1.5-Pro, and Claude-3.5-Sonnet when using IDE tools.Source: F ADDITIONAL RESULTS · p. 21, Table 7 · p. 22
  • Current models’ tool-use limitations are poor visual grounding, difficulty with complex interfaces, and failure to prioritize IDE solutions over manual approaches.Source: F ADDITIONAL RESULTS · p. 22
  • Explicitly prompting Claude to use VSCode’s rename feature increased refactoring accuracy from 25% to 75% across four tasks.The feature uses the IDE’s AST-based semantic understanding to rename symbols across the repository.Source: F ADDITIONAL RESULTS · p. 22
  • Agents often fail to use debuggers effectively because poor visual grounding prevents them from interpreting execution state and paused-line information.Source: F ADDITIONAL RESULTS · p. 22
  • When an action was suppressed, Gemini-1.5-Pro often continued as if it had succeeded, whereas Claude-Sonnet-4.0 ignored the screenshot in only one of five tested scenarios.Source: F ADDITIONAL RESULTS · p. 23
  • 250-step runs improved performance while preserving the overall ordering: file/bash agents reached 43.3%, mini-sweagent 41.7%, and pure computer-use agents 18.3%.Source: F ADDITIONAL RESULTS · p. 23, Table 8 · p. 23

G QUALITATIVE ANALYSIS

The qualitative analysis illustrates both failures and successful uses of IDE interaction, while the discussion explains why IDEs offer capabilities beyond simple Bash interfaces. PwP adds little computational overhead relative to the computer-use models themselves.

G QUALITATIVE ANALYSIS

  • Visual examples show agents misplacing edits, typing into the wrong panel, and failing to recognize the active editor or visual error indicators.Source: Figure 13 · p. 24, Figure 15 · p. 24

H DISCUSSION

  • PwP’s essential overhead comes from screenshots and the IDE, while VNC and media processing are optional; overall, the environment’s cost is small compared with model inference.Source: H DISCUSSION · p. 24
  • The ChartMimic example shows that assisted interaction can outperform pure computer-use interaction despite the same underlying image.Source: Figure 16 · p. 25
  • Interactive IDE debugging exposes breakpoints, variables, execution state, stack traces, and machine-parseable program context.Source: H DISCUSSION · p. 25
  • IDEs support AST-based refactoring that understands code context and avoids unintended changes to unrelated symbols.Source: H DISCUSSION · p. 25
  • IDE interfaces provide structured testing, coverage, profiling, semantic search, and extension integration that are harder to obtain by parsing CLI output.Source: H DISCUSSION · p. 25
  • IDE extensions can unify multiple structured tools and provide more reliable automation interfaces than separate command-line tools.Source: H DISCUSSION · pp. 25–26

Related papers

Loading 2502.18525v2…