PaperlayerView exact revision on arXiv

Source-linked AI summary

SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?

John Yang, Carlos E. Jimenez, Alex L. Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R. Narasimhan, Diyi Yang, Sida I. Wang, Ofir Press

arXiv:2410.03859v1cs.CLcs.AIcs.SE

TL;DR

Existing software-engineering benchmarks provide limited evidence about visual and non-Python software domains. This paper introduces SWE-bench Multimodal to evaluate real-world visual JavaScript tasks, finding that existing systems struggle while flexible SWE-agent configurations perform comparatively well.

  • Problem

    SWE-bench is predominantly text-based and Python-only, leaving visual software development and alternative programming-language settings underexplored.

  • Method

    The paper constructs SWE-bench M from real GitHub issues in visual, user-facing JavaScript repositories and evaluates adapted coding-agent systems on the resulting multimodal tasks.

  • Results

    Existing systems perform poorly on SWE-bench M; interactive SWE-agent systems resolve 11.5% on average versus 3.9% for Agentless and 5.5% for RAG baselines.

  • Takeaways & Limitations

    SWE-bench M exposes visual problem-solving and cross-language generalization challenges, while flexible language-agnostic agent features support stronger performance across these settings.

  • Takeaways & Limitations

    Many existing systems rely on Python-specific analysis tools, requiring redesign for other languages or failing when equivalent tools are unavailable.

Abstract

from arXiv · show

Autonomous systems for software engineering are now capable of fixing bugs and developing features. These systems are commonly evaluated on SWE-bench (Jimenez et al., 2024a), which assesses their ability to solve software issues from GitHub repositories. However, SWE-bench uses only Python repositories, with problem statements presented predominantly as text and lacking visual elements such as images. This limited coverage motivates our inquiry into how existing systems might perform on unrepresented software engineering domains (e.g., front-end, game development, DevOps), which use different programming languages and paradigms. Therefore, we propose SWE-bench Multimodal (SWE-bench M), to evaluate systems on their ability to fix bugs in visual, user-facing JavaScript software. SWE-bench M features 617 task instances collected from 17 JavaScript libraries used for web interface design, diagramming, data visualization, syntax highlighting, and interactive mapping. Each SWE-bench M task instance contains at least one image in its problem statement or unit tests. Our analysis finds that top-performing SWE-bench systems struggle with SWE-bench M, revealing limitations in visual problem-solving and cross-language generalization. Lastly, we show that SWE-agent's flexible language-agnostic features enable it to substantially outperform alternatives on SWE-bench M, resolving 12% of task instances compared to 6% for the next best system.

1 INTRODUCTION

SWE-bench evaluates autonomous software-engineering systems but covers predominantly Python, text-based tasks, leaving visual and other software domains underrepresented. SWE-bench Multimodal addresses this gap with visual JavaScript tasks, where existing systems perform substantially worse and generalizability remains a concern.

  • SWE-bench evaluates repository-level software tasks from GitHub issues and pull requests using unit tests, while its SWE-bench Lite performance rose from 3% to 43%.
  • SWE-bench covers only a fraction of software engineering, with predominantly Python repositories and just 5.6% of tasks containing an image.
  • SWE-bench M asks whether AI systems generalize to visual software domains, particularly front-end development with textual, visual, and interactive components.
  • 619 task instances form SWE-bench M, covering visual JavaScript applications and requiring images or videos in problem descriptions or testing scenarios; experts judged images necessary for 83.5% of tasks.
  • Existing systems perform significantly worse on SWE-bench M, as visual challenges and JavaScript’s varied object-oriented, functional, and procedural code structures resist standardized solutions.
  • Adapting existing systems highlights generalizability as an overlooked design consideration, because many rely on Python-only fault-localization tools.

2 SWE-BENCH MULTIMODAL

SWE-bench M extends repository-level benchmark evaluation to multimodal JavaScript software by collecting, filtering, testing, and manually validating real GitHub tasks. Its 619 instances span diverse visual challenges, many of which annotators judged essential for solving tasks, and cover a broad difficulty range.

  • Formulations: SWE-bench M adapts SWE-bench’s repository-level formulation, where agents modify codebases and must pass fail-to-pass and pass-to-pass tests.
  • Collection: The collection targets 17 user-facing JavaScript or TypeScript repositories covering visual libraries such as mapping, plotting, diagrams, and syntax highlighting, with no Python code.
  • Collection: 1,478 candidate task instances were identified by filtering issue–pull-request pairs for image or video links in issue text or testing code.
  • Collection: The dataset pipeline adds Node.js and Chrome for JavaScript execution, visual testing, and browser rendering, then uses repository-specific installation and testing scripts.
  • Collection: Repeated validation reduced 679 candidates to 643 viable instances, and human inspection removed 24 impossible tasks, yielding 619 final instances.
  • Dataset characteristics: The benchmark contains 619 tasks from 17 repositories, including visualization, diagram, UI, mapping, and syntax-highlighting libraries; its test split has 517 tasks from 12 repositories.
  • Features: SWE-bench M includes 862 problem-statement images spanning screenshots, code snippets, error messages, diagrams, art, maps, and data visualizations.
  • Features: 221 instances contain multiple pictures and 70 contain videos, while 67 carbon instances use actual-versus-expected image pairs to show visual discrepancies.

3 EVALUATING ON SWE-BENCH M

The evaluation adapts leading SWE-bench systems to multimodal JavaScript tasks, exposing how Python-centric workflows and specialized program-analysis tools constrain generalization.

  • Evaluation setup: The study adapts the top four open-source SWE-bench systems for evaluation on SWE-bench M.Adaptation feasibility is assessed for systems including SWE-agent, Agentless, Moatless, AutoCodeRover, and RAG.
  • Evaluation setup: SWE-agent is evaluated with Base, JavaScript-specific, and multimodal agent-computer interfaces.The multimodal configuration extends message processing and history functionality to support images.
  • System adaptations: Agentless requires a custom JavaScript parser and JavaScript-oriented prompts because replacing Python’s AST module alone yields 0% resolved.The custom parser was written from scratch to address Python-centric assumptions in localization.
  • System adaptations: Moatless and AutoCodeRover are not benchmarked because their Python-based AST or program-analysis tools require extensive redesign for JavaScript.Moatless’s language-agnostic code graph still depends on a Python-oriented AST representation, while AutoCodeRover relies on repository-specific Python analysis.
  • Evaluation setup: Evaluations use GPT-4o and Claude 3.5 Sonnet, reporting % Resolved and average per-instance inference cost.The models are selected for long-context, multimodal processing, and structured prediction capabilities.

4 RESULTS

Results show low overall performance on SWE-bench M, with interactive SWE-agent systems outperforming RAG and Agentless while visual inputs and multimodal tools help selectively.

  • Overall performance: 11.5% resolved on average for interactive SWE-agent systems, versus 5.5% for RAG and 3.9% for Agentless.Overall performance on SWE-bench M remains relatively low, despite the gap between interactive and non-interactive systems.
  • Overall performance: SWE-agent configurations show similar absolute performance, while JavaScript-specific customizations have minimal overall impact.Development-set ablations nevertheless indicate that multimodal tooling improves performance in some cases, leaving the overall picture ambiguous.
  • Visual inputs: Images substantially improve performance when visual information is necessary, especially for tasks containing non-textual visuals.For SWE-agent JS, non-textual-image performance drops from 13.0% to 8.7% without images, while primarily textual images remain at 23.1%.
  • Cross-language generalization: Python-centric localization modules fail on JavaScript patterns and reduce Agentless JS performance to 4.6%.Agentless JS achieves a file-localization F1 score of 0.142, compared with 0.367 for SWE-agent, and 28% of reference solutions edit multiple file types.
  • Implications: The results suggest that generalizable software-engineering agents should emphasize flexible interaction tools rather than manually engineered problem-solving pipelines.This conclusion follows from SWE-agent’s stronger performance and Agentless’s localization struggles.
  • Multimodal tools: Multimodal tools dedicate approximately 20% of SWE-agent actions to building and screenshotting websites.With GPT-4o, SWE-agent M builds websites and takes screenshots for 38.3% of instances, averaging 7.5 screenshots on those instances.
  • Multimodal tools: SWE-agent’s multimodal workflow increases complexity, producing an almost threefold rise in cost-limit terminations.The web-specific tools degrade performance with Claude 3.5 Sonnet, while GPT-4o improves despite some premature cost terminations.
  • Multimodal tools: Multimodal tools almost double GPT-4o’s correct-submission fraction to 19.6% from 10.4% without image inputs.This improvement is not observed for the Claude 3.5 Sonnet-based SWE-agent.

5 RELATED WORK

Prior work studies multimodal code generation, web interaction, and software engineering largely as separate tasks. SWE-bench M couples visual web interaction with repository-level coding through image-grounded agent iteration.

  • Multimodal code benchmarks extend code generation with dimensions such as robust evaluation and multilinguality as conventional performance plateaus.Claude Sonnet 3.5 is reported at 92% on HumanEval.
  • Earlier systems translated user interfaces into front-end code, while newer LM agents separately addressed web navigation and software engineering.
  • SWE-bench M is presented as the first benchmark meaningfully coupling web interaction and software engineering tasks.
  • Unlike text-only webpage tools, SWE-bench M supports meaningful iteration between code updates and their rendered visual effects.

6 CONCLUSION

The paper introduces SWE-bench M as a multimodal benchmark for real-world, user-facing JavaScript software engineering. Its diverse visual tasks expose challenges for current systems and motivate more general-purpose, language-agnostic agents.

  • SWE-bench M contains 619 task instances from 17 user-facing JavaScript repositories covering interfaces, visualization, art, mapping, and syntax highlighting.The benchmark evaluates real-world software engineering tasks involving visual elements.
  • The development split contains 100 tasks from five open-source repositories selected to provide roughly one-fifth of the test split and equivalent test repositories.
  • SWE-bench M includes visual testing, with 69 Chart.js and openlayers tasks using 273 reference screenshots.Pixel-level testing uses screenshots to verify functional correctness.
  • Visual-testing libraries accommodate browser-rendering discrepancies through pixel tolerances and region-specific comparisons.
  • SWE-bench M task instances span multiple years, with most drawn from 2019 to 2022.
  • Reference solutions modify multiple file types in 174 task instances, unlike SWE-bench reference solutions that exclusively edit Python files.Non-code files are excluded from this count.

A.3 ADDITIONAL ANALYSES

Additional analyses characterize SWE-bench M as more structurally diverse and generally more demanding than SWE-bench. The dataset combines larger, more distributed code changes with visual and repository-specific problem types.

  • 28% of SWE-bench M task instances modify two or more file types.
  • 40% of SWE-bench M tasks change one file and 32.5% change one function, compared with 83% and 65% in SWE-bench.SWE-bench M has a longer tail of multi-file and multi-function edits.
  • SWE-bench M’s median codebase has 535k lines of code versus 734k for SWE-bench, while its long tail includes larger codebases.
  • 75% of SWE-bench M tasks have two or fewer fail-to-pass tests, compared with 80% for SWE-bench.SWE-bench M issues also tend to be shorter, attributed to images complementing textual communication.
  • Bug fixes constitute a quarter of tags, alongside repository-specific issues such as accessibility, modeling, and WebGL rendering.
  • WebGL-tagged p5.js problems can require visual reasoning because behavior differs between the default P2D renderer and WebGL.

B.1 STATISTICS

The appendix details SWE-bench M’s filtering, asset collection, environment setup, and adaptations of existing agents for JavaScript and multimodal inputs. These changes address reproducibility, browser execution, visual testing, and language-specific tooling.

  • Data collection: Candidate task instances are run five consecutive times and removed when test cases or pass/fail outcomes vary across runs.
  • Data collection: Issue descriptions may include browser screenshots, animated images, and online IDE links that communicate visual behavior or reproduction steps.Around 17% of instances contain at least one online IDE link, and 15% contain more than one.
  • Data collection: The collection process extracts resources from CodeSandbox, JSFiddle, CodePen, StackBlitz, and the p5.js editor through scripts, browser automation, or manual downloading.
  • Agent adaptations: Agentless and AutoCodeRover require replacing Python-specific AST tooling, whereas Aider generalizes through Tree-sitter language support.
  • Agent adaptations: SWE-bench M supplies reproduction code in files such as index.html, script.js, and style.css, then initializes repositories as npm packages.
  • Agent adaptations: RAG configurations are selected by searching context lengths of 32K, 64K, and 100K with or without image inputs.

C.3 FURTHER ANALYSES

Further analyses characterize SWE-bench M images, assess potential solution leakage, and report repository- and category-level annotation results. The images are dominated by code and webpage screenshots, while other visual categories are concentrated in particular repositories.

  • Temporal Analysis: Temporal analysis finds no indication of solution leakage, although uneven repository distributions complicate year-based comparisons.All systems generally perform better on newer instances, with a pronounced peak in 2024.
  • Temporal Analysis: Post-cutoff performance exceeds reweighted pre-cutoff performance for all tested systems after accounting for repository-distribution shifts.The comparison uses October 2023 as GPT-4o’s training cutoff.
  • Image Categorization: Human annotation categorizes images into eight visual-content classes, applied to 862 problem-statement images sampled across SWE-bench M.The categories include code, webpage, geospatial visualization, diagram, data visualization, artwork or photography, error trace, and other.
  • Image Categorization: Most images are webpage or code screenshots, reflecting repositories focused on linting, syntax highlighting, and incorrectly rendered web elements.Other categories are more repository-specific: maps occur in openlayers, charts in Chart.js, and artistic outputs in p5.js.

D.2 IS AN IMAGE REPRESENTABLE AS TEXT?

The analysis tests whether task images can be faithfully represented as text without losing information relevant to solving the issue. Most images are not text-representable because visual properties such as color, spatial arrangement, and patterns can matter.

  • Annotation and Results: The criterion excludes images whose meaningful visuals or patterns, monochrome conditions, or non-text content cannot be conveyed without information loss.Annotators were instructed to answer whether the image could be represented faithfully as text.
  • Visual Information: OCR can process some code and webpage images, but diagrams, plots, and artwork typically cannot be reduced to text faithfully.Examples include icons, bar plots, graphics without text, syntax coloring, and error-message screenshots.
  • Results by Repository and Category: Across repositories, most images were not text-representable except in prettier, with some repositories having no “Yes” labels.Error messages were more often representable, whereas code and web screenshots retained important coloring or spatial organization.

D.3 IMAGE NECESSITY.

Human evaluation measures whether images are necessary for solving image-containing task instances by comparing judgments before and after viewing them. Most evaluated instances were judged to require their images.

  • Annotation and Results: Annotators judged necessity by reconsidering solvability after viewing the task’s associated images, with instances unsolvable both before and after viewing removed.The workflow examines the problem statement, optionally the codebase and patches, then the images.
  • Examples: A PrismJS screenshot supplies concrete evidence about incorrect syntax-highlighting colors that the textual issue description does not fully specify.The image shows which quotes should not be highlighted gray.
  • Examples: A Quarto image was judged unnecessary because it was only a reproduction-code link and conveyed no additional information about the task.The image remained convenient for reproducing the issue but did not aid task understanding.
  • Annotation and Results: 83.5% of 557 image-containing task instances were labeled as requiring the associated image to solve the problem.465 instances were labeled necessary and 93 were labeled not necessary.

D.4 TASK DIFFICULTY

SWE-bench M spans quick fixes through large refactorings, and human annotations identify patch size, task description, visual localization, and edit precision as difficulty signals. The sampled difficulty labels show substantial annotator agreement.

  • Annotation Procedure: Three annotators assign one of four time-to-fix labels to 100 repository-stratified task instances, using majority vote or the middle label when no majority exists.Annotators assess the problem statement, images, gold patch, and codebase without considering test-writing time.
  • Annotation Procedure: Fleiss’ kappa is 0.78, indicating substantial agreement among the three annotators.The agreement statistic summarizes the completed difficulty-labeling procedure.
  • Difficulty Signals: Smaller gold patches generally correspond to shorter estimated fixes, while the >4-hour tasks have substantially larger patch sizes.The <15-minute examples mostly change 1–10 lines, whereas >4-hour examples range from 186 to 2682 changed lines, with one auto-generated lockfile aberration among the short tasks.
  • Difficulty Signals: Detailed descriptions, reproduction code, and screenshots can help developers localize errant code faster, while precise edits may remain time-consuming despite being small.Annotators also considered the kinds of entities and symbols changed when estimating effort.
  • Implications: The resulting range from quick fixes to multi-module refactoring supports using SWE-bench M to track model and agent capability improvements.This conclusion follows the observed spread of estimated task difficulty.

E LIMITATIONS

SWE-bench M is limited in scope to 617 JavaScript tasks from 17 libraries, while future work could expand its languages, modalities, and task coverage; such expansion was labor-intensive.

  • Broader scope: 617 task instances from 17 JavaScript libraries leave scope for adding other programming languages, modalities, and JavaScript domains.The paper identifies Python, C++, Rust, audio, and additional JavaScript-library domains as possible extensions.
  • Improved models and environments: Future enhancements could improve SWE-bench M performance through stronger underlying models and richer agent environments.Suggested environmental improvements include better browsing capabilities and more tools.
Loading 2410.03859v1…