PaperlayerView exact revision on arXiv

Source-linked AI summary

Seeing is Fixing: Cross-Modal Reasoning with Multimodal LLMs for Visual Software Issue Fixing

Kai Huang, Jian Zhang, Xiaofei Xie, Chunyang Chen

arXiv:2506.16136v1cs.SE

TL;DR

GUIRepair addresses the limited ability of unimodal APR systems to resolve issues involving visual information. It combines Image2Code for fault comprehension with Code2Image for visual patch validation, resolving 157 SWE-bench M instances with GPT-4o and 175 with o4-mini.

  • Problem

    Existing APR systems mainly target unimodal textual issues, while multimodal issues require visual information to understand bugs and verify fixes.

    Source: §I · p. 1

  • Method

    GUIRepair uses Image2Code to generate reproduced code from issue visuals and Code2Image to render patched GUIs for visual validation.

    Source: §I · p. 2

  • Results

    GUIRepair resolves 157 SWE-bench M instances with GPT-4o and 175 with o4-mini, outperforming the reported open-source and commercial comparison systems.

    Source: §I · p. 2

  • Takeaways & Limitations

    Cross-modal reasoning through understanding and capturing visual information improves automated repair effectiveness on multimodal SWE-bench M issues.

    Source: §VIII · p. 11

    The evaluation manually installs dependencies and deploys runtime environments for each project, an inefficient process.

    Source: §VI · pp. 10–11

Abstract

from arXiv · show

Large language model-(LLM) based automated program repair (APR) techniques have shown promising results in resolving real-world GitHub issue tasks. Existing APR systems are primarily evaluated in unimodal settings (e.g., SWE-bench). However, these autonomous systems struggle to resolve multimodal problem scenarios (e.g., SWE-bench M) due to limitations in interpreting and leveraging visual information. In multimodal scenarios, LLMs need to rely on visual information in the graphical user interface (GUI) to understand bugs and generate fixes. To bridge this gap, we propose GUIRepair, a cross-modal reasoning approach for resolving multimodal issue scenarios by understanding and capturing visual information. Specifically, GUIRepair integrates two key components, Image2Code and Code2Image, to enhance fault comprehension and patch validation. Image2Code extracts relevant project documents based on the issue report, then applies this domain knowledge to generate the reproduced code responsible for the visual symptoms, effectively translating GUI images into executable context for better fault comprehension. Code2Image replays the visual issue scenario using the reproduced code and captures GUI renderings of the patched program to assess whether the fix visually resolves the issue, providing feedback for patch validation. We evaluate GUIRepair on SWE-bench M, and the approach demonstrates significant effectiveness. When utilizing GPT-4o as the base model, GUIRepair solves 157 instances, outperforming the best open-source baseline by 26 instances. Furthermore, when using o4-mini as the base model, GUIRepair can achieve even better results and solve 175 instances, outperforming the top commercial system by 22 instances. This emphasizes the success of our new perspective on incorporating cross-modal reasoning by understanding and capturing visual information to resolve multimodal issues.

I. INTRODUCTION

Multimodal software issues expose a gap in unimodal automated repair because screenshots are needed to understand visual symptoms and verify fixes. GUIRepair addresses this gap through bidirectional image-to-code and code-to-image reasoning, improving SWE-bench M performance.

Motivation

  • Visual issue reports combine text and screenshots, but omitting visual context makes root-cause identification and fix verification difficult.This challenge is especially relevant to front-end libraries whose bugs manifest in graphical user interfaces.Source: §I · p. 1
  • Existing APR systems mainly target unimodal textual tasks, and SWE-agent solves only 12% of SWE-bench M tasks despite strong SWE-bench performance.The cited benchmarks were not designed for issues combining visual and textual information.Source: §I · p. 1

Contribution

  • GUIRepair uses Image2Code to translate screenshots into reproduced code for fault comprehension and Code2Image to render patched GUIs for visual validation.Image2Code retrieves project documentation and generates reproduction code; Code2Image replays the scenario and assesses patched visual behavior.Source: §I · p. 2

Evaluation

  • GUIRepair resolved 157 instances with GPT-4o, achieving the best performance among open-source approaches using the same base model, and resolved 175 instances with o4-mini.With o4-mini, GUIRepair solved 22 more instances than the best commercial system.Source: §I · p. 2

Contribution

  • GUIRepair is presented as the first work explicitly addressing visual cues in front-end-library issue resolution through textual and visual modalities.The contribution frames multimodal reasoning as a new dimension for automated issue resolution.Source: §I · p. 2

II. MOTIVATION

Multimodal issue resolution is difficult because visual symptoms are poorly captured by text and patch correctness lacks reliable visual feedback. The motivation therefore centers on reconstructing code behavior from screenshots and assessing rendered repair effects.

A. Challenge 1: Fault Understanding.

  • 80% of visual symptoms cannot be fully described in text, and images are essential for issue-solving in 83.5% of cases.These figures motivate treating visual information as part of the issue-solving context rather than as optional metadata.Source: A. Challenge 1: Fault Understanding · p. 3
  • Only 17% of SWE-bench M reports include reproduced code, leaving most visual issues without the context linking symptoms to code behavior.The resulting challenge is inferring code behavior from visual symptoms for fault localization.Source: A. Challenge 1: Fault Understanding · p. 3

B. Challenge 2: Patch Validation.

  • Current systems often cannot validate candidate patches directly on visual scenes, even though only one top-ranked patch is selected for developers.SWE-bench uses Pass@1, making effective post-processing selection important.Source: B. Challenge 2: Patch Validation · p. 3
  • Pixel-level visual oracles are difficult to generate for issues such as Chart.js rendering bugs, where candidate fixes can produce subtle but distinct effects.The motivation is to capture and assess visual effects as repair feedback instead of relying solely on generated textual tests.Source: B. Challenge 2: Patch Validation · p. 3, Figure 2 · p. 3

III. APPROACH

GUIRepair augments the traditional APR pipeline with cross-modal reasoning that connects visual issue reports to code and rendered repair outcomes. Its Image2Code component begins by mining project-specific documentation to support reproduction and fault comprehension.

III. APPROACH

  • GUIRepair follows four APR stages: fault comprehension, fault localization, patch generation, and patch validation.It generates reproduction code early, uses injected knowledge for localization, samples candidate patches, and renders GUIs to select promising fixes.Source: §III · p. 3
  • The overall workflow uses reproduced code to replicate the reported scenario and rendered patched GUIs to provide visual feedback for patch selection.This replaces test-case generation as the primary validation mechanism for visual effects.Source: §III · p. 3

A. Fault Comprehension

  • Image2Code addresses fault comprehension through knowledge mining and reproduction generation.The component supplies project-specific context so models can reason about code components behind visual scenarios.Source: A. Fault Comprehension · p. 3

1) Knowledge Mining:

  • Knowledge Mining takes the issue report and project documentation as input, then heuristically selects contextually relevant documents.The selected documents provide project knowledge about components and their usage patterns for subsequent reasoning.Source: 1) Knowledge Mining · pp. 3–4

2) Repro Generation:

Repro Generation creates executable JavaScript context for visual issues by combining retrieved project knowledge with issue-specific reproduction code.

2) Repro Generation:

  • GUIRepair injects mined project documents into prompts before generating reproduction code, supplying domain knowledge for visual fault understanding.The resulting code is intended to expose the project-specific behavior behind the visual symptom.Source: 2) Repro Generation · p. 4
  • The generated reproduction code identifies called components and their organization, helping connect the visual symptom to responsible code elements.For next-1509, the code reveals both the invoked components and how they trigger the faulty scenario.Source: 2) Repro Generation · p. 4
  • GUIRepair generates only the JavaScript needed to reproduce dynamic behavior, then embeds it with text and images into an enhanced Issue Report+.HTML and CSS are treated as mostly static, while JavaScript captures the logic that produces the visual problem.Source: 2) Repro Generation · p. 5
  • Reproduction code may contain minor visual discrepancies, but remains useful when it captures the issue’s essential behavior and supports controlled re-rendering during validation.The controlled local rendering reduces noise from non-essential visual details.Source: 2) Repro Generation · p. 5

B. Fault Localization

GUIRepair localizes faults in two stages, using project-aware retrieval and reproduction context to narrow issues from files to code hunks before patch generation.

1) File Localization:

  • Fault localization first combines chat-model file identification with embedding-based retrieval, then merges their results into suspicious files.The embedding search is restricted to directories identified as likely relevant, reducing whole-repository scanning.Source: 1) File Localization · p. 5
  • When suspicious files exceed the context budget, GUIRepair selects Top-N key bug files using lightweight representations that preserve imports while removing variable declarations.This filtering stage retains project structure and dependencies while limiting the model’s input.Source: 1) File Localization · p. 5

2) Hunk Localization:

  • Hunk localization examines complete key files and identifies responsible classes or functions, while also handling configuration and global code outside those structures.Long class or function bodies are compressed, but the extracted localized hunk restores the complete body or a surrounding context window.Source: 2) Hunk Localization · p. 5

C. Patch Generation

  • Patch generation asks the chat model to produce patches from localized class- or function-level snippets using a Search/Replace format.The generated edits are backfilled into source files and converted into unified git diffs for later validation.Source: C. Patch Generation · pp. 5–6

D. Patch Validation

Patch validation renders each candidate’s GUI behavior, filters unchanged outputs, and uses visual comparison with the issue scenario to select an effective patch.

D. Patch Validation

  • Code2Image validates patches through GUI rendering and patch selection, targeting reliable top-ranked patch choice under Pass@1 evaluation.The process is designed to identify the most promising patch from the generated patch space.Source: D. Patch Validation · p. 6

1) GUI Rendering:

  • GUI rendering replays the reproduced issue in a browser, rebuilds each patched project, and captures one bug image alongside multiple patch images.The practical workflow manually prepares JavaScript environments and uses rebuilt packages to obtain the visual outputs.Source: 1) GUI Rendering · p. 6

2) Patch Selection:

  • Patch selection removes candidates with no pixel-level visual change, then asks the LLM to assess remaining outputs against the issue report.Validation stops when the model identifies an effective patch.Source: 2) Patch Selection · p. 6

A. Research Question •

  • The evaluation studies overall effectiveness, component contributions, and generalization across additional multimodal instances and base models.These are posed as RQ1, RQ2, and RQ3, respectively.Source: A. Research Question • · p. 6

B. Benchmark

The evaluation uses SWE-bench M test, compares GUIRepair with leading open and closed systems, and specifies models, retrieval settings, patch sampling, and browser replay procedures.

B. Benchmark

  • SWE-bench M contains 619 visual JavaScript repair instances from 17 libraries, with 517 test instances used for overall evaluation and 102 development instances for generalization.The benchmark targets visual, user-facing software issues involving images crucial to problem solving.Source: B. Benchmark · p. 7

C. Baseline

  • GUIRepair is compared with 13 state-of-the-art SWE systems, including open-source, commercial or closed-source tools, and a Computer Use Agent.The GPT-4o implementation of Agentless Lite is included as the best open-source complementary baseline.Source: C. Baseline · p. 7

D. Implementation

  • The implementation uses GPT-4o and text-embedding-3-small, with retrieval, localization, patch generation, and validation settings configured for bounded context and diverse candidates.The workflow retrieves Top-6 documents, returns Top-4 bug files, samples up to 40 patch candidates, and submits the Top-1 valid patch.Source: D. Implementation · p. 7
  • The reported benchmark tables cover SWE-bench M test results and resolved multimodal issues by repository.Table I presents test results, while Table II organizes resolved issues by repository.Source: Table I · p. 7, Table II · p. 7

V. EVALUATION

GUIRepair outperforms existing systems on SWE-bench M and combines Image2Code and Code2Image to address multimodal repair. Ablation results evaluate the individual and joint contributions of these components.

A. RQ1: Overall Effectiveness

  • GUIRepair solves 4 more instances than Globant Code Fixer Agent, 17 more than Zencoder, and 26 more than Agentless Lite on SWE-bench M.With the consistent GPT-4o base model, GUIRepair solves 30 more instances than Agentless Lite.Source: A. RQ1: Overall Effectiveness · pp. 7–8
  • GUIRepair resolves unique issues relative to Globant and achieves the best results in 7 repositories, compared with 5 for Globant and 6 for Zencoder.The systems are described as complementary rather than duplicative in repair capability.Source: A. RQ1: Overall Effectiveness · p. 8, Figure 8 · p. 8

B. RQ2: Ablation Study

  • The ablation study compares a basic agentless baseline with Image2Code-only, Code2Image-only, and full GUIRepair variants.The variants isolate visual understanding, visual patch validation, and their combined workflow.Source: B. RQ2: Ablation Study · p. 8

1) Basic Capability of the Agentless Workflow:

GUIRepairbase provides a basic agentless reference workflow for multimodal repair while simplifying fault localization to reduce retrieval and localization costs.

1) Basic Capability of the Agentless Workflow:

2) Improved Capability of the Image2Code Component:

Image2Code improves fault localization by mining project knowledge, while Code2Image validates patches through visual effects; the complete pipeline achieves the strongest result when both components are used together.

2) Improved Capability of the Image2Code Component:

  • GUIRepairI2C solves 10 more instances than GUIRepairbase, a 7.35% improvement, by mining project-specific knowledge to localize visually implicated files.In eslint-15243, Image2Code identifies cli.js and generates a developer-consistent patch, whereas the baseline localizes the wrong file.Source: 2) Improved Capability of the… · p. 9

3) Enhanced Capability of the Code2Image Component:

  • GUIRepairC2I solves 12 more instances than GUIRepairbase, an 8.82% improvement, but costs almost three times more because of multiple sampling and visual feedback.The authors still describe the cost as acceptable and visual-effect capture as effective for validating fixes.Source: 3) Enhanced Capability of the… · p. 9

4) Full Capability of the Complete Pipeline:

  • GUIRepairfull solves 157 instances, outperforming every other ablation variant because Image2Code and Code2Image jointly provide reproduction, localization, and visual patch validation.Most visual instances lack reproduction links, limiting Code2Image-only repair; the two components therefore operate as an interdependent workflow.Source: 4) Full Capability of the Complete… · pp. 9–10, Figure 10 · p. 10
  • For next-4182, Image2Code generates reproduction code and localizes cascader-select.jsx, after which Code2Image captures the visual effect and validates Patch 8.The case illustrates how reproduction code links visual understanding with patch validation.Source: 4) Full Capability of the Complete… · p. 10, Figure 10 · p. 10

C. RQ3: Generalizability Study

GUIRepair generalizes across additional repositories and base models, with stronger reasoning models producing larger gains over the basic workflow.

C. RQ3: Generalizability Study

  • With o4-mini, GUIRepair solves 175 instances, outperforming Globant by 22 instances on SWE-bench M.GUIRepairfull exceeds GUIRepairbase by 21, 13, and 15 instances with GPT-4o, GPT-4.1, and o4-mini, respectively.Source: C. RQ3: Generalizability Study · p. 10, Table V · p. 10, Table IV · p. 10
  • GUIRepair shows effectiveness across different model backbones and diverse projects, with larger gains over GUIRepairbase under stronger reasoning models.The evaluation follows prior work on SWE-bench M development data and includes additional repositories.Source: C. RQ3: Generalizability Study · p. 10

VI. THREATS TO VALIDITY

The paper identifies deployment effort and generalization context as important boundaries, situates GUIRepair among multimodal repair and GUI-testing research, and concludes with its highest benchmark performance.

VI. THREATS TO VALIDITY

  • Manual dependency installation and runtime deployment remain inefficient, although newly released Docker images may reduce this burden.Source: §VI · pp. 10–11

VII. RELATED WORK

  • GUIRepair extends agentless repair workflows with cross-modal reasoning for visual issues, while prior GUI-testing work motivates reproduced-code generation and visual replay for repair.Source: §VII · p. 11

VIII. CONCLUSION

  • GUIRepair combines Image2Code and Code2Image to understand and capture visual information for multimodal software-engineering problems.Source: §VIII · p. 11
  • GUIRepair achieves the highest performance among evaluated open- and closed-source systems on SWE-bench M.Source: §VIII · p. 11
Loading 2506.16136v1…