Source-linked AI summary
A Real-World Benchmark for Evaluating Fine-Grained Issue Solving Capabilities of Large Language Models
Ruida Hu, Chao Peng, Jingyi Ren, Bo Jiang, Xiangxin Meng, Qinyun Wu, Pengfei Gao, Xinchen Wang, Cuiyun Gao
TL;DR
Existing benchmarks such as SWE-Bench assess real-world issue solving end to end, leaving limited insight into individual subtasks. FAUN-Eval evaluates question answering, fault localization, and code editing using curated GitHub issue–pull-request data. Its results show that model strengths differ across tasks, while the study is limited to five programming languages.
Problem
Existing end-to-end benchmarks provide limited insight into model performance on individual issue-solving subtasks.
Source: §1 · p. 2Method
Results
Models excel at different tasks, and issue information may mislead models; proprietary models and larger models do not consistently outperform alternatives.
Source: §1 · p. 3, §7 · p. 19Takeaways & Limitations
FAUN-Eval’s task-level assessment is intended to support more informed model selection and optimization for specific software-engineering tasks.
Source: §1 · p. 3FAUN-Eval covers five popular programming languages and representative repositories with well-organized issue management.
Source: §6.2 · p. 18
Abstract
from arXiv · showhide
Automatically resolving software issues is crucial for software development in practice, impacting the software quality and user experience. The process of resolving real-world issues encompasses tasks such as question-answering (QA), fault localization, and code editing. Existing benchmarks such as HumanEval fall short in their ability to assess LLMs' proficiency in solving issues within a codebase. Although benchmarks like SWE-Bench are designed to evaluate the LLMs' capability to handle real-world GitHub issues, the end-to-end evaluation method cannot provide granular insights on the performance of subtasks involved in issue solving. To address existing deficiencies in benchmarking LLMs for practical software engineering tasks, we introduce FAUN-Eval, a benchmark specifically designed to evaluate the Fine-grAined issUe solviNg capabilities of LLMs. FAUN-Eval systematically assesses LLMs across three distinct tasks: QA, fault localization, and code editing. This benchmark is constructed using a dataset curated from 30 well-known GitHub repositories. For each entry, issue and pull request (PR) pairs are meticulously compiled and validated using cross-referencing and keyword verification methods. FAUN-Eval includes 300 entries and employs both LLM and manual checks to ensure data quality. We evaluate ten LLMs with FAUN-Eval, including four closed-source and six open-source models. Our experimental results reveal several key findings. We find that the top-performing LLMs differ across the different tasks. Additionally, features in issues may lead LLMs to generate incorrect information. Moreover, models may vary in their proficiency with texts of different lengths.
1 INTRODUCTION
Real-world issue resolution spans question answering, fault localization, and code editing, but existing end-to-end benchmarks provide limited insight into these subtasks. The paper introduces a task-specific evaluation framework and reports that model strengths differ by task, with issue features sometimes misleading models.
1 INTRODUCTION
- Existing benchmarks focus on small, self-contained problems or evaluate issue resolution end to end, limiting insight into individual subtasks and model weaknesses.Real-world maintenance can require navigating interconnected code and making multi-file changes; limited granular metrics make it harder to identify where models fall short and how to optimize them.Source: §1 · p. 2
- The paper evaluates GitHub issue solving across code question answering, fault localization, and code editing.The tasks respectively assess responses to issue descriptions, identification of a faulty file, and code changes based on issue and pull-request information.Source: §1 · p. 2
- Models excel at different issue-solving tasks, while issue content can mislead them; proprietary models and larger models do not consistently outperform alternatives.The experiments report strong task-specific performance, including Gemini-1.5-Pro and GPT-4o for fault localization and DeepSeek-Coder-V2 for code editing; real-world QA responses often lack accuracy, and some models fail to follow fault-localization instructions.Source: §1 · p. 3
- The framework breaks issue solving into three subtasks to provide clearer insights into model strengths and weaknesses.Its contributions also include a dataset derived from real-world GitHub issues and corresponding pull requests, and detailed per-task metrics intended to guide model selection and optimization.Source: §1 · p. 3
2 BACKGROUND AND RELATED WORK
Prior work explores LLMs and agents for software engineering, while existing benchmarks cover small coding tasks, general code questions, program repair, or end-to-end issue resolution. The cited work identifies a need for benchmarks that evaluate individual subtasks in issue-resolution workflows.
2.1 LLMs and LLM-based Agents
- LLMs and LLM-based agents have been applied to software engineering tasks, with agents using tools, planning, and collaboration to break complex work into specialized subtasks.Examples include assigning roles such as fault localization, code generation, or patch refinement.Source: §2.1 · pp. 3–4
2.2 Existing Benchmarks for Issue Resolving
- The comparison table covers existing benchmarks and FAUN-Eval, with DCM denoting dataset construction method and PQC denoting pair quality confirmation.The caption also defines MCQ as multiple-choice questions, AC as automated crawling, DG as data generation, MR as manual review, FF as feature filtering, and FL as fault localization.Source: Table 1 · p. 4
- Existing task-specific benchmarks cover small-scale code completion and generation, general code questions, or end-to-end program repair rather than issue-solving subtasks.The cited examples include HumanEval, APPS, and MBPP; MMLU, CodeQA, CS1QA, and CodeApex; and Defects4J, TFix, FixJS, and TypeBugs.Source: §2.2 · p. 4, §2.2.1 · p. 4
2.2.1 Task-specific benchmarks.
- SWE-Bench evaluates complex, real-world issue-solving end to end, but provides little guidance about performance on the subtasks underlying issue resolution.The passage reports a 4.33% success rate even for top-performing models and says meaningful model comparisons are difficult.Source: §2.2.2 · pp. 4–5
2.2.2 End-to-end Benchmarks.
- The background identifies a gap in evaluating subtasks within software issue-resolution workflows, motivating work intended to inform model selection and optimization.The cited passage says end-to-end results offer little guidance on how models perform on crucial subtasks.Source: §2.2.2 · p. 5
3 APPROACH
FAUN-Eval is constructed by crawling GitHub data, pairing issues with their repairing pull requests, filtering the collected data, and building task-oriented datasets.
3 APPROACH
- FAUN-Eval’s pipeline crawls issue and pull-request data, pairs issues with repairing pull requests, filters the data, and constructs task-specific datasets.The resulting datasets cover code question-answer pairs, bug reproduction, fault localization, and code editing.Source: §3 · p. 5
3.1 Raw Data Crawling
The raw-data collection selects popular repositories across five programming languages and crawls their GitHub issue and pull-request metadata. The crawl covers more than 800,000 entries, which are later filtered and curated for the benchmark.
3.1 Raw Data Crawling
- The study selects 30 popular repositories across Python, Java, JavaScript, TypeScript, and Go, requiring more than 5,000 stars per repository.Repositories are described as widely used and representative of their respective fields.Source: §3.1.1 · p. 5
3.1.2 GitHub Metadata Crawling.
- GitHub REST API crawling collects issue and pull-request entries from the 30 repositories, totaling over 800,000 entries with metadata and content information.Entries include base information, content information, interaction information, and pull-request-specific details.Source: §3.1.2 · p. 6, Table 2 · p. 6
3.2 Data Pairing
The pairing process links GitHub issues with PRs that fix them by extracting references and verifying fix-related keywords. It uses GitHub’s GraphQL API to retrieve cross-reference data and focuses on same-repository associations in the default branch.
3.2.1 Cross Reference Check.
- GraphQL CrossReferenceEvent queries and parsed entry content are used to identify candidate issue and PR references.The extraction process gathers entry content, parses possible references, then checks candidates against cross-reference events.Source: §3.2.1 · p. 7
- Reference patterns include full URLs, # or GH- identifiers, and owner/repo# identifiers; bare-number forms refer to entries in the current repository.GitHub’s REST API does not provide detailed information about both sides of cross-reference events, so the method uses GraphQL for retrieval.Source: §3.2.1 · pp. 6–7
3.2.2 Keyword Check.
- Issue–PR fix pairs are restricted to same-repository references on the default branch that include a linking keyword.The keyword must be followed by an issue identifier.Source: §3.2.2 · pp. 7–8
3.3 Data Filtering
FAUN-Eval filters crawled issues and fix pairs, then applies LLM-assisted and human semantic checks to select entries that present clear problem–solution pairs. The filtering rules constrain issue content and fix-pair scope, including changes to a single file.
3.3 Data Filtering
- After crawling about 500,000 issues and 100,000 fix pairs, filtering and semantic review select entries for the QA, fault-localization, and code-editing datasets.GPT-4o assesses problem–solution structure, and two researchers independently review its selections.Source: §3.3 · p. 8, §3.3.3 · p. 10
3.3.1 Issue Filtering.
- Issue filters remove reports under 200 characters, overly large entries, duplicates, and issues with more than ten events.Issues with external links are retained only when they contain internal GitHub links.Source: §3.3.1 · pp. 8–9
- Issue filtering also removes robot-created content and issues with only one participant after robot users are removed.The process identifies robot users through the author-info field and common robot-response patterns.Source: §3.3.1 · p. 9
3.3.2 Fix Pair Filtering.
- Fix-pair filtering requires a merged PR and a one-to-one issue–PR match, and retains only PRs that change a single file.The one-to-one check excludes PRs addressing multiple issues and issues addressed by multiple PRs.Source: §3.3.2 · p. 9
- Fix-pair filters exclude documentation-focused changes, test files, non-functional file types, and changes limited to comments or strings.Tree-sitter is used to identify changes limited to comments or strings.Source: §3.3.2 · pp. 9–10
3.3.3 Semantic Check.
- LLM automation and manual review are combined to validate dataset reliability and practical and scientific validity.The review checks whether selected issues and fix pairs actually present and resolve problems.Source: §3.3.3 · p. 10
3.4 Task-Oriented Data Construction
FAUN-Eval organizes data for QA, fault localization, and code editing as stages of solving programming challenges. The datasets pair issue and repository information with task-specific responses, file paths, or patches.
3.4 Task-Oriented Data Construction
- The three datasets represent a sequence from understanding a programming problem through locating and resolving it.Figure 2 presents the data-entry structures for QA, fault localization, and code editing.Source: §3.4 · p. 10, Figure 2 · p. 11
3.4.1 QA Dataset.
- The QA dataset stores dialogues as role–content entries, with issue information and repository metadata providing interaction context.The issue report includes the questioner’s author-association, title, and body; replies include the responder’s author-association and reply body.Source: §3.4.1 · pp. 10–11
3.4.2 Fault Localization Dataset.
- Fault localization uses an issue title and body plus a pruned repository folder tree as input, with the changed file path as output.The tree retains common programming-language files and removes empty folders and hidden files and folders.Source: §3.4.2 · p. 11
3.4.3 Code Editing Dataset.
- Code editing uses issue and PR descriptions plus the pre-change file to produce a patch in SEARCH/REPLACE blocks.The blocks identify the code to change and its replacement.Source: §3.4.3 · p. 11
4 EXPERIMENTAL SETUP
The experimental setup evaluates ten LLMs on QA, fault localization, and code editing, with research questions also addressing input modules and issue length. Task-specific prompts, metrics, and implementation settings define the evaluation procedure.
4 EXPERIMENTAL SETUP
- The study asks how models perform across the three tasks, how input modules affect performance, and how issue length relates to problem-solving performance.These are the three stated research questions.Source: §4 · p. 12
4.1 Model Selection
- Ten proprietary and open-source LLMs are selected, and input lengths are kept within each model’s maximum token limit.The models include GPT and Gemini series, as well as Mistral, CodeQwen, and DeepSeek Coder models.Source: §4.1 · p. 12
4.2 Task Design
- QA uses an issue report as input and a reply body as reference; fault localization predicts changed file paths, while code editing predicts the patch.Fault-localization inputs also include the repository folder tree; code-editing inputs include the pre-modification file and issue and PR descriptions.Source: §4.2 · pp. 12–13
4.3 Prompt Strategies
- Prompts specify task requirements and output formats, including file paths for fault localization and SEARCH/REPLACE diffs for code editing.QA prompts require code blocks to be enclosed in triple backticks when the answer includes code.Source: §4.3 · p. 13
4.4 Evaluation Metrics
- QA is scored with BLEU, ROUGE-L, ROUGE-1, and Edit Similarity; fault localization with Exact Match; and code editing with CodeBLEU.The metrics are selected according to each task’s output type.Source: §4.4 · p. 13
4.5 Implementation Details
- Experiments use temperature 0.2, Azure endpoints for GPT, Google Cloud for Gemini, and eight NVIDIA A100-80G GPUs for open-source models.The Linux machine is equipped with 256GB of memory.Source: §4.5 · pp. 13–14
5 EXPERIMENTAL RESULTS
Across QA, fault localization, and code editing, the leading models differ, and performance also varies with prompt components and input length. The results further show that model size and proprietary status do not consistently predict performance.
5.1 RQ1: Performance of LLMs
- Gemini-1.5-Pro, Gemini-1.5-Flash, and CodeQwen-1.5-Chat lead QA; GPT-4o, Gemini-1.5-Pro, and Mistral-large-2 lead fault localization; code editing leaders are Gemini-1.5-Flash, DeepSeek-Coder-V2, and CodeQwen-1.5-Chat.Table 4 presents the results across all three tasks.Source: Table 4 · p. 14, §5.1 · p. 14
- Closed-source models do not always outperform open-source models, and larger models do not consistently outperform smaller ones across tasks.DeepSeek-Coder-V2 leads in fault localization and code editing but is outperformed by its 33B and 6.7B counterparts in QA.Source: §5.1 · p. 15
- Even Gemini-1.5-Pro’s combined score across four QA metrics is only 0.1826, while CodeQwen-1.5-Chat and DeepSeek-Coder 33B receive zero fault-localization EM after adding explanations despite path-only instructions.Gemini-1.5-Pro’s BLEU score is 0.1208; the two models’ extra explanations prevented exact matches.Source: §5.1 · p. 15
5.2 RQ2: Effectiveness of Different Modules in FAUN-Eval Input on LLM Performance
- Removing the issue body reduced QA metrics by over 10% in most cases and fault-localization Exact Match by 35.93% on average, but changed code-editing CodeBLEU by only 0.85% on average.For DeepSeek-Coder-V2, fault-localization Exact Match fell by 42.14% when the issue body was removed.Source: §5.2 · pp. 15–16
- Removing the issue title improved QA across all tested models, while removing PR titles and bodies reduced code-editing CodeBLEU by 4.95% and 5.96%, respectively, on average.The authors suggest issue titles may misdirect predictions, whereas PR details often describe modifications directly.Source: §5.2 · pp. 16–17
5.3 RQ3: Impact of Input Text Length on Issue-Solving Efficacy
- Across short, medium, and long inputs, performance generally declines with length: all models do worst on long QA texts, and code editing is worse on long than medium texts.The reported length comparisons are shown in Figure 3.Source: Figure 3 · p. 17, §5.3 · p. 17
- Model rankings can cross as input length changes; in fault localization, GPT-4o leads Gemini-1.5-Pro on medium texts, but their order reverses on long texts.The authors report such crossovers frequently in Figure 3.Source: §5.3 · p. 17, Figure 3 · p. 17
6 DISCUSSION
A case study shows that a well-formulated issue title can guide GPT-4o to the correct file when the body and folder tree alone lead it elsewhere. The study’s scope is bounded by data-crawling and model-selection constraints.
6.1 Case Study: Influence of Issue Title
- With only the issue body and folder tree, GPT-4o mislocated an error in BERT modeling; adding a well-formulated title guided it to the correct trainer file.The title’s error phenomenon and the model’s understanding of transformers are described as guidance for the correct location.Source: §6.1 · p. 18
6.2 Threats to Validity
- A small amount of data could not be retrieved because of large files or permission restrictions, but the authors report that this portion often had inherent flaws and did not affect filtering or analysis.Source: §6.2 · p. 18
- FAUN-Eval covers five programming languages and representative repositories with well-organized issue management; the authors selected ten proprietary and open models ranging from 6.7B to 236B.The authors cite experimental-environment and time constraints and note that Codellama and StarCoder were not explored.Source: §6.2 · pp. 18–19
7 CONCLUSION
FAUN-Eval assesses LLMs’ atomic capabilities for real-world GitHub issues across question-answering, fault localization, and code editing. Its results show that models excel at different tasks and that issue information may misguide them.
7 CONCLUSION
- FAUN-Eval evaluates LLMs’ atomic capabilities for real-world GitHub issues across question-answering, fault localization, and code editing.It offers a more comprehensive assessment of issue-solving subproblems than previous benchmarks.Source: §7 · p. 19
- Different models excel at different issue-solving tasks.The benchmark’s results indicate that top performance varies across question-answering, fault localization, and code editing.Source: §7 · p. 19
- Information in GitHub issues may misguide LLMs and lead them to generate incorrect information.Source: §7 · p. 19