SWE-QA: Can Language Models Answer Repository-level Code Questions?

Weihan Peng; Yuling Shi; Yuhang Wang; Xinyun Zhang; Beijun Shen; Xiaodong Gu

Source-linked AI summary

SWE-QA: Can Language Models Answer Repository-level Code Questions?

Weihan Peng, Yuling Shi, Yuhang Wang, Xinyun Zhang, Beijun Shen, Xiaodong Gu

arXiv:2509.14635v2Sep 18, 2025cs.CLcs.PLcs.SE

TL;DR

Existing code-QA benchmarks underrepresent the cross-file, architectural, and multi-hop reasoning required by real repositories. SWE-QA addresses this gap with a taxonomy-driven repository-level benchmark and evaluates LLMs under retrieval and agentic context augmentation, with OpenHands achieving 70.79 using GPT-5.1 while revealing continuing difficulties on constrained and multi-hop questions.

Problem
Existing code-QA benchmarks mainly focus on isolated snippets and do not comprehensively capture repository architecture, cross-file dependencies, lifecycle flows, or design rationales.
Method
SWE-QA analyzes developer questions, builds a two-level taxonomy and seed questions, instantiates repository-specific questions, and manually validates the resulting benchmark.
Results
OpenHands achieves 70.79 with GPT-5.1, while direct prompting performs poorly, standard RAG improves performance, and agent frameworks further enhance results.
Takeaways & Limitations
The results highlight the promise of LLMs and especially agentic frameworks for repository-level QA, while exposing challenges in constrained and multi-hop reasoning.
Takeaways & Limitations
SWE-QA focuses on Python repositories, uses LLM-as-Judge with limited human evaluation, relies on static snapshots, and covers only 15 repositories.

Abstract

from arXiv · show

Understanding and reasoning about entire software repositories is an essential capability for intelligent software engineering tools. While existing benchmarks such as CoSQA and CodeQA have advanced the field, they predominantly focus on small, self-contained code snippets. These setups fail to capture the complexity of real-world repositories, where effective understanding and reasoning often require navigating multiple files, understanding software architecture, and grounding answers in long-range code dependencies. In this paper, we present SWE-QA, a repository-level code question answering (QA) benchmark designed to facilitate research on automated QA systems in realistic code environments. SWE-QA involves 576 high-quality question-answer pairs spanning diverse categories, including intention understanding, cross-file reasoning, and multi-hop dependency analysis. To construct SWE-QA, we first crawled 77,100 GitHub issues from 11 popular repositories. Based on an analysis of naturally occurring developer questions extracted from these issues, we developed a two-level taxonomy of repository-level questions and constructed a set of seed questions for each category. For each category, we manually curated and validated questions and collected their corresponding answers. As a prototype application, we further develop SWE-QA-Agent, an agentic framework in which LLM agents reason and act to find answers automatically. We evaluate six advanced LLMs on SWE-QA under various context augmentation strategies. Experimental results highlight the promise of LLMs, particularly our SWE-QA-Agent framework, in addressing repository-level QA, while also revealing open challenges and pointing to future research directions.

1 Introduction

SWE-QA targets repository-level code question answering, where realistic questions require cross-file reasoning, dependency tracing, and architectural understanding. The benchmark combines taxonomy-driven construction with evaluation of LLMs and agentic context augmentation, showing promise alongside persistent reasoning challenges.

Motivation: Existing code-QA benchmarks mainly target isolated snippets, missing repository-scale architecture, cross-file dependencies, lifecycle flows, and design rationales.Repository-level questions require deeper, multi-hop understanding of code structure, semantics, and intent.
Benchmark: SWE-QA constructs a repository-level benchmark from developer-question analysis, taxonomy-guided seed questions, LLM instantiation, and manual screening and refinement.The construction process analyzes GitHub issues and uses a two-level taxonomy to represent diverse repository reasoning requirements.
Evaluation: OpenHands achieves 70.79 with GPT-5.1, while direct prompting performs poorly, standard RAG improves performance, and agent frameworks further enhance results.The benchmark evaluates six advanced LLMs under multiple context augmentation strategies.
Evaluation: Models perform well on open-ended How and Why questions but struggle with constrained What and locational How questions requiring multi-hop reasoning.Performance also varies across repositories, with pylint identified as particularly challenging.
Contributions: SWE-QA’s contributions include a repository-level QA benchmark, a flexible seed-question generation pipeline, and evaluation with retrieval and agent frameworks.The reported benchmark contribution is described as covering comprehensive repository understanding through diverse context augmentation methods.

2 Benchmark Construction

SWE-QA is constructed through a taxonomy-driven, multi-stage pipeline that converts naturally occurring repository questions into human-validated benchmark items. Its design targets cross-file, multi-hop reasoning and supports extensible generation for new repositories.

2.1 Seed Collection and Taxonomy Construction: The benchmark pipeline comprises seed question collection, question instantiation, answer collection, and data validation.Repository structure is parsed, contextual questions are generated, answers are grounded in retrieved repository information, and experts review the resulting pairs.
2.1 Seed Collection and Taxonomy Construction: A two-level taxonomy of 12 user intentions was derived from manually coded GitHub issue questions and used to create reusable seed templates.The taxonomy organizes questions by interrogative type and fine-grained software-engineering intent, including dependency tracing and design rationale.
2.1 Seed Collection and Taxonomy Construction: How questions comprise 35.2% of extracted developer questions, followed by Where at 28.4%, Why at 23.1%, and What at 13.3%.The distribution emphasizes procedural and locational knowledge in repository-level developer queries.
2.2 Question Instantiation and Expansion: Question instances combine seed templates with compact repository subgraphs centered on focal classes or methods to provide targeted structural context.The subgraph is intended to provide sufficient context without overwhelming the generation prompt.
2.5 Statistics of SWE-QA: 77.6% of questions require cross-file knowledge, while 90.9% have reasoning-chain depth greater than one.On average, questions involve 8.71 functions across 3.19 files, with reasoning-chain depth 4.72 and dependency-chain depth 2.96.
2.5 Statistics of SWE-QA: SWE-QA’s multi-faceted design combines repository-level scope, categorized multi-hop questions, cross-file context, and human verification.The benchmark is also presented as a modular pipeline that can generate new repository-level QA instances using static analysis, prompting, and human filtering.

3 Evaluation

The evaluation compares six language models, retrieval methods, agent frameworks, and commercial tools on repository-level QA. Results show that richer context and agentic interaction improve performance, while question type and repository characteristics create substantial difficulty differences.

3.2 Performance of Language Models: Kimi K2 improves from 51.47 with direct prompting to 62.44 with Sliding Window RAG and 67.72 with SWE-agent.The progression illustrates steadily higher answer quality as repository context becomes richer and more structured.
3.2 Performance of Language Models: 70.79 is GPT-5.1’s score with OpenHands, the best reported model-framework combination, while GLM-4.6 with OpenHands reaches 70.15.The comparison shows performance gaps across model and method choices, with GPT-5.1 achieving the strongest overall result.
3.2 Performance of Language Models: Cursor scores 70.66 and Tongyi Lingma 69.07, placing both commercial programming tools among the strongest evaluated systems.These tools combine proprietary models with retrieval and orchestration as system-level baselines.
3.3 Taxonomy-Aware Analysis: OpenHands performs best on Why questions at 69.77 on average, whereas What questions average 65.81 and Where questions 66.76.Design rationale is the strongest Why subtype at 71.48, while Architecture exploration is a difficult What subtype at 61.84.
3.4 Repository-Level Analysis: Repository scores cluster near 70, but flask reaches 75.42 while pylint and conan score 62.01 and 64.51.The observed variance is associated with codebase size, architectural complexity, plugin or hook systems, API clarity, and unconventional patterns.
3.4 Repository-Level Analysis: Repositories from SWE-Bench average 68.59 versus 64.98 for SWE-Bench-Live, a decrease of 3.61 points.The passage relates this gap to reduced data leakage in SWE-Bench-Live, making evaluation more challenging.

4 Related Work

Prior code-QA benchmarks largely target snippets, functions, or other limited contexts, while newer work addresses repository-level understanding. SWE-QA is positioned as a repository-level benchmark emphasizing multi-hop, cross-file reasoning.

4 Related Work: Traditional benchmarks such as CoSQA and CodeQA focus primarily on isolated code snippets or methods rather than complete repositories.Related benchmarks include single- and multi-hop reasoning, real user queries, and template-generated questions within limited code contexts.
4 Related Work: SWE-QA’s evaluation includes question-type and repository-level analyses, with What and Where questions and some repositories presenting greater challenges.The supplied comparison captions identify Architecture exploration and Data / Control-flow as challenging subtypes and SWE-Bench-Live repositories as harder overall.
4 Related Work: Repository-level benchmarks have emerged, but SWE-QA targets realistic multi-file code-understanding tasks with deeper repository context.The paper frames repository-level QA as addressing limitations of snippet-focused evaluations.

5 Conclusion

The paper presents SWE-QA as a benchmark for realistic repository-level code questions across 15 Python repositories. It identifies future expansion to additional languages and dynamically evolving repositories.

5 Conclusion: SWE-QA contains 720 question–answer pairs across 15 Python repositories and targets multi-hop dependencies and cross-file context.The benchmark is intended to evaluate realistic repository-level code understanding.
5 Conclusion: Future work will extend SWE-QA to additional programming languages and dynamically evolving repositories.The stated goal is broader and more robust evaluation of AI-assisted code intelligence tools.

Limitations

SWE-QA’s findings are constrained by its language, evaluation, repository-state, and repository-selection scope.

The benchmark covers only Python repositories, which may limit generalizability to other programming languages.
LLM-as-a-Judge and limited human evaluation may introduce bias and miss nuances of answer quality.
Static code snapshots do not capture the dynamics of evolving repositories.
The 15 selected repositories may not represent smaller or highly domain-specific software projects.

A Benchmark Construction

SWE-QA is constructed by extracting repository structure and issue distributions, then instantiating and curating questions across repositories of varied scale.

The collected issue distribution is uneven, with Sympy contributing 17.8% and Xarray 15.2%, versus 3.5% for Flask and 5.3% for Requests.
Tree-sitter parsing produces a typed graph linking repository elements, containment relations, function calls, callers, and file imports.The graph includes repositories, files, snippets, classes, methods, attributes, functions, parameters, and variables.
Question instantiation combines a focal element’s compact contextual subgraph with seed templates from a taxonomy category.The compact subgraph is intended to provide sufficient context without overwhelming the prompt.
The benchmark spans 15 repositories ranging from 36 to 2,845 files and 598 to 33,994 functions.These repositories provide coverage across substantially different repository scales.
Questions average 28.62 words, answers average 266.64 words, and manually corrected answers have an average edit distance of 42.17 words.

B.1 Method Configurations

The evaluation compares direct prompting, retrieval methods, agent frameworks, and commercial tools under fixed experimental and judging configurations.

The studied approaches include Direct Prompting, Sliding Window RAG, Function Chunking RAG, OpenHands, and SWE-agent.
Direct Prompting answers without retrieved context, providing a baseline for gains from retrieval and agent-based methods.
Sliding Window RAG retrieves overlapping 500-line code segments with 100-line overlap and Top-K=10 retrieval.
Commercial tools are evaluated as current closed-source solutions, with Cursor operating in default auto mode and built-in retrieval and orchestration.
OpenHands and SWE-agent use at most 10 reasoning–action iterations, while decoding temperatures are fixed at 0.
Claude Sonnet 4.5 judges correctness, completeness, relevance, clarity, and coherence, using five evaluations per dimension with majority voting.

C.1 Human Evaluation

Human evaluation supplements LLM-as-a-Judge assessment and shows closely aligned trends across prompting, retrieval, and agent-based methods.

Table 7 reports human evaluation results and high agreement with LLM-as-a-Judge scores.
Agent-based methods such as SWE-agent and OpenHands require substantially more input and output tokens than Direct Prompting.
The study recruited three independent software engineers who rated GPT-5.1-generated answers across the same five dimensions.
Human evaluators found the same overall ordering as automated assessment: RAG improves over Direct Prompting, and Agent Frameworks add further gains.

C.2 Cost Analysis

Repository-level reasoning involves a substantial cost-performance trade-off: agent frameworks use far more tokens than direct prompting or RAG, while improving answer quality. Model-specific tool-calling and planning capabilities also produce large token-usage differences within the same framework.

Agent frameworks consume approximately 100× more tokens than direct responses and 10× more than RAG, while RAG uses over ten times more than direct prompting.
OpenHands improves GLM-4.6 Correctness from 9.48 to 11.91 and Completeness from 7.52 to 12.70 versus Function Chunking RAG.The completeness increase is described as nearly 70%.
The cost-performance trade-off requires practitioners to balance budget constraints against the need for exhaustive code comprehension.
Token usage varies substantially across LLMs using the same agent framework, likely reflecting differences in tool-calling efficiency and planning capabilities.

D Case Study

The case study shows that an agent can identify a repository’s high-level architecture while missing the specialized classes, methods, and dependency chain required for a complete answer. This surface-level understanding produces low correctness and completeness despite acceptable relevance and clarity.

D Case Study: GPT-5.1 with OpenHands identifies high-level architecture and trigger mechanisms but misses translation components and multi-hop implementation details.The missed details include the _NodeUpdater class and compare_references() method.
D Case Study: The case exemplifies a broader limitation: modern LLM agents often grasp surface structures but struggle to extract precise dependency chains across multiple files.The authors collected and annotated 100 typical error cases.
D Case Study: The benchmark’s case question asks how glossary-term consistency validation integrates with the translation processing transform system.
D Case Study: The reference mechanism runs through the Sphinx docutils transform pipeline, where Locale handles standard-domain term references and emits inconsistent_references warnings for mismatches.
D Case Study: The response’s failure to analyze the Locale transform, _NodeUpdater, update_pending_xrefs(), compare_references(), and inconsistent_references yields extremely low correctness and completeness.