PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepRepoQA: Code Repository Question Answering with Deep Agent Exploration

Weihan Peng, Yuling Shi, Yingwei Ma, Longfei Yun, Beijun Shen, Xiaodong Gu

arXiv:2608.24221v1cs.SEcs.CLcs.PL

TL;DR

Repository QA requires navigating interconnected codebases and tracing dependencies beyond surface-level retrieval. DeepRepoQA uses MCTS-guided agentic exploration, achieving consistent 4-7% improvements over state-of-the-art agent baselines across multiple LLMs.

  • Problem

    Existing repository QA methods rely mainly on surface-level retrieval and can miss evidence or limit reasoning across files and long-range dependencies.

  • Method

    DeepRepoQA casts repository QA as search-and-verify, using LLM agents and MCTS to navigate, inspect, evaluate, and synthesize cited code evidence.

  • Results

    DEEPREPOQA achieves consistent 4-7% improvements over SOTA agent baselines across multiple LLMs, with gains concentrated on Correctness, Completeness and Reasoning.

  • Takeaways & Limitations

    The paper shows systematic tree search is effective for repository-level reasoning and provides a foundation for more capable software agents.

  • Takeaways & Limitations

    Data contamination is a primary threat because benchmark repositories or similar code may have appeared in LLM pre-training.

Abstract

from arXiv · show

Answering developer questions about a software repository is a critical yet under-explored problem in software engineering. While existing repository understanding methods have advanced the field, they predominantly rely on surface-level code retrieval and lack the ability for deep reasoning over multiple files, complex software architectures, and grounding answers in long-range code dependencies. To address these limitations, we propose DeepRepoQA, a novel question answering (QA) framework for repository-level code understanding. DeepRepoQA builds on an agentic framework where LLM agents find answers through a systematic tree search over the repository structure. A Monte-Carlo Tree Search (MCTS) mechanism is employed to empower agents to dynamically search, navigate, and inspect code, enabling effective multi-hop reasoning over long-range code dependencies. Comprehensive experiments on the SWE-QA benchmark demonstrate substantial performance gains over strong baselines, validating the effectiveness of systematic MCTS-guided exploration for multi-hop repository reasoning.

1 Introduction

DEEPREPOQA frames repository question answering as systematic deep search-and-verification over repository structure, addressing the limitations of local retrieval and single-path exploration. Its MCTS-guided agentic framework enables verifiable multi-hop reasoning over cross-file dependencies and improves performance across SWE-QA evaluations.

  • Motivation: Repository question answering is critical yet underexplored because developers must navigate interconnected codebases, trace cross-file dependencies, and synthesize architectural knowledge.Repository-aware assistants are gaining interest as LLMs advance.
  • Limitations: Function-level methods and basic RAG surface local code fragments, making them ill-suited to questions requiring deep reasoning across files and call chains.Single-path agents can miss evidence outside the chosen trajectory and face limited reasoning depth, retrieval bias, and incomplete dependency synthesis.
  • Approach: DEEPREPOQA casts repository QA as a deep search-and-verify process in which LLM agents systematically explore repository structure and synthesize evidence before answering.Its action space includes semantic search and structured exploration, while collected evidence is curated into a memory path that supports grounded synthesis with cited code spans.
  • Evaluation: 7.08 points is DEEPREPOQA’s largest reported gain over SWE-agent, achieved on Qwen3-Coder-480B-A35B-Instruct in the SWE-QA evaluation.The framework improves consistently over ReAct-style agents across evaluated backbones and surpasses or matches leading commercial tools.
  • Evaluation: DEEPREPOQA shows strong computational cost and answer quality, while a case study indicates that MCTS-guided verification mitigates retrieval bias through cross-file evidence.The evaluation reports consistent improvements over state-of-the-art baselines on SWE-QA.
  • Approach: MCTS-guided exploration combines LLM feedback and memory to support efficient, repository-grounded search and address the limitations of single-path ReAct-style agents.Ablations identify MCTS as critical, with semantic search, the Evaluation Agent, and the Perception Agent further improving stability and efficiency.

2 Related Work

Repository-level code understanding supports code generation, translation, and issue resolution, but existing approaches generally emphasize retrieval, navigation, synthesis, or localized QA rather than comprehensive multi-hop reasoning. DeepRepoQA addresses this gap by formulating repository QA as planning with multi-module MCTS for systematic exploration and deliberate inference across entire codebases.

  • Repository-Level Code Understanding: Repository-level code understanding supports automated code generation, code translation, and issue resolution.Existing approaches include retrieval-augmented context gathering and agent-based repository navigation.
  • Repository-Level Code Understanding: Existing methods primarily target code synthesis and modification rather than comprehensive, multi-hop question answering.LongCodeZip, for example, compresses long code contexts by retaining task-relevant functions and code blocks.
  • Code Question Answering: Most code question-answering work focuses on snippet- or function-level understanding, while repository-level QA addresses architectural questions across entire codebases.Repository-level QA requires multi-file, multi-hop reasoning across repositories.
  • DeepRepoQA: DeepRepoQA formulates repository QA as a planning problem and uses multi-module MCTS to systematically explore complex repository reasoning spaces.This enables deliberate, multi-hop inference across entire repositories beyond simple retrieval or localized analysis.

3 Methodology

DeepRepoQA uses an MCTS-grounded agentic framework to transform repository question answering into iterative, multi-path exploration. It combines specialized agents, structured repository representations, and targeted actions to support reasoning over complex, multi-hop code dependencies.

  • Agentic MCTS Framework: DeepRepoQA coordinates Perception, Planning, Execution, and Evaluation agents within an MCTS framework for iterative repository exploration.The agents analyze state, propose actions, gather code evidence, and score results to guide subsequent exploration.
  • Agentic MCTS Framework: Each exploration step builds a compact search tree of possible search, navigation, inspection, or stopping actions, enabling multi-path reasoning over complex questions.The process replaces linear retrieval with a multi-path reasoning process for multi-hop repository questions.
  • Repository Representation: The repository pipeline parses files, directories, classes, functions, and snippets into structured representations using language-specific parsers, heuristics, and Tree-sitter.The representations are designed for efficient querying and downstream reasoning over the raw repository.
  • Repository Representation: DeepRepoQA combines index-based AST lookup for precise structural retrieval with a RAG-style semantic retriever for conceptual code search.Index-based operations include FindClass, FindFunction, and FindCodeSnippet, while SemanticSearch retrieves conceptually related code or documentation.
  • Action Space: The agent’s six core capabilities are FindClass, FindFunction, FindCodeSnippet, SemanticSearch, ViewCode, and Finish.These actions support precise identifier lookups, semantic retrieval, code inspection, and answer completion.
  • MCTS Simulation: DeepRepoQA uses a single-step simulation in which Perception, Planning, Execution with Filtering, and Evaluation directly set Q(s, a) ← V(s, a), reducing compute without multi-step rollouts.The iterative process focuses search on promising reasoning paths for more effective and efficient question answering.

4 Experimental Setup

The experiments assess DEEPREPOQA on repository-scale code question answering through five research questions, comparing it with retrieval, agent, prompting, and commercial baselines. Evaluation uses SWE-QA, multi-judge scoring across five answer-quality dimensions, and controlled implementation settings.

  • Research questions: The evaluation examines repository-scale QA, component contributions, exploration iterations, efficiency, and performance across question types.These five dimensions are formalized as RQ1–RQ5.
  • Baselines: DEEPREPOQA is compared with Direct Prompting, RAG-based Methods, Agent-based Methods, and Commercial Tools.The RAG baselines include Function Chunking RAG and Sliding Window RAG, while agent baselines include SWE-AGENT and OPENHANDS.
  • Implementation controls: Open-source methods use the same four underlying LLMs, while agent-based methods are limited to 15 interaction iterations and DEEPREPOQA uses only the adopted reasoning trajectory.The shared models are GLM-4.6, Kimi K2, Qwen3-Coder-480B-A35B-Instruct, and GPT-5.1.
  • Benchmark: All methods are evaluated on SWE-QA, a repository-level benchmark originally containing 576 question-answer pairs from open-source Python repositories.Three additional repositories—conan, streamlink, and reflex—were incorporated to improve coverage and mitigate potential data leakage.

5 Results

DEEPREPOQA achieves state-of-the-art open-source repository QA performance, approaching commercial tools through stronger correctness, completeness, and reasoning. Ablations, exploration-budget studies, action analysis, efficiency measurements, and question-type results support the value of its MCTS-guided, feedback-driven design.

  • Overall performance: With GPT-5.1, DEEPREPOQA scores 70.06 overall, surpassing Tongyi Lingma at 69.12 and approaching Cursor at 70.71.It remains the top open-source method across the reported backbones and approaches leading commercial tools.
  • Overall performance: DEEPREPOQA’s largest gains occur in Correctness, Completeness, and Reasoning, where it outperforms direct prompting, RAG variants, and agentic baselines.RAG variants can retrieve irrelevant code while missing critical segments, whereas MCTS-guided exploration supports more accurate, comprehensive, and rigorously reasoned answers.
  • Ablation studies: Removing any component reduces performance by −1.07 to −3.50, showing that MCTS, perception, evaluation, and semantic search each contribute to the integrated system.The MCTS ablation confirms the importance of exploration-exploitation balance, while evaluation-agent removal supports learned value estimation over expensive rollouts.
  • Exploration behavior: Trajectories emphasize SemanticSearch, FindClass, FindCodeSnippet, and ViewCode, indicating broad exploration, targeted navigation, and evidence verification.Less frequent FindFunction and Finish actions show that the agent spends most of its budget gathering and confirming evidence.
  • Efficiency and question types: DEEPREPOQA uses fewer input tokens than several agents and commercial tools at 15 iterations while maintaining comparable total cost, and performs strongly across question types.“Why” and “How” questions receive the highest average scores, reflecting strength in design rationale and system or algorithmic reasoning.

6 Discussion

The discussion supports the reliability of DeepRepoQA’s evaluation through strong agreement among LLM judges and human experts, while illustrating its repository-grounded advantages and recurring failure modes. It also identifies data contamination as a primary internal-validity threat and describes mitigations involving model diversity and a retrieval-free baseline.

  • Evaluation reliability: LLM judges achieved perfect agreement on system rankings, with ρoverall = 1.The ranking-consistency score uses a Spearman-style measure defined in Appendix C.
  • Evaluation reliability: LLM and human panels correlated strongly at Pearson r = 0.972 and agreed on 88.2% of pairwise comparisons, with Cohen’s κ = 0.725.The reported κ indicates substantial agreement between the panels.
  • Qualitative analysis: On a StandardScaler question, DEEPREPOQA correctly identified that fit ignores y, transform accepts only X, and TransformedTargetRegressor handles target transformation.The baseline instead retrieved feature-vector snippets and incorrectly treated the target vector the same way.
  • Error analysis: Appendix E identifies four recurring failure modes: wrong location (F1), adjacent-concept retrieval (F2), keyword-only exploration (F3), and open-ended design (F4).The analysis also characterizes symbol-anchoring and early-retrieval patterns in high-scoring answers.
  • Threats to validity: Data contamination is identified as a primary internal-validity threat, mitigated by evaluating open-weight and proprietary models and including a retrieval-free baseline.These measures reduce the chance of shared memorization and isolate gains from retrieval, respectively.

7 Conclusion … C LLM-as-a-Judge Reliability

DEEPREPOQA frames repository-level code question answering as Monte-Carlo Tree Search planning and reports consistent gains over state-of-the-art agent baselines. The accompanying material illustrates retrieval, token usage, and reliability evaluation procedures.

  • 7 Conclusion: DEEPREPOQA reformulates repository-level code question answering as a planning problem solved with Monte-Carlo Tree Search.The framework is agent-based and uses MCTS to support repository-level question answering.
  • 7 Conclusion: 4-7% improvements over SOTA agent baselines are reported across multiple LLMs, concentrated on Correctness, Completeness, and Reasoning dimensions.These dimensions are identified as critical for multi-hop repository understanding.
  • A MCTS Workflow Illustration: StandardScaler applies z = (x - u) / s, where u is the training-sample mean and s is the standard deviation.The implementation subtracts the learned mean when enabled and divides by the learned scale when standardization is enabled.
  • A MCTS Workflow Illustration: The MCTS workflow illustration contrasts retrieved evidence and generated answers from OpenHands and DEEPREPOQA.The case study concerns scikit-learn StandardScaler, whose transformation centers values and scales them by standard deviation.
  • B Token Usage: Token usage per question is reported as input/output tokens, with lower usage designated as better.The supplied material provides the reporting convention but no values.
  • C LLM-as-a-Judge Reliability: 360 answers are evaluated using human and LLM assessments from a GPT-5.1 backbone across 60 questions and 6 systems.The reliability study samples one question from each of four intent categories for each of 15 repositories, yielding 15 × 4 = 60 questions.

C.1 Intra-panel consistency — Krippendorff’s α (interval level)

The analysis reports Krippendorff’s α interval-level intra-panel reliability for human, LLM, and combined six-rater panels across evaluation dimensions. Both individual panels exceed substantial reliability in four of five dimensions, while clarity is consistently most difficult.

  • Intra-panel consistency: Krippendorff’s α is reported at the interval level for human, LLM, and combined 6-rater panels across evaluation dimensions.The results are broken down by evaluation dimension.
  • Intra-panel consistency: 4/5 dimensions exceed the 0.80 substantial-reliability threshold for both human and LLM panels.This indicates broadly substantial agreement within each panel.
  • Intra-panel consistency: Clarity is consistently the hardest dimension for both human and LLM panels, indicating a domain-inherent difficulty.The same pattern appears across both panel types.

C.2 Individual cross-group Spearman ρ matrix (per-answer, all p < 0.001)

The matrix reports per-answer Spearman ρ values between LLM judges and human experts, with all p-values below 0.001. Human-expert agreement averages 0.755, while the lower Claude–Expert1 correlation reflects Expert 1’s idiosyncratic rating style.

  • Cross-group matrix: All per-answer Spearman ρ associations between each LLM judge and each human expert are statistically significant at p < 0.001.The matrix reports these cross-group correlations for every LLM judge–human expert pair.
  • Within-human agreement: 0.755 is the mean within-human pairwise ρ across the three expert pairs.The pairwise correlations are E1–E2 = 0.680, E1–E3 = 0.668, and E2–E3 = 0.917.
  • Expert-specific variation: 0.661 is the lower Claude–Expert1 ρ, mirroring Expert 1’s weak correlations with the other human experts.Expert 1’s within-human correlations with Experts 2 and 3 are 0.680 and 0.668, respectively, suggesting an idiosyncratic rating style specific to Expert 1.

C.3 Panel-consensus agreement (mean-of-3 vs mean-of-3) · C.4 Pairwise preference agreement

The evaluation compares human and LLM panels both through mean-of-three consensus scores and through system-level pairwise preferences. The consensus scores closely align in absolute values and rankings, while pairwise agreement is measured by matching panel verdicts.

  • C.3 Panel-consensus agreement (mean-of-3 vs mean-of-3): Human-consensus and LLM-consensus scores average three ratings from their respective panels on a 0–100 scale.Table C3 compares these two panel-level consensus signals.
  • C.3 Panel-consensus agreement (mean-of-3 vs mean-of-3): Table C3 reports panel-level agreement between the human-consensus score and the LLM-consensus score.Both scores are defined as means of three experts or three LLM judges, respectively.
  • C.3 Panel-consensus agreement (mean-of-3 vs mean-of-3): 5.22 pts MAE is roughly one-fifth of the typical ~25-point gap between competing systems.This indicates that the panel-level signal is precise enough to distinguish methods reliably.
  • C.3 Panel-consensus agreement (mean-of-3 vs mean-of-3): 0.972 Pearson r and 0.942 Spearman ρ show that LLM consensus closely tracks human consensus in absolute level and ordering.The two panels therefore agree strongly on both score magnitude and system ranking.
  • C.4 Pairwise preference agreement: Pairwise preference agreement asks which system is better by converting each score pair into A > B, tie, or B > A.The comparison focuses on whether the human and LLM panels reach the same verdict.
  • C.4 Pairwise preference agreement: Table C4 reports agreement between human and LLM panels on system-vs-system comparisons.It measures how often the two panels produce the same pairwise preference verdict.

C.5 Length-bias analysis · C.6 Inter-judge ranking consistency

The analysis finds a small but significant association between answer length and LLM–human score differences, while agreement remains excellent across length bins. The three LLM judges show perfect ranking consistency, supported by human verification.

  • C.5 Length-bias analysis: Length-bias analysis uses OLS regression controlling for system and dimension effects plus stratified comparisons of short, medium, and long answers.The OLS specification is score ~ length + system + dim.
  • C.5 Length-bias analysis: 0.179 correlation links answer length with the LLM consensus−human consensus score difference (p = 0.006, r2 = 0.032).The correlation is computed after the specified OLS analysis context.
  • C.5 Length-bias analysis: LLM scores are uniformly lower than human scores across all length bins, indicating greater LLM stringency.The passage reports this pattern for the short, medium, and long answer bins.
  • C.5 Length-bias analysis: Human–LLM agreement is uniformly excellent across answer-length bins, with per-answer Pearson r ≥0.92.Table C5 reports bin shares, mean LLM−human score gaps, and per-answer Pearson correlations.
  • C.6 Inter-judge ranking consistency: Inter-judge analysis evaluates whether the three LLM judges rank baselines consistently, alongside SWE-QA’s human verification protocol.The paper frames this consistency as evidence that the judges are reliable.
  • C.6 Inter-judge ranking consistency: Judge rankings are represented by Ri,j(m), the rank assigned to baseline m by judge j for target model i, and aggregated through Sj,k = Pn i=1 ρi.These quantities define the ranking-consistency computation.
  • C.6 Inter-judge ranking consistency: ρoverall = 1 indicates perfect agreement among the LLM judges.The judge-level agreement complements the human-validation study reported in preceding subsections.

D Cross-language Generalization — Java Results … E.2 Failure modes

On the 30-pair Java subset, DeepRepoQA ranks first among open-source methods, trails only Cursor, and shows a ranking pattern consistent with Python. Its misleading-retrieval handling usually backtracks through low evaluation scores, but delayed correction can fail; manual inspection identifies four recurring failure modes.

  • D Cross-language Generalization — Java Results: DeepRepoQA ranks first among open-source methods on the Java subset and trails only the commercial Cursor system.Table D1 reports that this ranking mirrors the Python results.
  • D Cross-language Generalization — Java Results: 30 QA pairs cover Strata, Fineract, and Shiro under the same evaluation protocol as the Python experiments.The Java subset spans quantitative finance, banking infrastructure, and security framework repositories.
  • D Cross-language Generalization — Java Results: +3.78 over OpenHands, +2.28 over Tongyi Lingma, and a 1.81-point deficit to Cursor quantify DeepRepoQA’s Java results.These comparisons are reported as consistent with the Python setting.
  • E.1 Handling of misleading retrievals: When SemanticSearch returns a misleading snippet, the Evaluation Agent usually recognizes that the retrieved evidence is unhelpful and assigns a low score.This low score initiates backtracking through UCT.
  • E.1 Handling of misleading retrievals: If the Evaluation Agent assigns a high score incorrectly, the trajectory accumulates unhelpful evidence until a delayed UCT backtrack is triggered.The delayed backtrack occurs when the cumulative value falls below that of an unexpanded sibling.
  • E.1 Handling of misleading retrievals: In a minority of cases, the two-stage correction mechanism fails to recover and the trajectory terminates on the wrong path.This is the reported failure mode for misleading retrievals.
  • E.2 Failure modes: Manual inspection of the lowest-scoring trajectories identifies four recurring failure modes, labeled F1–F4, across question categories.Table E1 links each mode to affected categories and root causes, while Sections E.5–E.8 provide representative cases.

E.3 Success patterns … F Prompts

High-scoring repository answers depend on precise early retrieval, symbol-level anchoring, and final source rereading, while failures arise from wrong retrieval, ambiguity, stalled search, or reference-sensitive design evaluation. The prompts therefore enforce stepwise, code-grounded exploration, structured feedback, span identification, node evaluation, and strict JSON scoring.

  • E.3 Success patterns: 100% of top-scoring cases (score ≥73) reached the correct file within the first two hops, while 80% used symbol-level searches with explicit file patterns.Cases requiring 15+ node expansions almost never recovered from incorrect initial retrieval; ViewCode immediately before Finish also correlated with high Correctness.
  • E.4 Representative success case — why/purpose: A natural-language description mapped to one class, SemanticSearch found its file on the first hop, and read code plus docstring produced Correctness of 17/20.The winning path used 8 nodes and 4 steps: SemanticSearch, FindClass, ViewCode, and Finish.
  • E.5 Representative failure case F1 — where/iden-loca: The F1 failure substituted a semantically similar function for parse_args_spec because the agent stopped after one ViewCode without literal-identifier verification.The missing safeguard was a FindFunction(parse_args_spec) check for where-type questions.
  • E.6 Representative failure case F2 — why/purpose: Ambiguous “structured container” language led SemanticSearch to AppWrap instead of AppInfo, and the agent propagated that interpretation across all 17 MCTS nodes.Shared incorrect initial retrieval made backtracking ineffective.
  • E.7 Representative failure case F3 — why/design-rationale: The F3 agent varied SemanticSearch and FindCodeSnippet keywords for over 7 steps, hit the 20-node cap at 332 s, and never used structural navigation.It did not inspect directory structure, follow imports, or discover the HookPosition enum and ordering function.
  • E.8 Representative failure case F4 — how/system-design: For F4, a technically sound alternative redesign was penalized because the LLM judge compared it against one prescribed reference solution.The passage identifies both single-reference evaluation as an artefact and open-ended design questions as inherently underconstrained for code retrieval.
  • F Prompts: The prompts require one action at a time, explicit thoughts, observation before continuing, code-grounded investigation, structured tree-context feedback, relevant-span selection, and conservative five-criterion JSON evaluation.They distinguish exact-name tools from SemanticSearch, require ViewCode when needed, and instruct evaluators to compare candidates directly with references.
Loading 2608.24221v1…