PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepRepoQA: Code Repository Question Answering with Deep Agent Exploration

Weihan Peng, Yuling Shi, Yingwei Ma, Longfei Yun, Beijun Shen, Xiaodong Gu

arXiv:2608.24221v1cs.SEcs.CLcs.PL

TL;DR

Repository QA is critical but underexplored, as answering complex questions entails navigating interconnected code and tracing cross-file dependencies. DeepRepoQA uses MCTS-guided agentic tree search and achieves consistent 4-7% improvements over state-of-the-art agent baselines across multiple LLMs.

  • Problem

    Repository QA is critical but underexplored, and answering complex questions requires navigating interconnected codebases and tracing dependencies across files.

    Source: §1 · p. 1

  • Method

    DEEPREPOQA frames repository QA as a search-and-verify process, using LLM agents and MCTS to explore repository structure and synthesize evidence.

    Source: §1 · pp. 1–2

  • Results

    DEEPREPOQA achieves consistent 4-7% improvements over SOTA agent baselines across multiple LLMs, with gains concentrated on Correctness, Completeness and Reasoning.

    Source: §7 · p. 13

  • Takeaways & Limitations

    The results support systematic tree search as effective for repository-level reasoning, with MCTS identified as the primary performance driver.

    Source: §7 · p. 13

    The authors identify pretraining contamination as a threat to internal validity, since models may have seen benchmark repositories or similar code.

    Source: §6.3.1 · p. 12

Abstract

from arXiv · show

Answering developer questions about a software repository is a critical yet under-explored problem in software engineering. While existing repository understanding methods have advanced the field, they predominantly rely on surface-level code retrieval and lack the ability for deep reasoning over multiple files, complex software architectures, and grounding answers in long-range code dependencies. To address these limitations, we propose DeepRepoQA, a novel question answering (QA) framework for repository-level code understanding. DeepRepoQA builds on an agentic framework where LLM agents find answers through a systematic tree search over the repository structure. A Monte-Carlo Tree Search (MCTS) mechanism is employed to empower agents to dynamically search, navigate, and inspect code, enabling effective multi-hop reasoning over long-range code dependencies. Comprehensive experiments on the SWE-QA benchmark demonstrate substantial performance gains over strong baselines, validating the effectiveness of systematic MCTS-guided exploration for multi-hop repository reasoning.

1 Introduction

Repository QA requires reasoning across interconnected files, tracing dependencies, and synthesizing architectural knowledge, but existing approaches often retrieve local fragments or follow constrained single paths. DEEP-REPOQA uses MCTS-guided exploration and LLM feedback to search repositories systematically and ground answers in code evidence.

1 Introduction

  • Repository questions can require navigating large codebases, tracing cross-file dependencies, and synthesizing architectural knowledge, while function-level retrieval and single-path agents can miss evidence or limit reasoning depth.ReAct-style agents interleave action and reasoning, but constrained trajectories can leave cross-file evidence unexplored and make exploration-versus-exploitation difficult.Source: §1 · p. 1
  • DEEP-REPOQA frames repository QA as deep search and verification, with agents using semantic search, structural navigation, and targeted code inspection to gather verifiable evidence.Evidence is kept along a root-to-current-node memory path that structures prompts and supports final answers grounded in cited code spans.Source: §1 · pp. 1–2
  • Its two key innovations are iterative MCTS search-and-verify for multi-hop reasoning and LLM-feedback priors and values that reduce search depth and drift.The framework perceives the current state, plans and executes actions over an index-backed representation, and evaluates utility to steer exploration.Source: §1 · p. 2
  • On SWE-QA, DEEP-REPOQA’s largest gain over SWE-agent was 7.08 points on Qwen3-Coder-480B-A35B-Instruct; with GPT-5.1 and Kimi K2, it surpassed or matched Cursor and Tongyi Lingma.The evaluation covered Kimi K2, GLM-4.6, Qwen3-Coder-480B-A35B-Instruct, and GPT-5.1.Source: §1 · p. 2
  • Ablations identify MCTS as critical, while semantic search and the Evaluation and Perception Agents improve stability and efficiency; analysis reports strong computational cost and answer quality, and a case study shows reduced retrieval bias through cross-file evidence.The framework’s stated contributions include structured, verifiable multi-hop reasoning and comprehensive SWE-QA evaluation against state-of-the-art baselines.Source: §1 · p. 2

2 Related Work

Prior repository-understanding methods largely target code synthesis and modification, while repository-level QA requiring multi-file, multi-hop reasoning remains nascent. DeepRepoQA addresses this gap by treating repository QA as systematic planning and exploration.

Related Work

  • Repository-level code understanding supports code generation, translation, and issue resolution, using retrieval-augmented context, agent navigation, graph-based cross-file modeling, and context-aware completion.Examples include code generation (Shrivastava et al., 2023; Zhang et al., 2024), translation (Wang et al., 2024a, 2025a), issue resolution (Chen et al., 2025b), retrieval (Zhang et al., 2023), navigation (Ma et al., 2024; Yang et al., 2024), graph modeling (Ouyang et al., 2024), and completion (Shrivastava et al., 2023).Source: §2 · p. 2
  • These methods primarily address code synthesis and modification rather than comprehensive multi-hop question answering; LongCodeZip compresses long contexts by retaining task-relevant code blocks.LongCodeZip (Shi et al., 2025) retains task-relevant functions and code blocks when compressing long code contexts.Source: §2 · p. 3
  • Code QA has shifted from snippet- and function-level tasks to repository-level architectural questions, but the field remains nascent and direct LLM or RAG application has inherent limitations.Prior work includes subroutine QA (Bansal et al., 2021), code-QA pre-training, regulatory-code models, repository QA benchmarks and improvements (Chen et al., 2025a), and SWE-Bench ProMax for multilingual refactoring (Shi et al., 2026b); analyses identify limitations of direct LLM and RAG approaches (Andryushchenko et al., 2024).Source: §2 · p. 3
  • DeepRepoQA formulates repository QA as a planning problem and uses multi-module MCTS to systematically explore codebases for deliberate multi-hop inference beyond simple retrieval or localized analysis.The framework targets the vast reasoning space of complex codebases.Source: §2 · p. 3

3 Methodology

DeepRepoQA combines structured repository representations and a six-action toolkit with four collaborating agents in an MCTS-guided search-and-verify loop. Its process evaluates evidence and guides exploration toward promising branches before synthesizing an answer grounded in code citations.

3 Methodology

  • Four specialized agents repeatedly perceive, plan, execute, and evaluate MCTS search steps until evidence supports a citation-grounded answer.The system builds a compact tree of possible actions and uses feedback to guide subsequent exploration, turning repository QA into multi-path reasoning for complex, multi-hop questions.Source: §3 · p. 3, §3.4 · p. 6

3.1.1 Repository Parsing

  • Repository parsing extracts classes, functions, and snippets with language-specific parsers, Tree-sitter ASTs, and heuristics, while resolving calls, inheritance, and module dependencies.This pipeline transforms raw repositories into structured information for querying and analysis.Source: §3.1.1 · p. 3, §3.1 · p. 3

3.1.2 Repository Representation

  • DeepRepoQA combines AST-based structural lookup with embedding-based semantic retrieval to support both precise code-element location and conceptual search.The representation models repository directories and AST positions, and exposes FindClass, FindFunction, FindCodeSnippet, and SemanticSearch operations.Source: §3.1.2 · pp. 3–4

3.2 Action Space

  • The six actions comprise identifier-based lookup, semantic search, code inspection, and Finish, enabling targeted retrieval, conceptual queries, and answer-task termination.The name-based operations are FindClass, FindFunction, and FindCodeSnippet; ViewCode inspects retrieved code regions.Source: §3.2 · p. 4

3.3.1 Perception Agent; 3.3.2 Planning Agent

  • Perception summarizes the search trajectory and sibling branches to flag redundancy, evidence gaps, and promising targets, while planning proposes diverse next actions that avoid repeated trials.Plans are conditioned on long-term goals and expected immediate information gain.Source: §3.3.1 · p. 5, §3.3.2 · p. 5

3.3.3 Execution Agent

  • Execution runs planner-selected actions over the unified structural-semantic index, then filters, ranks, deduplicates, and normalizes outputs into citation-ready evidence with file and line ranges.The curated evidence becomes both the current node’s observation and context for later reasoning.Source: §3.3.3 · p. 5

3.3.4 Evaluation Agent; 3.4.1 Selection; 3.4.2 Expansion

  • The Evaluation Agent scores each action-evidence pair for utility, suggests targeted follow-up steps, and back-propagates feedback to update search-tree values and visit counts.UCT selection balances exploration and exploitation, while expansion uses perception’s summary of trajectory and sibling attempts to identify gaps and promising regions without retaining full sibling transcripts.Source: §3.3.4 · p. 5, §3.4.1 · p. 6, §3.4.2 · p. 6

3.4.3 Simulation; 3.4.4 Back-propagation; 3.4.5 Answer Synthesis and Termination

  • Simulation uses a single four-agent step whose evaluation value directly sets Q(s, a), avoiding multi-step rollouts; back-propagation updates values and visits along the path, and Finish triggers evidence-cited answer synthesis.The final answer must include at least one supporting code span with file and line spans and be checked against the question type; the process iterates until Finish is selected.Source: §3.4.3 · p. 6, §3.4.4 · p. 7, §3.4.5 · p. 7

4 Experimental Setup

The experimental setup evaluates DeepRepoQA across repository-scale answering, component and iteration effects, efficiency, and question types, using SWE-QA, multiple baseline families, and multi-judge scoring. It specifies the dataset, evaluation dimensions, and MCTS configuration.

4.1 Research Questions (RQs)

  • The evaluation asks how well DeepRepoQA answers repository-level questions and how components, exploration iterations, efficiency, and question type affect performance.These are the five research questions guiding the experiments.Source: §4.1 · p. 7

4.2 Baselines and Models

  • The evaluated baselines comprise Direct Prompting, RAG-based Methods, Agent-based Methods, and Commercial Tools; RAG uses two retrieval strategies, while ReAct-style code agents are limited to 15 interaction iterations.Function Chunking RAG uses function- or class-level chunks (Wang et al., 2024b); Sliding Window RAG uses overlapping 500-line chunks with 50-line overlap (Zhang et al., 2023).Source: §4.2 · p. 7

4.3 Dataset

  • SWE-QA comprises 720 question-answer pairs across 15 repositories, expanded from 576 pairs with conan, streamlink, and reflex.The benchmark targets repository-level code question answering in realistic software-development environments.Source: §4.3 · p. 8

4.4 Evaluation Metrics

  • Answer quality is assessed for correctness, completeness, relevance, and clarity, with the final score calculated by averaging outputs from three distinct judge models.Source: §4.4 · p. 9

4.5 Implementation Details

  • MCTS expands at most three children per node and runs for 15 iterations in the main results, with exploration temperature 0.7 and answer-generation temperature 0.SemanticSearch uses voyage-code-3 embeddings; other answer-generation decoding parameters remain at API defaults.Source: §4.5 · p. 9

5 Results

Across SWE-QA results, DeepRepoQA performs strongly against open-source and commercial systems, while ablations, exploration-budget tests, efficiency analysis, and question-type results show how its components and search behavior relate to performance.

5.1 RQ1: Main Results

  • 70.06 with GPT-5.1 puts DeepRepoQA above Tongyi Lingma (69.12), close to Cursor (70.71), and ahead of other open-source systems across the reported backbones.With Kimi K2, it further narrows the gap to commercial tools.Source: §5.1 · pp. 9–10
  • MCTS-guided, feedback-based search improves correctness, completeness, and reasoning over RAG and agentic baselines, while allowing course correction when trajectories become unproductive.RAG variants outperform direct prompting but trail agentic approaches, particularly on Correctness and Completeness; systematic search strengthens substantive dimensions across backbones.Source: §5.1 · pp. 9–10, §5.5 · p. 11

5.2 RQ2: Ablation Study

  • Removing the Evaluation Agent causes the largest drop (3.50 points), followed by MCTS (2.27), the Perception Agent (1.73), and Semantic Search (1.07).The Evaluation Agent guides MCTS path selection through learned value estimation, while the ablations show that every component contributes without redundancy.Source: §5.2 · p. 10

5.3 RQ3: Impact of Exploration Iterations

  • 55.06 at 5 nodes rises to 62.97 at 10 nodes (+7.91) and 65.33 at 20 nodes (+10.27), with performance nearly identical between 20 and 30 nodes.Gains are largest from 5 to 10 nodes and diminish at higher budgets, suggesting a trade-off between computation and performance.Source: §5.3 · pp. 10–11, Table 4 · p. 11
  • Trajectories combine SemanticSearch with FindClass and FindCodeSnippet, use ViewCode for verification, and less often use FindFunction or Finish.This broad exploration followed by verification is linked to improved Correctness and Completeness as the exploration budget increases.Source: §5.3 · pp. 10–11, Figure 3 · p. 11

5.4 RQ4: Efficiency Analysis

  • With 15 iterations, DeepRepoQA uses fewer input tokens than SWE-agent, OpenHands, and commercial tools without iteration limits, while total token cost is comparable to those systems.It uses slightly more output tokens, mainly due to the Perception and Evaluation Agents, while achieving better benchmark performance.Source: §5.4 · p. 11

5.5 RQ5: Performance across Question Types

  • DeepRepoQA performs strongly across question categories; “Why” and “How” have the highest average scores, while “What” and “Where” score somewhat lower.The results suggest strength in design-rationale and system-design or algorithmic-implementation reasoning, with room to improve definition, concept, and feature-location queries.Source: §5.5 · p. 11, Table 5 · p. 12

6 Discussion

The discussion examines LLM-judge reliability, illustrates answer-quality differences and failure modes, and considers threats to internal validity and the scope of model and language generalization.

6.1 Judge Reliability

  • ρoverall = 1 indicates perfect agreement among the three LLM judges on system rankings; LLM and human panels correlate at Pearson r = 0.972 and agree on 88.2% of pairwise comparisons (Cohen’s κ = 0.725).The reliability assessment also reports per-dimension Krippendorff’s α, per-answer human–LLM correlations, panel-consensus agreement, and length-bias analysis in Appendix C.Source: §6.1 · p. 12

6.2 Case Study

  • DEEPREPOQA answers the StandardScaler question by grounding in the API: fit documents “y: None, Ignored,” transform accepts only X, and TransformedTargetRegressor handles target transformation.The baseline retrieves feature-vector snippets and wrongly treats the target vector the same way, missing that StandardScaler ignores fit’s y argument.Source: §6.2 · p. 12
  • Appendix E identifies four recurring failure modes—wrong location (F1), adjacent-concept retrieval (F2), keyword-only exploration (F3), and open-ended design (F4)—and patterns in high-scoring answers.The reported high-scoring patterns are symbol-anchoring and early retrieval.Source: §6.2 · p. 12

6.3.1 Internal Validity

  • The internal-validity threat is pre-training contamination; mitigations are evaluating open-weight and proprietary models, including a retrieval-free baseline, and using diverse, relatively less popular repositories.Source: §6.3.1 · p. 12

6.3.2 External Validity

  • Evaluation of four open- and closed-source models supports generalization beyond one model type, while three-model judging, averaged scores, and judge-correlation checks address evaluation bias.Source: §6.3.2 · p. 13
  • On 30 Java QA pairs across Strata, Fineract, and Shiro, DEEPREPOQA ranks first among open-source methods and behind only Cursor, mirroring its Python ranking.Full Java results are reported in Appendix D.Source: §6.3.2 · p. 13

7 Conclusion … C LLM-as-a-Judge Reliability

DEEPREPOQA frames repository-level code question answering as planning with Monte-Carlo Tree Search, and evaluations show consistent gains over SOTA agent baselines. The section also presents workflow and case-study illustrations, token-usage reporting, and the sampling design for LLM-as-a-Judge reliability evaluations.

7 Conclusion

  • DEEPREPOQA improves 4-7% over SOTA agent baselines across multiple LLMs, with gains concentrated in Correctness, Completeness, and Reasoning.Ablations identify MCTS as the primary performance driver; Perception, Evaluation, and other agents also contribute to stability and efficiency.Source: §7 · p. 13
  • The conclusion frames repository-level code question answering as a planning problem solved with Monte-Carlo Tree Search.The paper presents systematic tree search as effective for repository-level reasoning and as a foundation for more capable software engineering assistants.Source: §7 · p. 13

A MCTS Workflow Illustration

  • Figure A1 illustrates the DEEPREPOQA MCTS workflow.The caption identifies the figure as an illustration of the MCTS workflow.Source: Figure A1 · p. 16
  • Figure A2 presents a scikit-learn StandardScaler case study comparing retrieved evidence and generated answers from OpenHands and DEEPREPOQA.The caption specifies the systems and comparison scope but provides no outcome values.Source: Figure A2 · p. 17

B Token Usage

  • Table B1 reports per-question token usage as input/output tokens, with lower usage designated as better.The supplied caption does not provide token counts.Source: Table B1 · p. 18

C LLM-as-a-Judge Reliability

  • Human and LLM evaluations use GPT-5.1-backbone answers for 60 questions across six systems, totaling 360 answers.The 60 questions are sampled from 15 evaluation repositories with stratification across the four intent categories: why, where, how, and what.Source: C LLM-as-a-Judge Reliability · p. 18
  • The reliability sample draws one question from each intent category in every repository, yielding 15 × 4 = 60 questions and equal representation across repositories and question types.This design is intended to prevent reliability statistics from being dominated by a single repository or question type.Source: C LLM-as-a-Judge Reliability · p. 18

C.1 Intra-panel consistency — Krippendorff’s α (interval level)

Both the human and LLM panels show substantial intra-panel reliability on most dimensions, while clarity is the hardest dimension for both.

C.1 Intra-panel consistency — Krippendorff’s α (interval level)

  • 4/5 dimensions exceed the 0.80 substantial-reliability threshold for both human and LLM panels.Table C1 reports interval-level Krippendorff’s α by evaluation dimension for the human, LLM, and combined six-rater panels.Source: §C.1 · p. 18, Table C1 · p. 18
  • Clarity is consistently the hardest dimension for both panels, indicating a domain-inherent difficulty.Source: §C.1 · p. 18

C.2 Individual cross-group Spearman ρ matrix (per-answer, all p < 0.001)

The human experts’ mean pairwise Spearman correlation was 0.755, while Expert 1’s correlations with the other humans were lower than the E2–E3 correlation. Claude’s lower correlation with Expert 1 mirrors that pattern, suggesting Expert 1 had an idiosyncratic rating style.

C.2 Individual cross-group Spearman ρ matrix (per-answer, all p < 0.001)

  • Table C2 reports per-answer Spearman correlations between each LLM judge and each human expert, with all p < 0.001.Source: Table C2 · p. 18
  • E2–E3 correlated at ρ = 0.917, compared with 0.680 for E1–E2 and 0.668 for E1–E3; the mean within-human ρ was 0.755.Source: §C.2 · p. 18
  • Claude–Expert 1’s ρ of 0.661 mirrors Expert 1’s lower correlations with the other human experts, suggesting an idiosyncratic rating style.Source: §C.2 · p. 18

C.3 Panel-consensus agreement (mean-of-3 vs mean-of-3) · C.4 Pairwise preference agreement

The sections compare human and LLM panels first through their mean-of-three consensus scores and then through system-pair preference verdicts. The reported consensus metrics indicate close agreement in both score level and system ordering.

C.3 Panel-consensus agreement (mean-of-3 vs mean-of-3)

  • The comparison averages three human ratings into a human-consensus score and three LLM ratings into an LLM-consensus score.Source: §C.3 · p. 19
  • Table C3 reports agreement between the two consensus signals on a 0–100 scale.Source: Table C3 · p. 19, §C.3 · p. 19
  • MAE was 5.22 points, roughly one-fifth of the typical ~25-point gap between competing systems.The passage interprets this panel-level signal as precise enough to distinguish methods reliably.Source: §C.3 · p. 19
  • Pearson r was 0.972 and Spearman ρ was 0.942, indicating that LLM consensus closely tracked human consensus in score level and ordering.Source: §C.3 · p. 19

C.4 Pairwise preference agreement

  • C.4 focuses on the practical question of which system is better by comparing human and LLM panel verdicts for system pairs.Source: §C.4 · p. 19
  • Each system pair is classified as A > B, a tie, or B > A, and agreement measures how often the panels reach the same verdict.Source: Table C4 · p. 19, §C.4 · p. 19

C.5 Length-bias analysis · C.6 Inter-judge ranking consistency

The analyses find perfect ranking agreement among the three LLM judges and high human–LLM agreement across answer-length bins, while LLM scores remain lower than human scores. They assess potential length bias through regression and stratified comparisons, and judge consistency through rankings across baselines and target models.

C.5 Length-bias analysis

  • Length bias is examined with two analyses: OLS regression that partials out system and dimension effects, and comparisons stratified by answer length.The stratified comparison uses short, medium, and long answer bins.Source: §C.5 · p. 19
  • The OLS analysis correlates the LLM–human consensus difference with answer length at r = 0.179, p = 0.006, r2 = 0.032.The model is score ~ length + system + dim, partialing out system and dimension effects.Source: §C.5 · p. 19
  • Across short, medium, and long answer bins, LLM scores are lower than human scores, while human–LLM agreement is uniformly excellent (r ≥0.92).Table C5 reports length-stratified agreement, the mean LLM −Human score gap, and per-answer Pearson r.Source: §C.5 · p. 19, Table C5 · p. 19

C.6 Inter-judge ranking consistency

  • The ranking-consistency analysis evaluates the reliability of the three LLM judges, alongside SWE-QA’s human verification protocol.The reported perfect judge-level agreement is ρoverall = 1.Source: §C.6 · pp. 19–20
  • Inter-judge consistency is assessed by comparing the ranks each judge assigns to baselines for each target model.The supplied computation passage defines Ri,j(m) as judge j’s rank for baseline m on target model i.Source: §C.6 · p. 20
  • ρoverall = 1, indicating perfect ranking agreement among the three LLM judges, complemented by SWE-QA’s human verification protocol.The paper presents this agreement, together with human verification, as evidence of judge reliability.Source: §C.6 · pp. 19–20

D Cross-language Generalization — Java Results … E.2 Failure modes

On the 30-pair Java subset, DeepRepoQA ranks first among open-source methods and trails Cursor; the failure analysis describes how misleading retrievals can lead to backtracking or wrong-path termination and identifies four recurring failure modes.

D Cross-language Generalization — Java Results

E.1 Handling of misleading retrievals

  • When SemanticSearch returns a misleading snippet, the Evaluation Agent usually recognizes the evidence is unhelpful and assigns a low score, triggering UCT backtracking.This is the first stage of the described correction mechanism.Source: §E.1 · p. 20
  • If the Evaluation Agent assigns a misleading snippet a high score, the trajectory may accumulate more unhelpful evidence before a delayed backtrack; sometimes it still terminates on the wrong path.The delayed backtrack occurs when the cumulative value signal falls below that of an unexpanded sibling.Source: §E.1 · p. 20

E.2 Failure modes

  • Manual inspection of the lowest-scoring trajectories across question categories identifies four recurring failure modes, with Table E1 mapping affected categories to root causes.Sections E.5–E.8 examine a representative case for each mode.Source: §E.2 · p. 20

E Failure Mode Case Studies

  • The failure-mode case studies walk through a representative example for each of the four modes identified from low-scoring trajectories.The examples are presented in Sections E.5–E.8.Source: §E.2 · p. 20

E.3 Success patterns … F Prompts

This appendix section presents success and failure patterns in repository exploration alongside prompts that guide agents’ code-grounded search, feedback, action scoring, answer evaluation, tool selection, and answer completion.

E.3 Success patterns

  • 100% of top-scoring cases reached the correct file within two hops, and 80% used symbol-level searches with an explicit file_pattern; immediate ViewCode before Finish correlated with high Correctness.Cases needing 15+ node expansions almost never recovered from an incorrect initial retrieval. In the full evaluation, Clarity averaged 15.1 versus Correctness at 10.5, showing fluent prose did not ensure factual accuracy.Source: §E.3 · pp. 20–21

E.4 Representative success case — why/purpose

  • 17/20 Correctness resulted when SemanticSearch found the target class on the first hop and ViewCode supplied the source and docstring used to answer.The eight-node, four-step path searched semantically, found ComputedVarShadowsBaseError in reflex/utils/exceptions.py, viewed the file, and finished.Source: §E.4 · p. 21

E.5 Representative failure case F1 — where/iden-loca

  • The F1 failure stopped at a semantically similar function without verifying the question’s literal identifier, parse_args_spec, using FindFunction.The retrieved _extract_func_kwargs_as_ast_nodes parsed kwargs but appeared in a different file and context.Source: §E.5 · p. 21, Table E2 · p. 21

E.6 Representative failure case F2 — why/purpose

  • Semantic ambiguity in “structured container” led SemanticSearch to AppWrap rather than AppInfo, and the agent carried that initial interpretation across all 17 MCTS nodes.The shared incorrect retrieval across branches made backtracking ineffective.Source: §E.6 · pp. 21–22, Table E3 · p. 22

E.7 Representative failure case F3 — why/design-rationale

  • The F3 search varied keywords for seven steps, hit the 20-node cap at 332 s, and never found the HookPosition enum or ordering function.It did not switch to directory inspection, import-following, or structural navigation when keyword search stalled.Source: §E.7 · p. 22

E.8 Representative failure case F4 — how/system-design

  • The F4 redesign was technically sound but penalized for diverging from the reference’s single prescribed solution, reflecting both a single-reference judging artifact and open-ended design ambiguity.The passage notes that additional retrieval would not have led to the reference-specific solution because code retrieval offers little guidance among multiple valid designs.Source: §E.8 · p. 22, Table E4 · p. 22

F Prompts

  • The prompts require stepwise, code-grounded searches and evidence-based completion, and specify feedback context, relevant code-span selection, and evaluation of the latest action.The action prompt directs the agent to search, inspect code as needed, and cite specific code; feedback describes the current node and tree relationships.Source: F Prompts · pp. 22–26
  • The action rubric judges search relevance, scope, result size, category fit, and appropriateness to the agent’s stage, with scores from 0–100.It treats exploratory actions as valuable even when they do not immediately find a result; 90–100 denotes excellent actions and 0–24 counterproductive ones.Source: F Prompts · p. 26
  • Exact-name tools locate classes, functions, or snippets, while SemanticSearch finds related functionality when names are unknown; ViewCode inspects code and Finish signals answer completion.The prompts give examples of exact-name searches and semantic queries for described functionality, tests, implementations, and error-handling patterns.Source: F Prompts · pp. 26–27
  • Answer evaluation compares candidate and reference answers strictly across correctness, completeness, relevance, clarity, and reasoning, with deductions for omissions or inaccuracies.Each criterion is scored from 1 to 20; high scores are reserved for answers closely matching the reference, and scoring guidance emphasizes conservative ratings.Source: F Prompts · pp. 27–28

Related papers

Loading 2608.24221v1…