PaperlayerView exact revision on arXiv

Source-linked AI summary

Prometheus: Towards Long-Horizon Codebase Navigation for Repository-Level Problem Solving

Yue Pan, Zimin Chen, Siyu Lu, Zhaoyang Chu, Xiang Li, Han Li, Yang Feng, Claire Le Goues, Federica Sarro, Martin Monperrus, He Ye

arXiv:2507.19942v2cs.SE

TL;DR

Large-codebase navigation remains difficult because relevant evidence is overwhelmed by irrelevant content and existing approaches do not retain exploration history. Prometheus combines a repository knowledge graph with working memory in a multi-agent issue-resolution system. It reports 74.4% and 33.8% resolution rates on SWE-bench Verified and SWE-PolyBench Verified, respectively.

  • Problem

    In large codebases, relevant context is overwhelmed by irrelevant content, while existing navigation approaches lack persistent memory of exploration.

    Source: §1 · p. 2

  • Method

    Prometheus combines a unified knowledge graph and working-memory context retrieval in a multi-agent system for issue resolution.

    Source: §1 · p. 3

  • Results

    Prometheus achieves 74.4% and 33.8% resolution rates on SWE-bench Verified and SWE-PolyBench Verified, ranking Top-6 and Top-1 among open-source agent systems, respectively.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Prometheus integrates repository structure, historical context, and specialized-agent coordination for long-horizon repository-level software engineering.

    Source: §7 · p. 19

    Each experiment was run once because of the high computational and API costs of LLM-based agents.

    Source: §6 · p. 18

Abstract

from arXiv · show

Large Language Models (LLMs) have shown remarkable capabilities in automating software engineering tasks, spurring the emergence of coding agents that scaffold LLMs with external tools to resolve repository-level problems. However, existing agents still struggle to navigate large-scale codebases, as the Needle-in-a-Haystack problem persists even with million-token context windows, where relevant evidence is often overwhelmed by large volumes of irrelevant code and documentation. Prior codebase navigation approaches, including embedding-based retrieval, file-system exploration, and graph-based retrieval, address parts of this challenge but fail to capture the temporal continuity of agent reasoning, rendering agents stateless and causing repeated repository traversals that hinder scalable planning and reasoning. To address these limitations, we present Prometheus, a memory-centric coding agent framework for long-horizon codebase navigation. Prometheus represents the repository as a unified knowledge graph to encode semantic dependencies and employs a context engine augmented with working memory that retains and reuses previously explored contexts to ensure continuity across reasoning steps. Built upon this engine, Prometheus integrates memory-enhanced navigation into a multi-agent system for automated issue resolution, encompassing issue classification, bug reproduction, patch generation, and verification. Comprehensive experiments are conducted on two widely used issue resolution benchmarks, i.e., SWE-bench Verified and SWE-PolyBench Verified. Powered by GPT-5, Prometheus achieves state-of-the-art performance with 74.4% and 33.8% resolution rates on the two benchmarks, ranking Top-6 and Top-1 among open-source agent systems, respectively. Our data and code are available at https://github.com/EuniAI/Prometheus.

1 Introduction

Prometheus addresses codebase navigation limits by combining a unified knowledge graph with working memory that reuses explored context across reasoning steps. It embeds this context engine in a multi-agent issue-resolution workflow and reports leading results on two benchmarks.

1 Introduction

  • Existing navigation methods struggle with large codebases: embedding retrieval can be noisy, filesystem exploration lacks a global architectural view, and graph methods lack persistent memory.Even million-token context windows face the Needle-in-a-Haystack challenge because relevant evidence is overwhelmed by irrelevant content.Source: §1 · p. 2
  • Prometheus combines a unified knowledge graph of semantic dependencies with working memory that retains and reuses explored contexts across reasoning steps.This targets the temporal continuity missing from structure-aware navigation, whose stateless agents repeatedly traverse repository regions.Source: §1 · pp. 2–3
  • Prometheus integrates memory-enhanced navigation into sequential agents for issue classification, bug reproduction, patch generation, and patch verification.Its evaluation also reports over 30% balanced resolution across Java, Python, JavaScript, and TypeScript on SWE-PolyBench Verified.Source: §1 · pp. 3–4
  • Prometheus achieves 74.4% and 33.8% resolution rates on SWE-bench Verified and SWE-PolyBench Verified, ranking Top-6 and Top-1 among open-source agent systems, respectively.The working-memory ablation reduces LLM invocation cost by 45.4% while improving issue resolution rate by 25%.Source: §1 · pp. 3–4

2 Methodology

Prometheus consists of a repository knowledge graph, memory-enhanced context retrieval, and a multi-agent architecture for issue resolution.

2 Methodology

  • The repository-to-knowledge-graph component builds a unified code index representing structural and semantic relationships in the target codebase.The index provides the repository representation used by the other components.Source: §2 · p. 4
  • Memory-enhanced context retrieval identifies and reuses relevant code and documentation to support reasoning.It is the context-retrieval component of the overall system.Source: §2 · p. 4
  • The multi-agent architecture coordinates specialized agents to collaboratively perform issue-resolution tasks.Together, the three components form Prometheus's system overview.Source: §2 · p. 4

2.1 Repository to Knowledge Graph

Prometheus represents repository structure, syntax, and text in a unified knowledge graph built from source files and persisted in a scalable database. Its schema uses three node types and five directed edge types to encode repository, syntactic, and lexical relationships.

2.1 Repository to Knowledge Graph

  • Graph construction defines a node-and-edge schema, constructs the graph from source files, and persists graph data in a scalable database.The unified representation integrates file structures, ASTs, and textual content.Source: §2.1 · p. 4
  • The graph schema uses FileNode, ASTNode, and TextNode nodes to represent files or directories, Tree-sitter syntax nodes, and unstructured text chunks.ASTNodes include source positions, covered code text, and the Tree-sitter grammar node type.Source: §2.1 · p. 4
  • TextNodes are extracted from documentation-oriented files and segmented into configurable chunks using adaptive paragraph-, line-, word-, or character-level separators.The split strategy depends on file content and structure.Source: §2.1 · p. 5
  • Five directed edge types connect repository hierarchy, files to syntax trees and text, AST syntax hierarchy, and sequential text chunks.The HAS_FILE, HAS_AST, PARENT_OF, HAS_TEXT, and NEXT_CHUNK relations integrate structural, syntactic, and lexical information.Source: §2.1 · p. 5
  • The schema averages 1.99 seconds to construct per instance across 500 SWE-bench Verified instances, and supports incremental updates as codebases evolve.The authors characterize it as minimal yet expressive relative to prior specialized or task-coupled graph designs [27].Source: §2.1 · p. 5

2.2 Memory-Enhanced Context Retrieval

Prometheus maps high-level intent to structured evidence through a memory-first retrieval cycle. Its working memory stores reusable contexts and matches later queries to prior exploration using multiple intent components.

2.2.1 Context Retrieval Engine.

  • The retrieval cycle synthesizes a structured sub-query, checks working memory first, and traverses the knowledge graph when relevant context is unavailable.Retrieved contexts are deduplicated, distilled, organized by source file, and stored for reuse.Source: §2.2.1 · pp. 5–6
  • Working memory stores minimal code-and-metadata context units in records linked to their synthesized sub-queries for fine-grained reuse.The persistent context layer is intended to reduce repeated queries and token cost, unlike retrieval from a static external corpus.Source: §2.2.2 · pp. 6–7

2.2.2 Working Memory.

  • Each memory record stores separate embeddings for the sub-query’s essential query, extra requirements, and purpose, alongside their text fields.Records link to context units and are inserted transactionally.Source: §2.2.2 · p. 7
  • Weighted cosine similarities across the three query components rank memory records, which are filtered by threshold and returned as reusable context.The paper states that this multi-vector retrieval improves alignment, precision, and recall compared with single-vector agentic memory baselines.Source: §2.2.2 · p. 7

2.3 Multi-Agent Architectural Design

Prometheus coordinates specialized agents from issue classification through verified resolution. Reproduction tests guide patch generation when available, while verification and aggregation handle both test-grounded and unverified cases.

2.3 Multi-Agent Architectural Design

  • Four agents share context and retrieved knowledge in a workflow spanning issue classification, bug reproduction, patch generation, and verification.The classification agent routes bug, feature, and refactoring issues to tailored workflows.Source: §2.3 · pp. 7–8, §2.3.1 · p. 8

2.3.2 Bug Reproduction Agent.

  • The Bug Reproduction Agent iteratively creates and executes reproduction artifacts until it establishes a reproducible failure or reaches its iteration budget.The resulting executable test and failure trace provide a concrete repair objective and a baseline for evaluating patches.Source: §2.3.2 · p. 9

2.3.3 Patch Generation Agent.

  • For verified bugs, patch generation uses the reproduction test and iteratively refines a candidate when verification fails; unverified cases generate multiple candidate patches.The latter pathway substitutes repeated sampling for test-grounded refinement.Source: §2.3.3 · pp. 9–10

2.3.4 Patch Verification Agent.

  • Verified-bug patches must pass both the reproduction test and selected regression tests or they are returned for further generation.Regression tests are retrieved for modified code regions, then an LLM selects a focused subset.Source: §2.3.4 · p. 10
  • For unverified bugs, feature requests, and refactoring, regression-tested patches are normalized, deduplicated, and selected through LLM aggregation and majority voting.If none passes verification, aggregation is applied to all generated candidates to return a plausible solution.Source: §2.3.4 · p. 10

3 Experimental Methodology

The evaluation examines issue-resolution effectiveness, context retrieval, and component ablations across two benchmarks. Its protocols compare reported agent results, analyze retrieval trajectories against manually constructed gold contexts, and disable components independently.

3.1 Research Questions

  • The study asks how Prometheus compares with other agents, how its context retrieval compares with prior techniques, and how four core components affect resolution and reliability.The ablated components are working memory, multiple patch selection, bug reproduction, and regression testing.Source: §3.1 · p. 10

3.2 Experimental Setup

  • Evaluation uses SWE-bench Verified’s 500 Python bug-fixing instances and SWE-PolyBench Verified’s 382 instances spanning multiple languages and task types.Both benchmarks provide the issue description and full repository without explicit fault localization or additional oracle information.Source: §3.2.1 · p. 11

3.2.2 Methodology.

  • For effectiveness, Prometheus is compared with prior-reported or leaderboard results for selected agents, using percentage of resolved issues as the primary metric.The study does not reimplement baseline systems.Source: §3.2.2 · p. 11
  • Context retrieval is compared on 100 randomly selected SWE-bench instances by measuring agent-retrieved artifacts against gold contexts traced by six professional developers.Runs use GPT-5 with Prometheus, Agentless, OpenHands, and SWE-agent; gold contexts are traced from files and code regions modified by benchmark gold patches.Source: §3.2.2 · pp. 11–12
  • Ablations disable working memory, multiple patch selection, bug reproduction, or regression testing independently while keeping other settings fixed.Each variant is evaluated under the same benchmark protocols using percentage of resolved issues.Source: §3.2.2 · p. 12

4 Experimental Results

Prometheus leads the reported issue-resolution comparisons and retrieves patch-relevant context more accurately than prior agents. Ablations associate working memory with lower inference cost and higher resolution, while removing other core components lowers performance.

4.1 RQ1: Effectiveness Comparison

  • 74.4% on SWE-bench Verified places Prometheus among top-performing systems, while 33.8% on SWE-PolyBench Verified ranks it first.With GPT-5 held constant, Prometheus resolves 29 unique instances, compared with 16 for OpenHands and 4 for mini-SWE-agent.Source: §4.1 · pp. 12–13, Table 3 · p. 13

4.2 RQ2: Analysis of Context Retrieval

  • Prometheus has higher file- and symbol-level context hit rates than the reported baselines, including 0.915 versus OpenHands’ 0.876 at file level and 0.850 at symbol level.The reported baselines remain below 0.58 at symbol level.Source: §4.2 · p. 14, Table 4 · p. 14
  • At span level, Prometheus reaches a 0.807 hit rate, versus 0.541 for OpenHands, 0.482 for SWE-Agent, and 0.054 for Agentless.The paper describes this as more precise localization of code corresponding to gold-patch modifications.Source: §4.2 · p. 14, Table 4 · p. 14

4.3.1 Ablation Study on Working Memory.

  • 70.00% resolution at $200.79 total inference cost with working memory contrasts with 56.00% at $367.73 without it across 50 SWE-bench Verified instances.Working memory reduces inference cost by approximately 45.4% and improves issue resolution rate by 25.0%; the case study describes storing and reusing retrieved context across iterations.Source: §4.3.1 · pp. 14–15, Figure 7 · p. 15

4.3.3 Ablation Study on Bug Reproduction.

  • 74.4 with bug reproduction falls to 64.0 without it, a 14.0% drop on the full SWE-bench Verified benchmark.The paper describes reproduction tests as an execution-based signal for whether a candidate patch addresses the root cause.Source: §4.3.3 · p. 16

4.3.4 Ablation Study on Regression Testing.

  • 74.4 with regression testing falls to 70.2 without it, a 5.6% reduction on the full SWE-bench Verified benchmark.Regression tests provide a complementary check for patches that fix the target bug but break existing functionality.Source: §4.3.4 · p. 16

4.3 RQ3: Ablation Study of Core Components

  • Removing core components lowers performance; the authors identify bug reproduction as having the largest impact, while patch selection and regression testing improve robustness.The authors attribute working memory’s role to reduced cost and improved resolution.Source: §4.3.4 · p. 16

5 Related Work

Prior work spans autonomous and workflow-based issue-resolution agents, repository-context retrieval, and agent memory. Prometheus combines repository-scale retrieval with working memory organized around repository artifacts and evolving repair states.

5.1 Agentic Issue Resolution

  • Issue-resolution agents include fully autonomous systems with dynamic tool workflows, such as SWE-Agent [44] and OpenHands [38], and workflow-based systems such as Agentless [39] and AutoCodeRover [53].The cited discussion contrasts autonomous flexibility and computational overhead with workflow constraints that reduce search space and mitigate hallucinated workflows.Source: §5.1 · p. 17

5.2 Repository-level Context Retrieval

  • Repository-context methods include iterative retrieval, documentation retrieval, and knowledge graphs, while Prometheus uses a language-agnostic graph and working memory to retain and selectively update salient context.Examples include RepoCoder [51], DocPrompting [54], RepoGraph [27], and KGCompass [43].Source: §5.2 · pp. 17–18

5.3 Memory for Agents

  • Unlike memory approaches centered on reusable skills, cross-issue experience, or episode-local state, Prometheus organizes repository artifacts and evolving repair states across iterative steps.The cited comparisons include Voyager [37], EXPEREPAIR [24], SWE-Exp [8], and RepairAgent [7].Source: §5.3 · p. 18

6 Threats to Validity

The evaluation faces internal, external, and construct validity threats: experiments were run once, benchmarks may not represent industrial systems, and metrics may miss qualitative aspects of software development.

Internal Validity

  • Each experiment was run once because of LLM-agent computational and API costs, leaving random factors or implementation variance as potential internal threats despite controlled settings.The authors used fixed random seeds, identical prompts, and consistent tool configurations across baselines.Source: §6 · p. 18

External Validity

  • The two benchmarks may not fully capture the scale and complexity of real-world industrial systems.SWE-PolyBench Verified covers multiple programming languages and diverse repository structures.Source: §6 · p. 18

Construct Validity

  • Standardized issue-resolution criteria and human-annotated gold contexts may not represent every qualitative aspect of real-world software development.The authors assess both task-level resolution accuracy and retrieval precision.Source: §6 · p. 19

7 Conclusion

Prometheus combines repository-level knowledge graphs and memory-enhanced retrieval in a multi-agent system for long-horizon software engineering. Its evaluations show stronger performance than existing baselines, especially in multi-language and multi-task scenarios.

7 Conclusion

  • Prometheus significantly outperforms existing baselines, particularly in challenging multi-language and multi-task scenarios.The evaluations use SWE-bench Verified and SWE-PolyBench Verified.Source: §7 · p. 19
  • Prometheus shifts automated issue resolution from local patch generation toward long-horizon, repository-level software engineering.The conclusion frames this shift as a focus beyond simple code generation.Source: §7 · p. 19
  • The system integrates repository-level knowledge graphs with memory-enhanced retrieval for deep program understanding and iterative reasoning across complex codebases.Source: §7 · p. 19
  • Specialized agents for retrieval, generation, and verification unify structural dependencies, historical context, and execution feedback.Source: §7 · p. 19
  • The authors conclude that robust automated maintenance requires principled long-horizon reasoning and repository-level abstraction beyond simple code generation.Source: §7 · p. 19

Related papers

Loading 2507.19942v2…