PaperlayerView exact revision on arXiv

Source-linked AI summary

Retrieval-Augmented Generation for Scientific Code Understanding

Aaron Nobile, Andreas Adelmann, Mohsen Sadr

arXiv:2609.12190v1cs.SEcs.AI

TL;DR

This work asks whether small local models can answer technical questions about specialized code repositories without relying on large cloud-hosted models. It builds a RAG system that prepares repository context offline for lightweight answering, and reports that model family and retrieval quality matter more than parameter count, with a 9B configuration achieving the highest average score.

  • Problem

    Large cloud-hosted models can be costly and raise privacy concerns, while scientific repositories contain specialized knowledge distributed across years of development.

    Source: §1 · p. 1

  • Method

    The system separates expensive offline retrieval preparation from lightweight answering with a local model, supplying structured and relevant codebase context.

    Source: §1 · p. 1, §3 · p. 4

  • Results

    Model family and retrieval quality mattered more than parameter count, and the 9B configuration achieved an average score of 0.795 ± 0.034 on the 100-question benchmark.

    Source: §6 · p. 28

  • Takeaways & Limitations

    The system is best suited to explaining and navigating a prepared scientific codebase, while active development tasks such as changing and verifying code call for more autonomous agents.

    Source: §4.3 · p. 22

    The system depends on parsing and retrieval quality, and it does not edit code, run tests, or iteratively verify its answers.

    Source: §4.3 · p. 22

Abstract

from arXiv · show

Large language models have become central to modern coding assistants, but state-of-the-art systems such as Claude Code or Codex rely on very large, cloud-hosted models with significant computational cost and data-privacy implications. This work investigates whether a useful, fully local coding agent can be built around small open-source models by shifting the computational burden away from inference. We develop a Retrieval-Augmented Generation (RAG) system for scientific code understanding that strictly separates an expensive offline ingestion stage parsing, structural graph construction, LLM-generated entity explanations, and embedding from a lightweight online answering stage. The system is evaluated on a 100-question benchmark spanning eleven categories over the IPPL scientific codebase written in C++, with answers scored by an independent frontier model as the judge. Across seven answering models, we find that model family and retrieval quality matter more than parameter count, i.e. a 9B model achieves the highest average score (0.795), outperforming both larger models within our pipeline and the same models embedded in the Claude Code retrieval architecture. The results indicate that front-loading code understanding into a reusable, codebase-specialised vector store enables small local models to deliver grounded and repository-specific answers, making the agent well suited as a privacy-preserving development tool for in-house scientific codebases.

1 Introduction

The introduction asks whether a smaller, locally run model can answer repository-specific technical questions when retrieval preparation handles more code understanding in advance. It presents a code-aware RAG system and evaluates answering models on questions about IPPL.

1 Introduction

  • Scientific repositories can be private, costly to query with large cloud models, and difficult to understand because their specialized C++ knowledge is distributed across years of development.The question is whether advance retrieval preparation can help a smaller local model answer technical repository questions.Source: §1 · p. 1
  • The system separates expensive offline ingestion from lightweight online question answering.Ingestion parses source and documentation, constructs structural relationships, generates entity-level explanations, and embeds records; answering combines retrieval and structural expansion before prompting a local model.Source: §1 · pp. 1–2
  • The evaluation covers 100 questions across eleven IPPL codebase categories and compares seven answering models with an agentic retrieval baseline.The contributions include representing functions, files, modules, and call neighborhoods, and retrieving concise, grounded context for small local models.Source: §1 · p. 2

2 Background

The background introduces language models and coding agents, then motivates retrieval-augmented generation for codebases too large to fit in a prompt. It presents IPPL as a scientific-software testbed combining code structure with high-performance computing and numerical concepts.

2.1 Large Language Models

  • Transformers use self-attention across token sequences, while trainable parameters are learned through prediction and optimization during training.The background describes parameters as coefficients shaping information flow and predictions, and notes that modern models can contain billions of them.Source: §2.1 · p. 2

2.2 Coding Agents

  • Coding agents inspect source code, modify files, execute commands, and support tasks such as debugging and refactoring.Scientific projects can be especially challenging because they combine long development histories, numerical algorithms, and domain knowledge; the work investigates local open-source models for code understanding and generation.Source: §2.2 · pp. 2–3

2.3 Retrieval-Augmented Generation (RAG)

  • RAG retrieves repository information relevant to a query and supplies it as prompt context, extending a model’s access beyond its training data and current prompt.The described pipeline embeds code and documentation, searches for relevant material, and appends retrieved information to the prompt; repository-level retrieval can expose cross-file context.Source: §2.3 · p. 3

2.4 IPPL as a Scientific-Software Testbed

  • IPPL is an open-source C++ framework for particle-mesh applications, with MPI, Kokkos, and heFFTe supporting parallelism and distributed Fourier transforms.Its mix of software structure, high-performance computing, and numerical concepts makes it a testbed for repository-level questions.Source: §2.4 · p. 4

3 System Design

The system is a locally executable scientific-code assistant that supplies structured, precise, relevant repository context to an answering model. Its overview separates preparation of retrievable chunks from inference-time retrieval and prompt building.

3 System Design

  • The system uses prompt enrichment: structured codebase excerpts, called chunks, are inserted into a prompt template before it is sent to the LLM.This design separates retrieval preparation from inference.Source: §3.1 · p. 4, §3 · p. 4

3.1 System Overview

  • Retrieval preparation stores structure graphs, base-LLM explanations, and higher-level entities as retrievable chunks; inference performs vector search, query-intent detection, reranking, context expansion, and prompt building.The two stages are the components shown in Figure 1.Source: Figure 1 · p. 4

3.2 Ingestion and Retrieval Preparation (Offline)

The offline preparation stage transforms source and documentation into structured, retrievable chunks, builds project relationships and summaries, and embeds content for search. It prepares a vector store that combines dense, exact, and filename-scoped retrieval and reuses the saved index unless the manifest changes.

3.2 Ingestion and Retrieval Preparation (Offline)

  • The offline stage traverses the codebase to collect source and documentation files, which become the building blocks for retrieval chunks.Collected types include C++ .cpp, .hpp, and .h files and documentation in .md, .rst, and .txt formats.Source: §3.2 · p. 5

3.2.1 Parsing and Chunking

  • Tree-sitter-based parsers represent C++ syntax structurally, extracting implementation functions and header interfaces while preserving leading comments and metadata for indexing and retrieval.The implementation parser also separates qualified names into namespaces and symbol names.Source: §3.2.1 · pp. 5–6

3.2.2 Documentation chunking & parsing

  • Documentation parsing extracts logical sections, and DocChunker turns them into metadata-bearing records, splitting oversized sections and suffixing titles to preserve their shared origin.It processes Markdown, reStructuredText, and plain text.Source: §3.2.2 · p. 7

3.2.3 Call and Dependendency Graph Building

  • ProjectStructureBuilder combines structure graphs into a project structure to represent relationships across files, folders, functions, and scientific-simulation call chains.The resulting project structure is stored in embeddings/project_structure/project_structure.json.Source: §3.2.3 · p. 7

3.2.4 Higher Entity Building

  • Function-level chunks reduce retrieval noise, while LLM-generated file- and module-level entities summarize broader scopes that would otherwise require retrieving many functions.Entity strategies also create call-chain chunks from direct incoming and outgoing calls, using symbol, file, and module summaries.Source: §3.2.4 · pp. 7–8

3.2.6 Vector Store

  • The embedder connects content chunks to a FAISS vector store by embedding chunks and user queries so semantically related content can be retrieved.A persisted manifest avoids rebuilding an unchanged store; FAISS uses exact L2-distance search on 32-bit embeddings to return top-k chunks.Source: §3.2.5 · p. 8, §3.2.6 · p. 10

3.2.5 Embedding and Vector Store

  • Chunk embeddings combine semantic explanations with structural metadata; prompt-length retries truncate explanations and code, while query prompts incorporate detected file names and other inferred details.The query prompt structure is used only for embedding; the original question sent to the answering LLM remains unchanged.Source: §3.2.5 · pp. 8–9

3.2.6 Vector Store

  • Hybrid retrieval supplements dense search because dense retrieval can miss direct keyword matches.Source: §3.2.6 · p. 10
  • Exact helpers retrieve file-, symbol-, and module-level entities using query mentions, with contextual symbol matches prioritized.File lookup handles full names and base names, including sibling extensions; symbol lookup returns at most two candidates, while exact module matches receive distance 0.0.Source: §3.2.6 · pp. 10–11
  • Filename-scoped semantic search returns the three closest function-level chunks within identified files.The vector store is saved under embeddings/vector_store and reused for retrieval unless the manifest changes.Source: §3.2.6 · p. 11

3.3 Retrieval

Retrieval selects a compact, relevant context from an indexed codebase before answer generation. The pipeline classifies and combines candidate chunks, applies query-aware ranking and filtering, and can add bounded structural context.

3.3 Retrieval

  • Retrieval selects relevant codebase material rather than sending the entire project to the language model, providing a bridge from the question to a compact context.Candidates are ranked using semantic similarity, lexical matches, and entity-level relevance.Source: §3.3 · p. 11

3.3.1 Chunk Classification

  • Candidate chunks are classified as primary evidence, structurally expanded context, or supplementary chunks from files referenced by primary chunks.Structural expansion can add related file-, module-, or call-chain-level chunks.Source: §3.3.1 · p. 11

3.3.2 Retriever

  • The retriever merges candidate sources, removes duplicates, reranks candidates, then refines primary chunks and applies exact-filename ordering safeguards.Table 2 describes configurable settings for choosing final primary chunks to avoid weak or repetitive context.Source: Table 2 · p. 12, §3.3.2 · p. 14

3.3.3 Test/Build Noise Sorting

  • Test- and build-scope chunks are excluded from general implementation answers but remain eligible when the query explicitly concerns tests or build configuration.The listed scopes include test, tests, unit_test, and unit_tests.Source: §3.3.3 · pp. 14–15

3.3.4 Structural Expansion

  • After primary retrieval, structural expansion follows ingestion-built relationships, with the query intent selecting a profile that bounds the types and number of added chunks.API-usage queries use a narrower profile than data-flow and workflow queries.Source: §3.3.4 · pp. 15–16

3.3.5 Query Intent Routing

  • Query-intent routing detects requested entity types and question intent, then returns retrieval preferences and an expansion mode.Table 4 covers intent categories and their corresponding structural expansion modes.Source: Table 4 · p. 16, §3.3.5 · p. 17

3.3.6 Metadata Reranker

  • The metadata reranker combines a semantic score with query matches for names, paths, source types, APIs, data flow, and requested entity types.It also applies query-specific boosts and penalizes low-signal symbols and test-scope chunks for non-test location queries.Source: §3.3.6 · p. 17

3.4 LLMAgent

The LLMAgent filters weak retrieved chunks before assembling the prompt context. It uses chunk quality signals and stronger evidence from the same local context to avoid passing redundant or low-value material onward.

3.4 LLMAgent

  • The LLMAgent removes a weak chunk only when it is low-information or trivial and a stronger chunk from the same local context meets score and margin thresholds.The stronger chunk must score at least 2 and beat the candidate by at least 3 points.Source: §3.4 · p. 18
  • A chunk's final-context quality score rewards explanations, leading comments, references, higher-level context, and substantive code, while penalizing trivial declarations.Examples of higher-level context include file-, module-, and call-chain-level chunks.Source: §3.4 · p. 18
  • Filtering can discard an empty class declaration when a richer file-level or method chunk from the same location is available.The selected chunks are then formatted into a structured text context for the language model.Source: §3.4 · p. 18

3.5 Prompting and Inference

Prompting uses distinct modes for entity explanations and final answers. At inference, the LLMAgent formats retrieved chunks with semantic and structural information before sending them to the answering model.

3.5 Prompting and Inference

  • Separate prompt modes support local code entities, file summaries, module summaries, call-chain neighborhoods, and final RAG answers.The active modes are configured in runtime_settings.json.Source: §3.5 · pp. 18–19, Table 5 · p. 19
  • Prompt signatures in the runtime manifest can invalidate generated artifacts and trigger a rebuild when a prompt changes.This keeps stored explanations consistent with the prompt version that generated them.Source: §3.5 · p. 19
  • The final answer prompt receives retrieved chunks with metadata, explanations, references, and structural relationships, and constrains answers to that retrieved context.The final answering stage uses retrieval_answer_v2, separately from preprocessing prompt modes.Source: §3.5 · p. 19

4 Implementation and Runtime Configuration

The architecture shifts repository understanding into a one-time offline stage, producing reusable retrieval context for lightweight online answering. Runtime configuration and cluster execution support this separation, which favors repeated question answering over stable scientific codebases while trading off preprocessing cost and agentic autonomy.

4 Implementation and Runtime Configuration

  • Prepared retrieval context lets the online stage use smaller or more efficient local models for repeated questions about a codebase.The offline stage generates entity explanations and stores retrieval-ready context for reuse during inference.Source: §4 · pp. 19–20

4.1 Configuration and Runtime Setup

  • Runtime settings centralize parser and model choices, stage-specific prompt modes, and retrieval and reranking controls without direct source-code changes.Separate prompt modes allow one stage's prompting strategy to change without affecting the others.Source: §4.1 · pp. 20–21

4.2 Execution on the Cluster

  • The initial vector-store build is the pipeline's most computationally expensive stage and should run on a GPU-equipped compute node; prepared artifacts can be reused afterward.A clean rebuild is needed when important inputs or configuration change.Source: §4.2 · p. 21

4.3 Architectural Advantages and Limitations

  • Offline chunk, file, module, and call-chain explanations specialize retrieval for the target codebase and support reproducible answering with smaller local models.Runtime settings, prompt modes, model names, and generated artifacts can be stored and compared across evaluation runs.Source: §4.3 · p. 22
  • The system depends on parsing and retrieval quality, may require rebuilding a stale index, and cannot modify code, run tests, or iteratively verify answers like agentic systems.Codex and Claude-style agents better suit active development; this system is optimized for repeated question answering over relatively stable scientific codebases.Source: §4.3 · p. 22

5 Evaluation

The evaluation tests seven answering models and compares retrieval architectures on a 100-question, eleven-category IPPL benchmark. Results vary by model family and retrieval setup, with category-level strengths and weaknesses and caveats about interpreting the scores.

5.1 Experimental Setup

  • The benchmark contains 100 questions across eleven categories, generated with GPT-5.4; GPT-5.5 judges answers as correct, partial, or incorrect.The answering model varies while explanation generation and chunk embeddings remain fixed.Source: §5.1 · pp. 22, 24

5.2 Results

  • Qwen3.5 was the clear winner among tested model families; older Qwen2.5 models scored 0.665–0.720, and newer or larger models tended to have greater latency.Latency was measured on merlin6.Source: §5.2 · p. 24, Table 10 · p. 25

5.3 Discussion

  • Qwen2.5 7B scored 0.720 in the dedicated RAG pipeline versus 0.520 under Claude Code, while the authors report that the dedicated retrieval context better serves small models.They attribute the difference tentatively to noisier or less targeted context from the agentic loop.Source: Table 11 · p. 25, §5.3 · p. 27

5.2.1 Claude Code Comparison Results

  • Under Claude Code, Qwen2.5 7B and 32B scored 0.520 ± 0.038 and 0.560 ± 0.036; Gemma4 scored 0.665 ± 0.040, while partial answers numbered 44/100, 46/100, and 27/100, respectively.The reported incorrect count for Gemma4 was around 20/100, similar to the other two models.Source: Table 11 · p. 25, §5.2.1 · p. 25

5.2.2 Category Breakdown

  • API Usage was strongest, averaging 0.881 across models; Definition Location, Class Responsibility, File Purpose, Boundary And Halo, and Data Flow also scored around 0.74 or higher.For Definition Location, 81 of 105 answers were correct; File Purpose had 9 incorrect answers out of 91.Source: §5.2.2 · pp. 25–26, Table 12 · p. 26
  • Build And Install was weakest, averaging 0.371 with 10 correct and 19 incorrect answers out of 35; Numerical Meaning averaged 0.548, while Parallelism And Kokkos and Testing And Workflow were around 0.61.The authors suggest the latter categories involve more cross-file reasoning and exact project conventions.Source: §5.2.2 · p. 26

5.3 Discussion

  • Model family mattered more than size in the reported comparisons: Qwen3.5 9B outscored Gemma4 31B, and the Qwen3.5 32B score was 0.03 lower than its 9B version.The reason for the size-related pattern is unclear, and shared generation settings may affect models differently.Source: §5.3 · p. 27
  • The authors caution that GPT-generated questions and GPT-based judging may introduce stylistic bias, and that 100 questions from one codebase make absolute scores indicative rather than definitive.They identify a larger multi-codebase benchmark and partial human validation as ways to strengthen the conclusions.Source: §5.3 · p. 27

6 Conclusion

The system demonstrates locally generated, grounded, repository-specific answers on IPPL, with its strongest configuration outperforming larger models and the same models in Claude Code’s retrieval architecture. The conclusion identifies offline-prepared context as a key part of small-model performance and bounds the system as a code-learning tool rather than an all-round coding agent.

6 Conclusion

  • 0.795 ± 0.034 was the 9B model’s average score on the 100-question benchmark, outperforming larger models in the same pipeline and the same models in Claude Code’s retrieval architecture.The authors report this as the strongest configuration.Source: §6 · p. 28
  • For this task, model family and retrieval quality mattered more than parameter count; offline-prepared explanations and structural metadata let small models use repository context without extensive reasoning.The authors describe this explanation of small-model competitiveness as appearing to follow from the prepared retrieval context.Source: §6 · p. 28
  • The system explains code but does not edit files, run tests, or iteratively verify its output; active software-engineering tasks require additional tooling.The conclusion characterizes it as a locally run, privacy-preserving “code teacher.”Source: §6 · p. 28
  • Re-ingestion is costly and required after codebase changes, making the design best suited to relatively stable repositories that are queried repeatedly.The conclusion gives education and corporate onboarding as examples of this setting.Source: §6 · p. 28

6.1 Future Work

  • Clearer code-block formatting and symbolization, such as explicit backticks, are proposed as ways to improve retrieval and context quality.The authors note that chunk formatting affects how well an LLM understands its input.Source: §6.1 · p. 28
  • Future component swaps include embedders, retrievers, vector stores, and reranking libraries; cross-encoder rerankers add retrieval computation but may recover relevant chunks with low initial vector similarity.The authors name BAAI/bge-reranker-v2-m3 and BAAI/bge-reranker-large as candidates for evaluation.Source: §6.1 · p. 28
  • The authors prioritize constant evaluation while experimenting with alternative frameworks, retrievers, models, and embedders; improvements should be evaluated in context because they could degrade the pipeline.They also identify implementation-helper tooling as a longer-term goal that would likely require revising the codebase.Source: §6.1 · pp. 28–29

A Evaluation Protocol

The protocol directs the selected Claude Code model to answer all IPPL benchmark questions using repository evidence and save the answers in a prescribed JSON format. It standardizes answer style and requires explicit acknowledgment when repository evidence is insufficient.

A Evaluation Protocol

  • The selected Claude Code model is instructed to answer every question in the benchmark using repository evidence, especially the raw IPPL data and documentation, and save the results as JSON.The run uses a designated label and timestamped output file, with progress saved incrementally before marking completion.Source: A Evaluation Protocol · p. 31
  • Answers must be concise, grounded in concrete repository files or symbols, and include evidence paths; when evidence is insufficient, the model must say so rather than guess.The protocol also prohibits external web search and long copied code blocks unless necessary.Source: A Evaluation Protocol · p. 31
  • Runs are intended to be comparable through consistent answer style and detail, with each response recorded in a prescribed output schema.The schema includes the question, answer, evidence files, error, and latency fields.Source: A Evaluation Protocol · p. 31

B Prompt Templates

The complete system and user prompts for explanation, summarization, and retrieval-based answering are provided in reproducibility artifacts rather than printed in the paper.

B Prompt Templates

  • Long machine-input prompt templates are omitted from the manuscript and distributed in the reproducibility artifacts and public repository.The omitted templates cover chunk, file, module, and call-chain explanations, as well as retrieval-based answering.Source: B Prompt Templates · p. 32

C Evaluation Questions

The complete evaluation benchmark is available separately, with its 100 questions organized into eleven categories summarized in Table 6.

C Evaluation Questions

  • The benchmark contains 100 questions across eleven categories summarized in Table 6 and is distributed as a separate artifact and with the public repository.The separate artifact preserves the exact benchmark wording while keeping the manuscript concise.Source: C Evaluation Questions · p. 32

Related papers

Loading 2609.12190v1…