PaperlayerView exact revision on arXiv

Source-linked AI summary

Code Comprehensor: Hierarchical Documentation and Cross-Modal Retrieval for Large Repositories

Sujoy Roychowdhury, Joseph H. R. Isaac, Jiangning Gao, Martin Klitte, Diarmuid Corcoran

arXiv:2610.30016v2cs.SEcs.AIcs.LG

TL;DR

Large repositories pose context and reliability challenges for LLM-based code understanding. Code Comprehensor combines call-graph-guided documentation, contrastive text-code retrieval, and agentic orchestration; its documentation-derived supervision improves retrieval across two repositories, with performance varying by query type.

  • Problem

    Large repositories can exceed LLM context windows, while processing whole repositories per question raises costs and large contexts can give poorer results.

    Source: §I · p. 1

  • Method

    Code Comprehensor combines CodeGraph, hierarchical CodeScribe documentation, and a CLIP-style dual-encoder CodeContrast model in an MCP-based orchestration system.

    Source: §III · p. 3

  • Results

    D3 improves overall Top-1 accuracy over D2 by 14.59% on OpenCV and 8.72% on Repo-A, while D2 scores 96.43% versus D3’s 93.94% on OpenCV doc_brief_merged tests.

    Source: §VI · p. 9

  • Takeaways & Limitations

    The evaluation reports complementary retrieval benefits across repositories and query distributions, with D2 stronger on human-authored descriptions and D3 improving synthetic-query results.

    Source: §VI · pp. 9–10

    Evaluation is limited to C/C++ systems programming, and because D3 combines parent/sibling context with synthetic QA, their individual contributions cannot be definitively isolated.

    Source: §VII · p. 10

Abstract

from arXiv · show

Understanding large, complex, and rapidly evolving code bases remains a core challenge for software engineering at scale, hindering onboarding, maintenance, refactoring, and impact analysis. Providing this information to a Large-Language Model (LLM) is another hurdle, as the LLM must understand not only the required code method but also its associated methods in the grand scheme of the repository. We propose Code Comprehensor, a hybrid retrieval-reasoning framework combining (i) CodeGraph, a code dependency callgraph, (ii) CodeScribe, which uses code-aware LLMs to generate hierarchical documentation across repository call graphs, and (iii) CodeContrast, a CLIP-style dual-encoder model trained on CodeScribe output to learn aligned text-code embeddings. CodeContrast enables fast, semantically precise retrieval of relevant code-context neighborhoods, which serve as grounded evidence for an MCP-based LLM orchestrator answering domain-specific queries. Evaluation on OpenCV and a proprietary 5M-LOC repository shows that CodeScribe-derived supervision significantly improves cross-modal retrieval in text-to-code accuracy over baseline approaches. Richer textual supervision (hierarchical documentation, parent/sibling context, and LLM-generated synthetic questions) consistently enhances performance across both repositories. LLM-as-a-judge evaluation using the popular RAGAS of the Code Comprehensor output yields above 90% faithfulness scores.

I. INTRODUCTION

Large repositories exceed practical LLM context limits, while similar code and probabilistic processing challenge reliable, repeatable navigation. Code Comprehensor combines call-graph-driven documentation and contrastive retrieval to provide repository context for domain-specific queries.

I. INTRODUCTION

  • Large repositories can exceed LLM context windows, and sending entire repositories for each question raises costs and can worsen results.Similar code across repositories also makes probabilistic LLM-based navigation less reliable and replicable.Source: §I · p. 1

A. Proposed Solution

  • CodeScribe generates hierarchical, call-graph-informed documentation, which CodeContrast uses to train a CLIP-style dual encoder for text-to-code and code-to-text retrieval.The model uses InfoNCE loss and supports multi-positive learning.Source: A. Proposed Solution · pp. 1–2
  • An MCP-based agentic layer coordinates CodeGraph traversal, CodeScribe retrieval, and CodeContrast queries to answer domain-specific repository questions.The system is intended to support interactive use.Source: A. Proposed Solution · p. 2, Figure 1 · p. 2
  • CodeScribe uses post-order chunking to document large call trees while preserving hierarchy, and its training-data pipeline incorporates parent-sibling context and synthetic questions.These are presented as contributions to scalable documentation and enhanced retrieval.Source: A. Proposed Solution · p. 2
  • At least 9% higher Top-1 accuracy over baseline approaches is reported for cross-modal code retrieval using hierarchical documentation-derived supervision.The system combines CodeGraph, CodeScribe, and CodeContrast as an integrated end-to-end framework.Source: A. Proposed Solution · pp. 1–2

II. RELATED WORK

Prior work covers neural code search, repository documentation, and contrastive code-text learning. Code Comprehensor combines these directions by turning hierarchical documentation into structured retrieval supervision and integrating retrieval with repository navigation.

A. Code Search and Retrieval

  • Earlier code-search work spans context-aware, cross-domain, cross-language, fusion, and contrastive approaches, alongside large-scale contrastive pretraining and surveys.The passage names Sun et al., Chai et al., Mathew and Stolee, Mao et al., Wang et al., Shi et al., Li et al., Di Grazia and Pradel, and Sun et al..Source: A. Code Search and Retrieval · p. 2

B. Repository-Level Documentation Generation

C. Contrastive Learning for Code

D. Differentiation from Prior Work

  • Unlike snippet-based retrieval methods, Code Comprehensor first generates hierarchical documentation that captures intermethod relationships, then trains a contrastive model on that enriched representation.The authors also note limited publicly available implementations for direct C/C++ comparison with CodeRetriever and CoCoSoDa.Source: D. Differentiation from Prior Work · p. 3

III. SYSTEM ARCHITECTURE

  • The system’s Master Agent uses specialized CodeScribe, CodeGraph, and CodeContrast agents to retrieve code, method hierarchies, and documentation for user queries.It can use text-to-code retrieval for natural-language questions and code-to-text retrieval for questions about code entities.Source: §III · p. 3

IV. METHODOLOGY

CodeGraph supplies a repository dependency structure that CodeScribe traverses to generate hierarchical documentation. CodeScribe processes start-method trees in context-limited chunks and can update documentation incrementally as code changes.

A. CodeGraph Creation

  • CodeGraph uses static analysis to identify files, methods, and dependencies, adding classes in object-oriented languages and extracting method comments, signatures, and bodies as node attributes.Tree-Sitter is used for parsing, while the graph structure and subsequent processing are described as language-agnostic.Source: A. CodeGraph Creation · p. 3

B. CodeScribe

  • CodeScribe creates configurable documentation chapters for repository directories, with custom prompts and optional domain abbreviation lists.The documentation structure replicates the repository’s directories.Source: B. CodeScribe · pp. 3–4
  • For each start method—one with zero in-degree in the method-only graph—CodeScribe retrieves its DFS tree, includes method definitions, and generates per-chapter documentation and directory summaries.Directory-hierarchy summaries are optional.Source: B. CodeScribe · p. 4
  • A recursive post-order procedure chunks oversized DFS trees, merging child remainders when they fit and splitting overflow when needed.Earlier chunk outputs are added to later contexts, while intermediate outputs are excluded from final documentation; the rolling summaries give parent callers functional summaries of callees.Source: B. CodeScribe · p. 4
  • CodeScribe can identify changed files and affected start methods from git commits, then update only those methods and their summaries.The resulting Markdown is parsed into a more structured format for CodeContrast.Source: B. CodeScribe · p. 4

1) Markdown-to-JSON Conversion:

The parsing pipeline converts CodeScribe’s hierarchical Markdown into nested, relational records that connect code entries with documentation context.

1) Markdown-to-JSON Conversion:

2) Code Block and Text Separation:

  • A recursive parser separates fenced code blocks from descriptions and assigns each node a depth identifier to recover parent-child and sibling relationships.Code blocks retain their language identifiers and source code.Source: 2) Code Block and Text Separation · p. 5

3) Code Block Classification:

  • Extracted blocks are categorized as annotated source code, PlantUML, or pseudocode according to their metadata and content.PlantUML is detected using its language identifier or an @startuml directive.Source: 3) Code Block Classification · p. 5

4) Relational Feature Construction:

  • Each code entry receives parent, sibling, and module-level textual context, and parsed records are joined with call-graph data using method name and file path.These features provide multi-granularity signals linking code to surrounding documentation.Source: 4) Relational Feature Construction · p. 5

D. Synthetic Data Generation

Synthetic questions broaden retrieval supervision to better reflect developer queries and sparse or inconsistent repository documentation. They are comment-driven, but use bounded code context and signature-derived terminology to improve grounding.

D. Synthetic Data Generation

  • Synthetic questions address sparse, inconsistent, or outdated repository documentation and the question-form of real developer queries.They are added to improve coverage and better match retrieval and reasoning workloads.Source: D. Synthetic Data Generation · p. 5

1) Design Principles:

  • Questions target varied developer intents, including behavior, parameter use, edge cases, preconditions, error handling, and local interactions.Repository-specific terms help retrieval, while the design discourages keyword dumping.Source: 1) Design Principles · p. 5

2) Generation Inputs:

  • Each question set is generated for one code entry using documentation and a signature plus bounded code snippet as inputs.The code provides terminology anchors rather than copied content, keeping the generated behavior description comment-driven.Source: 2) Generation Inputs · p. 5

3) Terminology Anchor Extraction:

The generation process anchors synthetic questions in repository terminology and documented behavior, while limiting verbatim code use and keyword stuffing.

3) Terminology Anchor Extraction:

  • Behavior anchors are action verbs that constrain question semantics, while parameter anchors come from names and salient types in the function signature.Context anchors are repository- and domain-specific terms mined from documentation and bounded code context.Source: 3) Terminology Anchor Extraction · p. 5
  • Key terms combine parameter and context anchors, with a cap to limit enforced tokens and avoid keyword stuffing.The generator is instructed to reflect documented behavior and constraints and include at least two terminology anchors when available.Source: 4) Prompting Strategy: Comment-Driven… · p. 5
  • Questions are generated from comment-driven intent with code-derived terminology anchors, while avoiding verbatim code and dataset-specific identifiers.The prompt requests exactly three developer-style questions per method; lightweight post-processing checks format, deduplicates, and filters for informativeness.Source: 4) Prompting Strategy: Comment-Driven… · p. 6

5) Post-processing:

The pipeline filters and organizes text–code supervision, then trains paired encoders to align textual queries with code snippets. It uses distributed global-batch comparisons to increase the available negatives.

5) Post-processing:

  • Post-processing checks JSON format and tag alignment, removes exact and near duplicates, and rejects questions lacking required anchors or minimum length.A function-name-stripped variant is evaluated to test generalization beyond lexical shortcuts.Source: 5) Post-processing · p. 6

E. CodeContrast Architecture

  • CodeContrast aligns natural-language documentation and source-code embeddings using a dual-encoder contrastive learning system.The system supports SBERT and BERT backbones and includes data preparation, training, embedding storage, and evaluation.Source: E. CodeContrast Architecture · p. 7

1) Data Preparation Pipeline:

  • Data preparation validates and filters entries into one-to-many supervision, pairing each code unit with raw documentation, hierarchical context, and synthetic questions.The resulting data is split into training, validation, and test sets; entries with insufficient documentation or overly long code are flagged.Source: 1) Data Preparation Pipeline · p. 7

2) Core Model Architecture:

  • Text and code encoders produce normalized embeddings in a shared dimension, and jointly trained similarity comparisons use a temperature parameter.Gathering embeddings across devices creates a global similarity matrix, increasing the negative examples compared with a local batch.Source: 2) Core Model Architecture · p. 7

3) Differentiable Global-Batch Training under FSDP:

The training and evaluation setup combines distributed embeddings with a symmetric, multi-positive retrieval objective. Tests assess both retrieval directions on held-out methods and include a separate judged-answer evaluation.

3) Differentiable Global-Batch Training under FSDP:

  • FSDP training gathers embeddings and labels into a global batch, using differentiable all-gather so gradients flow through the shared contrastive objective.Placeholder tensors let ranks with empty batches participate, avoiding synchronization issues and training stalls.Source: 3) Differentiable Global-Batch… · p. 7

4) Multi-positive Symmetric Contrastive Objective:

  • The contrastive objective treats code entries with matching identifiers as positives for each text anchor and applies the corresponding objective in reverse.The objective is optimized symmetrically across Text→Code and Code→Text retrieval.Source: 4) Multi-positive Symmetric… · p. 7

5) Evaluation Metrics:

  • Top-k accuracy and Recall@K measure retrieval quality in both Text→Code and Code→Text directions.Recall@K counts relevant items retrieved within the top K among all relevant items.Source: 5) Evaluation Metrics · p. 7

A. Test Methodology

  • Testing uses held-out methods and evaluates retrieval in both directions, with non-synthetic original descriptions and a synthetic-question variant.A separate RAGAS evaluation compares answers on a subset of synthetic questions with their ground-truth answers, using Answer Correctness and Faithfulness.Source: A. Test Methodology · pp. 7–8

B. Dataset

The experiments use OpenCV and a proprietary repository to assess CodeScribe output, deployment costs, and increasingly rich text-supervision configurations.

B. Dataset

  • The evaluation covers Repo-A, a proprietary C repository with over 5 million LOC, and OpenCV, with about 1.2 million LOC.Repo-A counts .c and .h files; OpenCV counts .c, .h, and .cpp files.Source: B. Dataset · p. 8

C. CodeScribe output on OpenCV

  • Subject-matter experts reviewed the proprietary-repository output, while the authors manually compared CodeScribe’s OpenCV documentation with CodeWiki.For OpenCV’s 3D calibration section, the authors describe CodeScribe’s modular account as more informative than the corresponding CodeWiki module.Source: C. CodeScribe output on OpenCV · p. 8
  • CodeScribe documentation was described as largely correct and useful, but somewhat verbose.Source: C. CodeScribe output on OpenCV · p. 8

D. Computational Efficiency and Costs

  • CodeScribe generation took approximately 48 hours and $3,200 for OpenCV, and approximately eight days and a few thousand dollars for Repo-A.CodeContrast training took about 24 hours on four A100 GPUs; retrieval latency was under 50 ms per query, while the full agent pipeline averaged 3–5 seconds.Source: D. Computational Efficiency and Costs · p. 8

E. Configurations

2) D2 — Method + Docstring + Self-Comments:

The experiments compare supervision configurations for code retrieval and assess the Master Agent with RAGAS. Parent/sibling context and synthetic questions improve results for complex synthetic queries, while D2 performs better on a human-authored documentation category.

2) D2 — Method + Docstring + Self-Comments:

  • D2 adds CodeScribe self-comments to method code and docstrings, while its training text also includes generated questions from documentation and context levels.The listed question types include raw-documentation, self, parent, and sibling questions.Source: 2) D2 — Method + Docstring +… · pp. 8–9

3) D3 — D2 + Parent–Sibling Comments and QA (Structured Context Expansion)

F. Evaluation of Master agent

  • RAGAS evaluates Master Agent answers using Answer Correctness and Faithfulness; reported high Faithfulness scores indicate grounding in retrieved context.The authors attribute comparatively lower Answer Correctness scores to sensitivity from few claims in the ground-truth and generated answers.Source: F. Evaluation of Master agent · p. 9, Table III · p. 9

VI. DISCUSSION

  • D3 improves overall Top-1 accuracy over D2 by 14.59% on OpenCV and 8.72% on Repo-A, while D2 leads on OpenCV’s doc_brief_merged category.On that category, D2 scores 96.43% and D3 93.94%; on synthetic queries, D3 improves over D2 across Top-K values for complex query categories.Source: §VI · pp. 9–10, Table I · p. 9, Table II · p. 9
  • D2 is recommended for queries matching method documentation, whereas D3 suits diverse queries requiring broader system context; the target query distribution should guide selection.Source: §VI · p. 10

VII. THREATS TO VALIDITY

The validity discussion bounds the evidence to C/C++ systems repositories and flags data leakage, measurement, and attribution uncertainties. The conclusion reports query-dependent retrieval results and calls for evaluation across more languages and domains.

VII. THREATS TO VALIDITY

  • Evaluation covers two systems-programming repositories in C and C++, and the authors say broader language and application-domain evaluation is needed to validate generalization.The conclusion notes that LLM generation may pose challenges for niche languages less represented in training data.Source: §VII · p. 10, §VIII · p. 10
  • CodeScribe uses Sonnet 4.5 while simpler generation tasks use Sonnet 4.0, a choice that may introduce distributional differences in training data.Source: §VII · p. 10
  • Training context may reference held-out methods, but sibling fields omit full code bodies, the code encoder sees only the target method, and removing cross-references shifts OpenCV Top-1 by less than 1%.The split is performed at the method level.Source: §VII · p. 10
  • Retrieval is evaluated in isolation with Top-K and Recall as proxies for code comprehension, while agent responses are evaluated using RAGAS FF and AC.Source: §VII · p. 10
  • D3 combines parent/sibling context with synthetic QA, so the authors cannot attribute its improvements to either source without an intermediate ablation; non-finetuned models may also affect RAGAS results.Source: §VII · p. 10

VIII. CONCLUSION AND FUTURE WORKS

  • D2 reaches 96.43% Top-1 on OpenCV for natural human-authored descriptions, while D3 reaches 80.64% Top-1 on synthetic text→code; agent evaluation reports FF above 90%.The conclusion says configuration choice depends on the expected deployment query distribution.Source: §VIII · p. 10
  • Future work proposes adding structural graphs, sequence diagrams, and visual artifacts, using graph-node proximity in retrieval, and refining synthetic-question generation.The authors suggest these additions could better capture relationships and control flows that are difficult to express in text and improve retrieval relevance.Source: §VIII · pp. 10–11

Related papers

Loading 2610.30016v2…