PaperlayerView exact revision on arXiv

Source-linked AI summary

CATCODER: Repository-Level Code Generation with Relevant Code and Type Context

Zhiyuan Pan, Xing Hu, Xin Xia, Xiaohu Yang

arXiv:2406.03283v2cs.SEcs.AI

TL;DR

Repository-level code generation must draw on information spread across files, while retrieval can miss needed API details. CatCoder combines retrieved code with type context extracted by static analyzers; on Java and Rust benchmarks, it outperforms RepoCoder, with further studies reporting generalizability and scalability.

  • Problem

    Repository-level generation requires cross-file context, but retrieved code may omit necessary field accesses and API calls, particularly for novel functionality or repositories with little duplication.

    Source: §1 · p. 2

  • Method

    CatCoder retrieves relevant repository code, uses static analyzers to extract dependent type context, and combines both in a prompt for an LLM.

    Source: §1 · p. 2

  • Results

    CatCoder outperforms RepoCoder by up to 14.44% in compile@k and 17.35% in pass@k, with improvements across selected LLMs and results supporting scalability in a large repository.

    Source: §1 · p. 3, §8 · p. 23

  • Takeaways & Limitations

    The authors report generalizability and scalability, and release the source code, benchmarks, and evaluation materials in a replication package.

    Source: §8 · p. 23

    A failure case passes relevant APIs to the model but still misses null-pointer handling and fails some unit tests, so repository context does not guarantee correctness.

    Source: §6.1.2 · p. 20

Abstract

from arXiv · show

Large language models (LLMs) have demonstrated remarkable capabilities in code generation tasks. However, repository-level code generation presents unique challenges, particularly due to the need to utilize information spread across multiple files within a repository. Specifically, successful generation depends on a solid grasp of both general, context-agnostic knowledge and specific, context-dependent knowledge. While LLMs are widely used for the context-agnostic aspect, existing retrieval-based approaches sometimes fall short as they are limited in obtaining a broader and deeper repository context. In this paper, we present CatCoder, a novel code generation framework designed for statically typed programming languages. CatCoder enhances repository-level code generation by integrating relevant code and type context. Specifically, it leverages static analyzers to extract type dependencies and merges this information with retrieved code to create comprehensive prompts for LLMs. To evaluate the effectiveness of CatCoder, we adapt and construct benchmarks that include 199 Java tasks and 90 Rust tasks. The results show that CatCoder outperforms the RepoCoder baseline by up to 14.44% and 17.35%, in terms of compile@k and pass@k scores. In addition, the generalizability of CatCoder is assessed using various LLMs, including both code-specialized models and general-purpose models. Our findings indicate consistent performance improvements across all models, which underlines the practicality of CatCoder. Furthermore, we evaluate the time consumption of CatCoder in a large open source repository, and the results demonstrate the scalability of CatCoder.

1 Introduction

Repository-level generation must use cross-file information, yet retrieved code can miss necessary API details. CatCoder combines relevant code with statically extracted type context and is evaluated on Java and Rust benchmarks.

1 Introduction

  • Retrieved code can miss cross-file field accesses and API calls, especially in repositories with little duplication or novel functionality.Approaches dependent on external resources or specialized retrieval-model training may also generalize poorly to unseen repositories or languages.Source: §1 · p. 2
  • CatCoder targets statically typed languages by combining relevant code with type context extracted using static analyzers.The type context exposes fields and methods of related types as repository-specific references for generation.Source: §1 · p. 2
  • The framework retrieves repository code, extracts dependent types, and combines both sources in a prompt for a frozen LLM without external databases or additional model training.Source: §1 · p. 2
  • CatCoder outperforms RepoCoder by up to 14.44% in compile@k and 17.35% in pass@k; results also report gains across selected LLMs and scalability in a large repository.The evaluation uses 199 Java tasks and 90 Rust tasks.Source: §1 · pp. 2–3
  • The authors release CatCoder’s source code and benchmarks as a replication package to facilitate future research.Source: §1 · p. 3

2 Motivating Example

The triu example shows why repository-level generation needs both a general algorithm and repository-specific knowledge. CatCoder supplies that knowledge through retrieved code patterns and type APIs, then combines both in an LLM prompt.

2 Motivating Example

  • For triu, the general algorithm copies matrix entries when c is at least r+k and sets other entries to zero, but valid code must follow RealMatrix APIs and construction conventions.The example identifies getEntry for element access and Array2DRowRealMatrix for creating the returned matrix.Source: §2 · p. 3, Figure 1 · p. 4
  • The two context sources contribute different clues: relevant code shows a repository-specific object-creation pattern, while type context exposes the related type’s fields and methods.The ones method illustrates creating an Array2DRowRealMatrix by passing a primitive double array to its constructor.Source: §2 · p. 4, Figure 1 · p. 4
  • CatCoder retrieves similar code using the target function’s docstring and signature, extracts dependent-type fields and method signatures with static analyzers, and merges both into an LLM prompt.Source: §2 · p. 5

3 Proposed Approach

CatCoder completes repository functions by combining retrieved code with type context extracted from the repository. Its workflow includes hybrid code retrieval, dependency-based type extraction, and prompting an LLM to generate the function body.

3 Proposed Approach

  • CatCoder takes a repository and an empty function, retrieves relevant code and extracts type context, then post-processes the LLM output into completed code.The framework's two core components are Relevant Code Retrieval and Type Context Extraction.Source: §3 · pp. 5–6

3.1.1 Code Splitting.

  • Code files are recursively split into size-limited chunks, preferring boundaries at type definitions, function definitions, control-flow statements, and newlines.The stated ordering protects code structure during splitting.Source: §3.1.1 · p. 6

3.1.2 Hybrid Retrieving.

  • A sparse keyword retriever and a dense semantic retriever each return top-k code chunks for a query built from the docstring and function signature.Sparse retrieval uses BM25; dense retrieval ranks embedding vectors by squared Euclidean distance, with lower distance indicating greater similarity.Source: §3.1.2 · pp. 6–7

3.1.3 Result Ranking.

  • Weighted Reciprocal Rank Fusion combines the retrievers’ rankings, after which duplicate documents are removed and the results are sorted by score.Each document's score uses the retriever weight and the document's original rank when present.Source: §3.1.3 · p. 7

3.2.1 Type Context.

  • The type dependency graph links repository types when one references or depends on another, and its related types supply fields and method signatures for type context.For the triu example, the graph is traversed from CMAESOptimizer and its resulting type context is serialized as text; the figure illustrates this graph and context.Source: §3.2.1 · pp. 7–8, Figure 3 · p. 8
  • Type context is limited to direct neighbors of the initial types, and standard-library types are pruned because the authors assume LLMs know them from pre-training.The scope limit addresses the size of full dependency graphs and the resulting context-window constraint.Source: §3.2.1 · p. 8

3.2.2 Building Type Context via Static Analyzer.

  • A parser and static analyzer iteratively discover direct-neighbor types from the initial set, then concatenate the discovered types’ fields and method signatures.The analyzer supplies type information and is used to navigate to type definitions and implementations.Source: §3.2.2 · pp. 8–9

3.3 Code Generation via LLM

  • The prompt’s CONTEXT combines type context and retrieved code, while CODE begins with the docstrings and signature for the frozen LLM to complete with a method body.Post-processing can truncate output, prepend function signatures, and remove Markdown syntax; these procedures may vary by model and language.Source: §3.3 · p. 9, Figure 4 · p. 9

4 Experimental Setup

The setup defines four research questions, compares context conditions, and evaluates CatCoder on Java and Rust benchmarks using compilation- and test-based metrics. It also specifies benchmark construction, implementation, model selection, and generation settings.

4 Experimental Setup

  • Four research questions cover effectiveness, component contributions, generalizability across models, and scalability; evaluations use Java and Rust.CodeLlama-13B-Instruct is the default model for RQ1 and RQ2.Source: §4 · p. 10, §4.4.2 · p. 12

4.1 Baselines

  • Vanilla, In-File, and RepoCoder provide no, local, and retrieved cross-file context comparisons, respectively.RepoCoder is adapted for Java and Rust, retrieving from docstrings initially and from generated code in later iterations.Source: §4.1 · p. 10

4.2 Evaluation Metrics

  • compile@k measures compilation rate, while pass@k measures test-passing rate using the unbiased score@k metric.The benchmark score is the mean of task-level score@k values.Source: §4.2 · pp. 10–11

4.3.1 Java Benchmark.

  • The Java evaluation adapts CoderUJB [52]’s 238 tasks by sequentially filtering 17 functions exceeding the 512-token output limit, 5 without modifiers or annotations, and 17 standalone functions, leaving 199 coding tasks.Correctness is evaluated by patching the repository with generated code and running Defects4J tests.Source: §4.3.1 · p. 11

4.3.2 Rust Benchmark.

  • RustEval fills the repository-level Rust benchmark gap with 90 documented functions gathered from 13 crates.io repositories.Correctness is tested with documentation tests.Source: §4.3.2 · p. 11

4.3.3 Benchmark Statistics.

  • The benchmarks report task counts, average NLOC, and average CCN; Rust is less verbose and structurally simpler than Java, but not necessarily easier semantically.Table 3 presents the benchmark statistics; the authors attribute Rust's reduced verbosity and simpler structure to idiomatic abstractions and less boilerplate.Source: §4.3.3 · pp. 11–12, Table 3 · p. 12

4.4 Implementation

  • CatCoder uses MPNet V2 for retrieval and Eclipse JDT.LS and rust-analyzer for type extraction, with ground-truth code removed from retrieval.CodeLlama-13B-Instruct is the default for RQ1/RQ2; RQ3 uses code-specialized and general-purpose models of varying sizes.Source: §4.4.1 · p. 12, §4.4.2 · p. 12

4.4.4 Configurations.

  • Retrieval uses different chunk sizes and result counts for Java and Rust; evaluations generate 10 samples per task with k values of 1, 3, and 5.Chunk size and k are 2,000 and 4 for Java, versus 1,000 and 8 for Rust; generation uses temperature 0.6, top_p 0.7, and a 512-token limit.Source: §4.4.4 · p. 13

5 Experimental Results

The section introduces its analysis of experimental results and research questions.

5 Experimental Results

  • The section analyzes experimental results and answers the research questions.Source: §5 · p. 13

5.1 RQ1: Effectiveness

CatCoder outperforms RepoCoder on compilation and test-passing rates across the Java and Rust benchmarks. Qualitative cases and CodeBLEU comparisons show why retrieved code and type context can matter beyond structural similarity alone.

5.1.1 Results.

  • Java compile@k and pass@k improve by up to 14.44% and 17.35%, and Rust compile@k and pass@k by up to 3.27% and 6.68%, over RepoCoder.Vanilla and In-File perform worse than RepoCoder and CatCoder.Source: §5.1.1 · p. 13

5.1.2 Qualitative Analysis.

  • In one case, type context helps distinguish paired fields and methods, avoiding a misleading retrieved example that otherwise led to a test failure.Both systems retrieved a similar method, but CatCoder's type context exposed the field-method pairings.Source: §5.1.2 · p. 14, Figure 5 · p. 14
  • In another case, type context could identify out-of-scope references that led RepoCoder to generate nonexistent fields and an API call.The misleading fields came from another file; the absent type context concerned JavaType and TypeFactory.Source: §5.1.2 · p. 14, Figure 5 · p. 14

5.1.3 Comparison of Java and Rust.

  • Java's evaluated projects are more verbose and structurally complex; the authors argue type context matters more there, alongside its larger pass@k improvement.The maximum pass@k improvement is 17.35% for Java and 6.68% for Rust.Source: §5.1.3 · pp. 14–15

5.1.4 Semantic Similarity vs. Test Pass Rate.

  • CatCoder and RepoCoder score similarly on CodeBLEU, with RepoCoder slightly ahead, although CatCoder has higher compile@k and pass@k, especially for Java.The authors use this contrast to underscore the importance of execution-based metrics for repository-level code generation.Source: §5.1.4 · p. 15, Table 6 · p. 15

5.2 RQ2: Ablation Study

Ablations show that removing either retrieval or type-context extraction lowers performance. Retrieval removal produces larger declines, while type context remains complementary on more challenging cases.

5.2 RQ2: Ablation Study

  • Removing Relevant Code Retrieval lowers Java compile@k and pass@k by up to 19.43% and 38.35%, and Rust scores by up to 46.24% and 56.03%.Source: §5.2 · p. 16, Table 7 · p. 16
  • Removing Type Context Extraction lowers Java compile@k and pass@k by up to 6.63% and 11.57%, and Rust scores by up to 8.52% and 5.3%.Source: §5.2 · p. 16, Table 7 · p. 16
  • Retrieval's larger decline reflects near-complete duplicated-code templates, while type context complements retrieval on harder cases with distracting or out-of-scope references.The authors describe type context as less critical when retrieved code closely matches the target, but essential when retrieval alone is insufficient.Source: §5.2 · p. 16

5.3 RQ3: Generalizability

CatCoder improves pass@1 across the tested models on both Java and Rust. The study also finds that relevant-code retrieval contributes more than type context, while model-specific gains vary.

5.3.1 Results on Generalizability.

  • CatCoder raises average pass@1 by 13.73% on Java and 8.4% on Rust versus RepoCoder, improving every tested model.The Java and Rust comparisons cover both benchmarks; the reported gains are averages across models.Source: §5.3.1 · pp. 16–17, Figure 6 · p. 17
  • Removing Relevant Code Retrieval lowers average pass@1 by 46.89% on Java and 57.94% on Rust, versus 17.17% and 8.21% without Type Context Extraction.The -CR variant has the largest degradation on both benchmarks; the -TC variant also drops on both.Source: §5.3.1 · p. 17, Figure 6 · p. 17

5.3.2 Comparison of different LLMs.

  • ML-8B falls behind the selected code-specialized models except CG-7B in these benchmarks.The authors relate the comparison to the benefit of fine-tuning on high-quality code datasets [42].Source: §5.3.2 · p. 17
  • CatCoder’s pass@1 score increase is more pronounced for ML-8B, CQ-7B, and DS-6.7B than for CL-13B, although larger models tend to have better absolute performance.Source: §5.3.2 · p. 18

5.4 RQ4: Scalability

CatCoder’s prompt-construction latency drops substantially after the first task in the closure-compiler repository. The scalability evaluation measures retrieval and type-context extraction across 29 sequential tasks, excluding LLM inference.

5.4.1 Experimental Design.

  • The evaluation runs 29 sequential code-generation tasks in closure-compiler, a roughly 400,000-line repository, and excludes underlying LLM inference latency.It measures the core Relevant Code Retrieval and Type Context Extraction components.Source: Figure 7 · p. 18, §5.4.1 · p. 19
  • Relevant Code Retrieval and Type Context Extraction can run in parallel, so total prompt-construction time is bounded by the slower component.Figure 7 reports component latency for the sequential tasks; Table 8 summarizes average latency.Source: Figure 7 · p. 18, Table 8 · p. 18, §5.4.2 · p. 19

5.4.2 Analysis of Scalability.

  • The first task takes 38.8 seconds, with Relevant Code Retrieval the main cold-start bottleneck.Setup includes generating code-chunk embeddings for retrieval and parsing the codebase and building type-hierarchy and dependency representations for type-context extraction.Source: Table 8 · p. 18, §5.4.2 · p. 19
  • After cold start, prompt construction averages 1.56s, with 36x faster Relevant Code Retrieval, 3.5x faster Type Context Extraction, and 25x faster total time.The reported component averages are 1.08s for retrieval and 1.32s for type-context extraction.Source: Table 8 · p. 18, Figure 7 · p. 18, §5.4.2 · pp. 19–20

6 Discussion

CatCoder’s scope is bounded by the reasoning abilities of its frozen LLM, retrieval and static-analysis limits, and an implementation currently supporting Java and Rust. The discussion describes a null-pointer failure and cases where reflection can leave Java type context incomplete.

6.1 Limitations of CatCoder

  • Because CatCoder uses a frozen LLM, it is inherently limited in enhancing the model’s logical reasoning abilities.The authors state that correct repository-level generation also depends on general syntax and algorithmic understanding.Source: §6.1.2 · p. 20
  • Even when generated code uses relevant APIs and resembles the ground truth, missing null-pointer handling can cause unit-test failures.The authors identify improving repository-level reasoning through fine-tuning or reinforcement learning as possible future directions [29].Source: §6.1.2 · p. 20, Figure 8 · p. 20

6.1.2 Limitation in Retrieval.

  • Hybrid retrieval can still return incomplete or misleading context, although type context can supply accessible API information for cross-reference.The authors suggest improving retrieval and reranking as future work.Source: §6.1.2 · pp. 20–21
  • When a Java repository relies heavily on reflection, the analyzer may fail to resolve runtime-loaded classes and members, leaving type context incomplete.The stated limitation concerns dependencies on classes loaded reflectively, such as through Class.forName.Source: §6.1.3 · p. 21

6.2 Threats to Validity

  • The static-analysis approach locates types and retrieves their definitions and implementations; the authors consider further verification unnecessary for this use.They describe the analysis as lightweight and more accurate than pointer or reflection analysis because it operates on explicit type names.Source: §6.2.1 · p. 21

6.2.1 Internal Threats.

  • The current implementation supports only Java and Rust, though the approach is described as applicable to statically typed languages more broadly.C++ and C# are named as possible future extensions.Source: §6.2.2 · p. 21

7 Related Work

Related work spans code-LLM architectures, repository-level benchmarks, and retrieval-based generation approaches. CatCoder combines repository code retrieval with type context for statically typed languages without requiring model training or multiple generation iterations.

7.1 Large Language Models for Code

  • Code LLMs include encoder-only, encoder-decoder, and decoder-only architectures; the evaluated models are all decoder-only.The passage associates encoder-only models with masked language modeling and decoder-only models with next-token prediction.Source: §7.1 · p. 21

7.2 Benchmarks for Repository-Level Code Generation

  • Repository-level benchmarks moved from same-file contexts toward real-world repositories with cross-file context, while RepoBench and EvoCodeBench address data leakage and memorization concerns.CoderEval and CrossCodeEval support multilingual tasks, while CoderUJB spans five practical programming-task types.Source: §7.2 · p. 22

7.3 Repository-Level Code Generation with LLMs

  • Retrieval approaches use documentation or external databases [34] [51], prompt proposals [40], or iterative repository-code retrieval.CatCoder adds type context to retrieval-based techniques; the paper reports that it requires neither model training nor multiple generation iterations.Source: §7.3 · p. 22
  • Because CatCoder does not modify the underlying LLM, the authors say it can be combined with model-training frameworks and decoding strategies [23] [48].Source: §7.3 · p. 23

8 Conclusion and Future Work

CatCoder addresses missing type-dependency information in repository-level code generation for statically typed languages by retrieving relevant code and extracting type context. Its evaluation reports improved pass@k over RepoCoder, alongside generalizability and scalability findings.

8 Conclusion and Future Work

  • CatCoder retrieves relevant code and uses static analyzers to extract type context for code generation in statically typed languages.It was evaluated on Java and Rust benchmarks built from real-world open-source repositories.Source: §8 · p. 23
  • Up to 17.35% higher pass@k scores were reported for CatCoder than for the RepoCoder baseline.Further studies demonstrated generalizability and scalability.Source: §8 · p. 23
  • The authors release CatCoder’s source code, benchmark datasets, and evaluation scripts, and identify benchmark data leakage as a possible future research topic.Source: §8 · p. 23
Loading 2406.03283v2…