PaperlayerView exact revision on arXiv

Source-linked AI summary

On Pretraining for Project-Level Code Completion

Maksim Sapronov, Evgeniy Glukhov

arXiv:2510.13697v1cs.SEcs.LG

TL;DR

Repository-level pretraining can capture cross-file context, but it can demand large datasets and costly long sequences. This paper tests repository-processing strategies for OpenCoder using curated repository data. The authors report that composer choice has only a marginal effect, while adapting to a longer context window is the primary factor in repository-level pretraining.

  • Problem

    Repository-level pretraining can capture cross-file relationships but may require huge datasets and computationally demanding long sequences.

    Source: §1 · p. 1

  • Method

    The paper compares repository context-composer strategies while pretraining OpenCoder 1.5B on repository-level data.

    Source: §1 · p. 2, §2.1 · p. 2, §3.2 · p. 3

  • Results

    Composer choice had only a marginal impact, with performance scores ranging from 45.2 to 48.8 out of 100; adapting to a longer context window was the primary factor.

    Source: §1 · p. 2, §3.2 · p. 4

  • Takeaways & Limitations

    A relatively small training dataset and simple composer, including file-level approaches, can yield performance comparable to recent state-of-the-art code LLMs.

    Source: §4 · p. 4

    Experiments cover only OpenCoder, so whether the findings generalize to other LLMs remains unclear.

    Source: §4 · p. 4

Abstract

from arXiv · show

Repository-level pretraining is commonly used to enable large language models for code to leverage codebase-wide context. This enhances their ability to generate accurate and context-aware code completions. In this work, we investigate how different repository-processing strategies affect in-context learning in OpenCoder, a 1.5B-parameter model. We extend its context window from 4,096 to 16,384 tokens by training on additional 1B tokens of curated repository-level data. Despite relying on a smaller dataset than competing models (which often use hundreds of billions of tokens), our model achieves comparable performance on the Long Code Arena benchmark. We find that various repository-processing techniques yield similarly strong results, with the primary gain coming from adapting to a new rotary positional embedding (RoPE) scaling parameter. Finally, we show that a simpler file-level training approach at the original sequence length remains highly effective, opening up repository-level code completion research to settings with more constrained data and compute resources.

1 INTRODUCTION AND MOTIVATION

Repository-level pretraining gives code models cross-file context but can require large datasets and costly long-sequence training. This work extends OpenCoder to 16K tokens and finds that composer choice has only a marginal effect on final quality.

Motivation

  • Repository-level pretraining extends context across files and shared dependencies, addressing file-level models’ limited project-wide context.The paper notes that this approach can also require very large datasets and computationally demanding long sequences.Source: §1 · p. 1

Approach

  • OpenCoder’s context window is extended from 4,096 to 16,384 tokens by training on repository-level input sequences.The extended model performs significantly better on 16K-token sequences than the initial configuration.Source: §1 · pp. 1–2

Evaluation

  • The evaluation uses Long Code Arena’s Project-Level Code Completion task to assess cross-file code completion.The study isolates repository-level pretraining and compares context-composer strategies.Source: §1 · p. 2

Main contributions

  • Composer choice had only a marginal impact, with performance scores ranging from 45.2 to 48.8 out of 100.The study evaluates repository-level pretraining on OpenCoder 1.5B.Source: §1 · p. 2

2 EXPERIMENT DESIGN

The experiments construct repository-derived completion examples, train OpenCoder with different context composers at an extended sequence length, and evaluate selected categories of cross-file code completion.

2.1 TRAINING DATA

  • The training data comes from open-source Python repositories and pairs pre-commit repository snapshots with files added in each commit.The raw dataset contains 1,640 repositories, 160,801 commits, and 361,052 completion files.Source: §2.1 · p. 2
  • A context composer ranks repository files or chunks by relevance, selects those that fit the context window, and concatenates them with the most relevant file last.The study includes an empty-context file-level composer and a Path Distance .py composer that orders Python files by path distance, then matching-line IoU.Source: §2.1 · p. 2

2.2 TRAINING

  • OpenCoder 1.5B is pretrained on each composer’s composed dataset with a 16,384-token context window.Training truncates context and completion files to maintain at least a 3:1 context-to-completion token ratio and changes RoPE’s base frequency from 10,000 to 500,000.Source: §2.2 · p. 3

2.3 EVALUATION

  • The evaluation tests next-line completion on Long Code Arena’s LCA-large, using infile and inproject categories and Exact Match.Models are evaluated with FL-4K and PD-16K composers; repository-context boost is the score difference between them.Source: §2.3 · p. 3

3 RESULTS

The results report comparable project-level completion quality with limited repository-level pretraining data. Across the tested composers, longer-context adaptation is the primary factor, while composer choice contributes only marginally.

3.2 IMPACT OF CONTEXT COMPOSER CHOICE

  • The results section examines how context-composer choice affects repository-level pretraining performance.Table 2 summarizes checkpoint results on LCA-large for inproject and infile using FL-4K and PD-16K evaluation composers.Source: §3 · p. 3, §3.2 · p. 4, Table 2 · p. 4

3.1 BENCHMARKING AGAINST STATE-OF-THE-ART

  • The study compares existing models and pretrained checkpoints on LCA-large’s inproject and infile categories.Table 1 reports Exact Match scores for file-level and Path Distance .py evaluation composers, plus repository-context boost.Source: §3.1 · p. 3, Table 1 · p. 4
  • With 1B tokens of training data, OpenCoder 1.5B reaches state-of-the-art project-level completion performance.The authors report that this result reaches Qwen2.5-Coder performance level; the nominal 1B-token pool was available, though only 72M tokens were used for training.Source: §3.1 · p. 3, §3.2 · p. 3, Table 1 · p. 4

3.2 IMPACT OF CONTEXT COMPOSER CHOICE

  • File-level pretraining yields a +19.3 repository-context boost, while other strategies yield boosts from +20.3 to +22.9.The authors attribute the primary gain to adapting to the longer context window with a new RoPE base frequency, rather than sequence composition.Source: §3.2 · p. 4, Table 2 · p. 4

4 CONCLUSION

The conclusion reports that a relatively small dataset and simple context composers can produce performance comparable to state-of-the-art code LLMs, reducing repository-level pretraining complexity. The appendices list composer variants and describe how processed files are formatted into context.

4 CONCLUSION

  • A small training dataset and simple composers such as file-level or path distance can yield performance comparable to state-of-the-art code LLMs while reducing pretraining complexity.The authors say this may encourage broader research on repository-level pretraining.Source: §4 · p. 4

A.1 COMPLETE LIST

  • The composer list spans empty file-level context, ranked Python-file selection, filtered code or text, non-code files, and randomized repository files.Shared preprocessing filters empty files and normalizes line separators to Line Feed.Source: §A.1 · p. 7
  • Four additional composers—random tokens, duplication, leak, and masked leak—are omitted from results because they do not reflect realistic scenarios.Duplication is disabled in evaluation mode.Source: §A.1 · p. 7
  • Two composer modifications reverse the selected-file order to place either the most relevant or the most irrelevant files first.The reversed variant puts the most relevant file at the beginning; the irrelevant variant reverses the order to retrieve the least relevant files.Source: §A.1 · p. 8

A.2 INPUT FORMATTING

  • Each processed file follows a predefined special-token formatting pattern, and the formatted files are concatenated into a composed context string.The completion file receives a similar transformation independently.Source: §A.2 · p. 8

B TRAINING DATASET

The training dataset is filtered to avoid benchmark-test repositories and retain recent, unique completion files within a specified length range.

B TRAINING DATASET

  • Repositories used in Long Code Arena’s Project-level code completion task are excluded to avoid training on test data.The repository snapshot otherwise remains without additional processing after the listed filtering.Source: §B.1 · p. 8
  • Completion files are retained only when they are 800–25,000 characters long and come from commits made in 2010 or later.Source: §B.1 · p. 8
  • After deduplication by completion-file name and repository name, up to 1,000 of each repository’s most recently updated unique completion files are selected.Source: §B.1 · p. 8

C TRAINING DETAILS

Training uses AdamW with gradient clipping and a cosine learning-rate schedule following linear warm-up.

C TRAINING DETAILS

  • AdamW training uses β1 = 0.9, β2 = 0.999, weight decay 0.01, batch size 128, and micro-batch size 1.Source: §C.1 · p. 8
  • Gradient clipping limits the gradient Euclidean norm to 2 for training stability.Source: §C.1 · p. 8
  • The learning rate peaks at 5 × 10^-5 after 256 warm-up iterations, then cosine-decays for 3,244 iterations to 5 × 10^-8.Source: §C.1 · p. 8

C.2 TRAINING MODE OF CONTEXT COMPOSERS

Training sequences combine independently tokenized composed context and completion text under context and total-length limits.

C.2 TRAINING MODE OF CONTEXT COMPOSERS

  • Context strings and completion files are tokenized independently before being combined into each training input sequence.Source: §C.2 · p. 8
  • Completion sequences are capped at 4,096 tokens and total concatenated inputs at 16,384 tokens, using left truncation for context and right truncation for completion.Source: §C.2 · p. 8
  • Because composed contexts are usually highly saturated, the training setup maintains a context-to-completion token ratio above 3:1.Source: §C.2 · p. 8

D EVALUATION DETAILS

Evaluation composers build model inputs from repository context and completion files, with sequence length adjustable at evaluation time. Four composers serve as baselines or test in-context learning at different context lengths.

D.1 EVALUATION MODE OF CONTEXT COMPOSERS

  • Each evaluation input concatenates the context string and completion file, tokenizes the result, and truncates it from the left.Evaluation maximum sequence length is adjustable rather than fixed as in training.Source: §D.1 · p. 9
  • FL-4K estimates task difficulty without repository context and supplies a reference point for calculating gains.It uses the File-Level composer with a maximum sequence length of 4K tokens.Source: §D.1 · p. 9
  • PD-4K and PD-16K measure in-context learning at the initial 4K and extended 16K input lengths, respectively.Or-16K applies only to the authors’ checkpoints and is used to identify the most promising composer overall.Source: §D.1 · p. 9

E COMPREHENSIVE COMPILATION OF EVALUATION RESULTS

The experiments find that RoPE adaptation is the dominant factor in long-context performance gains, while sequence composition plays a secondary role. File-level training recovers quality after RoPE adjustments and preserves shorter-context in-context learning.

E COMPREHENSIVE COMPILATION OF EVALUATION RESULTS

  • RoPE adjustments affect model quality: the experiments compare base frequencies θ=10,000 and θ=500,000, and the initial base frequency performs at zero-level for long contexts even after finetuning.The results include the base model and file-level pretraining at both frequencies.Source: E COMPREHENSIVE COMPILATION OF… · p. 9
  • File-level composer evaluation recovers model quality after RoPE adjustments, while the initial model already outperforms file-level inference with PD-4K.The PD-4K advantage persists after repository-level pretraining, retaining the model’s ability to use relevant context at shorter lengths.Source: E COMPREHENSIVE COMPILATION OF… · p. 9
  • +19 for file-level training and +22 for the best composer in the inproject category after further pretraining with PD-16K; composer choice adds only +3 points inproject.The initial model without RoPE adaptation fails completely on PD-16K, while RoPE scaling alone improves Exact Match scores.Source: E COMPREHENSIVE COMPILATION OF… · p. 9

F PERFORMANCE SCALING BEYOND TRAINING CONTEXT WINDOW

The pretrained checkpoints extrapolate beyond their training context lengths, reaching 16K and 32K tokens. Figure 2 compares Exact Match accuracy across context sizes for File-level and Path Distance .py approaches.

F PERFORMANCE SCALING BEYOND TRAINING CONTEXT WINDOW

  • Pretrained checkpoints extrapolate beyond their training context lengths, up to 16K and 32K tokens.Repository-level pretraining used File-level and Path Distance composers with maximum sequence lengths of 4K and 16K.Source: F PERFORMANCE SCALING BEYOND TRAINING… · p. 9
  • Figure 2 plots Exact Match accuracy across context sizes for File-level and Path Distance .py approaches, with inproject above infile and dashed lines marking pretraining context lengths.The figure supports comparing the approaches across context sizes relative to those pretraining lengths.Source: Figure 2 · p. 11

G MASKED LOSS AND FULL LOSS RESULTS

The study masks loss on context tokens to avoid distribution shift from out-of-distribution sequences. Masked-loss and full-loss pretraining results are comparable for each composer except Duplication.

G MASKED LOSS AND FULL LOSS RESULTS

Loading 2510.13697v1…