PaperlayerView exact revision on arXiv

Source-linked AI summary

Long-Range Modeling of Source Code Files with eWASH: Extended Window Access by Syntax Hierarchy

Colin B. Clement, Shuai Lu, Xiaoyu Liu, Michele Tufano, Dawn Drain, Nan Duan, Neel Sundaresan, Alexey Svyatkovskiy

arXiv:2109.08780v1cs.LGcs.SE

TL;DR

Transformers’ finite context windows limit access to relevant information in large source files. The paper introduces eWASH, which uses syntax hierarchy to select file-level context within a fixed window, and reports state-of-the-art Python results on CodeXGLUE tasks.

  • Problem

    Transformers have finite context windows, while source files can exceed them and contain task-relevant information at multiple scopes.

    Source: §1 · pp. 1–2

  • Method

    eWASH uses source-code syntax hierarchy to prioritize higher-level scopes and names when selecting file context for a fixed model window.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    The paper reports state-of-the-art Python performance on CodeXGLUE code completion, method completion, and code summarization.

    Source: §1 · p. 2, §8 · p. 8

  • Takeaways & Limitations

    The reported code-completion findings support syntax hierarchy as a useful signal even when context length is not limiting.

    Source: §8 · p. 8, §7.2.1 · p. 8

    The approach assumes that scope labels are more important for long-range modeling than the contents of those scopes.

    Source: §2.1 · p. 2

Abstract

from arXiv · show

Statistical language modeling and translation with transformers have found many successful applications in program understanding and generation tasks, setting high benchmarks for tools in modern software development environments. The finite context window of these neural models means, however, that they will be unable to leverage the entire relevant context of large files and packages for any given task. While there are many efforts to extend the context window, we introduce an architecture-independent approach for leveraging the syntactic hierarchies of source code for incorporating entire file-level context into a fixed-length window. Using concrete syntax trees of each source file we extract syntactic hierarchies and integrate them into context window by selectively removing from view more specific, less relevant scopes for a given task. We evaluate this approach on code generation tasks and joint translation of natural language and source code in Python programming language, achieving a new state-of-the-art in code completion and summarization for Python in the CodeXGLUE benchmark. We also introduce new CodeXGLUE benchmarks for user-experience-motivated tasks: code completion with normalized literals, method body completion/code summarization conditioned on file-level context.

1 Introduction

Transformers’ finite context windows constrain access to long source files, so the paper introduces syntax-hierarchy-based context compression and reports state-of-the-art Python results on several CodeXGLUE tasks.

1 Introduction

  • Finite transformer context windows limit how much source-file context models can consume, while expanding attention windows is impractical because its memory and compute costs scale quadratically.Prior work explores low-rank or sparse attention, kernel approximations, and retrieved documents.Source: §1 · pp. 1–2
  • eWASH compresses source-file context using syntax hierarchy, prioritizing higher-level scopes and names to surface distant information without requiring a new transformer architecture.The paper notes that the same general idea could be applied to natural-language documents by extracting key terms or distant features.Source: §1 · p. 2
  • The paper introduces code completion, method completion, and docstring generation as its three software-engineering tasks, and evaluates eWASH against transformer baselines.The experiments include GPT-C, PyMT5, and their eWASH-extended variants.Source: §1 · p. 2
  • The paper reports state-of-the-art performance for Python code completion, method completion, and code summarization, and introduces literal-normalized completion and file-context-conditioned method completion benchmarks.It also reports that eWASH excels at code completion even when the context window is not a limiting factor.Source: §1 · p. 2

2 Motivating example

The motivating examples show how fixed context windows can omit important file-level information; eWASH uses syntax hierarchy to prioritize what enters the window. Its prioritization assumes that higher-level scopes are more relevant to a task.

2 Motivating example

  • Method and code completion can both miss relevant file context, such as imports or class-level information, when a fixed context window excludes it.The examples concern predicting ConvNet.forward and completing an assignment in its body.Source: §2 · p. 2, Figure 1 · p. 3

2.1 Extended Window Access by Syntax Hierarchy

  • eWASH assumes that scope labels are more important for long-range modeling than the contents of those scopes.The paper motivates this assumption by how developers organize code into scopes and portable elements.Source: §2.1 · p. 2
  • For a focal method, eWASH prioritizes its signature and docstring and containing class before imports, class attributes, peer-method details, and global expressions.The priority list descends from higher-level context toward peer method bodies and global expressions.Source: §2.1 · p. 3, Figure 2 · p. 4
  • The implementation organizes concrete-syntax-tree elements by priority, tokenizes them, and adds them until the model’s context window is filled.For method completion, a control code identifies the target method; for docstring completion, the method signature and body form the source and the docstring is the target.Source: §2.1 · p. 3
  • For code completion, XGPT-C uses a rolling window over the focal method body, reserving 768 of 1,024 tokens for context and 256 for the rolling window.Methods longer than 256 tokens are split into multiple windows.Source: §2.1 · pp. 3–4

3 Dataset

The study trains on filtered, primarily Python GitHub repositories and uses separate datasets for code-completion and method/docstring evaluation.

3.1 Pre-training

  • Pre-training uses 5+ star GitHub repositories that are primarily Python, filtering for Python 3 compliance or successful lib2to3 conversion and excluding files that exceed a 10-second parsing limit.The time limit is intended to exclude files that are essentially data files, such as files containing very large lists.Source: §3.1 · p. 4

3.2 Fine-Tuning and Evaluation

  • Code completion is evaluated and fine-tuned on Py150, while method and docstring completion use CodeSearchNet.Py150 is larger than CodeSearchNet for Python; CodeSearchNet’s repositories were selected for good docstring coverage.Source: §3.2 · p. 4

4 Baseline Models

The paper compares its models with GPT-C, PyMT5, and memory-efficient transformer baselines, and describes the pre-training corpus alongside those models.

4 Baseline Models

  • The baselines include GPT-C and PyMT5, with Reformer and Performer added as memory-efficient transformers capable of modeling contexts beyond 1,024 tokens.The latter two models are presented as alternatives for longer contexts.Source: §4 · p. 4

4.1 GPT-C

  • GPT-C is a decoder-only autoregressive code language model with about 125M parameters and a 1,024-token context, using syntax-tree-derived token normalization and sliding-window training samples.Its preprocessing extracts token types, normalizes uncommon literals, trains a sub-token vocabulary, and encodes files into sub-token sequences.Source: §4.1 · p. 4

4.2 PyMT5

  • PyMT5 is an encoder-decoder model jointly pretrained on Python code and docstrings, then fine-tuned to translate among combinations of function signatures, docstrings, and bodies.It uses information from a single method and has 406M parameters with a 1,024-token context width for both encoder and decoder.Source: §4.2 · pp. 4–5
  • The paper’s Python parallel corpus is reported as 3× the size of the next largest corpus and over 15× the size of the next largest Python parallel corpus.The caption notes that CSN contains 500k Python methods with docstrings across six other languages.Source: Table 2 · p. 5

4.3 Memory-Efficient Transformers

  • Reformer combines reversible layers, axial positional embeddings, and bucketed attention, while Performer uses a linear attention approximation to improve computational efficiency.Performer approximates the softmax attention kernel and exploits linearity in its attention computation.Source: §4.3 · p. 5

5 Defining the Tasks at Hand

The section defines code completion, method completion, and docstring completion/code summarization, including how literal normalization and file-level context shape these tasks.

5.1 Code completion

  • Code completion predicts source-code tokens autoregressively, with a literal-normalized variant that replaces many literals with special tokens.The 200 most frequent string literals and 30 most frequent numeric literals are preserved.Source: §5.1 · p. 5

5.2 Method completion

  • Method completion predicts a method body from its signature, optional docstring, and other available context.The task is contributed to CodeXGLUE using CodeSearchNet and conditions completion on file-level information.Source: §5.2 · p. 5

5.3 Docstring Completion/Code Summarization

  • Docstring completion predicts a docstring from a focal method and optional context, while code summarization evaluates the summary in the docstring’s first paragraph.Training uses docstring completion; CodeSearchNet evaluation attempts to retain only the summary.Source: §5.3 · p. 5

6 Model Training

The section describes XGPT-C and XPyMT5 training configurations, alongside Reformer and Performer comparison models trained without eWASH.

6.1 XGPT-C

  • XGPT-C uses a 12-layer, 12-head Transformer decoder with 768 hidden dimensions and 125M parameters, trained for two weeks on 16 Tesla V100 GPUs.It follows GPT-C with a causal language-modeling objective and a 50,000-token SentencePiece BPE vocabulary.Source: §6.1 · p. 6

6.2 XPyMT5

  • XPyMT5 shares PyMT5’s architecture and hyperparameters—12 layers and 406M parameters—and uses its whitespace-augmented GPT-2 tokenizer (about 50,000 vocabulary); it was initialized with English pretrained BART weights (with whitespace embeddings).XPyMT5 was trained on 16 32GB Tesla V100 GPUs for 4 weeks, about 10 epochs total, then pre-trained using the BART de-noising objective for 5 weeks on the same hardware.Source: §6.2 · p. 6

7 Evaluation

The evaluation compares eWASH models with baselines across code completion, method completion, and docstring completion. Code completion improves across ExactMatch metrics, while method and docstring completion results favor models conditioned on eWASH features.

7.1 Metrics

  • The evaluation reports ROUGE-L, edit distance, and ExactMatch@N for code completion, plus BLEU-4, ROUGE-L, and syntax correctness for method completion.Exact matches are omitted for longer method and docstring outputs because the metric is too strict to interpret meaningfully.Source: §7.1 · p. 6

7.2 Experimental Conditions

  • For developer-intent evaluation, the authors sampled 833 Python functions and prompted at points within methods; inference used beam search with width 5.Method and docstring evaluations used re-downloaded CodeSearchNet repositories and commit hashes to extract eWASH features.Source: §7.2 · p. 6

7.2.1 Code Completion Evaluation Results

  • XGPT-C beat GPT-C and memory-efficient transformers on all computed metrics; normalized literals added 13% absolute ExactMatch@5, with benefits also observed at shorter context lengths.On files longer than 1,024 tokens, Performer beat XGPT-C only on EM@5, at 55.9%. With seven same-line context tokens, XGPT-C completed five tokens exactly over 80% of the time, versus just under 60% for GPT-C.Source: §7.2.1 · pp. 7–8
  • Over 12% more top-1 exact matches and 88.7% overall top-5 exact matches were reported for XGPT-C on CodeXGLUE.The comparison is between XGPT-C and GPT-C.Source: Table 3 · p. 7, §7.2.1 · p. 8

7.3 Method Completion Evaluation Results

  • XPyMT5 outperformed PyMT5 on all reported method source-target metrics, with dramatically improved ROUGE-L; syntax correctness was slightly lower.The authors note that the syntax-correctness difference may not be meaningful.Source: §7.3 · p. 8

7.4 Docstring Completion Evaluation Results

  • For docstring completion, XPyMT5 improved across all reported metrics and doubled the ROUGE-L F1 score relative to PyMT5.Source: §7.4 · p. 8

8 Conclusions

The conclusions report that code-completion results support the hypothesis that developer-imposed syntax hierarchies are useful signals, with the greatest relative benefit at shorter context lengths. They also report improved method completion and code summarization with XPyMT5 and propose eWASH as a general-purpose approach for structured documents.

8 Conclusions

  • eWASH produced large relative improvements in code completion under strict exact-match metrics, with the greatest relative benefit at shorter context lengths.The authors also report dramatic improvement in method completion and code summarization with XPyMT5.Source: §8 · p. 8
  • The code-completion results supported the hypothesis that developer-imposed syntax hierarchies signal importance for a task context.The authors report that eWASH also benefits code completion when context lengths do not exceed the model's context window.Source: §8 · p. 8
  • The authors describe eWASH as applicable to any programming language and, in principle, languages with hierarchical syntactic or stylistic structure.They state that models including GPT-3 can leverage the feature and suggest it could improve natural-language understanding tasks.Source: §8 · p. 8
  • The paper adds CodeXGLUE tasks for literal-normalized code completion, method-level code completion, and method/docstring completion conditioned on whole-file context.Source: §8 · p. 8

A.1 Code Completion Above and Below the Context Window

The appendix divides code-completion test samples by file length around the 1024-token context window. It describes two tables evaluating XGPT-C on methods sampled from files exceeding that length.

A.1 Code Completion Above and Below the Context Window

  • The test set is split into 90% of samples with fewer than 1024 tokens and 10% with more than 1024 tokens.Source: §A.1 · p. 10
  • Table 6 reports XGPT-C code-completion evaluation on sampled methods from files with more than 1024 tokens.The samples are from the CodeXGLUE code-completion test task.Source: Table 6 · p. 11
  • Table 7 also reports XGPT-C code-completion evaluation on sampled methods from files with more than 1024 tokens.The samples are from the CodeXGLUE code-completion test task.Source: Table 7 · p. 11

Related papers

Loading 2109.08780v1…