PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Networks for Modeling Source Code Edits

Rui Zhao, David Bieber, Kevin Swersky, Daniel Tarlow

arXiv:1904.02818v1cs.LGcs.CLcs.SEstat.ML

TL;DR

This paper asks how past source-code edits can reveal developer intent and support prediction of future edits. It models edit sequences with neural networks and evaluates them on synthetic patterns and fine-grained Google code changes. The two-headed attention model is particularly well-suited to accuracy, confidence calibration, and scalability on real data, providing preliminary evidence for edit-prediction tools.

  • Problem

    Predicting future source-code edits requires representing how each change relates to the code state where it occurred, rather than relying only on edit content or the resulting state.

    Source: §1 · p. 1

  • Method

    The paper develops neural models for explicit full-state and implicit position-content representations of edit sequences, including a two-headed attention model.

    Source: §1 · pp. 1–2, §4 · p. 4

  • Results

    The two-headed attention model was particularly well-suited to high accuracy, well-calibrated confidence, and scalability on real data.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results provide preliminary evidence that tools can learn to predict future edits from developers’ changes.

    Source: §1 · p. 2

    The formulation assumes edits between snapshots occur in left-to-right order, an assumption the authors call unrealistic.

    Source: §8 · p. 9

Abstract

from arXiv · show

Programming languages are emerging as a challenging and interesting domain for machine learning. A core task, which has received significant attention in recent years, is building generative models of source code. However, to our knowledge, previous generative models have always been framed in terms of generating static snapshots of code. In this work, we instead treat source code as a dynamic object and tackle the problem of modeling the edits that software developers make to source code files. This requires extracting intent from previous edits and leveraging it to generate subsequent edits. We develop several neural networks and use synthetic data to test their ability to learn challenging edit patterns that require strong generalization. We then collect and train our models on a large-scale dataset of Google source code, consisting of millions of fine-grained edits from thousands of Python developers. From the modeling perspective, our main conclusion is that a new composition of attentional and pointer network components provides the best overall performance and scalability. From the application perspective, our results provide preliminary evidence of the feasibility of developing tools that learn to predict future edits.

1 INTRODUCTION

The paper frames edit prediction as learning intent from the relationship between changes and the code state where they occur. It introduces edit-sequence representations and models, then evaluates them on synthetic patterns and fine-grained Google code edits.

1 INTRODUCTION

  • Edit prediction requires learning intent from the relationship between a change and the code state where it was made, not just the inserted or deleted content or resulting state.The representation of past edits must support generalization to future edits.Source: §1 · p. 1
  • Two histories can reach the same code state yet imply different next edits, so the model must use the preceding edits to predict what comes next.The example contrasts adding an argument at a call site with removing an argument from a function definition.Source: §1 · p. 1, Figure 1 · p. 2
  • The paper explores explicit and implicit edit representations, including a recurrent pointer network baseline and an implicit model with attention and separate heads for edit positions and contents.The models are intended to illuminate tradeoffs among edit-sequence formulations.Source: §1 · pp. 1–2
  • The study evaluates models on carefully designed synthetic data and fine-grained edits from Google code snapshots, finding the two-headed attention model particularly well-suited to accuracy, confidence calibration, and scalability on real data.The snapshots are saved at each file save, yielding finer-grained edits than typical Git commits.Source: §1 · p. 2

2 PROBLEM FORMULATION

The problem formulation represents an edit sequence either as successive full code states or as an initial state followed by compact position-and-content edits. The two formats trade ease of learning and generalization against scalability.

2 PROBLEM FORMULATION

  • The explicit format stores each edit sequence as successive token sequences, while the implicit format stores the initial token sequence and subsequent edits as position-content pairs.The explicit format indexes tokens within each state and time across states.Source: §2 · p. 2
  • Explicit representations are easier to model accurately but scale poorly; implicit representations scale more readily but baseline models struggle with complex edit patterns and generalization.The improved implicit model is presented as combining scalability with generalization ability.Source: §2 · p. 2
  • In the implicit format, inserted tokens receive indices after the initial-state tokens, allowing later edits to refer to earlier insertions.The example assigns index 6 to the first inserted B, which the next edit uses as its insertion position.Source: §2 · p. 3
  • The explicit learning task predicts the next edit from prior states, whereas the implicit task predicts it from the initial state and prior edits.Both formulations define the prediction target at each time step.Source: §2 · p. 3

3 BASELINE MODELS

The baseline models implement the explicit and implicit edit representations with different sequence architectures. The explicit model processes full states at each time step; the implicit model predicts a sequence of position-content edit pairs.

3 BASELINE MODELS

  • The explicit baseline uses a two-level LSTM: one level encodes tokens within each state, and the other processes state representations across time.Its output predicts edit content, while token-level hidden states support position prediction.Source: §3 · p. 3
  • The explicit baseline uses a pointer-network construction to predict edit positions from token-level hidden states.The model also provides the first-level LSTM with the previous token at the same position as context.Source: §3 · p. 3
  • The implicit baseline encodes the initial state and decodes the sequence of edits, treating each action as a position-content pair.The decoder predicts position first, then content conditioned on that position.Source: §3 · p. 3
  • For implicit position inputs, the model embeds each integer position and combines it with a content embedding to form decoder inputs.A learned output matrix produces the position distribution.Source: §3 · p. 3

4 IMPLICIT ATTENTION MODEL

The implicit attention model represents edits compactly while using attention to connect each edit with the context in which it occurred. It predicts edit positions and contents through separate decoder heads, with masking that prevents future edits from informing past predictions.

4 IMPLICIT ATTENTION MODEL

  • The model predicts all edits in one training pass, using an encoder for the initial state and edits and separate decoder heads for positions and contents.Masked attention prevents information from future edits flowing backward to past edits.Source: §4 · p. 4
  • The encoder embeds initial-state tokens and edits independently, then uses attention to exchange information across them into hidden representations.These representations are intended to retain relevant context for each initial and edit position.Source: §4 · p. 4
  • The decoder gathers hidden states at prior edit positions into context vectors, forming a compact subsequence of the contexts in which edits occurred.This context matrix is used by the decoder’s two prediction heads.Source: §4 · p. 4
  • The position head predicts a query from previously edited contexts and compares it with encoder states to assign probabilities to the next edit position.The paper illustrates the pattern with edits that append B after successive A characters.Source: §4 · p. 4
  • The content head predicts edit contents given prior edits and the current position, using either a shifted-content decoder or an analogical decoder.The analogical decoder attends over differences between shifted content and context representations, then applies predicted analogies to current contexts.Source: §4 · pp. 4–5

5 SYNTHETIC DATASETS

The synthetic datasets use regex-style replacements on random strings to test whether models can learn edit patterns that require generalization. Instances are constructed as sequences of complete-replacement snapshots and token-level edits.

5 SYNTHETIC DATASETS

  • The datasets use regular-expression replacement patterns and replacements over uniformly random initial strings drawn from a vocabulary.Patterns can include groups and meta characters, while replacements can refer back to captured groups; meta characters vary across examples.Source: §5 · p. 5
  • Each instance contains at least four snapshots, each marking a state after a complete pattern replacement; diffs between snapshots are converted into left-to-right, one-token edits.A Python diff library resolves the non-unique choices of edits between successive snapshots.Source: §5 · p. 5
  • Models receive conditioning edits until the replacement pattern becomes unambiguous, and a MultiTask dataset combines instances from the individual datasets.The number of conditioning steps depends on the dataset pattern.Source: §5 · p. 5

6 EXPERIMENTS

Across real Python edit sequences, the improved implicit attention model achieved the highest reported subtoken accuracy, while using less memory than the explicit baseline. Synthetic experiments also assessed generalization and scalability, and confidence-based evaluation found that the attention model performed best across thresholds.

6 EXPERIMENTS

  • The experiments evaluate E, IR, and IA for edit-pattern accuracy and scalability, using synthetic tasks first and real data afterward.Synthetic accuracy measures an edit as correct only when both its position and content are correct.Source: §6 · p. 5, §6.1 · p. 6

6.1 EXPERIMENTS ON SYNTHETIC DATA

  • On synthetic tasks, E and IA solved nearly all tasks, including meta-character and long-replacement cases, while POMP was near chance on many tasks.POMP’s low accuracy indicates that most test replacements were not seen during training; its results are reported alongside E, IR, and IA test accuracies.Source: §6.1 · p. 6, Table 1 · p. 6
  • At 100 insertions, E was already an order of magnitude slower despite sequences being 10 times smaller than real data; attention models took 50% to 75% of IR’s training time.The explicit model was consistently more expensive than implicit models, and the gap grew as data size increased.Source: §6.1 · pp. 6–7, Figure 4 · p. 7

6.2 EXPERIMENTS ON REAL DATA

  • The real-data corpus contains 8 million edits from 5,700 Google developers, grouped into instances of about 100 edits and split 80%/10%/10% for training, development, and test.The dataset uses Python snapshots tokenized into subtokens, and instances with states longer than 1,000 subtokens were pruned.Source: §6.2 · p. 7
  • 61.1% subtoken-level test accuracy for IA exceeded 55.5% for IR and 51% for E; IA also had fewer memory-constraint issues than E.The authors identify IA as a good tradeoff and express optimism about scaling it to larger datasets with longer initial states.Source: §6.2 · p. 7
  • For greedy autoregressive predictions through the next token boundary, IA outperformed the other models across confidence thresholds, with higher accuracy among predictions made at higher confidence.The authors suggest confidence could help decide when to trigger edit suggestions.Source: §6.2 · p. 7

7 RELATED WORK

Prior code models largely generate static snapshots, while earlier edit-history work either uses coarse-grained correlations or targets predefined goals. This work instead learns token-level position and content predictions from full edit histories.

7 RELATED WORK

  • Earlier iterative-refinement methods, including DeepFix, pursue predefined goals rather than inferring intent from past edits.Those methods propagate the resulting state, which may not preserve enough information to disambiguate an edit history.Source: §7 · p. 8
  • Static code models generate single snapshots, whereas this work models the temporal sequence of token-level edits.The paper contrasts its formulation with models of source code that generate one snapshot.Source: §7 · p. 8
  • Prior software-engineering edit studies track co-changed files or predict locations and API content at coarser granularity than tokens.The cited work uses frequency counts for edit-pair correlations; this paper predicts both position and content from full histories.Source: §7 · pp. 8–9
  • The paper treats code as token sequences, while identifying tree- or graph-based representations as a future direction.It describes source-code representation as mostly orthogonal to the edits-based formulation.Source: §7 · p. 8
  • Raza et al. (2014) predicts analogous presentation edits with a domain-specific language and heuristic, unlike this paper’s learned general edit space.The paper characterizes learning as better suited to noisy real-world settings.Source: §7 · p. 8

8 DISCUSSION

The paper formulates prediction of future edits from past edit histories and reports models applicable to large-scale source-code edit data. It identifies left-to-right edit ordering as an unrealistic assumption and suggests broader prediction tasks for future exploration.

8 DISCUSSION

  • The authors formulate learning from past edits to predict future edits and demonstrate applicability to large-scale source-code edit data.They report developing models capable of strong generalization.Source: §8 · p. 9
  • The formulation assumes edits occur left to right; the authors suggest treating their order as a latent variable under weak supervision.They describe the left-to-right assumption as unrealistic.Source: §8 · p. 9
  • Possible extensions include conditioning on cursor position for autocomplete or predicting a developer’s next code-search query from recent edits.The authors also propose studying other predictions about what a developer will do next.Source: §8 · p. 9

OVERVIEW

  • The appendix outlines additional synthetic-data, real-data, model-variant, and experiment details across Sections A–F.It includes further synthetic-data experiments and IA model-variant experiments.Source: OVERVIEW · p. 11

A ADDITIONAL SYNTHETIC DATASET INFORMATION

The appendix describes synthetic dataset properties and relates problem difficulty to edits per replacement, context displacement, and test-pattern novelty. Its Figure 5 examines accuracy over prediction timesteps for two illustrative problems.

A ADDITIONAL SYNTHETIC DATASET INFORMATION

  • Table 2 lists the regex patterns and replacements defining the synthetic datasets, alongside their computed properties.Its properties include edits per replacement and average context displacement.Source: A ADDITIONAL SYNTHETIC DATASET… · p. 11, Table 2 · p. 13
  • Higher edits per replacement corresponds to higher accuracy when repeated actions from past ground-truth edits are available.Problems repeating a context-based action only once are harder; MetaContextAppend11 is harder than MetaContextAppend13.Source: A ADDITIONAL SYNTHETIC DATASET… · p. 11
  • Higher average context displacement indicates harder problems and correlates with lower accuracy for E, IA-ag, and IR.ACD measures the distance between regex-pattern characters and where they are used in the replacement.Source: A ADDITIONAL SYNTHETIC DATASET… · p. 12
  • Accuracy of E, IA-ag, and IR moderately correlates with POMP accuracy on non-meta problems, suggesting test-pattern novelty relates to difficulty.IR tends to have lower accuracy on problems with more novel test-time patterns.Source: A ADDITIONAL SYNTHETIC DATASET… · p. 12
  • Figure 5 plots dev accuracy by prediction timestep for two illustrative synthetic problems and marks insertion locations in example edit sequences.The example sequences are shortened for illustration.Source: Figure 5 · p. 14

B.1 ADDITIONAL EXPERIMENT DETAILS

The appendix reports synthetic-experiment optimization settings, scalability measurement conditions, and error patterns across prediction timesteps. Accuracy dips most when the model must choose a new edit location at a cross-pattern jump.

B.1 ADDITIONAL EXPERIMENT DETAILS

  • Synthetic experiments select learning rate and hidden size by grid search, use ADAM with default TensorFlow settings, and clip gradients at 1.0.The search tests learning rates .005, .001, .0005, and .0001 and hidden sizes 128, 256, and 512.Source: §B.1 · p. 14

B.2 ADDITIONAL SCALABILITY EXPERIMENT DETAILS

  • Scalability is measured as average training time for 128 sequences on one P100 GPU over 1,000 steps, with batch size 64 across hidden sizes 128, 256, and 512.For larger datasets, hidden size 512 cannot fit batch size 64 for the explicit model, requiring batch size 32 steps.Source: §B.2 · p. 14

B.3 ANALYZING ERRORS

  • Accuracy dips when the target position changes non-trivially between timesteps, making cross-pattern jumps the hardest predictions.For the best-performing models, accuracy stays near 1 when the position increments by 1; fixed-size substitution patterns make the jumps and accuracy dips regular.Source: §B.3 · p. 14

C.1 ADDITIONAL EXPERIMENT DETAILS

The appendix details the implicit model’s parallel encoder, position-and-content decoder, and attention-module variants, alongside the hyperparameter search used for code-edit experiments.

C.1 ADDITIONAL EXPERIMENT DETAILS

  • Each model variant was trained for 48 hours on one P100 GPU after a grid search over learning rates, hidden sizes, and memory-constrained batch sizes.The search covered learning rates .005, .001, .0005, and .0001; batch sizes decreased as hidden size increased for explicit models.Source: §C.1 · p. 14

D.1 ENCODER

  • Edit embeddings combine content, explicit position, and a shared marker for edit-produced tokens before parallel multihead attention produces encoder outputs.The encoder avoids sequential dependence; explicit positions may repeat, with the shared marker distinguishing edited tokens from initial-state tokens.Source: §D.1 · p. 15

D.2 DECODER

  • The decoder predicts edit positions with a pointer over encoded contexts, then predicts content conditioned on the selected position and prior edits.The vanilla content decoder combines prior edit content with current edit context; the analogical decoder predicts transformations between prior content and context.Source: §D.2 · pp. 15–16

E ADDITIONAL MULTIHEAD ATTENTION MODULE DETAILS

  • Attention modules add positional encoding, apply multihead attention, and aggregate by summation or a GRU, whose learned gate can weight inputs instance by instance.The reported implicit variants combine vanilla or analogical decoders with either aggregation choice.Source: E ADDITIONAL MULTIHEAD ATTENTION… · p. 17

F ADDITIONAL IMPROVED IMPLICIT MODEL EXPERIMENTS

The appendix repeats the synthetic experiments across implicit-attention variants and reports their test accuracy at the hyperparameter and training step with best development accuracy.

F ADDITIONAL IMPROVED IMPLICIT MODEL EXPERIMENTS

  • The synthetic experiments compare four implicit-attention variants, with test accuracy reported at each variant’s best-development-accuracy hyperparameter and step.The variants differ by vanilla or analogical decoder and sum or GRU aggregation; the supplied passage does not provide their accuracy values.Source: F ADDITIONAL IMPROVED IMPLICIT MODEL… · p. 17, Table 3 · p. 18
Loading 1904.02818v1…