PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Generalizability of Neural Program Models with respect to Semantic-Preserving Program Transformations

Md Rafiqul Islam Rabin, Nghi D. Q. Bui, Ke Wang, Yijun Yu, Lingxiao Jiang, Mohammad Amin Alipour

arXiv:2008.01566v3cs.SEcs.LGcs.PL

TL;DR

The extent to which neural program models generalize to unseen programs is largely unknown. This paper evaluates three models on method-name prediction before and after semantic-preserving transformations, finding that even small changes can alter predictions. Models encoding data and control dependencies generalize better than AST-only models, and larger training datasets improve generalizability in most cases.

  • Problem

    How well neural program models generalize to unseen programs is largely unknown, motivating evaluation on semantically equivalent programs with different lexical appearances and structures.

    Source: §1 · pp. 1–2

  • Method

    The study applies six semantic-preserving transformations to Java method bodies and compares code2vec, code2seq, and GGNN predictions on original and transformed programs.

    Source: §1 · p. 2, §4.2 · p. 4

  • Results

    All evaluated models are sensitive to semantic-preserving transformations, while models encoding data and control dependencies generalize better than AST-only models and larger training datasets improve generalizability in most cases.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results indicate that neural program model generalizability remains limited and merits further attention, including work on more generalizable models or preprocessing such as canonicalizing program representations.

    Source: §1 · p. 2

    Evaluation covers code2vec, code2seq, and GGNN on one Java task, so results may not generalize to other models, tasks, or programming languages.

    Source: §9 · pp. 12–13

Abstract

from arXiv · show

With the prevalence of publicly available source code repositories to train deep neural network models, neural program models can do well in source code analysis tasks such as predicting method names in given programs that cannot be easily done by traditional program analysis techniques. Although such neural program models have been tested on various existing datasets, the extent to which they generalize to unforeseen source code is largely unknown. Since it is very challenging to test neural program models on all unforeseen programs, in this paper, we propose to evaluate the generalizability of neural program models with respect to semantic-preserving transformations: a generalizable neural program model should perform equally well on programs that are of the same semantics but of different lexical appearances and syntactical structures. We compare the results of various neural program models for the method name prediction task on programs before and after automated semantic-preserving transformations. We use three Java datasets of different sizes and three state-of-the-art neural network models for code, namely code2vec, code2seq, and GGNN, to build nine such neural program models for evaluation. Our results show that even with small semantically preserving changes to the programs, these neural program models often fail to generalize their performance. Our results also suggest that neural program models based on data and control dependencies in programs generalize better than neural program models based only on abstract syntax trees. On the positive side, we observe that as the size of the training dataset grows and diversifies the generalizability of correct predictions produced by the neural program models can be improved too. Our results on the generalizability of neural program models provide insights to measure their limitations and provide a stepping stone for their improvement.

1 INTRODUCTION

The extent to which neural program models generalize to unseen programs remains unknown, despite their use in program analysis. This paper evaluates generalizability through semantic-preserving transformations and analyzes how models respond to them.

1 INTRODUCTION

  • The extent to which neural program models generalize to unseen programs remains unknown, including programs in the same language.The authors note the importance of understanding this for prospective safety-critical uses such as malware detection and automated defect repair.Source: §1 · p. 1
  • Comparing predictions before and after controlled semantic-preserving transformations provides a systematic, quantifiable way to evaluate generalizability to semantically equivalent programs.The authors distinguish this evaluation from attempting to test every possible unseen program.Source: §1 · p. 2
  • The study evaluates code2vec, code2seq, and GGNN by transforming test programs into semantically equivalent counterparts using six transformations.The transformations range from variable renaming to changing for-loops into while-loops.Source: §1 · p. 2
  • All evaluated models are sensitive to semantic-preserving transformations, even small changes such as variable renaming or reordering independent statements.Models encoding data and control dependencies generalize better than models based only on abstract syntax trees; larger training datasets improve generalizability in most cases.Source: §1 · p. 2
  • The authors describe their evaluation as complementary to adversarial code generation and robustness work, examining many transformations and model behavior in depth.They also evaluate how dataset and program size affect generalizability.Source: §1 · p. 2
  • The paper introduces generalizability under semantic-preserving transformations and presents a large-scale evaluation with analysis of prediction changes and the effects of dataset and program size.It also discusses practical implications for existing neural program models.Source: §1 · p. 2

2 MOTIVATING EXAMPLE & DEFINITION

A variable-renaming example shows that code2vec can predict different method names for semantically identical methods. The paper defines generalizability in terms of consistent outputs under semantic-preserving transformations.

2 MOTIVATING EXAMPLE & DEFINITION

  • Renaming one variable from “other” to “var01” changes code2vec’s prediction from compareTo to getCount, despite unchanged method semantics.The example suggests the prediction relies substantially on identifier names.Source: §2 · p. 2
  • Generalizability means a neural program model returns the same results under semantic-preserving transformations.The paper frames its approach as a systematic, quantifiable way to measure this capability, with transformations and their prediction impact as its basis.Source: §2 · pp. 2–3
  • The paper distinguishes its use of generalizability from robustness, which it associates with adversarial examples and imperceptible input changes.Its source-code transformations can visibly change textual appearance and syntactic structure; quantifying imperceptibility in source code is left for future research.Source: §2 · p. 3

3 BACKGROUND

Neural program model performance depends on data quality, source-code representation, and neural-network characteristics.

3 BACKGROUND

  • Neural program model performance depends on data quality, representation, and neural-network characteristics and training parameters.The paper identifies these as three main factors affecting performance.Source: §3 · p. 3
  • Data quality concerns whether source code is representative and properly cleaned, while available datasets are described as immature, nonstandardized, and of somewhat unknown quality.The passage cites code duplication as a dataset issue that can greatly affect model performance.Source: §3 · p. 3
  • Source-code representations determine which program features enter neural-network vector embeddings, while network choices include architecture type, topology, and hyperparameters.The paper notes recurrent and graph neural networks among popular architectures in neural program models.Source: §3 · p. 3

4 EVALUATION APPROACH

The evaluation compares model outputs on original methods and transformed, semantically equivalent methods. It uses method-name prediction and measures both prediction changes and conventional subtoken scores.

4 EVALUATION APPROACH

  • The evaluation uses a metamorphic relation: predictions should remain unchanged between semantically equivalent programs, tested by generating transformed programs and comparing outputs.These two steps are used to compute generalizability metrics.Source: §4 · p. 3

4.1 Target Downstream Task

  • Method-name prediction asks models to predict a method’s name from its body, a task also used in code search, summarization, and analogies.The task supplies a method body and returns candidate names.Source: §4.1 · p. 3

4.2 Transformations

  • Six method-level transformations generate semantically equivalent methods, ranging from variable renaming and statement permutation to loop exchange, switch-to-if conversion, and boolean exchange.The set also includes inserting an unused string declaration; transformations differ in their impact on method AST structure.Source: §4.2 · p. 4

4.3 Generalizability Metrics

  • Generalizability metrics compare original and transformed predictions, including prediction change percentage and five correctness-change categories; precision, recall, and F1-score assess predicted method-name subtokens.The five categories distinguish correct predictions that stay correct or become wrong, and wrong predictions that stay the same, become correct, or change to a different wrong prediction.Source: §4.3 · p. 4

5 EXPERIMENTAL SETTING

The experimental setting compares three neural program models for method-name prediction across Java datasets of different sizes, using transformed test programs and research questions about prediction changes, method length, and metrics.

5.1 Subject Neural Program Models

  • code2vec represents methods with AST paths, code2seq encodes AST paths and generates names as sub-token sequences, and GGNN encodes AST graphs augmented with semantic edges.GGNN aggregates neighboring node embeddings through message passing before its decoder generates the method name.Source: §5.1 · p. 5

5.2 Datasets

  • The study used Java-Small, Java-Med, and Java-Large, containing about 700K, 4M, and 16M methods, respectively.The datasets span 9, 800, and 9,000 training projects, respectively.Source: §5.2 · p. 5

5.3 Training Models per Datasets

  • Table 1 reports test-set performance: code2seq was on par with, code2vec resembled but trailed, and GGNN differed from amid parser, edge, and dataset differences.The GGNN graphs included seven of the ten semantic edge types proposed in.Source: Table 1 · p. 5, §5.3 · p. 5

5.4 Population of Transformed Programs

  • The authors transformed 1,415,116 original test methods into 2,822,810 single-place transformed methods, applying each transformation separately at eligible locations.Two authors implemented and reviewed the transformations, and manually inspected samples for correctness.Source: §5.4 · pp. 5–6

5.5 Research Questions

  • The research questions address prediction changes and when they occur, method-length effects, transformation-type trends, and impacts on precision, recall, and F1-score.RQ1 concerns the single-place transformed dataset.Source: §5.5 · p. 6

6 RESULTS

Across the evaluated models, semantic-preserving transformations often change predictions, though GGNN is generally less affected than code2vec and code2seq. The results also vary with transformation type, dataset size, method length, and whether performance is measured by prediction changes or classic metrics.

6.1 RQ1: Impact of Transformations on the Predictions of Neural Program Models

  • In 14 of 18 cases, GGNN’s PCP is significantly lower than code2vec’s and code2seq’s, and its weighted-average PCP is lower for four of six transformations.All three models are susceptible to semantically equivalent transformations, but the impact differs across models and datasets.Source: §6.1 · p. 6, Table 2 · p. 7
  • For code2vec and code2seq, PCP is generally higher on Java-Small than larger datasets; GGNN’s PCP is usually lowest on Java-Small and highest on Java-Med.Transformation sensitivity also varies by model and transformation: code2vec is most sensitive overall to Permute Statement, code2seq to Boolean Exchange, and GGNN to Switch to If.Source: §6.1 · pp. 6–7

6.2 RQ2: When Transformations Affect Neural Program Models the Most?

  • All-place transformations generally change predictions more than single-place transformations for code2vec and code2seq, while GGNN’s results remain relatively similar across both.The code2seq Switch to If result on Java-Small is an exception, but only 13 transformed methods were available for that comparison.Source: §6.2.1 · pp. 7–8, Figure 2 · p. 8
  • For all models, transformations change predictions more often in originally incorrect methods; GGNN is more stable than the other models on originally correct methods.For GGNN, prediction changes range from 1.90% to 8.58% on originally correct methods and from 31.05% to 62.01% on originally incorrect methods.Source: Figure 3 · p. 8, §6.2.2 · p. 8
  • Under X%-transformations, GGNN has much lower PCP than code2vec and code2seq across transformations and datasets, with little change as X varies.For code2vec and code2seq, PCP for Variable Renaming tends to increase with X; other transformations show modest changes, but their transformed-program counts may be too low for statistical significance or comparison.Source: Table 3 · p. 9, §6.2.3 · p. 9

6.4 RQ4: Trends in the Types of Changes

  • For code2vec and code2seq, the proportion of correct predictions that remain correct rises with dataset size; transformations are more likely to turn correct predictions wrong than wrong predictions correct.The correct-to-wrong switch occurs on average in 23% of cases for code2vec and 20% for code2seq, versus less than 5% for GGNN.Source: §6.4 · pp. 9–10

6.3 RQ3: Impact of Method Length on Generalizability

  • As method length increases, code2vec and code2seq generally show notable PCP increases, while GGNN appears less sensitive to method length.Method length is measured by the number of executable lines before transformation.Source: §6.3 · p. 9, Figure 4 · p. 10

6.5 RQ5: Impact of the Transformations on Precision, Recall, and 𝐹1-Score

  • Average precision, recall, and F1-score decrease after transformations across all three models and Java datasets, but these metric changes do not correspond obviously to PCP.High PCP does not necessarily entail large changes in precision, recall, or F1-score, and vice versa.Source: §6.5 · pp. 10–11

7 DISCUSSION

The discussion weighs neural networks’ appeal for program analysis against their interpretability and generalizability challenges. It argues that transformation-based evaluation complements interpretability, while emphasizing the study’s narrow task scope and the need for better code representations.

7 DISCUSSION

  • Neural networks learn representations from raw data and can reduce feature-engineering burden with sufficiently large datasets, but whether they are the best way to process programs remains unknown.Their large hypothesis class and feature learning make them appealing, while their complexity makes them difficult to interpret.Source: §7 · p. 12

7.1 Generalizability vs. Interpretability vs. Robustness and Others

  • The study’s transformation-based evaluation offers a perspective complementary to interpretability and may help identify interpretable code elements by measuring the impact of code transformations.The authors frame this as a potential future use of the approach.Source: §7.1 · p. 12
  • Robustness tests based on imperceptible input changes in vision may not transfer directly to programs, where minor edits can drastically alter semantics and behavior.Quantifying source-code imperceptibility is left for future work.Source: §7.1 · p. 12

7.2 Are we there yet?

  • Semantic-preserving transformations changed model outputs in 26% to 73% of cases, but the findings cover only one task and therefore suggest caution about widespread use.The discussion also points to interpretability and testing work as needed steps toward rigorous evaluation.Source: §7.2 · p. 12

7.3 Code Representation

  • Models used in neural program analysis perform relatively poorly compared with neural models in natural-language understanding and text classification, motivating code representations that better capture program characteristics.Source: §7.3 · p. 12

8 RELATED WORK

Related work spans robustness evaluation, verification, and testing of neural networks, alongside code models that increasingly represent program structure, graphs, and runtime information.

8 RELATED WORK

  • Robustness research includes adversarial methods for image, language, and graph models, while verification and testing work studies guarantees and challenges for machine-learning systems.The cited approaches include bounded model checking, abstract interpretation, and satisfiability modulo theories.Source: §8 · p. 12
  • Code models progressed from NLP methods that miss program structure to AST-based approaches, graph neural networks, attention mechanisms, and models using runtime information.Source: §8 · p. 12

9 THREATS TO VALIDITY

The evaluation is limited to three model families, one task, and Java programs, so its results may not generalize to other models, tasks, or languages.

9 THREATS TO VALIDITY

  • The evaluation covers code2vec, code2seq, and GGNN on one task in Java, and its results may not generalize to other models, tasks, or programming languages.The transformations may also miss transformations from other domains; the authors plan to extend the approach, and code-review and manual inspections were used to reduce toolchain and implementation bug risk.Source: §9 · pp. 12–13

10 CONCLUSION & FUTURE WORK

The paper evaluates code2vec, code2seq, and GGNN using semantic-preserving transformations and finds that these changes frequently alter predictions. It concludes that generalizability remains limited, though dependency-aware models and larger training datasets may show better behavior.

10 CONCLUSION & FUTURE WORK

  • Semantic-preserving transformations frequently sway predictions from code2vec, code2seq, and GGNN, indicating serious generalization issues despite potentially better behavior from dependency-aware models and larger training datasets.Source: §10 · p. 13
  • The authors present a systematic evaluation approach and metrics for assessing neural program model generalizability.Source: §10 · p. 13
  • Future work may extend the approach with additional semantic-preserving and semi-semantic-preserving transformations, finer-grained prediction-change metrics, and transformed training data.Source: §10 · p. 13
Loading 2008.01566v3…