PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Reliability and Explainability of Language Models for Program Generation

Yue Liu, Chakkrit Tantithamthavorn, Yonghui Liu, Li Li

arXiv:2302.09587v3cs.SE

TL;DR

Language-model program generation has uncertain reliability and explainability, including concerns about opaque predictions and biased performance evaluation. The paper empirically studies eight models across program-generation tasks and datasets, using explainability methods to inspect token contributions. It finds that duplication skews performance estimates and that models show limited robustness to input changes.

  • Problem

    The reliability and explainability of language models for program generation remain uncertain, with opaque predictions and dataset bias complicating assessment.

    Source: §1 · p. 2

  • Method

    The study evaluates eight pre-trained models on program-generation tasks and datasets and applies SHAP-based explainability analysis to token contributions.

    Source: §1 · p. 3, §2.2.2 · p. 5

  • Results

    Duplicated and overlapping data skews performance evaluations, while models recognize code structure but show poor robustness to minor input changes.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The findings point to opportunities to improve program-generation reliability and show that explainable AI can help characterize model inference behavior.

    Source: §7 · p. 22

    Results are bounded by the studied datasets and generalizability, and the study does not seek performance gains through hyperparameter tuning.

    Source: §5.2 · p. 20

Abstract

from arXiv · show

Recent studies have adopted pre-trained language models, such as CodeT5 and CodeGPT, for automated program generation tasks like code generation, repair, and translation. Numerous language model-based approaches have been proposed and evaluated on various benchmark datasets, demonstrating promising performance. However, there is still uncertainty about the reliability of these models, particularly their realistic ability to consistently transform code sequences. This raises the question: are these techniques sufficiently trustworthy for automated program generation? Consequently, Further research is needed to understand model logic and assess reliability and explainability. To bridge these research gaps, we conduct a thorough empirical study of eight popular language models on five representative datasets to determine the capabilities and limitations of automated program generation approaches. We further employ advanced explainable AI approaches to highlight the tokens that significantly contribute to the code transformation. We discover that state-of-the-art approaches suffer from inappropriate performance evaluation stemming from severe data duplication, causing over-optimistic results. Our explainability analysis reveals that, in various experimental scenarios, language models can recognize code grammar and structural information, but they exhibit limited robustness to changes in input sequences. Overall, more rigorous evaluation approaches and benchmarks are critical to enhance the reliability and explainability of automated program generation moving forward. Our findings provide important guidelines for this goal.

1 INTRODUCTION

Prior work leaves the reliability and interpretability of language-model program generation uncertain, while experimental bias and limited practicality complicate assessment. This study evaluates eight models and finds that duplicated data skews performance estimates, while explanations show limited robustness to input changes.

1 INTRODUCTION

  • Prior work’s interpretability gap leaves practitioners without evidence about which input characteristics contribute to model predictions, hindering real-world adoption.The passage notes that complex model architectures make specific predictions difficult to explain.Source: §1 · p. 2
  • Noisy dataset composition can inflate reported performance: one prior method’s BLEU-4 score fell from 42.4% to 26.2% after noisy data was removed.The passage says this makes the actual capabilities of models difficult to evaluate accurately.Source: §1 · p. 2
  • A prior code-change approach reached 21.16% accuracy with one candidate and 36.02% with ten, but selecting the best output remained challenging for practitioners [72].The passage connects low accuracy with false negatives and difficulty judging candidate sequences.Source: §1 · p. 2
  • Benchmark duplication and overlapping test cases skew evaluations, while generated code sometimes exactly copies inputs and explanations show poor robustness to minor input changes.The study also reports that models can recognize code grammar and structure.Source: §1 · p. 3
  • The study evaluates eight pre-trained models across four program-generation tasks and five benchmark datasets to investigate reliability and explainability.Tasks include code repair, code review, code translation, and text-to-code generation.Source: §1 · pp. 2–3
  • The authors publish the studied dataset and a replication package on GitHub to support open science.Source: §1 · p. 3

2 BACKGROUND

Automated program generation maps a source sequence into an output sequence through an encoder and decoder. The paper frames explanations around token relevance and uses SHAP-based analysis rather than treating attention weights as definitive explanations.

2.1 Automated Program Generation

  • An encoder maps source tokens into a hidden state, which a decoder uses to generate output tokens; input and output sequence lengths may differ.The model parameters are updated using training data to maximize conditional probability.Source: §2.1 · p. 4

2.2 Model Explanation

  • For each generated token, feature importance reflects both source-input tokens and preceding tokens in the output sequence.The explanation represents relevance in the context of the input and the sequence generated so far.Source: §2.2 · p. 4
  • The setTitle example highlights source tokens such as setTitle and title, alongside the previously generated void token, as important to that next-token prediction.The example illustrates conditioning on both original input and generated context.Source: §2.2 · p. 4, Figure 1 · p. 5

2.2.1 Attention-based analysis.

  • Because attention heads and layers can focus on different code aspects, the study adopts a model-agnostic interpretation approach rather than relying on averaged attention weights.The passage identifies ambiguity in determining which attention weights indicate prediction saliency.Source: §2.2.1 · p. 5
  • The study applies SHAP’s GradientExplainer, which approximates values by sampling reference baselines and computing expectations of gradients.SHAP treats the model as a black box and provides a feature-importance vector for the source-to-target relationship.Source: §2.2.2 · p. 5

3 EXPERIMENTAL DESIGN

The study evaluates representative pre-trained models across program-generation tasks and datasets, using exact-match accuracy, BLEU-4, dataset-level similarity, and token-importance analysis. Its design frames questions about model performance, evaluation reliability, and explanations of generation behavior.

3 EXPERIMENTAL DESIGN

  • The study is designed to assess language-model program generation reliability and explainability through an empirical evaluation.Source: §3 · p. 5

3.1 Language Models

  • Eight publicly available models are selected, spanning T5, CodeT5, CoTexT, CodeTrans, CodeGPT, CodeBERT, CodeReviewer, and CodeT5+.Encoder-based and decoder-based models are included for completeness alongside encoder-decoder models.Source: §3.1 · pp. 5–6

3.2 Downstream Tasks and Corresponding Datasets

  • The datasets cover code review, repair, translation, and generation, including code-to-code, code+text-to-code, and text-to-code input-output paradigms.The study summarizes twelve subsets across five datasets.Source: §3.2 · pp. 6–8

3.3 Evaluation Metrics

  • Accuracy measures perfect code-snippet matches, BLEU-4 measures similarity and fluency, and modified BLEU-4 quantifies dataset-level similarity.Average feature-importance vectors are compared for explanation results.Source: §3.3 · p. 8

3.4 Implementation

  • For fair comparison, all models use 12-layer base versions, identical dataset splits, and fine-tuning procedures aligned with CodeXGLUE or the original papers.Encoder-based CodeBERT receives a transformer decoder for autoregressive generation; experiments use Captum and Ecco implementations for explainability.Source: §3.4 · p. 8

3.5 Research Questions

  • The research questions examine cross-task model performance, dataset representativeness and duplication as reliability concerns, and token contributions to generated sequences.The explanation question uses a model-agnostic explainable approach to interpret why models succeed or fail at generating sequences.Source: §3.5 · pp. 8–9

4 RESULT

The section introduces the results as answers to three research questions, based on analysis of generated code and explanation results.

4 RESULT

  • Results are organized around the three research questions and analyze generated sequences alongside their explanation results.Source: §4 · p. 9

4.1 (RQ1) How do language models perform on program generation tasks?

Encoder-decoder models such as CodeReviewer and CodeT5+ outperform other models across diverse program-generation datasets, but accuracy varies substantially across tasks and dataset sizes. The results include strong task-specific scores alongside low accuracy on more complex tasks.

4.1 (RQ1) How do language models perform on program generation tasks?

  • 30.43% accuracy on CodeReview (CodeReviewer), 22.65% accuracy on CONCODE (CodeReviewer), and 11.36% accuracy on Android_M (CodeT5+).CodeReviewer leads on the code+comment-to-code CodeReview and text-to-code CONCODE tasks; CodeT5+ performs better on multiple code-to-code scenarios.Source: §4.1 · pp. 9–10
  • T5 achieves 6.42% accuracy on Android_S and 2.75% accuracy on Android_M, while encoder-decoder architectures outperform decoder-only and encoder-only models across multiple datasets.The authors identify domain-specific pre-training as important for programming tasks.Source: §4.1 · p. 9
  • CodeT5+ accuracy ranges from 18.44% on B2F_S to 7.84% on B2F_M, and from 70.6% on C#2Java to 7.27% on Google_M.Accuracy is generally higher on small than medium-sized datasets, and overall levels are relatively low, especially on complex tasks.Source: §4.1 · p. 10
  • CodeReviewer and CodeT5+ show superior performance compared with other models across diverse datasets, but their accuracy varies substantially across tasks.The study reports that this variability raises concerns about reliability and motivates closer examination of model behavior.Source: §4.1 · p. 10

4.2 (RQ2) How reliable are automated program generation approaches?

The reliability analysis tests whether train–test overlap, duplicates within test sets, and outputs that copy inputs distort evaluations of program generation. Across these checks, the authors find substantial duplication and evidence that it can inflate performance estimates or make evaluation less reliable.

4.2.1 Data Duplication between Training and Testing Sets.

  • After removing high-similarity test cases, CodeT5+ accuracy falls from 15.2% to 13.09% on Android_S and from 70.6% to 53.09% on C#2Java.More than 20% of test samples in every Tufano et al. dataset are identical to training samples; CodeReview lacks highly similar instances.Source: §4.2.1 · pp. 11–13, Table 3 · p. 13

4.2.2 Data Duplication across Testing Sets.

  • 10 of 12 datasets contain duplicated test-source sequences even though the instances require different targets, including 11% of Android_S test instances.The authors describe this as an unfair evaluation scenario that can misrepresent model performance.Source: §4.2.2 · pp. 13–14, Figure 3 · p. 14

4.2.3 Output-Input Similarity Analysis.

  • Output-input copying varies by task and dataset: T5 outputs match inputs in 78% of Android_S and 80% of Android_M cases.CodeReview, Java2C#, and C#2Java have rates as low as 1–2%, while CONCODE has zero duplication; the authors identify identical outputs as a potential limitation in code review and repair.Source: Table 4 · p. 15, §4.2.3 · p. 15

4.3 (RQ3) Can we explain why automated program generation approaches can (or fail to) generate code sequences reliably?

The authors use SHAP to examine which code tokens language models prioritize and test how token removal affects generation performance. Models assign greater importance to identifiers and keywords, yet performance drops when input tokens are removed, indicating limited robustness to input changes.

4.3 (RQ3) Can we explain why automated program generation approaches can (or fail to) generate code sequences reliably?

  • The authors apply gradient-based SHAP to CodeT5, CodeReviewer, and CodeT5+, then examine token-type importance and performance after selective token removal.Tokens are ranked by importance, and a specified number of least-important tokens are removed with placeholders inserted to preserve code format.Source: §4.3 · p. 16, §4.3.1 · pp. 16–17

4.3.1 Understanding Token Importance.

  • On Android_S, removing one least-important token lowers CodeT5+ performance from 15.2% to 9.19%; removing five lowers it to 3%, and removing more than ten yields zero.Similar degradation occurs across all three studied models, suggesting that even tokens rated less important can be interdependent with other input features.Source: Figure 6 · p. 17, §4.3.1 · p. 17, §4.3.2 · p. 18
  • Identifiers and keywords receive higher importance scores than operators and separators, consistent with models recognizing code’s syntactic and semantic structure.CodeReview and CONCODE were excluded from this token-type analysis because comments and natural-language descriptions made token types difficult to distinguish.Source: §4.3.1 · pp. 16–17

4.3.2 Model Performance Under Input Token Reduction.

  • Across datasets, removing least-important tokens generally hurts performance less than random or highest-importance removal, while removing important tokens consistently reduces performance.For CodeT5+ on Android_S, accuracy falls to 3% after removing the least-important tokens and almost zero after random or most-important removal.Source: Figure 7 · p. 18, §4.3.2 · pp. 18–19

5 DISCUSSION

The discussion argues that data duplication and unchanged generated outputs complicate reliability assessments and raise concerns about deployment. It also reports that models prioritize identifiers and keywords but remain sensitive to input-token removal, while noting limits to the study’s validity and generalizability.

5.1 Implications

  • Training–testing and within-testing duplication, along with outputs copied from inputs, make performance evaluations unreliable and raise concerns about real-world deployment.The authors recommend rigorous standardized benchmarks and metrics beyond accuracy.Source: §5.1 · p. 19
  • Models assign greater importance to keywords and identifiers, indicating recognition of code grammar and structure, but performance declines when even a few tokens are removed.The authors describe explainable AI as a promising way to understand model inference behavior and reliability.Source: §5.1 · pp. 19–20

5.2 Threats to Validity

  • Validity is bounded by the chosen architectures and hyperparameters, and results may not generalize beyond the five datasets and four code-generation task types studied.The authors used original-paper model settings rather than tuning for best performance and provide a replication package for studies on more repositories and tasks.Source: §5.2 · p. 20

6 RELATED WORK

The related-work section distinguishes this study from prior work by focusing on reliability, robustness, and explanation for code-to-code program generation. It positions the empirical study as examining dataset and evaluation effects alongside model behavior across program-generation contexts.

6.1 Language Models for Program Generation

  • Earlier program-generation work proposed neural and pretrained approaches across code review and code translation, while this study evaluates their reliability and examines issues obscured by strong accuracy results.The cited examples include RNNs, Transformer and BPE approaches, and pretrained models for code review and refinement.Source: §6.1 · p. 20

6.2 Explainable LM-based Program Generation

  • Prior explainability work addressed code-to-text and text-to-code tasks or captured structural knowledge, whereas this study examines code-to-code generation behavior across experimental scenarios.The related work includes attention-based analysis and other explainable-AI approaches for code models.Source: §6.2 · pp. 20–21

6.3 Robustness of Language Models for Source Code

  • Against prior studies of adversarial attacks and model robustness, this work empirically examines pretrained code models’ robustness and uses explainable AI to investigate why they are not robust enough.The authors characterize this explanatory focus in software engineering as rarely explored.Source: §6.3 · p. 21

6.4 Reliability in Language Models for Source Code

  • Where prior reliability research considered broader experimental biases and dataset problems, this study focuses on program generation and examines five datasets across evaluation approaches and model robustness.The stated goal is to assess how these factors influence reliability and performance across program-generation contexts.Source: §6.4 · pp. 21–22

7 CONCLUSION

The conclusion reports that prior findings can be replicated but that program-generation models still have reliability concerns. It presents explainable AI as useful for understanding model behavior, including the greater importance assigned to identifiers and keywords.

7 CONCLUSION

  • Prior program-generation performance and findings can be replicated, but models remain unreliable and prior evaluations can produce unrealistic performance assessments.The conclusion identifies explainable AI as a promising approach for understanding program-generation models.Source: §7 · p. 22
  • Explainability results show distinct model patterns, with identifiers and keywords consistently receiving higher importance scores.The authors say explainable AI approaches help researchers better understand model inference behavior.Source: §7 · p. 22
Loading 2302.09587v3…