PaperlayerView exact revision on arXiv

Source-linked AI summary

Natural Is The Best: Model-Agnostic Code Simplification for Pre-trained Large Language Models

Yan Wang, Xiaoning Li, Tien Nguyen, Shaohua Wang, Chao Ni, Ling Ding

arXiv:2405.11196v1cs.SE

TL;DR

Code-oriented LLMs are computationally expensive, and attention-based input filtering can depend on model architecture and pre-training data. SlimCode instead simplifies code according to token characteristics, improving results over DietCode while greatly reducing runtime and GPT-4 costs. The study also finds task-specific but model-agnostic token effects across evaluated models and tasks.

  • Problem

    Attention-based code simplification depends on model architecture and pre-training data, creating a need for simplification based on the nature of the input code.

    Source: §1 · p. 2

  • Method

    SlimCode is a model-agnostic code simplification method that ranks and removes tokens according to their code-based categories and importance.

    Source: §1 · p. 4, §5.1 · p. 12

  • Results

    SlimCode improves DietCode by 9.46% in MRR for code search and 5.15% in BLEU for summarization, runs 133 times faster, and cuts GPT-4 invocation cost by up to 24% per query.

    Source: §1 · p. 4, §9 · p. 20

  • Takeaways & Limitations

    Token-category effects are task-specific but model-agnostic, supporting code-nature-based simplification across CodeBERT, CodeT5, and GPT-4 within the evaluated settings.

    Source: §1 · pp. 3–4, §7.2 · p. 19

    The reported generalizability requires more experiments across additional models and tasks.

    Source: §7.2 · p. 19

Abstract

from arXiv · show

Pre-trained Large Language Models (LLM) have achieved remarkable successes in several domains. However, code-oriented LLMs are heavy in computational complexity, and quadratically with the length of the input. Toward simplifying the input program of an LLM, the state-of-the-art approach has the strategies to filter the input code tokens based on the attention scores given by the LLM. The decision to simplify the input should not rely on the attention patterns of an LLM, as these patterns are influenced by both the model architecture and the pre-training dataset. Since the model and dataset are part of the solution domain, not the problem domain where the input belongs, the outcome may differ when the model is pre-trained on a different dataset. We propose SlimCode, a model-agnostic code simplification solution for LLMs that depends on the nature of input code tokens. As an empirical study on the LLMs including CodeBERT, CodeT5, and GPT-4 for two main tasks: code search and summarization, we reported that 1) the removal ratio of code has a linear-like relation with the saving ratio on training time, 2) the impact of categorized tokens on code simplification can vary significantly, 3) the impact of categorized tokens on code simplification is task-specific but model-agnostic, and 4) the above findings hold for the paradigm-prompt engineering and interactive in-context learning. The empirical results showed that SlimCode can improve the state-of-the-art technique by 9.46% and 5.15% in terms of MRR and BLEU score on code search and summarization. Moreover, SlimCode is 133 times faster than the state-of-the-art approach. Additionally, SlimCode can reduce the cost of invoking GPT-4 by up to 24% per API query, while still producing comparable results to those with the original code.

1 INTRODUCTION

The paper argues that code simplification should depend on source-code characteristics rather than model attention patterns, and proposes SlimCode to simplify inputs for downstream LLM tasks. Empirical findings show efficiency gains from removal and task-specific, model-agnostic effects across token categories, with SlimCode outperforming DietCode while running substantially faster.

  • The paper rejects attention-based simplification because attention patterns depend on model architecture and pre-training data rather than solely on the input program.This dependence may make simplification outcomes differ across datasets.Source: §1 · p. 2
  • The study evaluates random token removal, token-category removal, and structural removal for code search and code summarization.The empirical design uses multiple removal ratios, training-time measurements, and performance metrics with CodeBERT and CodeT5.Source: §1 · p. 3
  • The study finds that token-category effects are task-specific but model-agnostic across CodeBERT and CodeT5.Method signatures and variables have the greatest impact, whereas symbol-token removal has the least impact in code search and summarization.Source: §1 · p. 3
  • Removing 50% of code reduces CodeBERT training time by 51.5% for code search and 31.76% for summarization.The removal ratio has a linear-like relationship with training-time savings.Source: §1 · p. 3
  • SlimCode improves DietCode by 9.46% in MRR for code search and 5.15% in BLEU for code summarization, while running 133 times faster.Source: §1 · p. 4

2 TOKEN CATEGORY AND REMOVAL

The paper categorizes code tokens by lexical, syntactic, and semantic properties to identify which can be removed with limited impact on downstream tasks. These categories include symbols and identifiers, AST-selected structural tokens, and tokens selected using program dependencies.

  • The study divides code tokens into six categories: symbols, identifiers, control structures, method invocations, method signatures, and PDG tokens.Source: §2 · p. 4
  • Lexical categories use token features, syntactic categories require grammar through ASTs, and semantic categories require program semantics.Source: §2 · p. 4
  • Symbol tokens include brackets, separators, and operators that form program building blocks but generally carry less high-level task-related information.Source: §2.1.1 · p. 4
  • Identifiers convey developer intent through meaningful names, making them potential downstream-task hints through improved readability.Source: §2.1.2 · p. 4
  • AST-based syntactic removal targets control structures, method invocations, and method signatures because these tokens reflect code content and business logic.Method signatures describe inputs and outputs, while invocations can indicate a method’s purpose and functionality.Source: §2.2 · pp. 4–5, §2.2.1 · p. 5, §2.2.2 · p. 5
  • PDG-based selection uses data and control dependencies to identify tokens according to program semantics.Control dependencies reflect execution order, while data dependencies reflect how variables are modified and used.Source: §2.3 · p. 5

3 EXPERIMENTAL METHODOLOGY

The study evaluates code simplification across code search and summarization using fine-tuned CodeBERT and CodeT5, plus prompted GPT-4, with datasets, fixed input lengths, efficiency measures, and task-specific metrics.

Downstream Tasks

  • The experiments cover code search and code summarization, measuring how simplification affects pretrained models’ efficiency and effectiveness.Code search retrieves relevant snippets, while summarization generates natural-language descriptions of code.Source: §3.1 · p. 5

Models and Paradigms

  • The study evaluates CodeBERT, CodeT5, and GPT-4 under fine-tuning and prompt-based paradigms.CodeBERT and CodeT5 represent pre-train-and-fine-tune models, whereas GPT-4 is evaluated through prompts.Source: §3.2 · pp. 5–6

Datasets

  • The datasets are drawn from CodeBERT’s code search and summarization benchmarks, with Java-language experiments and separate training, validation, and testing portions.The study also samples 400 examples for GPT-4 evaluation because inference is expensive.Source: §3.3.1 · pp. 6–7, §3.3.2 · p. 7

Simplification and Efficiency

  • Simplification ratio measures the percentage of input tokens removed, while efficiency is measured by fine-tuning time or prediction cost.Original inputs are fixed at 200 tokens for search and 256 for summarization, with simplified lengths determined by the removal ratio.Source: §3.4 · p. 7

Evaluation Metrics

  • Code search is evaluated with MRR, summarization with BLEU-4, and prompted code search with Precision because MRR is computationally expensive there.Precision is defined as the proportion of correctly detected examples.Source: §3.4 · pp. 7–8

4 EXPERIMENTAL RESULTS

Random token removal produces near-linear training-time savings, while token-category effects differ by task and model; symbols are comparatively safe to remove, whereas identifiers matter more.

RQ1: Random Token Removal

  • 51.50% less CodeBERT code-search training time accompanies 50% code removal, while CodeT5 shows 31.55% less summarization training time at the same ratio.Training-time savings increase in a linear-like pattern as more tokens are removed.Source: §4.1.2 · p. 8, §4.1.3 · p. 8, §4.1.4 · p. 9
  • For CodeBERT code search, removing 10%-50% of tokens saves 10.39%-51.5% of training time while reducing MRR by 4.98%-17.77%.The percentage of time saved is consistently greater than the performance sacrifice in this setting.Source: §4.1.3 · p. 9
  • The time-saving-to-removal relationship is approximately 1:1 for code search and 0.6:1 for summarization, with CodeT5 outperforming CodeBERT under the same reductions.The summarization gap reflects additional decoder training time in CodeBERT’s generation setup.Source: §4.1.4 · p. 9

RQ2: Token Categories

  • Removing identifiers saves about 15% and 9% of training time on code search and summarization, but decreases CodeBERT MRR by 12.52% and 3.83%, respectively.CodeT5’s corresponding MRR and summarization-performance decreases are 8.81% and 4.88%.Source: §4.2.3 · pp. 9–10, §4.2.2 · p. 9
  • Removing symbols saves about 51% and 31% of training time while reducing CodeBERT performance by only 2.83% on search and 0.59% on summarization.The corresponding CodeT5 decreases are 2.67% and 0.73%, making symbols more removable than identifiers in these tasks.Source: §4.2.2 · p. 9, §4.2.4 · p. 10, §4.2.5 · p. 10

4.3 Impact of Removing Syntactic Tokens on the Performance of PLLMs (RQ3)

Syntactic token categories differ substantially in their effects on code search and summarization, with method signatures exerting the strongest influence, especially for generation.

4.3 Impact of Removing Syntactic Tokens on the Performance of PLLMs (RQ3)

  • Method-signature tokens have the greatest impact, reducing BLEU by 14.64% for CodeBERT and 18.94% for CodeT5 when removed.They account for 11.36% of tokens in the code-summarization dataset.Source: §4.3.3 · p. 11, §4.3.5 · p. 11
  • Method signatures affect code summarization more strongly than other syntactic categories, while their code-search impact resembles that of identifiers.The result holds across both CodeBERT and CodeT5.Source: §4.3.3 · p. 11, §4.3.5 · p. 11
  • Removing method-invocation and control-structure tokens reduces CodeBERT code-search MRR by 8.21% and 3.77%, respectively.These categories have similar effects overall, with control structures slightly more influential in code search.Source: §4.3.4 · p. 11, §4.3.2 · p. 11
  • The syntactic-token findings are task-specific: method invocations and control structures matter more for code search than for code summarization.Their impact still exceeds that of symbol tokens and identifiers in the reported comparisons.Source: §4.3.5 · p. 11, §4.3.4 · p. 11
  • Tokens outside program-dependence graphs have little downstream impact because they contain little semantic information.Removing them decreases CodeBERT code-search MRR by 4.04% and CodeT5 summarization BLEU-4 by 0.49%.Source: §4.4.2 · p. 12

5 SLIMCODE: MODEL-AGNOSTIC CODE SIMPLIFICATION FOR PLLMS

SlimCode simplifies code using token-category importance rather than model attention, greedily removing lower-importance tokens to meet a target ratio. It outperforms DietCode while requiring substantially less removal time.

5 SLIMCODE: MODEL-AGNOSTIC CODE SIMPLIFICATION FOR PLLMS

  • The observed importance order is method signatures > identifiers > control structures ≈ method invocations > symbol tokens, and is model-agnostic and task-independent.Tokens outside PDGs are excluded because constructing PDGs is time-consuming and may be impossible for incomplete code.Source: §5.1 · p. 12
  • SlimCode ranks token categories by importance and removes lower-importance tokens before higher-importance ones.Tokens belonging to multiple categories can receive greater importance than tokens in only one category.Source: §5.1 · pp. 12–13
  • SlimCode removes mostly symbol and other tokens, which together can exceed 95% of removed tokens, while removing fewer method-signature and identifier tokens.This distribution reflects its category-based ranking strategy.Source: §5.1 · p. 12, §5.4.2 · p. 15
  • SlimCode is up to 133 times faster than DietCode because it avoids model-based token-weight retrieval and associated pruning computations.DietCode requires 1 to 10 hours for token removal in the reported settings.Source: §5.4.3 · pp. 15–16
  • Token-weight rankings differ across CodeBERT and CodeT5, with a top-1,000-token Spearman correlation of 0.43.The difference is attributed to varying attention patterns, training data, architectures, loss functions, or tokenizers.Source: §5.4.3 · p. 16
  • SlimCode outperforms DietCode by up to 9.46% in MRR and 5.15% in BLEU score across code search and summarization.The comparison covers CodeBERT and CodeT5 experiments.Source: §5.4.1 · p. 14, Table 8 · p. 14, Table 9 · p. 14

6 USAGE OF SLIMCODE FOR GPT-4 WITH ZERO-SHOT-LEARNING

GPT-4 experiments show that code simplification reduces token usage while preserving or improving task performance, although token-category effects differ from CodeBERT and CodeT5. SlimCode can therefore reduce GPT-4 costs in the reported setting.

6 USAGE OF SLIMCODE FOR GPT-4 WITH ZERO-SHOT-LEARNING

  • As simplification increases, GPT-4 total token usage and task metrics decrease for both code search and summarization.The study uses total input-plus-output tokens as the resource measure because GPT-4 pricing depends on total tokens.Source: §6.2 · p. 16, §6.2.1 · p. 17, Table 10 · p. 17
  • GPT-4’s token-category sensitivity differs by task: method structures and invocations matter more in code search, while summarization categories appear more similar.In code search, these categories also have greater significance than method signatures.Source: §6.2.2 · pp. 17–18
  • SlimCode improves GPT-4 precision by 26% for code search and BLEU-4 by 10% for code summarization on average versus DietCode.The simplified code can even outperform the original code in the reported experiments.Source: §6.2.2 · p. 18, §6.2.3 · p. 19
  • A 50% SlimCode simplification ratio outperforms the original code on both GPT-4 tasks.Removing symbols and some control-structure and invocation tokens shifts GPT-4 toward method signatures, identifiers, and pre-trained knowledge.Source: §6.2.3 · p. 19
  • Removing 50% of tokens improves GPT-4 code-search precision by about 2.35% while reducing total tokens by 27.22%.Removing symbol tokens saves 24% of GPT-4 invocation cost in the reported 400-sample setting.Source: §6.2.3 · p. 19

7 DISCUSSIONS AND POTENTIAL USES OF SLIMCODE

SlimCode addresses code-oriented LLM complexity by simplifying inputs before processing, with code reduction tracking training-time savings. Its categorized-token effects are model-agnostic but task-specific, while broader validation remains necessary.

7 DISCUSSIONS AND POTENTIAL USES OF SLIMCODE

  • Code reduction has a linear-like relationship with training-time savings, making input simplification relevant to LLM efficiency.The discussion frames SlimCode as a way to reduce the computational burden of longer code sequences.Source: §7.1 · p. 19
  • SlimCode’s categorized-token effects are model-agnostic but task-specific across CodeBERT, CodeT5, and GPT-4.This supports applying the approach across architectures while preserving task-dependent token priorities.Source: §7.2 · p. 19, §7.3 · p. 19
  • The generalizability claim requires more experiments across other LLMs and tasks.The paper explicitly identifies broader validation as necessary before extending the reported benefits.Source: §7.2 · p. 19
  • SlimCode can reduce the cost of invoking LLMs through API queries, with practical implications for organizations and developers using code-related tasks.The discussion links lower token usage to resource management for API-based LLM use.Source: §7.4 · p. 20

Pre-trained Models in Software Engineering and its Understanding.

Pre-trained models have advanced code intelligence across software-engineering tasks, including code repair, generation, defect detection, summarization, and search.

  • Pre-trained models associated with programming languages have advanced code intelligence across multiple software-engineering tasks.The cited examples include code repair, code generation, defect detection, code summarization, and code search.Source: Pre-trained Models in Software… · p. 20

Program Simplification.

Program simplification methods reduce code into fragments that an auxiliary neural model processes for downstream testing tasks.

  • Delta-debugging-based approaches split code snippets into fragments and pass each fragment to an auxiliary neural network.Example tasks include method-name prediction and misused-variable detection.Source: Program Simplification · p. 20

9 CONCLUSION

The paper concludes that SlimCode is a model-agnostic code-simplification solution aimed at improving computational efficiency, generalizability, cost savings, and applicability across software engineering. It reports better benchmark results and substantially faster processing than DietCode, alongside lower GPT-4 invocation costs with comparable outputs to original code.

  • SlimCode improves over DietCode by 9.46% in MRR for code search and 5.15% in BLEU for summarization.Source: §9 · p. 20
  • SlimCode is 133 times faster than DietCode.Source: §9 · p. 20
  • SlimCode reduces GPT-4 invocation cost by up to 24% per API query while producing comparable results to original code.Source: §9 · p. 20
  • SlimCode is presented as a model-agnostic code-simplification solution with implications for computational efficiency, generalizability, cost savings, and broader software-engineering applicability.Source: §9 · p. 20
Loading 2405.11196v1…