PaperlayerView exact revision on arXiv

Source-linked AI summary

Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code

Ziyin Zhang, Chaoyu Chen, Bingchang Liu, Cong Liao, Zi Gong, Hang Yu, Jianguo Li, Rui Wang

arXiv:2311.07989v7cs.CLcs.AIcs.SE

TL;DR

Software-engineering language-model research spans applications and evaluation, but earlier reviews tended to cover only the NLP or SE perspective, and non-coding tasks remain understudied. This survey synthesizes models, tasks, datasets, and related work across the software lifecycle. It connects code-model development to NLP’s trajectory and discusses code-specific information and feedback mechanisms.

  • Problem

    Earlier reviews focused on either NLP or SE, and software-engineering tasks beyond coding have remained understudied.

    Source: §2 · p. 3

  • Method

    The paper systematically surveys language models across software engineering, including code models, lifecycle tasks, code-specific features, and techniques adapted from NLP.

    Source: §1 · p. 2, §8 · p. 29

  • Results

    The survey traces code modeling from SMT and NMT through pretrained Transformers to few-shot LLM use and autonomous agents, noting that code enables auxiliary views and interpreter or unit-test feedback.

    Source: §8 · p. 29

  • Takeaways & Limitations

    The survey aims to provide a global view of language-model applications in software engineering and connect research from NLP and SE.

    Source: §8 · p. 31

    Standalone-function benchmarks built from well-structured docstrings do not reflect real-world requirements or repository dependencies, motivating benchmarks with cross-file context.

    Source: §8 · p. 30

Abstract

from arXiv · show

In this work we systematically review the recent advancements in software engineering with language models, covering 70+ models, 40+ evaluation tasks, 180+ datasets, and 900 related works. Unlike previous works, we integrate software engineering (SE) with natural language processing (NLP) by discussing the perspectives of both sides: SE applies language models for development automation, while NLP adopts SE tasks for language model evaluation. We break down code processing models into general language models represented by the GPT family and specialized models that are specifically pretrained on code, often with tailored objectives. We discuss the relations and differences between these models, and highlight the historical transition of code modeling from statistical models and RNNs to pretrained Transformers and LLMs, which is exactly the same course that had been taken by NLP. We also go beyond programming and review LLMs' application in other software engineering activities including requirement engineering, testing, deployment, and operations in an endeavor to provide a global view of NLP in SE, and identify key challenges and potential future directions in this domain. We keep the survey open and updated on GitHub at https://github.com/codefuse-ai/Awesome-Code-LLM.

1 Introduction

The survey addresses the gap between NLP- and SE-focused reviews with a broad account of language models across software engineering. It connects the communities’ complementary uses of language models and reviews code-model development alongside applications beyond programming.

1 Introduction

  • The survey covers 70+ models, 40+ downstream tasks, 180+ datasets, and 900 related works to bridge NLP and SE perspectives.It analyzes code language models and their differences from general language models, including code-specific features and techniques adapted from NLP.Source: §1 · p. 2
  • Earlier surveys tended to focus on either NLP or SE, leaving models, tasks, and challenges from the other community less covered.For example, NLP-focused work centers on text-to-code generation, while some SE-focused reviews draw little from deep-learning and NLP venues.Source: §1 · p. 2
  • SE applies language models to automate tasks, while NLP adopts SE tasks to evaluate language models.The survey also describes how programming requirements provide real-world testbeds for language-model technologies and support their development toward production and deployment.Source: §1 · p. 2
  • The survey reviews language-model applications across software development, including requirement engineering, deployment, and operations beyond programming-related tasks.Its organization follows a taxonomy of pretrained language models for code and covers code-capable LLMs, specialized models, code features, and software-development integration.Source: §1 · p. 2

2 Downstream Tasks in Software Engineering

The survey organizes software-engineering language-model tasks across the software lifecycle and notes that programming-related tasks receive the most attention. It outlines applications from requirements and code search through code generation, while observing that non-coding tasks remain understudied.

2 Downstream Tasks in Software Engineering

  • Other software-engineering tasks, especially those not directly related to coding, have remained understudied despite CodeXGLUE’s broad task benchmark and HumanEval’s prominence.HumanEval brought NL-to-code synthesis into the NLP community and became a standard LLM evaluation task.Source: §2 · p. 3

2.1 Downstream Tasks in SE

  • The survey classifies tasks by software-development stage rather than input/output modality, spanning requirements, development, testing, deployment and operations, and newer LLM evaluation tasks.Each category may include both language-understanding and language-generation tasks.Source: §2.1 · p. 4

2.1.1 Requirement Engineering

  • Requirement engineering covers specification, analysis, and validation of requirements, software modeling, and UI/UX design.Requirement analysis ranges from earlier POS-tagging and dependency-parsing methods to LLM-based classification, summarization, and clause extraction; the survey focuses on NLP intersections such as generating markup for UI design.Source: §2.1.1 · p. 4

2.1.2 Development

  • Development contains the most programming activity and the most abundant LLM applications, grouped into code search, generation, editing, suggestion, and explanation.The survey points to Figures 4 and 5 for related works in these activities.Source: §2.1.2 · p. 5

Code Search

  • Code search includes retrieving code from natural-language queries, finding cross-language code clones, and identifying similar APIs or common code idioms.NL-to-code search commonly compares query and candidate-code embeddings; API mining may use similarity metrics or generate target APIs directly.Source: Code Search · p. 5

Code Generation

  • Program synthesis generates code, usually a function or method, from a natural-language description, while code completion predicts continuations from a code prefix.Transformer language models enabled autoregressive code generation without task-specific fine-tuning; grammar-aided statistical models performed better than early neural models, which became dominant after 2018.Source: Code Generation · pp. 5–7
  • Code infilling generalizes completion by using both left and right context, and Text-to-SQL generates SQL commands from natural-language queries.The survey notes a hypothesis that SQL’s declarative nature and restricted grammar make it easier to optimize than imperative languages such as C and Python.Source: Code Generation · p. 7

Code Editing

The survey describes development tasks including code editing, suggestion, and explanation, then notes a limited treatment of code-related document translation. It also places program testing and correctness analysis among software-engineering tasks.

Code Editing

  • Code translation converts code between programming languages and can support migration of projects written in obsolete languages, but such applications have not appeared at scale in the LLM era.The survey identifies limited context windows as a constraint when handling such projects; program repair, also called bug fixing, is another traditional sequence-to-sequence task.Source: Code Editing · p. 7

Code Suggestion

  • Type prediction estimates variable types, while identifier prediction predicts names that carry semantic information and can support summarization and model pretraining.Cloze tests for code understanding often select programming-language keywords such as min and max.Source: Code Suggestion · p. 7

Code Explanation

  • Code summarization and comment generation explain code for developers, whereas code documentation is more structured and targets a broader audience.The survey does not discuss document translation in detail because NLP already has abundant models, datasets, and prompting strategies for machine translation.Source: Code Explanation · pp. 7–9

2.1.3 Testing and Analysis

  • Testing and analysis address program correctness as well as other properties, including security.Source: §2.1.3 · p. 9

Testing

The survey maps language-model applications across software testing, analysis, deployment, and operations, describing tasks that range from generating tests to transforming and monitoring software.

Testing

  • Unit-test generation produces tests for a program; related tasks generate assertions, create mutants for mutation testing, or produce inputs that exercise fuzzing corner cases.The passage notes that nearly all unit-test-generation work predated code LLMs and that current tests may be insufficient for evaluating program synthesis.Source: Testing · p. 9

Analysis

  • Analysis tasks include predicting defects, malware, or code clones, while code classification and author identification have seen almost no pretrained-language-model application.Malware is distinguished from other vulnerable code by its intentional injection; semantic clones can be syntactically different while sharing functionality.Source: Analysis · p. 9

2.1.4 Deployment and Operations

  • After testing, deployed software is maintained or updated; related operations include analyzing logs, such as parsing them into templates or detecting anomalies.The cited surveys and comparisons cover log parsing and anomaly detection alongside traditional methods.Source: §2.1.4 · p. 10, Deployment · pp. 11–12

Deployment

  • Deployment tasks include optimizing compiled code, translating assembly or binaries back into source, and recovering meaningful identifiers from obfuscated programs.Compiler optimization has shifted from machine-learning approaches such as scheduling and pass ordering toward sequence-to-sequence generation; deobfuscation also serves as a code-model pretraining objective.Source: Deployment · p. 11

Operations

The survey covers operational uses of language models and newer programming-related evaluation tasks, then relates software-engineering tasks to NLP task forms and highlights gaps in language and domain coverage.

Operations

  • Operations tasks include automating peer code review, generating commit messages from pre- and post-change code, and automating Git workflows.Commit-message generation can be viewed as the dual task of program repair; workflow automation drew attention only after powerful LLMs emerged.Source: Operations · p. 12

2.1.5 LLM Evaluation

  • Programming-related LLM evaluation has gained momentum in both NLP and SE, including code reasoning, coding for reasoning, and analysis of AI-generated code.Coding for reasoning uses generated code executed by external interpreters; the model may also simulate that execution to support reasoning.Source: §2.1.5 · pp. 12–13

2.1.6 NLP Point-of-View

  • NLP framing distinguishes sequence-to-sequence generation, classification and token tagging, and retrieval or search, with code tasks mapped to these forms.Code completion and infilling target continuations or fills within the input, unlike sequence-to-sequence tasks that relate source and target sequences.Source: §2.1.6 · p. 13

2.1.7 Code LLMs for Low-Resource, Low-Level, and Domain-Specific Languages

  • Code modeling largely targets popular languages such as C, Java, and Python, while domain-specific languages can have active user communities and declarative or functional languages remain less studied.The survey notes abundant Verilog research, few studies beyond SQL and a recent Haskell work, and scarce transfer research for low-resource programming languages.Source: §2.1.7 · pp. 13–14

2.2 Evaluation Metrics

The section reviews how evaluation metrics for code generation developed toward functional correctness, traces major shifts in program synthesis, and describes emerging repository-level coding work.

2.2 Evaluation Metrics

  • Understanding tasks use measures such as accuracy, F1, and MRR, while code-to-text uses BLEU; code-generation evaluation has shifted from syntax and reference matching toward functional correctness.Content-overlap metrics can fail when functionally equivalent code has different lexical forms; pass@k estimates the chance that any of k samples passes all unit tests.Source: §2.2 · p. 14

2.3 Program Synthesis

  • Program synthesis has shifted from example-based programs in domain-specific languages to natural-language descriptions paired with general-purpose code, and from syntax-enhanced decoders toward execution-based feedback.HumanEval helped establish NL-to-code as a standard task; its language translations and domain-focused benchmarks broadened the evaluation set.Source: §2.3 · pp. 14–15

2.4 Repository-Level Coding

  • Repository-level coding extends beyond single-file or single-function tasks, using longer context for code completion and generation and introducing benchmarks for repository-level work.The survey also identifies repository-level API migration, temporal editing, and SWE-bench.Source: §2.4 · p. 15

3 Language Modeling Preliminaries

The section introduces language-model training objectives and Transformer implementation choices relevant to code modeling, from causal and masked prediction to denoising and efficiency-oriented design.

3.1 Causal Language Modeling

  • Causal language modeling factors a sequence into token probabilities conditioned on preceding tokens, enabling autoregressive generation with Transformer decoders.A causal attention mask restricts each token to attending to earlier tokens; training computes cross-entropy across input tokens in parallel.Source: §3.1 · pp. 15–16

3.2 Masked Language Modeling

  • Masked language modeling trains bidirectional representations by recovering randomly masked tokens, while ELECTRA instead predicts whether each input token was replaced.Because only masked positions contribute to BERT-style MLM training, the passage notes that this objective usually trains on about 15% of tokens; ELECTRA computes loss on all tokens.Source: §3.2 · p. 16

3.3 Denoising Objectives

  • Span corruption combines bidirectional encoding of corrupted input with autoregressive generation of masked spans; prefix language modeling can adapt such models for autoregressive modeling.UL2 combines span-corruption objectives with different corruption rates and span lengths; under the same computation budget, its encoder-decoder models perform better than decoder-only models.Source: §3.3 · pp. 16–17

3.4 Auxiliary Objectives

  • Auxiliary objectives target document structure beyond token-level modeling, including predicting whether segments are neighbors or whether adjacent sentences have been swapped.T5 also mixes supervised downstream samples into pretraining and uses conditional generation as the objective for both self-supervised and supervised tasks.Source: §3.4 · p. 17

3.5 Implementation Design

  • Transformer implementations change normalization placement and sublayer execution: pre-norm became standard in decoders, while GPT-J computes attention and FFN sublayers in parallel.Chowdhery et al. report limited performance degradation when applying the parallel design to larger models.Source: §3.5 · p. 18
  • Implementation choices also address precision, long-context positions, and attention cost, trading memory or speed against performance in some designs.BF16 avoids the numerical overflow and underflow issues described for FP16; RoPE enables position interpolation, MQA improves inference speed at small performance costs, and FlashAttention improves memory efficiency through tiling.Source: §3.5 · p. 18

4 General Language Models for Code

General language models can exhibit coding ability without code-specific design, while additional code pretraining and later multi-task finetuning further specialize models for code. Across off-the-shelf and specialized models, the section traces both benchmark results and architectural developments.

4 General Language Models for Code

  • General language models can demonstrate coding ability without code-specific training, and continual code pretraining has been reported to improve code performance.The section distinguishes this self-supervised process from supervised in-task or instruction finetuning.Source: §4 · p. 18

4.1 Off-the-Shelf Language Models

  • LLaMA and LLaMA 2, pretrained on datasets containing code, score 23.7 and 29.9 pass@1 on HumanEval, respectively.The cited datasets include 328GB of GitHub code for LLaMA; The Pile and ROOTS also contain substantial code collections used to support models with coding ability.Source: §4.1 · p. 19
  • GPT-4 scores 67.0 pass@1 on HumanEval, while an early version is reported at 82, exceeding specialized code models until recently.LaMDA and PaLM score 14.0 and 26.2 pass@1, respectively.Source: §4.1 · p. 19
  • Phi-1.5, a 1.3B model trained on textbook and filtered web data, attains 41.4 pass@1 on HumanEval.The section also notes a trend toward smaller models trained on larger datasets, following the revised scaling law.Source: §4.1 · p. 19
  • Mixtral 8x7B activates 13B parameters per token and scores 40.2 on HumanEval, surpassing much larger dense models such as LLaMA 2.DeepSeekMoE activates 2.8B parameters in a 16B model and scores 26.8 on HumanEval.Source: §4.1 · p. 19

4.2 Language Models with Additional Pretraining on Code

  • 74.4 pass@1 on HumanEval lets CodeFuse-CodeLLaMA surpass the GPT-4 performance published in OpenAI (2023), after multi-task finetuning of Code LLaMA.Earlier code-specialized models also advanced: PaLM-Coder set new state-of-the-art results on HumanEval and MBPP before PaLM 2-S*; Code LLaMA surpassed prior LMs except GPT-4.Source: §4.2 · p. 20
  • General and code-specialized models share Transformer-decoder and architectural advances, including RoPE, while BLOOM and 13B Baichuan 2 use ALiBi and Code LLaMA uses GQA.The section also identifies pre-norm across the models and traces parallel attention from GPT-J to PaLM, GPT-NeoX, and Phi-1.5.Source: §4.2 · p. 20

5 Specialized Language Models for Code

Specialized code models adapt language-model architectures, datasets, and training objectives to code, drawing on NLP while adding code-specific methods. The survey covers encoder and decoder families, newer objectives, and code-model training and deployment techniques.

5.1 Training Dataset for Code

  • Code pretraining draws on GitHub corpora, where code may include documentation and commit histories that support bimodal training and instruction data.CodeSearchNet contains 2.3M functions paired with natural-language documentation; CommitPackFT uses commit histories as instruction data.Source: §5.1 · pp. 20–21
  • A 1.3B model pretrained on 1B synthetic Python textbook and exercise tokens achieved HumanEval results comparable to much larger models trained on larger datasets.The synthetic data was generated by a powerful model.Source: §5.1 · p. 21

5.2 Encoders

  • Code encoders use code-specific objectives such as graph prediction, type tagging, and contrastive learning, while CuBERT and CodeBERT show gains over non-pretrained approaches.CodeBERT combines masked language modeling and replaced-token detection and yields a moderate gain when initializing a sequence-to-sequence model.Source: §5.2 · p. 23

5.3 Encoder-Decoders

  • Encoder-decoder code models adapt T5 and BART, add identifier-focused objectives, and use denoising and back translation to address limited parallel data for code translation.Transcoder uses an XLM-pretrained encoder before further pretraining with denoising auto-encoding and back translation.Source: §5.3 · pp. 23–24

5.4 Decoders

  • Fill-in-the-middle objectives enable code completion within an input and are reported to outperform CLM-only models on downstream tasks, though their exact benefits remain controversial.The comparison is with CLM-only models such as CodeGen; Table 4 reports pass@1 performance.Source: Table 4 · p. 24, §5.4 · p. 25

5.5 UniLMs

  • Code UniLM models combine objectives through alternating or varied attention masks, but the reviewed models are relatively small and their suitability at scale remains unexplored.Source: §5.5 · p. 25

5.6 Diffusion Models

  • A 75M-parameter diffusion model, CodeFusion, is reported to outperform StarCoder, CodeT5+, and GPT-3 on three code-synthesis datasets.The passage does not provide the individual dataset scores.Source: §5.6 · p. 25

5.7 Instruction Finetuning and Reinforcement Learning for Code

  • Code instruction tuning uses generated instructions, commit histories, and multitask data, while reinforcement-learning approaches use compiler and test feedback as rewards.CodeRL defines rewards for compilation errors, runtime errors, unit-test failures, and passing programs; RLTF also uses compiler error information and test-pass ratios.Source: §5.7 · pp. 25–26

6 Code Features for Language Models

Code models can use compiler-derived structure and semantics, such as syntax trees, intermediate representations, data flow, and types, as additional training signals. The section reviews how these features are encoded and where their use is constrained.

6 Code Features for Language Models

  • Compiler toolkits make it practical to extract code structure and semantic features, enabling training objectives beyond lexical manipulations such as masked or causal language modeling.Available features include ASTs, language-independent intermediate representations, types, and control- and data-flow graphs.Source: §6 · p. 26

6.1 Abstract Syntax Tree and Intermediate Representation

  • TreeBERT encodes AST paths alongside code and trains with masked-node and node-order objectives, while later approaches often flatten ASTs into input sequences.TreeBERT's method is described as complicated and not scaling well; SynCoBERT and SPT-Code use sequentialized AST representations.Source: §6.1 · pp. 26–27
  • Language-independent intermediate representations can serve as translation pivots; extending Transcoder with IR-based objectives produced promising results.The explored objectives include IR generation, decompilation, and generating code in one language from another language's IR.Source: §6.1 · p. 27

6.2 Control Flow and Data Flow

  • GraphCodeBERT incorporates data-flow edges into model attention and trains with edge prediction and node alignment, while dynamic execution semantics remain less studied than static features.AST and IR may fail to capture properties revealed only at runtime.Source: §6.2 · p. 27

6.3 Type

  • Type information can guide token prediction directly, while identifier tagging in CodeT5 and SynCoBERT serves as a coarse-grained type-prediction objective.Source: §6.3 · p. 27

6.4 Program Transformation

  • Function-preserving transformations support pretraining to recover original code and finetuning for robustness to transformed test samples.Examples include identifier renaming, loop or condition transformations, dead-code insertion, and statement swapping.Source: §6.4 · p. 27

7 LLMs in Software Development

The survey describes LLMs entering software development through coding tools and integration into development workflows, alongside research assessing generated code. These applications include interpreter-supported reasoning, interactive refinement, and code assistants with expanding features.

7.1 LLMs Extended with Coding Tools

  • Interpreters and tests can support numerical reasoning and interactive code refinement, while some systems use LLMs to simulate execution or provide feedback.Other approaches use planning or multiple LLM agents in roles such as programmer, reviewer, and manager.Source: §7.1 · p. 28

7.2 LLMs Integrated into Software Development

  • Code assistants have expanded beyond completion to generation, translation, commenting, and testing, while larger models raise client-side deployment and real-time performance challenges.Source: §7.2 · p. 29

7.3 Analysis of LLM-Generated Code

  • Studies assess AI-generated code for correctness, bugs, vulnerabilities, robustness, efficiency, and hallucinations, but report mixed security and code-complexity findings.The survey characterizes assistants as nascent and says their implications for software development remain to be investigated systematically.Source: §7.3 · p. 29

8 Conclusion and Challenges

Code modeling has followed NLP’s progression from statistical and neural translation models to pretrained Transformers, LLMs, and agents, while code also offers auxiliary views and test-based feedback. The survey identifies benchmark coverage, broader SE applications, data, code features, programming paradigms, architectures, and safety as challenges or future directions.

Benchmarking

  • HumanEval is small and its scoreboard has been manipulated to near-perfect scores, motivating more challenging benchmarks that reflect real-world requirements and repository context.SWE-bench exposes this gap; the most recent SWE-agent resolved 12.5% of real-world GitHub issues there.Source: §8 · pp. 29–30
  • Most existing benchmarks center on standalone Python functions and docstrings, while repository-level tasks expose limits on real-world applications.The cited SWE-agent resolved 12.5% of real-world GitHub issues in SWE-bench.Source: §8 · pp. 29–30

Beyond code generation

  • LLM evaluation and application remain concentrated on code generation and IDE suggestions, leaving other SE activities largely overlooked.The survey notes that code-trained models may suit programming and math, while general-domain models may be better suited to non-coding SE tasks.Source: §8 · p. 30

Data

  • Selecting high-quality training data and using synthetic data are expected to become increasingly prominent in pretraining and finetuning.Gunasekar et al. (2023) achieved SOTA performance with a 1.3B model trained on textbook data.Source: §8 · p. 30

Code features

  • Control-flow and data-flow graphs remain underused at scale, and attention-mask changes in the few employing data flow limit cross-task generalization and scaling.The survey identifies seamless integration of these features into textual input as a future research direction.Source: §8 · p. 30

Programming paradigms

  • Research focuses mainly on imperative languages such as C and Python, while declarative and functional languages receive less attention except for SQL.The survey argues that declarative and functional paradigms merit more research attention because they are more aligned with natural languages.Source: §8 · p. 30

Architectures and objectives

  • Code models with auxiliary objectives are encoder-only or encoder-decoder models, while decoder-only models have yet to be augmented with alternative objectives.The survey also points to diffusion models as a potential future direction for code modeling.Source: §8 · p. 30

Safety and ethics

  • Code LLM deployment raises risks from data contamination, toxic or biased generation, personal-information leakage, hallucinations, and insecure generated code.The authors call for caution with synthetic training data because the consequences of training models on AI-generated data have not been investigated at scale.Source: §8 · p. 30

A Benchmarks for Downstrem Tasks

The appendix lists benchmark datasets for downstream tasks, grouping them across text-to-SQL, program synthesis, translation and repair, and other code tasks. It also includes benchmarks for unit testing, log parsing, and repository-level coding.

Text-to-SQL

  • The appendix lists benchmarks for text-to-SQL generation.Table 5 identifies the benchmark category but these passages do not provide dataset names or counts.Source: Table 5 · p. 116

Synthesis, translation, and repair

  • The benchmark inventory covers program synthesis, code translation, and program repair.Table 6 covers synthesis; Table 7 covers translation and repair.Source: Table 6 · p. 117, Table 7 · p. 118

Other code tasks

  • Other listed benchmark categories include code summarization, completion, commit-message generation, defect and vulnerability detection, retrieval, reasoning, type inference, and clone detection.Tables 8 and 9 organize these tasks into benchmark groups.Source: Table 8 · p. 119, Table 9 · p. 120

Testing, logs, and repository tasks

  • Table 10 lists benchmarks for unit-test and assertion generation, log parsing, and repository-level coding.Its caption also defines markers for task variants and repository-related measures.Source: Table 10 · p. 121

B Results on Downstream Benchmarks

The appendix reports model performance across downstream code benchmarks, spanning search, clone and defect detection, summarization, translation, and reasoning. The supplied passages identify the datasets and metrics but do not provide the benchmark scores.

NL-code search

Clone detection

Defect detection

Code summarization

Code translation

Code reasoning

Related papers

Loading 2311.07989v7…