On the Effectiveness of Large Language Models in Domain-Specific Code Generation

1 INTRODUCTION

• The study defines domain-specific code as code tailored to a particular domain and typically built with domain-specific frameworks, where tailored training data is relatively scarce.Its dataset covers web development in Go and game development in C++, using six industry libraries.Source: §1 · pp. 1–2
• ChatGPT’s CodeBLEU score drops by 51.48% on average between general-purpose and domain-specific code corpora, often alongside misuse of third-party libraries.The study compares ChatGPT, PolyCoder, and CodeLlama across the two corpus types.Source: §1 · p. 2
• API sequences and docstrings improve results consistently across library-specific datasets compared with prompting with only a plain function signature.The prompt combinations were evaluated with ChatGPT.Source: §1 · p. 2
• DomCoder integrates domain knowledge through kg-GPT API-sequence inquiries, chain-of-thought prompting, and chain-of-thought fine-tuning.For chain-of-thought generation, each step produces a knowledge state suggesting relevant APIs and a task state describing the intended action.Source: §1 · p. 2
• All three DomCoder strategies improve domain-specific code-generation effectiveness under certain settings.They are applied to PolyCoder, StarCoder, and CodeLlama in zero-shot and fine-tuning paradigms.Source: §1 · p. 2

2.1 Code Generation

• Code-generation models include general-purpose LLMs and code-focused models trained on public code corpora, which primarily target general-purpose rather than domain-specific generation.Examples include Codex [8], PolyCoder, StarCoder [23], and CodeLlama [39].Source: §2.1 · p. 2

2.2 Prompting Large Language Models

• Prior prompting work explores prompt templates and contextual information such as Java class details, test cases, and API information [25]; this study injects chain-of-thought prompts into generation substeps.Chain-of-thought prompting uses intermediate steps, while this work uses a finer-grained unit than a whole function.Source: §2.2 · pp. 2–3

2.3 Code Generation with Domain Libraries

• Prior domain-library approaches include API retrieval and coding, repository-context prompts, and library-oriented generation; this work integrates APIs into internal coding states.It also proposes a GPT-based API recommender that generates API-knowledge prompts.Source: §2.3 · p. 3

2.4 Studies on Code Generation with LLMs

• Prior empirical studies examine code-generation usability, wording robustness, and benchmark adequacy; this study centers on LLM proficiency with domain-specific libraries.The authors describe their scope as evaluating how well LLMs acquire and use domain-specific libraries.Source: §2.4 · p. 3

3.1 Data Collection; 3.2 Evaluated Language Models

• The dataset covers web development in Go and game development in C++, using six libraries; the evaluated models are ChatGPT, PolyCoder, and CodeLlama.Repositories were filtered for use of the selected libraries, and functions with empty bodies or simple names such as main or init were excluded.Source: §3.1 · pp. 3–4, §3.2 · p. 4

3.3 Evaluation Metrics

• Because real-execution evaluation was infeasible for these complex functions, the study uses BLEU and CodeBLEU rather than pass@k.The authors cite interrelated functions, platform and library dependencies, and the impracticality of manually preparing tests and configurations.Source: §3.3 · p. 4

3.4.1 Methodology.

• For the domain-specific completion task, models receive function signatures and complete function bodies; the study samples 500 functions per library dataset and compares generations with reference code.HumanEval and CodeSearchNet (Golang) serve as open-domain comparison corpora.Source: §3.4.1 · p. 4

3.4.2 Quantitative Results.

• ChatGPT-3.5’s average BLEU score on domain-specific datasets drops 70.35% versus CodeSearchNet, and its average CodeBLEU drops 51.48%.Across the three models, performance declines on domain-specific datasets compared with open-domain datasets; Table 2 reports the comparisons.Source: §3.4.2 · pp. 4–5, Table 2 · p. 5

3.4.3 Qualitative Results.

• Only 16% of 200 sampled ChatGPT-generated functions were correct, and most observed mistakes involved API misuse or missing APIs.Examples include calling Gin’s router.GET rather than router.Group and omitting an API needed to supply the next call’s parameter.Source: §3.4.3 · pp. 5–6

3.5 RQ2: Prompting LLMs for Domain-Specific Code Generation; 3.5.1 Prompt Design.

• Prompting experiments examine how domain knowledge affects ChatGPT generation, including individual knowledge prompts and combinations of prompt elements.The tested elements include library imports, API sequences, and API docstrings, with combined API-plus-docstring prompts arranged in either order.Source: §3.5 · p. 6, §3.5.1 · p. 6

3.5.2 Results.

• Knowledge-enhanced prompts improved results across library-specific datasets, with API sequences outperforming other basic prompt types and combined-prompt order affecting results.Adding docstrings to APIs did not consistently improve on using APIs alone; long docstrings sometimes hindered API comprehension.Source: §3.5.2 · pp. 6–7

4 CODE GENERATION WITH DOMAIN KNOWLEDGE INTEGRATION

• The section tests API recommendation, chain-of-thought prompting, and chain-of-thought fine-tuning as strategies for automatically integrating domain knowledge into code generation.The motivation is that real-world input code context may contain no prior API knowledge.Source: §4 · p. 7

4.1 Prompting LLM by Inquiring External Knowledge

• kg-GPT uses an API knowledge inquirer to recommend a sequence of APIs from the input function signature, then supplies that sequence alongside the signature to generate the function body.The approach separates API inquiry from code generation.Source: §4.1 · p. 7
• The inquirer is trained on function signatures paired with API usages and adds its predicted API list as a comment above the function signature in the LLM prompt.The inquirer uses causal language modeling to predict the API list.Source: §4.1 · p. 7

4.2 Prompting LLMs with Chain-of-thought

• The chain-of-thought strategy is motivated as a more fine-grained alternative to the shallow integration of knowledge used by kg-GPT.Source: §4.2 · p. 7

4.2.1 Programming as a Chain-of-thought Process.

• The method frames programming as a chain of steps, with domain-specific API calls serving as separable sub-activities within code generation.A step is an action chosen from prior context and implemented in one or a few statements.Source: §4.2.1 · p. 7
• Figure 2 illustrates the three experimented strategies for knowledge integration, while Figure 3 presents a chain-of-thought view of code generation.Source: Figure 2 · p. 8, Figure 3 · p. 8
• Each knowledge state combines an API state suggesting APIs for a subtask with a task state describing the action to perform.Together, the states summarize the chain of operations associated with each coding step.Source: §4.2.1 · p. 8
• Generation loops through understanding the current input, predicting the next knowledge state, and generating the next code snippet until the function is complete or the maximum length is reached.Source: §4.2.1 · p. 8
• In the HandleGetRequest example, the first state pairs gin.Context.Query with retrieving a request parameter, after which the model generates the corresponding query code.If a step needs no API, its API state is empty and API prediction is skipped.Source: §4.2.1 · p. 8

4.2.2 Overall Design.

• The prompt generator predicts an API state from the state history, appends its API docstring as the task state, and combines these states with code history to prompt code generation.Source: §4.2.2 · pp. 8–9

4.3 Chain-of-Thought Fine-tuning

• CoT-FT conditions API-state prediction on code history because API-state history alone lacks code syntax and structure, which affects prediction accuracy.Source: §4.3 · p. 9
• At each step, the model reads the input and prior steps, predicts a knowledge state, appends it to the history, and generates the corresponding code snippet.The knowledge state contains an API sequence and a task description.Source: §4.3 · p. 9
• After the stepwise process, the knowledge states are removed to obtain the final generated code.Source: §4.3 · p. 9
• The method combines all steps into one training sequence and fine-tunes a causal language model to predict the next token from preceding tokens.Source: §4.3 · p. 9

4.4 Experimental Setup

• The API knowledge resource pairs each library API name with a docstring describing its behavior.Table 5 presents examples of this knowledge; the resource is organized as a dictionary.Source: Table 5 · p. 9, §4.4 · p. 9
• Target API calls are extracted from dataset functions and used to retrieve matching documentation from the knowledge base.The extraction identifies target-library variables and calls, while API descriptions come from official references or library source code.Source: §4.4 · p. 9
• For CoT fine-tuning, API calls and docstrings are added as comments above their corresponding code calls; fine-tuning uses PolyCoder.The other backbones are PolyCoder, StarCoder, and CodeLlama; fine-tuning experiments were conducted only on PolyCoder.Source: §4.4 · p. 9
• DomCoder is compared with zero-shot backbones, a few-shot method using 10 same-library examples, and domain-fine-tuned PolyCoder.The few-shot comparison follows Ahmed et al..Source: §4.4 · p. 10

4.5 Results

• 49.33% BLEU and 9.82% CodeBLEU gains with zero-shot kg-GPT, and 17.10% BLEU and 4.20% CodeBLEU gains with fine-tuning chain-of-thought; CoT-FT reaches the best scores.These are average score increases under their respective settings.Source: §4.5 · p. 10
• Improvements are larger on some weak zero-shot datasets, while CoT-FT gains are modest except for CodeBLEU on some libraries.The authors attribute differences between kg-GPT and CoT-PT to their distinct API-integration strategies; incremental API prediction can be unstable because it lacks code syntax and structure information.Source: §4.5 · p. 10
• Domain fine-tuning improves over zero-shot, and adding knowledge to training data in a chain-of-thought format raises performance further, though gains are relatively marginal.The authors note that this fine-tuning strategy still has room for improvement.Source: §4.5 · p. 10
• BLEU measures n-gram match, whereas CodeBLEU also combines syntax and dataflow matches, so their scores can differ substantially.For Unreal Engine and bgfx, CodeBLEU is ten times BLEU; correct code with different variable names can have low n-gram match but high syntax and dataflow match.Source: §4.5 · p. 10
• kg-GPT recommends domain API sequences with high accuracy, reaching 10 times PolyCoder's BLEU accuracy on bgfx.Table 7 reports the API-recommendation comparison using BLEU and Hit Ratio.Source: §4.5 · p. 11

4.6 Qualitative Analysis

• In the Gin example, CoT-FT correctly predicts the task description and API name at each step, then generates code meeting the requirements.Other methods make errors; PolyCoder-FT calls Gin APIs, but none are correct for the task.Source: §4.6 · p. 11
• The authors conclude that integrating domain knowledge into LLM generation can improve domain-specific code generation.This finding summarizes the qualitative analysis's broader implication.Source: §4.6 · p. 11

4.7 Industry Study

• The study selects 30 test functions using Tencent messenger-product developers' tRPC-go library, with functions containing 3 to 7 APIs.Functions with identical API sequences were grouped, one function was randomly selected from each class, and difficult-to-understand functions were filtered out.Source: §4.7 · pp. 11–12
• Triple the semantic correctness is reported for DomCoder-generated code compared with original CodeLlama-7B output.Table 8 presents the industry-study performance comparison.Source: Table 8 · p. 12, §4.7 · p. 12
• Three relevant-team developers independently rate task relevance and semantic correctness, resolving disagreements through discussion.DomCoder is built on CodeLlama-7B and evaluated using the kg-GPT strategy.Source: §4.7 · p. 12

5.1 Future Directions

• Future work could incorporate knowledge beyond API documentation, including library class relationships, code explanations, workflows, and templates.The discussion identifies these as possible sources of programming knowledge for prompts and knowledge integration.Source: §5.1 · p. 12
• Docstrings can improve prompt design, but increase code length, while code-oriented models have limited ability to understand and generate natural-language descriptions.The authors suggest mechanisms that bridge programming-language and natural-language information, such as jointly predicting code and knowledge.Source: §5.1 · p. 12
• The kg-GPT results motivate more sophisticated knowledge enquirers, such as a QA or search engine that finds factual knowledge and supplies it during code generation.Source: §5.1 · p. 12
• Because chain-of-thought works at fine-tuning and usage stages, future work could integrate knowledge during pre-training or generate code modules in a specific order.The proposed pre-training direction augments code with API knowledge; ordered submodules would simulate chain-of-thought.Source: §5.1 · p. 12

5.2 Threats to Validity

• The study covers three LLMs, code-specific models up to 2.7B parameters, web and game development, and BLEU/CodeBLEU metrics that may not fully represent human experience.The authors note that findings from two domains may offer insights applicable elsewhere, but may not represent all recent large-scale models.Source: §5.2 · p. 12

• LLMs such as ChatGPT exhibit sub-optimal performance on domain-specific code, while adding knowledge prompts about domain libraries improves performance.Source: §6 · p. 12
• DomCoder integrates domain knowledge through an external knowledge GPT, chain-of-thought prompting, and chain-of-thought fine-tuning.Source: §6 · p. 12
• The three strategies show effectiveness in both zero-shot and fine-tuning settings.Source: §6 · p. 12