A Survey on LLM-based Code Generation for Low-Resource and Domain-Specific Programming Languages

1 Introduction

• Limited training data for LRPLs contributes to lower LLM performance than in high-resource languages.The MultiPL-E results show significantly higher scores for Python than for LRPLs for most LLMs.Source: §1 · p. 2
• The survey systematically reviews 111 papers selected from a pool of over 27,000 studies on LLM code generation for LRPLs and DSLs.It examines strategies, metrics, benchmarks, and dataset collection and processing.Source: §1 · p. 2
• The authors identify a need for standard benchmark datasets and evaluation metrics for LRPL and DSL code generation.They also call for curating datasets and developing techniques and models to improve code generation.Source: §1 · p. 2
• The survey categorizes metrics and techniques, discusses challenges and future directions, and provides a roadmap for LLM code generation in LRPLs and DSLs.The authors also organize the surveyed papers in a GitHub repository.Source: §1 · p. 2

2.1 LRPLs and DSLs

• The survey notes that under-representation in LLM training data is associated with lower performance in LRPLs than in high-resource languages.The cited MultiPL-E comparison shows higher scores for Python and Java than for R and Julia.Source: §2.1 · p. 2
• Scores are consistently higher for high-resource languages such as Python and Java than for LRPLs such as R and Julia.Figure 1 presents model performance across high- and low-resource languages on MultiPL-E.Source: §2.1 · p. 2, Figure 1 · p. 3
• LRPLs and DSLs face data scarcity, and DSL syntax, semantics, and use cases are often poorly represented in general-purpose code datasets.The survey examines the two language categories together because they share challenges and may have synergies in LLM-based code generation.Source: §2.1 · p. 3

2.2 Related Surveys

• Prior surveys address LLMs in software engineering and code generation, while this work focuses on LRPL and DSL code generation and its models, evaluation, and performance-enhancement strategies.It also examines how datasets for these settings are collected, processed, and used.Source: §2.2 · p. 4
• The survey positions its specialized-language perspective as an extension of existing software-engineering literature, addressing challenges and opportunities beyond well-resourced languages.Earlier surveys cover topics including software-engineering tasks, LLM-based agents, code generation, and dataset curation.Source: §2.2 · pp. 3–4

3 Method

• The authors conducted a systematic literature review using a filtering approach outlined in prior work.The passage identifies as the source of the filtering approach.Source: §3 · p. 4

3.1 Study Goals

• The study uses the Goal Question Metric approach to define its objectives and investigate methodologies, performance, and evaluation for LRPL and DSL code generation.The stated issue is LLM underperformance associated with data scarcity and specialized syntax and semantics.Source: §3.1 · p. 4
• The review aims to analyze LLM use and techniques for LRPL and DSL code generation, identifying challenges and informing future research and development.Its stated perspective includes researchers and practitioners in software engineering, machine learning, and specialized programming domains.Source: §3.1 · p. 4

3.2 Research Questions

• The review asks which LLMs, metrics, and benchmarks are used to evaluate code generation in LRPLs and DSLs.It also asks what strategies and methodologies have been proposed to enhance performance.Source: §3.2 · pp. 4–5

Domains?

• Most research focuses on popular languages such as Python, leaving unclear which LLMs are used for LRPL and DSL code generation.The passage also identifies uncertainty about evaluation metrics and benchmark datasets in these languages.Source: Domains? · p. 5
• The review asks which metrics and benchmark datasets are used to evaluate LRPL and DSL code generation.Knowing this is framed as a way to understand model capabilities and whether specialized metrics are needed.Source: Domains? · p. 5
• A second research question asks what strategies and methodologies have been proposed to enhance performance.The passage links examining these approaches to identifying effective methods and guiding future research.Source: Domains? · p. 5

• Limited training data and specialized syntax are challenges that general-purpose LLMs may not effectively address in LRPLs and DSLs.These challenges are presented as barriers to leveraging AI-assisted coding tools in specialized and resource-constrained settings.Source: mance of LLMs for code generation in… · p. 5
• Understanding proposed strategies and methodologies is important for identifying effective approaches and guiding future research on LLM performance in LRPLs and DSLs.The stated aim is to improve performance in these specific contexts.Source: mance of LLMs for code generation in… · p. 5
• One research question asks how datasets for LRPLs and DSLs are collected, processed, and utilized to support code generation.Source: mance of LLMs for code generation in… · p. 5

tasks using LLMs?

• LRPLs and DSLs face data scarcity and imbalance that can hinder LLM performance, motivating careful dataset collection and processing.Understanding how datasets are collected, processed, and used is presented as important to generating accurate and reliable code in these languages.Source: tasks using LLMs? · p. 5

3.3 Search Strategy

• Searches covered arXiv, IEEE Xplore, Web of Science, and ACM Digital Library for studies published from January 1, 2020, to May 15, 2024.The search strategy paired LRPL/DSL language names and descriptive terms with LLM and code-generation terms, joining terms within each group by OR and the groups by AND.Source: §3.3 · p. 6

3.4 Eligibility

• Eligible studies investigated LLMs above 1 billion parameters for LRPL/DSL code generation and presented empirical evidence or a formal methodology.The survey excluded grey literature, papers of four pages or fewer, non-English work, tool demos, editorials, and duplicate or similar versions, but retained six relevant arXiv technical papers.Source: §3.4 · p. 7
• The survey excluded SQL to focus on underrepresented languages, since NL2SQL is extensively studied and already has comprehensive surveys.This choice aimed to direct attention to languages and domains that have received comparatively less focus in LLM-based code generation.Source: §3.4 · p. 7

3.5 Screening Process

• Screening started with 27,333 papers and proceeded through title, abstract, preliminary content, and final full-text review, including duplicate removal.Papers labeled uncertain at the abstract stage were examined more carefully.Source: §3.5 · p. 8
• 36 additional papers met the selection criteria through forward and backward snowballing, while works with over 300 citations were excluded from backward snowballing.The authors note that this prevented comprehensive backward snowballing for some seminal works.Source: §3.5 · p. 8
• Screening included studies generating First Order Logic, Lean theorem-proving tasks, CAD sketches in a designed sequential language, and a DSL for chemical-process questions.First Order Logic was included as a formal language with syntax and semantics used in software verification and theorem proving.Source: §3.5 · p. 8

3.6 Extracted Papers

• The final list contained 111 of 27,330 identified papers, and the authors read these papers to extract information for the research questions.Table 2 details the paper counts after filtering steps by database.Source: §3.6 · pp. 8–9, Table 2 · p. 9
• Research output on LLMs for LRPL and DSL code generation rose from one paper in 2020 and four in 2021 to 13 in 2022, 44 in 2023, and 49 in the first half of 2024.Source: §3.6 · p. 9
• The primary-search corpus covered 51 papers on LRPLs, 59 on DSLs, and 1 paper covering both.The survey also distributed analyzed papers across 39 conferences and journals.Source: §3.6 · p. 9
• A targeted search of major AI, NLP, and software-engineering venues from May 2024 to July 2025 added 5 eligible papers, integrated into findings but excluded from quantitative statistics.The primary systematic search covered January 1, 2020, to May 15, 2024.Source: §3.6 · p. 9

4.1 LLMs Used

• The LLaMA Family was the most popular choice for fine-tuning, used in 14 instances [70] [136] [204], followed by DeepSeek Family with 10 occurrences [24] [77] [136] [138] and then StarCoder family [13].Source: §4.1 · p. 11
• 60.4% of surveyed papers used proprietary models, while 39.6% used only open-source models.Proprietary models were often used for evaluation and comparison, while some studies fine-tuned open-source models on LRPL/DSL data.Source: Figure 3 · p. 10, §4.1 · p. 11
• Fine-tuning is the most frequently used enhancement, while other approaches rely heavily on proprietary GPT variants for complex LRPL and DSL tasks.Fine-tuning predominantly uses LLaMA, DeepSeek, and StarCoder families.Source: §4.1 · pp. 10–11

4.2 Metrics

• Automatic metrics include Pass@K, BLEU, ROUGE, Edit Similarity, and METEOR; Pass@K measures whether at least one of a model’s top-k predictions is correct.LLM Judge is also used for coding-assistant evaluation, but the survey notes reliability concerns.Source: §4.2 · pp. 11–12
• User-centric metrics capture practical use through acceptance rate, N-day retention, and #attempt_k, the average attempts needed for a satisfactory solution within k tries.Acceptance rate measures the proportion of generated completions users incorporate; retention measures users returning after N days.Source: §4.2 · p. 12
• Because Pass@k may miss specialized languages’ semantics and functionality, domain-specific evaluation methods are needed to assess their particular structures and purposes.The survey presents domain-specific metrics in Table 4.Source: §4.2 · p. 12, Table 4 · p. 13
• Manual evaluation supplements automated measures when assessing generated code, proofs, and domain-specific outputs that require human judgment.For HPC numerical-kernel suggestions, one approach rates quality and correctness from 0 to 1 across five levels; expert chemists assessed whether generated plans were scientifically sound and executable.Source: §4.2 · p. 12

4.3 Benchmarks Used to Evaluate LRPL and DSL Generated Code

• Table 5 catalogs benchmarks used to evaluate LRPL and DSL code generation and provides guidance on how each benchmark should be used.The supplied passage does not specify individual benchmarks or their recommendations.Source: §4.3 · p. 14
• DSL benchmarks span cross-language baselines, domain-specific and specialized-application testing, and formal reasoning; MultiPL-E is widely used across LRPLs including Rust, Julia, and Scala.The benchmark categories guide selection according to study objectives and the language capabilities being evaluated.Source: §4.3 · p. 14
• InterCode frames interactive coding as reinforcement learning, using code as actions and execution feedback as observations across Bash, SQL, and Python environments.Its environments draw on static datasets, while TLDR contains 1,879 Bash commands and 9,187 NL2Bash pairs.Source: §4.3 · p. 14
• VerilogEval [109] offers machine-generated and human-converted formats, while RTLLM [113] assesses 30 digital designs across syntax correctness, functional accuracy, and design quality.The human-converted format translates visual circuit and state-transition information into text.Source: §4.3 · pp. 14–15

4.4 Distinctive Features for LRPLs and DSLs

• Specialized DSL tasks need expert-aware metrics: pass rate is adapted to the language, and PLC outputs are judged for correctness, maintainability, and industry-standard adherence.CQLBLEU combines BLEU with semantic similarity, while FOL BLEU uses a tokenizer adapted to quantifiers, operators, and terms.Source: §4.4 · p. 16
• DSL correctness checks can be costlier than HRPL pass@k because hardware, chemistry, or configuration tasks may require simulators, domain solvers, or controlled environments.The survey describes Verify@k as requiring generated F* code to pass a type-checker and satisfy its specification.Source: §4.4 · p. 16
• DSL evaluation can extend beyond functional correctness: Syn-DC [110] checks RTL physical synthesizability, and Verilog assessment also uses power, performance, and area measures.PPA captures power consumption, clock speed, and area.Source: §4.4 · p. 17
• Chemistry and CAD metrics include NumberGenerated [159] and E_syntax and E_stat [135], while molecule generation measures validity, diversity, synthesizability, and target-class membership.Source: §4.4 · p. 17
• Because DSLs may have semantically equivalent but syntactically different forms, they need language-specific metrics, validators, simulators, executors, or symbolic checks.Source: §4.4 · pp. 17–18
• LLM-as-judge evaluation is difficult amid data scarcity and specialized domains, while manual LRPL and DSL assessment requires domain experts.Source: §4.4 · p. 18
• Benchmark coverage beyond LRPLs and some DSLs such as Verilog is low; data collection, augmentation, code translation, and domain expertise support benchmark creation.Source: §4.4 · p. 19
• Dataset validation relies on specialized tools such as Verilog synthesis, sandboxed Bash execution [173], and formal proof checkers, demanding infrastructure investment and domain expertise.Source: §4.4 · p. 19
• The LLaMA family is the most popular model choice for LRPL and DSL code generation, followed by DeepSeek and StarCoder families.Source: §4.4 · p. 19
• The literature uses four evaluation categories—automatic, user-centric, domain-specific, and manual—with Pass@k prevalent and domain-specific metrics also widely used.Source: §4.4 · p. 19
• The survey catalogs common benchmarks but notes that standardized benchmarks are unavailable for many LRPLs and DSLs.Source: §4.4 · p. 19
• Researchers have created their own evaluation datasets for DSLs.Source: §4.4 · p. 19
• Although some evaluation metrics and benchmarks resemble those for HRPLs, DSL evaluation involves differences and challenges that need to be addressed.Source: §4.4 · p. 19
• Code translation and compiler development are among the techniques adopted to make benchmarks for LRPLs.Source: §4.4 · p. 19

5.1 Model Adaptation Techniques

• A 60M-parameter FLAME model, pretrained exclusively on Excel formulas, competed with 175B-parameter Codex on last-mile repair and formula completion.This illustrates how domain-specific pretraining can make a small specialized model competitive on domain tasks.Source: §5.1 · p. 21
• A language-balanced training corpus improved low-resource-language pass@k by 66.48%, while high-resource-language performance decreased by 12.94% [133].Unimax limited how many times a language’s data could be duplicated.Source: §5.1 · p. 21
• LLVM IR-based transfer using nearly 4 million source-code/IR pairs across 12 languages yielded gains for high- and low-resource languages, without negative language interference [136].Reported gains included D, Ruby, and Swift; the authors describe IR as a semantically rich shared representation.Source: §5.1 · p. 21
• Fine-tuning on carefully filtered datasets like KStack-clean reduced syntax-error rates and enhanced code-completion capabilities; LLMs also handled unseen variable names, nouns, and operator descriptions.Augmentation techniques enabled effective domain-specific instruction tuning and contributed to BetterV achieving state-of-the-art performance on the VerilogEval benchmark [109].Source: §5.1 · p. 22
• Fine-tuning on LRPLs or DSLs improves code generation in their respective domains, and carefully curated domain-specific datasets can enable smaller specialized models to be competitive with or superior to GPT-3.5 or GPT-4.Source: §5.1 · p. 23
• LoRA [43] supports domain adaptation and task-specific tuning and has been used for LRPL/DSL generation, including GitHub workflows, PLC/ST [70], and FOL.Reported FOL tasks include direct translation, naive correction, and Chain of Thought correction using supervised fine-tuning and reinforcement learning.Source: §5.1 · p. 23
• LoRA enabled fine-tuning on a single GPU, making the process more accessible and cost-effective; studies also used QLoRA and proposed FLoRA.Source: §5.1 · p. 23
• FLoRA assigns each minibatch example unique low-rank adaptation weights, enabling efficient batching of heterogeneous requests while retaining LoRA’s parameter efficiency.This may improve throughput when serving diverse user queries in real-world applications.Source: §5.1 · p. 23

Prompting Strategies

• Few-shot prompting often outperformed zero-shot, but zero-shot did better for some tasks such as scatter plots, while VHDL gains were marginal under context-length constraints and verbose code.The cited studies also note that few-shot example selection and crafting require care because examples may not generalize across tasks.Source: §5.2 · p. 24
• Carefully paired specifications and OCL examples helped Codex generate more valid constraints, comments improved GPT-4 invariant completeness, and dynamically selected examples reduced overfitting risk.The GPT-4 example design particularly helped specify vector sizes in verification tasks.Source: §5.2 · p. 24
• ReAct generally achieved higher success rates with fewer turns and more admissible commands than Try Again, and elicited broader reasoning capabilities than Plan & Solve.The study suggests dynamic, adaptable reasoning can benefit interactive coding tasks.Source: §5.2 · p. 24
• Multi-PoT [114] executes few-shot code in multiple languages and votes across results, consistently outperforming single-language prompting across tasks and models.The approach includes low-resource languages such as R and uses language diversity to broaden reasoning.Source: §5.2 · p. 24
• Grammar Prompting augments examples with specialized grammars, while Intermediate Target Prompting [29] generates an intermediate solution before target-language code, with larger models benefiting more.Grammar Prompting predicts a grammar for the input and follows its rules when generating DSL output.Source: §5.2 · p. 25
• Multi-stage hardware prompting sequences interconnected prompts, and hardware verification decomposes assertion generation into specification analysis, signal mapping, and assertion generation for more accurate, reliable outputs.The verification stages use specialized language models to focus prompts at each step.Source: §5.2 · p. 25
• Self-planning [113] brought GPT-3.5 close to state-of-the-art, while hierarchical prompting [127] improved all tested LLMs and cut complex-module generation time by up to 73.73%.Self-planning first generates a design plan and reasoning steps; hierarchical prompting decomposes designs into submodules.Source: §5.2 · p. 25

Iterative Feedback

• Iterative feedback repeatedly gives the LLM error messages to correct; studies report that this loop can yield syntactically correct and functionally executable code [159] [194].The approach is used across domains including hardware design, verification, Rust proofs, and PDDL.Source: §5.2 · pp. 25–26

5.2 Prompting Strategies and Iterative Techniques

• For Verilog, GPT-4 corrected many errors with simple human input, but even advanced models often needed multiple iterations and substantial guidance for complex testbench creation.The study describes human feedback at three levels, from naming the general error type to identifying its location and suggesting a fix.Source: §5.2 · p. 26
• VeriRectify [166] and Mao et al. use simulator, compiler, and testbench diagnostics to prompt Verilog corrections, improving generated hardware-description quality and correctness.VeriRectify repeats enriched-prompt correction until errors disappear or an iteration limit is reached; Mao et al. return specific error messages and locations for regeneration.Source: §5.2 · p. 27
• LLM4PLC [70] combines grammar checkers, formal verifiers, user feedback, and automated error detection to iteratively refine PLC code, improving generation success and output quality.Source: §5.2 · p. 27
• For neuromorphic hardware, Vitolo et al. report that multiple Verilog refinement rounds improved handling of overflow and underflow and adherence to requirements.Source: §5.2 · p. 27
• CLARIFY [159] pairs an LLM plan generator with a rule-based verifier; feeding back the prior erroneous plan and detailed errors yielded the highest success rate, including for XDL procedures.It outperformed existing methods in successful plan generation and expert preference, and may help LLMs adapt to unfamiliar DSLs without extensive fine-tuning or domain-specific training.Source: §5.2 · p. 27
• A program verifier with Boolean feedback supports verified code transpilation across four DSL domains [8].Across domains, iterative methods generally prompt, evaluate generated code, and return detected errors for correction.Source: §5.2 · pp. 27–28
• GPT-4’s fully automated Rust proofs increased from 3 out of 20 programs to 14 out of 20 with an updated model and multiple refinement attempts [194].Verus error messages were fed back to refine initial proof attempts.Source: §5.2 · p. 27
• In PDDL, Smirnov et al. [160] use consistency checks and error correction to reduce domain errors and improve plan-generation success, especially in simpler scenarios.Human intervention may still be needed for complex domains, but the method reduces effort versus creating models manually.Source: §5.2 · p. 27

Section overview

• The survey groups these innovations into architectural modifications and novel training objectives intended to address LRPL and DSL challenges.The approaches are presented as addressing challenges not fully resolved by pretraining or prompting alone.Source: §5.3 · pp. 28–29

Novel Architectural Approaches

• MEV-LLM selects among expert Verilog models fine-tuned for four design-complexity levels, using an LLM classifier to assess prompt complexity.The design targets more tailored generation across basic, intermediate, advanced, and expert requirements.Source: §5.3 · p. 28
• Adjust QKNorm subtracts last-dimension means from Shellcode queries and keys before normalization and dot-product attention, targeting numerical differences as well as directional similarity.The adjustment is intended to help capture more nuanced relationships in low-resource Shellcode data.Source: §5.3 · p. 28
• MultiCoder combines MultiPL pretraining with programming-language-level expert routing, using shared experts for language-agnostic features and exclusive experts for each language.Dense lower layers learn transferable surface information, while upper Mixture-of-Experts layers capture language-specific semantic features to address capacity limits and support cross-lingual transfer.Source: §5.3 · p. 28

Novel Training Objectives

• DualSC treats Shellcode generation and summarization as complementary tasks, increasing the utility of limited training data and modeling code-comment interactions.The approach targets data scarcity and domain-specific vocabulary challenges.Source: §5.3 · p. 29
• For RTL generation, a comparative loss scores multiple candidate programs by quality, encouraging higher probabilities for stronger outputs and penalizing lower-quality ones.The objective uses reference code and relative quality of generated code to address discrepancies between training and inference behavior.Source: §5.3 · p. 29

5.4 Input, Output and Token Processing

• Variable replacement substitutes infrequent names with placeholders, reducing input length and simplifying prediction, while prefix generation handles mid-token autocompletion inputs.The section also describes HPC-oriented tokenization and early stopping intended to reduce token counts or improve Fortran training time.Source: §5.4 · p. 29
• Monitor-Guided Decoding improved compilation rates and agreement with ground truth across model sizes, with SantaCoder-1.1B outperforming text-davinci-003.It uses static-analysis suggestions to reshape token probabilities during decoding.Source: §5.4 · p. 29
• Constrained decoding improved program and execution accuracy in most cases, particularly for semantic parsing, but approximately tripled LLM API calls versus unconstrained decoding.The approach uses specialized BNF grammars and an Earley parser to identify valid prefixes and continuations.Source: §5.4 · pp. 29–30
• MCTS decoding significantly outperformed traditional methods in generating functionally correct Verilog, but took more time.Its search balances exploration and exploitation and prunes paths such as comment tokens.Source: §5.4 · p. 30
• YieldLang’s grammar-guided coroutine decoding achieved 100% parsing success for generated JSON strings across extensive tests.The approach also reduced JSON generation samples to 16.5% of the benchmark under optimal conditions.Source: §5.4 · p. 30
• Across decoding methods, improvements in code quality, syntactic validity, and correctness can trade off against computational efficiency and resource use.Other surveyed approaches include monitor-guided, constrained, MCTS, and coroutine-based decoding.Source: §5.4 · pp. 29–30

5.5 Retrieval Augmentation Generation (RAG)

• S3LLM combines retrieval from technical documents with FQL generation, using retrieved context to support more accurate, context-aware source-code analysis.The RAG component retrieves relevant information, while FQL generation supports querying source-code features.Source: §5.5 · pp. 30–31
• For F* program-and-proof generation, file context was the most crucial component, while the premise-selection model did not significantly affect performance.Higher overlap between prompt identifiers and the ground-truth solution correlated with better performance.Source: §5.5 · p. 31
• Combining example and knowledge retrievers yielded the best results in most cases, and the example retriever was particularly effective for Verilog coding on VerilogEval.The knowledge retriever extracts design principles and key-term explanations, while the example retriever searches for similar problems.Source: §5.5 · p. 31
• DocCGen retrieves library documentation and extracts schema information and grammar rules, performing well in out-of-domain and low-resource settings across metrics and model sizes.Its first stage uses sparse BM25 or dense ColBERTv2 retrieval.Source: §5.5 · p. 31
• Post-processing strategies achieved significant code-generation improvements without heavy reliance on external knowledge retrieval.The approach includes semantic filtering and interleaving, output-based score tuning, and temperature mixing, alongside lightweight KATE example selection.Source: §5.5 · p. 31
• RAG approaches can provide updated information and improve output accuracy, but depend on retrieved-information quality and add computation.Document segmentation can also cause information loss.Source: §5.5 · p. 31

5.6 Other Techniques

• A CAD-sketch DSL’s syntax-based input sequences significantly improved results over baselines, with ablations confirming their positive impact.The structured language represents sketch primitives and constraints uniformly for Transformer models.Source: §5.6 · p. 32
• A custom Qore-Base DSL reduced token usage by 77% versus the original API payload format, and the DSL-based approach implemented database schemas in 90% of experiments with optimal parameters.The DSL adapted elements from Prisma Schema Language to express Qore-Base features.Source: §5.6 · p. 32
• Temperature mixing improved top-1 accuracy for the M language and was most effective where models had less prior knowledge.It generates programs at low and high temperatures and combines them through reranking to balance diversity and accuracy.Source: §5.6 · p. 32
• Early stopping improved inference speed without compromising quality metrics by halting beam search when a lexer detects more than one program token.The approach also uses a hyperparameter to cap the number of complete tokens generated.Source: §5.6 · p. 33
• Code-to-code augmentation improved DeepSeek-Coder-7B-Instruct’s Verilog results, with smaller optimized training subsets more effective than the entire augmented dataset.It achieved 76.1% and 51.4% pass@1 on VerilogEval Human and Machine categories, respectively, and 87.3% syntactic correctness on VerilogFixEval.Source: §5.6 · p. 33
• ReflectionCoder improved results across benchmarks and languages, enhanced smaller models, and required no additional inference-time instructions.The approach uses reflection self-distillation and dynamically masked distillation.Source: §5.6 · p. 33

5.7 Comparison and Analysis of the Techniques

• RAG is favored when knowledge changes frequently, as with APIs and standards, whereas fine-tuning performs better in stable, pattern-heavy domains such as Verilog.RAG incurs ongoing retrieval costs per inference; fine-tuning has higher upfront training costs.Source: §5.7 · pp. 33–34
• LoRA provides 80-90% of full fine-tuning benefits at 10-20% of its computational cost, while full fine-tuning performs better when resources allow, particularly for complex DSLs such as Verilog.Novel architectures show promise but require substantial development resources and have less empirical validation than established techniques.Source: §5.7 · p. 34
• Iterative feedback improves correctness and compilation rates for DSLs with validators, but raises inference costs with iteration count; single-pass generation suits cost-sensitive settings or domains without automated validators.The comparison covers tools such as Verilog simulators and formal verifiers.Source: §5.7 · p. 34
• Tokenization can reduce token counts and improve training efficiency, while constrained decoding ensures syntactic validity; the survey presents them as complementary techniques.Prompting across studies improved Rust Pass@1, Pass@10, and Pass@25.Source: §5.7 · pp. 33–34

5.8 Distinctive Features for LRPLs and DSLs

• LRPL and DSL dataset strategies include domain-aware synthetic generation, transfer through intermediate representations such as LLVM IR, and expert-community curation.Template-based augmentation is noted as effective for configuration DSLs such as Ansible; expert networks are used for Coq datasets.Source: §5.8 · p. 34
• Specialized LRPL and DSL dataset creation faces confidentiality constraints because some code cannot be shared.The survey identifies a need for specialized approaches that account for these languages’ characteristics and constraints.Source: §5.8 · p. 34
• Specialized adaptation can require extensive human guidance or repeated iterations, while public-code abundance is associated with higher-quality outputs for C++ and Python than Julia or Fortran.Models also struggle with Coq syntax and semantics and may produce noncompliant Verilog designs or testbenches requiring human correction.Source: §5.8 · pp. 34–35
• Few-shot prompting generally outperforms zero-shot prompting, especially on complex tasks, although some task and domain settings favor zero-shot.The survey categorizes improvement methods into six groups and reports fine-tuning and prompting as the most prevalent.Source: §5.8 · p. 35

6.1 Dataset Curation

• Dataset strategies comprise collecting or curating sources, modifying existing datasets, synthesizing data with conversational LLMs, and manually creating datasets often used as evaluation benchmarks.Curated datasets may be used as-is, modified for study needs, or combined from repositories, literature, and forums.Source: §6.1 · p. 36
• 25 LRPL and 23 DSL studies used collected datasets; modified existing datasets appeared in 6 and 10 studies, while existing datasets were used in 3 and 7, respectively.GitHub was frequently used for LRPL curation.Source: §6.1 · p. 36
• Real-world and task-specific curation includes auto-completion usage from over 1,200 users, while most datasets contain code and comments and some add documentation, examples, and schemas.Bash datasets include human-written natural-language intents and use different commands in training and test splits.Source: §6.1 · pp. 36–37
• Nearly 4 million source-code–IR pairs across 12 languages formed SLTrans, using contest-platform samples to avoid dependency issues and ensure compilability.The languages include LRPLs such as Rust, Haskell, Fortran, D, Crystal, Nim, and Swift.Source: §6.1 · p. 36
• For DSLs, researchers most often compiled their own datasets, while others reused or adapted existing datasets to fit study objectives.Data scarcity, including licensing issues, motivates targeted collection for specialized languages.Source: §6.1 · p. 38
• DSL curation draws on language-specific sources, including Verilog repositories and hardware-security data, Ansible documentation, Coq libraries, Lean problems and courses, and PowerQuery forums.Other examples include Verilog textbooks, the OSCAT library for Structured Text, and datasets combining existing sources with newly translated inputs for Regex, FOL, and LTL.Source: §6.1 · pp. 38–39
• Dataset synthesis favored closed-source models such as GPT-3.5 and GPT-4, while a few studies used open-source StarCoder-15B [13] and DeepSeekMath-Base-7B.These models were used to generate, augment, and categorize code-related data.Source: §6.1 · p. 40
• For Lean auto-formalization, GPT-4 iteratively refined outputs using Lean proof-assistant errors or human feedback, stopping at verification or five iterations.The approach began with few-shot examples of informal-formal statement pairs.Source: §6.1 · pp. 40–41
• GPT-4 formalized over 60% of the IMO Shortlisted Problems in Algebra and Number Theory, producing the 149-problem FIMO dataset.Formalization used few-shot examples and iterative feedback from Lean verification or human experts, for up to five iterations.Source: §6.1 · pp. 40–41
• StarCoder-15B generated five test suites and 50 candidate translations per Python function for Julia, Lua, OCaml, R, and Racket in MultiPL-T [13].The candidates were filtered with the generated tests, and fine-tuning on the generated data significantly improved model performance for these languages.Source: §6.1 · p. 41
• GPT-3.5-Turbo generated approximately 75,000 problem-solution pairs for Wei et al.’s OSS-INSTRUCT dataset from code snippets using greedy decoding.The snippets came from StarCoderData, a filtered version of The Stack.Source: §6.1 · p. 42
• 4.5%: On miniF2F, the model trained on high-score proof data outperformed the model trained on low-score proof data.The authors used a fine-tuned DeepseekMath-Base-7B model to classify formal-statement quality.Source: §6.1 · p. 42
• For Verilog data, GPT-3.5 scored code quality and generated problem-solution pairs, while a Sentence-Transformer extended scoring and syntax and functional tests filtered generated code [77].The resulting dataset was used to fine-tune several code models for Verilog generation.Source: §6.1 · p. 42
• Manual curation supplies targeted evaluation data where high-quality large-scale datasets are scarce, including hand-written prompts and translated Kotlin HumanEval samples.Researchers manually wrote 50 Linux-task prompts, while 164 HumanEval samples were translated to Kotlin to address generic variable types in existing Kotlin benchmarks [63].Source: §6.1 · p. 43
• New DSLs and visual hardware inputs may require manual annotation or conversion into text, as with chemical-process DSL examples and HDLBits diagrams [109].HDLBits schematics, Boolean logic tables, Karnaugh maps, and state-transition graphs were converted into text descriptions and edge-list formats.Source: §6.1 · p. 43

6.2 Dataset Processing

• Raw LRPL and DSL datasets can contain irrelevant or noisy material, licensing problems, inconsistent styles, security risks or sensitive information, duplicates, outdated code, and inadequate documentation.These issues motivate preprocessing to create higher-quality data.Source: §6.2 · p. 43
• The survey groups common preprocessing into eight stages: collection, initial and fine-grained filtering, deduplication, code extraction and cleaning, quality checks, dataset-specific processing, and decontamination.The exact process varies with domain, data source, and language; collection is covered in previous sections.Source: §6.2 · pp. 43–44
• Initial filtering removes obviously irrelevant or low-quality data using language extensions [22], excludes files that are too small or too large [65] [106] [159], removes auto-generated files [16] [54] [138], and favors commercially friendlier licenses [63] [124] [133] [138].Source: §6.2 · pp. 44–45
• Deduplication helps prevent overrepresentation and bias from reused code, and training-set deduplication has been shown to improve LLM performance [42].Approaches include exact and near-duplicate detection, MinHash with Jaccard similarity, ROUGE-L, Levenshtein distance, file-level checks, and formula sketches.Source: §6.2 · p. 45
• Fine-grained filtering targets syntactic structure: self-contained Verilog modules are identified using ASTs and requiring module and endmodule with no instantiation [109]; other checks include procedural-block keywords [24] and excluding external references [204].The exclusion targeted files containing include or import keywords [204].Source: §6.2 · p. 45
• Studies filtered examples for syntax failures, irrelevant comments, domain incompatibility, and missing natural-language descriptions.For example, Zhao et al. [204] checked Verilog syntax and removed implementation-unrelated comments, while Yang et al. excluded unsupported commands and Zhang et al. removed workflows lacking natural-language descriptions.Source: §6.2 · pp. 45–46
• Quality checks removed erroneous samples and assessed code quality and compilability, filtered translations by similarity, and rejected inconsistent samples, including 22 erroneous FOLIO validation examples.One process removed 22 FOLIO validation samples due to various errors; another retained translations with a similarity score above 0.95. SKILL static analysis measured code quality and compilability.Source: §6.2 · pp. 46–47
• Extraction and cleaning isolated code or text, removed extraneous material, and standardized specialized-language or documentation data.Methods included extracting R code-comment pairs [32], OCR for Verilog textbooks, segmenting Unix manual pages, and normalizing an intermediate representation by removing comments and metadata while standardizing whitespace.Source: §6.2 · p. 46
• Other preparation supported multilingual and parallel data, including compiler-translated benchmarks and paired natural-language/formal representations.HumanEval and MBPP were translated into 18 additional languages for MultiPL-E; other work created parallel sentence and logical-representation data [84] and paired LTL formulas with natural-language descriptions.Source: §6.2 · pp. 46–47
• Dataset-specific transformations reshaped examples for target tasks, including formalizing math problems, adding metadata, and creating answer-choice variants.Mathematical problems were reframed as propositions for Lean [40], SKILL data yielded input-output pairs [26], and SimQA questions received three nearby incorrect answers for multiple-choice and open-ended evaluation.Source: §6.2 · p. 47
• Yang et al. added distinct prefixes to input data to handle multiple tasks, while another approach selectively added relevant UML classes to prompts to make them focused.For each example, the method parsed output y with full grammar G and extracted used rules, producing minimal specialized grammar G[y].Source: §6.2 · p. 48
• Processing includes converting sketches into primitive-and-constraint tokens [135] and applying atomic label changes, insertions, and deletions to train a Chain-of-Thought correction model.Source: §6.2 · p. 48
• Input formatting was task-specific: Excel inputs were lowercased to merge capitalization variants [95], while shell tasks used tokens for command endings, command sequences, and natural-language queries.Code snippets were marked with start/end tokens and language identifiers such as <ruby> and <en> [81].Source: §6.2 · p. 48
• To reduce benchmark contamination, Wei et al. removed problems matching popular benchmarks, while Liu et al. removed training samples with ROUGE-L above 0.5 against test cases [110].The latter approach checks similarity between training data and benchmark tests to reduce leakage and overfitting risk.Source: §6.2 · pp. 48–49

6.3 Distinctive Features for LRPLs and DSLs

• For LRPLs and DSLs, data quality and structure have greater impact than volume because real-world examples are limited, making domain knowledge and manual curation important.Some curation adds structured information such as schemas, intermediate representations, or formal specifications.Source: §6.3 · p. 49
• Manual curation can use LLM pipelines for data synthesis, prompting and refinement, scoring, iterative correction, and formal verification.Approaches also emphasize code diversity and semantic correctness, with evaluators such as verifiers in the feedback loop.Source: §6.3 · p. 49
• Dataset creation uses three approaches: curated datasets, manually created evaluation datasets, and LLM-synthesized datasets, with curated data divided into existing, modified, and newly collected sets.The survey notes that synthesized datasets are mostly generated using closed-source models.Source: §6.3 · p. 50
• Researchers source LRPL and DSL data from version-control and educational resources, domain-specific databases, forums, contests, industry, synthetic generation, and cross-lingual resources.Specialized sources, manual effort, and licensing constraints are common considerations.Source: §6.3 · p. 50
• Common preprocessing steps include initial and fine-grained filtering, deduplication, code and text extraction with cleaning, quality checks, and dataset-specific processing.Source: §6.3 · p. 50

7.1 Evaluation

• DSL evaluation is constrained by scarce domain experts and test data, syntax-sensitive outputs with multiple correct forms, and the need to assess semantic equivalence.Automatic metrics may miss factors such as feasibility or performance, making manual evaluation difficult to replace.Source: §7.1 · p. 50
• Researchers can improve evaluation consistency with shared domain-specific rubrics and semi-automated methods combining domain parsers with manual scoring.Reusing approaches across similar domains could reduce the effort of creating metrics and conducting manual evaluation.Source: §7.1 · pp. 50–51
• Researchers propose semantic-aware matching tools and portable, lightweight DSL sandboxes to support automated or hybrid evaluation across settings.Sandboxes could enable code execution or simulation and help develop reusable benchmarks and metrics; semantic matching can account for logic equivalence, such as symbol abstraction.Source: §7.1 · p. 51

7.2 Data and Benchmark

• Over 60% of publicly available Verilog code consists of learning exercises rather than production hardware designs [109], and proprietary use restricts access to many DSL datasets.Examples include Verilog designs containing intellectual property, Ansible playbooks exposing infrastructure details, and confidential pharmaceutical R scripts.Source: §7.2 · p. 51
• Dataset curation is costly because it requires domain expertise and must preserve specialized semantics and constraints while addressing licensing and data-access limits.Benchmark and data-preparation pipelines also require domain-specific techniques, and benchmark data must be checked for leakage from training datasets.Source: §7.2 · pp. 51–52
• Many DSLs lack standardized benchmarks, while Verilog-Eval [109] illustrates that widely adopted domain benchmarks can support reproducible and comparable research.Creating such benchmarks often requires domain-specific techniques, specialized execution setups, and expertise.Source: §7.2 · p. 51
• The survey calls for standardized, use-case-driven benchmarks for LRPLs and broader DSL suites, with automatic data augmentation and a performance leaderboard.Benchmarks should reflect each language’s paradigms, idioms, syntax, and semantics, while the leaderboard should cover benchmarks and model performance.Source: §7.2 · p. 52
• Researchers should build LRPL and DSL code-generation leaderboards and a collective platform for benchmarks and evaluation metrics.Related-domain translation and execution techniques, including for Verilog and RTL, can help develop benchmarks.Source: §7.2 · pp. 52–53
• Standardized data pipelines for LRPLs and DSLs, including feedback-based synthesis, error-driven correction, and targeted human-in-the-loop curation, could improve replicability and transferability.Active learning and few-shot sampling are examples of ways to focus limited human supervision.Source: §7.2 · p. 53
• Meta-resources documenting best practices, licensing constraints, structural requirements, and standard curation protocols could improve reproducibility and comparability across DSLs.A central platform for a more-studied language such as Verilog could be a first step.Source: §7.2 · p. 53
• Carefully curated synthetic LRPL and DSL examples could complement real-world data, but their impact on performance and generalization and their domain adaptation need evaluation.Adapting agentic workflows for synthesis and curation could also reduce manual evaluation.Source: §7.2 · p. 53

7.3 Performance-Enhancing Techniques

• Limited data, unfamiliar domain-specific grammars, scarce specialized tools, prompt sensitivity, and costly functional-equivalence assessment mean general code-generation techniques may need adaptation.The section notes that LLM-as-a-judge may not apply directly when the underlying models cannot generate DSLs.Source: §7.3 · p. 53
• Specialized approaches could bridge user specifications and LRPL or DSL code with LLM-friendly intermediate representations, formal methods, and iterative automated repair.The proposed direction also includes agentic frameworks connected to knowledge bases, specialized tools, and verifiers.Source: §7.3 · p. 54
• For DSLs with mathematical semantics, formal equivalence checkers, symbolic execution, and theorem proving could support code-generation techniques and evaluation metrics.The section also identifies cross-lingual transfer from HRPLs to LRPLs and further exploration of intermediate representations as research directions.Source: §7.3 · p. 54

Database selection

• The database search covered four selected sources, with backward and forward snowballing used to capture significant works the primary search might miss.The sources were ArXiv, IEEE Xplore, Web of Science, and ACM Digital Library.Source: §8 · p. 54

Scope

• SQL was excluded, leaving potential synergies between NL2SQL techniques and LRPL/DSL approaches for future study.Source: §8 · p. 54

Search coverage

• Time and resource limits left the initial resources incompletely covered, so some published work may have been missed.The summer 2025 extension added five papers from AI, NLP, and SE venues.Source: §8 · pp. 54–55

Language differences

• DSLs face more challenges than LRPLs because of specialized domains or narrow applications, and the distinction matters when interpreting results.The review tried to separate LRPL and DSL techniques where possible.Source: §8 · p. 55

Review scope

• The review filtered over 27,000 papers to study LLM code generation in low-resource and domain-specific programming languages.Source: §9 · p. 55
• The review examines evaluation metrics and benchmarks, improvement methodologies, and dataset curation and preprocessing strategies.Source: §9 · p. 55

Findings

• Most LRPLs and DSLs lack evaluation benchmarks, although studies and proposed evaluation metrics and improvement strategies have increased recently.Source: §9 · p. 55

Conclusion

• Challenges are more pronounced for DSLs because of specialized domain needs and required expertise, prompting discussion of related research opportunities.Source: §9 · p. 55