A Study on the Impact of Natural Language Differences in Prompts on Automatic Code Generation Using LLMs

• Building on Koyanagi et al., the study broadened the evaluation to AtCoder, LeetCode, and BigCodeBench and seven proprietary and open-source LLMs.The models include GPT-4o, o3-mini, DeepSeek-V3.2, GitHub Copilot, Llama3, and Qwen2.5-Coder 0.5B and 14B.Source: §1 · pp. 3, 5
• Language gaps appeared across all three datasets, ranging from 3.6%–6.6% for open-source models and 2.7%–15.6% for closed-source models; officially supported languages often had the highest median Accuracy.The language gap is the difference between the highest and lowest median Accuracy across languages for each dataset and model type.Source: §1 · p. 4
• Translation improved Accuracy on LeetCode and BigCodeBench but had mixed effects on AtCoder, where English translation changed open-source models’ median Accuracy by +3.2% and closed-source models’ by -5.3%.The study concludes that translation’s effectiveness depends on dataset and model type.Source: §1 · p. 4
• AtCoder statements were classified as narrative-style more often than LeetCode statements (74.0% vs. 21.2%), and their median length was more than twice as long.The study proposes that narrative structure and problem-statement length may relate to dataset differences in code-generation performance.Source: §1 · p. 5
• Additional analyses tested whether the findings generalized to C++ code generation on AtCoder and to low-resource natural languages.The study also reports that narrative aspects’ impact varies with context-information length.Source: §1 · p. 5

2.1 Code Generation Benchmarks for LLMs

• Benchmark studies evaluated code generation on standard datasets, while contest studies also examined programming-language differences and performance across difficulty levels.Source: §2.1 · p. 6

2.2 Biases in LLMs

• CodeGeeX and CodeGen-Multi-16B had different best-performing programming languages, indicating that code-generation performance depends on programming language.Source: §2.2 · p. 7
• Translation effects varied by target language and model in NLP [55], while minor English prompt perturbations degraded code generation.Source: §2.2 · pp. 7–8
• FuxiTranyu's results showed that a multilingual model could remain behind English-centric models, although its instruction-tuned version surpassed them in translation and summarization.The study attributed the findings to language-distribution considerations in training data.Source: §2.2 · p. 8
• Multilingual MBPP evaluations found code-generation performance varied across languages, while self-translation increased bias and external translation reduced it.Source: §2.2 · p. 8
• Studies found practical difficulties with non-English code elements and local naming conventions, while multilingual code-comment evaluations showed BLEU, ROUGE, METEOR, and BERTScore did not reliably reflect correctness across languages.Source: §2.2 · p. 9

2.3 Novelty of This Study

• Prior evidence focused on HumanEval/MBPP and often English and Chinese, leaving contest and specification-based tasks and Japanese underexplored.Source: §2.3 · p. 9
• The study extends an initial AtCoder/GitHub Copilot analysis to LeetCode, BigCodeBench, seven LLMs, and translation as a mitigation strategy.The initial analysis found language bias but was limited to one dataset and one model.Source: §2.3 · p. 9

3.1 Target Natural Languages

• English, Chinese, and Japanese were selected as high-resource languages, author-verifiable or widely used academically, and included in Koyanagi et al..The authors used native-speaker verification for Japanese and Chinese and assumed English could be verified to an acceptable degree.Source: §3.1 · p. 10

3.2 Data Collection

• The evaluation sampled 756 AtCoder problems, 100 LeetCode problems, and 288 BigCodeBench tasks, spanning contest problems and broader real-world code-generation scenarios.AtCoder included 189 problems each at difficulty levels A–D; the LeetCode subset had 16 easy, 52 medium, and 32 hard problems. The BigCodeBench sample was selected for a 95% confidence level and 5% margin of error.Source: §3.2 · pp. 10–13
• Because the benchmarks did not cover all three languages, the researchers translated AtCoder English into Chinese, LeetCode English into Japanese, and BigCodeBench English into both languages.DeepL produced draft translations that Japanese- and Chinese-speaking authors reviewed and refined; an error-coding book was developed through open coding until no new codes emerged over 50 consecutive samples.Source: §3.2 · pp. 13–14
• These translated prompts were not present in the LLMs’ training data, reducing training-data leakage in the evaluation.The authors state that this process generates problems absent from model training data.Source: §3.2 · p. 14

3.3 Studied LLMs

• The study selected seven models spanning closed-source and open-source systems: GPT-4o, o3-mini, DeepSeek-V3.2, GitHub Copilot, Llama-3.1-8B-Instruct, and Qwen2.5-Coder-0.5B and -14B.Selection criteria included programming-community use, commercial/API and locally deployable settings, and models developed in different countries.Source: §3.3 · pp. 15–16

3.4 Machine Translation for Improving the Code Generation Accuracy

• For mitigation experiments, the researchers translated problems from the lowest-accuracy language into the highest-accuracy language using DeepL, Google Translate, or GPT-4o, without human verification.This practical-use setup differs from the human-verified translations used to create semantically equivalent benchmark prompts.Source: §3.4 · p. 16

3.5 Experimental Workflow

• The workflow generated Python from language-specific prompts containing problem statements and input/output details, while Copilot received equivalent context as comments in a VSCode source file.When GPT-4o or DeepSeek-V3.2 produced two snippets, the researchers selected the one identified as more time-efficient; Copilot’s suggestions were generated from the prepared file context.Source: §3.5 · pp. 16–18
• When multiple code options were proposed, the study used the first option, assuming developers would typically choose it.Source: §3.5 · p. 19
• Accuracy was the percentage of generated code that passed all test cases for each language and problem combination.Code was marked “Pass” only if it passed every test case; failing at least one yielded “Fail.”Source: §3.5 · p. 20
• Five repeated generation and evaluation runs were summarized using median Accuracy to mitigate randomness in generated code.Source: §3.5 · p. 20
• Prompts in the lowest-Accuracy language were translated into the highest-Accuracy language using DeepL, Google Translate, and GPT-4o, then re-evaluated.The source and target languages were determined from the five-run results.Source: §3.5 · p. 20

4.1 Approach

• The analysis measured Accuracy, Pylint coding-rule violations, and cyclomatic complexity to compare functional correctness and code characteristics across prompt languages.Source: §4.1 · p. 20

4.2 Results

• DeepSeek-V3.2 achieved the highest median Accuracy in 17 of 24 dataset, difficulty, and language combinations.Source: Table 1 · p. 21, §4.2 · p. 23
• Officially supported languages most often yielded highest Accuracy: AtCoder English/Japanese scored 14 and Chinese 7; LeetCode English/Chinese scored 11 each and Japanese 2; BigCodeBench English scored 5 and Japanese/Chinese 0 or 1.Source: Table 3 · p. 22, §4.2 · p. 23
• o3-mini with Japanese and DeepSeek-V3.2 with English or Chinese each achieved highest Accuracy in three of eight cases.Source: Table 4 · p. 23, §4.2 · pp. 23–24
• Median language gaps were 6.6%, 6.3%, and 3.6% for open-source models in AtCoder, LeetCode, and BigCodeBench, versus 2.7%, 15.6%, and 11.5% for closed-source models.Source: Table 6 · p. 24, Table 5 · p. 24, §4.2 · p. 24
• Open-source models tended to perform better in English but remained below closed-source models overall; DeepSeek-V3.2 was competitive with Chinese prompts in AtCoder.Source: Table 1 · p. 21, §4.2 · p. 25
• Pylint warnings per line stayed below 0.101 across languages, similar generated-code LOC made LOC-related bias negligible, and complexity tracked difficulty and model capacity more than language.Source: §4.2 · p. 25, Table 7 · p. 26, Table 8 · p. 27

5.1 Approach

• The experiments translated prompts into dataset-level officially supported languages rather than selecting targets separately for each model or difficulty level.Targets were English and Japanese for AtCoder, English and Chinese for LeetCode, and English for BigCodeBench.Source: §5.1 · p. 29

5.2 Results

• 80.6% of open-source-model cases improved when translated into English, with a mean Accuracy increase of 5.2%.The table summarizes the effect across target languages for open-source models.Source: Table 16 · p. 32, §5.2 · p. 32
• 83.3% of AtCoder cases for closed-source models worsened after translation, with a mean Accuracy decrease of -11.2%.The authors note that translation can harm closed-source-model performance depending on the dataset.Source: §5.2 · p. 32, Table 17 · p. 33
• Translation tools produced similar performance trends, and their average BLEU scores ranged from 0.501 to 0.564.In LeetCode, median Accuracy improvements were +6.0% for Google, +5.8% for DeepL, and +6.2% for GPT; median BLEU scores across tools ranged from 0.484 to 0.572.Source: §5.2 · pp. 33–34, Table 18 · p. 33, Table 19 · p. 33
• Translation improved LeetCode and BigCodeBench and AtCoder for open-source models, but reduced AtCoder performance for closed-source models and when targeting Japanese.For AtCoder-to-English translation, median changes were -5.3% for closed-source and +3.2% for open-source models; LeetCode-to-English yielded +6.2% and +6.3%, respectively, while BigCodeBench-to-English yielded +2.4% and +0.2%.Source: §5.2 · pp. 30, 31, 34, Table 13 · p. 32, Table 14 · p. 32

6.1 Influence of Problem Statement Narrative

• Translation often improves Accuracy for LeetCode and BigCodeBench, but its effect on AtCoder is more limited and varies by model group and target language.The analysis asks whether narrative structure and problem-statement length help explain these differences.Source: §6.1.1 · p. 34, §6.1 · p. 34

6.1.1 Motivation

• The authors hypothesize that AtCoder’s more narrative-style statements make translation-based mitigation harder than for LeetCode, and focus on these two contest datasets for contrast.They investigate whether narrative aspects affect Accuracy, potentially through translation accuracy.Source: §6.1.1 · pp. 34–35

6.1.2 Approach

• A statement is narrative-style if it includes at least one of three elements: a specific subject, a contextualized objective, or an unrelated sentence.The criteria treat contextual elements as removable to abstract or decontextualize a problem.Source: §6.1.2 · p. 35
• The first and second authors manually reviewed 50 sampled AtCoder D-level problems and all 32 LeetCode Hard problems; inter-rater agreement was Cohen’s kappa 0.941.They discussed inconsistent labels and reached consensus.Source: §6.1.2 · p. 35
• Problems satisfying at least one criterion were classified as Narrative; the rest were Expository, and the authors calculated code-generation Accuracy for both groups.The comparison was limited to AtCoder D-level and LeetCode Hard problems because lower-difficulty problems generally have high Accuracy across languages.Source: §6.1.2 · p. 36
• Tables 22 and 23 report post-translation Accuracy for AtCoder D-level and LeetCode problems, respectively, for closed-source models only.The supplied passages give the tables’ scope but not their cell values.Source: Table 22 · p. 37, Table 23 · p. 37

• AtCoder had 74.0% narrative and 26.0% expository problems, versus LeetCode’s 50%/50%; 86.5% versus 43.7% of narrative problems met at least two criteria.On AtCoder, 70.3% met two narrative criteria and 16.2% met three; LeetCode’s corresponding shares were 37.5% and 6.2%.Source: Observation 14) AtCoder has a… · p. 37
• After translation, narrative problems had lower median Accuracy on AtCoder (overall difference -6.9%) but higher median Accuracy on LeetCode (overall difference 3.4%).The analysis focused on closed-source models; the reported differences are medians across translation tools.Source: Observation 14) AtCoder has a… · pp. 37–38
• AtCoder’s median sentence count was more than twice LeetCode’s, and the authors suggest long narrative contexts may impair comprehension, unlike shorter LeetCode contexts.They conclude that narrative aspects’ effect on Accuracy depends heavily on prompt context length.Source: Observation 14) AtCoder has a… · pp. 38–40
• Narrative-related factors accounted for 20 of 50 sampled DeepSeek AtCoder failures (Cohen’s kappa = 0.623), with misunderstanding and inappropriate abstraction identified as primary failure patterns.Two annotators independently reviewed the cases, and final labels were set by discussion and agreement.Source: Observation 14) AtCoder has a… · pp. 39–40

6.2.1 Motivation

• The extension examines whether language bias generalizes beyond Python and high-resource languages by broadening both programming-language and natural-language coverage.The motivation notes that C++ is widely used in competitive programming and that language bias may be more pronounced in low-resource languages.Source: §6.2.1 · p. 40

6.2.2 Approach

• The experiments add C++ and prompts in Thai, Slovak, Finnish, Bulgarian, and Norwegian Bokmål, using DeepL translations for the low-resource languages.All extensions use AtCoder; LeetCode and BigCodeBench are excluded because preparing reliable multilingual prompts and evaluation cases was impractical. The C++ setup retains the Python problem instances, test cases, and evaluation framework.Source: §6.2.2 · pp. 40–41

6.2.3 Results

• DeepSeek-V3.2 achieves the highest count overall in C++ (9 cases), while English has the highest count across languages (13), compared with 9 each for Japanese and Chinese.English leads on lower-difficulty problems, while Japanese shows improved performance on higher-difficulty problems, consistent with the Python observations.Source: Table 26 · p. 42, §6.2.3 · p. 43, Table 27 · p. 43
• In C++, DeepSeek-V3.2 is the most consistently top-performing model, especially on lower-difficulty A and B problems, while o3-mini leads in several harder settings, particularly D.The C++ experiments keep the main analysis’s model configurations, prompt formats, and evaluation metrics consistent.Source: Table 26 · p. 42, §6.2.3 · p. 43
• Performance differences persist among low-resource languages, with Slovak and Finnish often outperforming Bulgarian across models and difficulty levels.Tokenizer-language compatibility is proposed as a possible explanation, but the authors state that this hypothesis requires further investigation.Source: §6.2.3 · pp. 43–44, Table 28 · p. 44

Observation 21) Model capability does not eliminate language bias in low-resource

• Performance degrades in some low-resource languages even for high-performing models, and cross-language variance widens as task difficulty increases.The results indicate that improved model capability alone does not eliminate language bias.Source: Observation 21) Model capability does… · p. 44
• In C++, DeepSeek-V3.2 performs best overall, AtCoder-supported languages tend to yield higher Accuracy, and Chinese sometimes performs comparably to Japanese.Language bias also persists in low-resource languages, regardless of programming language or model capability.Source: Observation 21) Model capability does… · p. 45

7.1 Implications to LLM-based Code Generation for Multiple Natural Languages

• LLMs perform better in a dataset’s officially supported language, especially on difficult problems, while simple-context problems can achieve high accuracy across languages.The discussion attributes cross-language effectiveness partly to publicly available platform-specific resources and suggests that richer resources and simpler prompts may help when a language is unsupported.Source: §7.1 · p. 45
• AtCoder problems are more narrative and longer than LeetCode problems; narrative style harms AtCoder performance but helps LeetCode, where translation improves results, unlike AtCoder.The discussion identifies context length and narrative aspects as factors in translation efficacy, suggesting translation may work poorly when prompts are long and highly narrative.Source: §7.1 · p. 46

7.2 Avenues for Future Research

• Future work should compress prompts by removing unrelated sentences or paraphrasing problem statements while preserving task-relevant information.The discussion also points to identifying informative content rather than merely shortening prompts.Source: §7.2 · p. 46
• Future work should explore structuring longer prompts so LLMs can process them more efficiently without losing useful information.One proposed approach is converting problem statements into structured formats that guide the model to generate solution steps.Source: §7.2 · p. 46
• Researchers should quantify how context length and narrative aspects affect multilingual code generation and translation efficacy across datasets and model types.This could help determine appropriate prompt context, including the number of statements and narrative aspects.Source: §7.2 · pp. 46–47
• Future translation research should identify language-specific context through paraphrasing and human verification, since direct translation may omit information needed for code generation.The study notes that grammar, vocabulary, and common expressions differ across languages.Source: §7.2 · p. 47

8.1 Construct Validity

• Construct validity is limited by unvalidated machine translation for low-resource languages and Python-only generation outside supplemental AtCoder-only C++ experiments, since programming-language choice may affect performance.The authors recommend evaluating multiple programming languages across datasets.Source: §8.1 · p. 47

8.2 Internal Validity

• Closed-source models’ undisclosed internal structures limit reproducibility, while generation randomness was addressed with five repetitions, itself an experimental parameter.Source: §8.2 · p. 48
• Language-specific training-data contamination may influence observed performance differences, and translation controls semantic content but cannot eliminate this threat.Public solutions, blogs, repositories, and discussions may appear in training data; their availability and quality can vary by language.Source: §8.2 · p. 48
• LeetCode’s prepared tests may miss corner cases because hidden official tests are unavailable, despite 15 non-duplicated tests per problem validated on its website.Source: §8.2 · p. 48
• Translated statements were checked by native speakers using a coding book, but reviewer judgment may affect accuracy, which requires quantitative evaluation.Source: §8.2 · p. 48

8.3 External Validity

• External validity is limited because three datasets and seven selected LLMs do not represent all code-generation tasks or available models; the study extends prior work.The authors provide a replication package to encourage further research.Source: §8.3 · p. 48

9 Conclusion

• Officially supported languages frequently yielded the highest performance across AtCoder, LeetCode, and BigCodeBench, and code-generation performance varied by prompt language.The officially supported languages were English and Japanese for AtCoder, English and Chinese for LeetCode, and English for BigCodeBench.Source: §9 · p. 49
• Translating Japanese prompts into English improved LeetCode median performance by +6.2% for closed-source models and +6.3% for open-source models.Source: §9 · p. 49
• For English translation, AtCoder changes were +3.2% for open-source and -5.3% for closed-source models, while BigCodeBench changes were +2.4% and +0.2%, respectively.The BigCodeBench changes were for closed-source and open-source models, respectively.Source: §9 · p. 49
• Users should carefully select prompt language, favoring languages with ample data sources, because language choice affects code-generation performance.Source: §9 · p. 49
• Translation may enhance code generation depending on dataset and model type, so its effect should be evaluated in the target setting before practical use.Source: §9 · p. 49
• Long-context narrative-style prompts can hinder code generation, whereas short narrative-style prompts can facilitate it.The study found that narrative-style statements with long context are more challenging for LLMs and may be associated with larger language bias.Source: §9 · p. 49
• Future work will systematically analyze failure-case mechanisms and investigate dialogue-based code generation with iterative prompt refinement across languages.Source: §9 · p. 49