PaperlayerView exact revision on arXiv

Source-linked AI summary

A Study on the Impact of Natural Language Differences in Prompts on Automatic Code Generation Using LLMs

Haruka Tokumasu, Masanari Kondo, Alexander Serebrenik, Dong Wang, Kei Koyanagi, Kotaro Noguchi, Naoyasu Ubayashi, Yasutaka Kamei

arXiv:2609.18311v1cs.SE

TL;DR

Natural-language differences in code-generation prompts were insufficiently studied despite evidence of language bias in LLMs. The study evaluates seven LLMs across three datasets and tests translation into better-performing languages. Accuracy varied by language, officially supported languages often performed best, and translation helped inconsistently across datasets and model types.

  • Problem

    Prior research had not comprehensively established how natural-language prompt differences affect LLM code-generation performance or how to mitigate that bias.

  • Method

    The study compares English, Japanese, and Chinese prompts across seven LLMs and three datasets, then evaluates translation into the best-performing language.

  • Results

    Natural language affected code-generation Accuracy, with officially supported languages frequently performing best; translation improved LeetCode and BigCodeBench but had mixed AtCoder effects.

  • Takeaways & Limitations

    Language bias should be evaluated across datasets and model types, because translation can mitigate it in some settings but is not universally effective.

  • Takeaways & Limitations

    Training-data contamination from publicly available multilingual solutions may influence observed performance differences across languages.

Abstract

from arXiv · show

Large Language Models (LLMs) have demonstrated remarkable performance in automatic code generation tasks, thereby encouraging new research in this area. Although numerous studies have explored LLM-based code generation, the impact of the natural language in input prompts remains unexplored (language bias). This study aims to (1) quantify how the natural language of input prompts influences LLM-based code generation performance and (2) evaluate a mitigation strategy to reduce language bias in code generation. We assess code generation Accuracy on AtCoder, LeetCode, and BigCodeBench. To quantify the language bias on code generation, each problem is presented in English, Japanese, and Chinese. We use seven LLMs (GPT-4o, o3-mini, DeepSeek-V3.2, Llama-3, Qwen2.5-Coder-14B, Qwen2.5-Coder-0.5B, and GitHub Copilot) and assess their performance in terms of Accuracy (the number of problems for which generated code passes all test cases). We compare Accuracy before and after translation to evaluate the effectiveness of translation as a mitigation strategy. We observed that the natural language of problem statements affects LLM-based code generation performance. Specifically, the languages officially supported by each dataset achieved the highest median Accuracy. Also, translation improved Accuracy, but its effectiveness was not consistent across datasets and model types. We found that AtCoder contained a particularly high proportion of narrative-style problem statements and longer problem statements. Natural language significantly affects LLM code generation accuracy. Translation can mitigate language bias in some settings, but its effectiveness depends on the dataset and model type. Furthermore, the narrative aspects and context length of input prompts are important factors related to language bias and the effectiveness of translation as a mitigation strategy.

1 Introduction

This study examines how the natural language of problem statements affects LLM code-generation accuracy and whether translation can mitigate that bias. Across datasets, models, and complementary analyses, language effects varied, officially supported languages often performed best, and narrative structure and length emerged as relevant factors.

  • Scope and contributions: The study extends prior AtCoder-only, GitHub Copilot-only evidence with additional datasets, recent LLMs, low-resource languages, and C++ experiments.These complementary analyses were designed to examine generalizability across programming and linguistic conditions.
  • Research questions and contributions: The study evaluates translation into the best-performing language as a mitigation strategy for language bias in code generation.The strategy was motivated by prior evidence that translating prompts can improve LLM performance in natural-language tasks.
  • Research questions and contributions: Natural-language language gaps appeared across all three datasets, with officially supported languages frequently achieving the highest median Accuracy.The study expanded prior work across seven LLMs and AtCoder, LeetCode, and BigCodeBench.
  • Research questions and contributions: Translation improved Accuracy on LeetCode and BigCodeBench but had mixed effects on AtCoder and depended on model type.For AtCoder, open-source models improved by a median +3.2% after English translation, whereas closed-source models declined by a median -5.3%.
  • Complementary analyses: AtCoder contained 74.0% narrative-style problems versus 21.2% for LeetCode, and its median statement length was more than twice as long.Narrative-style statements were defined using a concrete subject, a contextualized objective, and sentences not directly required to solve the task.

2 Related Work

Prior work established biases related to natural and programming languages, but this study broadens code-generation evaluation across datasets, languages, models, and translation strategies. Its novelty is measuring natural-language effects in code-generation prompts rather than only comparing models, platforms, or programming languages.

  • Research gap and novelty: This study targets natural-language effects in problem statements, rather than only comparing programming languages, platforms, or difficulty levels.That focus distinguishes it from prior code-generation benchmark studies.
  • Natural-language bias: Prior evidence showed that LLM performance varies across natural languages, including lower performance for some low-resource languages and practical difficulties for Spanish-speaking developers.These limitations can affect developer efficiency, trust, and acceptance of generated suggestions.
  • Mitigation strategy: The paper evaluates translation as a mitigation strategy because prior work showed that translating prompts can improve performance in natural-language tasks.Its broader empirical analysis tests whether language bias and translation effects generalize across datasets and LLMs.
  • Related approaches: Related studies examined multilingual prompting, prompt translation, language ranking, multilingual model training, and realistic perturbations of code-generation prompts.These approaches include translated MBPP prompts, emphasized key phrases, and lexical, syntactic, and semantic perturbations.
  • Scope expansion: The study broadens an earlier AtCoder and GitHub Copilot analysis to include LeetCode, BigCodeBench, and additional LLMs.LeetCode provides another multilingual contest platform, while BigCodeBench extends evaluation to specification-based code generation tasks.

3 Study Design

The study evaluates language bias in code generation across three datasets using English, Chinese, and Japanese prompts. It combines contest and specification-based tasks, with dataset-specific sampling and test-case preparation.

  • The study targets English, Chinese, and Japanese, selected as high-resource languages that support prompt-quality verification and replication of prior work.
  • The evaluation covers AtCoder and LeetCode contest problems alongside BigCodeBench specification-based code-generation tasks.AtCoder and LeetCode provide structured multilingual statements and test cases, while BigCodeBench adds context-rich tasks beyond competitive programming.
  • AtCoder contributes 756 questions from contests 99–287, evenly divided across difficulty levels A through D.Each difficulty level contributes 189 questions.
  • LeetCode contributes 100 problems—16 easy, 52 medium, and 32 hard—with 15 manually prepared non-duplicated input-output test cases per problem.The subset was chosen because preparing test cases for all 3,313 problems required substantial manual effort.
  • The evaluation uses newly generated problems so measured performance is not biased by LLM training-data leakage.The passage states that these problems remedy the data-leaking problem.

4 Influence of Problem Statement Language

Problem-statement language affects LLM code-generation Accuracy, with officially supported languages generally performing better; the magnitude varies across datasets and model types.

  • 4.2 Results: DeepSeek-V3.2 achieved the highest median Accuracy in 17 of 24 dataset, difficulty-level, and language combinations.Copilot was excluded from subsequent comparisons because its manual, unspecified-model evaluation was not consistently reproducible.
  • 4.2 Results: Officially supported languages generally achieved higher Accuracy than unsupported languages across AtCoder, LeetCode, and BigCodeBench.AtCoder favored English over Chinese, LeetCode favored English and Chinese over Japanese, and BigCodeBench favored English.
  • 4.2 Results: Median language gaps were 6.6%, 6.3%, and 3.6% for open-source models versus 2.7%, 15.6%, and 11.5% for closed-source models on AtCoder, LeetCode, and BigCodeBench.Bias was largest on AtCoder for open-source models, but on LeetCode and BigCodeBench for closed-source models.
  • 4.2 Results: Japanese tended to perform best with o3-mini, whereas English and Chinese tended to perform best with DeepSeek-V3.2.English and Chinese each achieved the highest Accuracy with DeepSeek-V3.2 in three of eight cases, or 37.5%.
  • 4.2 Results: Natural language had limited effects on code quality measures: warning rates and median lines of code were consistent across languages, while cyclomatic complexity mainly reflected problem difficulty and model capacity.Pylint warning-rate differences were all below 0.101, and the observed LOC differences were negligible.

5 Translation for Performance Improvement

Translation can improve code-generation Accuracy, but its effect varies substantially by dataset, model group, target language, and translation tool.

  • 5 Translation for Performance Improvement: Translation improved Accuracy for LeetCode and BigCodeBench, while its effect on AtCoder varied across model groups and target languages.The study compares translated and original prompts across English, Japanese, and Chinese targets.
  • 5 Translation for Performance Improvement: For AtCoder, translating into English produced a median decrease of -5.3% for closed-source models but a median increase of +3.2% for open-source models.Translation into Japanese decreased performance for both groups, with median changes of -10.3% and -4.3%, respectively.
  • 5 Translation for Performance Improvement: For LeetCode, translation into English increased median Accuracy by +6.2% for closed-source and +6.3% for open-source models, while Chinese translation increased it by +1.9% and +6.9%.The corresponding improvement rates were high for both target languages and model groups.
  • 5 Translation for Performance Improvement: For BigCodeBench, closed-source models had a 72.2% improvement rate with a median increase of +2.4%, while open-source models had a 55.6% improvement rate and +0.2% median change.Translation often improved individual cases despite limited median gains.
  • 5 Translation for Performance Improvement: Translation tools produced similar performance trends, and their average BLEU scores ranged from 0.501 to 0.564.The reported tools were Google, DeepL, and GPT.

6.1 Influence of Problem Statement Narrative

The study defines narrative-style problem statements using contextual and irrelevant-text criteria, then examines whether narrative structure relates to code-generation Accuracy.

  • 6.1 Influence of Problem Statement Narrative: Narrative-style statements satisfy at least one criterion involving specific subjects, contextualized objectives, or sentences unnecessary for solving the problem.The criteria treat removing contextual elements as decontextualization and abstraction.
  • 6.1 Influence of Problem Statement Narrative: The authors manually reviewed 50 AtCoder D-level problems and all 32 LeetCode Hard-level problems, labeling each narrative criterion.They measured inter-rater agreement with Cohen’s kappa.
  • 6.1 Influence of Problem Statement Narrative: A problem was classified as narrative if it met at least one criterion; otherwise, it was classified as expository, and Accuracy was calculated for both groups.The analysis compares narrative and expository problem statements after translation.
  • 6.1 Influence of Problem Statement Narrative: The example narrative problem contains named entities, a contextualized objective, and an unrelated introductory sentence.These elements contextualize the computational task beyond an abstract numerical target.

Observation 14) AtCoder has a relatively large proportion of narrative problem

AtCoder contains more narrative and longer problem statements than LeetCode, and narrative effects on Accuracy depend on context length and dataset.

  • Observation 14) AtCoder has a relatively large proportion of narrative problem: AtCoder problems are 74.0% narrative and 26.0% expository, whereas LeetCode problems are evenly divided between the two categories.AtCoder narrative problems also more often satisfy multiple narrative criteria.
  • Observation 14) AtCoder has a relatively large proportion of narrative problem: Narrative AtCoder problems had lower Accuracy than expository problems, with median differences of -4.1% in English and -9.6% in Japanese, or -6.9% overall after translation.Narrative problems therefore remained more difficult in the AtCoder analysis.
  • Observation 14) AtCoder has a relatively large proportion of narrative problem: AtCoder problem statements have longer contexts than LeetCode statements, with AtCoder’s sentence counts exceeding LeetCode’s by more than twice in the reported comparison.The study relates longer contexts to greater difficulty in understanding problem statements.
  • Observation 14) AtCoder has a relatively large proportion of narrative problem: Among 50 sampled AtCoder narrative failures generated by DeepSeek, 20 were judged to involve narrative-related factors.The annotators’ Cohen’s kappa was 0.623, and examples appear in Figure 7.
  • Observation 14) AtCoder has a relatively large proportion of narrative problem: Narrative aspects lowered Accuracy for longer AtCoder statements but improved Accuracy for shorter narrative-style LeetCode statements.The authors interpret this pattern as context-length dependent rather than uniformly harmful.

6.2 Applicability and Generalizability

Additional experiments test whether the findings extend across C++ and low-resource natural languages, using the AtCoder dataset and consistent evaluation conditions.

  • 6.2 Applicability and Generalizability: The extension adds C++ code generation and prompts in Thai, Slovak, Finnish, Bulgarian, and Norwegian Bokmål.The main analysis used Python and English, Japanese, and Chinese.
  • 6.2 Applicability and Generalizability: The additional experiments keep problem instances, model configurations, prompt formats, and test-case evaluation consistent with the main experiments.This design attributes observed differences to programming-language or natural-language variation.
  • 6.2 Applicability and Generalizability: The extension used only AtCoder because preparing reliable multilingual LeetCode and BigCodeBench data was impractical under the experimental constraints.This limits direct generalization of the additional analysis to those datasets.
  • 6.2 Applicability and Generalizability: DeepSeek-V3.2 achieved the highest count of C++ Accuracy results, leading in 9 settings and particularly dominating the lower difficulty levels A and B.o3-mini achieved the highest Accuracy in several higher-difficulty settings, especially D.
  • 6.2 Applicability and Generalizability: Language bias persisted among low-resource languages, with consistent differences across models and difficulty levels.Slovak and Finnish often outperformed Bulgarian, while tokenizer compatibility was offered only as a hypothesis requiring further investigation.

Observation 21) Model capability does not eliminate language bias in low-resource

Higher model capability does not remove language bias: performance differences persist across languages, including for high-performing models and in C++ experiments.

  • Language bias persists even for high-performing models such as DeepSeek and o3, with degradation in certain low-resource languages.The variance across languages widens as task difficulty increases, so capability alone is insufficient to eliminate the bias.
  • DeepSeek-V3.2 achieves the best overall C++ performance, while AtCoder’s officially supported languages generally produce higher Accuracy.Chinese can nevertheless achieve Accuracy comparable to Japanese in some settings despite not being officially supported by AtCoder.

7 Discussion

The discussion links language bias and translation effectiveness to platform support, task difficulty, narrative structure, and context length. It also proposes context compression and structured representations as directions for handling longer prompts.

  • Officially supported prompt languages generally yield higher Accuracy, although unsupported languages can perform well when problem context is simple.For difficult problems, officially provided languages more often perform better because complex contexts require deeper understanding.
  • AtCoder problems contain more narrative content and longer context than LeetCode, while narrative style lowers AtCoder performance but improves LeetCode performance.These dataset differences qualify how narrative framing relates to code generation.
  • Translation improves LeetCode code generation but has mixed or limited effectiveness elsewhere, depending on dataset, model type, context length, and narrative content.Long contexts with many narrative aspects may reduce translation effectiveness.
  • Longer-context prompts do not benefit from translation in the same way as shorter prompts, motivating methods that appropriately handle extended context.The study identifies longer-context handling as an unresolved practical boundary.
  • Future work could compress prompts by removing unrelated sentences or paraphrasing problem statements, or structure information to help LLMs process longer context efficiently.The proposed approaches aim to reduce text length or preserve information while improving long-context processing.
  • Multilingual studies should quantify how context length and narrative aspects affect code generation and translation efficacy so developers can choose task-appropriate context.The discussion recommends evaluating translation in the target dataset and model setting before practical deployment.

8 Threats to Validity

The study’s validity is bounded by its model, language, programming-language, dataset, translation, test-case, randomness, and training-data choices. These factors constrain how broadly its findings should be interpreted.

  • Low-resource-language translations lacked native-speaker validation, potentially affecting translation quality and interpretation of those results.The authors explicitly identify this as a concern caused by the absence of native speakers among the authors.
  • The experiments primarily generate Python code, with C++ validation conducted only on AtCoder, so results may vary across programming languages and datasets.The authors recommend broader cross-language evaluation.
  • Training-data contamination may influence observed cross-language differences because public multilingual solutions could be included in model training.Translation controls semantic content partially but cannot eliminate this threat.
  • Direct translation may omit code-generation-relevant information, and machine-translated statements require language-specific paraphrasing or manual verification.The authors recommend human translators or manual checking for necessary context.
  • The three evaluated datasets do not represent the full landscape of code-generation tasks, limiting generalizability beyond AtCoder, LeetCode, and BigCodeBench.Additional datasets are needed for broader validation.
  • The seven-model evaluation excludes many available LLMs, so adding more models would provide a more comprehensive assessment.The study includes GPT-4o, o3-mini, DeepSeek-V3.2, GitHub Copilot, Llama3, and two Qwen coder models.

9 Conclusion

This study measures language bias in code generation across three datasets and seven LLMs, then evaluates translation as a mitigation strategy. Performance varies by language, translation helps selectively, and prompt narrative structure and length matter.

  • Across seven LLMs and three datasets, code generation performance varies by prompt language, with officially supported languages frequently achieving the highest performance.The supported languages are English and Japanese for AtCoder, English and Chinese for LeetCode, and English for BigCodeBench.
  • Translation can improve performance depending on the dataset and model type, including median LeetCode gains of +6.2% for closed-source and +6.3% for open-source models after translating Japanese prompts into English.The reported gains are dataset- and model-specific rather than universal.
  • Users should select prompt languages with ample data sources and evaluate translation in the target dataset and model setting before practical use.The conclusion treats language choice and translation as setting-dependent decisions.
  • Long-context narrative prompts can hinder code generation, whereas short narrative prompts can facilitate it.This contrast identifies context length as a key condition on the effect of narrative style.
  • Future work should examine failure mechanisms and dialogue-based code generation with iterative multilingual prompt refinement.The proposed direction extends the analysis beyond single-turn prompt evaluation.
Loading 2609.18311v1…