PaperlayerView exact revision on arXiv

Source-linked AI summary

Investigating the Influence of Prompt and Response Languages on LLM Content Generation

Thi Thanh Nhan Nguyen, Mai Khoi Tieu, Michael A. Riegler, Pål Halvorsen, Thu Nguyen

arXiv:2608.26186v1cs.CLcs.AI

TL;DR

The paper asks whether prompt and response language alter LLM generation beyond syntax. It evaluates five models across four English–Norwegian conditions using length, semantic, density, and lexical-overlap measures. Prompt language strongly shapes response length, while semantic content remains highly similar despite different surface vocabulary.

  • Problem

    The study asks whether language choice changes LLM information presentation and retrieval strategies, not merely output syntax.

  • Method

    Five models answer 68 non-translation questions under four English–Norwegian prompt×response conditions, analyzed with complementary structural, semantic, density, and lexical metrics.

  • Results

    Prompt language systematically changes response length and lexical realization; English prompt → Norwegian response is about 52% shorter, while semantic similarity remains approximately 0.83 and soft Jaccard is approximately 0.53 versus raw Jaccard approximately 0.02.

  • Takeaways & Limitations

    Prompt and response languages are not neutral settings: they jointly shape output length and surface vocabulary while largely preserving semantic content.

  • Takeaways & Limitations

    The study is limited to English–Norwegian Bokmål, five models, 68 questions, and capped generations, so broader generalization and truncation effects remain unresolved.

Abstract

from arXiv · show

This study examines how prompt and response language influence the behavior of large language models. Using five models, we evaluated answers to 68 non translation questions across four language conditions: English to English, English to Norwegian, Norwegian to Norwegian, and Norwegian to English. After removing refused items, the dataset contains 1348 responses. We measure length differences with Cohen d, semantic fidelity with LabSE cosine similarity, and cross lingual keyword overlap with both raw and soft Jaccard. Prompt language has a strong effect on response length. With English output, Norwegian prompts shorten responses by about thirty seven percent. With Norwegian output, English prompts shorten responses by about forty one percent. The largest cross lingual contrast shows a reduction in word count but a smaller reduction in tokens, reflecting tokenizer differences. Despite variation in length, semantic similarity remains high, and soft Jaccard reveals substantial conceptual overlap that raw Jaccard does not capture. Effect sizes vary across models, indicating heterogeneity. Prompt language is not neutral and systematically shapes output length and lexical realization, with implications for multilingual prompt design.

1. Introduction

This study tests whether prompt and response language alter LLM generation beyond syntax, using a four-condition, five-model evaluation and complementary structural and semantic measures.

  • Five models answer identical questions across four prompt×response language permutations spanning ethics, culture, health, and social topics.
  • The study measures translation compression with Cohen’s d, semantic fidelity with LaBSE cosine similarity, information density, and lexical divergence with Jaccard overlap.
  • The 2 × 2 design identifies translation compression as a prompt×response interaction rather than a uniform cross-lingual effect.
  • English prompt → Norwegian response is about 52% shorter and roughly 2× denser than English-to-English generation.
  • Semantic content remains largely preserved at approximately 0.83 similarity, while cultural presentation shifts qualitatively with language choice.

2. Related Works

Related work frames multilingual prompting as a question of reasoning, cultural framing, and semantic evaluation, motivating metrics that distinguish meaning preservation from lexical reuse.

  • Prior research links English-centric training data to stronger English reasoning and proposes high-resource-language prompting for low-resource-language outputs.
  • Research on semantic compression suggests models can preserve meaning while shortening text, but prior studies examined explicit compression tasks rather than spontaneous cross-lingual compression.
  • Prompt language has been studied as a factor in cultural values and generation style, including concerns about Western bias in English-prompted generations.
  • This study finds systematic shifts in length and lexical realization but does not directly quantify cultural framing.
  • Embedding-based metrics such as LaBSE are favored over simple lexical metrics for assessing semantic fidelity when output lengths vary.
  • Combining semantic similarity with information density is intended to characterize the trade-off between rhetorical elaboration and informational compactness.

3. Methodology

The methodology compares five models on a balanced multilingual question set under four prompt×response language conditions, while auditing refusals, provenance, and truncation.

  • Questions cover ethics, culture, health, and social topics, with mixed provenance and fewer health items than other topics.
  • Each question is issued under four conditions defined by prompt language crossed with required response language.
  • The labels ee, en, nn, and ne are internal shorthand for the four language conditions, not ISO language codes.
  • The study queries five named models once per question using fixed or default decoding settings, with token caps varying by model.
  • Three items were refused by some or all models in specified Norwegian-prompt conditions, and incomplete model-question pairs were removed to produce a balanced dataset.

4. Evaluation Strategies

Evaluation combines length, semantic, density, and lexical-overlap measures to separate response compression from tokenizer effects, truncation, meaning loss, and surface-form change.

  • Response Length: Word, character, and cl100k_base token counts jointly measure response length because Norwegian compounding and tokenizer fertility can make word and token comparisons diverge.
  • Length Effects: Cohen’s d quantifies standardized length differences, while a mixed-effects model accounts for repeated questions and reports interaction significance and per-model heterogeneity.
  • Semantic Fidelity: LaBSE cosine similarity compares responses in a shared multilingual embedding space, with chunk averaging checking robustness for long inputs.
  • Information Density: Information density is the number of TF-IDF keywords per 100 words, with keywords capped at 30 and extracted using sentence-level unfiltered unigram–bigram TF-IDF.
  • Information Density: Because keyword counts saturate near the cap, density is a near-perfect inverse of length and should be interpreted descriptively rather than as an independent quality measure.
  • Cross-lingual Keyword Overlap: Raw Jaccard measures exact top-k keyword overlap, whereas soft Jaccard matches keywords through LaBSE embeddings above cosine threshold τ = 0.70.

5. Results and Major Remarks

Across models and topics, language conditions systematically reshape response length and lexical realization, while semantic content remains highly aligned. Cross-lingual responses are shorter in words but show substantial concept-level overlap, with tokenizer differences and model heterogeneity qualifying the compression effect.

  • Word count analysis: 52% fewer words, 54% fewer characters, but only 25% fewer tokens in the ee →en comparison.Mean counts changed from 427.8 →203.8 words, 3072.9 →1419.6 characters, and 614.1 →459.9 tokens under cl100k_base.
  • Word count analysis: d ≈ 1.32 for the pooled ee–en length gap, with per-model values ranging from 0.78 to 4.42.Phi-4 had d = 0.78, Claude d = 1.52, and GPT-4o d = 4.42; the pooled effect was driven primarily by more verbose models.
  • Word count analysis: Norwegian-output conditions were generally more concise in words, with ee usually longest and en usually shortest across models and domains.Figure 2 summarizes the distributional pattern on the balanced dataset, with n = 337 per condition cell.
  • Prompt and response language effects: A large prompt×response interaction of +300.7 words shows that response-language effects reverse across prompt languages.Under English prompts, English responses were about 224 words longer than Norwegian responses; under Norwegian prompts, English responses were about 76 words shorter.
  • Semantic similarity and keyword overlap: LaBSE cosine similarity remained roughly 0.82–0.86, while ee↔en raw Jaccard averaged 0.017 versus 0.530 for soft Jaccard.The raw–soft gap supports conceptual paraphrasing across languages, and chunking produced nearly identical semantic-similarity means.
  • Information density: Information density was highest for en at 20.16 and closely inversely tracked response length rather than constituting an independent quality gain.Mean density followed en 20.16, ne 14.30, nn 12.60, and ee 9.67; the relation with inverse word count was near-perfect.

6. Discussion

The discussion finds that language choice changes response length and lexical realization while largely preserving semantic content. However, tokenizer fertility, truncation, refusals, experimental design choices, and model heterogeneity constrain interpretation and generalization.

  • Core findings: A large prompt×response interaction reverses across cells, with English prompts producing much longer English than Norwegian responses.Norwegian prompts produce slightly shorter English than Norwegian prompts.
  • Core findings: Per-model Cohen’s d ranges from 0.78 for Phi-4 to 4.42 for GPT-4o, showing that pooled compression is heterogeneous.The pooled estimate is dominated by more verbose models.
  • Core findings: ≈0.83 cosine similarity coexists with ≈0.02 raw Jaccard and ≈0.53 LaBSE soft Jaccard, indicating preserved concepts despite rearranged surface vocabulary.The similarity remains unchanged under chunk-averaged embedding.
  • Candidate mechanisms and limitations: Tokenizer fertility and truncation complicate the 52% ee–en word-count gap, because Norwegian uses more subword tokens and more often reaches the token cap.Mean nn responses reach 804 tokens versus 614 for ee; an uncapped rerun would be needed to separate effects.
  • Candidate mechanisms and limitations: The proposed RLHF and pretraining explanations remain speculative, while safety-conditioned refusals also limit naive cross-language comparisons.The authors state that these mechanisms are consistent with observations but are not established causally.
  • Candidate mechanisms and limitations: The study covers one language pair, five models, and 68 non-translation questions per model, leaving broader generalization unverified.Nynorsk, larger question sets, multi-turn settings, and systematic question-provenance effects were not tested.

7. Conclusion

The conclusion argues that prompt and response languages are not neutral: they jointly shape output length and lexical realization. Across robustness checks, semantic content remains stable while tokenizer effects and model heterogeneity qualify the compression result.

  • Conclusion: Prompt and response language jointly shape output length and lexical realization, while rhetorical framing is not quantified.The paper treats these as systematic effects of language choice rather than as purely syntactic differences.
  • Conclusion: A ∼52% ee→en word reduction versus ∼25% token reduction shows that Norwegian tokenizer fertility accounts for a substantial portion of apparent compression.The prompt×response interaction remains large and highly significant, with mixed-effects z = 25.6.
  • Conclusion: ≈0.83 cosine similarity, ≈0.02 raw Jaccard, and ≈0.53 LaBSE soft Jaccard support concept-level paraphrasing despite low surface overlap.The cosine result is unchanged under chunk-averaged embedding.
  • Conclusion: Per-model d ranges from 0.78 to 4.42, so the compression effect should be reported with model-specific breakdowns.The conclusion presents heterogeneity as a qualification on any single pooled number.

Appendix A. Lists of Experimental Questions

The appendix lists the single-turn questions used across ethics, culture, health, and social topics. It also documents the language-condition setup and design choices affecting comparability.

  • Experimental questions: The ethics set contains questions about moral concepts, ethical theories, dilemmas, technology, justice, leadership, and values.The listed items include genetic modification, environmental challenges, animal rights, and ethics without religion.
  • Experimental questions: The culture set asks about festivals, the Renaissance, clothing, storytelling, appropriation, food traditions, and tea ceremonies.These questions target cultural practices, identity, history, and cross-cultural variation.
  • Experimental questions: The health set includes questions about infant formula, fractures, headaches, stroke response, contraception, and related medical guidance.The appendix distinguishes these general health questions from the translation-task items documented separately.
  • Experimental questions: The social set covers social media, peer pressure, networks, diversity, empathy, stereotypes, inclusion, inequality, and technology.The questions address social interaction and societal behavior.
  • Experimental design: Each question was sent as one user turn under four prompt/output language permutations, with model-specific system-prompt differences.English and Norwegian question placeholders were used, and Gemini had no explicit system prompt.
  • Experimental design: The four conditions are not a clean language-only 2 × 2 because some rely on implicit language conventions while others add explicit cues.The cues themselves are written in different languages, introducing an additional design difference.

Appendix C. Translation-task health items (excluded from length analyses)

This appendix presents two health questions that explicitly request medical translation into Norwegian. Because these items over-determine the response language, they were excluded from the main analyses.

  • Exclusion rationale: Both translation items were excluded from length, density, similarity, and Jaccard analyses because their requested response language was over-determined.They were retained in the appendix for completeness.
  • Translation-task items: The first item requests a technically accurate Norwegian translation of a medical report describing aortic regurgitation and treatment.It includes an ejection fraction of 56%, ACE inhibitors, diuretic therapy, and renal-function monitoring.
  • Translation-task items: The second item requests an easy-to-understand Norwegian translation of a patient explanation about a heart valve, medicines, and kidney monitoring.The source explains backward blood flow, preserved pumping, vasodilation, fluid reduction, and follow-up testing.

Appendix D. Refusal and Truncation Audits

The audit documents refusals, truncations, and paired examples illustrating that language conditions affect both data availability and localized response content.

  • Refusal audit: 1,348 responses remained after removing 26 refusals and balancing the design by dropping 13 model-question pairs.All 26 refusals occurred in Norwegian-prompt conditions.
  • Truncation audit: 110 raw response files ended without terminal punctuation, with truncations concentrated in Norwegian-output conditions.Norwegian-output conditions accounted for 69% of truncations; Phi-4 and Claude contributed most, while DeepSeek-V3 had none except in nn.
  • Paired example: The paired infant-formula responses give broadly similar clinical guidance but recommend market-specific brands and regulators.The English response names US-market brands, whereas the Norwegian response names Norway-market brands.

Appendix F. Log-scale robustness check for the length model

A log(1 + words) mixed-effects model checks whether the length findings are robust to right-skewed word counts. The transformed analysis reproduces the raw-scale results, including direction, significance, and approximate magnitude.

  • Model specification: Log(1 + words) makes residuals more symmetric and fixed-effect coefficients interpretable as multiplicative raw-scale changes.The model uses prompt × response, model, and question-level random effects.
  • Robustness result: −21% is the back-transformed nn-cell effect relative to ee, compared with a direct-means value of −19%.The estimate combines the main effects and interaction on the log scale.
  • Robustness result: z = 23.01 for the log-scale interaction, comparable to z = 25.62 on the raw scale.Direction, significance, and magnitude of every effect match the raw-scale fit.
  • Reproducibility: The reproduction pipeline regenerates every number and figure in Sections 4–5 from the raw text responses.The analysis code and figure-generation scripts are publicly available on GitHub.
Loading 2608.26186v1…