PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languages

Libo Qin, Qiguang Chen, Fuxuan Wei, Shijue Huang, Wanxiang Che

arXiv:2310.14799v1cs.CLcs.AI

TL;DR

Zero-shot CoT is effective but remains limited in cross-lingual settings, where transferability and alignment are insufficient. The paper proposes CLP, combining cross-lingual alignment and task-specific solver prompting, plus CLSP for cross-language reasoning-path ensembling. Across several benchmarks, CLP and CLSP improve cross-lingual CoT and achieve state-of-the-art performance, with CLP gaining over 1.8% in AVG accuracy.

  • Problem

    Zero-shot CoT has limited ability to generalize across languages, while cross-lingual transfer and alignment remain under-addressed.

  • Method

    CLP aligns representations across languages and then uses task-specific solver prompting to generate final reasoning paths, while CLSP ensembles reasoning paths across languages.

  • Results

    Over 1.8% gain in AVG accuracy is reported for CLP over all baselines, while CLSP further enhances performance and both achieve state-of-the-art results.

  • Takeaways & Limitations

    Cross-lingual prompting provides a supported approach for improving zero-shot CoT reasoning across languages and leveraging knowledge from multiple languages.

  • Takeaways & Limitations

    Performance varies substantially across prompts, with differences exceeding 4%, leaving model-alignment robustness as an open issue.

Abstract

from arXiv · show

Chain-of-thought (CoT) is capable of eliciting models to explicitly generate reasoning paths, thus promoting reasoning accuracy and attracting increasing attention. Specifically, zero-shot CoT achieves remarkable improvements in a wide range of reasoning tasks by simply instructing the LLM with the prompt "Let's think step by step!". Despite the success of zero-shot CoT, the existing zero-shot prompting techniques remain limited to a single language, making it challenging to generalize to other languages and hindering global development. In this work, we introduce cross-lingual prompting (CLP), aiming to improve zero-shot CoT reasoning across languages. Specifically, CLP consists of two main components: (1) cross-lingual alignment prompting and (2) task-specific solver prompting. The cross-lingual alignment prompting is responsible for aligning representations across different languages, whereas the task-specific solver prompting is used to generate the final chain of thoughts and results for the reasoning task. In addition, we further introduce cross-lingual self-consistent prompting (CLSP) to ensemble different reasoning paths across languages. Our experimental evaluations on several benchmarks demonstrate that CLP and CLSP significantly outperform the existing prompting methods and achieve state-of-the-art performance. We hope this work will inspire further breakthroughs in cross-lingual CoT.

1 Introduction

Zero-shot CoT improves reasoning by prompting models to think step by step, but traditional prompting struggles to generalize across languages. The paper introduces CLP and CLSP, which improve cross-lingual reasoning and achieve state-of-the-art results.

  • Zero-shot CoT elicits explicit reasoning paths with the prompt “Let’s think step by step!” across several reasoning tasks.The cited example includes arithmetic reasoning and illustrates a traditional English CoT.
  • Traditional zero-shot CoT has limited cross-lingual generalization, motivating methods that support reasoning across languages.The paper emphasizes the need to generalize CoT across more than 7,000 languages and notes limited attention to zero-shot cross-lingual CoT.
  • CLP combines cross-lingual alignment prompting with task-specific solver prompting to improve zero-shot CoT reasoning across languages.The alignment stage aligns representations, while the solver stage generates the final reasoning path and result.
  • CLSP ensembles reasoning paths across languages to integrate different views of the reasoning process.This extends CLP by applying a cross-lingual form of self-consistency.
  • Over 1.8% improvement on AVG accuracy is reported for CLP and CLSP over existing prompting methods, with state-of-the-art performance.The evaluations cover several benchmarks.

2 Background

Traditional CoT prompts a model to generate reasoning step by step in one language, whereas cross-lingual CoT handles requests in any language and generates reasoning in English. The latter addresses traditional CoT’s limited cross-lingual transferability.

  • 2.1 Traditional Chain-of-Thought: Traditional CoT appends “Let’s think step by step!” to a request so the model outputs specific reasoning paths.
  • 2.2 Cross-lingual Chain-of-Thought: Traditional CoT is limited to generating reasoning within a single language and lacks effective cross-lingual transferability.
  • 2.2 Cross-lingual Chain-of-Thought: Cross-lingual CoT enables models to process requests in any language while generating the chain of thought in English.The target output language is specified as English in the cited background passage.

3 Cross-lingual Prompting

CLP aligns information between source and target languages before task-specific solving, while CLSP extends the framework by combining reasoning paths across languages. The solver stage produces target-language reasoning and formatted answers, with MGSM comparisons defining the evaluated prompting baselines.

  • 3 Cross-lingual Prompting: CLP combines cross-lingual alignment prompting with task-specific solver prompting to elicit cross-lingual reasoning.The alignment stage addresses cross-lingual transfer, and the solver stage follows after alignment.
  • 3.1 Step 1: Cross-lingual Alignment Prompting: Cross-lingual alignment prompting asks the model to understand a source-language request step-by-step in a target language.The framework simulates multilingual comprehension and represents intermediate semantic alignments across S alignment steps.
  • 3.1 Step 1: Cross-lingual Alignment Prompting: The alignment response A is selected by maximizing its probability conditioned on the input and source- and target-language settings.A denotes the alignment response produced in step 1.
  • 3.2 Step 2: Task-specific Solver Prompting: Task-specific solver prompting uses the target language and alignment text to resolve the target task through a multi-step reasoning path.The prompt asks the model to act as an expert in the target task and resolve the understood task step-by-step.
  • 3.2 Step 2: Task-specific Solver Prompting: The solver organizes potential reasoning paths into a final target-language reasoning path using the dialogue history, input variables, languages, and alignment response.The dialogue history C includes X, Ls, Lt, and A.
  • 3.2 Step 2: Task-specific Solver Prompting: The framework formats the final response as ‘Answer: [num]’ after extracting the answer text from potential reasoning results.The answer text Ft is generated from the potential reasoning results f.
  • 3.2 Step 2: Task-specific Solver Prompting: CLP and CLSP are evaluated on MGSM against Direct, Native-CoT, En-CoT, and Translate-En prompting settings.The table caption defines these settings and identifies Translate-En as Google-based translation of non-English requests.

4 Cross-lingual Self-consistent Prompting

CLSP addresses variation in language-specific reasoning by generating reasoning across target languages and combining the resulting answers. It retains answers that are most consistent through voting.

  • 4 Cross-lingual Self-consistent Prompting: CLSP integrates reasoning knowledge across different languages because LLMs show varying reasoning patterns across languages.The method is presented as an extension inspired by self-consistency prompting.
  • 4 Cross-lingual Self-consistent Prompting: For each reasoning step, CLSP generates alignment responses and corresponding reasoning steps in different target languages.The procedure produces potential reasoning results across the selected target languages.
  • 4 Cross-lingual Self-consistent Prompting: CLSP uses voting to retain highly consistent answers across inferred reasoning results as the final result.The formulation accounts for the number of target languages and potential reasoning results, with a 0-1 consistency function.

5 Experiments

Experiments show that CLP and CLSP improve cross-lingual reasoning accuracy, reasoning quality, benchmark transfer, and performance across model sizes and prompting settings. The results also indicate that language integration depends on resource availability rather than language count alone.

  • Main Results: CLP surpasses all previous baselines, outperforming PaLM-540B(Translate-En) by 16.4% and achieving 2.2% higher average accuracy than Translate-En.The authors interpret this as evidence that CLP improves cross-lingual understanding beyond simple translation.
  • Main Results: CLSP exceeds CLP across all languages by 6.1% in average accuracy, supporting cross-lingual integration of reasoning knowledge.CLSP further improves cross-lingual chain-of-thought reasoning by combining reasoning paths across languages.
  • CLP Results Better Reasoning Quality: CLP reasoning paths improve faithfulness by 1.6%, Informativeness by 2.8% for Step and 2.5% for Chain, and Miss-step by 2.8%.These changes indicate more consistent, informative, and complete reasoning chains.
  • Two-stage Interactive Prompting: Two-stage interactive prompting improves average performance by 10.4% over single-turn prompting.The authors suggest that interactive prompts better elicit the dialogue interaction abilities of language models.
  • Generalization: CLP improves accuracy by 3.1% on XNLI and 4.5% on PAWS-X relative to En-CoT, and by at least 6.8% on XCOPA with smaller LLMs.These results support generalization across multilingual reasoning benchmarks and model sizes.
  • Language Integration: Language integration improves performance in high-resource settings above 4%, but performance decreases as more low-resource languages are added.The results identify pretraining-data quantity as an important consideration when integrating languages.

6 Related Work

Prior work established CoT and cross-lingual learning as useful directions, but zero-shot cross-lingual alignment remained underexplored. This work addresses that gap with CLP and CLSP.

  • CoT prompts elicit intermediate reasoning steps for zero-shot and few-shot reasoning in large language models.
  • Cross-lingual learning has been studied in downstream tasks, while recent work increasingly examines nontraining-based approaches with large language models.
  • Few-shot examples are commonly used to guide cross-lingual alignment in existing research.
  • CLP explores zero-shot cross-lingual alignment CoT without constructing additional examples, and CLSP combines cross-lingual reasoning pathways to enhance performance.

7 Conclusion

The paper introduces CLP for cross-lingual CoT and adds CLSP to leverage knowledge across languages. Experiments report promising performance for both methods, with CLSP further improving over CLP.

  • CLP aligns representations across languages and generates final reasoning paths through cross-lingual alignment and task-specific solver prompting.
  • CLSP leverages knowledge across languages and further boosts performance over CLP.
  • Extensive experiments show promising cross-lingual CoT performance for both CLP and CLSP.

Limitations

CLP's reasoning improvements vary substantially across prompt formulations, indicating that robustness to prompt expression remains unresolved.

  • Prompt formulations produce performance differences exceeding 4%, despite all tested prompts improving reasoning performance.
  • The authors identify robust model alignment across prompt expressions as an important future research issue.

A.1 Robust Analysis Implementation

The robustness analysis generates synonymous guiding prompts and reports their corresponding results in Table 4.

  • GPT3.5 generates three synonymous guiding prompts to test CLP robustness across different expressions.
  • The prompt-generation instruction asks for three rewritten versions of a target-language step-by-step request.
  • The generated prompts and their corresponding results are presented in Table 4.

A.2 Chain-of-Thought Quality Scoring Implementation

The implementation evaluates chain-of-thought quality through alignment-based metrics for faithfulness, informativeness, missing steps, and reasoning alignment. It also extends these assessments cross-lingually and analyzes CLP strategies, few-shot use, and strategy diversity.

  • Faithfulness: Faithfulness averages alignment between hypothesis reasoning steps and source sentences to assess whether reasoning misinterprets or misuses information.Lower faithfulness can indicate vagueness, irrelevance, or misuse of information.
  • Informativeness: Info-Step measures how strongly reasoning steps use information from the source, assigning lower scores when steps overlook or diverge from it.The metric evaluates source utilization at the reasoning-step level.
  • Missing Step: Miss-Step compares reference reasoning steps with the hypothesis to verify whether each reference step has a corresponding hypothesis step.It is intended to identify significant steps missing from the generated reasoning chain.
  • Cross-lingual scoring: Cross-lingual evaluation expands ROSCOE beyond monolingual English using multilingual MP-Net for sentence-similarity computation.The extension targets assessment of Cross-lingual CoT inference quality.
  • Reasoning Alignment Scoring: RAS measures overlap between hypothesis and reference reasoning, and experiments calculate it only on correctly predicted samples to avoid penalizing incorrect reasoning.Figure 7 varies output temperature across seven generated reasoning results and averages RAS across samples.
  • CLP strategy analysis: CLP uses multiple alignment strategies, while few-shot experiments find that two-stage ICL improves performance and increasing strategy diversity gradually improves downstream results.The analyzed strategies include step-by-step translation, key-information extraction, preliminary solution, complete translation, problem restatement, step division, code-switching, and denial of service.
Loading 2310.14799v1…