PaperlayerView exact revision on arXiv

Source-linked AI summary

Eliciting Better Multilingual Structured Reasoning from LLMs through Code

Bryan Li, Tamer Alkhouli, Daniele Bonadiman, Nikolaos Pappas, Saab Mansour

arXiv:2403.02567v2cs.CLcs.AI

TL;DR

LLM reasoning research has largely focused on English or simpler tasks, leaving multilingual structured reasoning underexplored. This paper introduces xSTREET and code-based training and inference methods; the authors report improved performance across multilingual structured-reasoning tasks.

  • Problem

    Complex reasoning remains challenging for LLMs, while English-centric studies provide a limited assessment of their reasoning capabilities.

    Source: §1 · p. 1

  • Method

    The paper builds xSTREET and uses translated multilingual code comments for fine-tuning alongside code-structured prompts for inference.

    Source: §1 · pp. 1–2, §4.1 · p. 4

  • Results

    The authors report improved performance across multilingual structured-reasoning tasks, including for top-performing models, with the two techniques together achieving the best performance.

    Source: §1 · p. 2, §9 · p. 9

  • Takeaways & Limitations

    The fine-tuning results indicate that the code-and-reasoning hypothesis can apply to multilingual tasks.

    Source: §9 · p. 9

    Fine-tuning was limited to a 7B open-source model, so the recipe's effectiveness on 70B+ models remains undetermined.

    Source: Limitations · p. 9

Abstract

from arXiv · show

The development of large language models (LLM) has shown progress on reasoning, though studies have largely considered either English or simple reasoning tasks. To address this, we introduce a multilingual structured reasoning and explanation dataset, termed xSTREET, that covers four tasks across six languages. xSTREET exposes a gap in base LLM performance between English and non-English reasoning tasks. We then propose two methods to remedy this gap, building on the insight that LLMs trained on code are better reasoners. First, at training time, we augment a code dataset with multilingual comments using machine translation while keeping program code as-is. Second, at inference time, we bridge the gap between training and inference by employing a prompt structure that incorporates step-by-step code primitives to derive new facts and find a solution. Our methods show improved multilingual performance on xSTREET, most notably on the scientific commonsense reasoning subtask. Furthermore, the models show no regression on non-reasoning tasks, thus demonstrating our techniques maintain general-purpose abilities.

1 Introduction

Complex reasoning remains an open challenge for LLMs, and English-centric evaluations provide a limited assessment of their reasoning capabilities. The paper introduces xSTREET and code-based training and inference methods, reporting improved multilingual structured-reasoning performance.

1 Introduction

  • Prior complex-reasoning studies largely focus on English, limiting assessment of LLM reasoning capabilities across languages.The paper treats language as a surface-form representation of reasoning tasks.Source: §1 · p. 1
  • xSTREET covers four tasks across six languages and 5.5K entries, inheriting expert reasoning annotations with an average of 7.8 steps per answer.The benchmark extends English STREET to five additional languages and includes human post-editing of test translations.Source: §1 · pp. 1–2
  • The paper adapts code-based reasoning through translated multilingual comments for parameter-efficient fine-tuning and code-like prompts interleaving function calls with multilingual text.The prompt format is designed to bridge training and inference representations.Source: §1 · p. 2
  • Training-time and inference-time methods improve multilingual structured-reasoning performance, including for top-performing models; using them together achieves the best performance.The authors also report that the techniques preserve general-purpose language-model capabilities.Source: §1 · p. 2

2 Related Work

Related work motivates code as a structure for reasoning, surveys code prompts and multilingual reasoning benchmarks, and describes STREET and the source-code dataset used here.

2.1 Code & Reasoning Hypothesis for LLMs

2.2 Code Prompts for Complex Reasoning

  • Prior code prompts reformulate natural-language tasks into code, often requiring manual design; this work presents the first use of code prompts across multiple languages.Examples include structured commonsense, causal, and arithmetic reasoning prompts.Source: §2.2 · p. 3

2.3 Multilingual Reasoning for LLMs

  • MEGA covers 70 languages and 16 tasks but only simple reasoning; MGSM covers arithmetic in 10 languages, while xSTREET adds logic, science, and structured annotations (Ahuja et al., 2023; Shi et al., 2022).The cited MGSM work reports that English chain-of-thought outperforms native-language chain-of-thought on non-English questions.Source: §2.3 · p. 3

2.4 STREET Complex Reasoning Benchmark

  • STREET structures multipremise explanations as reasoning graphs and linearizes them for prompts, but the format is task-specific and diverges from LLM training data (Ribeiro et al., 2022).xSTREET omits STREET's SCONE task because its toy-world state tracking requires consideration beyond translation.Source: §2.4 · pp. 3–4
  • xSTREET omits STREET's SCONE task because its toy-world state tracking requires careful consideration beyond translation.Source: §2.4 · p. 3

2.5 Source Code Dataset

  • The source-code corpus is The Stack, a 3.1 TB permissively licensed dataset; this work uses 10,000 files each from Java, JavaScript, and Python.The selected files total 30,000.Source: §2.5 · p. 4

3 Multilingual Complex Reasoning Benchmark: xSTREET

xSTREET extends STREET from English into five additional languages, with human post-editing for its test split. Training and development translations use machine translation.

3 Multilingual Complex Reasoning Benchmark: xSTREET

  • Arabic, Spanish, Russian, Chinese, and Japanese add linguistic and script diversity and are languages used on many online programming-help websites.Source: §3 · p. 4
  • 914 English STREET entries become 5,484 xSTREET examples after expert translators post-edit test translations and the original reasoning-graph edges are reused.Translators post-edited machine-translated sentences with access to the full source entry for context.Source: §3 · p. 4
  • Machine-translated training and development splits were judged reasonable for training after native speakers reviewed 10 randomly sampled translations per language.Annotators noted some errors despite finding the translations suitable for training.Source: §3 · p. 4

4 Code with Multilingual Comments as Indirect Supervision for Reasoning

The training-time approach uses multilingual comments in code as indirect supervision for downstream reasoning, with a lightweight fine-tuning recipe.

4 Code with Multilingual Comments as Indirect Supervision for Reasoning

  • Multilingually commented code is used as indirect supervision for downstream reasoning through a lightweight fine-tuning recipe.Source: §4 · p. 4

4.1 Translated Code Comments Dataset (TCC)

  • TCC replaces source-code comments with translations, filtering for files with more than five comments and comments over 50% English.The filters reduce 30k source files to 20k; translations into five additional languages yield 120k files total.Source: §4.1 · p. 4

4.2 Train Time: fine-tuning on TCC

  • LoRA fine-tuning keeps base-model parameters frozen, while replaying 100k xP3 examples alongside TCC helps preserve original model capabilities.Source: §4.2 · p. 5

5 Multilingual Complex Reasoning as a Downstream Task

SIM code prompts use code structure to surface reasoning steps across task languages. They express structured facts and reasoning through function calls interwoven with multilingual text.

5 Multilingual Complex Reasoning as a Downstream Task

  • SIM code prompts aim to abstract away language-specific details so reasoning steps are more visible across languages.Source: §5 · p. 5
  • The single-prompt functions select facts, infer facts, check whether the problem is solved, and make choices; is_solved signals when generation should stop.make_choice is not used for non-multiple-choice tasks such as GSM8k.Source: §5 · p. 5
  • SIM-text returns selected facts as their text and works best for smaller models, while SIM-indexed references facts by array index and works best for larger models.The authors use each version for the model sizes where it performs best.Source: §5 · p. 5
  • A rule-based Python script converts structured graph annotations into SIM prompts that preserve the linearized format’s information without LLM-based conversion.The authors note that LLM-based conversion can introduce errors.Source: §5 · p. 5
  • SIM prompts list question facts in the task language and place new or selected facts as comments beside function calls.Source: §5 · p. 5

6 Experimental Setup

The experiments include BLOOMZ and BLOOMZ-TCC, with Falcon and GPT-3 evaluated only at inference time. Models receive few-shot prompts in several formats.

6 Experimental Setup

  • BLOOMZ (7.1B) is the base model, BLOOMZ-TCC its TCC-finetuned version, and Falcon (40B) and GPT-3 (175B) are used for inference-time experiments.Falcon is described as more performant than BLOOMZ on English tasks but limited in multilingual abilities.Source: §6 · p. 5
  • Experiments use up to five randomly sampled training exemplars, shared across models and prompt types, with greedy decoding and a 682-token generation limit.Source: §6 · p. 5
  • Figure 3 depicts direct, linearized, and SIM prompting formats, with inputs in grey boxes and expected outputs in white boxes.The SIM examples show two languages and color-code aligned facts.Source: Figure 3 · p. 6

7 Results

On ARC science commonsense reasoning, code prompts outperform linearized prompts across languages, while BLOOMZ-TCC still shows an English-to-non-English accuracy gap. BLOOMZ-TCC improves on BLOOMZ, but both models remain near chance on the other reported reasoning tasks.

7 Results

  • The results section reports xSTREET answer accuracy and directs readers to the appendix for results across tasks, languages, models, and prompt formats.The full results also include direct prompting.Source: §7 · p. 6

7.1 Results for BLOOMZ Models

  • Code prompts outperform linearized prompts across ARC languages; BLOOMZ-TCC accuracy falls from 76.2% in English to 61.1% averaged over non-English languages.Figure 4 compares linearized prompts (lines) and code prompts (dots); the average covers five non-English languages.Source: §7.1 · p. 6, Figure 4 · p. 7
  • BLOOMZ-TCC outperforms base BLOOMZ across languages and both prompt formats, with larger average non-English gains for code prompts (52.6 →61.1) than linearized prompts (33.5 →36.9).The authors suggest explicit separation of reasoning in code from multilingual understanding in comments may explain the larger gain.Source: §7.1 · p. 6

Results on GSM8K, AQUA_RAT, AR_LSAT

  • For GSM8K, AQUA_RAT, and AR_LSAT, BLOOMZ and BLOOMZ-TCC perform around random chance with or without the interventions.The authors hypothesize these arithmetic and logical reasoning tasks are too difficult for BLOOMZ-7B.Source: Results on GSM8K, AQUA_RAT, AR_LSAT · p. 6

Appendix §A.1.

The results compare code prompts and fine-tuning across reasoning and non-reasoning tasks, analyze when these methods help, and identify limits on the study’s scope and evaluation.

7.3 Non-Complex Reasoning Task Results

  • Differences between BLOOMZ and BLOOMZ-TCC were statistically insignificant on XNLI, XStoryCloze, and XQUAD.The authors attribute preserved natural-language abilities to LoRA and training-data replay.Source: §7.3 · p. 7

7.2 Results for Larger LLMs

  • 93.2→73.2 (linearized) versus 99.1→94.2 (SIM) on GPT-3 ARC; English remains higher than non-English, while SIM beats linearized on other tasks.For arithmetic and logical tasks, the relative gains from code prompts are larger in English; the authors note these tasks are more symbolic than ARC.Source: §7.2 · p. 7, Figure 5 · p. 8

7.4 Effect of Code Comments on Downstream Reasoning

  • On ARC, TCC ranks above TCC-en and TCC-del overall; diverse comments further boost performance across languages, with Russian an exception.Figure 6 compares these fine-tuning datasets using SIM prompts; BLOOMZ-TCC-en outperforms BLOOMZ-TCC in Russian, which the authors suggest may reflect missing Cyrillic tokenization in BLOOM.Source: §7.4 · pp. 7–8, Figure 6 · p. 8

8.1 Case Study on GSM8k English; 8.2 Case Study on ARC Arabic

  • In the examples, only GPT-3 with SIM answers the English GSM8k problem correctly, while GPT-3 with SIM also answers the Arabic ARC example correctly.BLOOMZ-TCC with linearized formatting performs two reasoning rounds on the GSM8k example but answers incorrectly; on Arabic ARC, SIM gives the correct answer after one fewer inference step than the gold.Source: §8.1 · p. 9, §8.2 · p. 9

9 Conclusion

  • The conclusion presents multilingual-comment code fine-tuning and code-structured inference prompts as methods that better elicit LLM reasoning abilities.The authors suggest multilingual comments during pre-training might amplify improvements, but state this as a possibility.Source: §9 · p. 9

Limitations

  • Fine-tuning was limited to a 7B model, and xSTREET uses machine-translated training data and few-shot exemplars; evaluation reports final-answer accuracy only.The study also calls for expansion to lower-resource languages and notes that reasoning processes may differ across languages.Source: Limitations · pp. 9–10

A Further Discussion

  • BLOOMZ-based models often copy input text rather than generate new text, while both interventions reduce this problem; GPT-3 does not show it in the reported formats.The authors link the reduced copying to BLOOMZ-TCC better eliciting the model’s underlying reasoning abilities.Source: A Further Discussion · p. 12

A.1 BLOOM Results for GSM8K, AQUA_RAT, AR_LSAT

BLOOM results are near chance on GSM8K, AQUA_RAT, and AR_LSAT, where the model lacks baseline ability; code-based training appears more useful for commonsense reasoning than math. The section also reports the TCC fine-tuning settings.

A.1 BLOOM Results for GSM8K, AQUA_RAT, AR_LSAT

  • Performance is around random chance across GSM8K, AQUA_RAT, and AR_LSAT; BLOOMZ-TCC slightly underperforms base BLOOMZ, while linearized and code prompts perform similarly.For GSM8K, random chance is 0, and the models solve nearly no math problems.Source: §A.1 · p. 12
  • The 7B BLOOMZ model lacks baseline ability on these three tasks, so indirect code supervision cannot elicit better reasoning there.The authors suggest complex reasoning may emerge with model scale.Source: §A.1 · p. 12
  • The code-and-reasoning hypothesis appears task-dependent: indirect code supervision helps ARC commonsense reasoning more than math, and AR_LSAT is deferred to future work with larger models.The authors note code rarely includes arithmetic equations.Source: §A.1 · pp. 12–13

B Hyperparameters

  • TCC fine-tuning used a 1024-token sequence limit, a 1e-5 learning rate, two epochs, batch size 3, and LoRA dimension 128.The run used cosine scheduling, no warm-up, 0.1 weight decay, and gradient accumulation over 20 steps.Source: B Hyperparameters · p. 13

C Dataset Statistics

The results contrast direct, linearized, and SIM prompting across STREET and xSTREET, while the dataset statistics table describes TCC. SIM leads on ARC, and intermediate reasoning is important for the reported arithmetic tasks.

C Dataset Statistics

  • Table 5 reports TCC statistics; its caption identifies the source as The Stack small subset for JavaScript, Python, and Java, with tokens counted using BLOOMZ’s 250k-vocabulary tokenizer.The full-results tables cover STREET and xSTREET averages and per-language results across models, tasks, prompt formats, and languages.Source: Table 5 · p. 13, C Dataset Statistics · p. 13, D Full Results · p. 13

D.1 Direct Prompting

  • On ARC, direct prompting outperforms linearized for all LLMs, but SIM outperforms both formats.The authors suggest ARC questions may be easy enough to solve directly, while linearized prompts can yield reasoning graphs without valid answers.Source: §D.1 · p. 14
  • For GPT-3 on GSM8K and AQUA_RAT, direct prompting fails; linearized prompting boosts arithmetic performance, and SIM boosts it further.The authors state that intermediate reasoning is essential for these arithmetic problems.Source: §D.1 · p. 14
  • The xSTREET results follow trends relatively consistent with those reported for STREET.Source: §D.1 · p. 14

D.2 Results for Falcon

Falcon’s STREET results vary by task and prompt, while its xSTREET results favor linearized prompts, which the authors attribute partly to context and tokenization limits. A separate SIM fine-tuning experiment improves ARC but leaves three tasks near chance.

D.2 Results for Falcon

  • Falcon is the 40B instruction-finetuned variant, selected as an intermediate size between BLOOMZ 7B and GPT-3 175B.Its 2048-token maximum context is shorter than the 4096-token context available to BLOOMZ and GPT-3.Source: §D.2 · p. 14
  • 34.7 direct, 76.5 linearized, and 81.5 SIM on ARC; on GSM8K, SIM reaches 19.6 versus linearized’s 28.9, attributed to context size.Falcon’s direct GSM8K result is 4.4.Source: §D.2 · p. 14, Table 6 · p. 15
  • AQUA_RAT stays near random at 20 across formats, while AR_LSAT reaches 34.0 with SIM versus near-random 20 for direct and linearized.Source: §D.2 · p. 14, Table 6 · p. 15
  • On multilingual xSTREET, linearized performs best across tasks, followed by code prompts and then direct prompting.The authors attribute this pattern to Falcon’s 2048-token context and byte-level tokenization of four non-Latin-script languages.Source: §D.2 · p. 14, Table 7 · p. 15

E.1 Finetuning on SIM Code Prompts

  • One LoRA-finetuned SIM model for all tasks and six languages reaches 85.9 versus 76.2 on ARC; the other three tasks remain near chance.The experiment omitted data replay; the authors suggest full fine-tuning might improve results but did not pursue it because of resource constraints.Source: §E.1 · p. 16

E.2 Improving the Code Comment Quality of TCC

The authors tested a larger-scale filtering process for code comments and human-reviewed translation quality. The resulting TCC-v2 model performed similarly to TCC, while annotators identified programming-specific translation as an unresolved issue.

E.2 Improving the Code Comment Quality of TCC

  • TCC began with 30,000 Stack files and was filtered to 20,000 files with more than five comments, over half of which were English.Source: §E.2 · p. 16
  • Human reviewers found non-useful comments, commented-out code, and mistranslated programming terms or code references in translated files.The last issue was left for future work because the authors used an online machine-translation API and programming-specific translation was out of scope.Source: §E.2 · p. 16
  • The expanded filtering experiment examined the first three million Stack examples across Java, Python, and JavaScript, deleting copyright, header, and linting comments and selecting files with high comment-to-line ratios.Files also had to contain more than five comments.Source: §E.2 · p. 16
  • TCC-v2 contained about 250,000 examples, but a 67,000-example subset yielded downstream reasoning performance similar to TCC and was not used in the main text.The authors hypothesize that program code contributes more to reasoning than comments.Source: §E.2 · p. 16
Loading 2403.02567v2…