PaperlayerView exact revision on arXiv

Source-linked AI summary

At Which Training Stage Does Code Data Help LLMs Reasoning?

Yingwei Ma, Yue Liu, Yue Yu, Yuanliang Zhang, Yu Jiang, Changjian Wang, Shanshan Li

arXiv:2309.16298v2cs.CL

TL;DR

The paper investigates when code data helps LLM reasoning, a question complicated by prior comparisons across models with different scales and unclear training details. It compares code data across training stages and finds broad reasoning benefits from pre-training, task-specific benefits from code instructions, and support for stepwise learning through dynamic mixing.

  • Problem

    Prior evaluations did not isolate code data's impact because models differed in parameter and data scales and had unclear training details.

    Source: §4 · p. 10

  • Method

    The paper compares code data at pre-training and instruction-tuning stages under the same parameter scale and training objective, evaluating reasoning across six tasks in five domains.

    Source: §1 · p. 2, §5 · p. 10

  • Results

    Code-text pre-training improves general reasoning with little negative transfer; code instructions improve task-specific code reasoning, while dynamic mixing supports stepwise learning.

    Source: §1 · p. 2, §3.3.2 · p. 7, §5 · p. 10

  • Takeaways & Limitations

    The findings deepen understanding of LLM reasoning for applications such as scientific question answering and legal support.

    Source: §1 · p. 2

Abstract

from arXiv · show

Large Language Models (LLMs) have exhibited remarkable reasoning capabilities and become the foundation of language technologies. Inspired by the great success of code data in training LLMs, we naturally wonder at which training stage introducing code data can really help LLMs reasoning. To this end, this paper systematically explores the impact of code data on LLMs at different stages. Concretely, we introduce the code data at the pre-training stage, instruction-tuning stage, and both of them, respectively. Then, the reasoning capability of LLMs is comprehensively and fairly evaluated via six reasoning tasks in five domains. We critically analyze the experimental results and provide conclusions with insights. First, pre-training LLMs with the mixture of code and text can significantly enhance LLMs' general reasoning capability almost without negative transfer on other tasks. Besides, at the instruction-tuning stage, code data endows LLMs the task-specific reasoning capability. Moreover, the dynamic mixing strategy of code and text data assists LLMs to learn reasoning capability step-by-step during training. These insights deepen the understanding of LLMs regarding reasoning ability for their application, such as scientific question answering, legal support, etc. The source code and model parameters are released at the link:~\url{https://github.com/yingweima2022/CodeLLM}.

1 INTRODUCTION

The paper asks when code data helps LLM reasoning and compares its use across training stages. It reports benefits from code-text pre-training, task-specific gains from code instructions, and stepwise learning with dynamic data mixing.

1 INTRODUCTION

  • The study asks at which training stage code data helps reasoning, motivated by code's logic and lower ambiguity and the importance of reasoning to LLM skills.The paper connects reasoning to capabilities such as translation, conversation, examination, and legal support.Source: §1 · p. 2
  • Code-text pre-training improves general reasoning with little negative transfer; code instructions support task-specific reasoning, and dynamic mixing supports stepwise learning.The evaluation covers six reasoning tasks in five domains.Source: §1 · p. 2

1.1 TRAINING DATA & TRAINING STRATEGIES

  • The experiments compare text-trained PanGu baselines with CodePanGu2.6B, trained from scratch on mixed code data, across pre-training and instruction tuning.The study uses both plain-text and text-code-mixed datasets.Source: §1.1 · p. 2

1.2 PRE-TRAINING CORPUS

  • The pre-training corpus combines 100GB of text matched to PanGu2.6B's scale and sources with CodeParrot, a roughly 50GB Python dataset.Figure 2 depicts the distribution of approximately 42B pre-training tokens.Source: §1.2 · p. 3, Figure 2 · p. 3

1.3 INSTRUCTION-TUNING CORPUS

  • The 500K instruction-tuning examples combine natural-language datasets with 150K code instructions, including expanded CodeAlpaca data.The code data includes CodeAlpaca and examples derived from CosQA and MBPP.Source: §1.3 · pp. 3–4

2 MODEL

The model follows autoregressive GPT-style training, maximizing next-token log-likelihood. Its architecture and tokenizer choices distinguish the mixed-code model from the text-only baseline.

2 MODEL

  • Training maximizes the log-likelihood of each token conditioned on the preceding tokens in the sequence.The model iteratively predicts the next token and compares it with the ground truth.Source: §2 · p. 4

2.1 MODEL ARCHITECTURE

  • The model uses a GPT-style autoregressive architecture with a 32-layer transformer decoder and an additional query layer for the final embedding.The 32-layer setting is retained for comparison with PanGu2.6B.Source: §2.1 · p. 4

2.2 TOKENIZATION

  • Text-only models use the 40,000-token PanGu vocabulary, while mixed-code models use the 130,044-token ChatGLM vocabulary.Both are BPE-based tokenizers; the ChatGLM vocabulary also encodes multiple spaces as extra tokens.Source: §2.2 · p. 4
  • Table 1 lists each task's input and prompt template, with `$` marking the input and other words marking the prompt.Source: Table 1 · p. 5

3 EXPERIMENTS

The experiments compare code and text data at different training stages across reasoning tasks and examine how evaluation methods and data-mixing strategies affect results. Code-mixed pre-training most consistently improves general reasoning, while instruction tuning and mixing strategies show task-specific effects.

3.1 TASK DESCRIPTION

  • The evaluation covers six reasoning tasks across logical, legal, scientific, analogical, and code-related scenarios.The code-related evaluations include code question answering and Python code generation.Source: §3.1 · p. 5

3.2 EVALUATION DETAILS

  • For understanding tasks, the authors use prompts and select the label with minimum perplexity; for generation tasks, models generate answers directly.Source: §3.2 · pp. 5–6

3.3.1 PRE-TRAINING STAGE

  • Code-mixed pre-training improved most reasoning tasks, including non-code tasks; CodePangu2.6B also surpassed text-trained PanGu13B across the reported reasoning scenarios.The authors summarize this as an improvement in general reasoning ability.Source: Table 2 · p. 6, §3.3.1 · p. 6

3.3.2 INSTRUCTION-TUNING STAGE

  • Instruction-tuning code data produced mixed reasoning trends, was less effective than pre-training for general reasoning, but improved CosQA and MBPP, especially code generation.The authors note that some tasks declined after instruction tuning and suggest domain-specific instructions for targeted reasoning capabilities.Source: Table 3 · p. 7, §3.3.2 · p. 7

3.3.3 CHAIN-OF-THOUGHT ABILITY

  • With Chain-of-Thought prompting, CODE (2.6B) achieved the best performance among the tested models on reasoning problems.The experiments evaluated the pre-trained models on ScienceQA(CoT) and E-KAR(CoT).Source: §3.3.3 · pp. 7–8, Table 4 · p. 8

3.3.4 EXPLORING WAYS TO MIX CODE AND TEXT DATA

  • A strategy that starts with a higher code-data ratio and gradually reduces it performed best on CosQA and MBPP while preserving performance on other reasoning tasks.The authors left validation of this strategy during pre-training for later work because those experiments require substantial resources.Source: Table 6 · p. 8, §3.3.4 · p. 8, Table 5 · p. 8

3.3.5 OTHER TASKS

  • Across most other comprehension and generation tasks, adding code data had little negative impact and sometimes helped, although some DuReader performance declined.The authors suggest the decline may reflect insufficient learning of code and text data, and plan to investigate with larger models and datasets.Source: Table 7 · p. 9, Table 8 · p. 9, §3.3.5 · p. 9

4 RELATED WORK

Related work describes language-only training and models that also use code, while noting that prior comparisons leave code’s effect on reasoning difficult to isolate.

4 RELATED WORK

  • Instruction tuning is presented as important for instruction following, alongside examples of models and approaches used in LLM training.The passage describes human-feedback tuning and instructions generated from public benchmarks.Source: §4 · p. 9
  • GPT-3 and PanGu-α use natural-language data, while LaMDA and LLaMA also include code data.The passage notes that the impact and specific origin of mixed-code data are unclear.Source: §4 · pp. 9–10
  • Prior evaluations used models with different parameter and data scales and unclear training details, making code’s exact impact on reasoning difficult to determine.The cited work suggests code may improve reasoning, but these comparison differences complicate attribution.Source: §4 · p. 10

5 CONCLUSION

The paper concludes that code data can help reasoning at both pre-training and instruction-tuning stages, and describes released training resources and MindSpore deployment.

5 CONCLUSION

  • Code-text pre-training can improve general reasoning, while code instructions at instruction tuning support instruction following and specific code reasoning.The paper also reports that dynamic code-text mixing assists step-by-step reasoning learning.Source: §5 · p. 10

A.1 TORCH AND MINDSPORE VERSION

  • Training and inference code is released in Torch, with a MindSpore deployment intended to make that framework easier to access for training and inference.The passage provides links to both implementations.Source: §A.1 · p. 13

A.2 INTRODUCTION TO MINDSPORE

  • MindSpore is described as newer and requiring further development time, though the passage reports 398 repositories using it since 2023 versus 192 using TensorFlow.The authors cite adoption by several research projects as evidence of community interest.Source: §A.2 · p. 13

B EXPERIMENTS DETAILS

The experiments use a shared model and sequence-length setup, with different Ascend 910 cluster sizes for pre-training and instruction tuning.

B EXPERIMENTS DETAILS

C THE TEMPLATE FOR OTHER TASKS

The appendix checks several comprehension and generation tasks, including C3, two NLI datasets, and duReader, and specifies their evaluation metrics.

C THE TEMPLATE FOR OTHER TASKS

D TRAINING LOSS

CodePanGu2.6B’s cross-entropy loss decreases steadily during training and converges around 2.25.

D TRAINING LOSS

  • CodePanGu2.6B’s cross-entropy loss decreases steadily during training and converges around 2.25.The paper presents these loss curves as a measure of training progress.Source: D TRAINING LOSS · p. 13

E CASE STUDY

The case study connects code’s step-by-step generation and dependencies with reasoning processes used in other tasks. The authors suggest these properties may help explain why code data improves general reasoning.

E CASE STUDY

  • Pre-training with code data can improve general reasoning and guide coherent Chain-of-Thought answer generation.The authors suggest code’s logic may help explain this improvement.Source: E CASE STUDY · p. 13
  • Figure 4 presents models with 2.6B and 13B parameters, each consisting of a 32-layer left-to-right transformer decoder and a top query layer.Source: Figure 4 · p. 14
  • Code requires step-by-step reasoning for correct generation, while its data and control-flow dependencies may benefit tasks requiring long inference chains.The authors suggest code data may implicitly teach a thinking chain; they illustrate dependencies with the calculate_average function’s data-flow graph.Source: E CASE STUDY · p. 14

F DATASET CONSTRUCTION

The dataset construction section describes text cleaning, deduplication, and sampling for CodePanGu2.6B, alongside supplementary training materials and figures.

F DATASET CONSTRUCTION

  • Cleaning removes documents with under 60% Chinese characters, fewer than 150 characters, or only a webpage title, and removes special symbols and duplicate paragraphs.These are among the stated rule-based filters applied to raw Common Crawl pages.Source: F DATASET CONSTRUCTION · p. 14
  • The cleaning rules also remove documents containing advertisements and webpage navigation bars.Advertisements are identified by keywords.Source: F DATASET CONSTRUCTION · p. 15
  • The authors supplement Common Crawl with open-source code and e-books as part of a strategy to mitigate potential data biases.They note that crawled data may exhibit biases and that ground-truth data can be difficult to access.Source: F DATASET CONSTRUCTION · p. 15
  • Fuzzy deduplication across data sources uses Spark’s MinHashLSH algorithm to remove documents with highly overlapping content.The paper describes MinHashLSH as a widely adopted technique.Source: F DATASET CONSTRUCTION · p. 15
  • The appendix includes CodePanGu2.6B training-loss curves, examples of code, a training-configuration table, and input-prompt templates for other tasks.The supplied figure and table captions identify these materials but do not provide further findings.Source: Table 9 · p. 15, Table 10 · p. 15, Figure 5 · p. 16
  • A 100GB text corpus, sampled across data sources, forms the first part of CodePanGu2.6B’s training set.The sampling uses source ratios shown in Figure 2.Source: F DATASET CONSTRUCTION · pp. 15–16

G URLS OF USED DATASETS

The appendix lists dataset URLs and shows code-dependency examples. It also notes that evaluation is limited to PanGu models and that the work targets general LLM reasoning rather than specialized code or text models.

G URLS OF USED DATASETS

  • The dataset-links section lists sources including Common Crawl, BAAI-WuDao, CodeParrot, GitHub code, and Stanford Alpaca.A separate figure in this section shows examples of code dependencies.Source: G URLS OF USED DATASETS · pp. 16–17
  • A figure in this section presents examples of code dependencies.The caption identifies the figure’s subject without describing the examples.Source: Figure 7 · p. 17

H.1 VERIFIED ON MORE LARGE LANGUAGE MODELS.

  • Experiments cover only PanGu models, at 2.6B and 13B parameters; verification on LLMs with architectures different from GPT is left for future work.The authors cite resource constraints and environmental concerns for the current scope.Source: §H.1 · p. 17

H.2 WHY NOT DESIGN A NEW MODEL FOR CODE?

  • The study examines how code helps general LLM reasoning across training stages, rather than developing a specialized code or text model.The authors frame the findings as guidance for future universal LLM development.Source: §H.2 · p. 17
Loading 2309.16298v2…