PaperlayerView exact revision on arXiv

Source-linked AI summary

On Code-Induced Reasoning in LLMs

Abdul Waheed, Zhen Wu, Carolyn Rosé, Daphne Ippolito

arXiv:2509.21499v2cs.CLcs.PL

TL;DR

The paper asks which properties of code help LLMs reason, a question left open despite evidence that code training can improve reasoning. It uses parallel instruction data and controlled code perturbations across models and tasks. The experiments find greater vulnerability to structural than semantic perturbations, while some abstractions can match or surpass code on non-code tasks.

  • Problem

    Although code training has been found to improve reasoning, which aspects of code produce those gains remains unclear.

    Source: §1 · p. 1

  • Method

    The study combines parallel instruction data, controlled code perturbations, and fine-tuning and evaluation across model families, scales, and reasoning tasks.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    Structural perturbations degrade performance more than semantic ones, especially on math and code tasks, while pseudocode or flowcharts can match or surpass unperturbed code on other tasks.

    Source: §4 · pp. 7–8

  • Takeaways & Limitations

    Some abstractions preserve useful algorithmic structure while removing superficial syntax, whereas code generation performs best with perturbations that retain explicit code structure.

    Source: §4 · p. 8

    The study focuses on small- to mid-scale base models, its perturbations may omit factors such as code complexity and data diversity, and its benchmarks capture only part of the reasoning spectrum.

    Source: §6 · p. 10

Abstract

from arXiv · show

Code data has been shown to enhance the reasoning capabilities of large language models (LLMs), but it remains unclear which aspects of code are most responsible. We investigate this question with a systematic, data-centric framework. We construct parallel instruction datasets in ten programming languages and apply controlled perturbations that selectively disrupt structural or semantic properties of code. We then finetune LLMs from five model families and eight scales on each variant and evaluate their performance on natural language, math, and code tasks. Across 3,331 experiments, our results show that LLMs are more vulnerable to structural perturbations than semantic ones, particularly on math and code tasks. Appropriate abstractions like pseudocode and flowcharts can be as effective as code, while encoding the same information with fewer tokens without adhering to original syntax can often retain or even improve performance. Remarkably, even corrupted code with misleading signals remains competitive when surface-level regularities persist. Finally, syntactic styles also shape task-specific gains with Python favoring natural language reasoning and lower-level languages such as Java and Rust favoring math. Through our systematic framework, we aim to provide insight into how different properties of code influence reasoning and inform the design of training data for enhancing LLM reasoning capabilities.

1 INTRODUCTION

Prior work finds that code training can improve reasoning, including on tasks unrelated to programming, but the code properties responsible remain unclear. This paper investigates those properties with parallel instruction data, controlled perturbations, and evaluations across reasoning tasks.

1 INTRODUCTION

  • Code training has been found to improve reasoning even on tasks unrelated to programming, with logical consistency, compositional structure, and reduced ambiguity proposed as possible signals (Ma et al., 2023b; Zhang et al., 2024b).Prior work has considered code in both pre-training and post-training.Source: §1 · p. 1
  • The paper asks which code properties—syntactic regularity, structural abstractions, or linguistic styles—drive reasoning improvements.The authors note that code training has broad effectiveness but lacks a systematic account of the responsible aspects.Source: §1 · p. 1
  • The study builds parallel natural-language and code instruction datasets, applies controlled rule-based and generative perturbations, and evaluates five model families across eight scales.The framework aims to disentangle code properties that contribute to reasoning.Source: §1 · pp. 1–2

2 RELATED WORK

Related work reports benefits from code training for reasoning and studies how training-data composition and quality affect model behavior. This paper is closest to work stress-testing reasoning with code perturbations.

2 RELATED WORK

  • Prior studies report that adding code during pretraining or instruction tuning improves reasoning across domains, model scales, and architectures (Zhang et al., 2024a).Other work connects code’s structured and verifiable properties with logical decomposition and intermediate-step generation (Bi et al.; Yang et al., 2024).Source: §2 · p. 2
  • The closest related work stress-tests LLM reasoning with structural and semantic code perturbations and reports that small corruptions can reduce performance.Related work also examines code’s effects on multilingual reasoning, alignment, and reward modeling.Source: §2 · p. 2
  • Research on training data finds that curation, composition, and noise can affect downstream performance and persistently alter model behavior (Longpre et al., 2024; Zhang et al., 2024c; Havrilla & Iyer, 2024).This literature also examines global accumulative errors in chain-of-thought training data.Source: §2 · p. 2

3 METHODOLOGY

The methodology compares parallel natural-language and code instruction data, systematically varies code representations, and fine-tunes and evaluates models across multiple tasks. Its perturbations target structural and semantic properties while keeping dataset example counts unchanged.

3 METHODOLOGY

  • The framework constructs parallel instruction datasets, modifies code data, then fine-tunes models on each variant and evaluates their performance.Figure 1 summarizes these stages, including language-specific ablations and evaluations on natural-language, code, and math reasoning tasks.Source: §3 · p. 2, Figure 1 · p. 3

3.1 INSTRUCTION DATA GENERATION

  • The two instruction datasets each contain 120,000 pairs; code responses span ten languages, while the natural-language subset is filtered to general-purpose, English examples.The code dataset is assembled from public sources, deduplicated, and filtered to remove language-specific instructions and domain-tied solutions.Source: §3.1 · pp. 2–3

3.2.1 RULE-BASED PERTURBATIONS

  • Rule-based perturbations target formatting and semantic cues through whitespace removal, identifier renaming, keyword replacement, comment removal, and comment swapping.The transformations are designed to disrupt patterns or signals without altering the code’s core logic.Source: §3.2.1 · pp. 3–4

3.2.2 GENERATIVE PERTURBATIONS

  • Generative perturbations rewrite code while preserving its intent, including versions with enhanced documentation or deliberately misleading comments.Enhanced comments document functions, classes, and key blocks; obfuscated comments can be irrelevant, nonsensical, or contradictory.Source: §3.2.2 · pp. 4–5
  • Other generative variants represent code as pseudocode, Markdown flowcharts, natural-language steps, or an imaginary language while retaining selected aspects of its logic or structure.These variants test reasoning across abstraction levels, from symbolic control flow to language-agnostic narrative procedures.Source: §3.2.2 · p. 5

3.3 MODEL TRAINING AND EVALUATION

  • Models are fine-tuned with the same supervised objective on original or perturbed data, including code-only, natural-language-only, mixed, and single-language configurations.The model suite spans five families and sizes from 0.6B to 8B parameters; single-language subsets contain about 12,000 examples each.Source: §3.3 · pp. 5–6
  • Evaluation covers natural-language and general-knowledge tasks, math, and code understanding and generation, with task-appropriate accuracy or exact-match scoring.Code outputs are assessed by an LLM judge using instance-specific rubrics because the small perturbed models often fail to produce fully executable code.Source: §3.3 · p. 6

4 RESULTS AND DISCUSSION

Code and mixed-data finetuning generally perform strongly, while controlled perturbations show that reasoning depends on how code structure and information are represented. The effects vary by task: suitable abstractions and compact variants can work well, and programming-language styles are associated with different strengths.

RQ1: Does incorporating code in finetuning improve task performance?

  • Across 14 model bases, code-ft or mixed-ft is best on 64% of natural language tasks, 86% of math and code understanding tasks, and all code generation tasks.Code-ft and mixed-ft generally achieve leading or competitive performance across tasks, especially code generation.Source: Figure 2 · p. 6, §4 · pp. 6–7

RQ2: How do our systematic perturbations affect performance?

  • Structural perturbations degrade performance more than semantic ones, especially on math and code tasks, and the discrepancy grows as models scale.Nearly all perturbations reduce performance versus unperturbed code finetuning; the authors hypothesize that math and code rely more on formatting and layout cues.Source: §4 · pp. 7–8

Explicitness of Code Structure

  • For code generation, forms preserving explicit code structure perform best; on other tasks, pseudocode and flowcharts can match or surpass code, while natural-language procedures generally perform worst.The comparison spans runnable code, intermediate abstractions, and natural-language step-by-step procedures.Source: §4 · pp. 7–8, Figure 4 · p. 7

Relative Information Density

  • Strong or moderate token reductions often match or outperform unperturbed code, except in code generation, where richer surface detail matters.Smaller models are more sensitive to information-density differences, while larger models remain robust.Source: §4 · pp. 7–8, Figure 5 · p. 7

Human Interpretability

  • Low-interpretability variants with noise or misleading signals often remain near baseline and can match or surpass medium-interpretability variants.The authors suggest models may exploit surface-level regularities and recurring structural cues that persist in noisy or opaque forms.Source: §4 · pp. 7–8, Figure 6 · p. 8

RQ3: How does performance vary across programming languages?

  • High-scripting languages underperform intermediate and low-system languages on math, while Python often leads on natural-language tasks and Java and Rust rank among the math leaders.Single-language finetuning improves code generation over zero-shot but lags full code finetuning; language-group differences on natural-language and code tasks are largely model-dependent.Source: §4 · pp. 8–9, Figure 7 · p. 9

5 CONCLUSION

The paper concludes that code’s structural properties matter for reasoning, but effective training signals need not preserve verbose or fully interpretable code. Language styles also show task-specific patterns, with lower-level languages benefiting math tasks.

5 CONCLUSION

  • Disrupting code structure consistently reduces performance, especially on math and code tasks.The experiments also find that appropriate abstractions and efficient encodings can be as effective as raw code.Source: §5 · p. 9
  • Appropriate abstractions and efficient encodings can be as effective as raw code.Models can remain robust to corrupted or low-interpretability code when statistical regularities persist.Source: §5 · p. 9
  • Lower-level programming languages provide more benefits for math tasks.The paper presents its findings as guidance for constructing effective training data beyond executable programs.Source: §5 · p. 9

6 LIMITATIONS

The study’s scope is constrained by model scale, the range of perturbations, and benchmark coverage; the authors identify larger models and additional domains as directions for future work.

6 LIMITATIONS

  • The study focuses on small- to mid-scale base models because of resource constraints.The authors suggest extending the framework to larger models.Source: §6 · p. 10
  • The perturbations may not cover enough factors, including code complexity and data diversity..Source: §6 · p. 10
  • The benchmarks capture only part of the reasoning spectrum, leaving additional domains for future analysis.Source: §6 · p. 10

7 REPRODUCIBILITY STATEMENT

Reproducibility details are distributed across the paper and supplementary materials, with dataset processing, training implementation, and prompts documented in specified sections and appendices.

7 REPRODUCIBILITY STATEMENT

  • Section 3.1 describes construction and processing of the code and natural language datasets.Source: §7 · p. 10
  • Appendix A.3 outlines model training and implementation details.Source: §7 · p. 10
  • Appendix A.4 includes prompts for data generation, perturbations, and LLM-as-Judge evaluation.Source: §7 · p. 10

A.1 EVALUATION SUITE DETAILS

The evaluation-suite details identify its task domains, organize perturbations across four analysis axes, and describe language-specification templates.

A.1 EVALUATION SUITE DETAILS

  • The evaluation suite spans natural language and general knowledge, math, and code tasks.Source: Table 2 · p. 13
  • Perturbations are categorized across structural versus semantic, explicitness of code structure, relative information density, and human interpretability axes.Source: Table 3 · p. 14
  • Language-specification templates use placeholders that can be instantiated with different programming languages.Source: Table 4 · p. 14

A.3 IMPLEMENTATION DETAILS

This appendix presents finetuning setup and prompt variants, along with task-performance figures for code-data configurations, perturbations, and programming-language choices, including a SmolLM2-1.7B breakdown by language group and individual language.

A.3 IMPLEMENTATION DETAILS

  • All models use full finetuning in BF16 for two epochs, with a 2048-token sequence limit, a fixed 1e−5 learning rate, and mostly batch size 64.Experiments run on a 4×A100 80G node; language-specific settings use batch size 32, and training uses cosine decay with 0.1 warmup, DeepSpeed ZeRO Stage 3, and LLaMA-Factory.Source: §A.3 · p. 14

A.4 PROMPTS

  • Prompts cover code generation, comment enhancement or obfuscation, pseudocode, flowcharts, step-by-step guides, and imaginary-language translation.The appendix also provides prompts for generating instance-specific rubrics and judging model-generated code responses.Source: §A.4 · pp. 14–15, Figure 8 · p. 15, Figure 9 · p. 15

A.5.1 TASK PERFORMANCE SHOWCASING CODE DATA IMPACT IN FINETUNING (RQ1)

  • Task-performance figures compare zero-shot, code-finetuned, natural-language-finetuned, and mixed code–natural-language models across five model families, alongside code-mixture-ratio ablations for two Qwen3 sizes.The mixture-ratio ablations cover Qwen3-0.6B-Base and Qwen3-1.7B-Base; the supplied passages identify the figures but do not give their result values.Source: §A.5.1 · pp. 16–17

A.5.2–A.5.5 PERTURBATION AGGREGATES (RQ2)

  • For Qwen3, task-performance figures aggregate perturbation results by structure versus semantics and by structural explicitness, relative information density, and human interpretability.The passages point to these aggregated figures without reporting specific performance comparisons.Source: §A.5.2 · p. 18, §A.5.3 · p. 19, §A.5.4 · p. 20

A.5.6 TASK PERFORMANCE FOR ALL INDIVIDUAL PERTURBATIONS (RQ2)

  • Figures report task performance for individual perturbation configurations across Qwen3, Llama-3.2, Gemma-3, OLMo-2, and SmolLM2 model families.The supplied passages identify the figures and model-family scope but do not state which configurations perform best.Source: §A.5.6 · pp. 22–23

A.5.7 TASK PERFORMANCE WITH DIFFERENT PROGRAMMING LANGUAGES (RQ3)

  • Programming-language results include grouped and individual-language performance figures for Qwen3, Llama-3, and SmolLM2.The Qwen3 grouped figure distinguishes low-system, intermediate, and high-scripting language groups.Source: §A.5.7 · pp. 23–24, Figure 48 · p. 37
  • Figure 51 reports SmolLM2-1.7B performance across programming-language groups and individual languages.Source: Figure 51 · p. 40
Loading 2509.21499v2…