PaperlayerView exact revision on arXiv

Source-linked AI summary

CodeI/O: Condensing Reasoning Patterns via Code Input-Output Prediction

Junlong Li, Daya Guo, Dejian Yang, Runxin Xu, Yu Wu, Junxian He

arXiv:2502.07316v4cs.CLcs.AI

TL;DR

Many reasoning domains have sparse and fragmented supervision despite abundant structured data for math and code. CodeI/O trains models to predict code inputs and outputs through natural-language rationales, and the paper reports balanced improvements across reasoning benchmarks.

  • Problem

    Logical, scientific, and symbolic reasoning domains have sparse and fragmented supervision compared with math and code generation.

    Source: §1 · p. 1

  • Method

    CodeI/O transforms code into input-output prediction tasks, training models to predict inputs or outputs from code and test cases using natural-language Chain-of-Thought rationales.

    Source: §1 · pp. 1–2

  • Results

    CodeI/O and CodeI/O++ consistently outperform existing baselines, delivering balanced improvements across benchmarks.

    Source: §3.2 · p. 6, §6 · p. 9

  • Takeaways & Limitations

    The results suggest code-centric training can improve reasoning on tasks beyond code reasoning, including diverse domains.

    Source: §3.2 · p. 6, §6 · p. 9

    Evaluation excludes code-generation and instruction-following subtasks.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Reasoning is a fundamental capability of Large Language Models. While prior research predominantly focuses on enhancing narrow skills like math or code generation, improving performance on many other reasoning tasks remains challenging due to sparse and fragmented training data. To address this issue, we propose CodeI/O, a novel approach that systematically condenses diverse reasoning patterns inherently embedded in contextually-grounded codes, through transforming the original code into a code input-output prediction format. By training models to predict inputs/outputs given code and test cases entirely in natural language as Chain-of-Thought (CoT) rationales, we expose them to universal reasoning primitives -- like logic flow planning, state-space searching, decision tree traversal, and modular decomposition -- while decoupling structured reasoning from code-specific syntax and preserving procedural rigor. Experimental results demonstrate CodeI/O leads to consistent improvements across symbolic, scientific, logic, math & numerical, and commonsense reasoning tasks. By matching the existing ground-truth outputs or re-executing the code with predicted inputs, we can verify each prediction and further enhance the CoTs through multi-turn revision, resulting in CodeI/O++ and achieving higher performance. Our data and models are available at https://github.com/hkust-nlp/CodeIO.

1. Introduction

CodeI/O turns contextual code into natural-language input/output prediction tasks to expose reusable reasoning patterns without requiring code-specific syntax. Its scalable, verifiable training data improves reasoning across diverse domains.

Motivation

  • CodeI/O targets sparse, fragmented supervision in reasoning domains beyond math and code by drawing on diverse reasoning patterns embedded in real-world programs.Raw-code training can leave reasoning implicit among noisy information, while text-to-code generation requires code-specific syntax that can hinder generalization beyond code tasks.Source: §1 · p. 1

Approach

  • CodeI/O trains models to predict outputs from inputs or feasible inputs from outputs in natural-language CoT, separating reasoning flow from code syntax while preserving logical rigor.The approach aims to expose skills including logic-flow orchestration, state-space exploration, recursive decomposition, and decision-making.Source: §1 · p. 1

Data and training

  • The intermediate training stage uses over 450K functions to produce 3.5M samples, with multiple executable input-output pairs generated for each function.The prompt combines a function, textual query, and given input or output; responses are collected from DeepSeek-V2.5 before general instruction tuning.Source: §1 · p. 2
  • Code execution verifies predictions, and DeepSeek-V2.5 revises initially incorrect responses to create CODEI/O++, which further enhances model performance.The multi-turn revisions are concatenated into longer responses.Source: §1 · p. 2

Results

  • Across four 7B–30B base models and 14 benchmarks, CODEI/O achieves higher average scores than several data baselines and consistent improvements across nearly all benchmarks.The reported gains span code-related and more general logic, symbolic, mathematical and numerical, scientific, and commonsense reasoning tasks.Source: §1 · p. 2

2. CODEI/O

CodeI/O builds a trainable input-output prediction dataset by collecting code, standardizing it, sampling executable examples, and generating natural-language reasoning responses, with execution feedback used for revision in CodeI/O++.

2. CODEI/O

  • The pipeline collects raw code, converts it to a unified format, samples input-output pairs, and assembles them into training data.The four stages are described in §2.1–§2.4.Source: §2 · p. 2

2.1. Collecting Raw Code Files

  • CodeI/O combines CodeMix, reasoning-focused PyEdu-R, and smaller reputable sources to obtain approximately 810.5K code files.PyEdu-R is a subset of Python-Edu (Ben Allal et al., 2024); pure algorithms are excluded from it to avoid overlap with CodeMix.Source: §2.1 · p. 2

2.2. Transforming to a Unified Format

  • DeepSeek-V2.5 preprocesses raw files into executable, self-contained functions with defined inputs and outputs, an input generator, and a concise functionality query.The unified format also cleans and refactors code to extract core logic while excluding non-essential visualization and file processing.Source: §2.2 · pp. 2–3

2.3. Collecting Input and Output Pairs

  • Executing generated inputs yields 3.5M instances from 454.9K code files, with input- and output-prediction instances roughly balanced.Functions involving randomness are skipped to ensure deterministic outputs; non-executable or over-limit samples are filtered.Source: §2.3 · p. 3

2.4. Building Samples for Input-Output Prediction

  • Prompts pair code and a query with a specific input or output, while DeepSeek-V2.5 generates natural-language CoTs for CODEI/O and execution feedback supports CODEI/O++ revisions.Incorrect responses are regenerated with feedback and rechecked; the final CODEI/O++ response concatenates both turns and their feedback, retaining correct and incorrect responses.Source: §2.4 · p. 4

3. Experiments

Across the evaluated benchmarks, CODEI/O delivers consistent gains over the single-stage baseline and other datasets, while CODEI/O++ improves average scores further. The experiments use two-stage training and assess reasoning across multiple domains and model families.

3.1. Settings

  • Experiments use Qwen 2.5 7B Coder (Hui et al., 2024), Deepseek v2 Lite Coder (MoE) (Zhu et al., 2024), LLaMA 3.1 8B (Dubey et al., 2024), and Gemma 2 27B (GemmaTeam et al., 2024), with about 1.18M instruction-tuning samples.The backbones vary in architecture, size, and pre-training focus; the tuning data spans languages and domains including math, coding, and writing.Source: §3.1 · p. 4
  • Most experiments train first on CODEI/O or CODEI/O++ and then instruction-tune, separating reasoning training from adaptation to diverse instructions.The authors say this avoids the larger CodeI/O dataset dominating a mixed training distribution and potentially limiting instruction-following.Source: §3.1 · p. 4
  • Evaluation spans science, math and numerical, symbolic, commonsense, logic, and code-understanding benchmarks, plus LiveBench and KorBench.Most evaluations use greedy zero-shot decoding; BBH-EN and BBH-ZH use three-shot evaluation, and the LiveBench split excludes code-generation and instruction-following subtasks.Source: §3.1 · pp. 4–5

3.2. Main Results

  • CODEI/O improves performance consistently across benchmarks, outperforming the single-stage baseline and other datasets, including larger ones.Competing datasets may excel on particular tasks while regressing on others; raw-code training yields only minor or sometimes negative gains relative to the single-stage baseline.Source: Table 1 · p. 5, §3.2 · p. 6

3.1. Settings

  • Baselines include single-stage instruction tuning and first-stage training on WebInstruct, OpenMathInstruct-2, OpenCoder-SFT-Stage-1, or PythonEdu.WebInstruct and OpenMathInstruct-2 use 3.5M-sample subsets in most experiments to align with CODEI/O's size; full-dataset scores are also reported for Qwen 2.5 7B Coder.Source: §3.1 · pp. 5–6

3.2. Main Results

  • CODEI/O++ further raises average scores without individual-task trade-offs, while both methods improve performance on most benchmarks across varied model sizes and architectures.A small number of tasks show nearly unchanged or decreased performance; the paper attributes CODEI/O++ gains to execution-feedback-based multi-turn revision.Source: §3.2 · p. 6

4. Analysis

The analysis probes CODEI/O’s data construction, synthesis source, scale, format, revision, training strategy, and leakage. It finds that code-I/O reasoning examples outperform response distillation alone, more training data helps, and gains persist on non-leaked samples.

4.1. Ablation Studies

  • Input and output prediction yield similar scores but favor different benchmarks, while rejection sampling or answer replacement does not improve overall performance balance.Input prediction excels on KorBench but slightly hurts GPQA, whereas output prediction benefits symbolic tasks such as BBH and the respective CRUXEval variants. Rejection sampling removes 50% of training data and causes a general performance drop; replacing incorrect responses with ground-truth answers helps LeetCode-O and CRUXEval-O but lowers scores elsewhere, and neither approach beats a comparable CODEI/O subset, so incorrect responses are retained unchanged.Source: §4.1 · pp. 6–7

4.3. Scaling Effect of CODEI/O

  • Figure 4a shows performance generally improves as CODEI/O training samples increase, with the full dataset yielding the most comprehensive and robust performance.The smallest data amount performs relatively weakly on most benchmarks, while intermediate amounts fall between the smallest and full dataset.Source: Figure 4 · p. 7, §4.3 · p. 7
  • Figure 4b shows that using more input-output pairs per code sample improves performance, particularly when increasing from 1/6 to 6/6.This effect is less pronounced than scaling the number of training samples, suggesting some reasoning patterns need multiple test cases to capture complex logic flow.Source: Figure 4 · p. 7, §4.3 · p. 7

4.2. Effect of Different Synthesis Model

  • WebInstruct-DS25 outperforms vanilla WebInstruct but still falls short of CODEI/O, showing that synthesis-model distillation alone does not match code-I/O prediction.Responses for the 3.5M-example WebInstruct subset were regenerated with DeepSeek-V2.5; the updated dataset improved results on Qwen 2.5 Coder 7B and LLaMA 3.1 8B but remained below CODEI/O.Source: §4.2 · p. 7

4.4. Different Data Format

  • The query-and-reference-code prompt with CoT in the response achieves the highest average score and most balanced benchmark performance; putting code in the response performs worst.Other formats have slightly lower but comparable results; the worst format resembles standard code generation with the query in the prompt and reference code in the response.Source: §4.4 · pp. 7–8, Table 3 · p. 8

4.5. Multi-turn Revision

  • Single-turn revision corrects about 10% of initially incorrect responses, but a second revision turn yields minimal gains, so CODEI/O++ stops after one turn.From turn 1 to turn 2, LLaMA 3.1 8B improves slightly while Qwen 2.5 Coder 7B regresses; the model often repeats the same incorrect CoT without useful new information.Source: Figure 5 · p. 8, §4.5 · p. 8

4.6. The Necessity of Two Stage Training

  • All two-stage training variants outperform single-stage training, but the best data mixture varies by model; the main experiments keep CODEI/O and instruction-tuning data separate.Qwen 2.5 Coder 7B does best with fully separate data, while LLaMA 3.1 8B does better with mixing in either stage.Source: Table 4 · p. 8, §4.6 · p. 8

4.7. Discussion on Data Leakage

  • 13-gram audits found little actionable leakage: apparent overlaps were generic on KorBench or sibling LeetCode-O descriptions, and CODEI/O gains persisted on non-leaked samples.The audit flags normalized 13-gram overlaps; manual inspection found KorBench overlaps were general descriptions, while LeetCode-O overlaps came from sibling problems despite removing original problems from training.Source: §4.7 · pp. 8–9, Table 6 · p. 9

5. Related Work

CodeI/O differs from prior code-execution work by training LLMs on large-scale, diverse code input-output predictions and demonstrating gains in general reasoning beyond code. The paper also describes CodeI/O as orthogonal to inference-time scaling.

Learning about Code Execution

  • Prior code-execution learning focuses on output prediction, execution feedback or traces for code generation, and benchmarks for predicting execution results.The cited benchmarks include CRUXEval (Gu et al., 2024) and LiveCodeBench-Exec (Jain et al., 2024).Source: §5 · p. 9
  • The paper claims CodeI/O is the first to train LLMs on large-scale, diverse code input-output predictions and demonstrate improved general reasoning beyond code.Jiang et al. (2024b) also use code execution to enhance general reasoning, but mainly with LeetCode functions.Source: §5 · p. 9

Inference Time Scaling

  • Inference-time scaling methods such as OpenAI’s o1 (Jaech et al., 2024) and DeepSeek’s R1 (DeepSeek-AI et al., 2025) use ultra-long reasoning encouraged by large-scale reinforcement learning; the authors view CodeI/O as orthogonal.The authors hope CodeI/O can provide a better basis for further incentivizing LLM reasoning.Source: §5 · p. 9

6. Conclusion · A. Details of Checking Execution Complexity

CodeI/O and CodeI/O++ are presented as improving LLM reasoning across diverse domains, while the appendix describes execution and input/output-size checks used to keep generated objects predictable.

6. Conclusion

  • CODEI/O and CODEI/O++ consistently outperform existing baselines, with balanced improvements across benchmarks and no sacrificed domain performance.The conclusion attributes their robustness and versatility to these results.Source: §6 · p. 9

A. Details of Checking Execution Complexity

  • Each code sample is limited to a 5-second runtime during execution.This check is part of the appendix’s execution-complexity safeguards.Source: A. Details of Checking Execution… · p. 13
  • The execution codes impose a 5-second runtime limit per sample and constrain input/output objects to under 1024 bytes, lists and dictionaries to under 20 items, and strings to at most 100 characters.Other object types are limited to 128 bytes, and checks apply recursively to sub-objects.Source: A. Details of Checking Execution… · p. 13
  • Objects outside the simple types are subject to a 128-byte size limit.The implementation checks object size with asizeof.Source: A. Details of Checking Execution… · p. 13
  • The strict_check_size function returns False for dictionaries with length at least 20; it also has a branch for lists, tuples, and sets, though these excerpts do not show their checks.Source: A. Details of Checking Execution… · p. 13

B. Details of Selected Benchmarks

This section describes the selected benchmarks by the reasoning abilities they test and introduces two author-created benchmarks. It also specifies evaluation conditions for LeetCode-O.

B. Details of Selected Benchmarks

  • WinoGrande (Sakaguchi et al., 2020) evaluates commonsense reasoning with expert-crafted pronoun-resolution problems.BBH (Suzgun et al., 2023) comprises 23 tasks previously found challenging for language models from BIG-Bench (Srivastava et al., 2022).Source: B. Details of Selected Benchmarks · p. 13
  • DROP (Dua et al., 2019), GSM8K (Cobbe et al., 2021), and MATH (Hendrycks et al., 2021b) cover numerical reading comprehension and grade-school through competition-level mathematics.DROP requires reference resolution and operations such as addition, counting, or sorting, and is scored by F1. GSM8K targets basic arithmetic and reasoning; MATH spans five difficulty levels and tests complex mathematical reasoning.Source: B. Details of Selected Benchmarks · p. 13
  • GPQA (Rein et al., 2024) tests deep expertise in biology, physics, and chemistry, while MMLU-STEM (Hendrycks et al., 2021a) uses the STEM subset of a 57-subject knowledge benchmark.The study uses the complete GPQA set; MMLU assesses breadth and depth of knowledge in an academic and professional testing context.Source: B. Details of Selected Benchmarks · p. 13
  • ZebraLogic (Lin et al., 2025) tests logical reasoning through Logic Grid Puzzles that require unique feature assignments for houses to be deduced from clues.The benchmark evaluates constraint-satisfaction problem solving, similar to reasoning tests such as the LSAT.Source: B. Details of Selected Benchmarks · p. 14
  • KorBench (Ma et al., 2024) assesses intrinsic reasoning and planning by using novel rules designed to minimize interference from pretrained knowledge.Its five categories—Operation, Logic, Cipher, Puzzle, and Counterfactual—each contain 25 manually defined rules.Source: B. Details of Selected Benchmarks · p. 14
  • LiveBench (White et al., 2024) aims to prevent test contamination with monthly releases of questions that have verifiable, objectively scorable answers.It comprises 18 tasks across six categories: reasoning, coding, mathematics, data analysis, language, and instruction following.Source: B. Details of Selected Benchmarks · p. 14
  • BBH-ZH translates nine BBH tasks into Chinese, covering tasks such as boolean expressions, logical deduction, object counting, and web of lies.The translated tasks include boolean expressions, date understanding, logical deduction with five objects, navigate, object counting, reasoning about colored objects, temporal sequences, tracking shuffled objects with five objects, and web of lies.Source: B. Details of Selected Benchmarks · p. 14
  • LeetCode-O tests code-output prediction from textual queries without supplying the function, unlike CRUXEval (Gu et al., 2024), which supplies a Python function without a textual query.The dataset excludes problems covered by training, balances 300 problems each across easy, medium, and hard, and provides English and Chinese versions. A problem earns credit only if all outputs are correct for all inputs in both languages; each typically has 2–3 test cases.Source: B. Details of Selected Benchmarks · pp. 13–14

C. Details of Processing Different Data Sources · C.1. Source Distribution · C.2. Input-Output Pairs for Each Source

The processing pipeline builds three code-source subsets, then creates source-specific numbers of input- and output-prediction instances from each.

C.1. Source Distribution

  • About 427K CodeMix files were retained by filtering for a DeepSeek-Coder-V2-Lite-Inst function-completion success rate of 10%–90%.CodeMix consists of curated raw Python files from an in-house code pre-training corpus; overly simple and excessively complex files are excluded.Source: §C.1 · p. 14
  • PyEdu-R contains about 369K files after classification removes algorithms and non-reasoning code, reducing overlap with CodeMix’s algorithm focus.Files were categorized using DeepSeek-Coder-V2-Lite-Inst and DeepSeek-V2; the source Python-Edu collection contains about 7.7M files.Source: §C.1 · p. 15
  • Other reputable sources, including algorithm repositories, mathematical problem collections, and online coding platforms, contribute about 14.5K code files.Source: §C.1 · p. 15

C.2. Input-Output Pairs for Each Source

  • CodeMix yields 1,674,345 instances from 300K samples, averaging 2.78 input- and 2.80 output-prediction instances per sample.At most 3 input-output pairs are selected per sample, yielding up to 3 instances of each prediction type.Source: §C.2 · p. 15
  • PyEdu-R yields 1,630,716 instances from 141K samples, averaging 5.77 input- and 5.79 output-prediction instances per sample.At most 6 input-output pairs are selected per sample, yielding up to 6 instances of each prediction type.Source: §C.2 · p. 15
  • Other sources yield 216,159 instances from 13.9K samples, averaging 7.70 input- and 7.87 output-prediction instances per sample.At most 10 input-output pairs are selected per sample, yielding up to 10 instances of each prediction type.Source: §C.2 · p. 15

C.3. The Effect of Using Different Sources

The ablation analysis finds that combining CodeMix and PyEdu-R yields the best overall performance, with their contributions varying by task type. PyEdu-R’s focus on complex calculations and relative lack of high-level reasoning flows may make it harder for models to learn from.

C.3. The Effect of Using Different Sources

  • Combining CodeMix and PyEdu-R yields the best overall performance, while removing CodeMix causes a larger decline than removing PyEdu-R in a similarly sized subset comparison.The source-specific effects vary by task: removing PyEdu-R reduces mathematical and scientific performance, whereas removing CodeMix has a greater negative impact on symbolic or logical tasks.Source: §C.3 · p. 15
  • PyEdu-R samples often emphasize complex floating-point calculations over high-level reasoning or problem-solving flows, making them challenging for models to learn from effectively.The authors suggest refining or cleaning PyEdu-R in future work to improve its learnability and utility.Source: §C.3 · p. 15

D. Detailed Statistics in Multi-turn Revision

Most correct predictions are made initially, and the benefit of revision diminishes rapidly; the main experiments therefore limit revision to one turn to save computing resources.

D. Detailed Statistics in Multi-turn Revision

  • Figure 7 shows most correct predictions arise initially; first-turn revisions fix approximately 16% of incorrect input predictions and 10% of incorrect output predictions, while second-turn gains diminish.The authors limit revision to one turn in the main experiments to save resources spent prompting DeepSeek-V2.5 and executing code for verification.Source: D. Detailed Statistics in Multi-turn… · p. 16

E. Training Hyper-parameters · F. The Effect of Using Other Instruction-Tuning Data

The method uses a two-stage training schedule with batch size 1024 and a maximum sequence length of 4096. On Qwen 2.5 Coder 7B, switching to Tulu-3 for stage two preserves a significant improvement over the single-stage baseline, although CODEI/O++ performs slightly worse than CODEI/O.

E. Training Hyper-parameters

F. The Effect of Using Other Instruction-Tuning Data

G. Examples Mentioned in the Main Text

The section illustrates CodeI/O’s unified-format conversion, response-collection prompt, and a CODEI/O++ training sample. The sample shows feedback enabling DeepSeek-V2.5 to revise an incorrect input prediction into a correct one on the second turn.

G. Examples Mentioned in the Main Text

  • In the CODEI/O++ sample, DeepSeek-V2.5 revises an incorrect initial input prediction into a correct prediction after feedback on turn two.The sample highlights the template and turn-one and turn-two feedback; the reference code is omitted from the prompt.Source: G. Examples Mentioned in the Main Text · p. 16, Table 11 · p. 19
  • Table 10 illustrates transforming a raw code file into the unified format, with an example question about deriving vehicle speed and displacement from vertical acceleration data and time points.The question specifies starting from initial speed and displacement.Source: Table 10 · p. 17, G. Examples Mentioned in the Main Text · p. 17
  • Figure 8 shows the response-collection and training prompt, whose input- and output-prediction sections largely match except for two gray blocks used as needed.The example asks for a feasible input without code and requires a JSON object whose input dictionary keys match the input variable names.Source: Figure 8 · p. 18, G. Examples Mentioned in the Main Text · p. 19
  • For this task, the output is the length of the shortest subarray whose sum is at least the target; an output of 4 therefore indicates a qualifying shortest length of 4.Source: G. Examples Mentioned in the Main Text · p. 19

Related papers

Loading 2502.07316v4…