PaperlayerView exact revision on arXiv

Source-linked AI summary

CodeI/O: Condensing Reasoning Patterns via Code Input-Output Prediction

Junlong Li, Daya Guo, Dejian Yang, Runxin Xu, Yu Wu, Junxian He

arXiv:2502.07316v4cs.CLcs.AI

TL;DR

Many reasoning domains lack the structured, scalable supervision available for math and code. CodeI/O converts executable functions into natural-language input-output prediction tasks and reports consistent cross-domain gains, with CodeI/O++ further improving data through verified revisions.

  • Problem

    Logical, scientific, symbolic, and other reasoning domains have sparse and fragmented supervision compared with math and code.

  • Method

    CodeI/O trains models to predict code inputs or outputs from executable functions and queries using natural-language Chain-of-Thought rationales, with CodeI/O++ adding execution-feedback revision.

  • Results

    CodeI/O and CodeI/O++ consistently outperform existing baselines across diverse reasoning benchmarks and models, with balanced improvements across domains.

  • Takeaways & Limitations

    Code-derived input-output prediction provides a scalable way to expose models to diverse structured reasoning patterns beyond code-specific tasks.

  • Takeaways & Limitations

    LiveBench evaluation excludes code-generation and instruction-following subtasks because they are not the study’s focus.

Abstract

from arXiv · show

Reasoning is a fundamental capability of Large Language Models. While prior research predominantly focuses on enhancing narrow skills like math or code generation, improving performance on many other reasoning tasks remains challenging due to sparse and fragmented training data. To address this issue, we propose CodeI/O, a novel approach that systematically condenses diverse reasoning patterns inherently embedded in contextually-grounded codes, through transforming the original code into a code input-output prediction format. By training models to predict inputs/outputs given code and test cases entirely in natural language as Chain-of-Thought (CoT) rationales, we expose them to universal reasoning primitives -- like logic flow planning, state-space searching, decision tree traversal, and modular decomposition -- while decoupling structured reasoning from code-specific syntax and preserving procedural rigor. Experimental results demonstrate CodeI/O leads to consistent improvements across symbolic, scientific, logic, math & numerical, and commonsense reasoning tasks. By matching the existing ground-truth outputs or re-executing the code with predicted inputs, we can verify each prediction and further enhance the CoTs through multi-turn revision, resulting in CodeI/O++ and achieving higher performance. Our data and models are available at https://github.com/hkust-nlp/CodeIO.

1. Introduction

Reasoning data is abundant for math and code but sparse and fragmented for many other domains. CODEI/O addresses this gap by converting executable code into natural-language input-output prediction tasks that expose diverse reasoning patterns while preserving procedural rigor.

  • Math and code benefit from abundant structured supervision, whereas logical, scientific, and symbolic reasoning often rely on sparse and fragmented signals.
  • CODEI/O transforms functions and textual queries into natural-language tasks that predict outputs from inputs or feasible inputs from outputs.
  • The approach separates reasoning flow from code-specific syntax while retaining logical rigor, enabling patterns learned from diverse code contexts.
  • Training uses a two-stage setup before general instruction tuning, with responses collected from DeepSeek-V2.5 and many executable input-output pairs generated per function.
  • Across 14 benchmarks and models from 7B to 30B parameters, CODEI/O improves diverse reasoning abilities beyond code-related tasks.
  • CODEI/O achieves balanced and generalizable reasoning abilities across all benchmarks.

2. CODEI/O

CODEI/O constructs executable, unified functions from diverse raw code, samples deterministic input-output pairs, and trains on natural-language reasoning responses. CODEI/O++ further uses execution feedback to revise incorrect predictions.

  • The data pipeline collects raw code, converts it into a unified format, samples input-output pairs, and assembles the training dataset.
  • CodeMix and PyEdu-R provide complementary sources spanning general Python code and complex STEM, system-modeling, and logic tasks.
  • 2.2. Transforming to a Unified Format: DeepSeek-V2.5 preprocesses irregular code into executable functions centered on core logical functionality, excluding non-essential visualization and file-processing elements.
  • 2.2. Transforming to a Unified Format: The coin-change example frames a function as predicting the minimum coin count or infinity when the target cannot be formed.
  • 2.3. Sampling I/O Pairs: Inputs are sampled with a generator and outputs are obtained by execution, while random functions and executions exceeding runtime or object-complexity limits are excluded.
  • 2.4. Constructing the Training Dataset: Training prompts combine the function, query, reference code, and an input or output, with natural-language CoTs explaining the predicted result or feasible input.
  • 2.4. Constructing the Training Dataset: CODEI/O uses DeepSeek-V2.5 to synthesize responses because deterministic reverse functions are impractical and fixed execution trajectories lack expressive natural-language reasoning.
  • 2.4. Constructing the Training Dataset: CODEI/O++ appends execution feedback to incorrect predictions and generates revised responses, retaining the multi-turn reasoning record.

3. Experiments

Experiments use two-stage training across varied models, datasets, and reasoning benchmarks. CODEI/O delivers broad, consistent gains, while CODEI/O++ further improves average performance through execution-feedback revision, with some tasks unchanged or lower.

  • 3.1. Settings: Experiments cover coder and general-purpose backbones ranging from 7B to 27B parameters, including Qwen, DeepSeek, LLaMA, and Gemma models.
  • 3.1. Settings: Models are generally trained first on CODEI/O or CODEI/O++ and then on approximately 1.18M general instruction-tuning samples.
  • 3.1. Settings: The two-stage design avoids the larger CODEI/O dataset dominating mixed training and preserves learning on diverse instruction-following data.
  • 3.1. Settings: Evaluation spans science, math and numerical, symbolic, commonsense, logic, and code-understanding benchmarks, with BBH-EN and BBH-ZH evaluated using three shots.
  • 3.1. Settings: The LiveBench evaluation excludes code-generation and instruction-following subtasks, which are outside the study’s focus.
  • 3.1. Settings: Table 1 compares CODEI/O and baselines using benchmark scores, training-set sizes, and color-coded changes relative to the single-stage baseline.
  • 3.2. Main Results: CODEI/O provides universal gains across benchmarks and improves non-code tasks despite using code-centric data, unlike baselines that often specialize or regress.
  • 3.2. Main Results: Raw PythonEdu training yields minor or negative improvements and underperforms CODEI/O, indicating that structured training tasks matter beyond dataset size.

4. Analysis

The analysis finds that performance depends on data scale, prediction format, revision, training stages, and leakage control. Results favor diverse CodeI/O data, natural-language CoTs, and two-stage training.

  • Ablations: Input and output prediction have generally similar scores, but input prediction favors KorBench while output prediction benefits symbolic reasoning tasks such as BBH.The two prediction directions show benchmark-specific strengths rather than a universal winner.
  • Ablations: Filtering incorrect responses by rejection sampling removes 50% of training data and causes a general performance drop, while replacing them with execution-derived answers helps some output-prediction benchmarks but lowers scores elsewhere.The results suggest that preserving data diversity matters for average performance.
  • Scaling: Increasing training samples generally improves benchmark performance, with the full dataset producing the most comprehensive results.Intermediate data amounts fall between the smallest and full-data settings.
  • Scaling: Using more input-output pairs also improves performance, especially when increasing coverage from 1/6 to 6/6.The weaker scaling effect suggests some reasoning patterns require multiple test cases to capture complex logic flow.
  • Data format: Placing the query and reference code in the prompt and CoT in the response achieves the highest average score and most balanced benchmark performance.Formats resembling standard code generation perform worst when the reference code is placed in the response.
  • Training strategy: Two-stage training consistently outperforms single-stage training, although the best data-mixing strategy varies across models.The main experiments therefore keep CodeI/O and instruction-tuning data fully separated.
  • Data leakage: Similar accuracy gains on full and non-leaked subsets indicate that improvements on LeetCode-O and KorBench are not attributable to data leakage.Manual inspection found general-description overlap in KorBench and sibling-problem overlap in LeetCode-O despite removing original problems.

5. Related Work

The paper positions CodeI/O as distinct from prior code-execution learning and inference-time scaling. Its contribution is large-scale code input-output prediction aimed at general reasoning beyond code.

  • Code execution: Prior code-execution studies mainly target output prediction or use execution feedback within narrow code-related tasks.The paper contrasts this scope with its broader reasoning objective.
  • Code execution: CodeI/O trains language models on large-scale, diverse code input-output predictions and evaluates benefits beyond code-related tasks.The paper presents this as its distinction from earlier code-execution work.
  • Inference-time scaling: CodeI/O is described as orthogonal to inference-time scaling methods that use long reasoning processes and reinforcement learning.The supplied passage introduces this relationship without reporting a direct comparison.

6. Conclusion

The paper concludes that CodeI/O trains reasoning through natural-language prediction of code inputs and outputs. CodeI/O and CodeI/O++ improve performance broadly while maintaining balanced results across reasoning domains.

  • Contribution: CodeI/O trains models to predict code inputs and outputs using pure natural-language Chain-of-Thought reasoning.The approach uses structured code to expose diverse symbolic, logical, mathematical, and commonsense reasoning patterns.
  • Results: CodeI/O and CodeI/O++ consistently outperform existing baselines across benchmarks without sacrificing performance in any domain.The conclusion characterizes the gains as balanced, robust, and versatile.
  • Data construction: The data pipeline constrains executed samples to a 5-second runtime and bounded input-output complexity.The stated limits include objects below 1024 bytes, collections shorter than 20 elements, and strings no longer than 100 characters.

B. Details of Selected Benchmarks

The evaluation suite spans reasoning benchmarks across commonsense, numerical, mathematical, scientific, symbolic, logical, coding, and bilingual tasks. It also includes newly constructed benchmarks for Chinese reasoning and code-output prediction.

  • The benchmark suite covers commonsense, numerical, mathematical, scientific, symbolic, logical, and code-understanding reasoning domains.
  • The evaluation includes established benchmarks such as BBH, CRUXEval, ZebraLogic, and the broader LiveBench and KorBench suites.
  • KorBench evaluates adaptation to novel rules across operation, logic, cipher, puzzle, and counterfactual tasks.
  • LiveBench uses newly released questions with verifiable answers to reduce test-set contamination and enable automatic scoring.
  • BBH-ZH translates nine BIG-Bench Hard tasks into Chinese for bilingual evaluation.
  • LeetCode-O tests bilingual code-output prediction from textual queries and test cases without providing the underlying function.

C. Details of Processing Different Data Sources

The data sources combine curated raw Python code, reasoning-focused educational code, and additional reputable repositories. These sources are converted into input-output prediction instances at different scales.

  • CodeMix retains approximately 427K Python files whose function-completion success rates fall between 10% and 90%.
  • PyEdu-R contains approximately 369K reasoning-focused files after excluding algorithmic and non-reasoning content from Python-Edu.
  • Other sources contribute approximately 14.5K code files from algorithm repositories, mathematical problem collections, and coding platforms.
  • CodeMix yields 1,674,345 input-output prediction instances from 300K filtered samples.
  • PyEdu-R yields 1,630,716 prediction instances from 141K filtered samples.
  • Other sources yield 216,159 prediction instances from 13.9K filtered samples.

C.3. The Effect of Using Different Sources

The ablations show that the two principal data sources contribute different reasoning strengths, while combining them produces the strongest overall performance. Multi-turn revision improves data quality, but its benefit diminishes rapidly after the first revision.

  • Removing PyEdu-R reduces performance on mathematical and scientific benchmarks, whereas removing CodeMix more strongly harms symbolic and logical tasks.
  • Combining CodeMix and PyEdu-R yields the best overall performance across the compared source configurations.
  • PyEdu-R emphasizes complex floating-point calculations more than high-level reasoning or problem-solving flows.
  • The second training stage can be switched to Tulu-3, as examined in the reported performance comparison.

D. Detailed Statistics in Multi-turn Revision

Most correct predictions are produced on the initial turn, while the first revision repairs a meaningful fraction of incorrect predictions and later revisions contribute much less. The main experiments therefore use one revision turn to limit computation.

  • Approximately 16% of incorrect input predictions and 10% of incorrect output predictions are revised during the first revision turn.
  • The second revision turn revises significantly fewer predictions, indicating rapidly diminishing benefits from additional revisions.
  • The main experiments limit revision to one turn to reduce prompting and code-execution costs.

E. Training Hyper-parameters

Training uses distinct two-stage schedules, with model-dependent learning rates and a fixed batch size. Experiments on Tulu-3 indicate CODEI/O remains effective when the second-stage dataset changes.

  • Batch size is 1024 in both training stages.
  • The first stage runs for 1 epoch with learning rates of 1e-5 for smaller models and 4e-6 for Gemma 2 27B.
  • The second stage runs for 700 steps, approximately 3 epochs, using cosine-decayed learning rates.The rates start at 3e-5 for smaller models and 1e-5 for Gemma 2 27B, decaying to 1e-6 and 3e-7, respectively.
  • Switching the second-stage data to Tulu-3 still yields a significant improvement over the single-stage baseline.

G. Examples Mentioned in the Main Text

The examples illustrate how raw code and prompts are converted into unified CodeI/O training samples, including natural-language input prediction and iterative revision. A worked example constructs an input whose shortest qualifying subarray has length 4.

  • Iterative revision: The example includes an incorrect initial response followed by feedback, revised reasoning, and a successful regenerated response.The revision seeks a length-4 subarray summing to at least 10 and verifies the final result.
  • Examples and data format: Raw code is transformed into a unified format, with Figure 8 showing the prompt used for response collection and training.Table 10 documents the raw-code transformation, while Table 11 presents a complete CODEI/O++ input-prediction sample.
  • Input prediction: The input-prediction prompt asks for a feasible input in JSON without writing code, using keys that match the input variables.
  • Worked input-prediction example: For the worked task, the target is 10 and the desired output is 4, meaning the shortest qualifying contiguous subarray has length 4.
  • Worked input-prediction example: The candidate list [1, 2, 3, 4, 5] contains a length-4 subarray, [2, 3, 4, 5], whose sum is 14 and exceeds the target.
  • Worked input-prediction example: A validated feasible input uses target 10 and numbers [1, 2, 3, 4, 5], satisfying the shortest-subarray condition.The example explicitly states that this input produces output 4.
Loading 2502.07316v4…