PaperlayerView exact revision on arXiv

Source-linked AI summary

Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMs

Dayu Yang, Tianyang Liu, Daoan Zhang, Antoine Simoulin, Xiaoyi Liu, Yuwei Cao, Zhaopu Teng, Xin Qian, Grey Yang, Jiebo Luo, Julian McAuley

arXiv:2502.19411v1cs.CLcs.AIcs.LGcs.SE

TL;DR

Systematic review of how code and reasoning interact in LLMs remains limited. This survey examines code as a reasoning medium and reasoning’s role in code intelligence, concluding that code enhances logical reasoning while reasoning improves code intelligence.

  • Problem

    Systematic review of how code and reasoning interact and reinforce each other in LLMs remains limited.

    Source: §1 · p. 2

  • Method

    The survey analyzes code as a medium for structuring and validating reasoning, examines how reasoning expands code intelligence, and summarizes challenges and future directions.

    Source: §1 · p. 2

  • Results

    The survey concludes that code enhances logical reasoning while reasoning improves code intelligence, including generation, comprehension, and debugging.

    Source: §5 · p. 10

  • Takeaways & Limitations

    Future research must deepen the integration of reasoning and programming to build more robust, interpretable, and adaptive AI systems.

    Source: §5 · p. 10

    The survey’s focus on recent arXiv categories and a confined publication window excludes older or less mainstream work.

    Source: §6 · p. 10

Abstract

from arXiv · show

In large language models (LLMs), code and reasoning reinforce each other: code offers an abstract, modular, and logic-driven structure that supports reasoning, while reasoning translates high-level goals into smaller, executable steps that drive more advanced code intelligence. In this study, we examine how code serves as a structured medium for enhancing reasoning: it provides verifiable execution paths, enforces logical decomposition, and enables runtime validation. We also explore how improvements in reasoning have transformed code intelligence from basic completion to advanced capabilities, enabling models to address complex software engineering tasks through planning and debugging. Finally, we identify key challenges and propose future research directions to strengthen this synergy, ultimately improving LLM's performance in both areas.

1 Introduction

The introduction frames code and reasoning in LLMs as mutually reinforcing: code structures and validates reasoning, while stronger reasoning expands code intelligence. The survey examines this interaction, its challenges, and future research directions.

1 Introduction

  • Code’s structured syntax, execution feedback, and modular design support task decomposition, reasoning-chain construction, and self-reflection, while reasoning advances code intelligence.The introduction presents this as a bidirectional relationship between code properties and reasoning capabilities.Source: §1 · p. 1
  • Chain-of-Thought prompting (Wei et al., 2022b; Zhang et al., 2024b) and Self-Reflection (Shinn et al., 2024) help extend code generation beyond autocompletion toward software development assistants (Labs, 2024; Yang et al., 2024d) and lifecycle management (Jimenez et al., 2024).The introduction also identifies code comprehension and modification, debugging, and optimization as advances enabled by improved reasoning.Source: §1 · pp. 1–2
  • The survey addresses a gap in systematic reviews by asking how code representations influence reasoning, how reasoning advances reshape code intelligence, and what challenges arise from their interplay.Its coverage includes code as a reasoning medium, expanded code-intelligence capabilities, and challenges in interpretability, scalable training, and multimodal fusion, with future research directions.Source: §1 · p. 2

2 Code-enhanced Reasoning

Code supports reasoning through executable programs, code-language integration, and non-executable program structures; code training also strengthens reasoning and planning across domains and training strategies.

2.1 Generating as Code Aids Reasoning

  • Code-enhanced reasoning turns problems into programmatic solutions to improve precision and reliability on complex tasks.Source: §2.1 · p. 2

2.1.1 Single Execution

  • PoT and PaL turn numerical problems into executable programs that provide deterministic solution paths, but their benefits vary with code complexity and problem type.Chen et al. (2022); Gao et al. (2023) introduce the methods, while Bi et al. (2023) find code transformation may not universally improve reasoning. Kabra et al. (2023) report better calibration for code-based than pure-language solutions.Source: §2.1.1 · p. 2

2.1.2 Dynamic Code-Language Integration

2.1.3 Non-Executable Program Representations

  • Non-executable code structures scaffold reasoning when execution is impractical, from Chain of Code’s LMulator to code-like templates, execution traces, and code-form plans.These representations support mixed computational and contextual reasoning, conditional logic, and program-state tracking without actual execution; CodePlan uses structured pseudocode to improve coherence and interpretability for problems resistant to pure algorithms.Source: §2.1.3 · pp. 2–3

2.2 Training with Code

  • Code data strengthens LLM reasoning and planning by providing structured patterns that guide logical thinking.Source: §2.2 · p. 3

2.2.1 Empowering Reasoning and Planning Through Code Training

2.2.2 Training Strategies Based on Code

  • Code-only and hybrid-data training strategies use execution traces, preferences, and stage- or domain-specific code mixing to improve reasoning and code-specific skills.MARIO (Liao et al., 2024) adds Python interpreter traces to GSM8K and MATH, while POET (Pi et al., 2022) trains with programs and execution results. CodePMP (Yu et al., 2024b) and SIAM (Yu et al., 2024a) use preference modeling and code-based quality control; hybrid strategies find code effects vary by domain and favor domain-specific mixing.Source: §2.2.2 · pp. 4–5

3 Reasoning-Enhanced Code Intelligence

The section traces code intelligence from direct code generation to reasoning-integrated systems and autonomous agents. It presents planning, code understanding, and execution feedback as complementary capabilities for handling complex development tasks.

3.1 Essential Code Intelligence

  • Direct-generation models handle completion and synthesis, but lack of explicit reasoning limits them on complex design, edge cases, and systematic planning.Their capabilities include next-line prediction, fill-in-the-middle, and repository-level tasks such as RepoBench (Liu et al., 2023b) and CrossCodeEval (Ding et al., 2023).Source: §3.1 · p. 5

3.2 Integration of Reasoning Capabilities

  • Code intelligence draws on reasoning to code—planning and decomposition—and reasoning over code—understanding behavior and properties—which converge in interactive programming.Interactive systems reason about what to generate and analyze execution results to guide fixes, optimization, and capability expansion.Source: §3.2 · p. 5

3.2.1 Reasoning for Code Generation

  • CoT, instruction tuning, and reinforcement learning have advanced code systems from completion and dialogue toward reasoning engines for complex programming tasks.Programming-oriented CoT structures thoughts around constructs, reusable modules, debugging, and natural-language plans; approaches also extend to lightweight models and repository-level planning with dependency analysis and editing.Source: §3.2.1 · p. 5

3.2.2 Reasoning Over Code

  • Code understanding remains difficult because syntactic correctness does not ensure semantic understanding, and code combines static structure, runtime behavior, and external dependencies.NExT (Ni et al., 2024a) analyzes runtime traces, while SelfPiCo (Xue et al., 2024) simulates partial-code execution and tracks state; benchmarks span function execution, broad comprehension, code QA, repository contexts, and specification adherence.Source: §3.2.2 · p. 6

3.2.3 Interactive Programming

  • Interactive programming makes executable feedback part of an iterative reasoning loop: models generate code, inspect execution results, and refine solutions.Self-Edit (Zhang et al., 2023), CodeChain (Le et al., 2024), LeTI (Wang et al., 2024e), and OpenCodeInterpreter (Zheng et al., 2025) exemplify iterative correction, modular refinement, natural-language feedback, and unified generation-execution-refinement; effectiveness depends on reasoning about program behavior and feedback.Source: §3.2.3 · pp. 6–7

3.3 Code Agents with Complex Reasoning

  • Code agents decompose development tasks, plan tool-mediated actions, and monitor environmental feedback in an ongoing decision cycle.CodeAct (Wang et al., 2024c), OpenHands (Wang et al., 2024d), SWE-agent (Yang et al., 2024b), CodeAgent (Zhang et al., 2024c), and AutoCodeRover (Zhang et al., 2024e) illustrate differing implementations. HyperAgent (Phan et al., 2024) coordinates specialized agents, while Agentless (Xia et al., 2024a) uses two-phase fault localization and repair; both reduce code redundancy relative to humans, but human-level performance remains challenging. Persistent challenges include reasoning about program behavior, safe codebase modification, and long-term planning.Source: §3.3 · p. 7

4 Challenges and Future Directions

Code-enhanced reasoning faces challenges in interpretability, modality choice, data quality, scalability, and tool use. Reasoning-enhanced code intelligence must improve repository understanding, long-form generation, agent capabilities, multimodal workflows, and evaluation while maintaining reliability and safety.

4.1 Code-enhanced Reasoning

  • Generated code may not reflect a model’s true chain of thought or reliable confidence, motivating systematic error handling, self-debugging, and formal verification.Proposed approaches include tree-based generation, reasoning-oriented self-reflection, reinforcement learning, and verification that checks code correctness and alignment with intended reasoning.Source: §4.1 · pp. 7–8
  • Code suits precise computation but can falter on subjective or abstract tasks, making hybrid code-language reasoning and context-aware selection of reasoning modes important.Interleaving executable code with textual interpretation can support tasks requiring both algorithmic precision and nuanced judgment.Source: §4.1 · p. 8
  • Code-enhanced reasoning needs calibrated code complexity, broader zero- and few-shot and cross-domain generalization, and cleaner, more diverse datasets beyond narrow task-specific training.Cyclomatic complexity and code length could guide complexity choices, while adaptive curricula, synthetic data, and unsupervised learning are proposed to improve training.Source: §4.1 · p. 8
  • A standardized code format for automated tool invocation and execution could help LLMs and agents construct complete tool-use chains for complex tasks.Current tool use typically relies on APIs or simple code, while sophisticated, complete tool-use chains remain challenging.Source: §4.1 · pp. 8–9

4.2 Reasoning-enhanced Code Intelligence

  • Repository-scale understanding requires balancing context expansion, retrieval, and precise parsing, while long-form generation needs stronger multi-file evaluation, coherence, correctness, and executability.Longer input contexts do not necessarily improve comprehension or long-form code output; errors can accumulate until a project becomes non-functional or logically inconsistent.Source: §4.2 · p. 9
  • Whether reasoning models improve code agents remains uncertain, and agent autonomy must be balanced with predictable behavior and safeguards against unsafe execution.Research should test new agent frameworks and reasoning-model capabilities in real-world programming scenarios, while providing safe execution environments that limit vulnerabilities and harmful actions.Source: §4.2 · p. 9
  • Future code intelligence should support visual-to-code workflows and graphical interface interaction, while reinforcement learning can use execution feedback to improve code reasoning and agent strategies.Proposed directions include datasets linking designs to code and trial-and-error refinement of code search, debugging, and refactoring strategies.Source: §4.2 · pp. 9–10
  • Evaluation frameworks should reflect real-world software challenges, resist data contamination, and assess code quality, maintainability, and design aesthetics beyond functional correctness.New benchmarks remain necessary as code intelligence models master existing ones.Source: §4.2 · p. 10

5 Conclusion

The survey describes a reciprocal synergy: code enhances reasoning, while reasoning advances code intelligence. It highlights unresolved challenges and calls for deeper integration of reasoning and programming.

5 Conclusion

  • Executable programs and structured code paths refine AI reasoning, while reasoning enables advanced code generation, comprehension, and debugging.Together, code and reasoning have driven significant advancements in AI.Source: §5 · p. 10
  • Ambiguity, scalability, and consistency remain challenges despite progress in code and reasoning.Source: §5 · p. 10
  • Future research must deepen reasoning-programming integration to build more robust, interpretable, and adaptive AI systems.As the fields converge, AI’s ability to think and code will continue to evolve, reshaping intelligent automation.Source: §5 · p. 10

6 Limitations

The survey’s breadth and publication scope limit its level of detail and exclude some earlier or less mainstream work. Its benchmark coverage is incomplete for repository-scale and specialized tasks, and its identified research gaps may change as the field develops.

6 Limitations

  • Covering approaches from single-execution reasoning to autonomous code agents requires implementation details and domain-specific nuances to be described only briefly.Source: §6 · p. 10
  • Focusing on recent arXiv categories and a confined publication window excludes older or less mainstream work that could provide alternative perspectives or historical context.Source: §6 · p. 10
  • Coverage of CRUXEval, CodeMMLU, RepoQA, and SWE-bench remains incomplete for real-world repository-scale tasks and specialized areas such as concurrency analysis and security verification.Source: §6 · p. 10
  • The challenges identified are ongoing research gaps rather than definitive conclusions, and future datasets, model architectures, and evaluation protocols may prompt survey revisions or expansions.Source: §6 · p. 10

A Technical Introduction for Important Methods · A.1 Code-enhanced Reasoning

Code-enhanced reasoning interleaves language with executable or pseudo-executable code, using task decomposition and external execution or model-based emulation to support precise, multi-step problem solving. PaL, PoT, MathCoder, and CoC illustrate complementary ways to combine textual reasoning with code.

A.1 Code-enhanced Reasoning

  • These approaches pair textual explanations with code to decompose tasks while delegating precise computations to interpreters or simulators.This scaffolding mechanism is described as supporting complex reasoning.Source: §A.1 · p. 19
  • PaL (Gao et al., 2023) delegates arithmetic and logical operations to an interpreter, grounding multi-step answers in verified execution outputs.The model emits natural-language reasoning alongside program statements, leaving higher-level step-by-step reasoning to the model rather than raw calculation.Source: §A.1 · p. 19
  • PoT (Chen et al., 2022) encodes substeps in semantically meaningful variables and partial code, then executes the code externally to produce numerical results.Breaking complex computations into small, interpretable snippets supports transparent and robust multi-step reasoning.Source: §A.1 · p. 19
  • MathCoder (Lu et al., 2024) alternates language rationales and code blocks, executing each snippet and feeding its output into the ongoing reasoning process.This iterative loop combines code-based calculation with textual reasoning for intricate math problems.Source: §A.1 · p. 19
  • CoC (Li et al., 2023a) directly executes code when possible and lets the model emulate segments that a standard interpreter cannot run.The model simulates non-executable code’s effect while maintaining a consistent state.Source: §A.1 · p. 19
  • CoC (Li et al., 2023a) combines symbolic execution with language-driven inference for tasks involving logical, numerical, and semantic reasoning.Actual code execution handles operations such as arithmetic, while model emulation handles segments that cannot run in a standard interpreter.Source: §A.1 · p. 19

A.2 Training with Code · A.3 Reasoning-enhanced Code Intelligence

Code-based training uses execution, quality checks, preferences, and structured plans to strengthen reasoning. Reasoning-enhanced code intelligence applies planning, iterative validation and refinement, and tool- or agent-based workflows to complex code tasks.

A.2 Training with Code

  • MARIO uses curated math data and a Python interpreter for exact calculations and error checks, while POET trains on programs and execution results to support stepwise inference and validation (Liao et al., 2024; Pi et al., 2022).MARIO reports improved GSM8K and MATH performance; POET instantiations cover numerical, logical, and multi-hop reasoning.Source: §A.2 · p. 20
  • CodePMP learns from synthesized code-preference pairs for reinforcement learning, while SIAM uses a code critic and self-generated instruction and preference data to improve math reasoning and generalization (Yu et al., 2024b; Yu et al., 2024a).The reported SIAM improvements span in-domain and out-of-domain math benchmarks and multiple languages.Source: §A.2 · p. 20
  • CODEPLAN uses structured pseudocode plans as intermediate representations, reporting gains across mathematical, symbolic, multi-hop question-answering, and decision-making tasks (Bairi et al., 2023).It is trained on paired plan-response examples and described as data-efficient and lightweight.Source: §A.2 · p. 20

A.3 Reasoning-enhanced Code Intelligence

B Introduction of Important Benchmarks · B.1 Code-enhanced Reasoning

This section characterizes three datasets that assess code-enhanced mathematical reasoning through complementary emphases: multi-step elementary reasoning, advanced competition problem solving, and robustness to variations in word problems.

B.1 Code-enhanced Reasoning

  • GSM8K contains 8.5K grade-school math word problems requiring 2–8 reasoning steps (Cobbe et al., 2021b).Its linguistically diverse problems rely on elementary mathematics and emphasize multi-step deduction over complex mathematical knowledge; natural-language solutions show the reasoning step by step.Source: §B.1 · p. 23
  • GSM8K tests multi-step deductive reasoning rather than complex mathematical knowledge, using problems with high linguistic diversity (Cobbe et al., 2021b).Its natural-language solutions explicitly demonstrate the step-by-step reasoning process.Source: §B.1 · p. 23
  • MATH comprises 12,500 competition problems from sources including AMC 10, AMC 12, and AIME (Hendrycks et al., 2021c).Unlike GSM8K, it requires more sophisticated problem-solving heuristics and domain knowledge, with detailed solutions demonstrating mathematical reasoning and domain-specific strategies.Source: §B.1 · p. 23
  • MATH’s detailed step-by-step solutions demonstrate mathematical reasoning alongside domain-specific problem-solving strategies (Hendrycks et al., 2021c).Source: §B.1 · p. 23
  • SVAMP is a 1,000-problem challenge set designed to test math word-problem solvers’ reasoning robustness (Patel et al., 2021).It maintains mathematical complexity similar to existing datasets while varying problem formulations.Source: §B.1 · p. 23
  • SVAMP varies question sensitivity, reasoning ability, and structural invariance to test understanding, adaptation, and consistency under superficial changes (Patel et al., 2021).The variations test whether models understand changed questions, adapt to subtle changes requiring different reasoning paths, and remain consistent across superficial changes.Source: §B.1 · p. 23

B.2 Training with Code

The section surveys benchmarks that test code-enhanced reasoning across open-ended problems, evidence chaining, logical and numerical inference, instruction following, dialogue, and interactive decision-making. Their varied tasks emphasize structured, stepwise reasoning and, in some cases, programmatic validation.

B.2 Training with Code

  • OCW (Lewkowycz et al., 2022) tests open-ended math, algorithmic puzzles, or coding snippets; APE (Zhao et al., 2020) asks for justified arithmetic or algorithmic steps.Both emphasize reasoned, stepwise solutions rather than unsupported answers.Source: §B.2 · pp. 23–24
  • HotpotQA (Yang et al., 2018) tests multi-hop evidence chaining, while LogiQA (Liu et al., 2020) targets deductive and inductive inference in reading comprehension.Both assess whether models can connect facts or analyze logical structures to reach coherent conclusions.Source: §B.2 · p. 24
  • DROP (Dua et al., 2019) evaluates arithmetic, entity tracking, and multi-step symbolic derivation, where program-like parsing and computation can support answers.Its tasks require numerical or symbolic manipulation over text.Source: §B.2 · p. 24
  • MathShepherd-pair (Wang et al., 2024b), ReClor-pair (Yu et al., 2020), and LogiQA2.0-pair assess systematic comparison or validation of mathematical steps and logical arguments.Their paired formats foreground checking correctness, consistency, and subtle logical cues.Source: §B.2 · p. 24
  • CMATH (Wei et al., 2023b) tests language-specific parsing and reasoned mathematical steps, including non-English contexts such as Chinese.Programmatic solution paths can help handle textual variation while solving the math tasks.Source: §B.2 · p. 24
  • AlpacaEval-2 evaluates instruction following with structured reasoning, while MT-Bench (Zheng et al., 2023) tests coherence and reasoning across multi-turn dialogue.Code-infused or planning-based approaches may support stepwise instruction comprehension and logical continuity.Source: §B.2 · p. 24
  • ALFWorld (Shridhar et al., 2020) tests sequential decisions in interactive text environments, requiring agents to use environmental cues and plan multi-step goals.The tasks involve cause-and-effect reasoning and strategies resembling small programs or scripts.Source: §B.2 · pp. 24–25

B.3 Reasoning-enhanced Code Intelligence · C Paper Collection

The section surveys code-intelligence benchmarks spanning functional correctness, reasoning, and repository-level challenges, and describes a systematic search and screening process that produced a 63-paper collection. Together, these passages outline evaluation coverage and the literature-selection method.

B.3 Reasoning-enhanced Code Intelligence

  • HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), and APPS (Hendrycks et al., 2021a) use automated tests to assess code generation from natural-language tasks, from basic programming to broad algorithmic challenges.HumanEval uses 164 handcrafted problems and pass@k; MBPP contains approximately 1,000 Python problems, and APPS around 10,000 problems.Source: §B.3 · p. 25
  • DS-1000 (Lai et al., 2022) evaluates data-science code generation, while CRUXEval (Gu et al., 2024) tests reasoning and execution behavior on short Python functions.CRUXEval uses input prediction for code reasoning and understanding, and output prediction for execution behavior.Source: §B.3 · pp. 25–26
  • RepoBench (Liu et al., 2023c) and CrossCodeEval (Ding et al., 2023) assess cross-file retrieval and completion, with CrossCodeEval covering four languages and real-world repositories.RepoBench separates retrieval, completion, and pipeline tasks; CrossCodeEval selects cases where multiple files are necessary for accurate completion.Source: §B.3 · p. 25
  • BigCodeBench (Zhuo et al., 2024) targets large-scale, heterogeneous coding scenarios, while LiveCodeBench (Jain et al., 2024) adds continuously collected unseen problems and evaluates self-repair, execution, and test-output prediction.The benchmarks address real-world coding complexity and contamination-resistant evaluation, respectively.Source: §B.3 · pp. 25–26
  • RepoQA (Liu et al., 2024c) tests long-context code search across 500 tasks from 50 repositories in five languages, measuring retrieval accuracy and multi-file comprehension.Its needle-in-a-haystack setup requires locating specific snippets in extensive code context.Source: §B.3 · p. 26
  • SWE-bench (Jimenez et al., 2024) evaluates fixes for real GitHub issues; its multimodal extension adds visual inputs, while SWE-bench Verified screens ambiguous tasks and provides reliable tests.The tasks require patch generation in multi-file repositories; the multimodal version focuses on JavaScript front-end work, and Verified uses human-validated instances.Source: §B.3 · p. 26

C Paper Collection

  • The paper collection searched post-January 2021 arXiv papers in cs.AI and cs.CL using code/program and reasoning/planning terms, then manually screened 110 results to retain 63 papers.The screening considered relevance, technical depth, and contributions to code reasoning.Source: C Paper Collection · p. 26

D Additional Tables and Figures

This section adds a taxonomy figure and two benchmark tables covering code-aided reasoning and reasoning-enhanced code intelligence. The captions describe each table’s scope and reporting conventions but do not provide individual result values.

D Additional Tables and Figures

  • Table 4 reports the percentage of problems solved correctly by code-aided reasoning methods across multiple benchmarks.A dash indicates no reported result; Direct and CoT results are from Chen et al. (2022).Source: Table 4 · p. 27
  • Figure 3 presents a full taxonomy illustrating the interplay between code and reasoning.Source: Figure 3 · p. 27
  • Table 5 reports reasoning-enhanced code intelligence methods across multiple benchmarks, with HumanEval and MBPP scored using pass@1.Its caption identifies result sources and qualifiers, including execution feedback without human involvement and results from the official SWE-bench leaderboard accessed Feb 15, 2025.Source: Table 5 · p. 28

Related papers

Loading 2502.19411v1…