Source-linked AI summary
Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMs
Dayu Yang, Tianyang Liu, Daoan Zhang, Antoine Simoulin, Xiaoyi Liu, Yuwei Cao, Zhaopu Teng, Xin Qian, Grey Yang, Jiebo Luo, Julian McAuley
TL;DR
The survey addresses limited systematic review of how code and reasoning interact in LLMs and organizes this relationship across reasoning and code intelligence. It examines code-based reasoning, reasoning-driven code systems, and open challenges, concluding that the synergy has advanced code generation, comprehension, debugging, and autonomous development while remaining constrained by interpretability, scalability, ambiguity, and survey coverage.
Problem
Limited systematic review has examined how code representations influence LLM reasoning, how reasoning reshapes code intelligence, and what challenges arise from their interplay.
Method
The survey analyzes code as a reasoning medium, reasoning-enhanced code intelligence, and open problems in interpretability, scalable training, and multimodal fusion.
Results
The survey concludes that code enhances logical reasoning while reasoning enables advanced code generation, comprehension, debugging, and autonomous code agents.
Takeaways & Limitations
The code–reasoning synergy supports more robust, interpretable, and adaptive AI systems as reasoning and programming continue to converge.
Takeaways & Limitations
The survey’s confined publication window and incomplete benchmark coverage exclude some older, less mainstream, repository-scale, concurrency, and security-verification work.
Abstract
from arXiv · showhide
In large language models (LLMs), code and reasoning reinforce each other: code offers an abstract, modular, and logic-driven structure that supports reasoning, while reasoning translates high-level goals into smaller, executable steps that drive more advanced code intelligence. In this study, we examine how code serves as a structured medium for enhancing reasoning: it provides verifiable execution paths, enforces logical decomposition, and enables runtime validation. We also explore how improvements in reasoning have transformed code intelligence from basic completion to advanced capabilities, enabling models to address complex software engineering tasks through planning and debugging. Finally, we identify key challenges and propose future research directions to strengthen this synergy, ultimately improving LLM's performance in both areas.
1 Introduction
Code and reasoning form a bidirectional relationship in LLMs: code structures and validates reasoning, while improved reasoning expands code intelligence. The survey addresses how this interaction works, what it enables, and which challenges remain.
- Code–reasoning synergy: Code’s structured syntax, execution feedback, and modular design support task decomposition, reasoning-chain construction, and self-reflection.These properties provide a structured medium for organizing and checking reasoning.
- Code–reasoning synergy: Improved reasoning advances code intelligence from task decomposition and code understanding to debugging, optimization, and end-to-end software development.The survey links reasoning improvements to increasingly capable software systems.
- Survey scope: The survey addresses three questions: how code representations influence reasoning, how reasoning reshapes code intelligence, and what challenges arise from their interplay.These questions define the survey’s organizing problem.
- Survey scope: Its framework analyzes code as a reasoning medium, reasoning-enhanced code intelligence, and open problems in interpretability, scalable training, and multimodal fusion.The survey also proposes future research directions across these dimensions.
2 Code-enhanced Reasoning
Code-enhanced reasoning uses executable programs, hybrid code-language representations, and code-like structures to organize problem solving. Training with code further supports reasoning, planning, and decision-making across domains, although execution is not always suitable for ambiguous tasks.
- Generating as Code Aids Reasoning: Program of Thoughts and program-aided language models express numerical reasoning as executable programs, providing deterministic solution paths and reducing calculation errors.These approaches differ from natural-language Chain-of-Thought by representing the reasoning process as a self-contained program.
- Generating as Code Aids Reasoning: Hybrid methods interleave code with explanatory text, while execution feedback guides natural-language reasoning and iterative refinement.REPL-based and workflow approaches address code-language integration through runtime results.
- Non-Executable Program Representations: Code-like representations can organize ambiguous or abstract reasoning without execution, including structured pseudocode that improves coherence and interpretability.Code-form plans provide programmatic organization when pure algorithmic solutions are unsuitable.
- Training with Code: Code training strengthens reasoning and planning across commonsense, mathematics, decision-making, visual structure, multilingual tasks, and autonomous driving.The surveyed applications use code-derived structure or execution to support capabilities beyond direct code generation.
- Training with Code: Execution traces and programs can improve natural-language reasoning, while code instructions during instruction tuning improve code-specific skills.The survey distinguishes benefits associated with code execution data and with the stage at which code data is added.
3 Reasoning-Enhanced Code Intelligence
Reasoning has transformed code intelligence from completion and synthesis toward planning, code understanding, interactive refinement, and autonomous agents. These systems can manage complex development workflows, but repository-scale reasoning and human-level performance remain challenging.
- Evolution of Code Intelligence: Code intelligence evolved from sequence prediction and program synthesis toward systems combining dialogue, planning, debugging, and reasoning.The surveyed progression includes code completion, dialogue applications, and sophisticated reasoning engines.
- Reasoning for Code Generation: CoT-based programming strategies decompose problems, organize logic around program constructs, modularize solutions, and guide implementation with natural-language plans.These strategies support generation, iterative refinement, and debugging.
- Reasoning Over Code: Reasoning over code remains difficult because models must combine static syntax, dynamic execution, and external dependencies; syntactic correctness can coexist with semantic misunderstanding.This creates a gap between surface manipulation and actual code understanding.
- Interactive Programming: Interactive programming forms an optimization loop in which models generate code, execute it, receive objective feedback, and refine their outputs.Execution feedback triggers additional reasoning cycles and supports error correction.
- Code Agents with Complex Reasoning: Autonomous code agents decompose tasks, act through tools, monitor execution states, and evaluate outcomes throughout a dynamic development cycle.This distinguishes them from static code generators.
- Code Agents with Complex Reasoning: Multi-agent and agent-free approaches both reduce code redundancy relative to humans, but matching human-level performance remains challenging.The passage identifies effective task decomposition as important to success.
4 Challenges and Future Directions
The survey identifies interpretability, modality, scalability, context, and agent-performance challenges that constrain code-enhanced reasoning and reasoning-enhanced code intelligence. It proposes debugging, hybrid code-language systems, broader data, improved repository understanding, and new agent frameworks as future directions.
- Code-enhanced Reasoning: Generated code may not faithfully expose an LLM’s reasoning, while manual inspection is costly, motivating systematic error detection, self-reflection, debugging, and formal verification.The survey also notes that models may not reliably assess their own confidence.
- Code-enhanced Reasoning: Code is effective for precise computation but inadequate for some commonsense, semantic, and subjective tasks, requiring a deliberate split between executable code and free-form language.Interleaving code generation with textual reasoning is presented as one route to combining symbolic computation with natural-language interpretation.
- Code-enhanced Reasoning: Narrow-domain training data restricts real-world applicability, so zero- and few-shot learning, synthetic data, unsupervised learning, and cross-domain training are proposed to broaden generalization.The survey specifically identifies mathematical calculation and code manipulation as examples of narrow domains.
- Reasoning-enhanced Code Intelligence: Repository-scale understanding remains difficult despite context windows exceeding 1 million tokens, because models may miss dispersed information and retrieval may not align with complex code structures.The survey calls for balancing context expansion, retrieval augmentation, and precise parsing for global comprehension and accurate localization.
- Reasoning-enhanced Code Intelligence: Reasoning models have shown limited improvement on agent-based tasks, motivating new agent frameworks, better reasoning utilization, and empirical validation in real-world programming scenarios.The survey leaves open whether the limitation comes from existing agent frameworks or from reasoning models themselves.
- Future Directions: Future systems should connect visual specifications, graphical interfaces, and code generation to bridge design and implementation.The survey points to aesthetic-aware models, vision-based coding agents, and interface manipulation technologies as relevant directions.
5 Conclusion
The conclusion presents code and reasoning as mutually reinforcing: code strengthens logical reasoning, while reasoning advances code generation, comprehension, and debugging. It emphasizes persistent ambiguity, scalability, and consistency challenges and calls for deeper integration to build more robust, interpretable, and adaptive systems.
- Code enhances logical reasoning, while reasoning improves code intelligence across generation, comprehension, and debugging.
- Ambiguity, scalability, and consistency remain challenges for code-reasoning systems despite substantial progress.
- The survey calls for deeper integration of reasoning and programming to build more robust, interpretable, and adaptive AI systems.
6 Limitations
The survey’s breadth limits the depth of implementation and domain-specific coverage, while its publication scope and incomplete benchmark coverage constrain historical, alternative, and repository-scale perspectives. Its identified challenges are framed as ongoing research gaps rather than definitive conclusions.
- Broad coverage from single-execution reasoning to autonomous code agents leaves some implementation details and domain-specific nuances only briefly described.
- The focus on recent arXiv categories and a confined publication window excludes older or less mainstream work that could provide alternative perspectives or historical context.
- Coverage of CRUXEval, CodeMMLU, RepoQA, and SWE-bench remains incomplete for real-world repository-scale tasks and specialized areas such as concurrency analysis and security verification.
- The survey’s challenges reflect ongoing research gaps, and future datasets, architectures, and evaluation protocols may revise or expand its conclusions.
A.1 Code-enhanced Reasoning
Code-enhanced reasoning methods interleave textual reasoning with executable or simulated code to decompose tasks and delegate precise computation. Representative approaches differ in how they execute, feed back, or emulate program steps while combining symbolic and language-based inference.
- Shared Design: Together, these approaches transform reasoning problems into programmatic solutions that improve precision and reliability in complex reasoning tasks.
- Program-Aided Language Models: Program-Aided Language Models interleave natural-language reasoning with program statements and delegate arithmetic or logical operations to an interpreter.This lets the model focus on higher-level step-by-step reasoning while grounding calculations in verified outputs.
- Program of Thoughts: Program of Thoughts encodes sub-steps as meaningful variables and partial code, then executes the program externally to produce reliable numerical results.Its small, interpretable code snippets make multi-step computation more transparent and robust.
- MathCoder: MathCoder alternates language rationales and executable code blocks, feeding each execution result back into the ongoing reasoning chain.This iterative loop supports more reliable handling of intricate mathematical problems.
- Chain of Code: Chain of Code executes feasible code directly and lets the language model emulate non-executable segments while maintaining a consistent state.The hybrid approach combines symbolic execution with language-driven inference for logical, numerical, and semantic tasks.
A.2 Training with Code
Code-enhanced training methods structure reasoning through executable programs, plans, execution feedback, and iterative refinement. They target mathematical, logical, multi-step, and code-generation tasks by improving decomposition, verification, and correction.
- Code-generation methods use code data to structure thinking, verify intermediate steps, and produce more precise final answers.
- MARIO combines enriched mathematical data with a Python interpreter for exact calculation, systematic error checking, and reproducible fine-tuning.Its dataset uses GPT-4 annotations, human review, and self-training, with source code and trained models released publicly.
- POET pretrains models on programs and execution results, aligning natural language with code to support step-by-step inference and conclusion validation.POET-Math, POET-Logic, and POET-SQL cover numerical, logical, and multi-hop reasoning.
- Preference-model and critic-based approaches reduce reliance on human labels and improve reasoning across mathematical and logical tasks.CodePMP uses synthesized code-preference pairs, while SIAM applies a code-based critic and self-generated instruction and preference data.
- Reasoning-enhanced code generation uses plans, programming constructs, self-feedback, execution, and iterative repair to handle complex code tasks.Examples include high-level planning, structured pseudocode, syntax self-testing, repository-wide edit planning, and execution-trace analysis.
B Introduction of Important Benchmarks
The section introduces representative mathematical reasoning benchmarks that vary in scale, difficulty, and robustness demands. Together, they evaluate multi-step deduction, advanced problem-solving, and sensitivity to changes in wording or structure.
- GSM8K contains 8.5K grade-school word problems requiring 2-8 reasoning steps and emphasizing multi-step deduction with elementary mathematics.
- MATH comprises 12,500 competition mathematics problems requiring more sophisticated heuristics and domain knowledge than GSM8K.Each problem includes a detailed step-by-step solution.
- SVAMP contains 1,000 challenge problems that test question sensitivity, adaptation to changed reasoning paths, and structural invariance.
B.2 Training with Code
Code-enhanced reasoning is evaluated across textual, mathematical, conversational, interactive, and multi-hop settings. These benchmarks emphasize structured inference, evidence integration, sequential decisions, and program-like solution procedures.
- HotpotQA tests multi-hop question answering by requiring models to connect information across documents or sentences.
- LogiQA evaluates deductive and inductive inference in reading comprehension through questions requiring analysis of logical structures.
- DROP requires numerical and symbolic manipulation, including arithmetic, entity tracking, and multi-step logic derivation.
- AlpacaEval-2 measures instruction following on tasks involving reasoning and structured thinking, including multi-turn reasoning and systematic instruction analysis.
- MT-Bench assesses conversational coherence, reasoning depth, and consistency across extended multi-turn dialogues.
- ALFWorld evaluates sequential decision-making in interactive text environments that require language understanding, environmental cues, and multi-step goals.
B.3 Reasoning-enhanced Code Intelligence
Code-intelligence benchmarks span functional correctness, data science, repository-level completion and understanding, real-world issue resolution, and multimodal software engineering. The review also uses systematic literature collection to cover recent code-and-reasoning work.
- HumanEval evaluates functional correctness on 164 handcrafted programming challenges using unit tests and pass@k.
- MBPP tests Python code synthesis from natural-language descriptions with approximately 1,000 problems and automated test cases.
- APPS contains around 10,000 programming problems ranging from one-line solutions to complex algorithmic challenges, evaluated with unit tests.
- DS-1000 evaluates domain-specific code generation for data manipulation, statistical analysis, and data visualization.
- RepoBench and CrossCodeEval assess repository-level and cross-file completion through retrieval, contextual integration, and next-line prediction.CrossCodeEval covers Python, Java, TypeScript, and C# using examples that require multi-file information.
- SWE-bench evaluates fixes for real GitHub issues in multi-file repositories using original unit tests, while its Verified version screens out underspecified cases.SWE-bench Multimodal adds visual inputs from JavaScript repository tasks involving interfaces, diagrams, and data visualization.
- The review collects post-January-2021 papers from arXiv categories cs.AI and cs.CL using combined code/program and reasoning/planning keywords.
D Additional Tables and Figures
The supplementary materials organize code-enhanced reasoning and reasoning-enhanced code intelligence through benchmark tables and a taxonomy of their interaction. Table 4 reports code-aided reasoning performance across multiple benchmarks, while Table 5 covers reasoning-enhanced code intelligence methods.
- Additional Tables and Figures: Table 4 reports the percentage of problems solved correctly by various code-aided reasoning methods across multiple benchmarks.A dash indicates no reported result, and Direct and CoT results are attributed to Chen et al. (2022).
- Additional Tables and Figures: The taxonomy includes code-enhanced reasoning, training with code, and reasoning for code generation as major organizational categories.Examples listed include Program of Thoughts, PaL, MARIO, POET, SCoTs, Self-Planning, and CodeCoT.
- Additional Tables and Figures: The taxonomy also identifies essential code intelligence as a distinct area within the code–reasoning interplay.The listed benchmarks include Codex&HumanEval, MBPP, CodeXGLUE, RepoBench, CrossCodeEval, and HumanEval-FIM.
- Additional Tables and Figures: Figure 3 presents a full taxonomy illustrating the interplay between code and reasoning.
- Additional Tables and Figures: Table 5 reports reasoning-enhanced code intelligence results, using pass@1 for HumanEval and MBPP and incorporating results from specified original papers and leaderboards.Its annotations distinguish Direct, Self-Collaboration, Self-Planning, execution-feedback, SWE-bench, and HyperAgent sources.