Source-linked AI summary
MoTCoder: Elevating Large Language Models with Modular of Thought for Challenging Programming Tasks
Jingyao Li, Pengguang Chen, Bin Xia, Hong Xu, Jiaya Jia
TL;DR
Complex programming problems challenge LLMs that generate monolithic code blocks rather than decomposing work into sub-tasks. MoTCoder uses module-of-thought instruction tuning to guide modular solutions, improving pass@1 on APPS and CodeContests and self-correction against current SOTA.
Problem
On intricate coding problems such as APPS and CodeContests, models that generate a monolithic solution in one shot struggle with tasks involving multiple sub-tasks.
Source: §1 · p. 1Method
MoTCoder instruction-tunes LLMs to outline sub-modules, implement them, and combine them into a final solution.
Source: §1 · pp. 1–2Results
MoTCoder improves pass@1 by 5.8% on APPS and 5.9% on CodeContests, and surpasses current SOTA in self-correction by 3.3%.
Source: §1 · p. 2Takeaways & Limitations
MoTCoder enhances solution modularity and improves the maintainability index of generated code.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Large Language Models (LLMs) have showcased impressive capabilities in handling straightforward programming tasks. However, their performance tends to falter when confronted with more challenging programming problems. We observe that conventional models often generate solutions as monolithic code blocks, restricting their effectiveness in tackling intricate questions. To overcome this limitation, we present Module-of-Thought Coder (MoTCoder). We introduce a framework for MoT instruction tuning, designed to promote the decomposition of tasks into logical sub-tasks and sub-modules. Our investigations reveal that, through the cultivation and utilization of sub-modules, MoTCoder significantly improves both the modularity and correctness of the generated solutions, leading to substantial pass@1 improvements of 5.9% on APPS and 5.8% on CodeContests. MoTCoder also achieved significant improvements in self-correction capabilities, surpassing the current SOTA by 3.3%. Additionally, we provide an analysis of between problem complexity and optimal module decomposition and evaluate the maintainability index, confirming that the code generated by MoTCoder is easier to understand and modify, which can be beneficial for long-term code maintenance and evolution. Our codes are available at https://github.com/dvlab-research/MoTCoder.
1 INTRODUCTION
MoTCoder addresses the difficulty of complex programming tasks by training LLMs to decompose solutions into sub-modules. It improves benchmark performance and self-correction, while analysis also reports benefits for code maintainability.
1 INTRODUCTION
- Earlier iterative code-inference methods add inference costs and offer only marginal gains, motivating MoT instruction fine-tuning as an alternative.The paper frames MoT as a more efficient way to improve coding LLM modularization.Source: §1 · p. 1
- 5.8% on APPS and 5.9% on CodeContests pass@1, with a 3.3% improvement over current SOTA in self-correction.The figure compares pass@1 on both benchmarks across models, with model size indicated by scatter size.Source: Figure 1 · p. 1, §1 · p. 2
- MoTCoder trains LLMs to decompose solutions into modules for logical sub-tasks, then implement and combine those modules into a final solution.The approach uses MoT instruction transformation and fine-tuning to cultivate modularization capabilities.Source: §1 · pp. 1–2
- MoTCoder analyzes complexity and module decomposition, reports reduced memory consumption, and finds its generated code easier to understand and modify.The authors connect maintainability to long-term code maintenance and evolution.Source: §1 · p. 2
2 RELATED WORKS
Related work traces progress in general and coding LLMs, instruction tuning, and prompting. The paper distinguishes its training-time guided modularization from approaches that plan code only during generation.
2.1 Large Language Models
2.2 Instruction Fine-Tuning
- Instruction tuning spans broad task corpora and human- or model-generated instructions, while related work also applies it to reasoning, coding, and tool use.The cited approaches include T5, FLAN, InstructGPT, and instruction-tuned reasoning work [52], [53].Source: §2.2 · p. 2
2.3 Prompting Techniques
- CoT generates intermediate reasoning steps, while least-to-most prompting sequences sub-problems; MoT instead guides modularization during training rather than only planning code at inference.The paper describes MoT instruction transformation as a two-step process of outlining sub-modules and then implementing and integrating them.Source: §2.3 · pp. 2–3, Figure 2 · p. 3
3 METHODS
The methods transform ordinary coding instructions into a staged module-of-thought process, assess transformed examples, and fine-tune a coding model on clean and MoT data.
3 METHODS
- The methods section covers module-of-thought instruction transformation followed by instruction tuning for MoTCoder.The section introduces the transformation in Section 3.1 and the tuning strategy in Section 3.2.Source: §3 · p. 3
3.1 Module-of-Thought Instruction Transformation
- The transformation section introduces normal instructions, MoT instructions, and assessment of the transformed instructions.These components form the instruction-transformation procedure.Source: §3.1 · p. 3
3.1.1 Normal Instruction
- A normal instruction conditions autoregressive token generation; the text identifies I as the input instruction and o_t as the t-th output token.The output is described as a flattened sequence sampled from the model's conditional distribution.Source: §3.1.1 · p. 3
3.1.2 Module-of-Thought Instruction
- MoT transforms normal instructions into sequential generation: first outline sub-module headers and docstrings, then implement and combine them into a solution.The prompt includes a one-shot example; the approach targets coding detail and broader strategic planning.Source: §3.1.2 · pp. 3–4, Figure 3 · p. 4
3.1.3 Instruction Assessment
- Assessment flags instructions that violate the two-phase protocol, omit sub-modules or main code, create multiple main-code blocks, or fail equivalence tests.The test criterion checks that transformed programs preserve functional equivalence with the original codes.Source: §3.1.3 · pp. 4–5
3.2 Module-of-Thought Instruction Tuning
- The section first introduces the module-of-thought dataset, followed by details of MoT instruction tuning, in that order.Source: §3.2 · p. 5
3.2.1 MoT Dataset
- Training data comes from APPS and CodeContests training sets, is deduplicated against APPS test overlap, and comprises 183K clean and 174K MoT examples.Generated examples are tested against training-set input-output examples, and failing examples are discarded.Source: §3.2.1 · p. 5, Table 1 · p. 5
3.2.2 MoT Instruction Tuning
- MoTCoder is instruction-tuned from Qwen2.5-Coder-7B-Instruct [39] for one epoch on the MoT dataset, using a 2048-token maximum input length.The passage also specifies batch sizes, learning-rate scheduling, AdamW, and memory and compute optimizations.Source: §3.2.2 · p. 5
4 EXPERIMENTS
Experiments evaluate MoTCoder on challenging APPS and CodeContests problems, including comparisons with fine-tuning baselines and self-reflection. The reported results show improvements across these evaluations.
4 EXPERIMENTS
- The evaluation focuses on intricate code-generation tasks using APPS and CodeContests pass@k, rather than short-task benchmarks such as HumanEval and MBPP.Source: §4 · p. 5
4.1 Ablation Experiments
- 34.67% versus 32.21%: MoT fine-tuning improved overall APPS pass@1, while uncurated normal fine-tuning lowered the score to 27.70%.Source: §4.1 · p. 5, Table 2 · p. 6
4.2 Results on APPS
- On APPS, MoTCoder-7B scored 34.67%, surpassing Qwen2.5-Coder-Instruct-7B by 2.46%; MoTCoder-32B scored 45.95%, surpassing the baseline by 5.82%.MoTCoder scored higher across difficulty levels.Source: Table 3 · p. 6, §4.2 · pp. 6–7
- 12.61%: MoTCoder-6.7B outperformed GPT-4 on APPS, with larger gains on interview- and competition-level problems.Source: §4.2 · p. 6
4.3 Results on CodeContests
- On CodeContests, MoTCoder-7B achieved 19.09% (+3.89% over Qwen2.5-Coder), and MoTCoder-32B reached 23.85% (+5.93%).Source: Table 4 · p. 7, §4.3 · p. 7
4.4 Results on Self-Reflection
- With self-reflection, MoTCoder-7B improved from 19.09% to 23.08%, surpassing Qwen2.5-Coder-Instruct by 3.28%.Source: §4.4 · p. 7
5 FURTHER ANALYSIS
The analysis examines how module count relates to problem difficulty, compares resource use, and evaluates the maintainability of generated code.
5.2 Time and Memory Consumption
- MoT code uses comparable time but consistently less memory than normal code on passed samples across problem levels.The comparison uses samples from APPS and CodeContests where both approaches passed.Source: §5.2 · p. 7, Figure 4 · p. 8
5.1 Influence of modules on Accuracy
- With more functions, accuracy declines for introductory problems, stays relatively stable for interview problems, and trends upward for competition problems.The authors relate this pattern to simpler problems benefiting from fewer modules and complex problems from decomposition.Source: §5.1 · p. 7, Figure 4 · p. 8
5.3 Maintainability Analysis
- MoT code has significantly higher maintainability than normal-finetuned and baseline code across all levels, making it easier to understand and modify.The authors identify long-term maintenance and evolution as settings where this may be beneficial.Source: Figure 4 · p. 8, §5.3 · p. 8
6 CONCLUSION
MoTCoder addresses monolithic solutions to complex programming tasks by encouraging decomposition into sub-tasks and sub-modules. The paper reports gains in accuracy, self-correction, and maintainability.
6 CONCLUSION
- MoTCoder uses instruction tuning to encourage breaking complex programming tasks into manageable sub-tasks and sub-modules.Source: §6 · p. 8
- The approach enhances solution modularity and accuracy, with considerable pass@1 improvements on APPS and CodeContests.Source: §6 · p. 8
- MoTCoder also achieves notable improvements in self-correction capabilities.Source: §6 · p. 8
- Generated code is easier to comprehend and modify, and the authors describe MoT instruction tuning as a promising direction for future research.Source: §6 · p. 8