SemCoder: Training Code Language Models with Comprehensive Semantics Reasoning

1 Introduction

• SEMCODER combines high-level functionality, program properties and constraints, statement-level execution effects, and input/output behavior to model comprehensive program semantics.Its monologue reasoning verbalizes semantics from abstract descriptions to dynamic execution states.Source: §1 · p. 2
• PYX provides executable code samples with functional descriptions, tests, and execution traces, while PYX-R adds buggy programs, debugging rationales, and refinement plans.These datasets support semantic learning and human-like debugging or patching.Source: §1 · p. 2
• The approach trains Code LMs to summarize functionality, identify constraints, and reason forward or backward about execution states in a human-like debugging style.Forward monologues simulate execution step by step, while backward monologues infer possible prior states from outputs and constraints.Source: §1 · p. 2
• 79.3% pass@1 on HumanEval versus GPT-3.5-turbo’s 76.8% shows SEMCODER’s competitive code-generation performance despite having 6.7B parameters.The paper also reports strong execution-reasoning, debugging, and self-refinement capabilities.Source: §1 · pp. 2–3
• Monologue reasoning improves execution understanding over scratchpad-style traces by focusing on relevant semantic properties rather than redundant concrete states.The method is presented as purely static, compact, and human-readable, while handling abstract and nondeterministic states.Source: §1 · pp. 2–3

2 Program Semantics

• Program semantics describes what a program does during execution, including input processing, computation, and output.Understanding these behaviors supports checking whether programs meet their intended purpose.Source: §2 · p. 3
• Approximate semantics captures a program’s overall objective through natural-language descriptions such as docstrings or documentation.It provides a high-level view of intended functionality and expected results.Source: §2 · p. 4
• Symbolic semantics captures functionality and logic through the symbols, syntax, and structure of source code.It describes how source constructs represent behaviors and operations.Source: §2 · p. 4
• Operational semantics explains how individual code steps transform program state during concrete execution.It supports reasoning about dynamic behavior in a step-by-step manner.Source: §2 · p. 4
• Abstract semantics focuses on properties and constraints that characterize program behavior while ignoring low-level execution details.Unlike concrete semantics, it reasons about essential behavior across executions rather than one specific input.Source: §2 · p. 4

3 PYX: Semantic-aware Training Dataset

• PYX collects synthetic natural-language-to-code samples and retains executable programs using interpreter-based filtering.The dataset targets semantic learning with code, tests, and execution information.Source: §3 · p. 4, §3.1 · p. 4

3.1 Synthesizing Executable Code

• 26.9% of 43.1k Python solutions from prior instructional data were inexecutable, motivating PYX’s execution-based quality filtering.PYX improves generation by parsing AST subtrees, executing generated code, and retrying failed samples through LLM debugging.Source: §3.1 · p. 4

3.2 Dataset with Operational Semantics

• Operational-semantics data excludes programs with external resource interactions or randomness so traces can represent predictable variable-state changes.Input diversity is expanded through type-aware mutation and LLM-based input generation.Source: §3.2 · p. 5

3.3 PYX-R: Training Code LLMs to Rubber-duck Debug and Self-refine

• PYX-R contains buggy Code-LM solutions, rubber-duck debugging rationales, and patches verified by differential testing against ground truth.Only rationales leading to correct patches are retained for debugging and self-refinement training.Source: §3.3 · p. 5

4.1 Natural Language to Code

• Natural-language-to-code training supplies task objectives, implementation constraints, and expected outcomes through executable descriptions and test cases.These descriptions provide the high-level basis for semantic understanding.Source: §4.1 · p. 5

4.2 Monologue Reasoning to Comprehensively Understand Code Semantics

• Monologue reasoning progresses from functional summaries to invariant properties and constraints, then to operational state changes for given inputs and outputs.It omits the original problem description during monologue generation and focuses on code semantics.Source: §4.2 · p. 5

4.2.1 Forward Monologue

• Forward monologues simulate execution in natural order, explaining side effects, repeated loop iterations, variable changes, control flow, and final output.The process emphasizes lines with side effects and predicts the output after the walkthrough.Source: §4.2.1 · p. 6

4.2.2 Backward Monologue

• Backward monologues use abstract constraints to reason about multiple possible prior states when outputs do not uniquely determine inputs.They can connect abstract constraints to concrete input values for cases such as sorting or aggregation.Source: §4.2.2 · p. 6

4.2.3 Monologue Annotation Using LLM

• LLM-generated monologues are accepted when execution-based checks confirm their predicted outputs, with few-shot examples and system instructions guiding the format.The same rejection-sampling strategy is applied to forward and backward annotations.Source: §4.2.3 · pp. 6–7

4.3 Joint Training with Comprehensive Semantics

• Joint training combines natural-language-to-code samples and both monologue types under a next-token objective applied to code and monologue tokens.Task-specific prefixes identify the semantic modality to learn for each sample.Source: §4.3 · p. 7

Evaluation Tasks

• SEMCODER is evaluated on code generation with EvalPlus and LiveCodeBench-Lite, and on execution reasoning with CRUXEval and LiveCodeBench code execution.Baseline models use chain-of-thought prompting, while SEMCODER uses zero-shot monologue reasoning under the benchmarks’ original settings.Source: §5 · p. 7

Rubber-duck Debugging and Self-refine

• Iterative programming is tested through five rounds in which models generate, test, verbally debug, and refine code using EvalPlus.The study compares zero-shot prompting with fine-tuning on PYX-R.Source: §5 · p. 7

Models

• SEMCODER starts from DeepSeekCoder-6.7B and is trained in base and advanced forms with progressively broader coding datasets.The base model uses PYX, whereas SEMCODER-S additionally uses Evol-instruct and partial CodeContest data.Source: §5 · p. 7

Configuration and Empirically Settings

• All SEMCODER variants train for two epochs with cosine-decayed learning rates, while self-refinement fine-tuning uses separate settings and synthetic data generation relies on GPT-3.5-turbo and GPT-4o-mini.Training uses eight RTX A6000 GPUs, batch size 512, and a 2,048-token context length.Source: §5 · p. 7

6.1 Overall Performance

• 23.0%, 23.9%, and 23.6% absolute gains over DeepSeekCoder-6.7B occur on input prediction, CRUXEval-O, and LCB-Exec, respectively.Both SEMCODER variants outperform GPT-3.5-turbo on execution reasoning, including models with up to twice as many parameters.Source: §6.1 · p. 8
• 79.3 pass@1 on HumanEval base and 74.4 on HumanEval plus are achieved by SEMCODER-S, while SEMCODER reaches 79.9 on MBPP and execution reasoning surpasses GPT-3.5-turbo.The main results are reported in Table 1, with code-generation metrics using pass@1.Source: §6.1 · pp. 7–8, Table 1 · p. 8
• 21.7% absolute improvement in output prediction is obtained with monologue reasoning, whereas free-form chain-of-thought provides little benefit for execution reasoning.The result supports systematic reasoning over comprehensive code semantics rather than merely generating more reasoning tokens.Source: §6.1 · p. 8

6.2 Effectivenss of Monologue Reasoning

• Monologue reasoning outperforms scratchpad, NeXT trace, and concrete-state formats in both input and output prediction.Its natural-language transitions retain key properties and values while expressing causal relations that concrete traces omit.Source: §6.2 · pp. 8–9, Table 2 · p. 9
• Monologues can contain flawed intermediate logic despite reaching correct outcomes, unlike the correct intermediate steps guaranteed by realistic execution traces.This limitation motivates emphasizing key-property correctness and model-friendly formats when training across semantic dimensions.Source: §6.2 · p. 9

6.3 Debugging and Self-Refinement

• SEMCODER-S outperforms all evaluated baselines in iterative debugging and self-refinement, especially under zero-shot prompting.Fine-tuning on PYX-R improves iterative programming through monologue-style debugging rationales and aligned patches, but offers limited gains for some already strong models.Source: §6.3 · p. 9, Table 3 · p. 9

• SEMCODER-S outperforms all baselines in debugging and self-refinement, especially in the zero-shot setting.This indicates that monologue reasoning augments general-purpose instruction tuning with code-semantics reasoning capabilities.Source: HEval (+) MBPP (+) · p. 9
• SEMCODER demonstrates continuous code refinement across iterations, highlighting the potential of learned program semantics for complex programming tasks.Source: HEval (+) MBPP (+) · p. 10
• Fine-tuning on PYX-R significantly improves iterative programming through monologue-style debugging rationales and well-aligned patches.PYX-R helps models analyze bugs using source code and execution traces, although gains are limited for already strong reasoning models and can be constrained by occasional rationale errors.Source: HEval (+) MBPP (+) · p. 10
• Monologue reasoning is comparably effective to attaching execution traces for debugging while avoiding dynamic tracing and redundant concrete states.It focuses on bug-relevant properties and provides human-readable explanations.Source: HEval (+) MBPP (+) · p. 10

• Existing Code LMs are trained mainly on source code and related text, while instruction datasets often contain little execution coverage and unverified solutions.These limitations motivate training on static code together with dynamic execution information.Source: §7 · p. 10
• Earlier execution-reasoning approaches predicted outputs or generated traces, but generally did not combine broad semantic understanding with code-generation evaluation.SemCoder is trained for code generation, output prediction, and input prediction while targeting execution reasoning.Source: §7 · p. 10

• SEMCODER or SEMCODER-S achieves SOTA performance among open-source models below 15B parameters across generation, prediction, debugging, and self-refinement tasks.The reported tasks require knowledge of both source code and execution reasoning.Source: §8 · p. 10
• Joint training on approximate, symbolic, operational, and abstract semantics cultivates comprehensive program-semantics understanding.Source: §8 · p. 10

A Limitations and Future Work

• Intermediate monologue steps can contain flawed assumptions despite correct outputs or patches, potentially limiting further improvements in code reasoning.The authors encourage automatic and efficient process supervision for code reasoning.Source: A Limitations and Future Work · p. 15
• SEMCODER relies on GPT-3.5-Turbo or GPT-4o-mini to generate crucial monologue annotations because its base model has only 6.7B parameters.Future work could use a larger base model to generate its own annotations.Source: A Limitations and Future Work · p. 15
• Execution reasoning currently benefits code generation and self-refinement indirectly, motivating direct self-verification with forward monologues before finalizing solutions.The authors propose using the model’s own execution reasoning directly in generation and refinement.Source: A Limitations and Future Work · p. 15

B Broader Impacts

• The publicly released artifacts could support automated programming assistants, but the authors recommend using them as copilots rather than relying on full automation.Source: B Broader Impacts · p. 15
• The synthetic data is generated with GPT-3.5-Turbo, which the authors characterize as having minimal misuse risk because of its alignment and synthetic-data setting.Source: B Broader Impacts · p. 15

C.1 Debug and Self-refine w/ Real Execution Traces

• SEMCODER is comparably effective to real execution traces for debugging and self-refinement, while remaining static and more compact.Source: Table 4 · p. 15

C.2 Input/Output Prediction Without Reasoning

• Monologue reasoning improves execution output prediction by up to 21.7% absolute over reasoning-free prediction.Source: §C.2 · p. 16, Table 5 · p. 16

C.3 Comparison with Larger Open-Sourced Models and Closed-Source Models

• Table 6 compares SEMCODER with larger open-source and closed-source Code LLMs using pass@1 across code generation and execution reasoning.Source: Table 6 · p. 16

D SEMCODER Continuously Refines Code Qualities

• SEMCODER continuously improves its code across refinement steps, with gains continuing at high sampling temperatures.Source: D SEMCODER Continuously Refines Code… · p. 16, Figure 3 · p. 17

• Table 7 reports the top ten error types among inexecutable Python samples from OSS-INSTRUCT.Source: Table 7 · p. 17, E Executability Analysis of OSS-INSTRUCT · p. 17

F Details on PYX

• PYX generates executable training data by restricting problems and solutions to self-contained code, expanding inputs through type-aware mutation and LLM generation, and retaining only inputs that execute successfully.The two input-generation methods alternate until a predefined threshold, and the resulting inputs and outputs become unit tests.Source: F Details on PYX · p. 17
• 93% average branch coverage and 96% average line coverage show that PYX’s lightweight input generation is effective for its short, single-function programs, despite not targeting full execution-path diversity.The method varies input values rather than explicitly using coverage-guided testing, while the dataset’s programs generally lack complicated branches.Source: F Details on PYX · p. 18
• Training on executable PYX performs only marginally better than OSS-INSTRUCT and its Python subset, indicating executability alone does not necessarily improve natural-language-to-code performance but is needed to expose full code semantics.PYX selects only parsable code seeds, producing a slightly higher executable-code yield.Source: F Details on PYX · p. 18, Table 8 · p. 18
• PYX’s benchmark similarity is comparable to OSS-INSTRUCT, with most samples below 0.4 similarity, while executability filtering shifts categories toward algorithmic, data-structure, data-science, machine-learning, mathematical, and computational problems.The category shifts reflect excluding tasks that commonly require external resources unavailable in the execution environment.Source: F Details on PYX · p. 18, Figure 5 · p. 19, Figure 6 · p. 19
• PYX contains 32,489 natural language-to-code pairs, 29,945 forward monologues, 31,022 backward monologues, and PYX-R’s 18,473 debugging samples; SEMCODER and SEMCODER-S use 93.4k and 214.1k samples, respectively.The monologue samples were decontaminated against CRUXEval by removing shared input-output pairs.Source: F Details on PYX · p. 19
• In the HTML-parser example, monologue execution tracing identifies that `_parse_input` incorrectly aggregates child text into parent tags, causing the test assertion to fail; the fix processes text only for leaf elements and initializes parent entries as empty lists.The corrected logic prevents parent elements such as `div` from duplicating descendant text while preserving separate child-element outputs.Source: F Details on PYX · pp. 22–23

G Detailed Example of Monologues

• The detailed monologue example explains that `unique_sorted_indices` maps each unique energy value to its first index, sorts unique values ascending, and returns the corresponding original indices while handling duplicates.Its stated constraints include variable-length inputs, duplicate values, and first-occurrence indexing.Source: G Detailed Example of Monologues · pp. 23–24
• The detailed monologue example maps each unique energy to its first index, sorts unique energies ascending, and returns [3, 1, 0] for the demonstrated input.For [10.5, 8.2, 10.5, 7.1, 8.2], the final dictionary is {10.5: 0, 8.2: 1, 7.1: 3}, and sorted unique energies are [7.1, 8.2, 10.5].Source: G Detailed Example of Monologues · pp. 24–25

H Task-specific Prefix

• Task-specific prefixes guide natural-language-to-code generation, execution simulation, semantic-constraint deduction, and debugging with self-refinement.The prompts respectively request code from a description, execution outputs, an input producing an expected output, or a simulated diagnosis followed by corrected code.Source: H Task-specific Prefix · pp. 25–26

I Baseline Trace Formats

• The baseline trace presents line-labeled code states, intermediate dictionary and sorted-energy states, and an output marker yielding [3, 1, 0].It records sorted_unique_energies as [7.1, 8.2, 10.5] before producing the index list.Source: I Baseline Trace Formats · pp. 26–27