InCoder-32B: Code Foundation Model for Industrial Scenarios

Motivation

• Industrial software demands specialized semantics, hardware reasoning, strict timing and resource constraints, and rigorous verification beyond conventional code tasks.Existing models achieve only 28.80% call success rate of G and 41.57% of T on Triton operator generation, and 33.3% accuracy on Verilog code passing simulation while failing formal equivalence checking.Source: §1 · p. 2

Contribution

• InCoder-32B is presented as the first purpose-built industrial code LLM, unifying chip design, GPU kernel optimization, embedded systems, compiler optimization, and 3D modeling in one model.The model has 32B parameters and targets hardware constraints, timing behavior, synthesis requirements, and low-level performance optimization.Source: §1 · p. 2

Method

• The Code-Flow pipeline combines curated-code pre-training and annealing, mid-training with synthetic reasoning and agentic trajectories while scaling context from 8K to 128K, and execution-grounded post-training.Figure 3 summarizes the three stages, including data curation, industrial-aware foundation building, progressive context scaling, and simulation-grounded supervised fine-tuning.Source: §1 · p. 3, Figure 3 · p. 3

Results

• InCoder-32B achieves 74.8% on SWE-bench Verified, 49.14% on LiveCodeBench, and 60.99% on BFCL, while establishing the strongest open-source results across evaluated industrial domains.The evaluation spans chip design, GPU kernel optimization, embedded systems, compiler optimization, and 3D modeling.Source: §1 · p. 3

Evaluation and ablations

• The study evaluates 14 general and 9 industrial benchmarks across 4 specialized domains, finding repository transitions outperform static snapshots, trajectory data improves distribution-shift robustness, and feedback-conditioned multi-turn trajectories unlock capabilities absent from standard instruction tuning.These findings come from extensive ablations accompanying the broad benchmark evaluation.Source: §1 · p. 3

2. Scaling Industrial Data under Simulation Environments

• The framework reconstructs four industrial environments in software so training signals reflect deployment toolchains and correctness criteria.These include RTL simulation, GPU execution, embedded firmware, and manufacturable CAD geometry.Source: §2 · p. 3

2.1. Chip Design

• Chip-design data uses Icarus Verilog, Verilator, and Yosys in a containerized RTL flow producing simulation, compilation, synthesis, area, and timing signals.The flow mirrors RTL authoring, behavioral simulation, and synthesis criteria used before silicon implementation.Source: §2.1 · p. 4

2.2. GPU Optimization

• GPU-kernel data runs CUDA and Triton through their standard compilation paths on NVIDIA A100 hardware, validating numerical correctness and performance with deployment-matched allocation and timing.CUDA uses nvcc through PyTorch runtime compilation, while Triton uses its official JIT compiler stack.Source: §2.2 · p. 4
• CAD data executes CadQuery scripts against production-matched OpenCascade, exports STEP or STL solids, and evaluates geometric fidelity volumetrically against references.The environment supports scripted primitives and transformations while checking that outputs are geometrically faithful, not merely syntactically valid.Source: §2.2 · p. 4

2.4. Code Optimization

• Code-optimization data covers STM32F407 firmware in Renode and native x86-64 assembly benchmarking under controlled execution conditions.The embedded setup models peripherals, registers, and interrupts, while assembly tests use fixed CPU frequency, pinned affinity, and repeated measurements.Source: §2.4 · p. 5

3. Training Strategy

• The strategy addresses the scarcity of training corpora across digital design, GPU computing, systems programming, FPGA synthesis, CAD integration, and embedded systems, each with specialized constraints and verification needs.Detailed procedures for pre-training, mid-training, and post-training are provided in Appendices A, C, and D, respectively.Source: §3 · pp. 5–6

3.1. Stage 1 Pre-Training

• Stage 1 pre-training collects industrial code from repositories, technical literature, and web data, then applies license, privacy, validation, deduplication, annotation, AST, and recompilation checks.The model is trained with autoregressive language modeling and fill-in-the-middle completion using a decoder-only Transformer on 4,096 GPUs.Source: §3.1 · p. 6

3.2.1. Context Extension

• Stage 2 extends context from 8K to 32K and then 128K tokens, progressing from file-level RTL completion to long-context debugging.Its data include synthetic industrial reasoning question-answer pairs, agent trajectories, and code artifacts grounded in real-world development scenarios.Source: §3.2.1 · p. 6, §3.2.2 · p. 6

3.2.2. Industrial Data Synthesis and Curation

• Industrial data synthesis combines engineer-specified scenarios, realistic seed code, automatically verified reasoning pairs, tool-feedback debugging trajectories [128], and professional hardware-development artifacts.Artifacts include SystemVerilog/UVM testbenches, SDC timing constraints, synthesis scripts, GPU profiling traces, and memory sanitiser logs [7, 104], exposing the model to operational context scarce in public data.Source: §3.2.2 · pp. 6–7

3.3. Stage3 Post-Training

• Stage 3 post-training constructs 2.5M execution-grounded industrial coding samples, verifies diverse candidates in real environments, and includes feedback-driven repairs and optimization examples.Tasks specify requirements, interfaces, platforms, toolchains, dependencies, and verification scripts; failed executions yield repair trajectories containing compiler, runtime, counterexample, waveform, and profiling feedback.Source: §3.3 · p. 7

4.1. Baselines

• The evaluation compares InCoder-32B with extensive open-weight and proprietary models spanning general-purpose code and specialized industrial domains.Baselines include dense and mixture-of-experts systems across a wide parameter range.Source: §4.1 · pp. 7–8

4.4. Main Results

• InCoder-32B, a 32B dense model, achieves broadly competitive results with significantly larger open-weight models across code generation, code reasoning, and Text2SQL benchmarks.Source: Table 1 · p. 8, Table 2 · p. 8, §4.4 · p. 11

4.2.1. General Code Evaluation

• General-code evaluation covers generation, library-intensive and full-stack coding, bidirectional execution reasoning, competitive programming, efficiency, Text-to-SQL, agentic coding, and tool use.The benchmarks include EvalPlus, BigCodeBench, FullStackBench, CRUXEval, LiveCodeBench, Mercury, Spider, BIRD, Terminal-Bench, SWE-bench, Mind2Web, BFCL, and τ2-bench.Source: §4.2.1 · p. 9

4.3. Industrial Code Benchmarks

• Industrial evaluation targets chip design, GPU kernel optimization, code optimization, and 3D modeling, requiring hardware-constraint reasoning, low-level performance trade-offs, and domain-specific correctness.These tasks differ materially from conventional software engineering.Source: §4.3 · p. 9

4.3.1. Chip Design

• Chip-design evaluation combines VeriScope, RealBench, ArchXBench, and VeriRepair, covering Verilog generation, production IP, complex digital systems, and error diagnosis, localization, and repair.Metrics include simulation scores, Syn@k, Func@k, (n,t) outcomes, and testbench-verified repair success.Source: §4.3.1 · pp. 9–10

4.3.2. GPU Optimization

• GPU optimization uses KernelBench and TritonBench to assess correctness, speedups over PyTorch baselines, and execution validity for generated Triton operators.KernelBench reports fast1 at three levels, while TritonBench reports call and execution accuracy for curated and PyTorch-aligned tracks.Source: §4.3.2 · p. 10

4.3.3. Code Optimization

• Code-optimization evaluation covers resource-constrained embedded C generation with EmbedCGen and assembly superoptimization with SuperCoder.EmbedCGen uses cross-compilation and Renode verification, whereas SuperCoder measures functional accuracy and speedup over gcc -O3.Source: §4.3.3 · pp. 10–11

4.3.4. 3D Modeling CAD-Coder.

• CAD-Coder evaluates executable CadQuery generation over 110K verified text-to-3D-model triplets using compilation success and voxel IoU for geometric fidelity.The data are stratified into high-, medium-, and hard-quality cases, with additional chain-of-thought annotations.Source: §4.3.4 · p. 11

5.1. Large Language Models for Code

• General code LLM research spans open-weight families such as DeepSeek [27, 68], GLM [6, 135], Kimi [111, 112], MiniMax [78–80], and Qwen [97–99, 113], alongside closed-source Claude [8–11], GPT [84–86], and Gemini [41, 42].These models target code generation, complex reasoning, multi-step planning, and tool-augmented agentic workflows, supported by benchmarks covering correctness, multilingual ability, real-world tasks, contamination, preference alignment, and agentic coding [22, 69, 19, 57, 32, 74, 140, 53, 125, 29, 55, 90, 114, 130].Source: §5.1 · pp. 12–13
• Industrial code intelligence requires specialized syntax, hardware-aware optimization, and strict functional correctness beyond general-purpose programming, but remains largely underexplored.Relevant domains include hardware description languages, high-performance GPU kernels, compiler optimization, and computer-aided design.Source: §5.1 · p. 13, §5.2 · p. 13

5.2. Industrial Code Intelligence

• Industrial research has progressed in chip design, GPU and NPU kernels, compiler optimization, embedded systems, and parametric or industry-standard 3D modeling, with domain-specific models, rewards, and benchmarks.Chip design includes VeriGen [115] and RTLCoder [72]; GPU work includes Kevin [14], ConCuR [60], and CUDA Agent [25]; compiler work includes SuperCoder [118] and LLM-VeriOpt [36]; 3D work includes CAD-Coder [44], ReCAD [62], STEP-LLM [103], and BrepCoder [59].Source: §5.2 · p. 13
• InCoder-32B unifies industrial code intelligence across hardware design, GPU kernels, compiler optimization, embedded systems, and 3D modeling, addressing fragmented single-domain coverage.Existing industrial models and benchmarks generally target one industrial sub-domain, whereas InCoder-32B bridges general-purpose code LLMs and industrial software development.Source: §5.2 · p. 13, §5.1 · p. 13

6.1. Error Analysis

• Across 1,882 failures, compilation and syntax errors dominate several benchmarks, including 71% of RealBench and 51% of ArchXBench failures, while 46% of VeriScope outputs are unparseable.Figure 5 categorizes failures across all nine industrial benchmarks into compilation/syntax, functional/logic, format, and performance errors.Source: §6.1 · pp. 13–14, Figure 5 · p. 14
• Functional correctness remains a major bottleneck: 79% of VeriRepair failures pass compilation but fail tests, 49% of ArchXBench failures simulate incorrectly, and 93% of CAD-Coder failures are geometric.CAD-Coder errors largely reflect systematic misinterpretation of Euler-angle conventions, causing extrusions to rotate around the wrong axis.Source: §6.1 · p. 14
• Optimization failures show that correct code is insufficient: 33% of KernelBench failures are functionally correct but too slow, while 83% of SuperCoder failures copy the input assembly unchanged.The analysis attributes these gaps to insufficient low-level performance reasoning and recommends rare-API and hardware-semantic curation with verification signals in training.Source: §6.1 · p. 14

6.2. Effects of different Stages

• Scaling industrial SFT data from 83M to 250M tokens consistently improves most benchmark results, with only minor late-stage regressions in some RealBench and TritonBench sub-metrics.The largest-scale results remain above or close to the 83M baseline, suggesting verification-related understanding may saturate early with smaller high-quality SFT data.Source: Figure 6 · p. 15, §6.2 · p. 15

• InCoder-32B uses a three-stage Code-Flow pipeline to reason about hardware behavior and industrial constraints without sacrificing general programming performance.The model unifies instruction tuning and analytical reasoning within one framework.Source: §7 · p. 15
• Across 14 general and 9 industrial benchmarks, InCoder-32B matches leading models competitively on general tasks and establishes strong baselines in chip design, embedded systems, GPU optimization, and compiler optimization.Source: §7 · p. 15
• Ablations show repository transition data outperforms static snapshots for planning, mid-training trajectory data improves robustness under distribution shift, and execution-feedback-conditioned multi-turn training unlocks capabilities absent from standard SFT.Source: §7 · pp. 15–16

A.1. Model Architecture

• InCoder-32B adopts a standard decoder-only Transformer architecture, with detailed configurations provided in Table 6.Source: §A.1 · p. 29

A.2. Pre-training Data

• Industrial code is underrepresented in existing corpora, motivating a dedicated pipeline for collecting, cleaning, and refining domain-specific data.Verilog and VHDL occupy negligible token fractions, while CUDA, Triton, drivers, and firmware are often mixed with general-purpose code.Source: §A.2 · p. 29

A.2.1. Data Collection.

• Data collection combines repository-level recall, OCR from technical literature, and domain-specific web sources to cover industrial code and practical usage patterns.Repository recall progresses from rules to FastText classification and domain-adapted semantic retrieval across four industrial domains.Source: §A.2.1 · p. 29

A.2.2. Data Cleaning.

• Collected data is cleaned through license and privacy filtering, invalid-file removal, four-level deduplication, and domain-specific syntax and format checks.Failed samples are repaired during refinement or removed.Source: §A.2.2 · p. 29

A.2.3. Data Refinement.

• Data refinement normalizes formatting, removes stale or dead code, adds dependency, platform, constraint, and natural-language annotations, then verifies samples by AST comparison and recompilation.Source: §A.2.3 · p. 30

A.3. Training

• Training uses autoregressive language modeling and fill-in-the-middle completion [40, 46] on 4,096 GPUs for 15T tokens, with a 3 × 10−4 learning rate and 2,048 global batch size.A curriculum advances from function-level single-file to multi-file project data; general code and text remain in every batch, while held-out industrial benchmarks guide sampling adjustments.Source: §A.3 · p. 30

B. Mid-Training Loss Curves

• Stable convergence is achieved in both mid-training stages: Stage 2.1 with cosine decay and Stage 2.2 after graduated warm-up for long-sequence samples.Figure 7 reports the training dynamics for 8K-to-32K and 32K-to-128K extension stages.Source: B. Mid-Training Loss Curves · p. 30, Figure 7 · p. 30

C. Mid-Training Details

• Mid-training addresses industrial hardware gaps by continuing pretraining on hardware-aware data for timing, resource optimization, and performance-critical system reasoning.General-purpose repositories underrepresent specialized languages, stringent timing and resource constraints, and industrial verification methodologies.Source: C. Mid-Training Details · p. 31

C.1. Progressive Context Extension Strategy

• The progressive curriculum extends context from pretrained 8K to 128K tokens, supporting file-level tasks first and long-context debugging, refactoring, and cross-module dependencies later.Stage 2.1 trains sequences up to 32K without positional interpolation; Stage 2.2 introduces sequences over 32K with replay from Stage 2.1.Source: §C.1 · p. 31

C.2. Industrial Data Synthesis and Curation

• Training data combines curated industrial code with synthetic reasoning QA, pairing authentic codebase examples with systematic coverage of underrepresented specialized scenarios.Synthetic data complements curated samples by providing structured reasoning signal where public repositories are distributionally biased toward web applications and high-level software.Source: §C.2 · p. 31, §C.2.1 · p. 31

C.2.1. Synthetic Industrial Code QA

• The three-stage synthesis pipeline specifies industrial scenarios, generates representative seed code, and synthesizes reasoning QA pairs grounded in realistic hardware and systems contexts.Scenarios include timing failures, RTL optimization, GPU memory access, formal verification, FPGA synthesis, memory safety, waveform debugging, and interface changes.Source: §C.2.1 · pp. 31–32
• Automated verification filters synthetic reasoning through code execution, static analysis, and logical consistency checks, preventing fluent but semantically incorrect reasoning from entering training.The verification layer checks outputs, semantic properties, and stepwise consistency against contradictions.Source: §C.2.1 · p. 32
• The synthetic QA corpus covers RTL, GPU, systems, FPGA, CAD, and embedded domains alongside timing, resource, safety, concurrency, precision, and operational reasoning tasks.Its broad coverage fills gaps in organic training data and supplies dense signal for structured industrial reasoning.Source: §C.2.1 · p. 32

C.2.2. Curated Industrial Code Data

• The dataset includes code commits pairing developer intent with before-after states, covering RTL timing optimisation, kernel memory refactoring, ABI preservation, and hardware fixes with testbench updates.General software commits are supplemented with hardware-characteristic samples from large-scale open-source repositories. [73]Source: §C.2.2 · p. 33
• Industrial artefacts extend source code with test suites, API specifications, configurations, logs, hardware testbenches, timing constraints, synthesis scripts, profiling traces, and sanitiser logs.These artefacts represent the operational context of professional hardware and systems development. [7, 104]Source: §C.2.2 · p. 33
• Structure-aware file-level FIM masks function bodies, methods, and code blocks while preserving AST boundaries and domain-specific units such as Verilog always blocks and kernel loop bodies.The masking strategies target syntactically valid, semantically coherent completions in C/C++ and Triton kernels. [40, 47]Source: §C.2.2 · p. 33
• Stage 2.1 weights reasoning QA at 40%, agent trajectories at 20%, commits and artefacts at 15% each, and FIM at 10%, while Stage 2.2 shifts to 30% trajectories, 25% FIM, 25% QA, and 10% each for other categories.This reweighting first strengthens reasoning foundations and then emphasizes long-context industrial workflows.Source: §C.2.2 · p. 33

D.1. Data Construction

• 2.5M SFT samples are constructed from real industrial code tasks, covering direct solutions, defect repairs, and performance optimizations, with training lasting approximately 4.9k steps at global batch size 512.The pipeline targets industrial tasks requiring compilability, testbench success, and measurable area or latency improvements.Source: §D.1 · p. 33

D.2. Task Extraction and Normalization.

• Industrial tasks are normalized into instructions containing requirements, interface constraints, target platforms and toolchains, dependencies, and verification scripts.The tasks span hardware design, GPU development, systems programming, and embedded firmware, despite varying from terse commit messages to multi-page design documents.Source: §D.2 · p. 34
• Candidates are synthesized through reference rewriting, template perturbation, cross-language migration, retrieval-augmented generation [48], and direct generation from frontier models to diversify implementation strategies.Combining channels broadens coding styles and optimization trade-offs beyond the narrow solution clusters produced by one generation method.Source: §D.2 · p. 34
• Every candidate is validated in real execution environments using compilation, simulation, tests, profiling, or formal checking, defining correctness through production toolchains rather than proxy scoring [22, 67].Candidates passing all applicable checks become high-confidence samples.Source: §D.2 · p. 34

D.3. Feedback-Driven Repair

• Failed candidates initiate closed-loop repair trajectories [54, 131] that feed compiler errors, logs, counterexamples, waveform differences, or profiling bottlenecks back into generation and re-verification.These trajectories explicitly teach diagnosis and iterative fixing from real tool output.Source: §D.3 · p. 34

D.4. Quality Filtering and Final Composition

• The final corpus combines direct solution, defect repair, and performance or structural optimization samples to teach implementation, debugging, repair, and refinement.Optimization samples improve functionally correct solutions in efficiency, readability, or architectural quality.Source: §D.4 · pp. 34–35

E. Detailed Synthetic Data Generation Pipeline

• The appendix describes the synthetic industrial code QA generation pipeline introduced earlier.It presents the pipeline as a detailed appendix component rather than introducing a separate generation objective.Source: E. Detailed Synthetic Data Generation… · p. 35

E.1. Industrial Scenario Specification

• The pipeline targets industrial scenarios including timing debugging, RTL optimisation, GPU kernels, formal verification, FPGA synthesis, memory safety, waveform analysis, and interface changes.Scenarios are identified through practising-engineer consultation and systematic analysis of industrial design-verification reports.Source: §E.1 · p. 35
• Each scenario is specified by design scale, language, timing or resource constraints, verification requirements, and common failure modes.Source: §E.1 · p. 35

E.2. Seed Code Generation

• Seed-code generation produces representative hardware and systems fragments reflecting realistic design patterns, timing-critical implementations, memory idioms, and domain conventions.It prioritises frequent but underrepresented industrial patterns such as RTL state machines, pipelined datapaths, GPU memory access, SIMD kernels, firmware, and CAD scripting.Source: §E.2 · p. 35

E.3. QA Pair Synthesis with Automated Verification

• For each seed, QA pairs probe root-cause analysis, performance diagnosis, correctness verification, refactoring trade-offs, and impact analysis through step-by-step reasoning traces [120].The traces undergo code-execution validation, static analysis, and logical-consistency checks to ensure factual correctness.Source: §E.3 · pp. 35–36
• Automated verification distinguishes the corpus from naive prompt-based synthesis by restricting training data to industrially grounded, factually correct reasoning patterns.Source: §E.3 · p. 36

E.4. Coverage and Scale

• The resulting synthetic data spans RTL, GPU, systems, FPGA, CAD, and embedded domains while covering timing, optimisation, memory safety, concurrency, precision, and operational debugging contexts.It complements curated samples by filling coverage gaps and supplying dense training signal for hardware-aware structured reasoning.Source: §E.4 · p. 36

F. Model Configuration

• Table 6 summarizes the architectural details of InCoder-32B.The supplied passages identify the table but do not provide its configuration values.Source: F. Model Configuration · p. 37, Table 6 · p. 37

G.1. VeriScope Benchmark – Level 3 Example Case

• The VeriScope case specifies a synthesizable, resource-efficient UART transmitter that serializes 8-bit data LSB-first with start, stop, idle, and busy signaling.The interface includes clk, synchronous rst, tx_start, 8-bit tx_data, tx, and tx_busy; the reference uses a shift register and bit counter.Source: §G.1 · pp. 37–38

G.2. Verilog Bug Fix – Example Case

• The Verilog debugging case identifies a missing end closing the 2’b00 begin block, while noting that the Boolean function does not depend on A.The structural defect is explicitly distinguished from the functional observation that AB=00 matches AB=10 and AB=01 matches AB=11.Source: §G.2 · p. 41

G.3. CAD-Coder – Example Case

• The CAD-Coder example case documents rectangular block generation as an industrial CAD task.The supplied material identifies the complete example through its figure caption but does not provide implementation details or outcome values.Source: Figure 10 · p. 43

G.4. C Compile – Example Case

• The C Compile case asks for compilable, fully functional FP32 CUDA optimization of matrix multiplication using a custom tiled kernel and a ModelNew architecture.The implementation uses PyTorch C++/CUDA extensions, shared-memory tiles, and a CUDA kernel interface for matrix multiplication.Source: §G.4 · pp. 44–45

G.5. TritonBench – Example Case

• The TritonBench case generates and verifies a wrapper for fused batch matrix multiplication, RMS normalization, GELU, dropout, and subtraction.Inputs have shapes (B, N, M) and (B, M, P), the output is (B, N, P), dropout defaults to 0.5, and RMS normalization uses eps=1e-5.Source: §G.5 · p. 47
• The TritonBench example wraps a fused BMM-RMSNorm-GELU-Dropout-Sub kernel with normalized-shape, output-shape, broadcasting-stride, and grid handling.The wrapper launches one kernel instance per (b, n) pair using grid = lambda META: (B * N, ).Source: §G.5 · pp. 50–51

G.6. KernelBench – Example Case

• KernelBench demonstrates replacing PyTorch operators with custom CUDA implementations, including operator fusion or algorithmic changes, and shows tiled shared-memory matrix multiplication with CUDA validation.The matrix-multiplication implementation checks CUDA placement, tensor dimensionality, and matching inner dimensions before launching a tiled grid.Source: §G.6 · pp. 52, 54, 55

G.7. EmbeddedC – Example Case

• The EmbeddedC case specifies STM32F407 TIM12 channel 2 PWM on PH7 at 800 Hz and 55% duty cycle with an update interrupt.The reference code enables GPIOH and TIM12 clocks, configures PH7 for TIM12 alternate function 9, and sets TIM12->ARR = 104999UL from an 84 MHz timer clock.Source: §G.7 · pp. 56, 58

G.8. RealBench – Example Case

• The RealBench case asks for synthesizable Verilog implementing an AES cipher top that coordinates 10 rounds over 128-bit data through explicit control states and transformations.The specification requires 16 S-box modules, a 4x4 state-matrix register array, one round per clock cycle, and rising-edge register updates.Source: §G.8 · pp. 59–60
• RealBench presents an AES Cipher Top design specification alongside model output covering declarations and control, state matrix and ShiftRows, MixColumns and AddRoundKey, output generation, and submodules.Source: Figure 21 · p. 61, Figure 22 · p. 62, §G.8 · p. 62

G.9. ArchXBench – Example Case

• The ArchXBench example specifies an 8-to-3 binary encoder that assumes one-hot input, is purely combinational, outputs 0 for invalid multiplicity, and propagates X or Z inputs to corresponding output bits.The objective is to output the binary index of the highest active input under the one-hot assumption.Source: §G.9 · p. 66
• The ArchXBench model output implements unknown-state detection, one-count validation, one-hot encoding, and conditional final output, while also showing an alternative casez implementation.The generated module is named binary_encoder_8to3 with an 8-bit input and 3-bit output.Source: §G.9 · p. 67