PaperlayerView exact revision on arXiv

Source-linked AI summary

InCoder-32B: Code Foundation Model for Industrial Scenarios

Jian Yang, Wei Zhang, Jiajun Wu, Junhang Cheng, Shawn Guo, Haowen Wang, Weicheng Gu, Yaxin Du, Joseph Li, Fanglin Xu, Yizhi Li, Lin Jing, Yuanbo Wang, Yuhan Gao, Ruihao Gong, Chuan Hao, Ran Tao, Aishan Liu, Tuney Zheng, Ganqu Cui, Zhoujun Li, Mingjie Tang, Chenghua Lin, Wayne Xin Zhao, Xianglong Liu, Ming Zhou, Bryan Dai, Weifeng Lv

arXiv:2603.16790v3cs.SEcs.AI

TL;DR

General code models leave industrial programming demands underexplored because these tasks require hardware semantics, specialized constructs, and strict resource constraints. InCoder-32B addresses the gap with staged training and execution-grounded verification, achieving competitive general-code performance and strong open-source results across industrial domains. Ablations further identify data and feedback strategies associated with planning and robustness.

  • Problem

    Existing code LLMs are predominantly trained and evaluated on general software tasks, while industrial domains requiring hardware semantics, low-level optimization, and specialized correctness remain underexplored.

  • Method

    InCoder-32B uses a three-stage pipeline combining curated industrial data, context extension from 8K to 128K tokens, synthetic reasoning and trajectories, and execution-grounded post-training.

  • Results

    InCoder-32B combines broad coding competence with specialized industrial capabilities, including 74.8% on SWE-bench Verified and strongest open-source results across evaluated industrial domains.

  • Takeaways & Limitations

    The model provides a unified open-source baseline spanning chip design, GPU optimization, embedded systems, compiler optimization, and 3D modeling.

  • Takeaways & Limitations

    Industrial code intelligence remains challenging: 33% of KernelBench failures are functionally correct but too slow, and 83% of SuperCoder failures copy the input assembly unchanged.

Abstract

from arXiv · show

Recent code large language models have achieved remarkable progress on general programming tasks. Nevertheless, their performance degrades significantly in industrial scenarios that require reasoning about hardware semantics, specialized language constructs, and strict resource constraints. To address these challenges, we introduce InCoder-32B (Industrial-Coder-32B), the first 32B-parameter code foundation model unifying code intelligence across chip design, GPU kernel optimization, embedded systems, compiler optimization, and 3D modeling. By adopting an efficient architecture, we train InCoder-32B from scratch with general code pre-training, curated industrial code annealing, mid-training that progressively extends context from 8K to 128K tokens with synthetic industrial reasoning data, and post-training with execution-grounded verification. We conduct extensive evaluation on 14 mainstream general code benchmarks and 9 industrial benchmarks spanning 4 specialized domains. Results show InCoder-32B achieves highly competitive performance on general tasks while establishing strong open-source baselines across industrial domains.

1. Introduction

General code models perform strongly on mainstream programming tasks, but industrial software requires specialized semantics, hardware reasoning, and strict resource constraints. InCoder-32B addresses this gap with a three-stage training pipeline and competitive general and industrial benchmark results.

  • Industrial programming requires specialized language semantics, hardware reasoning, timing behavior, resource constraints, and rigorous verification beyond conventional software engineering.
  • InCoder-32B is a 32B-parameter model designed to unify code intelligence across chip design, GPU kernels, embedded systems, compiler optimization, and 3D modeling.
  • Its Code-Flow pipeline combines curated industrial-code pre-training and annealing, context scaling from 8K to 128K tokens, synthetic reasoning data, and execution-grounded post-training.
  • 74.8% on SWE-bench Verified, 49.14% on LiveCodeBench, and 60.99% on BFCL demonstrate competitive general coding performance against comparable or larger models.
  • InCoder-32B establishes the strongest open-source results across the evaluated industrial domains and covers 14 general and 9 industrial benchmarks.

2. Scaling Industrial Data under Simulation Environments

The paper builds industrial simulation environments that reproduce deployment toolchains and execution semantics across chip design, GPU optimization, CAD, embedded systems, and assembly optimization. These environments evaluate correctness and, where relevant, performance or geometric fidelity using domain-specific criteria.

  • Industrial code correctness is established in deployment-like environments: RTL is simulated, GPU kernels run on hardware, firmware boots on microcontrollers, and CAD scripts produce manufacturable geometry.
  • 2.1. Chip Design: Chip-design evaluation composes Icarus Verilog, Verilator, and Yosys to produce compilation status, simulation results, synthesis reports, and area or timing estimates.
  • 2.2. GPU Optimization: GPU optimization evaluation compiles CUDA or Triton kernels, launches them on NVIDIA A100 hardware, and measures numerical correctness and performance through the production-like software stack.
  • 2.3. 3D Modeling: The CAD environment uses CadQuery and OpenCascade to execute scripts and compares tessellated solids volumetrically against references for geometric fidelity.
  • Embedded and assembly environments respectively simulate STM32F407 firmware and benchmark x86-64 assembly under controlled native execution conditions.
  • Across domains, the environments replicate industrial toolchains and execution semantics rather than simplified proxies so training signals reflect practical deployment criteria.

3. Training Strategy

InCoder-32B scales industrial data through collection, refinement, synthetic reasoning, agent trajectories, operational artifacts, and execution-grounded post-training. The strategy progressively expands context and filters verified samples into solution, repair, and optimization categories.

  • Industrial training data are collected from public repositories, technical literature, and domain-specific web data, then license-filtered, deduplicated, validated, and annotated.
  • Context length is progressively extended from 8K to 32K and then 128K tokens, moving from file-level completion toward extended debugging sessions.
  • Synthetic industrial reasoning data are generated from engineer-specified scenarios, realistic seed code, and automatically verified question-answer pairs.
  • Multi-step agent trajectories capture closed-loop reasoning with feedback from simulators, synthesis tools, compilers, and formal verification engines.
  • The post-training corpus includes industrial artifacts such as testbenches, timing constraints, synthesis scripts, profiling traces, and memory-sanitizer logs.
  • Execution-grounded data construction decomposes tasks into requirements, interfaces, platforms, dependencies, and verification scripts before generating and validating diverse solutions.
  • 3.3. Stage3 Post-Training: Quality filtering by executability, stability, and information density yields direct solutions, defect repairs, and performance or structural optimization samples.

4. Evaluation

The evaluation compares InCoder-32B with open-weight and proprietary models across general coding, reasoning, tool-use, and industrial benchmarks. It shows broadly competitive general performance and strong industrial results, while some optimization and embedded tasks remain challenging.

  • Evaluation scope: The evaluation spans general-purpose code, reasoning, efficiency, Text2SQL, agentic coding, tool use, and specialized industrial tasks.Industrial evaluations cover chip design, GPU kernel optimization, code optimization, 3D modeling, and embedded systems.
  • Industrial benchmarks: Chip-design benchmarks use simulation, compilation, testbench, or repair-oriented evaluation to assess specialized Verilog generation and debugging.VeriScope contains 568 problems, RealBench targets production-grade IP-level Verilog, ArchXBench contains 51 digital designs, and VeriRepair contains approximately 22,000 training and 300 test samples.
  • Industrial benchmarks: TritonBench evaluates generated operators through call accuracy and execution accuracy on GitHub-derived and PyTorch-aligned tracks.The four reported metrics are G-call, G-exe, T-call, and T-exe.
  • General benchmarks: InCoder-32B ranks first among open-weight baselines on SWE-bench Verified, Mind2Web, and τ2-bench despite being a 32B dense model.A performance gap remains on some reasoning and efficiency benchmarks relative to the largest mixture-of-experts models.
  • Industrial benchmarks: On industrial benchmarks, InCoder-32B achieves the best open-weight RealBench module-level results and leads open-weight baselines on CAD-Coder and KernelBench across all three levels.It also surpasses Claude-Sonnet-4.6 on CAD-Coder IoU and KernelBench L1/L2/L3, while EmbedCGen and SuperCoder remain challenging.

5. Related Work

Existing code LLMs have advanced general software engineering but leave industrial programming comparatively underexplored. Prior work addresses individual industrial domains, whereas InCoder-32B targets unified coverage across multiple domains.

  • Research gap: Existing code LLMs are predominantly trained and evaluated on general software tasks, while industrial programming demands remain largely underexplored.The underexplored domains include hardware description, GPU kernel programming, compiler optimization, and computer-aided design.
  • Industrial challenges: Industrial software requires domain-specific syntax, hardware-aware optimization, and strict functional correctness beyond general-purpose programming.Recent work has begun addressing these challenges, mostly through domain-specific approaches.
  • Prior domain-specific work: Chip-design research spans RTL generation, debugging, multimodal synthesis, verification, and reinforcement learning with hardware-specific rewards.Associated benchmarks include VerilogEval, RealBench, and ArchXBench.
  • Prior domain-specific work: GPU kernel and compiler optimization research has developed reinforcement-learning, profiling-based, superoptimization, and verification-oriented approaches with dedicated benchmarks.Examples include KernelBench and TritonBench for GPU optimization and multiple systems targeting compiler optimization.
  • Unified direction: Prior models and benchmarks generally address one industrial sub-domain, producing fragmented coverage.InCoder-32B is presented as a unified industrial code LLM spanning multiple industrial programming domains.

6. Analysis

Error analysis across nine industrial benchmarks shows that InCoder-32B still struggles with precise syntax, functional correctness, formatting, and optimization requirements, although larger industrial SFT datasets generally improve performance.

  • Failure patterns: 1,882 failures across nine industrial benchmarks reveal five recurring error themes in InCoder-32B.Figure 5 categorizes failures by compilation and syntax, functional and logic, format, and performance-related errors.
  • Compilation and syntax: 71% of RealBench failures involve compilation or syntax errors, while ArchXBench reports 51% syntax errors.Observed issues include malformed literals, incorrect port declarations, bit-width mismatches, named-port misuse, and indefinite-width signed literals.
  • APIs and formats: 47% of EmbedCGen failures are linker errors, TritonBench has 33% NameErrors and 24% TypeErrors, and 46% of VeriScope failures are unparseable outputs.These failures reflect incomplete knowledge of industrial APIs and required output formats.
  • Functional correctness: 79% of VeriRepair failures compile but fail tests, while ArchXBench has 49% functional failures and CAD-Coder has 93% geometric failures.The failures involve subtle repair logic, incorrect Verilog simulation behavior, and systematic Euler-angle misinterpretation.
  • Optimization: 33% of KernelBench failures produce correct but insufficiently fast kernels, and 83% of SuperCoder failures copy the input assembly unchanged.Optimization tasks therefore require performance reasoning beyond code generation alone.
  • Data scaling: Industrial SFT scaling from 83M to 250M tokens improves most benchmark results, with only minor regressions in some RealBench and TritonBench sub-metrics.Verification-related understanding may saturate earlier with a smaller high-quality SFT dataset.

7. Conclusion

InCoder-32B bridges general code intelligence and industrial software demands through a three-stage training approach. Its ablations further identify transition data, mid-training trajectories, and execution-conditioned multi-turn trajectories as important training signals.

  • Contribution: InCoder-32B bridges general code intelligence and industrial software development through a unified code foundation model.The conclusion frames the model as addressing stringent industrial requirements while retaining general programming performance.
  • Ablation findings: Multi-turn trajectories conditioned on execution feedback unlock capabilities absent from standard SFT.

A. Pre-training Details

The pre-training pipeline combines a decoder-only Transformer with broad industrial-data collection, rigorous cleaning and refinement, and large-scale autoregressive training that retains general-purpose code.

  • Model architecture: InCoder-32B uses a standard decoder-only Transformer architecture.Detailed model configurations are provided in Table 6.
  • Data motivation: Industrial code is underrepresented in existing corpora, especially for Verilog, VHDL, CUDA, Triton, drivers, and firmware.Common large-scale code datasets are dominated by web-oriented languages and high-level software.
  • Data collection: Industrial code is collected from public repositories, technical literature, and domain-specific web sources.The sources provide complementary coverage of code, reference knowledge, practical samples, and usage patterns.
  • Repository-level recall: Repository retrieval uses rule-based filtering, a FastText classifier, and a domain-adapted semantic encoder in successive recall stages.The strategy targets both obvious industrial files and code that is syntactically similar to general-purpose programs.
  • Cleaning and validation: Cleaning removes restrictive-license files, personal information, credentials, invalid artifacts, and duplicates across four deduplication levels.Validation checks cover domain-specific syntax, header consistency, and GPU-kernel configuration formats.
  • Data refinement: Refinement standardizes formatting, adds dependency and constraint metadata, and verifies samples through AST comparison and recompilation.
  • Pre-training: Training uses autoregressive language modeling and fill-in-the-middle completion on 4,096 GPUs for 15T tokens.The curriculum progresses from function-level single files to multi-file projects while retaining general-purpose code and text in every batch.

B. Mid-Training Loss Curves

Mid-training progressively extends context from 8K to 128K tokens while adapting the model to hardware-aware industrial reasoning. Its synthetic data pipeline specifies scenarios, generates seed code, and verifies reasoning with execution, static, and logical checks.

  • Loss dynamics: Both mid-training stages converge stably under cosine decay and graduated warm-up strategies.Stage 2.2 introduces long-sequence samples from 10% to 50% during warm-up before stable convergence.
  • Industrial motivation: Industrial hardware tasks require reasoning about timing, resources, performance, and verification beyond general-purpose code training.The motivation includes domain-specific languages and constraints rarely represented in general software repositories.
  • Progressive context extension: Mid-training extends context from 8K to 128K tokens through two progressive stages.Stage 2.1 targets file-level tasks, while Stage 2.2 supports extended debugging and multi-file dependencies.
  • Training data: The method combines curated industrial code with synthetic industrial reasoning QA to cover specialized scenarios underrepresented in public repositories.
  • Synthesis pipeline: Synthetic QA is built by specifying industrial scenarios, generating representative seed code, and synthesizing verified reasoning pairs.Questions target diagnosis, verification, refactoring trade-offs, and impact analysis.
  • Verification: Automated verification checks code execution, static properties, and logical consistency before reasoning traces enter training.The verification layer is intended to exclude fluent but semantically incorrect reasoning patterns.

C.2.2. Curated Industrial Code Data

The curated industrial code data combines authentic development artefacts with agentic debugging trajectories and structure-aware completion tasks. Its mixture is progressively reweighted to emphasize reasoning first and long-context workflows later.

  • Agent Trajectories: Agent trajectories capture multi-step debugging and repair using feedback from simulators, synthesis tools, compilers, and formal verification engines.They expose industrial signals such as synthesis warnings, simulation mismatches, timing violations, and memory-sanitizer reports.
  • Code Commits: Code commits pair developer intent with before-and-after states and include hardware-specific timing, kernel, ABI, and bug-fix scenarios.The curated commits extend general software coverage with industrial optimization and repair examples.
  • Industrial Code Artefacts: Industrial artefacts include test suites, API specifications, configurations, logs, hardware testbenches, timing constraints, synthesis scripts, profiling traces, and sanitizer logs.These artefacts represent the operational context surrounding professional hardware and systems development.
  • File-Level Fill-in-the-Middle: Structure-aware file-level fill-in-the-middle masks coherent code units while respecting Verilog blocks and kernel implementation boundaries.AST-guided masking targets function bodies, methods, code blocks, always blocks, port declarations, loops, and SIMD intrinsic blocks.
  • Data Mixture: Stage 2.1 weights synthetic reasoning QA at 40%, while Stage 2.2 increases agent trajectories to 30% and FIM to 25% for long-context training.Reasoning QA receives 25% in Stage 2.2, while commits, artefacts, and FIM-related categories are rebalanced as specified.

D. Post-training Details

The post-training pipeline constructs industrial tasks with explicit interfaces and toolchains, synthesizes diverse candidate solutions, and verifies them through real execution. Failed candidates become repair trajectories, while filtering and data composition teach direct solution, debugging, and optimization skills.

  • Task Construction: The pipeline builds 2.5M SFT samples from industrial tasks rather than relying primarily on general-purpose coding datasets.Industrial prompts require compilability, testbench success, and measurable resource or performance behavior under specific constraints.
  • Task Construction: Industrial tasks are normalized into requirements, interface constraints, target platforms, toolchains, dependencies, and verification scripts.This structured representation standardizes heterogeneous task specifications.
  • Candidate Synthesis: Candidates are synthesized through reference rewriting, template perturbation, cross-language migration, and retrieval-augmented generation.The complementary channels provide diversity across industrial implementation settings.
  • Execution-Based Verification: Every candidate is validated by compilation, simulation, testing, profiling, or formal checking in a real execution environment.Only solutions passing the applicable checks become high-confidence samples, defining correctness through domain toolchains rather than model judgment.
  • Feedback-Driven Repair: Failed candidates are paired with compiler, runtime, waveform, counterexample, or profiling feedback to create closed-loop repair trajectories.The repaired version is re-verified, teaching diagnosis and iterative correction from real tool output.
  • Filtering and Composition: The verified pool is filtered for executability, stability, and information density, then composed from direct solutions, defect repairs, and optimization samples.This composition targets correctness, debugging, and refinement rather than direct code generation alone.

E. Detailed Synthetic Data Generation Pipeline

The synthetic data pipeline grounds industrial reasoning tasks in realistic engineering scenarios, toolchains, and verification procedures. It combines broad domain coverage with automated checks so the resulting data supplies dense, industrially grounded reasoning signals.

  • Scenario Design: The pipeline targets timing violations, RTL optimization, GPU operator tuning, formal kernel verification, FPGA synthesis errors, memory safety, and related industrial scenarios.These contexts are identified through engineer consultation and analysis of industrial verification reports.
  • Scenario Design: Each scenario specifies design scale, hardware language, timing or resource constraints, verification requirements, and common failure modes.The specifications encode concrete industrial characteristics rather than generic programming requirements.
  • Seed Code: Seed code covers finite-state machines, pipelined datapaths, GPU memory-access patterns, SIMD vectorization, interrupt-driven firmware, and CAD scripting.The selected patterns are frequent in practice but underrepresented in standard data.
  • Reasoning Tasks: Synthetic question-answer pairs probe root-cause analysis, performance diagnosis, correctness verification, refactoring trade-offs, and impact analysis.Answers provide step-by-step reasoning traces for industrial code situations.
  • Verification: Reasoning traces are checked through code execution, static analysis, and logical consistency validation before inclusion.These checks verify outputs, semantic properties, and stepwise soundness.
  • Coverage and Scale: The corpus spans RTL, GPU computing, systems, FPGA synthesis, CAD automation, and embedded firmware, with timing, resource, safety, concurrency, and precision reasoning.Operational contexts include waveform debugging, kernel tuning, synthesis-error resolution, hardware-software co-design, and design-for-test integration.
  • Coverage and Scale: The resulting synthetic data complements curated samples by filling coverage gaps and supplying dense signals for hardware-aware industrial reasoning.

G.1. VeriScope Benchmark – Level 3 Example Case

The examples illustrate industrial benchmark tasks spanning UART design, Verilog debugging, CUDA optimization, and fused Triton operations. They pair concrete specifications or code with compilation, simulation, correctness, and performance-oriented requirements.

  • VeriScope UART Example: VeriScope Problem 221 asks for a synthesizable UART transmitter implementing the 8N1 protocol with LSB-first serialization and busy signaling.The task specifies an 8-bit parallel input, start and stop bits, an idle-high TX line, and resource efficiency.
  • VeriScope UART Example: The UART example’s reference implementation loads input data, emits a start bit, shifts eight data bits, then emits a stop bit and clears busy.The example includes the module interface, reset behavior, transmission state, and testbench connections.
  • Verilog Bug Fix: The Verilog bug-fix example identifies a missing end closing the begin block in the 2’b00 case.The associated analysis also notes that the function does not depend on A, but the structural bug is the missing block terminator.
  • CUDA Optimization: The CUDA example targets a square FP32 matrix multiplication model and requests a compilable custom CUDA implementation named ModelNew.The surrounding code includes tiled-kernel, shared-memory, extension-loading, and wrapper components.
  • Triton Example: The Triton example specifies a fused batch matrix multiplication, RMS normalization, GELU, dropout, and subtraction operation.Its wrapper accepts three tensors plus normalized-shape and dropout-related parameters.
Loading 2603.16790v3…