Source-linked AI summary
Compilation Is Not Correctness: Scan-by-Scan Evaluation of LLM-Generated PLC Programs
Jinyun Chen, Juntao Hu, Haochen Li, Wentao Song, Ningbo Gu, Jianwei Niu, Qing Gao, Rui Wang
TL;DR
Compilation and checks at selected scans can miss intermediate behavioral errors in generated PLC programs. CycleBench compares ST candidates with hidden references after every scan, and CycleGen uses candidate generation, bounded repair, and runtime self-tests. Across the evaluated settings, execution trails compilation by 11.92 percentage points, while CycleGen improves execution pass rate over Direct generation.
Problem
Compilation success and checks on selected properties or scans can leave behavioral errors undetected in PLC programs whose outputs evolve across scans.
Source: §1 · pp. 1–2Method
CycleBench compares generated ST and hidden references after every scan under aligned inputs, initialization, and logical time; CycleGen selects among candidates using bounded repair and runtime self-tests.
Source: §1 · pp. 2–3Results
Execution pass rate trails compilation pass rate by 11.92 percentage points, and CycleGen raises execution pass rate from Direct’s 74.63% to 83.31%.
Source: §1 · pp. 2–3Takeaways & Limitations
Every-fifth-scan checking misses 21.66% of 157 Direct behavioral failures, supporting scan-by-scan evaluation for the supplied traces.
Source: §8 · p. 18Execution pass rate measures agreement on supplied scenarios, not full requirement coverage, unseen inputs, physical behavior, or termination.
Source: §7 · p. 18
Abstract
from arXiv · showhide
Programmable logic controllers (PLCs) automate industrial processes through repeated scans, each reading inputs, executing control logic, and updating outputs. Engineers write this logic in Structured Text (ST), a high-level textual language defined in IEC 61131-3. Large language models (LLMs) can now generate ST from natural-language requirements, yet compilation success and checks on selected properties or sampled scans can leave behavioral errors undetected. We introduce CycleBench, a benchmark of 301 ST tasks with behavioral requirements, public interfaces, reference programs, and executable scenarios. Its runtime executes each generated program and its reference under aligned inputs, initialization, and logical time, comparing their outputs after every scan. Across eight LLMs and baseline generation methods, execution pass rate trails compilation pass rate by 11.92 percentage points; observed failures include output-update, arithmetic, and timing errors. On fixed traces of programs directly generated by five LLMs, checking outputs only at every fifth scan misses 21.66% of the 157 failures detected by per-scan comparison; across sampling phases, the missed-failure rate ranges from 12.10% to 28.03%. These misses stem from transient output discrepancies, whose detection depends on the sampling phase. Building on this evaluation, we propose CycleGen, which generates multiple candidate ST programs, repairs their compilation errors with bounded compiler feedback, and selects one using runtime self-tests generated in a separate model context. CycleGen improves the execution pass rate over direct generation by 8.68 percentage points. Component ablations distinguish repair and self-test selection; candidate-count analysis shows diminishing returns, with no execution-pass-rate improvement beyond five candidates in our experiments despite 38.8% higher token consumption at eight.
1 Introduction
Compilation and selected checks can miss incorrect intermediate PLC outputs, motivating evaluation against shared behavioral requirements after every scan. The paper introduces CycleBench and CycleGen to measure and improve generated ST behavior.
1 Introduction
- A forward-assignment shift register compiles yet produces 0000 instead of 0111 at scan 12; reset restores agreement despite earlier mismatches.The reference assigns outputs in reverse order to preserve old values; the generated program’s outputs differ at scans 12–13.Source: §1 · p. 2, Figure 1 · p. 2
- CycleBench evaluates 301 ST tasks against hidden references after every scan, addressing the difficulty of comparing prior methods with differing task sets and acceptance criteria.The evaluator aligns inputs, initialization, and logical time.Source: §1 · p. 2
- CycleGen raises execution pass rate from Direct’s 74.63% to 83.31% by generating candidates, repairing compilation failures, and selecting with runtime self-tests.Tests are authored beforehand in a separate model context; hidden references and evaluation scenarios remain unavailable during generation and selection.Source: §1 · pp. 2–3
- The paper contributes CycleBench, CycleGen, and empirical analyses of the compile–execution gap and failures hidden by transient discrepancies and scan sampling.Its stated contributions include an eight-model comparison and a five-model fixed-trace analysis.Source: §1 · p. 3
2 Background and Related Work
PLC program behavior depends on scan order, retained state, and timing, so verification evidence and generation workflows address different aspects of correctness. Related work spans formal analysis, reference-based testing, and LLM methods that use feedback.
2.1 Scan Semantics and PLC Verification
- PLC scans read inputs, execute control logic, and update outputs; assignment order, retained function-block state, and timer elapsed time can affect behavior.Behavioral tests therefore need fixed inputs, initialization, time progression, and observed scans.Source: §2.1 · p. 3
- Formal and symbolic tools analyze programs against properties or semantics, while comparing generators also requires a shared task suite with expected behavior.CycleBench supplies that shared suite and compares reference outputs during scan execution.Source: §2.1 · p. 3
2.2 LLM Generation, Testing, and Repair
- CodeT, Self-Debugging, MBR-EXEC, and DOCE use generated tests or execution feedback for candidate selection, repair, or refinement; generated expectations and repairs can be wrong.This motivates separating feedback-based selection and repair from reference-based final scoring.Source: §2.2 · p. 3
2.3 LLM-Based PLC Methods
- PLC generation approaches add retrieval, specialized training, or structured representations, while related test and mutation datasets examine test quality.Early work directly generated PLC control logic.Source: §2.3 · p. 4
- LLM4PLC, Agents4PLC, AutoPLC, and SemaPLC combine generation with compiler, formal, or runtime feedback, but their workflows differ.SemaPLC combines sampled traces with persistent assertions; its native verification gate controls delivery.Source: §2.3 · p. 4
3 Benchmark: CycleBench
CycleBench evaluates generated ST by comparing candidates with references after every scan under aligned inputs, initialization, and logical time. Its 301 tasks are constructed and reviewed for behavioral evaluation, and the formative study finds execution failures across task groups.
3 Benchmark: CycleBench
- CycleBench pairs reviewed public specifications with hidden references and scenarios, comparing candidate and reference outputs after every scan under aligned conditions.The benchmark contains 301 ST function blocks and programs; references and evaluation scenarios remain hidden from generation.Source: §3 · p. 4
3.1 Benchmark Construction
- Six ST collection routes feed preprocessing, task construction, and joint engineer review; reference execution supplies scenario expected outputs for the final 301-task roster.Two PLC engineers review requirements alongside reference ST, interfaces, and scenario suitability.Source: Figure 2 · p. 4, §3.1 · p. 5
3.2 Benchmark Evaluation
- The evaluator drives candidate and reference with identical inputs, initialization, scan intervals, and reset boundaries, then compares outputs after each scan.Each scenario begins in fresh sessions, state persists between scans, and logical time follows the configured schedule rather than host elapsed time.Source: §3.2 · pp. 5–6
3.3 Benchmark Characteristics
- CycleBench has 287 function blocks and 14 programs, across 684 scenarios and 61,424 scans; syntax-feature counts do not establish semantic coverage.Six tasks exceed 1,000 scans in total, while most tasks have short scenarios.Source: §3.3 · p. 6, Table 2 · p. 6
3.4 Comparison with the Agents4PLC Benchmark
- CycleBench brings 90 Agents4PLC-derived tasks into a common per-scan evaluator, rather than evaluating only formal properties or sampled traces.SemaPLC combines sampled runtime traces with persistent assertions; the paper describes an interface-adapted comparison separately.Source: §3.4 · pp. 6–7
3.5 Formative Study
- Pooled execution pass rate falls from 87.17% in the shortest reference-length group to 48.61% in the longest, while the compile–execution gap grows from 3.67 to 24.07 points.All eight models perform worse in the longest group, although intermediate rates need not decrease monotonically; every source group has a positive pooled gap.Source: §3.5 · p. 7, Table 3 · p. 7
4 Generation Framework: CycleGen
CycleGen generates multiple ST candidates, applies bounded compiler-guided repair, and uses one shared suite of model-generated tests to select a candidate. The suite is fixed before code generation, and hidden-reference scoring remains separate.
4 Generation Framework: CycleGen
- CycleGen supplements bounded compiler-guided repair with behavioral comparison of multiple candidates using one shared suite of model-generated scan-level tests.Tests are fixed before code generation, and the selected program is frozen before hidden evaluation.Source: §4 · p. 7, Figure 3 · p. 8
4.1 Candidate and Test Generation
- The model authors tests and code in separate contexts; tests are fixed first and withheld from code generation, while both stages use the same specification.Candidate-blind authorship controls information flow, but a shared misunderstanding can affect both code and predicted outputs.Source: §4.1 · p. 8
4.2 Compiler-Guided Repair
- Each nonempty candidate failing the ST-to-C front-end check receives at most one repair request; a revised program replaces it only if the interface is preserved and the check passes.This gate does not perform native GCC compilation or linking.Source: §4.2 · p. 9
4.3 Runtime Self-Testing
- Self-Test builds eligible candidates and runs shared test inputs; scored cases pass only with complete matching execution, while runtime validity means cases complete, not that predictions are correct.Wrong predictions can penalize correct code or favor incorrect code.Source: §4.3 · p. 9
4.4 Candidate Selection
- Selection ranks eligible programs by complete execution, then more passing scored cases, then fewer mismatched output-field assertions, with generation order breaking ties.With no usable scored suite, it returns the earliest eligible program; if no candidate is eligible, it returns the first original candidate and its observed status.Source: §4.4 · p. 9
5 Experimental Settings
The experiments evaluate CycleBench and CycleGen across eight model deployments and compare adapted baselines under shared tasks and interfaces. They also specify candidate-generation settings, runtime evaluation criteria, failure analyses, and statistical and inference-cost procedures.
5 Experimental Settings
- Eight model deployments were evaluated on the 301-task roster, with one returned program per task and model; the models were pretrained and used without retrieval or benchmark fine-tuning.The roster served as the evaluation set, and unsuccessful attempts remained in the denominator.Source: §5 · p. 10
- The comparison included Direct, three baseline adaptations from pinned public materials, and a SemaPLC reconstruction; results characterize adaptations rather than complete native reproductions.The Agents4PLC retrieval workflow was reconstructed with TF–IDF because its retrieval implementation and private vector store were unavailable.Source: §5 · p. 10, Table 4 · p. 10
- Initial generation used temperature 0.7, while planning, test authoring, and repair used 0 except AutoPLC’s code editor; five-candidate calls shared candidate one with Direct.Ablations shared initial candidates and candidate-blind tests, and invalid test suites invoked a fallback.Source: §5 · p. 11
- CycleGen self-test suites were limited to six cases and forty scans, with admission filters requiring scored cases and scans, complete inputs and expected outputs, and distinct writable-input values.The fixed filters bound cost and excluded degenerate suites.Source: §5 · p. 11
- Execution pass rate requires interface preservation, compilation, completed scenarios, and scan-by-scan output/inout matches; the reported gap is compilation rate minus execution pass rate.Each condition scores one returned program, and unsuccessful outcomes remain in the denominator.Source: §5 · pp. 11–12
- Failure analyses classified Direct compilation and behavioral failures, while observation-granularity tests compared fixed traces at every scan, sampled strides, and final scans.The study also used task-cluster bootstrap intervals, corrected sign-flip tests for specified contrasts, and counted model calls and tokens while excluding local compilation and execution.Source: §5 · p. 11
6 Evaluation
The evaluation uses a common evaluator across the 301-task roster to compare compilation, execution behavior, CycleGen components, and observation granularity. It finds that compilation success does not ensure behavioral agreement, while CycleGen improves execution pass rates and coarser scan checks miss failures on fixed traces.
6.1 RQ1: How Effective Is CycleGen at Generating Executable ST Programs?
- 83.31% execution pass rate for CycleGen exceeds Direct by 8.68 points, while the overall compilation–execution gap is 11.92 points.The gap appears across every model–method combination, including some with compilation rates above 99%; CycleGen retains an 11.59-point gap.Source: §6.1 · pp. 12–13
6.2 RQ2: Which Failures Occur During Compilation and Execution?
- 53.42% of Direct compilation failures involve malformed statements or declarations, while output-update, arithmetic, and timing/call-order errors account for 65.6% of behavioral failures.These are symptoms and reviewer-assigned categories, not definitive root causes; Table 6 summarizes the failure categories.Source: Table 6 · p. 13, §6.2 · p. 14
6.3 RQ3: How Do CycleGen’s Modules Contribute to Execution Correctness?
- On controlled pools, Compile-Fix adds 2.28 points and self-test selection adds 1.62 points; the ablations evaluate these components under fixed controls.Table 7 reports ablations on 2,408 task–model pairs; self-test and selection-gate effects are evaluated jointly.Source: §6.3 · pp. 14–15, Table 7 · p. 14
- Extending CycleGen from five to eight candidates raises token use by 38.8% without improving selected execution pass rate.The unconstrained oracle rate rises over the same range, but its hidden behavioral matches need not satisfy the public interface or be selectable.Source: §6.3 · pp. 15–16, Figure 4 · p. 15
6.4 RQ4: How Much Does Observation Granularity Change Correctness Verdicts?
- Among 157 Direct behavioral failures, checking every fifth scan misses 34 (21.66%), even when scenario endpoints are retained.On these fixed traces, final-scan-only checking misses 98 failures; the apparent pass-rate increase reflects missed evidence, not better code.Source: §6.4 · pp. 16–18, Figure 6 · p. 17
- 144 of 157 failures show within-scenario output re-agreement, and every fixed sampling phase misses failures on the audited traces.Across phases, every-fifth-scan checks miss 19–44 failures; visible output agreement does not establish internal-state equivalence.Source: §6.4 · pp. 17–18
7 Threats to Validity
The paper qualifies its findings by the benchmark’s scope, evaluation setup, and study design. These boundaries limit what execution pass rates and comparisons can establish beyond the evaluated tasks and conditions.
7 Threats to Validity
- Execution pass rate measures agreement on supplied scenarios, not full requirement coverage, unseen inputs, physical behavior, or termination.Exact equality may also count numerical rounding differences as failures.Source: §7 · p. 18
- The eight deployments cover 301 tasks, while taxonomy and observation analyses cover five models; source concentration and limited scenarios constrain diversity and coverage.Training exposure is unknown, and fixed budgets, one runtime, and provider-specific reasoning limit transfer to other dialects and physical controllers.Source: §7 · p. 18
- CycleGen was developed using this benchmark, so independent tasks and repeated generations are needed to assess generalization and sampling stability.Ablations estimate conditional effects under the reported controls.Source: §7 · p. 18
8 Conclusion
CycleBench evaluates generated Structured Text with scan-by-scan execution, and the results show a gap between compilation and behavioral correctness. CycleGen improves execution pass rate, while coarser observation can miss transient failures.
8 Conclusion
- Across eight models and four baseline methods, execution pass rate trails compilation by 11.92 percentage points.CycleBench evaluates 301 PLC tasks through scan-by-scan execution.Source: §8 · p. 18
- Every-fifth-scan checking misses 21.66% of 157 behavioral failures in the five-model Direct analysis, as transient discrepancies and sampling phase affect detection.These results concern fixed traces with scenario endpoints retained.Source: §8 · p. 18
- CycleGen raises execution pass rate from 74.63% to 83.31%, while increasing the candidate count beyond five adds token cost without improving execution pass rate.Controlled-pool ablations support contributions from repair and self-test selection.Source: §8 · p. 18