PaperlayerView exact revision on arXiv

Source-linked AI summary

PLCWorld: Benchmarking LLM-Generated PLC Programs in Closed-Loop Plant Simulation

Yunji Kim, Yunseok Lee, Hyunwoo Seo, Jaerim Choi, Woojin Lee

arXiv:2610.02982v1cs.AI

TL;DR

Evaluating LLM-generated PLC programs requires observing how their commands affect device and workpiece states, not just the commands themselves. PLCWorld couples ST execution with simulated plant responses and sensor feedback, using tasks organized by control-dependency scope and separate completion and safety measures. Across evaluated task groups, Hard tasks had substantially lower Task Success, and completion and safety outcomes did not always align.

  • Problem

    Evaluating generated PLC programs requires evidence about whether they complete tasks and satisfy safety constraints as commands affect device and workpiece states.

    Source: §1 · pp. 1–2

  • Method

    PLCWorld couples cyclic Structured Text execution with simulated plant updates and sensor feedback, and evaluates tasks using separate Task Success and Safety Violation measures.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Hard tasks had substantially lower Task Success across evaluated models and methods; direct GPT-5.5 achieved 82.70% SR on Easy tasks versus 25.10% SR on Hard tasks.

    Source: §1 · p. 2, §5.2 · p. 8

  • Takeaways & Limitations

    Task completion and observed safety violations are distinct outcomes, supporting their separate reporting in closed-loop PLC evaluation.

    Source: §5.2 · p. 8, §6 · p. 10

    PLCWorld’s safety metric covers specified constraints under evaluated conditions and is not safety certification; physical deployment requires independent engineering review and equipment-specific validation.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Programmable logic controllers (PLCs) coordinate industrial equipment by reading sensor inputs and issuing control commands. Evaluating whether large language model (LLM)-generated PLC programs satisfy task requirements and safety constraints requires observing how their commands affect device and workpiece states. We introduce PLCWorld, a common closed-loop execution environment and benchmark that couples Structured Text (ST) execution with simulated plant responses and sensor feedback. Grounded in control relations identified in industrial PLC programs and engineering documentation, PLCWorld contains 100 synthetic tasks and 473 registered task-condition pairs across Motion Control and Material Handling, with difficulty defined by control-dependency scope. A common protocol reports Task Success and Safety Violation separately. Validation combines practitioner review, reference and alternative programs, targeted counterexamples, specification-evaluator alignment checks, and comparisons with independent ST runtimes. Reference and alternative programs satisfy their applicable cases, while all 542 targeted counterexamples activate their designated evaluator rules under at least one registered condition. Execution Gap relates submission-profile acceptance to subsequent task failure or observed Safety Violation. Across the constructed task groups, direct GPT-5.5 achieves 82.70% Task Success on Easy cases but 25.10% on Hard cases. Evaluations of six LLMs and four adapted generation-and-verification workflows further expose differences between completion, safety, and generation cost. Our code, simulation environment, benchmark tasks, and baseline implementations are publicly available at https://yunji0516.github.io/PLCWorld/.

1 INTRODUCTION

PLC programs must coordinate commands with actual device and workpiece states, because issuing commands in sequence does not ensure that required operations have completed. PLCWorld evaluates generated ST programs through closed-loop plant execution and reports completion and safety outcomes separately.

1 INTRODUCTION

  • PLC control dependencies govern when operations may start, which states must be maintained, and when operations count as complete.These dependencies arise within functions, between functions in a job, and between jobs sharing equipment.Source: §1 · p. 1
  • A command does not guarantee that an operation has completed; dependent actions must account for actual device and workpiece states.A fixed wait can end before grasping is complete, allowing a lift to start without the workpiece even when commands are issued in the intended order.Source: §1 · p. 2
  • PLCWorld combines cyclic ST execution, simulated device and workpiece behavior, and sensor feedback in a benchmark with separate Task Success and Safety Violation measures.Its tasks cover Motion Control and Material Handling, with difficulty based on the scope of control dependencies.Source: §1 · p. 2
  • The benchmark’s task suites are grounded in industrial PLC programs and engineering materials, and its protocol compares upstream checks with plant outcomes on the same submissions.That comparison is used to analyze the Execution Gap.Source: §1 · p. 2
  • Hard tasks show substantially lower Task Success than Easy and Medium tasks across the evaluated models and generation-and-verification methods.Failures to establish valid completion by the deadline were the most frequent recorded failure signature.Source: §1 · p. 2

2 RELATED WORK

Prior work evaluates generated PLC code using methods including formal verification, runtime tests, and physical demonstrations, while development tools offer simulation and program execution. PLCWorld combines closed-loop plant execution with common tasks, conditions, and completion and safety criteria.

2 RELATED WORK

  • Prior PLC-generation studies use retrieval, compiler feedback, formal verification, expert review, and runtime testing, with reported validation scope summarized in Table 1.Examples include LLM4PLC, Agents4PLC, AutoPLC, Spec2Control, and SemaPLC.Source: §2 · p. 3
  • Prior dependency assessments include formal verification and runtime tests, while LLM4PLC also reports a physical handling demonstration.PLCWorld evaluates Task Success and Safety Violation from plant states and events under specified operating conditions.Source: §2 · p. 3
  • PLC development tools support controller simulation, virtual commissioning with equipment models, and execution of IEC 61131-3 programs through open runtimes.Comparing generated programs with these tools requires common tasks, operating conditions, and completion and safety criteria.Source: §2 · p. 3
  • Execution-based evaluations in other domains use interactive databases, physics simulation, and simulated motion control, while PLC evaluation must preserve dependencies between plant states and subsequent control.Plant states both change in response to outputs and constrain later control.Source: §2 · p. 3

3 PLCWORLD

PLCWorld evaluates ST programs through interaction with a simulated plant, using task specifications and execution conditions to assess completion and safety. Its benchmark organizes tasks by control-dependency scope and reports Task Success, Safety Violation, and Execution Gap separately.

3.1 PLANT EXECUTION ENVIRONMENT

  • Each ST scan reads sensor inputs and produces control outputs; the plant then advances, and updated-state observations feed the next scan.The cycle retains program state and applies execution conditions; sensor feedback comes from the updated plant rather than a prerecorded trace.Source: §3.1 · p. 4

3.2 BENCHMARK TASKS

  • Tasks specify goal plant states, required intermediate events and precedence, plus separate safety constraints; any implementation meeting them is valid.The task information also includes operating conditions and distinguishes the completion deadline from the evaluation endpoint.Source: §3.2 · p. 5
  • Difficulty reflects dependency scope—from coordination within a basic function to coordination across functions or between jobs—not device count or code length.Short sequences or retained state within one basic function remain Easy, and serial or fixed-priority control can still satisfy Hard tasks.Source: §3.2 · pp. 5–6

3.3 EVALUATION

  • Task Success measures required completion, while Safety Violation measures observed breaches separately; a case can succeed despite a violation or fail without one.Task Success requires the goal state and required events, precedence, and hold conditions within the deadline; violation monitoring continues past completion claims.Source: §3.3 · p. 6
  • Execution Gap is the fraction of cases passing an upstream check that later show task failure or an observed violation.Cases with both outcomes count once, and each check is evaluated separately using the same submitted program as the plant evaluation.Source: §3.3 · p. 6

4 BENCHMARK CONSTRUCTION AND VALIDATION

PLCWorld constructs synthetic tasks from control relations identified in industrial programs and engineering materials, then validates the benchmark through complementary checks. Reference and alternative programs satisfy applicable cases, targeted counterexamples activate evaluator rules, and runtime comparisons show close agreement with independent ST execution.

4.1 INDUSTRIALLY GROUNDED TASK CONSTRUCTION

  • Task construction identifies recurring control relations in 2,201 PLC and related automation project files and engineering documents, then instantiates them as synthetic tasks without reproducing source implementations.The relations include feedback-conditioned actuation, interlocks, support and handover, workpiece-state dependencies, and shared-equipment coordination.Source: §4.1 · pp. 6–7

4.2 BENCHMARK VALIDATION

  • Four practitioners review all 100 specifications for information sufficiency, completion and safety clarity, and dependency-based difficulty; feedback informs revisions before evaluation.The review covers six suite–difficulty groups and records individual task issues.Source: §4.2 · p. 7
  • Reference programs achieve Task Success without observed Safety Violation across all 100 tasks and 473 registered cases; targeted counterexamples trigger their designated rules under at least one condition.Alternative implementations also achieve the same outcomes in their tested executions, supporting acceptance beyond a single solution.Source: §4.2 · p. 7
  • A bidirectional audit checks that published requirements appear in executable rules and that rules are supported by specifications, with automated checks of inputs and interfaces.The checks also cover scan periods, evaluator bindings, and submission constraints; findings guide revisions and regression checks.Source: §4.2 · p. 7
  • MATIEC/GCC comparisons agree on traces in 1,162/1,199 cases, Task Success in 1,195, and Safety Violation in all cases.Four Task Success differences involve one program calling a TON instance twice per scan.Source: §4.2 · p. 7

5 EXPERIMENTS

Across the evaluated models and workflows, Hard tasks show substantially lower Task Success, while completion and safety do not move together consistently. Execution Gap and pooled failure signatures describe distinct aspects of plant-level outcomes.

5.1 EXPERIMENTAL SETUP

  • The evaluation compares six direct-generation LLMs and four adapted generation-and-verification workflows, using three independent single-turn generations per task for direct models.Final programs are evaluated under all registered task conditions at task-specific PLC scan periods, with a common upstream check separate from framework-internal compilation and verification.Source: §5.1 · pp. 7–8

5.2 MAIN RESULTS

  • 82.70% and 84.09% SR on Easy and Medium tasks, respectively, versus 25.10% on Hard tasks for direct GPT-5.5 across both suites.Other models and frameworks show similar declines; these comparisons describe task groups rather than isolating dependency scope.Source: §5.2 · p. 8, Figure 5 · p. 8
  • 35.25% vs. 27.87% SR for Gemini 3.1 Pro versus GPT-5.5 on Hard Material Handling, alongside 51.95% vs. 32.24% SVR.Task Success alone does not capture observed safety violations, so the paper reports Task Success and Safety Violation separately.Source: §5.2 · p. 8, Table 5 · p. 8
  • AutoPLC improves both SR and SVR on Hard Material Handling but worsens both on Hard Motion Control relative to matched GPT-5.5.All evaluated frameworks use more calls, tokens, and time than direct GPT-5.5 on the shared 18-task subset; higher overhead is not accompanied by consistent plant-level gains.Source: Table 4 · p. 8, §5.2 · p. 9

5.3.1 EXECUTION GAP

  • 32.10% vs. 37.88% Execution Gap for Gemini 3.8 Flash versus GPT-5.5, but Flash has fewer accepted cases and fewer assigned cases achieving Task Success without an observed Safety Violation.Execution Gap is conditional on passing the local submission-profile check; lower Execution Gap alone does not imply better overall outcomes.Source: §5.3.1 · p. 9, Figure 6 · p. 9

5.3.2 FAILURE ANALYSIS

  • Completion timing is the most frequent recorded signature at 73.0%, followed by completion retention at 36.7%, actuation violations at 24.7%, and resource constraint violations at 18.8%.The pooled percentages describe adverse-case composition, not overall failure rates, model differences, or code root causes.Source: §5.3.2 · p. 9, Figure 7 · p. 9, Table 6 · p. 9

6 CONCLUSION

PLCWorld is presented as a common resource for reproducible closed-loop evaluation of LLM-generated ST programs, with separate Task Success and Safety Violation judgments. Its simulation results are bounded to specified constraints and evaluated conditions, not safety certification.

6 CONCLUSION

  • PLCWorld provides 100 tasks and 473 registered task–condition pairs, with separate Task Success and Safety Violation judgments.Reference and alternative programs support task feasibility and acceptance of tested valid implementations, while controlled counterexamples assess targeted violation detection.Source: §6 · p. 10
  • The reported safety metric covers specified constraints under evaluated conditions and does not constitute safety certification.Physical deployment requires independent engineering review and equipment-specific validation.Source: ETHICS STATEMENT · p. 10
  • Code, simulation environment, benchmark tasks, evaluator, submission interface, and baseline implementations are publicly available; confidential industrial source materials are excluded.Four practitioners reviewed public task specifications for plausibility, sufficiency, clarity, and difficulty appropriateness.Source: REPRODUCIBILITY STATEMENT · p. 10, ETHICS STATEMENT · p. 10

A LIMITATIONS

PLCWorld’s evidence is bounded by its source coverage, modeled plant behavior, and registered execution conditions.

A LIMITATIONS

  • Limited source coverage constrains generalization across industrial settings, while source confidentiality restricts independent inspection of the industrial grounding.Source: A LIMITATIONS · p. 14
  • Physical plant-response validation against equipment measurements remains future work, and runtime comparisons cover selected constructs and programs with residual cross-runtime differences.Source: A LIMITATIONS · p. 14
  • Evaluation covers specified requirements under registered conditions; unmodeled hazards and untested executions remain outside its scope.Source: A LIMITATIONS · p. 14

B BENCHMARK CONTENTS AND TASK SPECIFICATIONS

PLCWorld documents 100 tasks and their specifications, execution conditions, and task-specific plant abstractions. Difficulty is based on control-dependency scope, while composition statistics are descriptive.

B.1 BENCHMARK COMPOSITION

  • Table 7 summarizes the number of tasks, standard execution conditions, public I/O addresses, and operational device-family counts by suite–difficulty group; composition statistics describe diversity and do not assign difficulty.Devices use an author-defined 17-family taxonomy, counting each family once per task. Requirement-family assignments follow specification obligations, are non-exclusive, and are separate from difficulty.Source: §B.1 · p. 14

B.2 PUBLIC TASK SPECIFICATION FORMAT

  • Each public task specification defines required observable behavior while leaving internal program structure open within the supported ST profile.It specifies objectives, deadlines or termination conditions, I/O semantics, initial state, operating conditions, device-response semantics, completion requirements, and safety constraints.Source: §B.2 · p. 15, Table 9 · p. 16

B.3 STANDARD EXECUTION CONDITIONS

  • The 100 tasks define 473 standard task–condition pairs, varying permitted initial states, device responses or timing, and external inputs while preserving each task’s requirements.Sensor feedback is generated from the evolving plant state and the program’s preceding outputs, not supplied as a fixed trace.Source: §B.3 · p. 15

B.4 REPRESENTATIVE TASK

  • Representative tasks illustrate progression from dependencies within a basic function to dependencies between functions and coordination between independent jobs.Examples are illustrative; evaluation uses each task ID’s complete frozen public specification.Source: §B.4 · pp. 15–16, Table 10 · p. 16, Table 11 · p. 17

C PLANT MODEL DETAILS

  • The plant model uses task-specific abstractions and an author-defined taxonomy of 17 operational device families, not a general-purpose physics model or industry-standard taxonomy.The plant instantiates only the devices and state needed to realize the task specification and evaluate completion and safety.Source: C PLANT MODEL DETAILS · p. 16, §C.1 · p. 17, Table 12 · p. 18

C.2 TRANSITION RULES

  • Device states change through task-specific response dynamics, timing, and preconditions; a command does not instantly establish the corresponding physical state.An inadmissible command may be blocked, have no physical effect, or be recorded as a safety violation; success depends on required goal states and event history, not command bits alone.Source: §C.2 · p. 19

C.3 WORKPIECE AND TRANSFERABLE-FIXTURE RELATIONS

  • Material-handling models track task-relevant properties such as item location, holder, support, identity, quantity, process state, and transition history.Motion Control tasks may instead track a fixture, carrier, jig, or request-associated load, or only device motion and feedback.Source: §C.3 · p. 19

C.4 MODELING SCOPE

  • Plant models combine discrete events with task-relevant numerical state; physical effects such as friction or deformation are represented only when explicitly encoded.Each model is an executable abstraction of its task specification, not a complete physical model of the equipment.Source: §C.4 · p. 19

D INDUSTRIAL GROUNDING AND TASK CONSTRUCTION

PLCWorld abstracts control relations from industrial materials into synthetic tasks, then evaluates programs through recorded plant behavior and separately defined completion and safety judgments. Practitioner review and executable checks address specification and evaluator validity.

D.1 INDUSTRIAL SOURCE MATERIALS AND RELATION ABSTRACTION

  • Industrial PLC programs and engineering documents are cross-referenced to abstract actions, observable feedback, prerequisite states, and states to maintain, while source-specific details and source artifacts are excluded from the release.Source: §D.1 · p. 20, Table 13 · p. 20

D.2 SYNTHETIC TASK INSTANTIATION

  • Each synthetic task instantiates an industrially informed objective in a new plant configuration with specified devices, workpieces, initial state, I/O semantics, and model parameters.Task requirements define event ordering and states to maintain without prescribing internal program implementation; Figure 9 illustrates abstraction rather than physical or implementation equivalence.Source: §D.2 · pp. 20–21

E.1 EXECUTION RECORDS AND TEMPORAL CONDITIONS

  • Execution records capture scan inputs, committed outputs, updated plant states, and events, enabling evaluation against state, event, precedence, and hold requirements.Completion claims are checked after outputs take effect and against the recorded state and event history.Source: §E.1 · p. 21

E.2 TASK SUCCESS AND SAFETY VIOLATION

  • Task Success requires all functional requirements, including goal state, events, ordering, and retention, by the task deadline; Safety Violation records any observed breach of a specified constraint.The metrics are separate: a case can succeed with a violation or fail without one, and an observed violation remains counted even if the plant recovers.Source: §E.2 · p. 21

E.3 ERRORS, TERMINATION, AND METRIC ELIGIBILITY

  • Metric eligibility distinguishes cases entering plant execution from those that do not, excludes infrastructure errors from aggregation, and conditions Execution Gap on passing an upstream check.The common check covers the supported ST profile and required I/O interface, not external compilation or formal verification.Source: §E.3 · pp. 21–22

F.1 TASK SPECIFICATION VALIDATION

  • Four experienced practitioners review all 100 task specifications for industrial plausibility, information sufficiency, completion and safety clarity, and dependency-based difficulty.They assess the public specifications, provide group-level assessments, and record task-specific concerns.Source: §F.1 · p. 22, §F.1.1 · p. 22, §F.1.2 · pp. 22–23
  • The practitioner review examines whether task assumptions and control relations are plausible and sufficiently specified, including properties not established solely by executable checks.Source: §F.1 · p. 22

F.1.2 REVIEW SCOPE AND PROCEDURE

  • The review process records concerns about task wording, implementation information, I/O or device behavior, completion and safety requirements, and difficulty assignment.Source: §F.1.2 · pp. 22–23

F.1.3 REVIEW RESULTS BY GROUP

Validation checks cover practitioner assessments, known-valid implementations, targeted counterexamples, specification–evaluator alignment, and selected ST-runtime comparisons. The evidence supports the audited tasks and conditions rather than exhaustive correctness over arbitrary programs or plant states.

F.1.3 REVIEW RESULTS BY GROUP

  • Reviewers rated information sufficiency and completion and safety clarity Appropriate in all 24 assessments per criterion; industrial plausibility received 16 Appropriate and 8 Minor Revision Required ratings.Comments led to revisions clarifying operational rationale, completion requirements, permitted action sequences, and difficulty explanations; 18 specifications were revised before the benchmark was frozen.Source: §F.1.3 · p. 23, §F.1.4 · p. 23, §F.1.6 · pp. 24–25

F.1.5 FEEDBACK ON DIFFICULTY LEVELS

  • Three of four reviewers considered dependency-based difficulty generally appropriate, and seven task explanations were clarified without reassigning any task.The authors treat this as practitioner-perspective evidence, not proof of an objective or universal ordering of PLC difficulty.Source: §F.1.5 · p. 24

F.2.1 FEASIBILITY AND VALID ALTERNATIVES

  • All reference programs succeeded without observed safety violations across 473 task–condition pairs, and 72 alternative programs passed all 378 corresponding executions.The alternative-program test checks known valid implementations beyond the references, not every possible valid program.Source: §F.2.1 · p. 25

F.2.2 TARGETED VIOLATION DETECTION

  • All 542 targeted counterexample programs activated their designated evaluator rule under at least one registered condition; 723 of 739 executions were rejected.The remaining 16 executions occurred under conditions without the targeted violation; those programs activated their rules under other registered conditions.Source: §F.2.2 · p. 25

F.3 SPECIFICATION–EVALUATOR ALIGNMENT VALIDATION

  • All 100 tasks passed public-input and execution-path checks, and comparison of 461 numerical and operating-range items found no unresolved mismatch.A bidirectional semantic audit compared public obligations with executable judgments and corrected identified discrepancies; the checks are not exhaustive over arbitrary ST programs or continuous plant states.Source: §F.3 · p. 25

F.4 ST RUNTIME VALIDATION

  • Frozen-evaluator reruns matched trace hashes and judgments across 473 conditions; three diagnostic programs also agreed with MATIEC/GCC on all 1,320 observations over 360 scans.The diagnostics exercised selected ST constructs, and a Siemens TON scan-boundary discrepancy was corrected and rechecked.Source: §F.4 · p. 25
  • Across 1,199 paired generated-program cases, complete scan-level records agreed in 1,162, Task Success judgments in 1,195, and Safety Violation judgments in all 1,199.All four Task Success differences involved a Qwen3.5-Plus MOT-E08 program calling the same TON instance twice per scan; PLCWorld marked these false and MATIEC/GCC true, with no Safety Violation in either runtime.Source: §F.4 · pp. 25–26

G EXPERIMENTAL SETUP AND REPRODUCIBILITY

  • Direct-generation evaluations used three independent single-turn outputs per task without repair or best-of selection, while adapted generation-and-verification frameworks used method-specific implementations and budgets.Full-benchmark framework evaluations retained final candidates even after failed internal checks, timeouts, or budget exhaustion; plant outcomes were not used for candidate selection or repair.Source: §G.2 · p. 26, §G.3 · pp. 28–29

G.5 STATISTICAL ANALYSIS AND EFFICIENCY MEASUREMENT

Framework comparisons show no paired SR interval excluding zero against matched direct runs, while only three generation repetitions provide a descriptive view of variability. The appendix also specifies task-cluster uncertainty estimates, joint task-and-safety outcomes, and efficiency measurement boundaries.

G.5 STATISTICAL ANALYSIS AND EFFICIENCY MEASUREMENT

  • The 95% SR intervals use 10,000 stratified task-cluster bootstrap replicates, with paired SR differences using identical task resamples.Intervals are conditional on generated programs and are not adjusted for multiple comparisons.Source: §G.5 · p. 29
  • Efficiency is measured on 18 shared tasks, and latency comparisons are limited by differing timing boundaries across methods.Reported medians cover calls, revisions, tokens, and elapsed time; the measurements are API usage and latency, not normalized compute or monetary cost.Source: §G.5 · p. 29
  • The reported results use 473 registered task–condition pairs and omit a separate robustness evaluation under additional heldout operating conditions.Source: §G.5 · p. 29

H.1 EXECUTION COVERAGE AND JOINT OUTCOMES

  • Task completion can coincide with a safety violation, while task failure can occur without one; interrupted cases without observed violations are not established safe through the full endpoint.The joint outcome table covers plant-entering cases; pre-entry failures are excluded.Source: §H.1 · p. 30, Table 21 · p. 30

H.2 CONFIDENCE INTERVALS AND GENERATION VARIABILITY

  • All framework-versus-matched-run SR intervals include zero; Agents4PLC and SemaPLC intervals versus the three-run direct baseline are below zero.The paired task-cluster intervals exclude framework rerun variability.Source: §H.2 · p. 31, Table 24 · p. 32
  • Three independent generation repetitions provide only a descriptive view of variability, with all repetitions included in pooled SR.Each repetition covers 100 programs and 473 evaluation cases.Source: §H.2 · p. 31, Table 25 · p. 32

H.3 FRAMEWORK EVALUATION OUTCOMES

Framework outcomes are reported against a common submission-profile check, with Execution Gap conditioned on its acceptance and later task failure or observed safety violation. The appendix distinguishes framework-specific checks, efficiency measurements, and evaluator-recorded failure signatures from causal or source-code diagnoses.

H.3 FRAMEWORK EVALUATION OUTCOMES

  • Framework-internal check statuses have method-specific meanings, and status-group outcome differences do not identify the effect of successful verification.The checks can inspect different artifacts or acceptance criteria, and the status groups differ in task composition.Source: §H.3 · pp. 32–33, Table 27 · p. 33
  • Among compiler-accepted final programs, 208/469 LLM4PLC cases and 179/452 AutoPLC cases have task failure, an observed safety violation, or both.These adverse counts are not complements of SR because successful cases can also violate safety constraints.Source: §H.3 · p. 33
  • MATIEC rejects 31 reference programs for conversion notation or mixed-type numeric expressions, while diagnostic rewrites compile; this supports compatibility comparison only, not execution equivalence.The common submission-profile check remains the cross-method upstream check.Source: §H.3 · p. 33

H.4 GENERATION EFFICIENCY

  • Efficiency results use fresh runs on an 18-task subset, where all 18 LLM4PLC runs timed out, errored, or exhausted budget; these are not full-benchmark termination rates.Six of 18 Gemini 3.8 Flash runs reached the configured output-token limit; the reported costs include incomplete verification workflows.Source: §H.4 · p. 33, Table 28 · p. 34

I.1 COMMON UPSTREAM CHECK AND EXECUTION GAP COUNTS

  • Execution Gap uses accepted task–condition cases under PLCWorld’s local submission-profile check, and subsequent program-attributable initialization or execution errors count as failures.The check parses ST and verifies structure, I/O types, read-only inputs, and supported constructs before runtime initialization and plant execution.Source: §I.1 · p. 34, Table 29 · p. 34

I.2 RULE-TO-SIGNATURE MAPPING

  • Failure signatures are evaluator-recorded manifestations, assigned through a fixed rule-to-signature mapping; adverse cases without failed-rule detail receive no inferred signature.Support-prohibited rules are split by recorded reason, and unseen rules stop the mapping audit.Source: §I.2 · p. 35, Table 30 · p. 35

I.3 OUTCOME AND SIGNATURE DISTRIBUTIONS

  • 41 GPT-4o, 11 Gemini 3.8 Flash, and 14 Qwen3.5-Plus accepted cases never enter plant execution, counting as Execution Gap failures without safety judgments or plant-level signatures.Among adverse plant-entered cases, signature labels may overlap, and 724 of 4,339 cases lack failed-rule detail.Source: §I.3 · pp. 35–36, Table 31 · p. 36

I.4 REPRESENTATIVE EVALUATION OUTCOMES

The representative outcomes are selected to illustrate distinct outcomes and multi-label mapping, not their frequency. Recorded evaluator outcomes establish reported case results, while source-code root-cause identification requires inspection of the submitted ST and scan-level trace.

I.4 REPRESENTATIVE EVALUATION OUTCOMES

  • Representative evaluator-outcome examples illustrate distinct outcomes and multi-label mapping rather than outcome frequency.Source: §I.4 · p. 36, Table 32 · p. 36
  • Stored case records identify the frozen program, condition, evaluator version, failed rules, and execution-trace hash, but root-cause identification also requires inspecting ST and scan-level traces.The table’s final column gives observed rule outcomes, not inferred source-code causes.Source: §I.4 · p. 36, Table 32 · p. 36
Loading 2610.02982v1…