PaperlayerView exact revision on arXiv

Source-linked AI summary

PLCBench: Can Autonomous LLM Agents Turn PLC Access into Sustained Physical Impact?

Yitian Zhou, Jingyu Zheng, Qiliang Jiang, Linkang Du, Haoming Liu, Lichao Wu, Shiyi Zhao, Mengxiang Liu, Ruilong Deng

arXiv:2608.26882v1cs.CRcs.AI

TL;DR

Autonomous LLM agents may convert network reachability to sustained physical impact on PLC-controlled processes, but prior evaluations lacked comprehensive real-PLC, closed-loop evidence. PLCBENCH provides a real-PLC HIL framework with vendor-native interaction and independent deterministic scoring, finding sustained impact in 75 of 240 episodes while exposing deployment-level barriers and observation-sensitive conversion.

  • Problem

    Existing evaluations do not comprehensively measure online autonomous agents across vendor-native PLC interaction, real PLC execution, closed-loop process integration, and independently verified physical outcomes.

  • Method

    PLCBENCH combines vendor-native commercial PLC execution, closed-loop HIL process simulation, modular agent interfaces, hidden multi-source evidence, and deterministic evaluation of six diagnostic flags.

  • Results

    75 of 240 episodes, or 31.3%, reached sustained physical impact across five LLMs, four commercial PLCs, and four workloads.

  • Takeaways & Limitations

    The measurements localize failures between PLC-interface acquisition and physical-impact conversion and identify intervention points for future defense evaluation.

  • Takeaways & Limitations

    The matrix cannot isolate a protocol-only cause because each native interface appears on one controller, so runtime, protocol, session requirements, and client support vary together.

Abstract

from arXiv · show

Industrial control systems (ICSs) rely on programmable logic controllers (PLCs) to connect networked computation with physical control. Tool-using large language model (LLM) agents represent an emerging attack threat: can an autonomous agent convert a network-reachable PLC into sustained adverse physical impact? However, existing evaluations focus on digital tasks or individual stages of PLC testing. In ICSs, evaluations that stop at software exploitation, an accepted write, or tool access may therefore mischaracterize physical risk. We present PLCBENCH, to our knowledge, the first real-PLC hardware-in-the-loop (HIL) framework for characterizing this cyber-to-physical capability and its boundaries. It combines vendor-native interaction, commercial PLC execution, closed-loop reduced-order process simulation, and independent outcome verification. A deterministic evaluator applies fixed rules to runner, communication, PLC-object, and process records to assign six hidden diagnostic flags, distinguishing usable PLC interaction, process-linked manipulation, and sustained physical impact. We instantiate PLCBENCH on four commercial PLCs crossed with four closed-loop workloads. Across five LLM families and 240 real-PLC episodes, 75 episodes (31.3%) sustain their respective physical objectives. Stagewise results show that 98 episodes stop before a valid native read, whereas 62 reach a process-linked write but do not sustain the final objective. Notably, richer process observation is associated with an increase in conditional objective attainment after a process-linked write from 44.2% to 64.0%. These measurements localize failure in configured PLC-process deployments and identify intervention points for future defense evaluation. To support reproducibility, we release the safely disclosable PLCBENCH code and a software-only reproduction pipeline through the accompanying artifact.

1 Introduction

PLCBENCH addresses the lack of comprehensive evaluation for autonomous agents that move from vendor-native PLC interaction through closed-loop process feedback to independently verified physical impact. It combines real PLCs, process workloads, deterministic evidence evaluation, and stagewise diagnostics to characterize capability and failure modes.

  • Motivation: A configured, network-accessible PLC-process deployment must be evaluated beyond open services or software exploitation because PLC compromise can affect physical operations.PLCs execute control logic while exposing network services to engineering and supervisory systems.
  • Evaluation gap: Existing evaluations rarely assess online agents across vendor-native PLC interfaces, real PLC execution, closed-loop processes, and independent physical validation.These four dimensions define the comprehensive end-to-end gap identified by PLCBENCH.
  • Framework: PLCBENCH uses a modular agent framework, closed-loop PLC-process platform, and deterministic evaluator with hidden multi-source evidence.Explicit interfaces allow models, controllers, and workloads to be recombined without changing the episode loop or evaluator.
  • Motivation and contribution: 31.3% of episodes reached sustained physical impact across five LLMs, four commercial PLCs, and four workloads.The benchmark recorded 75 successful episodes out of 240.
  • Measurement: The benchmark separates interface acquisition from physical conversion rather than treating an accepted write or tool access as end-to-end success.Six diagnostic flags localize progress and failure across the cyber-to-physical path.

2 Background

PLCs form a cyber–physical boundary by repeatedly translating sensor inputs and application data into actuator commands. PLCBENCH evaluates agents through vendor-native access and process feedback, distinguishing controller interaction from physical response.

  • PLC control loop: PLCs repeatedly execute input–compute–output scans that connect sensor values and stored control state to actuator commands.The physical process responds and supplies inputs for the next scan, forming a closed loop.
  • Native interfaces: The same logical control variable may use different native representations across PLC vendors, such as named symbols or numbered registers.PLCBENCH keeps this logical-to-native mapping hidden and records native transactions separately from process traces.
  • Agent interaction: Tool-using agents select actions, receive observations, and adapt subsequent actions during live interaction rather than static question answering.The interaction follows a repeated reason–act–observe cycle.
  • Evaluation boundary: Read-back can establish that a PLC stored a candidate value, whereas process feedback is needed to determine whether the change produced a physical effect.Separating these signals motivates distinct evaluation stages.

3 PLCBENCH Design

PLCBENCH is a real-PLC HIL design in which agents use vendor-native runtime access and bounded process observations while hidden evidence independently scores progress toward physical impact. Its interfaces, prompts, tools, diagnostic flags, and deterministic context management support long-horizon evaluation.

  • 3 PLCBENCH Design: PLCBENCH evaluates whether an agent can reconstruct usable PLC interaction and a process-relevant intervention from network reachability and a high-level objective.The active protocol, native object map, controller program, deployment binding, and attack procedure remain hidden.
  • Architecture: The framework separates model backends, PLC deployments, and process workloads through explicit interfaces and a deterministic evaluator outside the agent path.Each PLC integration supplies native access, object binding, reset, and evidence configuration.
  • Long-horizon interaction: Long-horizon interaction requires multi-step protocol discovery, delayed write-to-process attribution, and context control without losing temporal order or recent observations.These requirements arise because physical effects may emerge after the triggering action and raw traces can exceed context limits.
  • Observation: Process tools distinguish native storage from physical response by pairing read-back with current and bounded-interval process observations.The resulting observations guide hypotheses about objects and values.
  • Context management: Deterministic compaction preserves a fixed prefix and recent protected steps while progressively compressing older ordered history; the raw transcript remains intact.Compaction uses canonicalized actions, bounded monitoring samples, and explicit head–tail truncation rather than semantic ranking.
  • Diagnostic evaluation: The six hidden flags distinguish discovery, valid native reads, accepted writes, and later process-linked manipulation or impact using evidence recorded outside the model-facing path.The evaluator applies fixed rules to archived execution evidence and records the first qualifying step when assignable.

4 Implementation

The implementation separates one-time deployment admission from repeated measured episodes on stateful physical PLC lanes. Automated reset, closed-loop verification, evidence capture, deterministic scoring, recovery, and reruns make the experiments repeatable and protect against invalid testbeds or state carryover.

  • Deployment admission: Deployment admission verifies the installed program, native data path, closed loop, baseline stability, and withheld impact feasibility before measurement.Only PLC–workload bindings passing all checks enter the agent evaluation.
  • HIL execution: The HIL runtime writes process sensors to mapped PLC objects, reads controller outputs, and returns them to the process server to close the loop.Deployment bindings provide the native endpoint, object mapping, and reset profile.
  • Measured episodes: Each measured episode follows reset, run, evaluate, and recover, with native read-back and runtime-mode verification before launch and after completion.Failed verification stops the launch, while infrastructure failures are invalidated and rerun under the same configuration.
  • Scoring and recovery: Evidence is finalized before recovery, and the deterministic evaluator derives six diagnostic flags from runner, communication, PLC, and process records.This preserves the episode’s outcome before state restoration.
  • Execution scale: 240 valid physical-PLC episodes accumulated 118.0 aggregate PLC-hours, with per-PLC state-changing operations serialized and different lanes able to run concurrently.Each PLC has separate process, sandbox, evidence, and reset-lock resources.

5 Evaluation

Across 240 real-PLC episodes, sustained physical impact was limited and varied substantially by model, PLC, workload, and evaluation stage. Failures concentrated first in native-interface acquisition and later in converting process-linked manipulation into sustained objectives.

  • RQ1: End-to-End Impact, Coverage, and Repeatability: 75 of 240 episodes reach sustained physical impact, for an overall rate of 31.3%.GPT 5.5 accounts for 38 successful episodes and reaches impact in all 16 PLC–workload cells.
  • RQ1: End-to-End Impact, Coverage, and Repeatability: 42 of 80 model-specific PLC–workload cells have no successful repetition, while only 12 succeed in all three repetitions.GPT 5.5 is the only model with at least one success in every cell, but it succeeds in all three repetitions for only 9 of 16 cells.
  • RQ2: Barrier I–Native-Interface Acquisition: 98 episodes stop before valid native interaction is established, including 49 stopping before discovery and another 49 between discovery and read.Across the full chain, 191 reach discover, 142 reach read, 141 reach write, and 137 reach manipulate.
  • RQ2: Barrier I–Native-Interface Acquisition: Native-interface losses depend on the deployment: P3 exposes ADS in all 60 episodes but yields only 12 valid reads, while P4 enables discovery and read in only 12 episodes.Only GPT 5.5 and Gemini 3.5 Flash cross either deployment, and the matrix supports a deployment-level conclusion rather than a protocol-only claim.
  • RQ3: Barrier II–Physical Conversion: 62 episodes reach process-linked manipulation but not sustained impact, because physical conversion requires entering and maintaining hazardous states against coupled dynamics and protection.S2 and S3 mainly lose episodes after disruption, whereas S4 loses most before disruption; model families also stop at different stages.
  • RQ5.2: Richer Process-Observation Condition: Conditional impact after manipulation increases from 44.2% to 64.0% with richer process observation, while early discovery and read attainment remain unchanged.In the richer-observation comparison, sustained impact rises from 38 of 86 to 57 of 89 episodes.

6 Discussion

PLCBENCH shows that process-relevant PLC writes can expose sustained adverse states, but physical impact depends on process dynamics, protection logic, feedback, and available information. The results also identify stage-specific defensive intervention points and a scope limited to configured single-controller HIL systems.

  • Workload Findings: 20 episodes sustain S2's objective for 30 s after a process-relevant write, while S3 and S4 expose coupled-state gaps in recovery and lockout logic.S3 sustains both level and temperature objectives in 19 of 34 episodes; S4 sustains its asymmetric objective in 11 of 33 episodes.
  • Security Implications: Runtime write access can expose adverse process states that configured protection logic does not fully exclude, without guaranteeing sustained physical impact.Whether a process-relevant write becomes sustained impact depends on control logic, feedback, protection, process coupling, and attacker information.
  • Remaining Barriers: Physical conversion is more closely tied to process dynamics and information than native-interface acquisition, with richer telemetry increasing post-write impact.The authors characterize protocol and client dependence as friction rather than a durable security boundary, while noting telemetry's dual-use role with control authority.
  • Defensive Strategies: Defensive controls should restrict engineering-service reachability, constrain process-relevant writes, guard hazardous regions, and separate detailed monitoring data from write permission.The proposed strategies include state-aware write checks, independent safety mechanisms, and process-aware anomaly detection.
  • Scope: The conclusions apply to configured single-controller HIL systems and do not evaluate equipment damage, plant-wide effects, operator-led response, or the proposed defenses.Real PLC runtimes and vendor-native traffic are coupled to simulated plant dynamics.

7 Related Work

Prior work established interactive cyber evaluation, PLC security testing, process-aware analysis, and LLM-assisted PLC tasks. PLCBENCH addresses the remaining gap of measuring sustained closed-loop physical effects through heterogeneous vendor-native PLC interfaces.

  • Interactive Cyber Benchmarks: Executable agent benchmarks evaluate command execution, penetration testing, vulnerability exploitation, and related software or range objectives.These benchmarks establish increasingly realistic autonomous cyber capability but do not center sustained closed-loop physical effects.
  • PLC and ICS Security: PLC security research covers controller behavior, program semantics, process constraints, real-controller testing, industrial protocols, and process-aware attack payloads.This line of work addresses complementary stages of the cyber-to-physical path.
  • LLM-Assisted PLC Systems: LLM-assisted PLC systems support multi-vendor testing, context-conditioned attack synthesis, code generation, verification, and protocol-state recovery.Their evaluated units are generated programs, test cases, protocol behaviors, synthesized attacks, or vulnerabilities rather than autonomous end-to-end physical impact.

8 Conclusion

PLCBENCH provides a reproducible real-PLC HIL framework for measuring autonomous LLM-agent progression from reachable controllers to sustained physical impact. The study demonstrates validated laboratory capability with uneven reliability, while defining defenses, release boundaries, and a software reproduction path.

  • Conclusion: PLCBENCH preserves vendor-native PLC interfaces and uses independent evidence to derive six hidden diagnostic flags for repeatable cyber-to-physical measurement.A separate hardware workflow supports repeatable evaluation on stateful controllers.
  • Conclusion: Current agents complete end-to-end cyber-to-physical attacks in validated laboratory HIL settings, but reliability is uneven.Protocol complexity, undocumented object semantics, and scarce target-specific knowledge can slow exploration without preventing impact once knowledge is reconstructed online.
  • Safety Boundary: The experiments use researcher-owned PLCs and simulated processes in an isolated network to preserve real controller execution while avoiding equipment damage.No production systems, Internet path, human participants, personal data, or third-party systems are involved.
  • Ethics and Release: The study's dual-use risk motivates separating auditable scientific evidence from operational details that could make target-specific attack workflows replayable.The artifact plan withholds credentials, endpoints, runnable attack scripts, exact successful commands, and complete target-specific trajectories.
  • Ethics Considerations: The experiments use documented runtime application interfaces and do not claim a previously unknown vulnerability.The ethical analysis therefore centers on dual use and operational safety rather than human-subject risk.

B Failure Distributions and Sanitized Trajectories

Selected trajectories expose where PLCBench runs stop across PLC–workload cells, while sanitized S4 cases illustrate representative failure locations without replacing the aggregate analysis.

  • Trajectory selection: Figure 10 selects successful repetitions with earliest impact, or otherwise the repetition reaching the deepest flag first, so it favors stronger runs and does not measure repeatability.
  • Failure distributions: All 40 selected P1/P2 traces reach manipulate, 37 reach disrupt, and 28 reach impact.Impact declines across workloads from 10 for S1 to 8 for S2, 6 for S3, and 4 for S4.
  • Failure distributions: On P4, 13 of 20 selected traces stop before discovery, while seven reach disruption and six reach impact.The six impacts come from GPT 5.5 and Gemini 3.5 Flash.
  • Sanitized trajectories: Table 5 reconstructs five S4 trajectories while omitting executable protocol arguments, exact PLC objects, and model-generated write values.Sanitized episode records are planned for a separate artifact release.
  • Sanitized trajectories: The five cases span completion, process-linked failure, later physical-control failure, and service-discovery stopping points, but do not replace the aggregate 240-episode analysis.

C Scenario Dynamics, Controller Laws, and Evaluation Details

The four scenarios couple real PLC controller laws with reduced-order process dynamics and evaluate progress using independently recorded evidence and sustained physical predicates.

  • Physical objectives: Scenario objectives require physical predicates to remain continuously true for 30 s; any violating sample resets the candidate dwell interval.
  • S1: Level process: S1 models tank level with inlet, drain, passive leak, automatic hysteresis, manual control, and immutable high-high and low-low guards.Its objective is level above 85% for 30 s, near the high-high recovery boundary.
  • S2: PID temperature process: S2 models temperature with heating, fan cooling, ambient exchange, PID control, an overheat latch, and a manual-mode watchdog.The objective is temperature above 130°C for 30 s.
  • S3: Coupled thermal mixing: S3 couples liquid inventory and temperature dynamics, with safety lockout behavior and an objective requiring level above 85% and temperature above 60°C simultaneously for 30 s.
  • Controller–process coupling: The process model evolves physical state from PLC actuator outputs, while the real controller applies discrete modes, feedback, hysteresis, watchdogs, and safety logic.
  • Evaluation: The evaluator recovers six flags from archived system evidence, using canonical process traces for disruption and impact rather than agent reports or accepted writes alone.Runner, network, and PLC-object records establish interface and object-linked progress, while process traces establish physical consequence.

G Task and Environment Prompt Templates

The task templates keep the physical objectives fixed across deployments while withholding controller-specific routes, and the appendix documents prompt conditions and cross-design results.

  • G Task and Environment Prompt Templates: Across PLCs, each scenario uses the same task text with only the target endpoint rendered for the selected deployment.Target-specific protocol and object knowledge must therefore come from interaction rather than prompt engineering.
  • G.1 Canonical Scenario Tasks: The canonical scenarios cover level control, thermal processing, coupled water treatment, and a coupled quadruple-tank process with stated physical objectives.
  • G.1 Canonical Scenario Tasks: Agents are instructed to investigate with available tools, use episode observations to identify a viable influence path, and achieve the stated objective.
  • G.1 Canonical Scenario Tasks: The templates provide authorized networking, protocol-client, and Python capabilities while requiring bounded, rate-limited observations.
  • G.1 Canonical Scenario Tasks: The tasks prohibit crashing or rebooting controllers, modifying controller programs, and using anything beyond normal controller communication.The S3/S4-related constraints also prohibit high-frequency repeated writes or background loops for holding target conditions.
  • G.2 Environment Conditions: The main evaluation uses a standard environment condition, while supplementary studies alter only process-view conditions and preserve the controller task and agent framework.
  • H Complete Real-PLC Flag Tables: The appendix provides complete six-flag tables for the physical PLC matrix and matched SoftPLC reference, complementing the headline impact results.Across deployments, P1/P2 mainly expose post-manipulation conversion differences, P3 concentrates loss after discovery, and P4 at discovery.

J Richer-Observation Detailed Results

The richer-observation study compares sustained-impact outcomes under expanded process-side feedback, while appendix references document the benchmark and supplementary interface variants.

  • Observation comparison: The balanced sensitivity slice compares final and richer observation surfaces on 90 episodes using P1/P2 and S2–S4, excluding P3/P4 because of early interface attrition.
  • Observation comparison: Figure 11 and Tables 16–17 report sustained-impact rates and n/3 outcomes for each model–scenario cell under the observation conditions.
  • What the richer observation exposed: The richer condition exposes intermediate process quantities and bounded recent-window summaries without revealing PLC addresses, object mappings, diagnostic flags, or evaluator state.
  • Reference and appendix records: The 60-episode SoftPLC reference is separate from the physical-PLC benchmark, and its complete six-flag counts are listed in Table 15.
  • Interpretation and boundary: The richer study is a sensitivity analysis: additional physical state can improve control-side conversion but does not replace protocol interaction or guarantee sustained impact.
  • Reference and appendix records: Supplementary S4 variants vary one interface or observation factor at a time, lack a complete model–condition three-seed grid, and remain diagnostic rather than headline results.
  • Supplementary S4 variants: Coarse observation often loses runs during physical conversion, whereas port 1502 can move attrition earlier toward discovery and read acquisition.These patterns do not modify the headline 240 real-PLC or 60-run SoftPLC results because the subsets are small.

M Experiment Blocks and Runtime

The experiment blocks cover real-PLC, SoftPLC, and richer-observation episodes under a common action and time budget. Runtime is reported for the physical deployments, but agent steps—not wall-clock latency—are the comparable efficiency measure.

  • 240 real-PLC episodes, 60 matched SoftPLC episodes, and 90 richer-observation episodes use seeds 0, 1, and 2 under a common 100-action/3600-s contract.
  • The reported experiment blocks retain exact executed model identities and provider routes in their manifests.
  • Agent steps remain the comparable efficiency measure because model-response latency is not a controlled benchmark variable.
  • Appendix tables cover richer-observation, SoftPLC S4 interface/observation, and appendix-only S4 six-flag attainment conditions.
  • Figure 13 reports per-episode wall-clock runtime across the four physical PLC deployments.

N Reproducibility Map

The reproducibility map ties the benchmark’s empirical claims to recorded data and artifact provenance while separating audit materials from executable attack instructions. The release includes sanitized benchmark evidence and compilation materials, but omits raw successful command sequences and exact model-generated write values.

  • The appendix maps each major empirical statement to its corresponding artifact record without exposing operational attack scripts.
  • The source ZIP is self-contained for paper compilation, with the PDF rebuildable from the root directory using the listed LaTeX and BibTeX commands.
  • Rendered appendix figures are included in the arXiv source package, while regeneration scripts are planned for the separate artifact release.
  • Result tables use five paper-level model families, while artifact provenance retains the exact model and provider used in each block.
  • The planned artifact includes process/configuration sources, case bindings, prompt/tool definitions, episode results, runtime, admission evidence, and sanitized behavioral summaries.
  • The data-to-evidence audit map and prompt/tool-contract provenance are provided as dedicated appendix artifacts.
  • Raw successful command sequences and exact model-generated write values are omitted; case bindings are limited to auditing the benchmark evaluation contract in the isolated testbed.
Loading 2608.26882v1…