PaperlayerView exact revision on arXiv

Source-linked AI summary

Inverting Self-Triggered Control: Adversarial Reinforcement Learning for Sparse Denial-of-Service Attacks

Adam Haroon, Erick J. Rodríguez-Seda, Tristan Schuler, Cody Fleming

arXiv:2609.12016v1cs.LGcs.SYeess.SY

TL;DR

The paper asks how a resource-efficient DoS adversary can destabilize a self-triggered controller, a setting not addressed by prior scheduling work. It constructs a self-triggered adversary and proves a jam-count lower bound for STC under a Lyapunov contract. Across three plants, the learned adversary crashes every evaluated defender, with robustness results also reported for Quadrotor2D sensing constraints.

  • Problem

    Prior DoS scheduling work uses fixed, grid-scheduled, or energy-budgeted attacks, leaving the self-triggered-adversary/self-triggered-defender setting unaddressed.

    Source: §I · p. 1

  • Method

    The paper inverts RL-STC’s self-triggered structure and proves a plant-property lower bound on jam count for an immediate hold-last MAC adversary against an STC satisfying a Lyapunov contract.

    Source: §I · p. 1

  • Results

    The learned adversary crashes all 12 plant-defender cells at 100%, while baselines miss in specified cells; it strictly beats the best baseline on 4/4 CartPole cells and 2/4 Quadrotor2D cells, and matches greedy on Pendulum.

    Source: H. Discussion · p. 6

  • Takeaways & Limitations

    Under the hold-last MAC model, jam time, attack energy, and jam fraction are equivalent affine measures of jam count, connecting the wa sweep to count-budget DoS scheduling.

    Source: E. Two-player game and metrics · p. 3

    The count bound is stated on the linearization while the empirical table measures the nonlinear plant, and the evaluation uses one DQN architecture on three plants.

    Source: H. Discussion · p. 8

Abstract

from arXiv · show

Self-triggered reinforcement learning control (RL-STC) learns the sparsest control schedule that preserves Lyapunov-decreasing stability under a Run-Time Assurance (RTA) override. We invert this: an adversarial RL agent learns the sparsest jamming or Denial-of-Service (DoS) schedule that destabilizes the closed loop, with a Lyapunov-increase admissibility predicate mirroring the defender's safety certificate. We prove a plant-property lower bound on the minimum jam count required for an immediate hold-last medium-access-control adversary to force a crash against a self-triggered controller (STC) satisfying a Lyapunov contract, and recover a certificate-level analog of the consecutive-grouping optimality of prior count-budget DoS scheduling as a corollary. This extends the DoS-scheduling count-budget analysis from periodic and linear-time-invariant to STC controllers. Empirically, we train against four fixed defenders per plant (one Linear Quadratic Regulator (LQR) and three RL-STC) on Pendulum, CartPole, and Quadrotor2D. The learned adversary is the only adversary that crashes every defender on every plant at $100\%$: greedy misses Quadrotor2D LQR on $42\%$ of episodes and periodic misses Pendulum LQR on $97\%$. On jam-time-per-failure it beats baselines by up to $2.8\times$, and shows its widest absolute margin on Quadrotor2D LQR. Robustness ablations show that Gaussian observation noise exceeding the initial-state magnitude and position-only observation both preserve $100\%$ failure rate and keep the learned adversary strictly ahead of both baselines on jam-time-per-failure.

I. INTRODUCTION

The paper inverts self-triggered reinforcement learning control to study resource-efficient DoS attacks, addressing the absence of a self-triggered adversary against a self-triggered defender. It combines an adversarial construction and a jam-count lower bound with evaluations across three plants.

I. INTRODUCTION

  • A DoS adversary destabilizes a networked cyber-physical plant by withholding control updates without physically touching it.The paper frames communication as a shared resource: defenders spend sensing, computation, and bandwidth, while adversaries spend presence and energy.Source: §I · p. 1
  • Prior DoS scheduling studies cover fixed, grid-scheduled, or energy-budgeted adversaries, but not a self-triggered adversary against a self-triggered defender.Related work includes count-budget scheduling,, time-varying interference, and frequency- and duration-limited DoS –.Source: §I · p. 1
  • The adversarial construction inverts RL-STC’s self-triggered structure, using a Lyapunov-increase predicate analogous to the defender’s RTA certificate.The learned DQN is unshielded; the predicate instead defines the predicate-greedy baseline.Source: §I · p. 1, Figure 1 · p. 2
  • A plant-property lower bound gives the minimum jam count for an immediate hold-last MAC adversary to force a crash against an STC satisfying a Lyapunov contract.A certificate-level analog of consecutive-grouping optimality follows as a corollary, extending count-budget analysis from periodic and LTI controllers to STC.Source: §I · p. 1
  • 100%: the learned adversary crashes all defenders on all plants, while predicate-greedy misses Quadrotor2D LQR on 42% of episodes and periodic misses Pendulum LQR on 97%.On jam-time-per-failure, it strictly beats the best baseline on all four CartPole defenders and Quadrotor2D LQR, with a 2.6× margin on Quadrotor2D LQR.Source: §I · p. 1

II. PROBLEM FORMULATION

  • The formulation adds a self-triggered jammer to the plant, defender, and clock of the prior RL-STC framework.Both agents use the same discretization grid, and a jam latches the last applied input at the actuator.Source: §II · p. 2, Figure 1 · p. 2

A. Nonlinear plant

The setup uses a sampled nonlinear plant with a stabilizing bounded control law and a defender protected by an RTA layer. The adversary commits jam durations on the same grid, withholding updates while the actuator holds its last input.

A. Nonlinear plant

  • The plant is a continuously differentiable nonlinear system with state x, input u, and a bounded sampled control law assumed capable of stabilizing an equilibrium.The defender, adversary, and clock all act through the sampled input.Source: A. Nonlinear plant · p. 2

B. Defender (RL-STC baseline)

  • The defender selects a sampling interval from a discrete grid, holds its control with a zero-order hold, and uses an RTA layer to override unsafe predictions with a clipped LQR backup.The backup is certified Lyapunov-decreasing when unsaturated; the defender reward includes Lyapunov decrease, communication cost, RTA activation, and crash terms.Source: B. Defender (RL-STC baseline) · p. 2

C. Self-triggered adversary

  • The adversary commits grid-aligned jam durations; during a jam, the actuator latches the most recently delivered control input.A zero-duration action passes the fresh defender update, and the learned reward mirrors the defender’s while penalizing jam activity; the admissibility predicate is not enforced on the learned policy.Source: C. Self-triggered adversary · p. 2

D. Lyapunov-increase admissibility predicate

The adversary-side admissibility predicate uses a one-tick prediction under the held input to test for Lyapunov increase. With the defender fixed, the game becomes a single-player adversary MDP, evaluated using jam count, jam fraction, and jam-time-per-failure.

D. Lyapunov-increase admissibility predicate

  • A jam passes the adversary-side predicate only when a one-tick probe under the held input predicts a Lyapunov increase.The probe integrates the nonlinear plant for one minimum tick; with θatt fixed at zero, any predicted increase passes.Source: D. Lyapunov-increase admissibility… · p. 2

E. Two-player game and metrics

  • Fixing the defender reduces the interaction to a single-player MDP, and evaluation reports the defender’s mean inter-sample interval, jam fraction, and jam-time-per-failure.Under the hold-last MAC model and constant jammer power, jam time, jam fraction, and energy are affine in jam count, making jtf attack energy per successful destabilization.Source: E. Two-player game and metrics · pp. 2–3

III. MINIMUM-JAM-COUNT BOUND FOR STC

The paper extends count-budget DoS analysis to self-triggered controllers by deriving a plant-property lower bound for a hold-last attacker under a Lyapunov contract. Its corollaries give a certificate-level consecutive-jam result and describe how the bound varies with plant growth.

A. Lower bound and corollaries

  • G1 is the plant-dependent P-norm operator growth of the linearized dynamics over τmin and can capture transient amplification even when the spectral radius is one.Source: A. Lower bound and corollaries · p. 3
  • Under an exponential-decay Lyapunov contract, the minimum contiguous jam count is bounded below by ceil(log(Vcrit/V0)/(2 log G1)) when G1 > 1.The theorem considers a hold-last attacker that starts at t = 0 with V0 < Vcrit; if G1 ≤ 1, no finite jam count crashes the plant.Source: A. Lower bound and corollaries · pp. 3–4, §III · p. 3
  • The bound certifies the linearized model, while Table I measures full nonlinear dynamics; the observed margin does not isolate higher-order effects from affine corrections.A non-equilibrium held input adds a bounded affine translation absorbed into the empirical margin.Source: A. Lower bound and corollaries · p. 3
  • The certificate-level terminal Lyapunov bound is maximized by one contiguous jam block among schedules with the same jam count.This extends consecutive-grouping optimality from periodic and LTI controllers to STC, using the STC contract and P-norm growth factor rather than an LTI closed-loop transition matrix.Source: A. Lower bound and corollaries · pp. 3–4, §III · p. 3
  • For fixed V0 and Vcrit, the pre-ceiling bound log(Vcrit/V0)/(2 log G1) decreases strictly as G1 increases, so N* is non-increasing.Across plants, variation in Vcrit/V0 means N* need not be ordered by G1 alone.Source: A. Lower bound and corollaries · p. 4

B. Empirical validation

The empirical comparison places the certified count floor well below the measured count needed for a high crash rate. The gap reflects a worst-case bound compared with a typical outcome under one schedule.

B. Empirical validation

  • The certified count floor is at least 3.3× below the measured threshold throughout the reported comparisons.The two counts answer different questions: the bound applies across directions and schedules, while the measured threshold is for one contiguous block on reset-distribution episodes.Source: B. Empirical validation · p. 4
  • The measured threshold is the count at which one contiguous block crashes 95% of reset-distribution episodes, unlike the bound's worst case over schedules and directions.Source: B. Empirical validation · p. 4
  • The bound is a plant property that can be computed before running an experiment.Source: B. Empirical validation · p. 4

C. Learned adversary vs. the immediate-jam bound

The learned adversary's advantage over immediate jamming is concentrated on Quadrotor2D, not on Pendulum or CartPole. Its Quadrotor2D result comes from a multi-step schedule rather than a better starting state.

C. Learned adversary vs. the immediate-jam bound

  • The learned adversary fires on the first tick in every Pendulum and CartPole episode and at median tick 1 on Quadrotor2D, where its firing-state Lyapunov value is 0.87 V0.Thus its Quadrotor2D advantage is not from obtaining a better starting state.Source: C. Learned adversary vs. the… · p. 4
  • On Quadrotor2D, the learned adversary averages 18.9 jam ticks versus 58 for greedy, which has 58% reliability; learned waits one tick, uses 1.79 blocks, and crashes every episode.The learned adversary's jam time is 0.755 s, compared with 2.7× as much for greedy.Source: C. Learned adversary vs. the… · p. 4
  • On Pendulum and CartPole, the learned adversary's 95th-percentile counts of 34 and 35 ticks are no cheaper than immediate jamming from t = 0.It reproduces predicate-greedy's schedule to within about one tick, adding reliability rather than efficiency on those plants.Source: C. Learned adversary vs. the… · p. 4

IV. EXPERIMENTS

The experiments train adversaries against fixed LQR and RL-STC defenders on three plants. They evaluate attack cost primarily by jam-time-per-failure and use a strict comparison to both baselines to identify strategic behavior.

A. Setup

  • The evaluation covers Pendulum, CartPole, and Quadrotor2D, with four fixed defenders per plant: periodic LQR and three RL-STC checkpoints.The study trains 84 adversaries per plant across seven jam-activity penalties and three seeds.Source: A. Setup · p. 4
  • Jam-time-per-failure is the mean jammed seconds over crashing episodes and is treated as attack energy per successful destabilization.Source: A. Setup · p. 4
  • The study calls an adversary strategic on a defender when its mean jam-time-per-failure is strictly below both baselines and its cross-seed standard deviation is small relative to the smaller gap.Jam fraction is secondary because it can conflate structural sparsity with strategic timing.Source: A. Setup · pp. 4–5

B. Adversary training and reward calibration

The adversary uses a DQN with a calibrated crash bonus, and is evaluated against nominal, periodic, and predicate-greedy baselines. The predicate-greedy baseline waits for a Lyapunov-increase predicate before continuously jamming.

B. Adversary training and reward calibration

  • The DQN adversary uses actions from {0, . . . , Nτ = 8}, learning rate 10−3, γ = 0.99, and an episode-bounded training budget of 25,000.The best-per-step checkpoint is typically reached within ∼10% of the budget.Source: B. Adversary training and reward… · p. 5
  • Setting resc = 100 avoids terminal-dominated Q-targets while keeping the crash bonus above worst-case non-terminal accumulation.Mirroring the defender’s −1000 penalty would overwhelm the differential jam cost between full-jam and half-jam schedules.Source: B. Adversary training and reward… · p. 5

C. Baselines

  • The comparison includes a no-jam baseline measuring nominal defender behavior in isolation.The learned adversary and non-learned baselines are compared on every defender and wa combination, with 100 evaluation episodes per combination.Source: C. Baselines · p. 5
  • The periodic baseline jams in 4τmin bursts every other decision, while predicate-greedy idles until its predicate fires and then jams continuously.Predicate-greedy uses the same observed state as the learned adversary, making their comparison a same-information test of strategic timing.Source: C. Baselines · p. 5

D. Pendulum

On Pendulum, the learned adversary crashes every defender but does not beat predicate-greedy on jam-time-per-failure. On CartPole, two defenders meet the strategic-adversary criterion, with the LQR result making learned the only reliable crasher.

D. Pendulum

  • On Pendulum, learned crashes all four defenders at 100%, but matches greedy on jtf and beats periodic by 1.2–2.3×.Predicate-greedy misses LQR on 13% of episodes and periodic misses on 97%; the learned policy’s headline advantage is reliability, not strict jtf dominance.Source: D. Pendulum · p. 5
  • Pendulum’s continuous-jam optimum follows because MSIctrl is irrelevant during held-actuator jam windows, so learned and greedy settle near the lower bound.The per-defender MSIctrl values are 0.050 s for LQR and 0.276, 0.398, and 0.382 s for the RL-STC defenders.Source: D. Pendulum · p. 5

E. CartPole

  • Two of four CartPole defenders admit a strategic learned adversary; against LQR, learned crashes every episode and is 1.5% faster than greedy.Periodic never crashes CartPole LQR, while predicate-greedy reaches 95% failure.Source: E. CartPole · pp. 5–6
  • On CartPole wc = 1.0, learned jtf is 0.545 ± 0.188 s; at wc = 6.0 it is 0.320±0.000 s, beating greedy by 2.7×.At wc = 14.0, learned jtf is 0.396 ± 0.014 s, 35% faster than periodic and 2.3× faster than greedy.Source: E. CartPole · p. 5
  • At CartPole wc = 14.0, learned idles ∼0.24 s while θ drifts, then crashes at 0.56 s with one sustained burst.In the compared episode, periodic crashes at 1.32 s and predicate-greedy at 0.64 s.Source: E. CartPole · p. 6
  • Across the CartPole wa sweep, wc = 14.0 and wc = 1.0 remain below the greedy line for every wa ≥0.1.For wc = 6.0, learned enters the below-greedy regime from wa = 2.Source: E. CartPole · p. 6

F. Quadrotor2D

On Quadrotor2D, the learned adversary crashes every defender and achieves its widest learned-versus-baseline gap against LQR. Its advantage persists under noisy and position-only observations.

F. Quadrotor2D

  • Against Quadrotor2D LQR, learned jtf is 0.755±0.099 s, 2.7× faster than greedy, and its 100% failure rate exceeds greedy’s 58%.This is the paper’s largest learned-versus-baseline gap.Source: F. Quadrotor2D · p. 6
  • At wc = 14.0, learned jtf is 0.654 ± 0.195 s, 29% faster than periodic and 2.8× faster than greedy.Its spread is 73% of the margin to periodic, making it the weakest of the four strategic cells.Source: F. Quadrotor2D · p. 6

FAILURE RATE IS 100% IN EVERY ROW; BOLD MARKS THE LEARNED

G. Deploy-realism ablation (Quadrotor2D)

H. Discussion

The discussion shows that attack reliability and efficiency vary with plant and defender structure, while state-uniform reward shaping and the evaluation scope bound the findings. The conclusion presents a defender-agnostic adversary and identifies co-training and predicate masking as follow-ons.

H. Discussion

  • 100%: the learned adversary crashes all 12 plant–defender cells, strictly beats the best baseline on 4/4 CartPole and 2/4 Quadrotor2D cells (largest gap 2.6× on LQR), and matches greedy on Pendulum.Source: H. Discussion · p. 6
  • At wa = 10, CartPole’s three RL-STC defenders reach 68%, 80%, and 52% jam fractions below the 90% gate; Quadrotor2D sparsifies three RL-STC defenders across most of the sweep.Source: Figure 2 · p. 7
  • Prior consecutive-grouping results cover count-budget packet-dropping under periodic or LTI assumptions [2],; RL-STC timing exposes strategic windows even on single-scalar CartPole.Source: H. Discussion · p. 7
  • CartPole’s wc = 14.0 adversary stays below greedy across wa ∈[0.1, 10], while wc = 6.0 does so only from wa = 2 despite similar MSIctrl, indicating policy structure matters beyond nominal budget.Source: H. Discussion · pp. 7–8
  • State-uniform reward shaping biases two Quadrotor2D cells toward near-equilibrium jamming: they retain 100% crashes but have 1.6× jtf and 5.5× cross-seed spread versus that plant’s strategic cells.Short-MSIctrl RL-STC on Quadrotor2D also has wider jtf variance.Source: H. Discussion · pp. 7–8

V. CONCLUSION AND FUTURE WORK

  • The conclusion presents the adversary as a defender-agnostic black-box framework and proposes co-training plus hard-masking inadmissible jams, which would forbid equilibrium-time jams needed to drive the system out.Source: §V · p. 8
Loading 2609.12016v1…