Source-linked AI summary
Energy as a Concealable State in Adversarial UAV Patrolling: Formulation, an Energy-Security Threshold, and the Limits of Self-Play
Sai Krishna Reddy Mareddy
TL;DR
The paper studies adversarial patrolling by a battery-limited UAV whose hidden energy can be inferred from trajectory and exploited during recharge. It formalizes the interaction as a zero-sum POSG, finds unstable solver behavior, and structurally identifies an energy threshold for defensibility. The remaining inference-and-deception question is whether recharge-window exploitation can be prevented by learned timing deception.
Problem
Battery-limited UAV patrolling lacks a model combining strategic attack timing with periodic recharge and hidden, behavior-correlated energy.
Method
The paper formalizes persistent energy-constrained adversarial patrolling as a zero-sum partially observable stochastic game and studies inference-capable versus memoryless attackers.
Results
0.25 to 0.09 mean thwart rate under IDQN over training, while structural security rises monotonically from 0 below threshold to about 0.7 with ample energy.
Takeaways & Limitations
Energy budget is the primary determinant of defensibility, and the model frames recharge timing as an information vulnerability that deception may defend.
Takeaways & Limitations
The model uses a discrete graph setting with one patroller and attacker, approximate uncertified equilibria, a known-target-on-alarm assumption, and incomplete deception results.
Abstract
from arXiv · showhide
We study energy-constrained adversarial patrolling on a graph, in which a battery-limited UAV defends a cluster of high-value targets against a strategic attacker who chooses when and where to strike. Unlike prior adversarial patrolling, the patroller must periodically return to a base to recharge; unlike prior energy-aware patrolling, it faces a self-interested adversary. Our central observation is that the remaining energy is a hidden state: the attacker never observes the battery directly, but observes the patroller's trajectory and can infer when a recharge excursion, and thus a vulnerability window, is imminent. We formalize the interaction as a zero-sum partially observable stochastic game and report a negative result on the solver side: neither independent deep Q-learning nor Neural Fictitious Self-Play reaches a stable equilibrium at this scale; each improves transiently and then collapses, with the co-trained thwart rate falling from about 0.25 to about 0.09 over training. Using a structural analysis independent of the learning dynamics, we show that achievable security rises monotonically with the energy budget, from zero below a threshold to about 0.7 when the budget is ample, establishing the energy budget as the primary determinant of defensibility. We set out the program the model is built to answer: whether an inference-capable attacker concentrates its successful strikes in the recharge window, and whether the defender can learn deceptive recharge timing to keep that window closed.
I. INTRODUCTION
The paper combines finite-energy UAV patrolling with strategic attack timing, treating remaining energy as hidden information that shapes recharge vulnerability. It formalizes this interaction and evaluates both solver stability and the energy conditions for defensibility.
- Motivation: Finite battery forces the UAV to interrupt patrol and return to base, creating recurring, localized exposure windows during recharge.A strategic attacker can choose attack timing and location, giving it an incentive to strike during those windows.
- Research gap: Existing adversarial patrolling assumes unlimited mobility, while energy-aware patrolling models recharge against nature rather than a self-interested opponent.The paper targets the interaction in which the recharge cycle itself becomes the object of adversarial reasoning.
- Core idea: Remaining energy is modeled as a concealable hidden state that the attacker infers from observable trajectory behavior, including movement toward base.This reframes recharge vulnerability as an information vulnerability that deceptive timing may contest.
- Contributions: The interaction is formalized as a zero-sum partially observable stochastic game with hidden energy and persistent multi-attack episodes.The formulation also identifies why a natural single-attack model is degenerate.
- Contributions: Independent deep Q-learning exhibits limit cycles and defender collapse, while Neural Fictitious Self-Play is adopted to obtain average policies that stabilize.The solver comparison motivates equilibrium-seeking learning rather than reliance on transient independent-learner snapshots.
- Contributions: Achievable security rises monotonically with the energy budget, with a threshold below which no defense is possible.This structural result identifies energy budget as the determinant of whether the game is defensible.
- Contributions: The proposed central program tests whether inference-capable attackers exploit recharge windows and whether defenders learn deceptive recharge timing.It includes memoryless-versus-recurrent attacker comparisons and belief-based deception measures.
II. RELATED WORK
The paper bridges three related strands: adversarial security games, energy-aware patrolling, and neural equilibrium computation, while positioning hidden battery inference as a distinct deception setting.
- Adversarial patrolling: Adversarial patrolling permits strategic choices over attack timing, location, and duration, but typically gives defenders unlimited mobility and omits recharge dynamics.The paper instead imposes a physical energy budget and recharge at base.
- Energy-aware patrolling: Energy-aware multi-agent patrolling models automatic recharging, but its objective is coverage in a nonadversarial environment rather than defense against a strategic attacker.Here, the attacker’s payoff is exactly the defender’s loss.
- Equilibrium computation: Independent learners can cycle in zero-sum games, whereas Neural Fictitious Self-Play approximates fictitious play with neural networks and produces average policies aimed toward equilibrium.The paper uses NFSP and exploitability as its equilibrium-solving and equilibrium-gap framework.
- Deception and hidden-state inference: The paper instantiates deception through beliefs about battery state, trajectory-based inference, and recharge timing as the deceptive action.Because the concealed quantity is a resource that must be replenished, physical energy bounds how much deception is possible.
III. PROBLEM FORMULATION
The formulation is a graph-based zero-sum POSG with a battery-limited patroller, persistent attack status, and asymmetric observations: the attacker sees behavior but not remaining energy.
- A. Environment: The environment is a finite, connected, undirected graph G = (V, E) with n nodes, a base node v0, and k valued target nodes.Shortest-path distance d(u, v) is defined on the known graph.
- B. State and Observations: The global state is st = (xt, et, ϕt), combining patroller position, remaining energy, and attack status.Energy ranges from 0 to E0, while attack status is either inactive or an active target attack with remaining duration.
- B. State and Observations: The patroller observes its position and energy, plus an alarm and attacked-target identity when an alarm is active.The observation design gives the patroller direct access to its own battery state.
- B. State and Observations: The attacker observes patroller position, attack status, and its trajectory history, but never directly observes remaining energy.This hidden, behavior-correlated energy creates the game’s defining information asymmetry.
C. Actions and Dynamics
Actions and dynamics couple movement, recharge, attack timing, and persistent episode scoring: energy limits patrol reach, while repeated attacks make recharge effects observable.
- Actions and dynamics: With positive energy, the patroller moves to a neighboring node at a cost of one energy unit; at base, it may stay and recharge up to E0.Reaching zero energy away from base strands the patroller.
- Actions and dynamics: When no attack is active, the attacker chooses a target or WAIT; an attack lasts D steps and is thwarted if the patroller visits the target during that interval.Otherwise the attack succeeds.
- Persistent episodes and payoffs: Attacks are scored over a continuing horizon Tmax, with thwarted and successful attacks producing opposite zero-sum rewards plus efficiency and stranding terms.The primary metric is the thwart rate, the fraction of attacks stopped.
- Persistent episodes and payoffs: Persistent multi-attack episodes prevent the full-information attacker from ending the game immediately at t = 0 before the patroller leaves base.This structure is essential for making energy effects observable.
E. Solution Concept
The paper seeks an approximate Nash equilibrium for a zero-sum partially observable stochastic game, using masked Double DQN agents and equilibrium-gap evaluation.
- E. Solution Concept: The target is an approximate Nash equilibrium for the zero-sum POSG, with exploitability measuring the value of a best response against a frozen policy.The game assumes known attacked targets on alarm, clustered targets distant from base, and hidden remaining energy.
- E. Solution Concept: Action masking sets invalid-action Q-values to −∞ before both policy selection and bootstrapping.This handles position-dependent patroller actions and the attacker’s WAIT-only restriction during active attacks.
- E. Solution Concept: Independent DQN is the baseline, with both agents learning simultaneously in the zero-sum game.The implementation uses Huber loss, a Polyak-averaged target network, and ϵ-greedy exploration over valid actions.
B. Neural Fictitious Self-Play
Because independent learners cycle in this zero-sum game, the paper uses Neural Fictitious Self-Play to produce average policies intended to approximate equilibrium.
- B. Neural Fictitious Self-Play: IDQN fails to converge because each player best-responds to the other’s current policy, producing cycles.NFSP is introduced to address this learning pathology.
- B. Neural Fictitious Self-Play: NFSP trains a masked Double DQN best-response network alongside an average-policy network that imitates accumulated best responses.Only best-response episodes enter the reservoir used to train the average policy.
- B. Neural Fictitious Self-Play: Under anticipatory dynamics, agents use their best response with probability η and their average policy otherwise.The reported approximately-Nash policy is the average policy.
C. Hidden-State Inference and Deceptive Defense
The model gives the attacker recurrent memory to infer hidden energy from trajectories and tests whether deceptive recharge timing emerges, while reporting unstable IDQN training evidence.
- C. Hidden-State Inference and Deceptive Defense: A recurrent encoder over the attacker’s history represents beliefs about hidden energy and anticipates recharge.A memoryless attacker provides the comparison needed to isolate the value of inference.
- C. Hidden-State Inference and Deceptive Defense: The evaluation uses a 5×5 grid with diam = 8, k = 4 targets, attack duration D = 3, recharge rate Ech = 3, and horizon Tmax = 200.NFSP uses η = 0.1, and the reported results use completed runs while some experiments remain in progress.
- C. Hidden-State Inference and Deceptive Defense: 0.25 to 0.09: the mean co-trained IDQN thwart rate falls from the first half to the second half of training.The defender progressively collapses as the attacker locks in a robust exploit, so mid-training snapshots are not meaningful policies.
B. Energy budget determines achievable security
A structural sweep shows that achievable security increases monotonically with the energy budget, while IDQN’s cycling obscures this energy dependence.
- B. Energy budget determines achievable security: 0 to about 0.7: the structural thwart-rate ceiling rises from zero at or below the base-to-cluster distance to about 0.7 with ample energy.The ceiling uses a strong on-station heuristic defender against a worst-case attacker.
- B. Energy budget determines achievable security: A defensible regime exists only above an energy threshold.The structural analysis is independent of the learning pathology.
- B. Energy budget determines achievable security: 0.25–0.30: IDQN’s time-averaged thwart rate remains nearly flat across energy budgets and does not track the structural ceiling.Cycling washes out the energy dependence, so energy–security relationships require a converged solver rather than IDQN.
D. NFSP stabilizes where IDQN collapses (in progress)
NFSP shows preliminary stabilization at ample energy, while the recharge-window, inference, and deception tests remain unfinished. These results are intended to determine whether attackers exploit inferred battery state and whether defenders can obscure recharge timing.
- NFSP results: 0.87 average-policy thwart rate stabilizes after about 2500 episodes on a representative NFSP seed at E0 = 20.A second seed remains noisier, so the authors treat this as preliminary evidence pending a five-seed run and exploitability certification.
- Planned tests: The ongoing experiments measure successful-strike concentration in recharge phases, recurrent-attacker energy inference, and defensive deception.These tests compare a memoryless attacker with a recurrent GRU attacker and examine whether learned recharge timing reduces vulnerability.
- Interpretation: The structural question of defensibility is separated from the learning question: energy thresholds determine whether defense is possible, while solver convergence determines whether learners can reach it.IDQN cycles and collapses, whereas NFSP shows early signs of stabilizing; the central information-theoretic tests require a converged solver.
A. Limitations
The study is bounded by a discrete graph model with one patroller and attacker, approximate equilibria, and a known-target-on-alarm assumption. Several extensions and the central deception and inference results remain incomplete.
- Scope: The model uses discrete time and graph abstraction with a single patroller and single attacker.The paper identifies multiple patrollers, continuous space, and real facility or road networks as natural extensions.
- Assumptions and guarantees: Computed equilibria are approximate rather than certified, and the base model assumes the target is known upon alarm.The binary-alarm case is treated only as an ablation, while certified equilibria are left as future work.
- Experimental status: The inference and deception experiments are still in progress rather than complete.Consequently, the paper has not yet settled whether attackers concentrate strikes in recharge windows or defenders learn deceptive timing.