When to Intervene? State-Aware Sparse Manipulation in Federated Reinforcement Learning

• FRL supports collaborative policy optimization while keeping local trajectory data decentralized, but exposes global policy learning to Byzantine clients.Robust aggregation methods address malicious updates, including coordinate-wise statistics and FRL-specific filtering and ensemble architectures.Source: §1 · pp. 1–2
• Existing FRL poisoning work has emphasized malicious-signal or update construction, although intervention states can alter subsequent trajectories, state visitation, and learning signals.The paper identifies which states along a local trajectory receive intervention as a largely implicit design choice.Source: §1 · p. 2
• Changing selected trajectory states materially alters downstream degradation even when update construction and intervention budget are fixed.Controlled experiments motivate treating WHEN as distinct from HOW.Source: §1 · p. 2
• V-BSA selects sparse intervention states using local policy entropy and applies target-directed steering scaled to the nominal update and constrained by a client-local empirical envelope.Its design does not rely on peer updates or server-side defense information.Source: §1 · p. 2
• Across three discrete-action environments, V-BSA degrades performance under robust aggregators and ensemble defenses, with task- and aggregation-dependent transfer boundaries.The evaluation frames intervention timing as a distinct perspective on sequential robustness in FRL.Source: §1 · p. 2
• The paper's contributions are a WHEN–HOW attack formulation, local-only sparse manipulation, and empirical characterization across environments and aggregation settings.V-BSA is proposed under a strictly local, defense-agnostic threat model.Source: §1 · p. 2

• Model-poisoning work in federated learning steers global models by maximizing divergence from benign updates or bounding perturbations within empirical dispersion envelopes.The section contrasts static FL with FRL's sequential, Markovian sampling dynamics.Source: §2 · p. 2
• In the LunarLander pilot, entropy gating produces substantially higher terminal degradation than dense and budget-matched baselines under FedPG-BR.The figure holds update construction fixed across intervention schedules.Source: Figure 2 · p. 3
• Prior FRL attacks target local environments or rewards, malicious policy updates, and selected model components, but primarily focus on how malicious signals are constructed.The section frames trajectory-level intervention timing as a largely unaddressed design choice in these approaches.Source: §2 · p. 3
• Existing robust aggregation and ensemble defenses operate mainly on submitted updates or resulting policies, without explicitly accounting for the trajectory states that generate them.Examples include robust statistics, FedPG-BR, worker replication, and client-grouping ensembles.Source: §2 · p. 3
• Prior temporal-selectivity work addresses communication-round selection or multi-round consistency in FL and selected decision steps in single-agent RL, rather than their joint treatment in FRL.The authors state that prior FRL poisoning work does not explicitly formulate trajectory-level timing as a distinct attack dimension.Source: §2 · p. 3

3.1 FEDERATED REINFORCEMENT LEARNING

• The FRL setup has agents interact with local MDPs under a shared parameterized policy and optimize global performance.The server broadcasts parameters, agents compute local rollout updates, and the server aggregates them using FedAvg or a Byzantine-robust alternative.Source: §3.1 · p. 3

3.2 MOTIVATION

• The pilot holds malicious-update construction fixed and varies dense, random, budget-matched random, and entropy-based intervention schedules on LunarLander under FedPG-BR.The schedules use intervention rates of 100%, 15%, 5.7%, and approximately 5.7%, respectively.Source: §3.2 · p. 4
• D = 148.17 under entropy-based selection versus D = 91.99 for budget-matched random selection, with both intervening on about 5.7% of states.Entropy selection used about 17.5× fewer interventions than the dense baseline while producing D = 148.17, compared with D = 36.65 for dense intervention.Source: §3.2 · p. 4

4.1 PROBLEM SETTING

• The attacker observes its local rollout and benign update, but not peer updates or server-side aggregation mechanics, and seeks terminal degradation under bounded interventions and locally constrained updates.The formulation separates intra-round intervention timing from update construction across communication rounds.Source: §4.1 · p. 4
• V-BSA makes WHEN explicit by filtering rollouts into a sparse intervention set using policy-entropy gating, while HOW performs viability-constrained behavioral steering.The WHEN operator selects states without peer or server information.Source: §4.1 · p. 4, §4.2 · p. 4

4.2 INTERVENTION TIMING

• Under centered isotropy and binary actions, entropy and Fisher-trace susceptibility induce the same state ranking, supporting entropy as an output-only selection proxy in that setting.The idealized budgeted rule selects states with the largest Fisher trace; the online method uses entropy rather than Jacobians.Source: §4.2 · p. 5
• For multi-action policies, exact ranking equivalence does not generally hold, and a fixed entropy threshold induces an intervention rate rather than enforcing a chosen budget.Entropy captures output dispersion but not representation sensitivity.Source: §4.2 · p. 5

4.3 VIABILITY-CONSTRAINED BEHAVIORAL STEERING

• The HOW operator constructs a target-directed update from selected states, with target actions fixed per compromised subgroup and unit steering weight used in evaluations.The local update constraint uses historical benign-update EMA statistics to define a client-local empirical envelope; the envelope is a compatibility constraint, not a formal defense-bypass guarantee.Source: §4.3 · pp. 5–6
• V-BSA reuses rollout logits for its entropy gate and confines additional steering computation to selected states, with cost proportional to intervention rate and parameter dimension.The method does not require Jacobian evaluation, peer-update exchange, or additional communication rounds.Source: §4.3 · p. 6

5.1 EXPERIMENTAL SETUP

• Experiments span CartPole, Acrobot, and LunarLander with standard and robust aggregators; V-BSA uses fixed attack coefficients and evaluation reports mean return over three seeds.The setup uses 30 clients, including 10 compromised clients, and measures degradation as D = Jclean − Jattacked.Source: §5.1 · pp. 6–7

5.2 EXPERIMENTAL RESULTS

• D = 79.90 (FedAvg) to D = 218.75 (FedPG-BR) across ensemble LunarLander; evaluation curves separate over rounds, with degradation persisting under FedPG-BR filtering.Table 1 reports terminal degradation across tasks and aggregation rules; Figure 4 shows the corresponding ensemble LunarLander evaluation dynamics.Source: Table 1 · p. 7, §5.2 · p. 7, Figure 4 · p. 7
• D = −57.19 under non-ensemble FedAvg marks an empirical transfer boundary, while Acrobot degradation reaches D = 351.17 under non-ensemble FedAvg and D = 323.97 under ensemble Trimmed Mean.The results indicate that downstream degradation depends jointly on task dynamics and aggregation.Source: §5.2 · p. 7, Table 1 · p. 7
• D = 110.63 under ensemble FedPG-BR contrasts with D = 8.80 under coordinate-wise Median on CartPole; binary-target steering directions can cancel near maximum-entropy states.Trimmed Mean has an intermediate mean effect, D = 66.40, concentrated on d1 (D = 184.3) versus d0 (D = 0) and d2 (D = 14.9).Source: §5.2 · pp. 7–8
• ¯D = 218.75 for entropy gating versus ¯D = 115.92 for random selection across three dynamics, a 1.89× larger mean degradation under matched HOW components and budgets.Entropy gating used a 5.7% intervention rate on average versus 7.2% for random selection; the direction and magnitude of the effect varied across dynamics.Source: Figure 6 · p. 8, §5.2 · p. 8
• Median susceptibility rises from 0.24 to 1.62 (6.6×) under nominal policies and from 0.33 to 1.74 (5.3×) under attacked policies for entropy-selected states.The similar separation under nominal execution supports entropy as an output-only proxy for local behavioral susceptibility in evaluated LunarLander, though multi-action entropy does not generally give exact Fisher-trace rankings.Source: Figure 5 · p. 8, §5.2 · p. 9

5.3 TEMPORAL COUNTERMEASURES AND DEFENSE OUTLOOK

• The epoch auditor rejects only 31.5%–33.0% of Byzantine updates per round, while the temporal accumulator’s maximum degradation remains 82.26 on d2, above the 49.7 threshold.The accumulator also fails the clean-training gate with relative return 0.886 < 0.9; the results motivate complementing parameter-space auditing with rollout-level behavioral monitoring.Source: §5.3 · p. 9, Figure 7 · p. 9

6 CONCLUSION

• V-BSA concentrates behavioral steering on sparse, state-selected interventions, and the conclusion reports effectiveness in discrete control alongside operational boundaries in continuous settings.The stated continuous-setting boundary is variance-head saturation; cumulative temporal auditing and robust intra-group estimators also maintained substantial resilience.Source: §6 · pp. 9–10

A.2 SUSCEPTIBILITY SURROGATES

• The direction-agnostic surrogate Tr(F(s)) aggregates sensitivity across isotropic perturbation directions, whereas λmax(F(s)) describes the most susceptible single direction.The analysis uses the trace surrogate because the attacker cannot anticipate which parameter directions an unknown aggregation rule will emphasize.Source: §A.2 · p. 13

A.3 IDEALIZED BUDGETED SELECTION

• For a fixed intervention budget, the oracle susceptibility ranking selects the B states with the largest Tr(F(s_k)).This characterizes the ideal local ranking, not the online V-BSA procedure or optimal long-horizon return.Source: §A.3 · p. 13

A.4 ENTROPY–SUSCEPTIBILITY RANKING UNDER CENTERED ISOTROPY

• Under centered isotropy, binary-policy Shannon entropy and Tr(F(s)) induce identical state rankings, allowing entropy thresholding to select top-ranked states without computing the Fisher trace.The ranking equivalence requires a binary-action policy and state-independent c > 0; V-BSA uses entropy as an online gate.Source: §A.4 · p. 14

A.5 ANALYTICAL FAILURE OF EXACT ENTROPY RANKING IN MULTI-ACTION REGIMES

• For multi-action policies, Shannon entropy and Fisher-trace susceptibility can rank states in opposite orders, so entropy is a practical proxy rather than an exact susceptibility estimator.The explicit example has H(pA) > H(pB) while Tr(F(sA)) < Tr(F(sB)).Source: §A.5 · p. 15

Binary target geometry

• At π = 1/2, the two binary target-specific steering directions cancel exactly, and the entropy gate selects a neighborhood where cancellation is structurally possible.For the realized alternating assignment, the matched-state aggregate at π = 1/2 is one fifth of the fully aligned reference magnitude.Source: §A.6 · p. 16

Cancellation under alternating assignment

• The cancellation calculation assumes matched states, policy probabilities, susceptibility directions, and steering strengths, so its quantitative effect depends on downstream aggregation.Actual compromised clients encounter different states and accumulate updates over their own trajectories.Source: §A.6 · p. 16

Relation to the empirical CartPole behavior

• On tested CartPole configurations under the original modulo assignment, coordinate-wise trimmed-mean and median blocks did not meet the pre-registered damage criterion, while FedPG-BR showed seed-dependent damage (D = {164.9, 0, 0}) across three dynamics seeds.Source: §A.6 · p. 17

Target assignment

• Assigning all five compromised groups to action 0 passed its pre-registered damage criterion on 2/3 dynamics, yielding D = {139.5, 0, 192.4}; it did not amplify largest observed single-seed damage relative to the original modulo configuration, since 139.5 < 164.9.Source: §A.6 · p. 17

Local update envelope

• The client-local empirical envelope bounds each malicious-update coordinate by that client’s historical range, but guarantees neither indistinguishability from honest updates nor acceptance by an aggregation rule.The construction norm-matches the malicious update before coordinate-wise clamping.Source: §A.6 · p. 17

Scope of the analysis

• The analysis is limited to tested configurations: it is not an aggregation-independent impossibility result, nor does it imply that action spaces with at least three actions must permit successful penetration.The reported penetration pattern expands from binary to three- and four-action settings under the evaluated configurations.Source: §A.6 · p. 17

Definition of Behavioral Susceptibility

• Behavioral susceptibility L(s) is measured by the total variation distance between the nominal action distribution and the distribution induced by isotropic state perturbations.Larger values indicate that small input perturbations induce larger shifts in the action distribution.Source: B BEHAVIORAL SUSCEPTIBILITY METRIC · p. 17

Empirical Monte-Carlo Estimator

• The offline Monte-Carlo estimator evaluates susceptibility without model updates or backpropagation using 50 independent noise samples.The section distinguishes this audit from the online state-selection procedure.Source: B BEHAVIORAL SUSCEPTIBILITY METRIC · p. 17

Entropy as a Low-Overhead Computational Proxy

• V-BSA uses normalized policy entropy as a low-overhead online proxy; it exactly matches Fisher-trace rankings only for binary policies under centered isotropy.For multi-action tasks, entropy is a practical output-uncertainty proxy rather than an exact susceptibility measure; the LunarLander gate induces an average intervention rate of ρ ≈ 5.7%.Source: B BEHAVIORAL SUSCEPTIBILITY METRIC · p. 18

C.1 BENCHMARK ENVIRONMENTS AND CONTROL DYNAMICS

• The benchmarks include binary-action CartPole, three-action Acrobot, and four-action LunarLander, plus an exploratory continuous LunarLander setting.The discrete environments have state dimensions of four, six, and eight, respectively.Source: §C.1 · pp. 18–19
• All benchmark evaluations use three fixed dynamics realizations, d0, d1, and d2, matched between clean baselines and attacked runs.This permits degradation D to be reported per condition alongside summary statistics.Source: §C.1 · p. 19

C.2 SERVER-SIDE ROBUST AGGREGATION RULES

• The server-side baselines include FedAvg, coordinate-wise Trimmed Mean and Median, FedPG-BR, and ensemble subgroup voting.FedPG-BR filters gradients by alignment to a robust reference; ensemble policies are combined at test time by majority voting.Source: §C.2 · pp. 19–20

C.3 FEDERATED LEARNING AND TRAINING HYPERPARAMETERS

• The discrete and continuous benchmarks use a two-layer MLP with 64 hidden units per layer and Tanh activations; alternative policy parameterizations remain future work.The paper holds architecture constant as a controlled experimental baseline.Source: §C.3 · p. 20

C.4 COMPUTE ENVIRONMENT

• Training runs used Python, PyTorch, Linux compute nodes, and RTX 5090 GPUs or 48GB GPU instances; a 261-round run typically took 1.5–3.5 hours.The 30 simulated clients executed rollouts and policy updates sequentially within each communication round.Source: §C.4 · p. 21

D COMPLETE TRAINING DYNAMICS ACROSS AGGREGATIONS

• The complete training-dynamics section provides detailed numerical breakdowns, timing-schedule ablations, and convergence trajectories across tasks and aggregations.It underlies the terminal degradation comparisons reported in Table 1.Source: D COMPLETE TRAINING DYNAMICS ACROSS… · p. 21

Penetration via Trajectory-Level Behavioral Steering

• V-BSA intervenes on about 5.7% of states, and its updates can pass FedPG-BR filtering before altering later state-action visitation.Routine states remain unperturbed, keeping the malicious update close to nominal gradient scales and within the empirical coordinate envelope.Source: §D.1 · p. 21

Empirical Realization and Microscopic Telemetry on LunarLander

• On d2, split votes rise to 98.2% and deadlocked top-two ties to 38.0%, while mean returns fall to −66.71 and trajectory length to 185.1 steps.Nominal rollouts had top-two ties in 7.9% of decisions and averaged 420.0 steps.Source: §D.1 · p. 22
• At step 27, a tie resolves to idling during descent, and the resulting missed-thrust window precedes a crash at return −207.28.The paper contrasts this trajectory-level pathway with update-space attacks that seek large parameter deviations (Fang et al., 2025b).Source: §D.1 · p. 22
• Under ensemble FedPG-BR, LunarLander degradation across d0, d1, and d2 is 148.17, 137.48, and 370.59, respectively, with mean degradation 218.75.The d2 attacked return is −125.2, compared with a clean return of 245.4.Source: Table 4 · p. 22, §D.1 · p. 22

D.2 ANALYSIS OF TIMING ABLATIONS

• At about 5.7% intervention, entropy gating yields mean degradation 218.75, versus 27.38 for dense poisoning at 100% intervention.The comparison holds the malicious-update construction fixed.Source: §D.2 · p. 22
• On d0, budget-matched random selection yields degradation 91.99, compared with 148.17 for entropy gating; 15% uniform random selection has mean degradation 102.64.The 15% uniform schedule yields degradations of −15.89, 50.01, and 273.79 across the three dynamics.Source: §D.2 · p. 23

D.3 COMPLETE CONVERGENCE TRAJECTORIES

• Across configurations, large terminal degradation generally accompanies early, sustained separation of attacked and clean evaluation trajectories; small or negative degradation accompanies close tracking or non-persistent gaps.Figures 8–12 show 10-round rolling mean returns, with terminal points at round 261 matching the degradation statistics.Source: §D.3 · p. 23, Figure 8 · p. 24, Figure 9 · p. 24

E EXPLORATORY STUDY ON CONTINUOUS ACTION SPACES

• The exploratory study extends state-selective intervention to LunarLanderContinuous-v2, which shares LunarLander-v2 physics and reward specifications but uses continuous action bounds.Its action space is Box(2), and policies parameterize a Gaussian action distribution.Source: E EXPLORATORY STUDY ON CONTINUOUS… · p. 27

E.1 LIMITATIONS OF CONTINUOUS UNCERTAINTY GATING

• With state-independent Gaussian variance, differential entropy is constant across states and cannot discriminate trajectory-level intervention states.A state-conditioned variance head can remain elevated across the state space, driving the intervention rate toward 1.0 and collapsing sparse gating into dense poisoning.Source: §E.1 · p. 27

E.2 CONTINUOUS BEHAVIORAL STEERING

• Continuous behavioral steering perturbs predicted action means toward saturated control corners instead of using categorical cross-entropy.The paper frames this as the continuous-action steering mechanism.Source: §E.2 · p. 27
• Adaptive scaling yields degradation of 82.05 on d0 and 82.91 on d1, but −33.72 on d2; fixed scaling at λ = 0.03 has median degradation −23.12.The reported outcomes show sensitivity to transition dynamics.Source: §E.2 · p. 27, Table 6 · p. 27
• Direct output-space uncertainty proxies do not reliably produce sparse state selection, leaving principled continuous-control susceptibility proxies an open direction.The discussion names representation sensitivity and local value gradients as possible proxy bases.Source: §E.2 · p. 27