Brittle by Design? Multi-Scenario Reinforcement Learning for Robust Mission Coordination in Systems of Systems

1 Introduction

• Static SoS architectures are costly to modify and sensitive to operational changes, motivating a mission-centered view of how constituent systems are coordinated.ME links operational needs and capabilities to desired mission outcomes, including adaptive task allocation as conditions unfold.Source: §1 · p. 2
• Mission coordination faces unpredictable threats, nonlinear asset interactions, and combinatorially growing task assignments that can make exact methods too slow for operational timelines.The paper frames digital simulation and fast decision methods as complementary capabilities for developing and deploying coordination policies.Source: §1 · p. 2
• The paper integrates high-fidelity digital mission models with RL, using simulation outcomes to train coordinators to map mission states to tactical actions.The case study uses an aerial firefighting simulation and trains under either single-scenario or multi-scenario exposure.Source: §1 · pp. 2–3
• The paper distinguishes coordinator robustness across conditions from the broader objective of SoS resilience to disruptions during a mission.It uses brittleness for performance loss under conditions absent from training and robustness for retention of that performance.Source: §1 · p. 3
• Trained coordinators significantly outperform random tactics and a fixed heuristic in the scenarios on which they were trained.The paper evaluates both transfer from single-scenario training and concurrent training across scenarios.Source: §1 · pp. 3–4
• A coordinator trained on the more forgiving scenario can fall to random-tactics performance in a more demanding scenario, while the demanding-scenario coordinator retains most of its advantage after transfer.The comparison holds the fleet, its capabilities, and available tactics constant while changing the mission scenario.Source: §1 · pp. 3–4
• Within the same training budget, multi-scenario training matches scenario-specific coordinators in their own scenarios and significantly outperforms other policies across both.The proposed analytical structure is mission-agnostic, but state, action, and reward definitions must be re-derived for each mission.Source: §1 · p. 4

2.1 SoS and Mission Engineering

• SoS constituents retain operational and managerial independence while being orchestrated toward shared purposes, and changing missions make static architectures brittle.Mission Engineering connects engineering activities to operational needs and desired mission outcomes.Source: §2.1 · pp. 4–5
• ME identifies adaptive task allocation and reconfiguration as needs, while coordination must balance efficiency with constituent autonomy and cope with changing conditions.Because deployed architectures are costly to modify, adaptation often falls to coordination among constituent systems.Source: §2.1 · p. 5

2.2 SoSE and ME Using Analytical Methods

• Prior design-time analytical methods provide tools for SoS decisions but generally treat operations statically, leaving operational coordination unaddressed.The reviewed work spans capability-risk trade-offs, dependency analysis, and multi-stakeholder evolution.Source: §2.2 · pp. 5–6

2.3 Learning-Based Methodologies for SoSE

• Learning-based governance can steer autonomous constituents toward system-level objectives, but the reviewed studies use abstract environments rather than mission outcomes.Mission-related RL studies use realistic simulations but are not framed in ME terms or examine how training scenarios shape post-deployment performance.Source: §2.3 · p. 7

2.4 Synthesis and the Research Gap

• The literature gap is operational RL coordination in ME using high-fidelity mission simulation, including explicit study of brittleness across scenarios.The synthesis contrasts static design-time analysis, abstract learning-based governance, and mission planners that do not study training-scenario effects.Source: §2.4 · p. 8

3 Methodology

• The five-stage approach builds a digital mission model, simulates interactions, formulates coordination as an MDP, trains coordinators under alternative scenario curricula, and evaluates them in Palisades and Pyrenees.The approach targets real-time tactical decisions using high-fidelity mission models and RL.Source: §3 · p. 8

3.1.2 Firefighting Fleet and Infrastructure

• The model uses a fixed fleet of five seaplanes and five eVTOLs, whose payload, endurance, and access to water sources differ.The scenarios cover 10×10 km areas with varying geography; fires ignite uniformly within a 4 km² region and are detected after scenario-specific response windows.Source: §3.1.2 · p. 10, §3.1.3 · pp. 10–11

3.1.3 Mission Scenarios

• Weather varies across time, with wind speed and direction identified as the main uncertainty in fire propagation.Temperature and humidity follow a more stable daily profile, while fire spread also depends on fuel type and terrain.Source: §3.1.3 · p. 11, §3.2.1 · p. 12

3.2 The Agent Based Simulation

• The agent-based simulation links geography, weather, wildfire, and mission assets, with water releases changing fire spread and outcomes emerging from their interactions.Aircraft operate independently, releasing suppressant, reloading, and repeating sorties according to assigned tactics and class-specific preferences.Source: §3.2 · p. 12, §3.2.2 · p. 12

3.2.4 The Firefighting Fleet as an SoS

• The fleet is modeled as a directed SoS: the coordinator assigns tactics centrally, while aircraft retain independent target selection and class-specific preferences.The study fixes the fleet across scenarios and defers fleet selection to future work.Source: §3.2.4 · p. 13

3.3 Markov Decision Process Formulation and RL Implementation

• The coordinator’s state includes scenario, mission progression, weather, fire conditions, and aircraft positions; its actions assign each aircraft a tactic.Tactics combine point-of-interest selection, tracking, and suppression mode; constraints on indirect tactics reduce the action set to 17 valid tactics.Source: §3.3.1 · p. 14, §3.3.2 · pp. 14–15

3.3.3 The Reward Structure Based on a Measure of Effectiveness

• The reward is the change in a scenario-bounded measure of effectiveness that aggregates burnt area, cost, emissions, casualties, and whether the fire escapes the region.The measure ranges from −1 to 1, with values near 1 indicating minimal losses and negative values denoting escape beyond the region boundary.Source: §3.3.3 · p. 15

3.3.4 Proximal Policy Optimization

• PPO trains an actor-critic policy for the discrete tactic space, constraining updates through a clipped objective to keep the policy near its previous version.The critic estimates state value and supplies the advantage estimate used by the objective.Source: §3.3.4 · pp. 15–16

3.4.1 Simulation Workflow

• Every 10 minutes, the coordinator observes the mission state, assigns aircraft tactics, and receives the resulting change in MoE as reward until a termination condition is met.An episode ends when the fire is contained, reaches the 24-hour limit, or escapes the region boundary.Source: §3.4.1 · p. 16

3.4.2 Training Setup

• All coordinators use the same PPO configuration and 150,000-episode budget, with 128 parallel workers and separate actor and critic networks.Each network has three hidden layers; worker rollouts are shuffled into mini-batches for five optimization epochs per policy update.Source: §3.4.2 · p. 16

3.4.3 Baselines and Evaluation Protocol

• Evaluation compares fixed trained policies on the same 1,000 episodes per scenario against random tactics and a fixed role-based heuristic assignment.Paired tests use per-episode MoE differences with Holm correction.Source: §3.4.3 · p. 17

4 Findings

• The findings section compares trained coordinators with baselines, examines cross-scenario transfer, evaluates concurrent training, and includes sensitivity analyses of ignition location, weather, and curricula.The analyses are organized around three research questions.Source: §4 · p. 19

4.1 RQ1: Integrating RL with a High-Fidelity Mission Model

• Both single-scenario coordinators learned beyond random tactics, with gains of 0.103 MoE in Palisades and 0.112 in Pyrenees.Palisades reached approximately 0.97 while its baseline remained 0.867; Pyrenees finished at −0.008 versus −0.120, and negative MoE reflects an undersized fleet in that scenario.Source: Figure 4 · p. 19, §4.1 · p. 20

4.2 RQ2: Brittleness of Single-Scenario Coordinators

• In Pyrenees, PA performed no better than Random Tactics, whereas PY retained most of its advantage when transferred to Palisades.In Palisades, PA and PY achieved mean MoE values of 0.972 and 0.943, respectively; in Pyrenees, PA’s −0.113 was statistically indistinguishable from the −0.122 random baseline.Source: §4.2 · pp. 20–21, Figure 5 · p. 21

4.3 RQ3: Multi-Scenario versus Single-Scenario Training

• The pooled comparison combines 1,000 evaluation episodes from each scenario, giving the scenarios equal weight to assess a policy intended for deployment when the scenario is unknown.MS is trained concurrently on Palisades and Pyrenees under the same 150,000-episode budget as the single-scenario coordinators.Source: §4.3 · pp. 21–22

4.3.1 Training

• MS converged to nearly PA’s Palisades MoE, 0.967 versus 0.970, and finished above PY in Pyrenees, 0.010 versus −0.008.MS used half its workers for each scenario during training.Source: §4.3.1 · pp. 22–23, Figure 6 · p. 23, Figure 7 · p. 23

4.3.2 Evaluation

• 0.495 mean MoE: MS significantly outperformed every other policy when pooled across both scenarios, while matching each scenario’s native coordinator within that scenario.Overall regret was zero for MS, compared with 0.0154 for PY, 0.0663 for PA, 0.1131 for Heuristics, and 0.1261 for Random Tactics.Source: Figure 8 · p. 24, §4.3.2 · pp. 24–25, Figure 9 · p. 26
• MS achieved the highest mean MoE in both scenarios and pooled, but its within-scenario differences from the native coordinator were not statistically significant.The figure compares MoE distributions for PA, PY, MS, Heuristics, and Random Tactics across the two scenarios and pooled episodes.Source: Figure 8 · p. 24, §4.3.2 · pp. 24–25, Table 4 · p. 25

4.3.3 Multi Scenario Training Leads to Lowest Overall Regret

• MS had zero regret in Palisades, Pyrenees, and pooled evaluation; its pooled regret advantage was strongest against the baselines.The analysis defines regret as the shortfall from the best approach’s mean outcome in a scenario; the study does not test whether multi-scenario training improves performance in scenarios absent from training.Source: §4.3.3 · pp. 25–26, Figure 9 · p. 26

4.4.1 Sensitivity to the Ignition Location During Training

• 0.927 under fixed ignition drops to −0.069 under varying ignition for PYfix; PY (0.017) and MS (0.018) significantly outperform it under varying ignition.The authors interpret this as evidence that training on one ignition point can tailor tactics to that fire rather than the scenario as a whole.Source: §4.4.1 · pp. 27–28, Figure 10 · p. 27

4.4.2 Sensitivity to Adverse Weather Conditions

• Adverse weather lowers every policy’s mean MoE, mild weather raises it, and MS matches or exceeds every policy across the tested weather conditions.The coordinators’ relative advantage vanishes in mild Palisades, shrinks in adverse Pyrenees, and is largest in adverse Palisades; harsh unseen weather sharply degrades absolute performance.Source: §4.4.2 · pp. 28–29, Figure 11 · p. 29

4.4.3 Sensitivity to the Multi-Scenario Training Curriculum

• Concurrent multi-scenario training yields the best mean MoE in both scenarios and pooled, while the even sequential split performs worst overall.The comparison indicates that scenario order matters in addition to budget share; the authors suggest sequential training may partially forget the first scenario and conclude scheduling and allocation must be appropriate.Source: §4.4.3 · p. 29, Table 5 · p. 30

4.4.4 Sensitivity to an Unseen Scenario

• In unseen Salamis, MS reaches the highest mean MoE (0.824), ahead of PY (0.785), while PA (0.381) falls below Heuristics (0.588) and Random Tactics (0.420).PY, trained on the hardest scenario, retains an advantage over the baselines in easier scenarios; PA, trained on the easiest, falls to or below baseline performance in harder scenarios.Source: §4.4.4 · pp. 30–31, Figure 13 · p. 31

Efficacy of the approach

• In Pyrenees, successful coordinators contained two to three times more fires than random and heuristic baselines.In Palisades, where most missions succeeded under every policy, their advantage mainly came from reducing catastrophic failures.Source: §5 · p. 32

Single-scenario training

• The Palisades-trained coordinator performed no better than Random Tactics in Pyrenees and Salamis, while the Pyrenees-trained coordinator retained most of its advantage in both.The Palisades-trained policy offered no benefit in the new environments despite 150,000 training episodes.Source: §5 · p. 32

Multi-scenario training

• Within the same simulation budget, the multi-scenario coordinator matched the best single-scenario coordinator in each scenario and outperformed every other policy in pooled evaluation.It also attained the highest mean MoE in Salamis, a scenario unseen during training.Source: §5 · p. 32

Mission uncertainty

• None of the trained coordinators eliminated mission failures; some fires exceeded what the fixed fleet could suppress under challenging ignition-location and weather combinations.The authors stress rigorous validation and ongoing monitoring because conclusions from limited data can overstate a method’s value.Source: §5 · pp. 32–33

Training variability

• Training on fixed ignition locations or nominal weather left coordinators brittle when those conditions varied, while scenario difficulty also shaped transfer performance.Performance under training conditions alone was not a reliable indicator of operational performance; evaluation beyond training conditions was necessary to uncover brittleness.Source: §5 · p. 33

From robustness to mission resilience

• The study assesses robustness across scenarios and conditions, not resilience to in-mission disruptions such as asset failures, airport closures, or communication delays.Assessing resilience would require testing disruptions during missions and measuring outcomes, degradation, and recovery.Source: §5 · pp. 33–34

Study limitations

• The proof of concept assumes perfect information and complete asset adherence to orders, uses discretized tactics, and fixes the fleet rather than selecting assets.The authors also note that their single static heuristic may underestimate the potential value of heuristics.Source: §5 · pp. 31, 34

6.1 Conclusion

• Repeated simulations train coordinators to map SoS and environmental states to tactical actions without explicitly modeling scenario uncertainty.The study frames this as a proof of concept for integrating high-fidelity digital mission models with RL.Source: §6.1 · p. 34
• A coordinator trained concurrently across both scenarios matched the specialized coordinators in each and performed best overall when the training budget was scheduled appropriately.The more-forgiving-scenario coordinator proved brittle in the more demanding scenario.Source: §6.1 · pp. 34–35

6.2 Future Work

• Future work targets imperfect information sharing, communication delays, misinformation, broader scenario and threat variability, and held-out-scenario evaluation.The authors also propose principled curricula that adapt training shares and ordering to scenario difficulty.Source: §6.2 · p. 35
• The study addresses tactical management of a fixed fleet, while future work would extend the coordinator to portfolio selection, infrastructure and system design, explainability, and resilience to real-time disruptions.Proposed disruptions include asset failures and airport closures.Source: §6.2 · p. 35

• Table 6 is titled “Description of State Variables.”Source: Table 6 · p. 36

• The figure uses green for vegetation, black for burned area, red for the fire line, and blue for water-resource locations.It also depicts Track POI options to prioritize water resources or urban areas, with suppression actions shown in blue and green lines, respectively.Source: B Tactics · pp. 36–37
• Tactical decisions affect fire propagation alongside wind, temperature, humidity, and the type of cell toward which the fire spreads, shaping mission outcomes measured by the MoE.Source: B Tactics · p. 37
• Figure 14 presents simulation progression and two example Track POI and suppression operations.Source: Figure 14 · p. 37

• Table 7 describes per-scenario MoE normalization constants as maximum damage values for each term.Source: Table 7 · p. 37

• Generalized advantage estimation computes advantages from critic temporal-difference errors over a rollout of length T.Source: D PPO Advantage and Critic Losses · p. 37
• Actions with positive advantage become more likely under the updated policy, while actions with negative advantage become less likely.Source: D PPO Advantage and Critic Losses · p. 38
• The critic minimizes squared error between value estimates and return targets, improving its estimates and the advantages it provides as training progresses.Source: D PPO Advantage and Critic Losses · p. 38
• The parameter λ balances bias and variance in the advantage estimate.Source: D PPO Advantage and Critic Losses · p. 38