Planning or Learning: Reliability and Cost in Multi-Asset Maintenance

• Fleet maintenance decisions are coupled because multiple degrading assets share technicians, tools, and downtime windows.This makes maintenance a fleet-level scheduling problem rather than a set of independent asset-level decisions.Source: §1 · p. 1
• The study presents a controlled benchmark that evaluates planning and RL under a unified environment, cost structure, and evaluation protocol for multi-asset maintenance.The protocol is proposed as a reusable template for other maintenance and PHM settings.Source: §1 · p. 2
• Planning enforces failure avoidance as a hard constraint, while RL optimizes discounted soft cost and can favor zero maintenance when asset lifetimes exceed its effective discount horizon.The resulting behavior can differ even when both methods observe the same state.Source: §1 · p. 2
• Reward shaping has negligible effect on RL failure behavior, while action masking reduces failures at significant cost overhead without closing the reliability gap to planning.Source: §1 · p. 2

2.1 Planning for Maintenance Scheduling

• Rolling-horizon planning jointly optimizes across assets, but online re-planning can be computationally costly at scale.Receding-horizon control executes only the first planned action before recomputing from the updated state.Source: §2.1 · p. 2

2.2 RL for Maintenance Scheduling

• RL learns cost-minimizing maintenance policies through interaction without an explicit system model, but matched comparisons with planning remain scarce and delayed failures make credit assignment difficult.Shared crew constraints add dependencies between action consequences in multi-asset settings.Source: §2.2 · p. 2
• Reward shaping, action masking, constrained Markov decision processes, and shielding represent distinct approaches to reliability constraints in learned policies.This work evaluates reward shaping and action masking, leaving CMDPs and shielding for future study.Source: §2.2 · pp. 2–3
• Average-reward and risk-sensitive RL are proposed as unevaluated directions for reducing horizon-related bias or accounting for costly failure risk.Source: §2.2 · p. 3

3.1 Multi-Bearing System

• A fleet contains bearings with independent degradation trajectories, while one shared crew can service at most one bearing per step.The scheduling goal is to minimize cost and failures subject to crew availability.Source: §3.1 · p. 3
• At each step, the action is no maintenance or service of a selected bearing; maintained bearings reset to a post-maintenance state while others continue degrading.Source: §3.1 · p. 3

3.2 Mapping to Discrete Health Indicator

• The algorithms can observe either estimated continuous remaining useful life or a discrete system health state.Source: §3.2 · p. 3
• A composite health indicator is computed from horizontal and vertical vibration signals, then smoothed and discretized into four health categories using ground-truth remaining-life thresholds.The categories are healthy above 75% of initial lifetime, degrading from 25% to 75%, critical below 25%, and failed at zero remaining life.Source: §3.2 · p. 4

3.3 Cost Model

• Episode cost combines operating cost, a fixed charge for maintenance, and a penalty for newly failed bearings.The failure penalty varies across experiments as the risk-tolerance parameter, while other cost parameters remain fixed.Source: §3.3 · p. 4

3.4 Action Space and Scheduling Constraint

• The action space includes no-op and maintenance of any bearing, with only one maintenance action allowed per step and no operating cost for a bearing already failed.Source: §3.4 · p. 4

3.5 Post-Maintenance Dynamics

• Oracle post-maintenance dynamics replay each bearing’s recorded degradation trajectory, whereas estimated dynamics sample lifetime from a fitted log-normal distribution.The latter introduces stochasticity and mismatch between expected and realized lifetime.Source: §3.5 · p. 4

4.1 Planning: Dijkstra Search

• Dijkstra search finds a minimum-cost maintenance schedule over a finite-horizon joint state, assigning infinite cost to any state with a bearing failure.It executes only the first action, then replans; failure cost does not affect its decisions because zero failures are enforced as a hard constraint.Source: §4.1 · pp. 4–5

4.2 Deep Reinforcement Learning

• DQN learns action values with a neural network, while PPO directly optimizes a stochastic policy; both use the same continuous observation space.DQN uses decaying ϵ-greedy exploration and a periodically updated target network; PPO uses an on-policy clipped objective and an entropy bonus.Source: §4.2 · p. 5

4.2.1 Constraint Mechanisms

• Reward shaping amplifies the failure penalty during training, whereas action masking forbids no-op when a non-failed bearing's normalized remaining life falls below δ.Evaluation uses the original cost for reward shaping; the action mask uses δ = 0.25 and applies during both training and evaluation.Source: §4.2.1 · p. 5

4.3 Rule-Based Baselines

• The threshold baseline maintains the most recently threshold-crossing bearing when the crew is available, prioritizing lowest remaining useful life when thresholds coincide.It is designed to avoid failures under oracle dynamics and requires no training.Source: §4.3 · p. 6

5.1 Dataset and Environment

• Six run-to-failure bearing trajectories operate in parallel for 8000-step episodes, with at most one bearing serviced per time step.The episode horizon exceeds the maximum observed lifetime, so failures occur without maintenance.Source: §5.1 · p. 6
• Failure penalties vary across cf = {100, 500, 1000, 5000, 10000}, while operating and maintenance costs remain fixed at 1.0 and 25.0.Experiments use both oracle and estimated post-maintenance dynamics to represent idealized and uncertain conditions.Source: §5.1 · p. 6

5.2 Planning and Heuristic Protocol

• Dijkstra planning uses a 20-step finite horizon, while the threshold heuristic maintains at the critical health state or worse and prioritizes the bearing closest to failure.Planning and heuristic policies are deterministic, so each configuration reports one run.Source: §5.2 · p. 6

5.3 RL Training and Evaluation Protocol

• DQN and PPO are trained across three seeds, with 1500 and 1000 episodes respectively, and final performance is evaluated over 10 episodes.The checkpoint with the lowest mean cost on five evaluation episodes is selected; reward shaping multiplies the training failure penalty by 10, while evaluation uses the original reward.Source: §5.3 · pp. 6–7

6.1 Main Comparison

• Dijkstra costs less than the threshold heuristic; DQN costs less than PPO across most configurations but tends toward more failures, and method rankings remain largely stable under estimated dynamics.PPO generally shows fewer failures, while Dijkstra retains zero failures under both post-maintenance dynamics.Source: §6.1 · p. 7
• Planning and heuristic methods avoid failures, while RL may trade failures for lower costs at low penalties; at high penalties, planning can dominate on cost and reliability.The cost–failure trade-off is shown across methods and penalty levels.Source: §6.1 · p. 7, Figure 2 · p. 7, Table 1 · p. 8

6.2 Planning Cost and Runtime

• Dijkstra requires approximately 90 minutes of inference per episode, versus 28 seconds for the heuristic and approximately 9 and 10 seconds for DQN and PPO.RL shifts computation to a one-time training phase of approximately 40–68 minutes in these experiments.Source: §6.2 · p. 8, Table 2 · p. 8

6.3 Constraint mechanisms

• Reward shaping does not consistently change failure behavior; action masking reduces failures more, but raises total cost through more frequent maintenance.Neither mechanism consistently matches planning and heuristic methods’ zero-failure performance.Source: §6.3 · pp. 8–9, Table 3 · p. 9

6.4 Further analysis of DQN failures

• DQN converges to a near-zero-maintenance policy across state representations; the proposed discount–horizon mismatch is a plausible explanation, and γ = 0.999 does not resolve it.The authors note that other training and reward factors may contribute, and formally disentangling them remains future work.Source: §6.4 · pp. 9–10, Table 4 · p. 9

6.5 Discussion and broader impact

• The comparison deliberately assigns different reliability objectives; isolating algorithmic differences requires matched-objective tests, such as constrained RL against finite-penalty planning.Planning’s zero-failure performance follows from its imposed hard constraint, not a general demonstrated advantage over RL.Source: §6.5 · p. 10
• The study uses six bearings from one run-to-failure dataset and a fixed horizon, with accurate RUL or ground-truth-derived health-state observations.The authors note that behavior under other data distributions, system scales, operating horizons, and noisy prognostics estimates remains uncharacterized.Source: §6.5 · p. 10

• Planning enforces reliability as a hard constraint, while vanilla RL optimizes discounted expected cost that may tolerate failures.Source: §7 · p. 10
• DQN collapses to zero maintenance amid a discount-horizon mismatch, while PPO partly mitigates the issue but reaches near-zero failures only with additional constraint mechanisms.Source: §7 · p. 10
• Both heuristic and planning policies achieve zero failures, but planning achieves lower operational cost at significantly higher computational time.Source: §7 · p. 10
• Planning suits strict reliability requirements, while RL suits settings where limited failures are acceptable and long-term efficiency matters.Future work should explore constrained MDPs, scaling behavior, and hybrid planning–RL approaches.Source: §7 · p. 10