PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement Learning-Guided Evolutionary Policy Optimization for Preference-Adjustable Heterogeneous Agile Earth Observation Satellite Scheduling

He Wang, Junyu Wu, Hui Li, Yanjie Song, Witold Pedrycz, Liang Li

arXiv:2608.24470v1cs.AI

TL;DR

Heterogeneous AEOS scheduling must handle satellite-dependent access, maneuvering, and resource constraints while balancing task gain, energy saving, and load balance. The paper proposes RLOSMEA, which decouples decoding and evolutionary search from online actor-critic operator control. Across heterogeneous scenarios, RLOSMEA achieves higher weighted utility and more stable convergence than representative metaheuristic baselines, with sensitivity analyses supporting its robustness.

  • Problem

    Heterogeneous AEOS scheduling requires unified optimization of assignments and sequences despite satellite-dependent visibility, transition, energy, storage, and load constraints.

  • Method

    The paper combines indirect assignment encoding, decoder-based equivalent-cost evaluation, weighted utility modeling, population search, and online actor-critic selection of high-level operator modes.

  • Results

    RLOSMEA achieves higher weighted utility and more stable convergence than representative metaheuristic baselines across different heterogeneous AEOS scenarios.

  • Takeaways & Limitations

    The results support reinforcement-learning-guided operator selection as an effective approach for preference-adjustable heterogeneous AEOS scheduling.

Abstract

from arXiv · show

Heterogeneous agile Earth observation satellite (AEOS) scheduling requires task selection, satellite assignment, and observation sequencing under satellite-dependent visibility windows, attitude maneuvering requirements, energy consumption, and onboard storage constraints. Since satellites differ in orbital access, maneuvering capability, and payload resources, the same task may have different feasible windows, transition costs, and resource-consumption patterns on different platforms, which increases the difficulty of unified modeling and efficient optimization. To address this problem, this paper proposes an evolutionary policy optimization framework for heterogeneous AEOS scheduling with preference-adjustable weighted objectives. In the modeling layer, assignment-based indirect encoding is combined with decoder-based equivalent-cost evaluation to retain satellite-dependent constraints while integrating task gain, energy saving, and load balance into an interpretable scalar utility. In the optimization layer, schedule decoding, population-based search, and online actor-critic operator control are decoupled, so that reinforcement learning selects high-level search operators rather than constructing schedules directly. Based on this framework, a reinforcement-learning-assisted operator-selection memetic evolutionary algorithm (RLOSMEA) is developed to coordinate global exploration, feasibility recovery, and local refinement under a limited function-evaluation budget. Experiments on different heterogeneous AEOS scenarios show that RLOSMEA achieves higher overall weighted utility and more stable convergence than representative metaheuristic baselines. Sensitivity and learning-behavior analyses further confirm the robustness of the proposed method and the effectiveness of reinforcement-learning-guided operator selection.

I. INTRODUCTION

Heterogeneous AEOS scheduling must coordinate task selection, satellite assignment, and sequencing under satellite-dependent visibility, maneuvering, and resource constraints. The paper addresses this challenge with a preference-adjustable evolutionary policy optimization framework that uses reinforcement learning to select search operators rather than construct schedules directly.

  • Related work: Existing AEOS formulations and algorithms often emphasize task selection, sequencing, and feasibility, while heterogeneous multi-satellite modeling remains insufficiently addressed.Exact methods can solve medium-scale or structured cases, whereas heuristics and metaheuristics are more practical for large-scale scheduling.
  • Related work: Fixed operator probabilities and manually tuned adaptive rules can cause premature convergence, weak feasibility recovery, or unstable performance across scenarios.These limitations motivate adaptive control of evolutionary search behavior.
  • Proposed approach: The paper introduces an Evolutionary Policy Optimization framework combining weighted utility modeling, decoder-based evaluation, population search, and online actor-critic operator-mode selection.The policy selects high-level modes coordinating exploration, feasibility recovery, and schedule refinement instead of constructing schedules directly.
  • Proposed approach: RLOSMEA adaptively coordinates global exploration, feasibility recovery, refinement, and diversity maintenance under different heterogeneous scheduling scenarios and preference settings.The framework separates schedule decoding, evolutionary search, and policy control through a modular design.
  • Problem setting: Heterogeneous satellites give the same task different feasible windows, transition requirements, and resource-consumption patterns.Scheduling must account for satellite-specific orbital access, maneuvering capability, energy, storage, and load characteristics.

B. Notation

The scheduling representation encodes each task’s satellite assignment indirectly, while a decoder determines feasible observation timing and sequencing. Schedule quality is evaluated through normalized task-gain, energy-saving, and load-balance components.

  • Assignment representation: Each candidate schedule is represented by a task-to-satellite assignment vector rather than explicit timing and sequencing decisions.The vector uses x_j = 0 for unassigned tasks and x_j = i for assignment to satellite i.
  • Schedule decoding: The schedule decoder constructs observation intervals and satellite-specific sequences while enforcing visibility, timing, conflict, maneuvering, and resource constraints.This leaves exact observation timing and execution order to decoding.
  • Utility evaluation: Decoded schedules are evaluated using normalized task gain, energy saving, and load balance to represent different operational preferences.Task gain measures scheduled observation value, energy saving reflects resource efficiency, and load balance characterizes workload distribution.
  • Utility evaluation: Task-gain utility increases when more high-priority observation tasks are successfully scheduled.The normalization uses a small positive ε to avoid division by zero.
  • Utility evaluation: Satellite energy consumption includes both observation and attitude-transition energy, which are platform-dependent in heterogeneous systems.Different observation rates, maneuvering efficiencies, and energy budgets produce different energy-consumption patterns.

2) Energy-Saving Utility:

The model combines task gain, energy saving, and load balance through a preference weight vector whose components sum to one. Changing the weights represents different operational priorities while preserving a single weighted scheduling objective.

  • Energy-Saving Utility: A larger normalized energy-saving utility corresponds to lower schedule energy consumption and better resource efficiency.The energy utility uses E_ref as its normalization scale.
  • Load-Balance Utility: Load balance is computed from satellite workload and resource usage relative to service capability, with larger utility indicating more balanced resource utilization.The formulation uses average load and load dispersion across satellites.
  • Preference-adjustable objective: The preference vector w = (w_g, w_e, w_b) assigns weights to task gain, energy saving, and load balance, with nonnegative components summing to one.These weights allow the scheduler to emphasize different utility dimensions.
  • Preference-adjustable objective: The weighted scheduling objective maximizes the combined utility of task gain, energy saving, and load balance.The same formulation supports gain-prioritized, energy-saving, and load-balanced operation through weight adjustment.
  • Experimental evaluation: The reported experimental value is the best weighted utility obtained in one independent run.The candidate schedule set is denoted by Ω in the evaluation formulation.

F. Constraints and Constraint Handling

Constraint handling preserves task-assignment, visibility, timing, conflict, maneuvering, and onboard-resource feasibility. Because assignments are encoded indirectly, these constraints are checked during schedule decoding rather than embedded explicitly in the chromosome.

  • Constraint handling: The decoder evaluates assignment-based candidates against temporal conflicts, attitude-transition feasibility, and onboard resource limits.This approach enforces feasibility during decoding instead of explicitly embedding all constraints in the chromosome.
  • Task assignment constraint: Each task can be assigned to at most one satellite.The integer assignment encoding naturally satisfies this uniqueness condition because each task has one assignment variable.
  • Visibility constraint: An assigned task must have at least one feasible satellite-task visibility window.Visibility feasibility is represented through the set W_ij and indicator v_ij.
  • Timing constraint: A scheduled observation must fit simultaneously within the task time window and a satellite-task visibility window.The required observation duration must be accommodated by the decoded interval.

3) Temporal and Attitude-Transition Constraint:

The scheduler enforces satellite-specific temporal, transition, energy, storage, and feasibility constraints while converting weighted utility into an equivalent minimization cost.

  • Temporal and Attitude-Transition Constraint: Each decoded satellite sequence must leave enough time between consecutive tasks for the required attitude transition.For consecutive tasks jk and jk+1, the next start must follow the previous finish plus satellite-specific transition time.
  • Energy and Storage Constraints: Energy budgets, storage capacities, and resource-consumption processes are evaluated separately for each heterogeneous satellite during decoding.The model tracks remaining energy and onboard data against satellite-specific planning-horizon capacities.
  • Constraint-Violation Measure and Equivalent Cost: Total constraint violation sums time-window, duration, transition, energy, and data-storage violations, with CV(x) = 0 indicating strict feasibility.The normalized violation guides search toward feasible regions instead of simply discarding infeasible schedules.
  • Constraint-Violation Measure and Equivalent Cost: The optimizer converts weighted utility into an equivalent cost and penalizes constraint violations and unscheduled tasks.This provides a minimization-compatible evaluation while retaining the scheduling objective and feasibility penalties.
  • Constraint-Violation Measure and Equivalent Cost: The schedule with the largest weighted utility is reported as the final scheduling result.

III. EPO FRAMEWORK AND RLOSMEA

The EPO framework searches compact task-to-satellite assignments, decodes them into heterogeneous schedules, and uses online actor–critic control to select high-level evolutionary operator modes.

  • III. EPO FRAMEWORK AND RLOSMEA: Individuals encode task-to-satellite assignments, while a shared decoder constructs intervals, execution sequences, resource consumption, and constraint violations.The decoder preserves satellite-dependent feasibility while allowing search in a compact assignment space.
  • III. EPO FRAMEWORK AND RLOSMEA: The evolutionary module updates candidate assignments, whereas the policy module observes population state and selects operator modes for offspring generation and schedule modification.Reinforcement learning controls search operators rather than constructing schedules directly.
  • III. EPO FRAMEWORK AND RLOSMEA: RLOSMEA combines memetic evolutionary search with warm start, feasibility repair, conflict resolution, local improvement, elite intensification, immigrant injection, and diversity rescue.Schedule modifications occur before decoding, so each modified offspring is evaluated once by the shared decoder.
  • III. EPO FRAMEWORK AND RLOSMEA: The online MDP represents operator selection through population states, operator-mode actions, transition dynamics, rewards, and a discount factor.The controller learns generation by generation without an offline training dataset, adapting to the current search trajectory and scenario.
  • III. EPO FRAMEWORK AND RLOSMEA: Population states summarize search progress, scheduling quality, feasibility, and population structure, with phase descriptors adding stagedependent search information.Bootstrap, shaping, repair, and intensification phases emphasize different search conditions; recovery safeguards severe stagnation or diversity collapse.

E. Operator-Mode Action Space

The actor selects interpretable high-level operator modes, which are decoded into crossover, mutation, and schedule-specific control rates for adaptive offspring modification.

  • E. Operator-Mode Action Space: The action space contains coordinated high-level operator modes for modifying offspring assignment vectors.The selected action is decoded into crossover, mutation, and schedule-specific operator rates.
  • E. Operator-Mode Action Space: Schedule-specific operators include warm start, feasibility repair, conflict resolution, local improvement, elite intensification, immigrant injection, rare-task shaping, and diversity rescue.
  • E. Operator-Mode Action Space: Sparse control activates one dominant problem-oriented operator while retaining other operators as weak safeguards.This improves action interpretability and provides a clearer reward signal for policy learning.
  • E. Operator-Mode Action Space: The actor computes action preferences from controller features using parameterized logits, temperature-controlled softmax, phase priors, and epsilon mixing.The critic estimates state value using a linear approximation, and temporal-difference error supports online updates.
  • E. Operator-Mode Action Space: Recovery-forced actions do not update the actor policy, preventing safeguard decisions from biasing policy learning.The update is skipped only when recovery mode replaces the sampled action.

G. Reward Function

The reward function aligns operator selection with weighted scheduling utility and equivalent-cost improvement while incorporating offspring success into online learning.

  • G. Reward Function: The reward combines weighted-utility improvement, best-cost improvement, mean-cost improvement, and offspring success rate.Feasibility, diversity, and stagnation inform the state and phase descriptors rather than entering the reward as separate terms.
  • G. Reward Function: Weighted-utility improvement compares population-level utilities before and after applying the selected operator mode.Ut and Ut+1 denote the population-level weighted utilities at consecutive stages.
  • G. Reward Function: Best- and mean-cost improvements measure reductions in the population’s best and average equivalent costs under the minimization interface.
  • G. Reward Function: Offspring success rate contributes to reward by measuring offspring performance relative to the parent population.The parent and offspring populations are denoted by Pt and Qt, and I(·) is the indicator function.
  • G. Reward Function: The compact reward preserves consistency with both the weighted-utility objective and equivalent-cost minimization.

H. Policy-Controlled Evolutionary Search

The EPO framework separates schedule evolution from policy learning, using reinforcement learning to select high-level operator modes that guide exploration, feasibility recovery, and refinement. A shared decoder evaluates each modified offspring once, supporting fixed evaluation budgets.

  • Policy-Controlled Evolutionary Search: After action selection, the decoder maps each action to an operator-control vector specifying offspring-generation and refinement strategies.Preliminary offspring undergo mating selection, crossover, and mutation before schedule-specific refinement.
  • Policy-Controlled Evolutionary Search: Schedule-specific operators include warm start, feasibility repair, conflict resolution, local improvement, elite intensification, immigrant injection, and diversity rescue.
  • Policy-Controlled Evolutionary Search: All schedule modifications occur before objective evaluation, so each offspring is decoded and evaluated once by the shared schedule decoder.This avoids repeated evaluations after individual sub-operators and preserves a clear interface between operator control and schedule evaluation.
  • Policy-Controlled Evolutionary Search: Feasibility-aware environmental selection ranks feasible individuals by equivalent cost and infeasible individuals by normalized constraint violation.The selected population defines the next evolutionary state.
  • Policy-Controlled Evolutionary Search: The EPO framework separates evolutionary search over assignment vectors from policy learning that selects high-level operator modes rather than constructing schedules directly.
  • Policy-Controlled Evolutionary Search: RLOSMEA uses population state, phase information, and compact rewards to switch among construction, repair, exploitation, and exploration under a fixed evaluation budget.Optimization uses equivalent cost, while reported scheduling performance uses weighted utility.

IV. SIMULATION EXPERIMENTS AND ANALYSIS

Experiments evaluate RLOSMEA across heterogeneous scenarios with increasing task and satellite scales under common decoding, objective, constraint, and stopping settings. It achieves the best weighted utility across scenarios, with stable convergence and gains driven mainly by task-gain utility while preserving load balance.

  • Experimental Settings: Six scenarios increase from 100 to 350 candidate tasks and from 5 to 12 agile satellites, enlarging assignment and scheduling coupling.
  • Experimental Settings: Heterogeneity arises from satellite-specific orbital access, energy budgets, storage capacities, and execution costs, giving the same task platform-dependent feasibility and resource patterns.
  • Experimental Settings: All algorithms use identical scenario files, visibility windows, decoder, objective function, constraint rules, and stopping criterion, without scenario-specific tuning.
  • Experimental Settings: 30 independent runs per scenario use a budget of 10000 objective evaluations, with convergence curves reporting average best-so-far weighted utility and standard-deviation bands.
  • Convergence Behavior: RLOSMEA achieves the highest weighted utility in all scenarios, while AESSPSO and ALNS improve rapidly early but stagnate sooner and MemeticEA remains the strongest baseline.
  • Convergence Behavior: The relatively narrow shaded bands indicate stable run-to-run performance, while policy-guided operator selection redistributes tasks when feasibility or diversity becomes limiting.
  • Component-Wise Utility Analysis: RLOSMEA’s main advantage comes from task-gain utility while maintaining high load-balance utility, rather than excessively concentrating tasks on a few satellites.
  • Component-Wise Utility Analysis: Energy-saving utility is not always best because the gain-oriented weighting permits moderate energy increases when they produce larger overall weighted utility.

3) Discussion:

RLOSMEA remains superior across tested preference settings and problem scales, while its utility components respond smoothly to changing weights. The results support robust preference adjustment through tradeoffs among task gain, energy saving, and load balance.

  • Overall Performance Across Weight Settings: RLOSMEA achieves the best performance across all tested weights in small-, medium-, and large-scale representative scenarios.
  • Overall Performance Across Weight Settings: Its ranking remains generally consistent as preference shifts from task-gain-oriented scheduling toward balanced consideration of gain, energy consumption, and load distribution.The relatively narrow bands suggest repeatability across independent runs.
  • Component-Wise Response Under Different Preferences: As gain weight decreases and energy-saving and load-balance weights increase, task-gain utility decreases gradually, energy-saving utility improves, and load-balance utility remains high.
  • Component-Wise Response Under Different Preferences: RLOSMEA adjusts tradeoffs among task gain, energy consumption, and load distribution according to prescribed weights rather than optimizing one dominant component in every setting.The same response pattern appears in scenarios 01, 03, and 05.
  • Discussion: Across all tested preference settings, RLOSMEA maintains superior weighted utility and its component responses vary smoothly with the weight vector.These results support preference-adjustable multi-objective scheduling in heterogeneous AEOS systems.
  • Discussion: The internal analysis further examines parameter sensitivity and reinforcement-learning operator-selection dynamics on scenario 04.

3) Discussion:

RLOSMEA remains robust across tested parameter ranges while its actor–critic controller adaptively diversifies operator selection. The analyses support stronger weighted-utility performance, stable convergence, and reinforcement-learning-guided coordination in heterogeneous AEOS scheduling.

  • Parameter Sensitivity: RLOSMEA maintains stable performance across the tested parameter ranges without abrupt degradation, indicating limited dependence on delicate parameter tuning.The default setting lies in a relatively stable region rather than near an isolated optimum.
  • Parameter Sensitivity: The softmax temperature has a stronger influence than the learning rates and exploration rate because it controls action-selection sharpness.Conflict-cluster and local-improvement rates also affect results, highlighting the role of structural refinement.
  • Learning Behavior of Operator Selection: Weighted utility increases steadily as gain utility improves while load-balance utility remains high, showing stronger task-gain capture without seriously damaging workload distribution.
  • Learning Behavior of Operator Selection: Selected operator modes remain diversified throughout optimization rather than being dominated by one action, preserving exploration, feasibility recovery, and exploitation.The actor–critic controller adjusts operator-selection tendencies according to the current search condition.
  • Discussion: RLOSMEA integrates satellite-dependent feasibility with task gain, energy saving, and load balance through assignment-based encoding, decoding, and equivalent-cost evaluation.
  • Discussion: Across heterogeneous scenarios, RLOSMEA achieves higher weighted utility and more stable convergence than representative metaheuristic baselines under the same function-evaluation budget.Component-wise analysis attributes the improvement mainly to stronger task-gain capture while maintaining acceptable energy-saving and load-balance performance.
Loading 2608.24470v1…