Source-linked AI summary
Implicit Q-learning-bootstrapped ant colony optimization for maritime moving-target observation scheduling with agile satellites
He Wang, Junyu Wu, Yeye Liu, Yifan Zhou, Jie Zhang, Hui Li, Yanjie Song, Liang Li
TL;DR
Maritime moving-target observation scheduling couples time-varying windows, satellite assignments, sequencing, and operational constraints. IQACO combines offline implicit Q-learning with ant colony optimization to adapt search control, achieving higher objective values and faster convergence across 14 scenarios.
Problem
Maritime moving-target scheduling is a sequence-dependent, resource-constrained optimization problem because vessel and orbital motion make observation windows time dependent.
Method
IQACO uses an offline implicit Q-learning policy to adapt ACO parameters while deterministic construction preserves feasible schedule generation.
Results
Across 14 scenarios, IQACO achieves higher objective values and faster convergence than competing methods, with stronger gains in larger-scale scenarios and competitive weight sensitivity.
Takeaways & Limitations
Offline value learning can regulate exploration and exploitation as an adaptive search-control mechanism for constrained maritime moving-target scheduling.
Takeaways & Limitations
The study relies on simulated target distributions, simplified cloud-availability modeling, and an offline policy trained within a fixed scenario distribution.
Abstract
from arXiv · showhide
Maritime moving-target observation scheduling with agile Earth observation satellites is a dynamic, sequence-dependent combinatorial optimization problem. Sea-surface targets move continuously, causing feasible observation windows to vary with target motion and satellite orbital geometry. The scheduler must jointly determine task selection, satellite assignment, observation-window selection, and observation ordering under time-window, attitude-maneuvering, onboard-resource, and cloud-affected availability constraints. This paper proposes an implicit Q-learning-bootstrapped ant colony optimization method, termed IQACO, for multi-satellite maritime moving-target observation scheduling. Rather than directly learning a task-selection policy, IQACO embeds an offline implicit Q-learning module into constructive ant colony optimization to adaptively adjust the pheromone factor, heuristic factor, and evaporation rate. A compact search-state representation captures pheromone distribution, current and historical-best solution quality, and iteration progress. During online scheduling, ant colony optimization constructs feasible observation sequences, while the learned policy regulates exploration and exploitation according to the current search state. Experiments on 14 scenarios with different scales and satellite configurations show that IQACO obtains the highest mean observation benefit in every scenario, improves the result of conventional ant colony optimization by 3.40\%--9.40\%, accelerates convergence, and remains stable under different objective-weight settings. These results demonstrate that offline value learning provides an effective adaptive search-control mechanism for constrained maritime moving-target observation scheduling.
1. Introduction
The paper addresses the dynamic, sequence-dependent scheduling of maritime moving-target observations by agile Earth observation satellites under changing windows and operational constraints. It proposes IQACO, which uses offline implicit Q-learning to adapt ant colony optimization control parameters while ACO constructs feasible schedules.
- Problem motivation: Maritime moving-target scheduling is difficult because target geometry makes observation windows time dependent, while feasibility also depends on task order and maneuvering requirements.Frequent slews and high-resolution imaging additionally consume onboard resources.
- Method motivation: Conventional ACO uses fixed pheromone factor α, heuristic factor β, and evaporation rate ρ, which may poorly balance exploration and exploitation across changing scenarios.Scenario differences include target density, satellite number, feasible-window distribution, and resource constraints.
- Proposed method: IQACO uses an offline-trained IQL policy to adaptively adjust α, β, and ρ from a compact five-dimensional search state.ACO continues constructing feasible schedules under time-window, attitude-maneuvering, energy, storage, and cloud-affected availability constraints.
- Problem formulation: The scheduling model integrates time-varying observation windows, satellite-task-window assignment, sequence-dependent attitude maneuvering, onboard energy and storage resources, and cloud-affected availability.
- Experimental evaluation: Experiments cover 14 scenarios with different problem scales and satellite configurations, evaluating convergence, final observation benefit, objective-weight sensitivity, and training behavior.
2. Scheduling Model · 2.1. Problem Description
The model addresses multi-satellite scheduling of maritime moving-target observations over a finite horizon, jointly selecting and sequencing tasks under dynamic visibility and operational constraints. Feasibility depends on target–satellite geometry, cloud-affected availability, and execution order for agile satellites.
- 2.1. Problem Description: The planning problem covers agile satellites and maritime moving targets over a finite planning horizon TH.
- 2.1. Problem Description: Target trajectories and satellite ephemerides are propagated before optimization to generate feasible task–satellite observation windows.
- 2.1. Problem Description: The scheduler jointly determines task selection, satellite assignment, observation-window selection, and task ordering on each satellite.
- 2.1. Problem Description: Scheduling must satisfy time-window, attitude-maneuvering, onboard-resource, and cloud-affected availability constraints.
- 2.1. Problem Description: Relative satellite–target geometry changes with both vessel motion and orbital motion, making feasible observation windows time dependent.
- 2.1. Problem Description: A geometrically visible window may have low practical availability because of ocean cloud cover, while agile-satellite feasibility also depends on selected-task execution order.
2.2. Sets and Parameters
The scheduling model expands each moving target’s multiple observation requirements into independent tasks and represents feasibility through task–satellite window sets. Binary variables encode task assignment and window selection, while successor variables define satellite-specific task order and maneuvering costs.
- Task and feasibility sets: Each original moving target’s multiple observation requirements are expanded into independent scheduling tasks.The expansion applies within the planning horizon.
- Task and feasibility sets: If task i cannot be observed by satellite s, its feasible-window set W_i,s is empty.Feasibility is therefore represented separately for each task–satellite pair.
- Decision variables: The binary variable x_i,s,k equals 1 when task i is assigned to satellite s and executed in window k, and 0 otherwise.This variable jointly represents task selection, satellite assignment, and observation-window selection.
- Decision variables: The binary variable y_s,ij equals 1 when satellite s executes task j immediately after task i, and 0 otherwise.It defines task order on each satellite and supports calculating attitude-maneuvering time and energy between consecutive observations.
2.3. Constraints
A feasible schedule must satisfy assignment, observation-window, sequencing, attitude-maneuvering, onboard-resource, and cloud-affected-availability constraints. These rules are enforced during constructive schedule generation, including exclusion of impractical optical windows.
- General feasibility: Feasibility covers task assignment, observation windows, task sequencing, attitude maneuvering, onboard resources, and cloud-affected availability.These constraints serve as feasibility rules in the constructive decoder.
- Assignment and windows: Each task is executed at most once, and each observation interval must be contained within its selected window.Assignment and window constraints apply to selected task–satellite–window decisions.
- Task sequencing: Successor relations are satellite-specific, with each scheduled task having at most one immediate successor and predecessor.Constructive ACO checks successor relations, temporal feasibility, and resource feasibility during schedule construction.
- Attitude maneuvering: Consecutive observations on the same agile satellite require rate- and acceleration-limited attitude maneuvers, using triangular or trapezoidal angular-rate profiles.The required slewing angle is computed from payload-pointing-direction separation, and transition time is reserved between observations.
- Cloud-affected availability: Cloud-affected optical windows below cmin are excluded, while retained windows use cloud availability as a benefit attenuation factor.Geometric visibility alone may not imply practical observation value under cloud cover.
2.4. Objective Function
The objective maximizes overall constellation observation performance through normalized observation-benefit, energy-efficiency, and workload-balance components. Its weighting expresses mission preferences while favoring practical observations, lower relative energy use, and balanced satellite workloads.
- Objective formulation: The objective maximizes overall observation performance using normalized observation benefit, energy efficiency, and workload balance.Normalization lets the weights represent mission preferences rather than compensate for different physical scales.
- Objective formulation: The component weights satisfy η1 + η2 + η3 = 1 and η1, η2, η3 ≥0.
- Observation benefit: The observation-benefit term measures effective benefit from selected tasks while reducing the benefit of windows with low cloud-affected availability.The reduction is applied through factor ci,s,k.
- Energy efficiency: The energy-efficiency term accounts for imaging and attitude-maneuvering energy, with larger Fe indicating lower relative energy use.
- Workload balance: The workload-balance term compares satellite observation workloads and rewards more balanced distributions, using ϵ to avoid division by zero.Workload is defined by the total duration of each satellite’s selected observations.
3. Method
IQACO combines offline implicit Q-learning with online ant colony optimization to adaptively control ACO parameters while retaining deterministic feasible-schedule construction. Its state captures search quality, pheromone distribution, and iteration progress, and its continuous policy adjusts α, β, and ρ with low online overhead.
- IQACO framework: IQACO separates offline IQL training from online ACO scheduling, using the learned policy to adjust α, β, and ρ after each iteration.Offline transitions are collected from exploratory ACO runs; online ants construct schedules, update the best solution and pheromone matrix, and apply policy adjustments.
- IQACO framework: Unlike direct reinforcement-learning schedulers, IQACO learns continuous ACO parameter adjustments while a deterministic decoder constructs feasible schedules and enforces constraints.This design uses IQL as a value-guided search controller rather than replacing the scheduling algorithm.
- Hierarchical solution encoding: A candidate node z = (i, s, k) encodes a task, satellite, and feasible observation window, while complete schedules decompose into satellite-specific ordered sequences.Candidates are pre-filtered by visibility, time-window feasibility, and cloud-affected availability; construction excludes operationally infeasible candidates.
- Adaptive ACO control: The adaptive-control MDP represents pheromone statistics, current and global-best objective values, and normalized iteration progress, then outputs continuous adjustments for α, β, and ρ.The reward balances solution improvement, search diversity, and global-best progress, with clipping used to stabilize offline training.
- Offline IQL training: The offline dataset records exploratory transitions containing states, actions, rewards, next states, and terminal indicators, from which IQL learns value functions and a policy.IQL uses a value function, two Q-functions, and advantage-weighted policy regression.
- Online scheduling: The five-dimensional policy input and once-per-iteration inference make IQACO’s additional online overhead small relative to schedule construction and feasibility checking.This overhead assessment is stated for the online scheduling stage.
4. Simulation Experiments
Experiments across 14 maritime moving-target scheduling scenarios show that IQACO consistently improves observation benefit and convergence over competing algorithms, including conventional ACO. Additional sensitivity and offline-training analyses indicate robust performance across objective weights and stable learning behavior.
- Experimental setup: 14 scenarios with 100–240 maritime moving targets and 3–6 satellites evaluate IQACO against GA, PSO, WOA, and conventional ACO under matched budgets and constraints.Each algorithm is executed 20 times per scenario with NFE = 20000.
- Convergence behavior: IQACO reaches higher best-so-far objective values than conventional ACO in moderate-scale scenes and exceeds all four competitors in larger scenes.The final separation from competing methods increases as target and satellite counts grow.
- Statistical results: IQACO obtains the highest mean observation benefit in every scenario, with gains over conventional ACO increasing from 3.40% in Scene 01 to 9.40% in Scene 14.Wilcoxon signed-rank testing identifies significant superiority over all baseline algorithms at p < 0.05.
- Weight sensitivity: IQACO achieves the highest or near-highest values under most of five objective-weight configurations across four scenarios with three to six satellites.The sensitivity study reports mean and standard deviation over 10 runs.
- IQL training behavior: Offline IQL training remains stable: losses decrease rapidly before entering a bounded fluctuation range, while value estimates and action weights stabilize without evident divergence.The learned policy provides adaptive parameter control for online ACO scheduling.
5. Conclusion
The paper presents IQACO, an IQL-guided adaptive ACO framework that preserves ACO schedule-construction feasibility while using offline value learning to regulate search behavior. Across 14 scenarios, IQACO achieves higher objective values and faster convergence, while future work targets more realistic operational settings and adaptive learning.
- Method: IQACO adjusts key ACO parameters through offline implicit value learning rather than directly generating scheduling decisions.This preserves the feasibility advantage of ACO-based schedule construction while regulating exploration–exploitation behavior during search.
- Results: 14 scenarios show that IQACO achieves higher objective values and faster convergence than competing methods.The performance gain is more evident in larger-scale scenarios with complex task-window distributions and sequence-dependent constraints.
- Limitations and future work: The study uses simulated target distributions, simplified cloud-availability modeling, and an offline policy trained within a fixed scenario distribution.Future work will incorporate real AIS trajectories, time-varying cloud fields, more realistic satellite constraints, transfer learning, online adaptation, and hybrid learning-search mechanisms.