PORL: Pretrained Offline Reinforcement Learning for the Job Shop Scheduling Problem

I. INTRODUCTION

• Classical optimization and handcrafted dispatching rules have limited adaptability or generalization across changing production environments and scheduling scenarios.Online RL learns from experience but requires simulation, which can create a simulation-to-reality gap.Source: §I · p. 1
• Offline RL learns from historical trajectories without environment interaction, but its performance depends strongly on dataset quality and coverage.PORL initializes offline learning from a policy that has explored a large portion of the scheduling state space.Source: §I · p. 1
• PORL first learns general scheduling strategies through simulated online interaction, then adapts the policy offline to production-specific data.The reverse training order is motivated by the cost and operational risk of exploratory scheduling in deployed industrial systems.Source: §I · pp. 1–2
• A policy-level constraint limits fine-tuning deviations from the pretrained policy, while offline adaptation targets the production distribution.The contribution summary describes adapting pretrained Q-networks while controlling divergence from the pretrained policy.Source: §I · pp. 1–2
• Under substantial distributional shift, PORL achieves lower optimality gaps than dispatching-rule and standalone offline RL baselines, and performs strongly on low-return datasets.The introduction also reports that PORL outperforms standalone offline RL on low-return data.Source: §I · p. 2

A. RL for JSSP

• JSSP RL methods use graph-based representations and GNNs, while DQN policies commonly train through simulator interaction and depend on simulator fidelity.Prior graph-based approaches report strong benchmark performance and generalization [7], [8], [13], [14].Source: A. RL for JSSP · p. 2

B. Offline RL for scheduling

• Offline RL avoids environment interaction, but distributional shift can make value estimates unreliable for unsupported actions; CQL uses value regularization to address this issue.JSSP studies report competitive CQL scheduling using optimal, heuristic, and noisy-trajectory datasets [12],.Source: B. Offline RL for scheduling · p. 2

C. Research gap

• The identified gap is that online-pretraining-then-offline-adaptation had not been systematically explored for JSSP, although target-environment exploration is often unrealistic in industrial scheduling.Online RL enables exploration but may not reflect real production, while offline RL is constrained by available dataset support.Source: C. Research gap · p. 2

A. Model Architecture

• Each JSSP state is represented as a heterogeneous graph with operation and machine nodes, connected by precedence and assignment edges.Precedence edges connect consecutive operations within jobs, while assignment edges link operations to their machines.Source: A. Model Architecture · p. 2
• Feature-based homogenization uses one-hot node-type information to let operation and machine nodes share a feature space while retaining their structural distinctions.The resulting graph is processed by a K-layer GIN whose stacked layers aggregate K-hop neighborhoods.Source: A. Model Architecture · pp. 2–3

III. ALGORITHM

• The model architecture is held fixed across experiments, with its GNN hyperparameters listed in Table I.Fig. 1 illustrates the training pipeline for the offline RL baseline and PORL.Source: §III · p. 2, A. Model Architecture · p. 3, Table I · p. 3

1) Positioning:

• PORL is intended to specialize a well-trained general policy to a production setting using limited offline data, with fixed online and offline training budgets.The paper evaluates the feasibility of this adaptation principle.Source: 1) Positioning · p. 3

2) Initialization and setup:

• The pretrained online DQN weights initialize the hybrid agent, requiring the same Q-network architecture in both training stages.The initialized agent is then fine-tuned offline.Source: 2) Initialization and setup · p. 3

3) Policy Constraint Formulation:

• The offline objective adapts the pretrained Q-function to target instances while limiting large deviations from the original policy.The constraint is intended to allow targeted specialization without catastrophic degradation of previously learned behavior.Source: 3) Policy Constraint Formulation · p. 3
• A stochastic policy is derived from each Q-function by softmax, and the pretrained policy remains fixed as the reference during fine-tuning.The softmax temperature controls distribution smoothness, and the action mask defines valid actions.Source: 3) Policy Constraint Formulation · p. 3
• Forward KL penalizes probability assigned to actions unlikely under the pretrained policy while allowing action preferences to be reweighted; training also retains CQL alongside DQN and KL losses.The KL constraint strength is controlled by λKL.Source: 3) Policy Constraint Formulation · p. 3

• Online pretraining uses uniformly distributed 15 × 15 instances, while offline fine-tuning uses custom distributions with significant distribution shift and 10 × 10 instances.The size convention is J × M, where J is jobs and M is machines.Source: §IV · p. 4
• Target templates vary machine-specific processing times using Gaussian or uniform distributions and assign partial or full precedence constraints to 50% of jobs.Remaining jobs follow the standard unconstrained JSSP formulation.Source: §IV · p. 4
• The specialized instances differ from generic uniform benchmarks and are intended to resemble shop floors with repeated product flows and heterogeneous machines.The paper describes this shift as a way to bridge simulation-based training and practical manufacturing environments.Source: §IV · p. 4

• Two ϵ-greedy priority dispatching policies follow heuristics but choose a random valid action with probability ϵ.This introduces variability similar to real production logs.Source: A. Behavioral policies · p. 4
• Noisy-expert policies follow an exact-solver schedule with probability 0.5; otherwise each scheduled action has a 0.1 chance of being replaced by a random valid action.If a random action blocks the scheduled optimal action, another random valid action is scheduled.Source: A. Behavioral policies · p. 4
• The third behavioral policy chooses random valid actions to represent low-quality schedules, while hyperparameters remain constant across all three datasets.The fixed hyperparameters are used to test robustness to reduced-quality datasets.Source: A. Behavioral policies · p. 4

V. AGENT TRAINING

• All three approaches use the same underlying network architecture, while their data, optimization procedures, training budgets, and hyperparameters differ.Source: §V · p. 4

A. Online RL training

• The online RL baseline trains with DQN on 100 fixed, uniformly distributed 15 × 15 instances with operation durations from 0 to 100.Source: A. Online RL training · p. 4

B. Offline and PORL

• Offline RL and PORL use target-specific datasets generated by stochastic behavioral policies rolled out on 50 instances sampled from the target distribution.Source: B. Offline and PORL · p. 4
• Standalone offline RL trains CQL-DQN from scratch, whereas PORL initializes from online-pretrained weights and fine-tunes on the same offline dataset.This setup compares learning exclusively from offline data with adapting a pretrained policy to the target distribution.Source: B. Offline and PORL · p. 4

1) Offline RL:

• Offline CQL-DQN is trained with fixed hyperparameters listed in Table III.Source: 1) Offline RL · p. 4, Table III · p. 5

2) Pretrained Offline RL:

• PORL uses the offline agent’s dataset and training-step budget, while its learning rate and CQL α change over time to reduce multiobjective optimization instability.Table V presents the PORL hyperparameter ramping.Source: 2) Pretrained Offline RL · pp. 4–5, Table V · p. 5

VI. EVALUATION PROTOCOL

• Each model uses five random seeds, and evaluation uses 100 new instances generated with five seeds different from the training seeds.Evaluation instances remain constant across approaches.Source: §VI · p. 5
• Performance is measured by average makespan and optimality gap, with the gap referenced to an exact solver’s optimum under a 300-second time limit.Source: §VI · p. 5

A. Benchmarking

• Optimality gap implicitly compares performance with the optimal solver; additional benchmarks are Priority Dispatching Rules and a PPO agent.Source: A. Benchmarking · p. 5
• The PPO benchmark was trained on uniformly distributed 20 × 20 JSSP instances, following Hoss, Link, and Klarmann [14].Source: A. Benchmarking · p. 5

VII. RESULTS

• PORL outperforms standard offline CQL across heuristic, noisy-expert, and random dataset configurations, with its advantage widening particularly for noisy-expert and fully random data.Table VII reports makespan optimality gaps for the agents across dataset compositions.Source: §VII · p. 5, Table VII · p. 6
• 7.5% lower optimality gap than classical offline RL and 28.7% lower than a PPO agent trained on uniformly distributed instances; PORL also beats FDD/MWR and online RL baselines.The relative reductions are reported against classical offline RL and the PPO agent, respectively.Source: §VII · p. 5, Figure 2 · p. 6

VIII. DISCUSSION AND LIMITATIONS

• Under substantial distributional shift, results suggest simulator-acquired knowledge transfers to offline adaptation; broader state and action exposure may explain more informative initial values and representations.The authors describe this explanation as possible, not established causation.Source: §VIII · p. 6
• FDD/MWR outperforms online RL agents trained on uniform instances, while both offline RL and PORL outperform the heuristic baseline in the evaluated setting.The discussion characterizes dispatching rules as more robust to changing conditions because they are largely distribution-agnostic.Source: §VIII · p. 6
• Limiting divergence from the online pretrained policy helps maintain PORL's advantage over priority dispatching rules on two of three datasets as offline RL degrades faster with lower dataset quality.Source: §VIII · p. 6
• PORL has greater training instability than conventional offline RL, plausibly because DQN, CQL, and KL objectives can give conflicting signals when pretrained and behavioral policies differ.Loss scheduling and hyperparameter ramping partly mitigated the effect but added tuning complexity.Source: §VIII · p. 6
• Preliminary experiments without KL regularization showed catastrophic forgetting and optimality gaps reaching 25%, but a full ablation study was outside the work's scope.Source: §VIII · p. 6
• Evaluation is limited to synthetic 10 × 10 JSSP instances and one distribution-shift scenario; larger problems, real production data, and simulation-to-deployment mismatch remain open boundaries.The authors also leave KL strength and alternative policy constraints for further study.Source: §VIII · p. 6

IX. CONCLUSION

• The framework first learns a general policy through simulation-based online RL, then adapts it offline to production-specific data while a KL constraint limits deviation from the pretrained policy.Source: §IX · p. 7
• PORL consistently outperforms standalone offline RL, online RL baselines, and priority dispatching rules on JSSP instances under substantial distributional shift.Source: §IX · p. 7
• The results indicate that online pretraining provides prior knowledge for adaptation to specialized scheduling environments and reduces offline RL's dependence on dataset quality and coverage.Source: §IX · p. 7
• The authors identify larger-scale scheduling problems, real-world production datasets, and varied levels of distribution shift as future evaluation targets.Source: §IX · p. 7