Source-linked AI summary
OR-Transformer: Scaling Real-Time Decision-Making to 1,000 Items
Shuze Daniel Liu, David Simchi-Levi, Claire Chen, Chutong Gao, Shangtong Zhang
TL;DR
Large-scale stochastic joint replenishment is difficult because high-dimensional, coupled decisions combine shared fixed costs, correlated demand, and heterogeneous lead times. OR-Transformer uses a permutation-equivariant Transformer with pathwise-gradient training through inventory dynamics. Across up to 1,024 items, it increasingly outperforms learning-based and rolling-horizon MILP baselines while reducing online decision time by more than 4 million times relative to MILP solvers.
Problem
More than 1,000-item replenishment creates observation and action spaces exceeding 10^4 and 10^3 dimensions, while stochastic dependencies and shared fixed costs challenge rolling-horizon MILPs and standard RL credit assignment.
Method
OR-Transformer combines an item-permutation-equivariant Transformer with pathwise-gradient training through differentiable inventory dynamics for continuous order quantities.
Results
Across problem sizes up to 1,024 items, OR-Transformer increasingly outperforms learning-based and rolling-horizon MILP baselines and reduces online decision-making time by more than 4 million times relative to MILP solvers.
Takeaways & Limitations
The results suggest a path toward efficient, real-time decision-making for large-scale stochastic inventory systems.
Takeaways & Limitations
The formulation models correlated demand through an observed common stochastic factor and supplies full demand-specification details in Appendix B.
Abstract
from arXiv · showhide
Modern supply chain operations can require coordinating replenishment across thousands of heterogeneous items under correlated stochastic demand, heterogeneous lead times, and shared fixed ordering costs, yielding observation spaces exceeding $10^4$ dimensions. At this scale, rolling-horizon stochastic mixed-integer linear programs (MILPs) become prohibitively slow, while standard reinforcement learning (RL) methods face increasingly challenging credit assignment in high-dimensional action spaces. We introduce OR-Transformer, a deep reinforcement learning framework for joint replenishment under stochastic demand, with an item-permutation-equivariant Transformer architecture and pathwise-gradient training through the inventory dynamics. Across problem sizes up to 1,024 inventory items, OR-Transformer increasingly outperforms learning-based and rolling-horizon MILP baselines as scale grows. It also reduces online decision-making time by over 4 million times relative to MILP solvers, enabling real-time, large-scale deep RL in supply chain operations.
1 Introduction
Large-scale stochastic joint replenishment couples thousands of item decisions under shared costs, correlated demand, and heterogeneous lead times, challenging both optimization and learning. OR-Transformer addresses this setting and increasingly outperforms learning-based and rolling-horizon MILP baselines as problem size grows.
- $1.7 trillion annually, or about 6.2% of global retail sales, is attributed to retailers’ combined out-ofstocks and overstocks.
- Joint replenishment coordinates products under a shared fixed ordering cost, requiring decisions about which items to include and their quantities.
- More than 1,000 items create observation and action spaces exceeding 10^4 and 10^3 dimensions, respectively, while heterogeneous lead times and correlated demand add dependencies.
- OR-Transformer uses an item-permutation-equivariant Transformer and differentiable inventory dynamics to assign gradients directly to continuous order quantities.
- Approximately 0.35M discounted cost at 1,024 items represents reductions of 74.7%–90.8% versus learning baselines and about 96% versus rolling-horizon MILP controllers.
2 Problem Formulation
The paper formulates stochastic joint replenishment as discounted-cost control over inventories, outstanding orders, demand factors, and item-specific constraints. Each action jointly determines whether to order and how much to order for each item.
- Each item has holding cost, backlog cost, lead time, and maximum order quantity, while any joint order incurs shared fixed cost K.
- The state comprises net inventories, outstanding orders scheduled for future arrival, an observed common stochastic demand factor, and time-invariant instance parameters.
- The action is a binary order-opening decision Y_t together with item quantities Q_t constrained by 0 ≤ Q_i,t ≤ q̄_i.
- Orders arrive after item-specific lead times L_i, with outstanding orders tracked in the state.
- The objective is to minimize expected infinite-horizon discounted cost under the replenishment policy.
3 Method
OR-Transformer combines permutation-equivariant Transformer policies with hybrid gradient training for discrete order openings and continuous item quantities. Its architecture preserves item-order invariance while modeling cross-item dependencies.
- Architecture: Three independent Transformer networks produce the order-opening decision, item quantities, and a value estimate.
- Training: Score-based policy gradients train the discrete order-opening decision, while pathwise gradients train continuous quantities through differentiable inventory dynamics.
- Architecture: One global token and one token per item encode the state, while self-attention captures dependencies across item states.
- Permutation structure: Arbitrary item reordering leaves global outputs unchanged and reorders quantity outputs accordingly, without positional or item-index embeddings.
- Training: Pathwise gradients connect an order quantity to later costs through its effects on future inventory and outstanding orders, providing more direct credit assignment.
4 Experiments
Across correlated-demand replenishment problems from 1 to 1,024 items, OR-Transformer remains stable as learning baselines diverge at larger scales. At 1,024 items, it achieves lower cost than learning and rolling-horizon MILP baselines while making decisions substantially faster than Gurobi.
- Online decision quality and computation: At 1,024 items, OR-Transformer achieves 19.1% lower inventory cost than Gurobi when Gurobi receives up to six hours per decision.
- Scaling performance: OR-Transformer remains stable through 1,024 items, while PPO, HPO, and Transformer-PPO diverge at 4, 16, and 64 items, respectively.
- Scaling performance: Approximately 0.35M discounted cost versus 1.39M for the best learning baseline yields an approximately 75% reduction at 1,024 items.
- Online decision quality and computation: 0.0432 seconds versus 48 hours 49 minutes makes OR-Transformer more than four million times faster in online decision-making than Gurobi.
5 Conclusion
OR-Transformer is presented as a framework for large-scale stochastic joint replenishment, combining permutation-equivariant modeling with pathwise-gradient training. Across problems up to 1,024 items, it increasingly outperforms learning and MILP baselines while substantially reducing online decision time.
- OR-Transformer combines an item-permutation-equivariant Transformer architecture with pathwise-gradient training through inventory dynamics.
- More than 4 million times faster online decision-making than MILP solvers supports real-time control at large scale.
- The formulation includes correlated demand, heterogeneous lead times and costs, shared setup costs, backlogging, and an infinite-horizon discounted objective.
- The state includes net inventory, in-transit quantities, observed market factors, and fixed instance parameters that remain constant along each trajectory.
- The hybrid action consists of a binary order-opening decision and item-level proposed quantities, with executed quantities given by O_i,t = Y_tQ_i,t.
C Detailed OR-Transformer Method
OR-Transformer represents inventory items as an unordered set and uses separate Transformer-based networks for order opening, quantities, and value estimation. Its symmetry constraints preserve consistent behavior under item reordering while self-attention models cross-item dependencies.
- Architecture: Three separate networks parameterize order opening, continuous quantities, and the critic, allowing their outputs and training roles to differ.
- Permutation symmetry: Symmetric pooling leaves the global token unchanged when item indices are permuted.
- Architecture: The architecture uses one global token and one token per item, with full attention and no positional or item-index embeddings.
- Hybrid policy: The complete hybrid policy factorizes into a Bernoulli order-opening distribution and a deterministic proposed-quantity vector.
- Permutation symmetry: The order-opening probability and critic value are permutation invariant, while the proposed-quantity vector is permutation equivariant.
C.3 Hybrid pathwise training
Hybrid pathwise training separates optimization of the discrete order-opening decision from the continuous quantity decision. Conditional on sampled discrete actions and exogenous realizations, gradients propagate through quantities, inventory transitions, and future costs.
- Discrete decision: The sampled binary order-opening decision blocks ordinary end-to-end differentiation, so its network uses a score-based gradient.
- Continuous quantities: Pathwise gradients update the quantity network while holding sampled order openings, demands, and factor innovations fixed.
- Continuous quantities: The quantity gradient propagates through every proposed quantity, inventory transition, and future rollout cost.
- Truncated rollouts: A frozen critic supplies terminal-value information beyond the truncated rollout while preserving gradients through the terminal state.
- Training objective: The complete objective combines pathwise, critic-regression, and entropy terms, while detached simulator states implement truncated backpropagation through time.
D.1.1 Method and training configuration
The appendix specifies OR-Transformer’s architecture, hybrid training procedure, experimental protocol, and rolling-horizon MILP formulation. It also defines scenario construction and inventory dynamics for the MILP baseline.
- Architecture and training: OR-Transformer uses three independent Transformer encoders for order opening, order quantities, and the critic.The order-opening decision uses a score-based estimator, while continuous quantity gradients propagate pathwise through differentiable inventory dynamics.
- Architecture and training: Pathwise training treats sampled order openings and exogenous realizations as constants while propagating gradients through simulator transitions and future costs.Each transition includes exogenous sampling, cost evaluation, normalization, and inventory dynamics.
- Experimental protocol: The evaluation protocol uses a fixed held-out set of 128 episodes for all methods.Table D.2 contains the environment setup, baseline architectures, and training and evaluation protocol for Figure 1.1.
- MILP formulation: The rolling-horizon MILP conditions on the observed state, constructs demand scenarios over a planning horizon, and executes only the common root action.Item-specific lead times determine when in-transit orders become available, while the shared root action is enforced across scenarios.
- MILP formulation: The MILP models correlated demand through sampled common-factor paths and item-specific demand generation, with sampled demands treated as data in the deterministic equivalent.The factor innovations are sampled independently across scenarios and look-ahead steps.
D.3 Full scaling results
OR-Transformer’s cost advantage increases with problem size, reaching approximately 75% lower cost than the best learned baseline and 96% lower cost than Gurobi at 1,024 items. At that scale, both MILP solvers hit their time limits and return the same no-order trajectory rather than a common optimized solution.
- Cost comparison: At 1,024 items, OR-Transformer reduces cost by approximately 75% relative to the best learned baseline and 96% relative to Gurobi.The comparison covers a 50-decision evaluation horizon.
- Cost comparison: OR-Transformer achieves the lowest cost from 16 items onward, while HPO is best at one and four items.At 64 items, OR-Transformer reduces cost by approximately 20% relative to Gurobi.
- Scaling stability: Several learned baselines diverge as problem size increases, indicating difficulty maintaining stable learning at scale.The reported divergence occurs at different problem sizes across the learned methods.
- Solver boundary: At 1,024 items, both solvers reach the 10-minute time limit for all 6,400 evaluated decisions and return zero replenishment quantities.The corresponding MILP contains approximately 20 million columns, so identical solver costs reflect a common no-order trajectory rather than common optimized solutions.