PaperlayerView exact revision on arXiv

Source-linked AI summary

OR-Transformer: Scaling Real-Time Decision-Making to 1,000 Items

Shuze Daniel Liu, David Simchi-Levi, Claire Chen, Chutong Gao, Shangtong Zhang

arXiv:2609.01933v1cs.LG

TL;DR

Large-scale stochastic joint replenishment is difficult because high-dimensional, coupled decisions combine shared fixed costs, correlated demand, and heterogeneous lead times. OR-Transformer uses a permutation-equivariant Transformer with pathwise-gradient training through inventory dynamics. Across up to 1,024 items, it increasingly outperforms learning-based and rolling-horizon MILP baselines while reducing online decision time by more than 4 million times relative to MILP solvers.

  • Problem

    More than 1,000-item replenishment creates observation and action spaces exceeding 10^4 and 10^3 dimensions, while stochastic dependencies and shared fixed costs challenge rolling-horizon MILPs and standard RL credit assignment.

    Source: §1 · p. 2

  • Method

    OR-Transformer combines an item-permutation-equivariant Transformer with pathwise-gradient training through differentiable inventory dynamics for continuous order quantities.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Across problem sizes up to 1,024 items, OR-Transformer increasingly outperforms learning-based and rolling-horizon MILP baselines and reduces online decision-making time by more than 4 million times relative to MILP solvers.

    Source: §5 · p. 5

  • Takeaways & Limitations

    The results suggest a path toward efficient, real-time decision-making for large-scale stochastic inventory systems.

    Source: §5 · p. 5

    The formulation models correlated demand through an observed common stochastic factor and supplies full demand-specification details in Appendix B.

    Source: §2 · p. 2

Abstract

from arXiv · show

Modern supply chain operations can require coordinating replenishment across thousands of heterogeneous items under correlated stochastic demand, heterogeneous lead times, and shared fixed ordering costs, yielding observation spaces exceeding $10^4$ dimensions. At this scale, rolling-horizon stochastic mixed-integer linear programs (MILPs) become prohibitively slow, while standard reinforcement learning (RL) methods face increasingly challenging credit assignment in high-dimensional action spaces. We introduce OR-Transformer, a deep reinforcement learning framework for joint replenishment under stochastic demand, with an item-permutation-equivariant Transformer architecture and pathwise-gradient training through the inventory dynamics. Across problem sizes up to 1,024 inventory items, OR-Transformer increasingly outperforms learning-based and rolling-horizon MILP baselines as scale grows. It also reduces online decision-making time by over 4 million times relative to MILP solvers, enabling real-time, large-scale deep RL in supply chain operations.

1 Introduction

Large-scale stochastic joint replenishment couples thousands of item decisions under shared costs, correlated demand, and heterogeneous lead times, challenging both optimization and learning. OR-Transformer addresses this setting and increasingly outperforms learning-based and rolling-horizon MILP baselines as problem size grows.

  • $1.7 trillion annually, or about 6.2% of global retail sales, is attributed to retailers’ combined out-ofstocks and overstocks.Source: §1 · p. 1
  • Joint replenishment coordinates products under a shared fixed ordering cost, requiring decisions about which items to include and their quantities.Source: §1 · p. 2
  • More than 1,000 items create observation and action spaces exceeding 10^4 and 10^3 dimensions, respectively, while heterogeneous lead times and correlated demand add dependencies.Source: §1 · p. 2
  • OR-Transformer uses an item-permutation-equivariant Transformer and differentiable inventory dynamics to assign gradients directly to continuous order quantities.Source: §1 · p. 2
  • Approximately 0.35M discounted cost at 1,024 items represents reductions of 74.7%–90.8% versus learning baselines and about 96% versus rolling-horizon MILP controllers.Source: §1 · p. 2

2 Problem Formulation

The paper formulates stochastic joint replenishment as discounted-cost control over inventories, outstanding orders, demand factors, and item-specific constraints. Each action jointly determines whether to order and how much to order for each item.

  • Each item has holding cost, backlog cost, lead time, and maximum order quantity, while any joint order incurs shared fixed cost K.Source: §2 · p. 2
  • The state comprises net inventories, outstanding orders scheduled for future arrival, an observed common stochastic demand factor, and time-invariant instance parameters.Source: §2 · p. 2
  • The action is a binary order-opening decision Y_t together with item quantities Q_t constrained by 0 ≤ Q_i,t ≤ q̄_i.Source: §2 · p. 2
  • Orders arrive after item-specific lead times L_i, with outstanding orders tracked in the state.Source: §2 · p. 2
  • The objective is to minimize expected infinite-horizon discounted cost under the replenishment policy.Source: §2 · p. 2

3 Method

OR-Transformer combines permutation-equivariant Transformer policies with hybrid gradient training for discrete order openings and continuous item quantities. Its architecture preserves item-order invariance while modeling cross-item dependencies.

Architecture

  • Three independent Transformer networks produce the order-opening decision, item quantities, and a value estimate.Source: Figure 3.1 · p. 3, §3 · p. 3

Training

  • Score-based policy gradients train the discrete order-opening decision, while pathwise gradients train continuous quantities through differentiable inventory dynamics.Source: Figure 3.1 · p. 3, §3 · p. 3

Architecture

  • One global token and one token per item encode the state, while self-attention captures dependencies across item states.Source: §3 · p. 3

Permutation structure

  • Arbitrary item reordering leaves global outputs unchanged and reorders quantity outputs accordingly, without positional or item-index embeddings.Source: §3 · p. 3

Training

  • Pathwise gradients connect an order quantity to later costs through its effects on future inventory and outstanding orders, providing more direct credit assignment.Source: §3 · pp. 3–4

4 Experiments

Across correlated-demand replenishment problems from 1 to 1,024 items, OR-Transformer remains stable as learning baselines diverge at larger scales. At 1,024 items, it achieves lower cost than learning and rolling-horizon MILP baselines while making decisions substantially faster than Gurobi.

Online decision quality and computation

  • At 1,024 items, OR-Transformer achieves 19.1% lower inventory cost than Gurobi when Gurobi receives up to six hours per decision.Source: Figure 4.1 · p. 4, §4 · p. 4

Scaling performance

  • OR-Transformer remains stable through 1,024 items, while PPO, HPO, and Transformer-PPO diverge at 4, 16, and 64 items, respectively.Source: §4 · p. 4
  • Approximately 0.35M discounted cost versus 1.39M for the best learning baseline yields an approximately 75% reduction at 1,024 items.Source: §4 · p. 4

Online decision quality and computation

  • 0.0432 seconds versus 48 hours 49 minutes makes OR-Transformer more than four million times faster in online decision-making than Gurobi.Source: §4 · p. 4

5 Conclusion

OR-Transformer is presented as a framework for large-scale stochastic joint replenishment, combining permutation-equivariant modeling with pathwise-gradient training. Across problems up to 1,024 items, it increasingly outperforms learning and MILP baselines while substantially reducing online decision time.

  • OR-Transformer combines an item-permutation-equivariant Transformer architecture with pathwise-gradient training through inventory dynamics.Source: §5 · p. 4
  • More than 4 million times faster online decision-making than MILP solvers supports real-time control at large scale.Source: §5 · p. 5
  • The formulation includes correlated demand, heterogeneous lead times and costs, shared setup costs, backlogging, and an infinite-horizon discounted objective.Source: B Detailed Problem Formulation · pp. 9–10
  • The state includes net inventory, in-transit quantities, observed market factors, and fixed instance parameters that remain constant along each trajectory.Source: A Notation · p. 8, B Detailed Problem Formulation · p. 10
  • The hybrid action consists of a binary order-opening decision and item-level proposed quantities, with executed quantities given by O_i,t = Y_tQ_i,t.Source: A Notation · p. 8, B Detailed Problem Formulation · p. 10

C Detailed OR-Transformer Method

OR-Transformer represents inventory items as an unordered set and uses separate Transformer-based networks for order opening, quantities, and value estimation. Its symmetry constraints preserve consistent behavior under item reordering while self-attention models cross-item dependencies.

Architecture

Permutation symmetry

  • Symmetric pooling leaves the global token unchanged when item indices are permuted.Source: §C.1 · p. 11

Architecture

  • The architecture uses one global token and one token per item, with full attention and no positional or item-index embeddings.Source: §C.2 · p. 12

Hybrid policy

  • The complete hybrid policy factorizes into a Bernoulli order-opening distribution and a deterministic proposed-quantity vector.Source: §C.2 · p. 12

Permutation symmetry

  • The order-opening probability and critic value are permutation invariant, while the proposed-quantity vector is permutation equivariant.Source: §C.2 · p. 12

C.3 Hybrid pathwise training

Hybrid pathwise training separates optimization of the discrete order-opening decision from the continuous quantity decision. Conditional on sampled discrete actions and exogenous realizations, gradients propagate through quantities, inventory transitions, and future costs.

Discrete decision

  • The sampled binary order-opening decision blocks ordinary end-to-end differentiation, so its network uses a score-based gradient.Source: §C.3 · pp. 12–13

Continuous quantities

  • Pathwise gradients update the quantity network while holding sampled order openings, demands, and factor innovations fixed.Source: §C.3 · p. 13
  • The quantity gradient propagates through every proposed quantity, inventory transition, and future rollout cost.Source: §C.3 · p. 13

Truncated rollouts

  • A frozen critic supplies terminal-value information beyond the truncated rollout while preserving gradients through the terminal state.Source: §C.3 · p. 13

Training objective

  • The complete objective combines pathwise, critic-regression, and entropy terms, while detached simulator states implement truncated backpropagation through time.Source: §C.3 · pp. 13–14

D.1.1 Method and training configuration

The appendix specifies OR-Transformer’s architecture, hybrid training procedure, experimental protocol, and rolling-horizon MILP formulation. It also defines scenario construction and inventory dynamics for the MILP baseline.

Architecture and training

  • OR-Transformer uses three independent Transformer encoders for order opening, order quantities, and the critic.The order-opening decision uses a score-based estimator, while continuous quantity gradients propagate pathwise through differentiable inventory dynamics.Source: §D.1.1 · p. 14
  • Pathwise training treats sampled order openings and exogenous realizations as constants while propagating gradients through simulator transitions and future costs.Each transition includes exogenous sampling, cost evaluation, normalization, and inventory dynamics.Source: §D.1.2 · pp. 14–15

Experimental protocol

  • The evaluation protocol uses a fixed held-out set of 128 episodes for all methods.Table D.2 contains the environment setup, baseline architectures, and training and evaluation protocol for Figure 1.1.Source: §D.1.3 · p. 15

MILP formulation

  • The rolling-horizon MILP conditions on the observed state, constructs demand scenarios over a planning horizon, and executes only the common root action.Item-specific lead times determine when in-transit orders become available, while the shared root action is enforced across scenarios.Source: §D.2 · pp. 15–16
  • The MILP models correlated demand through sampled common-factor paths and item-specific demand generation, with sampled demands treated as data in the deterministic equivalent.The factor innovations are sampled independently across scenarios and look-ahead steps.Source: §D.2 · p. 16

D.3 Full scaling results

OR-Transformer’s cost advantage increases with problem size, reaching approximately 75% lower cost than the best learned baseline and 96% lower cost than Gurobi at 1,024 items. At that scale, both MILP solvers hit their time limits and return the same no-order trajectory rather than a common optimized solution.

Cost comparison

  • At 1,024 items, OR-Transformer reduces cost by approximately 75% relative to the best learned baseline and 96% relative to Gurobi.The comparison covers a 50-decision evaluation horizon.Source: §D.3 · p. 18
  • OR-Transformer achieves the lowest cost from 16 items onward, while HPO is best at one and four items.At 64 items, OR-Transformer reduces cost by approximately 20% relative to Gurobi.Source: §D.3 · p. 18

Scaling stability

  • Several learned baselines diverge as problem size increases, indicating difficulty maintaining stable learning at scale.The reported divergence occurs at different problem sizes across the learned methods.Source: §D.3 · p. 18

Solver boundary

  • At 1,024 items, both solvers reach the 10-minute time limit for all 6,400 evaluated decisions and return zero replenishment quantities.The corresponding MILP contains approximately 20 million columns, so identical solver costs reflect a common no-order trajectory rather than common optimized solutions.Source: §D.3 · p. 18

Related papers

Loading 2609.01933v1…