Source-linked AI summary
AI-Driven Digital Twin Framework for Warehouse Logistics Automation Optimization
Md Mahinur Alam, Kanita Jerin Tanha, Md Tanjin Alam, Shadman Sakib, Min Gyu Kim
TL;DR
Static placement can be inadequate amid volatile warehouse demand and service constraints, while many RL approaches remain disconnected from high-fidelity digital twins. This paper integrates a DQN agent with a FlexSim digital twin through AAS; against static slotting, it reports lower picking time, higher throughput, and shorter travel distance.
Problem
Static slotting and rule-based placement are inadequate for sustaining low picking time and high throughput under volatile demand and service constraints, while most cited RL solutions operate in isolated simulations.
Source: §I · p. 1Method
The framework couples a DQN-based dynamic-slotting agent with a FlexSim digital twin through AAS middleware for real-time decisions and standardized asset interaction.
Source: §I · pp. 1–2, §II · p. 2Results
29.66% reduction in average picking time (169.22 to 118.32 s), 35.42% increase in throughput (23.35 to 31.62 orders/hour), and 18.54% decrease in travel distance versus static slotting.
Source: A. Overall DQN Agent Performance · p. 3Takeaways & Limitations
The agent learns demand-aware, distance-efficient placement strategies, with reported gains in picking time, throughput, and travel distance.
Source: §IV · p. 4
Abstract
from arXiv · showhide
In recent years, warehouse logistics optimization has increasingly focused on intelligent, data-driven systems to enhance operational efficiency. Conventional static product placement strategies are being progressively replaced by dynamic slotting approaches that can adapt to real-time variations in warehouse conditions. In this work, we present a novel framework that employs a reinforcement learning agent integrated within a FlexSim-based Digital Twin environment for real-time product placement optimization. The proposed agent continuously learns from streaming operational data, enabling it to reduce picking time while improving overall system throughput. The simulation environment models a four-zone warehouse layout with forklift-based material handling and leverages the Asset Administration Shell framework to ensure seamless interaction between the learning agent, physical asset, and the digital twin. Experimental evaluation demonstrates substantial performance gains, with the proposed approach achieving a 29.66\% reduction in average picking time, a 35.42\% increase in throughput, and an 18.54\% decrease in travel distance.
I. INTRODUCTION
Static slotting struggles to accommodate volatile demand and service constraints, while existing learning-based approaches often lack integration with high-fidelity digital twins. The paper introduces a DQN-based dynamic-slotting framework linked to a FlexSim digital twin through AAS.
I. INTRODUCTION
- Static slotting and rule-based placement are described as inadequate for sustaining low picking time and high throughput under volatile demand, labor shortages, and tight service constraints.Digital twins offer an updatable virtual replica for monitoring and safe evaluation without interrupting physical workflows.Source: §I · p. 1
- Prior MILP, genetic-algorithm, and pathfinding approaches offer coordination or task-time benefits but are limited by static formulations or computational overhead.The passage notes these limits constrain real-time adaptability to changing warehouse conditions.Source: §I · p. 1
- Most cited RL warehouse solutions operate in isolated simulations, limiting continuous reflection of real warehouse states and practical deployment.The related work describes PPO, DQN, and other deep RL applications for adaptive slotting, task assignment, shelf reallocation, and travel-distance reduction.Source: §I · p. 1
- The proposed framework couples a DQN agent to a FlexSim digital twin through AAS middleware for real-time dynamic slotting and standardized asset interaction.Its contributions include a demand-aware zone-assignment policy, a CNN–BiLSTM–Attention Q-network, and safe shadow evaluation before deployment.Source: §I · pp. 1–2
II. PROPOSED FRAMEWORK
The framework links a reinforcement-learning agent and FlexSim digital twin through AAS middleware. Physical assets are mirrored and synchronized, while slotting decisions are evaluated in the digital environment before deployment.
II. PROPOSED FRAMEWORK
- AAS mirrors IoT-instrumented physical assets into the digital twin and standardizes asset identification, state representation, and control interfaces.This supports bidirectional synchronization between the physical facility and its simulated counterpart.Source: §II · p. 2
- The reinforcement-learning agent operates within the digital-twin environment and issues dynamic slotting decisions.The agent’s actions are translated into adaptive reslotting plans for deployment after evaluation in simulation.Source: §II · p. 2
- The framework evaluates slotting decisions in simulation before deploying them as adaptive reslotting plans.The digital twin and agent are coupled through an AAS middleware layer.Source: §II · p. 2
A. RL Agent Formulation
The agent observes warehouse state and assigns candidate items to zones, using action masking to keep choices operationally feasible. Its reward balances picking-time savings, relocation cost, and zone-level KPI changes, while DQN training uses replay and a target network.
A. RL Agent Formulation
- At each step, the agent encodes warehouse state and selects a zone assignment from A, B, C, or D for a candidate item.The state includes order ID, rack number, pick time, and stock level.Source: A. RL Agent Formulation · p. 2
- Invalid zone actions receive Q-values of −∞, preventing assignments to full-capacity zones or zones exceeding a stock threshold.This constrains exploration to operationally feasible placements.Source: A. RL Agent Formulation · p. 2
- The reward combines picking-time savings, relocation cost, and change in zone-level KPI, with λ=0.3 and µ=0.2 selected by grid search.The KPI aggregates normalized fulfillment rate, stockout rate, turnover, and holding cost; picking time is prioritized as the dominant signal.Source: A. RL Agent Formulation · p. 2
- DQN training uses a periodically synchronized target network and minibatch temporal-difference targets with mean-squared loss.The supplied training description specifies Adam, replay, exploration decay, and stability measures across 2,000 episodes.Source: A. RL Agent Formulation · p. 2
B. Deep Q-Network
The Q-network combines convolutional, recurrent, and attention components to represent warehouse-state features across space and time. Attention-weighted context is projected into values for the four zone actions.
B. Deep Q-Network
- A 1D-CNN extracts local spatial patterns among warehouse state features such as rack occupancy, stock levels, and picking frequency.The resulting feature maps are passed to the BiLSTM.Source: B. Deep Q-Network · p. 2
- The BiLSTM produces forward and backward hidden representations that encode past operational trends and anticipated future demand signals.Its gating mechanisms include forget, input, cell, and output components.Source: B. Deep Q-Network · p. 2
- Bahdanau attention weights state transitions before a context vector is projected into Q-values for all four zone actions.The passage says the mechanism lets the network weight transitions where demand shifts or congestion events influence slotting decisions.Source: B. Deep Q-Network · p. 2
C. Digital Twin Closed-Loop Integration
The DQN agent sends zone-assignment actions through an executor to FlexSim via AAS, then uses operational traces to update its policy. The supplied figure compares average picking time across Q-network variants.
C. Digital Twin Closed-Loop Integration
- The agent’s zone-assignment action is translated into FlexSim simulator commands via AAS.FlexSim advances the epoch as part of the interaction loop.Source: C. Digital Twin Closed-Loop Integration · p. 2
- Table I is identified as the configuration parameters for the FlexSim digital-twin warehouse assets.Source: Table I · p. 3
- Fig. 3 compares average picking time across Q-network architecture variants.Source: Figure 3 · p. 3
- Operational traces feed reward and next-state computation; the resulting transition is stored for minibatch sampling and network updates.The traces include picking durations, travel distances, and queue states.Source: C. Digital Twin Closed-Loop Integration · p. 3
D. Warehouse Logistics Digital Twin Environment
The FlexSim digital twin models a four-zone warehouse with racks, transporters, queues, conveyors, and human operators. Operational traces support the agent loop and safe shadow evaluation before deployment.
D. Warehouse Logistics Digital Twin Environment
- The simulation represents four zones, 16 racks, and 500 SKUs, with four autonomous transporters, inbound and outbound queues, conveyors, two human operators, and a sink.Two Picker units retrieve products and two Placer units handle replenishment.Source: D. Warehouse Logistics Digital Twin… · p. 3
- Table II is a performance comparison averaged over five independent runs with different random seeds.Source: Table II · p. 3
- After each epoch, picking durations, transporter trajectories, and queue states feed reward computation and close the agent-environment loop.Source: D. Warehouse Logistics Digital Twin… · p. 3
E. Experimental Setup
Experiments used a FlexSim digital twin and a Python/PyTorch DQN agent, trained for 2,000 episodes. Evaluation metrics were averaged across independent runs with distinct random seeds.
E. Experimental Setup
- The DQN agent was trained for 2,000 episodes, each spanning a full order cycle across the four-zone warehouse.Source: E. Experimental Setup · p. 3
- The digital twin ran on FlexSim 25.1, and the agent was implemented in Python 3.10 using PyTorch 2.1.Source: E. Experimental Setup · p. 3
- Evaluation metrics were averaged over several independent runs with distinct random seeds to ensure reproducibility.Source: E. Experimental Setup · p. 3
III. PERFORMANCE ANALYSIS
The framework outperformed static slotting across picking time, throughput, and travel distance. The results were averaged across five runs with distinct random seeds.
A. Overall DQN Agent Performance
- 29.66% lower average picking time (169.22 to 118.32 s), 35.42% higher throughput (23.35 to 31.62 orders/hour), and 18.54% lower travel distance versus static slotting.The static baseline uses fixed zone assignment by SKU category.Source: A. Overall DQN Agent Performance · p. 3
- Low standard deviations across all metrics indicate policy stability across varying initializations.Results were averaged over five runs with distinct random seeds.Source: A. Overall DQN Agent Performance · p. 3
- The learned policy prioritizes high-frequency items near dispatch zones while internalizing spatial efficiency.Source: A. Overall DQN Agent Performance · p. 3
B. Q-Network Architecture Ablation
The full CNN-BiLSTM-Attention Q-network outperforms its ablated variants, while runtime analysis reports lower transporter travel, higher productive utilization, and increased source and sink throughput.
B. Q-Network Architecture Ablation
- The full CNN-BiLSTM-Attention architecture has the lowest median picking time and tightest interquartile range among the tested variants.Removing CNN increases learning variance; replacing BiLSTM with GRU reduces temporal modeling fidelity; removing attention weakens state-transition identification; replacing CNN with MLP performs worst overall.Source: B. Q-Network Architecture Ablation · p. 3
C. Runtime Operational Analysis
- Approximately 19% lower pertransporter travel across all four units accompanies source throughput rising from 6,629 to 8,543 units and sink throughput from 5,709 to 7,512 units.Fig. 4 compares the DQN-Agent with Static Slotting across travel distance, utilization, and throughput.Source: Figure 4 · p. 4, C. Runtime Operational Analysis · p. 4
- All four transporters exceed 98% productive utilization under DQN-Agent, while static slotting leaves them idle or repositioning for up to 40% of their time.The passage attributes the static-slotting inefficiency to poorly placed inventory forcing long empty traversals.Source: C. Runtime Operational Analysis · p. 4
IV. CONCLUSION
The conclusion presents a DQN dynamic-slotting framework integrated with a FlexSim Digital Twin and AAS for adaptive warehouse decisions. In simulation, it reports improved operational efficiency through demand-aware placement and reduced unnecessary transporter movement.
IV. CONCLUSION
- The study integrates DQN-based dynamic slotting with a FlexSim Digital Twin and standardized AAS for real-time adaptive decision-making and interoperable asset modeling.The framework is validated through simulation against traditional static slotting methods.Source: §IV · p. 4
- The conclusion reports that the DQN agent consistently outperforms traditional static slotting methods in simulation.The stated improvements concern operational efficiency, including picking speed, throughput, and travel distance.Source: §IV · p. 4
- The agent learns demand-aware, distance-efficient placement strategies that prioritize high-demand items and reduce unnecessary transporter movement and non-productive states.The conclusion reports faster picking, increased throughput, and reduced travel distances.Source: §IV · p. 4