PaperlayerView exact revision on arXiv

Source-linked AI summary

Integrated Optimization of Automated Warehouse Operations and Last-Mile Transport for Differentiated On-Demand Delivery

Xiaozhu Sun, Bilal Farooq

arXiv:2609.19048v1cs.LG

TL;DR

Warehouse and last-mile decisions are difficult to coordinate amid dynamic urban logistics, and traditional decompositional or single-objective approaches do not synchronize cross-stage information. This paper integrates multi-objective reinforcement learning for warehouse scheduling with heterogeneous fleet routing; its joint strategy outperforms FIFO and historical strategies across both phases, improving delivery times and high-priority service.

  • Problem

    Traditional decompositional or single-objective methods do not synchronize internal and external information or capture cross-stage influences in integrated warehouse and last-mile optimization.

    Source: §7 · p. 33

  • Method

    The framework combines multi-objective reinforcement learning in a hierarchical decision structure with heterogeneous vehicle routing to jointly optimize warehouse scheduling and last-mile delivery.

    Source: §1 · p. 2, §7 · p. 33

  • Results

    The integrated strategy outperforms FIFO and historical strategies across warehousing and transportation, reducing delivery times and improving high-priority service.

    Source: §7 · p. 34

  • Takeaways & Limitations

    Cross-decision-level coordination using information from inside and outside the warehouse is crucial for system performance, particularly in high-density and time-sensitive logistics scenarios.

    Source: §7 · p. 34

    The framework relies on regional traffic information and historically predefined vehicle characteristics, which may not reflect real-time operational volatility.

    Source: §7 · p. 34

Abstract

from arXiv · show

In the context of differentiated on-demand goods delivery services, this study proposes an integrated optimization method for automated guided vehicles (AGVs) based smart warehouse operations and the last-mile multi-modal transport. A deep reinforcement learning algorithm for multi-objective joint scheduling is designed to establish a dynamic connection between two systems, solving key challenges such as achieving high-throughput continuous order scheduling, meeting competing requirements, and improving the overall system sensitivity and adaptability. For warehouse optimization within this framework, we propose an improved algorithm based on multi-objective, Multi-Reward Machines-A* Guided Deep Q-Network (MORM-AGDQN), which combines service level, system cost, and external transportation demand. For external optimization, we propose an improved algorithm based on a Multi-Reward, Multi Head attention-Heterogeneous Capacity Vehicle Routing Problem (MRMH-HCVRP) framework, which incorporates the optimized scheduling order sequence and grouping, combined with customer location, demand, and priority, vehicle capacity, speed, and service range. The results show that the proposed framework significantly outperforms traditional methods, achieving 100% on-time delivery rate for warehousing operations. After joint optimization, the average delivery time for the last mile was reduced by 29.3% to 53.2%, the total transportation distance was reduced by 46.4%, the high-priority service rate was increased to over 92%, and a balance was maintained between operating costs and customer satisfaction.

1. Introduction

The paper addresses the disconnect between warehouse operations and last-mile transport, which can lead to suboptimal performance under dynamic constraints. It proposes a coordinated deep reinforcement learning framework and a dual-network architecture for multi-objective logistics optimization.

1. Introduction

  • E-commerce growth and demand for same-day or instant delivery have made warehouse efficiency, cost, service priorities, and tight deadlines central design concerns.AGV applications are concentrated in indoor warehousing environments, according to the cited work.Source: §1 · p. 1
  • Warehouse operations and last-mile delivery are deeply interconnected but traditionally optimized independently, risking suboptimal global performance under dynamic constraints.The paper identifies scalability challenges for traditional mathematical approaches under stochastic demand and heterogeneous fleets.Source: §1 · p. 2
  • The proposed framework synchronizes warehouse AGV scheduling with heterogeneous last-mile routing and balances safety, energy, traffic, and customer-priority objectives.Its dual-network architecture combines CNNs, multi-head attention, and reward machines.Source: §1 · p. 2

2. Related work

The review covers AGV scheduling and routing, vehicle routing outside warehouses, and integrated optimization. Across these areas, the paper identifies a need for adaptive, multi-objective methods that coordinate warehouse and delivery decisions.

2. Related work

  • The review organizes prior work around AGV task allocation and path optimization, external vehicle routing, and integrated optimization.These are the three literature strands identified by the authors.Source: §2 · p. 2

2.1. Application of RL in AGV scheduling and path optimization in warehouse

  • Prior warehouse studies address scheduling, routing, and multi-objective AGV control, but multi-load settings and explicit objective trade-offs remain underdefined.The review also notes limitations in prior methods, including static layouts or small maps and fleets.Source: §2.1 · p. 3

2.2. Application of RL in vehicle routing optimization external the warehouse

  • External-routing research uses DRL to adapt to traffic, vehicle state, and demand, while scalability, training complexity, and heterogeneous-fleet coordination remain challenges.The review identifies integrated warehouse and external-routing frameworks as a response to unresolved multi-objective and coordination challenges.Source: §2.2 · pp. 3–4

2.3. Integrated Process Optimization

  • Integrated optimization can improve coordination and resource use, but many existing approaches rely on deterministic or heuristic methods rather than adaptive reinforcement learning.Some reviewed work connects warehouse operations with last-mile delivery, while other studies address only internal processes.Source: §2.3 · pp. 4–5

3. Problem Statement

The problem is framed as two connected stages: smart-warehouse processing and external vehicle routing and delivery. The described workflow and fleet grouping set the scope for the integrated optimization problem.

3. Problem Statement

  • The study targets delivery settings such as e-commerce smart warehouses and urban warehouse-to-client services, dividing the process into internal warehouse operations and external vehicle routing.Figure 1 is identified as depicting the integration of warehouse operations and last-mile delivery.Source: §3 · p. 5, Figure 1 · p. 5

3.1. Internal Components and Workflow

  • The warehouse workflow moves from order reception and batching through scheduling, AGV assignment and picking, then verification, packaging, and dispatch.The system includes multi-capacity AGVs coordinated by a warehouse management system.Source: §3.1 · pp. 5–6, Figure 2 · p. 6

3.2. External Vehicle Routing and Delivery

  • External routing matches vehicles with different speeds, capacities, and mileage to delivery clusters, while grouping fleets by service distance, capacity, and speed.The stated scope excludes complex variables such as EV charging and vehicle leasing, deferring fully hybrid fleet scenarios.Source: §3.2 · p. 6, Figure 3 · p. 7

3.3. Key Notations and Definition

  • Tables 1 and 2 list the key notations and definitions used for the integrated warehouse and last-mile optimization problem.Source: §3.3 · p. 6

4. Methodology

The methodology couples internal AGV scheduling with external fleet routing in a multi-objective framework. It represents dynamic objectives over time and accounts for operational trade-offs in both warehouse and last-mile systems.

4. Methodology

  • A unified multi-objective reinforcement-learning framework links internal AGV scheduling and external fleet routing through dynamic reward mapping, neural networks, and attention heads.Source: §4 · p. 7

4.1. Multi-objective Optimization Function

  • The framework represents objectives over decision vectors and time, updating the current optimum between time windows.The running time is divided into steps, with each window’s summed objective carried into the next window.Source: §4.1 · p. 7

4.1.1. Multi-Objectives for Internal Warehouse System

  • Internal AGV objectives address cost, service quality, and safety, including energy use, ABC-class waiting inventory, external delivery influences, and collision or deadlock avoidance.Source: §4.1.1 · pp. 8, 10, 11

4.1.2. Multi-Objectives for External Multimodal Delivery System

  • External routing balances fleet allocation, operating costs, timely priority service, and travel time across vehicles with differing speeds and dynamic traffic conditions.Source: §4.1.2 · pp. 12–13

4.2. Dual-Architecture Mechanism

This section describes a dual architecture linking warehouse AGV operations with external heterogeneous-capacity vehicle routing through reinforcement learning, while accounting for time, region, and vehicle type in last-mile travel efficiency and using a multiplicative reward to satisfy reward components simultaneously.

4.2. Dual-Architecture Mechanism

  • The framework models warehouse scheduling and path planning alongside external HCVRP routing as a Dual-Architecture MDP for dynamic information exchange.The two processes are connected so information and influencing factors can function across both systems.Source: §4.2 · p. 13

4.2.1. RL Architecture for Internal Warehouse System

  • The internal cooperative AGV MDP combines warehouse conditions, vehicle status, order details, and downstream customer priority, with deterministic movement and state updates.Actions include moving forward, backward, left, right, or staying; updates track battery use, order completion, and AGV positions for collision-free navigation.Source: §4.2.1 · pp. 13–14
  • Four internal reward machines separately track energy consumption, inventory management cost, external influence, and safety/stability.External-priority rewards decay exponentially by Pareto-front index, ensuring orders in higher-priority fronts receive greater rewards than those in lower-priority fronts.Source: §4.2.1 · p. 15

4.2.2. RL Architecture for Last-Mile Delivery on External Multimodal System

  • The external fleet state combines vehicle attributes, customer demand and priority, traffic conditions, and the periodically refreshed warehouse order sequence.New orders are appended every five minutes; transitions update vehicle, customer, traffic, and sequence states while enforcing capacity, distance, time-window, and deadline feasibility.Source: §4.2.2 · pp. 17–18
  • External routing rewards combine priority satisfaction, transportation-time efficiency, and congestion adaptability.The priority component reflects responsiveness to high-priority customers, while the other components measure transit time and the effect of current traffic conditions.Source: §4.2.2 · p. 18
  • Travel efficiency accounts for time period, region, and vehicle type, with congestion represented by regional influence, congestion propagation, and a penalty coefficient.Time-period factors distinguish normal and busy periods; regional factors distinguish metropolitan, urban, and suburban areas.Source: §4.2.2 · p. 18
  • The time-window total reward is the product of individual components, so a higher reward requires their simultaneous satisfaction for multi-objective optimization.Source: §4.2.2 · p. 18

4.3. Interactions in Integrated Internal/External Systems

The framework links warehouse scheduling and path planning with external fleet allocation and routing. It uses Pareto optimization to represent trade-offs and translate external factors into warehouse picking sequences.

4.3. Interactions in Integrated Internal/External Systems

  • Warehouse path planning considers external factors, while last-mile allocation and routing use the warehouse order sequence.The paper describes this as an interactive bridge between the two systems.Source: §4.3 · p. 19
  • A 3D Pareto front translates external factors into time-based AGV picking sequences that serve as neural-network inputs.The paper notes that high-dimensional Pareto fronts can be computationally expensive and difficult to interpret.Source: §4.3 · p. 19
  • Linear programming is used for linear objectives, while Pareto optimization addresses nonlinear or non-convex objectives without convexity assumptions.The examples given are energy and inventory costs for linear optimization, and external influences and the final multi-objective function for Pareto optimization.Source: §4.3 · p. 19

4.4. Framework of Integrated Optimization

The integrated framework combines an A*-guided, multi-objective AGV controller for warehouse operations with an attention-based heterogeneous fleet routing policy. The warehouse network encodes operational context, and the external network selects vehicles and customers using routing and service features.

4.4.1. Inner Warehouse Framework

  • MORM-AGDQN combines multichannel state representation, hierarchical feature extraction, A*-guided decisions, and multi-objective reward optimization for warehouse AGVs.The hybrid is designed to navigate dynamic warehouse environments while optimizing competing objectives.Source: §4.4.1 · p. 19
  • The warehouse state is a four-channel spatial grid encoding infrastructure and AGVs, external constraints, AGV status, and order attributes.The representation is intended to encode operational context and its relation to decisions.Source: §4.4.1 · pp. 19–21
  • A*-recommended paths guide action selection, while prioritized experience replay and policy regularization support learning around the A* policy.The framework also decays exploration rates during training while retaining a bias toward A*-guided actions.Source: §4.4.1 · pp. 21–22

4.4.2. External Vehicle Routing Framework

  • The external policy uses a transformer encoder with attention heads for transportation time, service priority, and regional traffic delay.Its encoding jointly represents proximity, priorities, temporal constraints, capacity use, efficiency, and traffic patterns.Source: §4.4.2 · pp. 22–23
  • Vehicle and customer decoders assign customers to vehicles and select the next customer, accounting for fleet status and multi-objective compatibility.Masked softmax excludes already served customers, and capacity-aware decisions use demand normalized by vehicle capacity.Source: §4.4.2 · pp. 22, 24, 25

5. Case Study and Simulations

The case study uses Amazon last-mile delivery records and a discrete-event simulation. Three scenarios vary demand, traffic, fleet conditions, and disruptions to assess the integrated system.

5.1. Data Description

  • Amazon delivery records provide customer and warehouse locations, delivery timing, order and customer characteristics, and operational timestamps.The paper uses the dataset as its empirical base.Source: §5.1 · p. 25

5.2. Simulation Scenarios

  • The simulation uses Python discrete-event principles, Google Maps geographic data, historical traffic patterns, and autonomous agent-based decision-making.Source: §5.2 · p. 25
  • Three scenarios test normal operations, peak-season surges of 2 to 5 times regular order volumes, and uncertainty from congestion, emergency orders, and special days.The scenarios are intended to assess scalability, efficiency, and resilience.Source: §5.2 · p. 25

6. Results

The results examine Pareto-front optimization, compare warehouse and last-mile methods, and assess task allocation and delivery KPIs. They report gains in routing efficiency, priority service, and integrated delivery performance.

6.2.1. Inner Warehouse Algorithm Comparison

  • Over 1,000 episodes, MORM-AGDQN converged smoothly to higher rewards than PG, DQN, AGDQN, and MO-AGDQN, whose trajectories were lower or less stable.Its reward fluctuations narrowed after episode 600; larger scenarios reached higher cumulative rewards but required longer training and showed more mid-training fluctuation.Source: §6.2.1 · p. 26, Figure 10 · p. 28

6.1. Pareto Fronts Optimization

  • Figure 9 traces orders from goods categories through traffic and service classes to Pareto solutions, with stream widths encoding order volumes.The diagram describes a funnel from diverse conditions to the multi-objective decision space.Source: §6.1 · p. 26, Figure 9 · p. 27

6.2.1. Inner Warehouse Algorithm Comparison

  • After 10,000 rounds, models trained on 17- and 96-order datasets both completed order tasks, while large-batch training yielded higher average rewards per order.Rewards eventually plateaued rather than increasing indefinitely with scale.Source: §6.2.1 · p. 27, Figure 11 · p. 28

6.2.2. External Vehicle Routing Algorithm Comparison

  • Distance-aware vehicle allocation reached optimal solutions by iterations 44–49; transport costs fell 44.2% for scooters (24 orders), 42.6% for motorcycles (30 orders), and 63.6% for vans (48 orders).The allocation matches vehicle capacity, speed, and battery limits to delivery ranges; a rolling baseline updates only after significant improvement on an independent test set (p < 0.05).Source: §6.2.2 · pp. 28–29, Figure 12 · p. 29
  • MRMH-HCVRP achieved 272.67 km and 3.41 h, versus Time Sequence & Clustering’s 408.13 km and 5.10 h, while attaining 92% rapid delivery and 93.62% priority first ratio.It also beat HCVRP’s 390.45 km and 4.88 h on distance and time, and slightly undercut Shortest Path & Clustering’s 279.43 km distance.Source: Table 3 · p. 30, §6.2.2 · p. 30

6.3.1. Task Sequence and Allocation

  • During peak hours, targeted weight design improved grouping, pickup sequencing, and vehicle allocation for high-priority customers while managing congestion, complex routing, and dense orders.Baseline models performed adequately during low-volume periods when resources were easier to allocate.Source: §6.3.1 · pp. 30, 32

6.3.2. Key KPIs Comparison

  • Table 4 reports 100% on-time departure and pickup, 58-minute average delivery, 209-km distance (46.4% saving versus FIFO), 97.06% on-time delivery, and 92.31% high-priority satisfaction.Integrated average order completion time fell 27.5% to 66.38 minutes; warehouse high-priority scheduling reached 84.62%.Source: Table 4 · p. 33, §6.3.2 · p. 33

7. Conclusions

The study presents an integrated multi-objective approach to warehouse scheduling and last-mile delivery, linking hierarchical reinforcement learning with heterogeneous vehicle routing. Its reported findings include improved performance across both phases, with limitations around the use of regional traffic information and historically defined vehicle characteristics.

7. Conclusions

  • The framework jointly addresses warehouse scheduling and last-mile delivery through hierarchical multi-objective reinforcement learning and heterogeneous vehicle routing.It is presented as overcoming limitations of decompositional or single-objective approaches that fail to synchronize internal and external information and capture cross-stage influences.Source: §7 · p. 33
  • MORM-AGDQN uses structured multi-objective reward shaping to stabilize learning while remaining sensitive to competing operational objectives.Compared with similar reinforcement learning algorithms, it achieves smoother convergence, higher cumulative rewards, and better scalability, particularly under high dynamic demand and multi-agent collaboration.Source: §7 · pp. 33–34
  • MRMH-HCVRP combines distance- and traffic-aware heterogeneous fleet allocation with priority-sensitive service, reducing travel distance and delivery time while improving differentiated service.Compared with traditional route planning, it balances operating costs and customer responsiveness.Source: §7 · p. 34
  • KPI analysis finds the integrated strategy outperforms FIFO and historical strategies in both warehousing and transportation.The study reports near-perfect just-in-time warehousing, reduced delivery times, and improved high-priority service rates.Source: §7 · p. 34
  • The authors recommend prioritizing high-value or time-sensitive orders and deploying heterogeneous fleets according to delivery range and capacity constraints.They also identify synchronizing upstream and downstream decisions as a way to support adaptive, resilient operations under demand uncertainty and congestion.Source: §7 · p. 34
  • The framework relies on regional traffic information and historically defined vehicle characteristics, which may not adequately reflect real-time operational volatility.The authors also note that exact optimization is computationally prohibitive at large scale because the problem is NP-hard.Source: §7 · p. 34

A.1 Algorithm for Warehouse Operation

The warehouse algorithm initializes agent networks, a CNN feature extractor and guider, replay storage, and reward machines, then interacts with the environment and updates the learned policies.

A.1 Algorithm for Warehouse Operation

  • The algorithm initializes per-agent Q-networks and target networks, CNN feature extraction and guidance, a replay buffer, and reward machines.The listed reward machines cover environmental, internal, external, and safety rewards.Source: §A.1 · p. 39
  • At each step, agents select actions using Q-values augmented by reward-machine rewards, execute them, and observe the next state and rewards.Reward-machine states are updated from the action and next state; the transition and reward-machine states are stored in the replay buffer.Source: §A.1 · p. 39
  • Training samples mini-batches from the replay buffer and updates agent and target networks.The algorithm description specifies updates to the Q-network parameters and target networks.Source: §A.1 · p. 39

A.2 Algorithm for Last-Mile Vehicle Routing

The routing algorithm encodes orders, fleet, and traffic information, then selects feasible vehicle-customer actions and trains a policy and value network from replayed transitions.

A.2 Algorithm for Last-Mile Vehicle Routing

  • The routing encoder combines order sequence, time, and location with fleet type, capacity, location, speed, and state, plus traffic congestion, delays, and routes.Multi-head attention is applied to the encoded representation.Source: §A.2 · p. 40
  • The decoder selects a vehicle using its representation, traffic, and feasibility, then scores customers using compatibility and masks infeasible choices.Customer feasibility includes demand within vehicle capacity and feasible delivery time.Source: §A.2 · p. 40
  • Training uses a policy network, a value network, and replayed batches to compute a temporal-difference target and update the networks.The algorithm description computes squared value loss and updates parameters by gradient descent.Source: §A.2 · p. 40
  • Each routing action selects a vehicle and customer, and the reward combines travel time, travel cost, and priority.The resulting transition is stored in replay memory for training.Source: §A.2 · p. 40

Related papers

Loading 2609.19048v1…