PaperlayerView exact revision on arXiv

Source-linked AI summary

CoRe-MARL: Cooperative Redistribution Under Unknown Dynamics Using Recurrent Multi-Agent Reinforcement Learning

Naimur Rahman Chowdhury, Shatabdi Sen Prapti, Md. Salehin Seyam, Limon Bin Hossain

arXiv:2609.18639v1cs.LGcs.AI

TL;DR

Relief networks face mismatches between local supply and need, while centers operate with limited information and evolving conditions. The paper uses recurrent cooperative multi-agent learning for redistribution; its reported comparisons show improved equity and worst-served-center service, with trade-offs in overall network service depending on the comparator.

  • Problem

    Unpredictable supply and shifting demand leave relief centers with local imbalances, while decision-makers have limited information about other centers and evolving conditions.

    Source: §1 · p. 3

  • Method

    The paper combines recurrent learning with cooperative multi-agent redistribution, using a shared reward for service equity and centralized training with decentralized decisions.

    Source: §1 · p. 4, §2 · p. 6, §3.3.1 · p. 11

  • Results

    MAPPO improves equity over the local-only heuristic, raising worst-center service from 0.2369 to 0.3562, while the heuristic can achieve higher overall network service.

    Source: §5.1 · p. 16, §6 · p. 17

  • Takeaways & Limitations

    The results indicate that cooperative learning can balance local service, reserves, and redistribution under limited and delayed information without observing true dynamics.

    Source: §6 · p. 17

    Redistribution is learned from reward without accounting for its costs, which may produce two-way transfers that incur an initial cost.

    Source: §6 · p. 17

Abstract

from arXiv · show

Emergency management assistance programs, such as relief distribution, are essential for delivering necessary supplies to affected communities. However, these programs operate in a decentralized network of local centers that face uncertain local demand and supply dynamics, resulting in inconsistent avail- ability of local services. Redistribution of supplies among these local centers reduces these imbalances, but the centers often make decisions independently, with limited information and disrupted transportation. This study develops CoRe-MARL, a cooperative multi-agent reinforcement learning (MARL) framework, by formulating a decentralized partially observable Markov decision process (Dec-POMDP). We treat each center as an agent that learns a redistribution policy to improve the service in the worst-case region and reduce the service gap across regions while protecting network-wide service. We incorporate a recurrent network that captures evolving supply and demand dynamics without direct observation, while multi-agent proximal policy optimization (MAPPO) enables centralized training and decentralized execution (CTDE). We evaluate the framework in a simulated environment with diverse trajectories, where exact dynamics are not observed by actors and the MAPPO critic. We compare the recurrent MAPPO with the recurrent independent PPO (IPPO) and a local only heuristic, and find that MAPPO reduces the service gap across local centers and enhances service for the worst-served center while maintaining competitive network-wide service. The recurrent MAPPO also shows consistent performance across diverse trajectory patterns, demonstrating its ability to adapt to evolving dynamics. The findings demonstrate the capability of cooperative learning for decentralized redistribution and improving equitable service under uncertain and evolving dynamics.

1. Introduction

Relief centers can hold surplus while others face shortages because supplies arrive unpredictably and disruptions weaken information about network conditions. The paper frames redistribution as cooperative, partially observable decision-making, using recurrent policies and a shared reward to address shifting dynamics and service imbalance.

1. Introduction

  • Unpredictable arrivals and disrupted roads or communications leave some centers with surplus while others face acute shortages, making redistribution important, especially for perishable supplies.Unused surplus may expire while another center lacks supplies.Source: §1 · p. 3
  • Centers have limited network information and face emergency trajectories that shift across locations and time, motivating cooperation under partial observability.A surplus center may not know which partner needs supplies, and a shortage center may not know which partner has stock.Source: §1 · p. 3
  • The paper models centers as cooperative agents that allocate supply among local service, reserves, and transfers, using recurrent belief states to infer evolving emergency phases.The recurrent state summarizes observation history without a direct signal of the trajectory.Source: §1 · p. 4
  • Recurrent MAPPO uses local observation histories and limited network information to allocate inventory dynamically without explicit knowledge of the trajectory.The study compares it with recurrent IPPO on the same trajectories to assess centralized training's effect on action performance.Source: §1 · p. 4
  • A shared reward penalizes service dispersion across centers and rewards minimum service, allowing evaluation of the equity–network performance trade-off across supply and demand realizations.The research question asks whether coordination can improve service for the worst-served communities without deteriorating overall network performance.Source: §1 · p. 4

2. Literature Review

Prior work addresses emergency resource allocation through centralized optimization and reinforcement learning, but fully decentralized coordination for redistribution under unknown dynamics has received insufficient attention. This study distinguishes itself by combining decentralized center decisions, equity-oriented rewards, recurrent learning, and explicit treatment of transport uncertainty, shipment losses, and perishability.

2. Literature Review

  • Prior work studies allocation efficiency and cost, supply adequacy under uncertainty, and equity; Gutjahr and Fischer (2018) add a Gini-coefficient correction to deprivation cost to represent inequity.The review also notes that minimal total deprivation cost alone does not adequately account for inequity.Source: §2 · pp. 4–5
  • Redistribution studies include centralized inventory models and forecast-based prepositioning, whereas this paper studies centers deciding autonomously from local observations and limited knowledge of others' stocks and demands.Examples include Rottkemper et al. (2012), Pacheco and Batta (2016), and Sakiani et al. (2020).Source: §2 · p. 5
  • Humanitarian-logistics reinforcement learning spans allocation, inbound logistics, and disaster response, but cited multi-agent work does not address physical resource redistribution in this decentralized setting.The paper incorporates service-gap penalties and worst-case service, with centralized training and decentralized execution.Source: §2 · p. 5
  • The review identifies insufficient attention to fully decentralized redistribution under unknown dynamics and distinguishes this work through recurrent learning and models of transport uncertainty, disruption-dependent shipment losses, and perishability.The cited coordination literature includes decentralized inventory management and multi-echelon inventory work.Source: §2 · pp. 5–6

3. Methodology

The methodology models relief centers as cooperative agents allocating inventory under partial observability, then trains recurrent MAPPO policies to use local histories and centralized training information. Its shared reward targets worst-center service and service equity while protecting network service below a threshold.

3.1. Problem Formulation

  • The network has autonomous centers connected by directed transfers; each period brings supply and completed shipments, and inventory beyond storage capacity is removed before allocation.Inventory beyond its shelf life is disposed of, while remaining inventory is kept.Source: §3.1 · pp. 6–7, §3.2.1 · p. 9

3.2. Cooperative Redistribution Model

  • Centers form a cooperative Dec-POMDP: each acts on local observations, while the model includes local observation and action spaces, state transitions, a shared reward, and a discount factor.The environment state is not directly observed by agents.Source: §3.2 · p. 7

3.2.2. MARL Action

  • Post-arrival, post-overflow inventory is allocated through a continuous simplex action across local service, holding, and intended transfers to other centers.Local service is capped by realized demand, and outbound transfers may be scaled when they exceed daily capacity.Source: §3.2.1 · p. 9, §3.2.2 · p. 9

3.2.3. Local Observations

  • Agents observe local inventory, current demand and supply relative to historical averages, prior unmet demand, and inbound transfers still in transit, but not partners’ inventories, demands, supplies, or current actions.The current observation alone does not characterize the evolving system state.Source: §3.2.3 · p. 10

3.2.4. Shared Reward

  • The shared reward improves service in the least-served center and penalizes disparities, while penalizing network service only when it falls below threshold ϕ = 0.60.The stated reward weights satisfy w3 ≥ w2 ≥ w1.Source: §3.2.4 · p. 11

3.3. Learning Framework

  • MAPPO uses centralized training with a shared critic and decentralized actors; at execution, each center acts on its own observations and the critic is discarded.During training, the critic observes all agents’ local observations, the previous transfer matrix, and origin-destination pipeline redistribution.Source: §3.3 · p. 11

3.3.1. Recurrent Actor and Policy Update

  • A recurrent memory summarizes each agent’s observation history, combining current observations with prior memory to represent trajectory dynamics without a direct observation signal.The memory state is updated at each period.Source: §3.3.1 · p. 11
  • The policy uses a Dirichlet distribution over simplex shares, and PPO updates actors with a clipped objective and entropy regularization; the critic is trained against a return target.The framework also uses generalized advantage estimation to compute advantages.Source: §3.3.1 · p. 12

4. Experimental Design

The experimental design evaluates six relief centers over 20-period emergency episodes, with hidden trajectory families and center roles. Recurrent MAPPO is compared with recurrent IPPO and a local-only heuristic using shared episodes and trajectories.

4. Experimental Design

  • Each episode has 6 centers and 20 periods; agents use 29-dimensional local observations and choose 7-dimensional simplex actions allocating service, holding, and transfers.The MAPPO critic uses a 246-dimensional centralized operational state.Source: §4 · p. 13
  • Hidden trajectory families and episode-specific center roles are not observed by actors or the critic, and demand, supply, and transportation realizations are generated by the described process.Source: §4 · pp. 13–14
  • Evaluation tracks service gap, worst-center service, network service, and episodic return, comparing recurrent MAPPO with recurrent IPPO and a local-only policy on the same generated episodes and trajectories.IPPO replaces MAPPO’s centralized critic with local critics; the local-only heuristic serves local demand and never redistributes inventory.Source: §4 · p. 14

5. Results and Discussion

MAPPO achieves stronger training results than IPPO and improves held-out service equity, though Local-only has the highest network service. Across unseen trajectory families, MAPPO retains service and equity gains over the baselines.

5. Results and Discussion

  • MAPPO’s mean last-100-episode reward was −5.68 versus IPPO’s −17.81, a 68.1% smaller reward penalty; network service was 0.599, 16.2% higher, and service gap fell 14.1%.Both learners improve over their initial policies.Source: §5 · p. 14, Figure 2 · p. 14

5.1. Validation and Held-Out Performance

  • At the final validation checkpoint, MAPPO had 11.4% higher network service, 31.0% higher worst-center service, and 4.5% lower service gap than IPPO.During learning, MAPPO’s service gap peaked at 0.213 around episode 960 while network service had reached 0.604.Source: §5.1 · p. 15, Figure 3 · p. 15
  • Local-only had the highest held-out network service at 0.6686, while MAPPO reduced service gap by 33.2% and raised worst-center service from 0.2369 to 0.3562 versus Local-only.The MAPPO differences were statistically significant; Table 3 reports means ± standard deviations across 20 held-out realizations.Source: §5.1 · pp. 15–16, Table 3 · p. 16
  • Against IPPO on held-out tests, MAPPO increased network service from 0.5602 to 0.6276 and worst-center service from 0.2767 to 0.3562, while service gap fell 1.9% nonsignificantly.Daily reward improved from −0.3786 to −0.0120; the other reported differences were statistically significant.Source: §5.1 · p. 16
  • Across five held-out trajectory families, MAPPO’s network service exceeded IPPO’s by 10.6%–15.3%, and worst-center service improved by 19.5%–41.6%.Worst-center service differences were significant for four families, with Early recovery the exception.Source: §5.1 · p. 16
  • Across unseen trajectories, MAPPO significantly reduced service gap and improved worst-center service versus Local-only; both MAPPO and IPPO generalized across trajectory types.The policies did not observe the trajectory types.Source: §5.1 · p. 16

6. Conclusion and Future Work

Cooperative MAPPO balances service and redistribution to improve equity, including service to the worst-served center, while trading off some overall network service. The authors identify unmodeled redistribution costs and other network-wide effects as future-work considerations.

6. Conclusion and Future Work

  • MAPPO trades off some overall network service for significantly better equity by reducing the service gap.The local-only heuristic can achieve higher overall network service despite a large service gap.Source: §6 · p. 17
  • The recurrent MAPPO policy balances local service, reserve, and redistribution without observing the true dynamics.The authors characterize this as reasonable decision-making under limited and delayed information.Source: §6 · p. 17
  • MAPPO significantly increases worst-case service relative to local-only operation by prioritizing the worst-served locations with marginal network-performance impact.Source: §6 · p. 17
  • Future work should account for volunteer availability, transportation, efficiency, and redistribution costs that may lead to costly two-way transfers.Source: §6 · p. 17

Declaration of Generative AI

The authors used Grammarly, ChatGPT, and Claude to improve writing quality and check grammar, then reviewed and edited the content.

Declaration of Generative AI

Appendix A. Synthetic Emergency Environment

The synthetic environment represents uncertainty through heterogeneous baseline demand and supply profiles and stochastic shocks that vary conditions across centers.

Appendix A. Synthetic Emergency Environment

  • Stochastic shocks applied to heterogeneous center-level baseline demand and supply profiles generate time-varying shortage, surplus, and balanced conditions.Source: Appendix A · p. 18

Appendix A.1. Hidden trajectories

Each episode samples an emergency onset day and an unobserved trajectory family, with five families representing distinct disruption and recovery patterns. Their unobserved temporal differences motivate recurrent inference from observation histories.

Appendix A.1. Hidden trajectories

  • Each episode samples an onset day from 2–6 and one hidden trajectory family, whose intensity is zero before onset and follows a family-specific path afterward.The five families represent early recovery, delayed peak, progressive worsening, persistent disruption, and aftershock.Source: Appendix A.1 · p. 18
  • Trajectory-family intensity values are clipped to [0, 1], and Table A.5 presents the training-episode dynamics and parameter values.Source: Appendix A.1 · p. 18, Table A.5 · p. 19
  • Because the distinct temporal dynamics are unobserved by the policy during training, recurrent MAPPO must infer the evolving state from observation history.Source: Appendix A.1 · p. 19

Appendix A.2. Demand and Supply Dynamics

Appendix A.2 describes an environment with heterogeneous center roles and evolving demand and supply. It incorporates stochastic variation and delayed relief-supply responses.

Appendix A.2. Demand and Supply Dynamics

  • Baseline demand and supply conditions vary by center, while β_i and δ_i govern disaster effects on demand growth and supply reduction, η_i controls delayed supply response, and χ_i controls transport vulnerability.Source: Appendix A.2 · p. 19
  • Two centers are randomly assigned to each of the shortage, surplus, and balanced roles, which have distinct demand, supply, and transport-vulnerability profiles.Table A.6 gives the center-role parameter ranges.Source: Appendix A.2 · p. 19, Table A.6 · p. 19
  • Episode severity is sampled from U(0.80, 1.20), and demand includes common and center-specific disturbances whose combined variance is denoted by σ_D^2.Source: Appendix A.2 · p. 20
  • Relief supplies do not rise immediately: a 2–6 day response delay is sampled, after which supply initially increases and then gradually declines.Source: Appendix A.2 · p. 20
  • Realized supply is defined relative to bootstrapped baseline supply, with disturbance terms capturing event-caused supply reduction and delayed incoming relief; a maximum operator keeps supply nonnegative.The supply disturbance parameter σ_S is 0.06.Source: Appendix A.2 · p. 20

Appendix A.3. Hyperparameter Setting

Table A.7 identifies the learning configuration for the two policy-gradient methods.

Appendix A.3. Hyperparameter Setting

  • Table A.7 presents the learning configuration for the two policy-gradient methods.Source: Table A.7 · p. 21

Related papers

Loading 2609.18639v1…