PaperlayerView exact revision on arXiv

Source-linked AI summary

Optimizing Team Behavior via Extremum-seeking Control in Multi-agent Reinforcement Learning

Svar Rajankumar Patel, Kshitij Jerath

arXiv:2610.18808v1cs.MAcs.AIcs.ROcs.SYeess.SY

TL;DR

Cooperative MARL needs coordination, but the useful level of behavioral diversity depends on the task and is difficult to tune. ADiCo combines DiCo with ESC to adapt the diversity target from performance feedback during training. Across two tasks, it adapts diversity to task requirements and matches or exceeds the reported DiCo baselines.

  • Problem

    Behavioral diversity can support complementary policies, but its appropriate level varies by task and existing approaches generally do not explicitly regulate its magnitude.

    Source: §I · p. 1

  • Method

    ADiCo combines fast-timescale DiCo diversity regulation with slower-timescale ESC target adaptation using scalar performance feedback, without an explicit model or analytical gradient.

    Source: §I · p. 1

  • Results

    Across six configurations of two cooperative tasks, ADiCo adapts diversity to task requirements and matches or exceeds grid-searched and unconstrained DiCo baselines.

    Source: §I · p. 2, A. Performance Across Task… · p. 4

  • Takeaways & Limitations

    Experiments with different initial setpoints show that users can start with a rough diversity estimate and let ADiCo refine it without an explicit model of team dynamics.

    Source: §VII · p. 6

    Mean-return feedback is confounded by policy learning, environmental shifts, and adapting opponents, complicating adaptation in adversarial or mixed-motive domains.

    Source: §VI · p. 6

Abstract

from arXiv · show

In cooperative Multi-Agent Reinforcement Learning (MARL), agents must learn to coordinate their actions to achieve shared objectives. A key indicator of effective coordination is behavioral diversity, i.e., the agents' potential to enact distinct yet complementary control policies given the same set of observations. Although behavioral diversity can enable role specialization and richer joint behaviors, its appropriate level is highly context- and task-dependent and often costly to tune. We propose Adaptive Diversity Control (ADiCo), a two-timescale framework that adapts behavioral diversity online during training. On a fast timescale, Diversity Control (DiCo) regulates system diversity toward a target value. On a slower timescale, an Extremum Seeking Controller (ESC) adjusts this target using only mean episodic returns, without requiring an explicit model or analytical gradient of the diversity-performance landscape. We evaluate ADiCo on six configurations across two cooperative MARL tasks, where it maintains task-appropriate diversity levels: low diversity when homogeneous policies are beneficial and higher diversity when heterogeneous policies are required. Through this adaptation, ADiCo matches or exceeds the mean performance of DiCo (best grid-searched diversity target) and Unconstrained DiCo (diversity allowed to emerge without target regulation). These results demonstrate that incorporating classical feedback control into MARL training enables effective coordination by providing a systematic alternative to task-specific manual selection of the diversity target.

I. INTRODUCTION

Behavioral diversity can support complementary roles, but its useful level depends on the task and is generally not explicitly regulated. ADiCo combines DiCo with ESC to adapt a diversity target during training, matching or exceeding the reported DiCo baselines across six configurations.

I. INTRODUCTION

  • The appropriate diversity level depends on task requirements, from low diversity for similar delivery routes to greater diversity for routes serving different areas.The paper motivates methods that can measure and regulate diversity accordingly.Source: §I · p. 1
  • The work measures behavioral diversity with SND and uses DiCo to regulate it toward a prescribed target, whose appropriate value is generally unknown beforehand.ESC perturbs the target and uses resulting performance changes to estimate a local direction of improvement.Source: §I · p. 1
  • ADiCo combines DiCo’s diversity regulation with slower-timescale ESC target adaptation, using scalar performance feedback without a model or analytical gradient.The framework preserves compatibility with on- and off-policy actor-critic algorithms without modifying the MARL learning objective.Source: §I · p. 1
  • ADiCo matches or exceeds grid-searched and unconstrained DiCo baselines across six configurations of two cooperative tasks.The framework adapts diversity without manual selection of the target.Source: §I · p. 2

II. BACKGROUND AND RELATED WORK

Prior approaches encourage behavioral heterogeneity, but generally do not make its magnitude an explicit control variable or locate an unknown reward-maximizing target. ADiCo uses ESC to adapt DiCo’s diversity target from performance feedback within cooperative MARL training.

II. BACKGROUND AND RELATED WORK

  • Earlier methods encourage specialization or diverse behavior, while SND measures policy heterogeneity and DiCo regulates it toward a prescribed target; locating an unknown reward-maximizing target remains open.Adaptive hyperparameter methods use populations, candidate policy updates, or meta-gradients, whereas this setting calls for model-free target adjustment from scalar performance feedback.Source: §II · p. 2

III. ADAPTIVE DIVERSITY CONTROL

  • ESC uses periodic target perturbations and performance variation to estimate a local improvement direction, adapting ADiCo’s system-level diversity target more slowly than DiCo and MARL.It seeks a locally high-performing diversity level along a single training run.Source: §II · p. 2

A. Problem Formulation

  • The cooperative problem is modeled as a Dec-POMDP with a shared team reward, and training follows CTDE: centralized information may be used in training while agents execute from local observations.Each agent selects actions from its local observation under a decentralized policy.Source: A. Problem Formulation · p. 2

B. Measuring and Controlling Behavioral Diversity

The section measures behavioral diversity as a system-level average of pairwise policy distances, then uses DiCo to regulate it toward a scalar target. The target’s task-appropriate value remains to be determined.

B. Measuring and Controlling Behavioral Diversity

  • System Neural Diversity averages pairwise 2-Wasserstein policy distances across agents and observations.For stochastic Gaussian policies, the distance has a closed-form expression; for deterministic policies, it reduces to Euclidean distance between policy outputs.Source: B. Measuring and Controlling… · pp. 2–3
  • DiCo decomposes each policy into shared homogeneous and agent-specific heterogeneous components, then rescales the latter toward a desired diversity target.The target is denoted θ, and a smoothed estimate of the unscaled heterogeneous components’ diversity is used in the scaling factor.Source: B. Measuring and Controlling… · pp. 2–3
  • The architecture retains DiCo’s rescaling pipeline and adds an ESC signal chain that uses mean reward to estimate a gradient and update the diversity setpoint.The current diversity is estimated using an exponential moving average.Source: Figure 1 · p. 3

C. Diversity-Performance Landscape

The diversity target is task-dependent, and its relationship to expected team performance may change as policies learn. The section frames target selection as black-box optimization using episodic return feedback.

C. Diversity-Performance Landscape

  • The diversity–performance map J(θ) relates a diversity target to expected team performance, but is unknown and may change as policies learn.The MARL/DiCo training process is treated as the faster subsystem, with target selection handled on the slower ESC timescale.Source: C. Diversity-Performance Landscape · p. 3
  • At each cycle, the target stays fixed during training, then updated policies are evaluated over M complete episodes to obtain mean return Rk for the next target update.Diversity adaptation therefore follows each training-and-evaluation cycle rather than every environment timestep.Source: C. Diversity-Performance Landscape · p. 3
  • ESC perturbs the target and processes the resulting return changes to estimate a local direction of performance improvement without an analytical gradient.The search seeks a locally optimal target within Θ = [0, θmax].Source: C. Diversity-Performance Landscape · p. 3

D. Extremum Seeking Diversity Adaptation

ADiCo uses periodic setpoint perturbations and filtered episodic returns to estimate a local performance-improvement direction, then updates the nominal diversity target. The supplied experiment material includes the evaluation’s scope and algorithm procedure.

D. Extremum Seeking Diversity Adaptation

  • ESC applies a sinusoidal perturbation to the nominal setpoint and projects the resulting target onto the admissible range Θ before supplying it to DiCo.The perturbation phase advances by ω∆t at each ESC update.Source: D. Extremum Seeking Diversity Adaptation · p. 3
  • A high-pass filter, demodulation, and a low-pass filter process episodic return to produce an estimate of the local performance response to the perturbation.The resulting signal ˆξk estimates the local direction of performance improvement.Source: D. Extremum Seeking Diversity Adaptation · p. 3
  • The controller uses ˆξk to update the nominal setpoint with integrator gain η and projection onto Θ, seeking a locally high-performing diversity level.The complete procedure is summarized in Algorithm 1.Source: D. Extremum Seeking Diversity Adaptation · p. 3

IV. EXPERIMENTS

  • The experiment section evaluates ADiCo’s diversity adaptation across task configurations and analyzes convergence behavior.Algorithm 1 organizes the procedure into setpoint selection, DiCo policy training, and ESC setpoint update phases.Source: §IV · pp. 3–4

A. Environments and Task Configurations

The evaluation uses Navigation and Balance, two cooperative continuous-control tasks, with three partially observed agents. ADiCo is compared with grid-searched and unconstrained DiCo under a shared training budget.

A. Environments and Task Configurations

  • All experiments use three agents under partial observability, with episodes lasting at most 100 steps.Team size is fixed for analysis, though ADiCo is not limited to a fixed number of agents.Source: A. Environments and Task Configurations · p. 4
  • Navigation uses continuous 2D forces to reach goal locations, while Balance coordinates forces on a shared platform to move a packet toward a goal.The figure background encodes SND diversity, with yellow for high and purple for low diversity.Source: Figure 2 · p. 4, A. Environments and Task Configurations · p. 4
  • Grid-searched DiCo uses the best fixed SND target from nine values, while unconstrained DiCo lets diversity emerge with agent-specific scaling fixed at one.Each baseline and ADiCo run uses the same MARL algorithm, network architecture, and 12M-frame budget; each grid-search target receives a separate run.Source: A. Environments and Task Configurations · p. 4

B. Training and Evaluation

Training alternates policy updates with evaluation, using mean episodic return to update ADiCo’s diversity setpoint.

B. Training and Evaluation

  • Each cycle collects 60,000 frames from 600 parallel environments, updates the policy, evaluates 200 episodes, and uses mean return to update the setpoint.Training runs for 12M frames over five random seeds.Source: B. Training and Evaluation · p. 4

V. RESULTS AND ANALYSIS

The results section examines task-config performance, adaptation from different initial targets, and training with continued ESC perturbations.

V. RESULTS AND ANALYSIS

  • The evaluation covers performance across task configurations, diversity adaptation from different initial targets, and training behavior under continuous ESC perturbations.Source: §V · p. 4

A. Performance Across Task Configurations

Across task configurations, ADiCo adapts diversity to task needs and performs comparably to tuned DiCo; tested initializations converge similarly, and continued ESC adaptation can remain beneficial.

A. Performance Across Task Configurations

  • Across Navigation, ADiCo reaches SND ≈1.2–1.3 for 3A-3G, ≈0.8 for 3A-2G, and near zero for 3A-1G, while maintaining low diversity across Balance masses.In Navigation it achieves reward comparable to grid-searched DiCo and lower reward variance than Unconstrained DiCo; in Balance it matches DiCo at SND = 0.Source: A. Performance Across Task… · p. 4, Figure 3 · p. 5
  • For 3A-4M, Unconstrained DiCo’s reward collapses as diversity rises, whereas ADiCo matches manually tuned DiCo without configuration-specific grid search.Source: A. Performance Across Task… · p. 4

B. Convergence From Different Initializations

  • From initial SND targets of 0, 0.5, and 1.0, ADiCo reaches ≈1.2–1.3; the 0.5 and 1.0 starts arrive faster than the 0 start.All three ADiCo initializations achieve mean returns near 1.0 and episode lengths of approximately 45–50 steps; DiCo at SND = 0 or 0.5 performs worse, while SND = 1.0 produces longer episodes.Source: B. Convergence From Different… · p. 5, Figure 4 · p. 5

C. Effect of Continued ESC Adaptation

VI. LIMITATIONS AND FUTURE DIRECTIONS

The authors identify limits in ADiCo’s feedback signal, optimization target, and runtime scope, while concluding that team-performance feedback can adapt diversity during training.

VI. LIMITATIONS AND FUTURE DIRECTIONS

  • Mean return can change for reasons beyond diversity adjustments, ESC may find a local rather than global optimum, and one team-wide SND target may not fit distinct roles.ADiCo adapts diversity during training, not runtime execution.Source: §VI · p. 6

VII. CONCLUDING REMARKS

  • In tested configurations, users can start from a rough diversity estimate and let ADiCo refine it without an explicit model of team dynamics.The authors frame training as adjustable components optimized through feedback, drawing on established control techniques.Source: §VII · p. 6

Related papers

Loading 2610.18808v1…