PaperlayerView exact revision on arXiv

Source-linked AI summary

Safe Multi-agent Reinforcement Learning with Natural Language Constraints

Ziyan Wang, Meng Fang, Tristan Tomilin, Fei Fang, Yali Du

arXiv:2405.20018v1cs.MAcs.CLcs.LG

TL;DR

Safe MARL commonly relies on predefined mathematical costs, while free-form human constraints are difficult to incorporate. SMALL uses language models to interpret constraints and guide policy learning, and LaMaSafe benchmarks this setting. Across evaluated environments, SMALL achieves comparable rewards and significantly fewer constraint violations.

  • Problem

    Safe MARL approaches commonly rely on fixed barriers or predefined costs, while diverse, context-specific natural-language constraints are difficult to estimate and incorporate.

    Source: §1 · p. 1

  • Method

    SMALL uses fine-tuned language models to interpret natural-language constraints, estimate violations from semantic alignment with environment descriptions, and guide policy learning.

    Source: §1 · p. 2

  • Results

    SMALL achieves comparable rewards to other MARL algorithms while significantly reducing constraint violations across the evaluated settings.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    LaMaSafe provides a benchmark suite for evaluating algorithms under free-form natural-language constraints in multi-agent environments.

    Source: §1 · p. 2, §7 · p. 9

    Scalability to larger multi-agent systems with complex constraints remains to be explored.

    Source: §7 · p. 9

Abstract

from arXiv · show

The role of natural language constraints in Safe Multi-agent Reinforcement Learning (MARL) is crucial, yet often overlooked. While Safe MARL has vast potential, especially in fields like robotics and autonomous vehicles, its full potential is limited by the need to define constraints in pre-designed mathematical terms, which requires extensive domain expertise and reinforcement learning knowledge, hindering its broader adoption. To address this limitation and make Safe MARL more accessible and adaptable, we propose a novel approach named Safe Multi-agent Reinforcement Learning with Natural Language constraints (SMALL). Our method leverages fine-tuned language models to interpret and process free-form textual constraints, converting them into semantic embeddings that capture the essence of prohibited states and behaviours. These embeddings are then integrated into the multi-agent policy learning process, enabling agents to learn policies that minimize constraint violations while optimizing rewards. To evaluate the effectiveness of SMALL, we introduce the LaMaSafe, a multi-task benchmark designed to assess the performance of multiple agents in adhering to natural language constraints. Empirical evaluations across various environments demonstrate that SMALL achieves comparable rewards and significantly fewer constraint violations, highlighting its effectiveness in understanding and enforcing natural language constraints.

1 Introduction

SMALL addresses the difficulty of applying free-form human constraints in Safe MARL by using language models to estimate constraint violations and guide policy learning. The paper also introduces LaMaSafe and reports comparable rewards with fewer violations.

1 Introduction

  • Safe MARL often uses fixed barriers or predefined costs, while diverse, context-specific language constraints are difficult to estimate and incorporate.Source: §1 · p. 1
  • In multi-agent settings, agents must respond to language instructions while cooperating, and pre-designed costs may fail to anticipate evolving constraints, increasing risks.Source: §1 · p. 2
  • SMALL summarizes and disambiguates natural-language constraints, encodes them with environment descriptions, and learns violation costs from semantic alignment to guide policy updates.Source: §1 · p. 2, Figure 1 · p. 2
  • Across discrete and continuous action settings, SMALL achieves comparable rewards and significantly reduces constraint violations, while LaMaSafe benchmarks language-constrained MARL.Source: §1 · p. 2

2 Related Work

Related work covers safe MARL, natural-language constraints, and language models. The cited approaches use predefined costs or structured constraint mechanisms, while SMALL applies fine-tuned language models to Safe MARL constraints.

Safe Multi-Agent Reinforcement Learning

  • Safe MARL methods based on fixed costs or shielding may not handle free-form, simultaneous constraints, and barriers can require prior knowledge and fail to generalize to new scenarios [10] [9] [14] [31] [8] [3].Source: §2 · p. 3

Constraints with Natural Language

  • Prior natural-language constraint methods either require ground-truth costs or model relevant environment entities, which can limit adaptability or handling of free-form language [30].Source: §2 · p. 3

Language Models

  • Transformer language models support text representations or generation, and the authors identify fine-tuned language models for Safe MARL constraints as their first application in this area [28] [6] [26] [4].Source: §2 · p. 3

3 Preliminaries

The preliminaries define a constrained Markov game and its cooperative objective. They also identify predefined cost functions as difficult to design and adapt to dynamic, unstructured constraints.

3 Preliminaries

  • A constrained Markov game is specified by agents, states, actions, transitions, team rewards, costs, a violation budget, and a discount factor [9].Source: §3 · p. 3
  • In the fully cooperative setting, agents maximize expected team reward while minimizing accumulated cost and satisfying the constraints.Source: §3 · pp. 3–4
  • Predefined cost functions require extensive domain knowledge and lack flexibility for dynamic, unstructured constraints.Source: §3 · p. 4

4 Methodology

SMALL models language constraints as unknown costs, then combines language-based cost prediction with multi-agent policy learning. Its cost-learning module condenses constraints, embeds their semantics, and predicts violations for policy updates.

4.1 Language Constrained Markov Game

  • The Language Constrained Markov Game treats costs as derived from free-form language, with a constraint sampled at each episode's start and used throughout training.The simplified constraint lc removes redundant or irrelevant information from the original language constraint l.Source: §4.1 · p. 4

4.2 Cost Learning Module

  • The Cost Learning Module uses an LLM to condense instructions, then fine-tunes a BERT-based encoder with triplet loss to embed constraint semantics.Triplets pair an anchor constraint with a semantically similar positive example and a different or unrelated negative example.Source: §4.2 · p. 4
  • For each agent, SMALL combines constraint–observation cosine similarity with an LLM decoder's binary violation judgment to predict cost.The decoder is queried with the current text observation and constraint; the two signals are combined into the predicted cost.Source: §4.2 · p. 5

4.3 Multi-Agent Policy Learning with Constraints

  • SMALL combines predicted constraint costs with MAPPO or HAPPO and a Lagrange multiplier to maximize reward while minimizing cost, without ground-truth costs.The policy is updated using predicted costs from the cost-learning module.Source: §4.3 · pp. 5–6

5 LaMaSafe Benchmark

LaMaSafe provides Grid and Goal environments for evaluating agents under free-form natural-language safety constraints. The benchmark spans discrete grid navigation and continuous robot control, with hazards, vases, and collisions as constraint settings.

5 LaMaSafe Benchmark

  • LaMaSafe introduces Grid and Goal environments for evaluating multi-agent navigation and objective completion under free-form natural-language constraints.Grid is a 2D discrete-action Mini-Grid environment; Goal is a 3D continuous-action Gymnasium environment.Source: §5 · p. 6
  • In LaMaSafe-Grid, agents seek their own target balls while avoiding an episode-specified hazardous area and collisions; episodes last at most 300 timesteps.The Random layout scatters hazards, while One-Path leaves a single safe route through a lava-filled grid.Source: §5 · p. 6
  • LaMaSafe-Goal uses Point, Car, and Ant robots, with hazards, contactable vases, and close-proximity collisions as constraints.Agents navigate toward targets that relocate after arrival; experiments measure cost by the number of hazard contacts, with a 1000-timestep limit.Source: §5 · pp. 6–7
  • Figure 2 depicts Grid with 20 randomly placed lava, water, and grass, and Goal(Ant) with two agents in a Hard layout.The figure also shows example natural-language constraints and environment descriptions; Goal difficulty increases with hazard and vase counts.Source: Figure 2 · p. 7

6 Experiments

Experiments evaluate SMALL on LaMaSafe-Grid and LaMaSafe-Goal under natural-language constraints; SMALL maintains near-backbone rewards while substantially reducing costs.

6.1 Setup

  • The experiments assess reward and cost performance under natural-language constraints, using three random seeds for average reward evaluation.Baselines include MAPPO, HAPPO, and their Lagrangian variants; LaMaSafe-Grid and LaMaSafe-Goal provide the evaluation environments.Source: §6.1 · p. 7, §6 · p. 7

6.2 Main Results

  • SMALL-based algorithms generally earn slightly lower rewards than their backbones but converge to extremely low costs in almost all environments.They maintain similar reward performance and excel in the more challenging Ant Medium and Hard layouts.Source: Figure 3 · p. 8, §6.2 · p. 8

6.3 Ablation Study

  • With four agents, SMALL substantially reduces constraint violations while keeping rewards slightly below baselines; baseline costs converge in Easy and Medium layouts.The setting doubles the original agent count and retains constraints against blue hazards and collisions.Source: §6.3 · p. 8, Figure 4 · p. 9
  • On the four-agent Hard Goal(Ant) layout, SMALL converges similarly in cost to methods using ground-truth costs, which act as oracles.Most SMALL-based algorithms perform similarly or slightly worse than those oracle methods.Source: §6.3 · p. 8, Figure 4 · p. 9
  • Ablations report that removing encoder fine-tuning harms prediction, removing the decoder weakens constraint adherence, and removing the descriptor degrades performance.The fine-tuning uses a separate set from subsequent training; the study uses SMALL-HAPPO in two-agent Easy Goal(Ant).Source: §6.3 · p. 9, Table 1 · p. 9
  • Directly predicting cost from the similarity score produces more conservative performance, with lower costs and significantly lower rewards.This is the final ablation described in the supplied passage.Source: §6.3 · p. 9

7 Conclusion

The paper concludes that SMALL can incorporate context-specific natural-language constraints into MARL, with comparable rewards and fewer violations. It identifies larger systems and ambiguous or conflicting constraints as future challenges, while noting possible applications in intuitive human-agent interaction.

7 Conclusion

  • SMALL uses fine-tuned language models to interpret natural-language constraints during policy learning, and the LaMaSafe benchmark evaluates algorithms under free-form textual constraints.The reported empirical outcome is comparable rewards alongside significantly fewer constraint violations.Source: §7 · p. 9
  • Future work should examine scaling to larger multi-agent systems and handling complex, ambiguous, or conflicting constraints.The conclusion presents these as limitations to address.Source: §7 · p. 9

A Broader Impact Statement

  • The authors describe potential for more intuitive human-agent interaction, with users specifying safety requirements and operational boundaries in natural language.Potential application domains include robotics, autonomous vehicles, and industrial automation.Source: A Broader Impact Statement · p. 12

B Reproducibility Statement

The reproducibility statement describes controlled experimental conditions and points to the paper's descriptions of settings and implementations.

B Reproducibility Statement

C Implementation of the LaMaSafe-Grid

LaMaSafe-Grid tests cooperative ball collection under natural-language constraints across layouts with different hazard patterns. Its cost rules and texted observations turn collisions, hazards, and nearby rewards into information relevant to agent decisions.

C Implementation of the LaMaSafe-Grid

  • The section describes the layouts, state and action spaces, obstacle types, ground-truth costs, and rule-based texted observations used in LaMaSafe-Grid.The texted-observation approach is also presented as applicable to continuous and discrete multi-agent settings.Source: C Implementation of the LaMaSafe-Grid · p. 12, §C.3 · p. 13

C.1 Layouts

  • The Random layout is a 14x14 grid with randomly scattered lava, water, and grass, while One-Path is an 8x8 grid of lava with a single hazard-free, turn-filled path.Avoiding lava requires strict adherence to the safe path; more flexible constraints allow freer navigation.Source: §C.1 · p. 12
  • Both layouts require agents to cooperate in collecting balls worth 3 points each; episodes end when all balls are collected or 300 timesteps elapse.The collective objective is to maximize reward while following the natural-language constraint.Source: §C.1 · p. 12

C.2 Ground Truth Cost Functions

  • Grid collisions and occupying a hazard cell specified by the constraint each incur cost 1, and costs accumulate over timesteps spent violating constraints.Accumulation makes cost proportional to the duration of collisions or constraint violations.Source: §C.2 · p. 13

C.3 Texted Observation

  • Raw texted observations report agent positions, hazard locations, and reward-ball positions; rule-based descriptions classify safe, hazardous, or adjacent tiles and nearby balls.These concise descriptions provide immediate surroundings for decisions about hazards and rewards.Source: §C.3 · p. 13

D Implementation of the LaMaSafe-Goal

LaMaSafe-Goal includes three agent types and three layouts with varying difficulty, yielding nine agent-layout types for experimental comparison.

D.1 Type of agents and Layouts

  • LaMaSafe-Goal’s three agent types and three layouts of varying difficulty produce nine agent-layout types in the experimental setting.The section identifies agent state and action spaces, obstacles, ground-truth costs, and rule-based texted observations as implementation topics.Source: §D.1 · p. 13, D Implementation of the LaMaSafe-Goal · p. 13

D.2 Ground Truth Cost Functions

LaMaSafe-Goal assigns costs to collisions and blue-hazard violations, accumulating them over time. Its texted-observation pipeline converts environment information into segmented descriptions of nearby obstacles.

D.2 Ground Truth Cost Functions

  • In the continuous 3D environment, a collision costs 1 when the centers of two agent models are less than 1 meter apart.The cost is assigned to the agents involved.Source: §D.2 · p. 14
  • An agent violates the blue-hazard constraint when its model edge is less than 1 meter from a blue hazard’s center, incurring a cost.Blue hazards are identified as the primary obstacles in LaMaSafe-Goal.Source: §D.2 · p. 14
  • Collision and constraint-violation costs accumulate across timesteps in proportion to violation duration.This calculation encourages agents to minimize time spent in violation.Source: §D.2 · p. 14

D.3 Texted Observation

  • Raw texted observations describe state, observations, actions, obstacles, and their spatial relations; rule-based descriptors segment them using textual radar information about obstacle position.The resulting descriptions support decisions based on nearby obstacles’ proximity and nature.Source: §D.3 · pp. 14–15

E Implementation Details

SMALL processes natural-language constraints and textual observations to predict costs and update multi-agent policies. Its implementation uses HAPPO and MAPPO backbones, compares against four baselines, and prompts an LLM for binary violation judgments.

E.1 Algorithm

  • SMALL condenses each sampled natural-language constraint, encodes it as an embedding, predicts costs from encoded texted observations, and updates value, policy, and Lagrange networks.The method uses PPO-based updates and is instantiated with HAPPO and MAPPO backbones as SMALL-HAPPO and SMALL-MAPPO.Source: §E.1 · p. 15

E.2 Baselines / Backbones

  • The comparisons include MAPPO [31], HAPPO [14], MAPPO-Lagrange [9], and HAPPO-Lagrange.MAPPO-Lagrange extends MAPPO with Lagrangian constraint adjustment; HAPPO-Lagrange applies the analogous approach to HAPPO.Source: §E.2 · p. 15

E.3 Query Prompt

  • The validation prompt supplies the natural-language constraint and agent’s texted observation, then asks the LLM whether that agent violated a constraint.The LLM returns a binary Yes/No judgment used in calculating predicted cost.Source: §E.3 · p. 16

E.4 Hyper-parameters

The section specifies the training hyperparameters and computational setup, including per-episode BERT fine-tuning for natural-language constraint semantics. It also lists varied LaMaSafe-Goal constraint formulations concerning hazards and other robots.

E.4 Hyper-parameters

  • Training uses Adam with a 3 × 10−4 learning rate; policy learning rates vary by algorithm, while the 0.95 discount factor is shared across tasks.SMALL-specific training hyperparameters are listed in Table 2; experiments ran on a system with 128 Intel Xeon processors at 2.2 GHz, 5 TB of memory, and an Nvidia A100 PCIE-40G GPU.Source: §E.4 · p. 16

F Natural Language Constraints and Fine-tuning

  • At each episode’s start, SMALL fine-tunes BERT on 30 randomly sampled triplets from a set distinct from the constraints used in subsequent training.Fine-tuning runs for 95 rounds to align BERT with constraint semantics, supporting its generalization to constraints encountered during training.Source: F Natural Language Constraints and… · pp. 16–17
  • LaMaSafe-Goal examples repeatedly prohibit approaching or contacting blue circular hazards and warn agents to avoid collisions with other robots.The formulations range from avoiding blue circles to treating them as risk areas or no-entry zones.Source: F Natural Language Constraints and… · p. 21
  • Other listed constraints describe water avoidance through metaphorical phrases such as “moon reflects on water” and “places where the sky touches the ground.”These examples are also paired with instructions to avoid collisions with other robots.Source: F Natural Language Constraints and… · p. 22
Loading 2405.20018v1…