Source-linked AI summary
Learning to Teach in Cooperative Multiagent Reinforcement Learning
Shayegan Omidshafiei, Dong-Ki Kim, Miao Liu, Gerald Tesauro, Matthew Riemer, Christopher Amato, Murray Campbell, Jonathan P. How
TL;DR
Prior cooperative MARL communication focuses on task coordination, while learning to teach raises additional challenges in measuring advice’s effects on learning. LeCTR trains agents to request and provide action advice, and the paper reports faster learning and coordination in tasks where existing methods fail.
Problem
Prior MARL communication targets task coordination rather than teaching, while learning to teach compounds delayed reward, credit assignment, and partial observability challenges.
Source: Introduction · p. 1Method
LeCTR trains cooperative agents to learn when and what to advise, switch between student and teacher roles, and use received advice to improve local learning.
Source: Introduction · p. 2, Teaching in Cooperative MARL · p. 2Results
LeCTR agents learn significantly faster and coordinate in tasks where existing teaching methods fail.
Source: Introduction · p. 2Takeaways & Limitations
LeCTR trains advisors using task-level learning progress, improving learning rate without harming final performance and without assuming expert teachers.
Source: Contribution · p. 7The paper focuses on pairwise teaching; fusing advice from multiple teachers in the n-agent setting remains for future investigation.
Source: LeCTR: Algorithm for Learning to… · p. 4
Abstract
from arXiv · showhide
Collective human knowledge has clearly benefited from the fact that innovations by individuals are taught to others through communication. Similar to human social groups, agents in distributed learning systems would likely benefit from communication to share knowledge and teach skills. The problem of teaching to improve agent learning has been investigated by prior works, but these approaches make assumptions that prevent application of teaching to general multiagent problems, or require domain expertise for problems they can apply to. This learning to teach problem has inherent complexities related to measuring long-term impacts of teaching that compound the standard multiagent coordination challenges. In contrast to existing works, this paper presents the first general framework and algorithm for intelligent agents to learn to teach in a multiagent environment. Our algorithm, Learning to Coordinate and Teach Reinforcement (LeCTR), addresses peer-to-peer teaching in cooperative multiagent reinforcement learning. Each agent in our approach learns both when and what to advise, then uses the received advice to improve local learning. Importantly, these roles are not fixed; these agents learn to assume the role of student and/or teacher at the appropriate moments, requesting and providing advice in order to improve teamwide performance and learning. Empirical comparisons against state-of-the-art teaching methods show that our teaching agents not only learn significantly faster, but also learn to coordinate in tasks where existing methods fail.
Introduction
The paper addresses the lack of algorithms for learning to teach in multiagent systems, where agents could share local knowledge without relying on expert teachers. It introduces peer-to-peer action advising in cooperative MARL and reports faster learning and coordination on tasks where existing methods fail.
Introduction
- Agents may have useful local knowledge without being experts, motivating learned peer teaching rather than hand-crafted, domain-specific heuristics.Communication can share knowledge and skills to improve system-wide learning.Source: Introduction · p. 1
- Prior cooperative MARL communication supports task coordination, whereas this work uses action advice to improve learning and final task performance.The paper identifies the lack of algorithms for learning to teach in multiagent systems.Source: Introduction · p. 1
- Learning to teach must address when and what to teach, how to learn from advice, privacy constraints, and delayed estimation of advice’s effects amid nonstationarity.These challenges compound delayed reward, credit assignment, and partial observability, and increase computational complexity.Source: Introduction · p. 1
- LeCTR agents learn when and what to advise, switch between student and teacher roles, and learn faster and coordinate in tasks where existing methods fail.The approach also supports heterogeneous teammates and settings with communication costs.Source: Introduction · p. 2
Background: Cooperative MARL
The paper frames cooperative MARL as a Dec-POMDP in which agents act jointly and receive local observations and shared reward. Agents use policies based on their observation histories to maximize discounted value.
Background: Cooperative MARL
- Cooperative MARL is formalized as a Dec-POMDP with agents, states, joint actions, transitions, rewards, observations, and a discount factor.The joint action determines state transitions, and observations are generated from the next state and action.Source: Background: Cooperative MARL · p. 2
- Agents choose actions using policies conditioned on their observation histories, which may be represented through recurrent internal states or belief states.The joint policy is parameterized by θ.Source: Background: Cooperative MARL · p. 2
- The team maximizes discounted expected reward, while each agent’s action-value estimates the expected value of an action given its local observation and internal state.The vector of action-values covers the agent’s available actions.Source: Background: Cooperative MARL · p. 2
Teaching in Cooperative MARL
The paper defines teaching as advice that influences teammates’ task-level learning, with agents learning to request or provide actions during cooperative MARL. The advising problem is coupled to task learning and, in the main treatment, focuses on pairs of agents.
Teaching in Cooperative MARL
- Agents learn to advise one another to improve joint task-level learning, deciding when and what to advise rather than relying on hand-crafted heuristics.The paper names this higher-level problem ePAdvise and aims for task policies that no longer need advice at every timestep.Source: Teaching in Cooperative MARL · p. 2
- Action advising assumes task-level algorithms support off-policy exploration and that agents receive advising-level observations about teammates’ learning progress.The paper notes that many earlier frameworks focus on single-agent RL, assume optimal advice, or use advising heuristics.Source: Teaching in Cooperative MARL · pp. 2–3
- Each agent learns a task policy, a student policy for requesting advice, and a teacher policy for advising a teammate; roles can change over time.Task-level policies summarize learned behavioral knowledge, and tildes mark advising-level properties.Source: Teaching in Cooperative MARL · pp. 2–3, Figure 1 · p. 2
- In action advising, a student may replace its intended action with a teammate’s advised action and use the resulting experience to update its task policy.The student policy decides whether to request help, and the teacher policy decides whether to respond.Source: Teaching in Cooperative MARL · p. 3
- Task-level learning optimizes task coordination, while ePAdvise seeks to influence teammates’ learning; the objectives are coupled as policies change.The paper focuses on pairs of agents and describes extensions to n agents.Source: Teaching in Cooperative MARL · p. 3
- Unlike communication methods aimed at task coordination, this approach studies how advice affects task-level learning; the approaches may be combined.Source: Teaching in Cooperative MARL · p. 3
- LeCTR iterates task-level learning and advising-level learning, with advising policies trained using rewards correlated with task-level learning.Source: Figure 2 · p. 3
LeCTR: Algorithm for Learning to Coordinate and Teach Reinforcement
LeCTR alternates task-level learning with training advising policies, so agents learn when to request or provide advice and how to improve teammates’ learning. Its policies use compact advising-level observations and reward signals tied to task-level learning rather than assuming teachers know the best actions.
LeCTR: Algorithm for Learning to Coordinate and Teach Reinforcement
- LeCTR iterates two phases: agents learn the task using black-box algorithms and current advising policies, then update advising policies with rewards correlated to teammates’ task-level learning.Repeating the phases is intended to train increasingly capable advising policies.Source: LeCTR: Algorithm for Learning to… · p. 3
- Task-level algorithms may be heterogeneous, and advising policies use compact observations rather than full policy parameters, which may be costly or undesirable to share.The stated concerns include decentralized learning, communication costs, and privacy.Source: LeCTR: Algorithm for Learning to… · p. 3
- Each agent learns student and teacher policies, allowing it to request advice, advise a teammate, or occupy both roles simultaneously.Student policies choose whether to request advice; teacher policies choose whether and what to advise using advising-level observations.Source: LeCTR: Algorithm for Learning to… · pp. 3–4
- Teacher policies can advise an action or decline, and may learn to suggest exploratory actions rather than simply imitate what the teacher would do.When advice is declined, the student executes its originally intended action.Source: LeCTR: Algorithm for Learning to… · p. 4
- Advising rewards are designed to reflect student learning after advice is executed; task reward alone may fail to capture learning and produce poor advice.The paper evaluates reward choices corresponding to different measures of task-level learning.Source: LeCTR: Algorithm for Learning to… · p. 4, Figure 3 · p. 5
- Advising policies are trained centrally with an actor-critic approach, while task-level policies train online and reservoir sampling helps reduce advising-reward nonstationarity.The paper says these measures stabilize advising-level learning.Source: LeCTR: Algorithm for Learning to… · p. 4
- For more than two agents, students must still decide how to fuse advice from multiple teachers; the paper leaves this for future work.Majority voting and learned teacher scores are offered as possible approaches, not established solutions.Source: LeCTR: Algorithm for Learning to… · p. 4
Evaluation
Evaluation compares LeCTR with existing teaching methods and no teaching across cooperative games, transfer settings, and communication costs. LeCTR outperforms the compared approaches with the VEG reward, learns faster than no teaching, and performs well with heterogeneous teammates and transferred advice policies.
Counterexample demonstrating Remark 1
- Using task reward as the advising reward can produce poor standalone task policies because teachers advise optimal actions without prompting exploration.The counterexample shows that student learning progress, rather than task-level reward, is critical for useful advising.Source: Evaluation · p. 5
Comparisons to existing teaching approaches
- With the VEG advising-level reward, LeCTR outperforms all approaches; across games, several heuristics over-advise and perform poorly, while probabilistic methods fare better.In Repeated, LeCTR also achieves the best final value and learning rate; in Hallway and Room, Importance Advising and Correct Important reach suboptimal goals but retain positive value through domain symmetries.Source: Evaluation · p. 6, Table 2 · p. 6
- LeCTR learns faster than no teaching, stabilizing after roughly 4–6 training epochs in the Hallway game.Figure 4b measures episodes needed to converge to the maximum value reached.Source: Figure 4 · p. 5, Evaluation · p. 6
Teaching for transfer learning
- For transfer from T3 to T4, fixed advising policies achieve AUC 68 ± 17, compared with 24 ± 26 without advising.The fixed advising policies were trained for transfer from T1 to T2; the T3-to-T4 test uses related Room tasks flipped vertically and horizontally.Source: Evaluation · p. 6, Figure 6 · p. 6
Advising heterogeneous teammates
- LeCTR attains the highest AUC across all tested action-space rotations in heterogeneous Room games.Importance Advising and Correct Important attain 0 AUC in heterogeneous cases; AdHocVisit performs reasonably well across rotations.Source: Figure 5 · p. 5, Evaluation · p. 7
Effect of communication cost on advice exchange
- With no communication cost, agents learn to advise opposite actions; at cost c = 0.5, advising becomes unidirectional.The paper attributes the change to binary advising rewards: two-way advising nullifies positive rewards and penalizes excessive advice.Source: Evaluation · p. 7, Figure 7 · p. 7
Related Work
Prior work spans knowledge transfer, curriculum and active-learning policies, and action advising, but advising methods largely rely on experts or heuristics. AdHocVisit and AdHocTD are identified as the only action-advising methods that do not assume expert teachers.
Related Work
- Related work spans inverse reinforcement and apprenticeship learning, learning from demonstration, curriculum learning, and meta-learned supervised active-learning policies; this work applies advising-level meta-learning in MARL [Ng and Russell 2000; Abbeel and Ng 2004; Argall et al. 2009; Bengio et al. 2009; Graves et al. 2017; Bachman, Sordoni, and Trischler 2017; Fang, Li, and Cohn 2017; Pang, Dong, and Hospedales 2018; Fan et al. 2018].In this MARL regime, agents must advise teammates without destabilizing coordination.Source: Related Work · p. 7
- Prior advising approaches use state-importance heuristics or learn when to advise, with Q-Teaching rewarding teachers for advising on important states [Clouse 1996; Amir et al. 2016; Torrey and Taylor 2013; Fachantidis, Taylor, and Vlahavas 2017].The approaches include student-initiated requests, teacher-initiated advice, and Q-Teaching.Source: Related Work · p. 7
- AdHocVisit and AdHocTD are described as the only action-advising methods that do not assume expert teachers, using local actions and visit-count heuristics [da Silva, Glatt, and Costa 2017].Wang et al. (2018) uses their algorithm with minor changes.Source: Related Work · p. 7
Contribution
LeCTR introduces learning to teach in cooperative MARL, training advisors with task-level learning progress while avoiding hand-crafted advising policies and expert-teacher assumptions. The work focuses on pairwise teaching and identifies n-agent extensions as future work.
Contribution
- LeCTR uses agents’ task-level learning progress as feedback for advising policies intended to improve learning rate without harming final performance.The paper introduces a new paradigm for learning to teach in cooperative MARL.Source: Contribution · p. 7
- Unlike prior approaches, LeCTR avoids hand-crafted advising policies and does not assume expert teachers [Torrey and Taylor 2013; Taylor et al. 2014; Zimmer, Viappiani, and Weng 2014].Source: Contribution · p. 7
- The work focuses on pairwise teaching and identifies extending the approach to n-agent settings as a natural avenue for future work.Its target issues are when and what to teach.Source: Contribution · p. 7
Supplementary Material
The supplementary material details advising rewards, prior heuristics, LeCTR’s scope and implementation, experimental procedures, notation, and algorithm steps. It contrasts reward signals and advising strategies while documenting the pairwise algorithm and evaluation setup.
Details of Advising-level Rewards
- Joint Value Gain measures task-level value improvement after advice, but may have high variance and require costly performance evaluation after each advising step.Source: Details of Advising-level Rewards · p. 9
- Advising-level rewards include signals based on TD-error reduction, loss improvement, parameter updates, and value estimates.The TDG and LG rewards target reductions in task-level TD error and loss, respectively; VEG uses a value threshold τ.Source: Details of Advising-level Rewards · p. 9
Details of Heuristics-based Advising Approaches
- Prior advising heuristics use state importance, uncertainty, advice budgets, or policy disagreement to determine when to request or provide advice.Q-Teaching learns when to advise by rewarding the teacher’s state-importance value.Source: Details of Heuristics-based Advising… · p. 9
Optimal Action Advising
- Optimal-action advice can yield suboptimal joint return when teammates have heterogeneous optimal policies, while LeCTR learns both when and what to advise.LeCTR can learn advice within teammates’ action spaces, beyond passive exploration in heterogeneous-action settings.Source: Optimal Action Advising · p. 10
Architecture, Training Details, and Hyperparameters
- Advising actors and critics use three-layer MLPs with 32 hidden units per layer; the training setup uses Adam at 1e−3 and distinct task- and advising-level discounts.The discounts are γ = 0.95 at task level and eγ = 0.99 at advising level.Source: Architecture, Training Details, and… · p. 10
Experimental Procedures
- Table 2 reports final joint-policy value and learning-curve AUC, with at least 10 independent runs and statistical significance assessed using a Student’s t-test at α = 0.05.AUC evaluates task-level policies after advising actions, measuring student learning progress without teacher advice actions.Source: Experimental Procedures · p. 10
Notation
- Notation uses superscripts for agent-specific properties, bold for joint team properties, and tilde accents for advising-level properties.Source: Notation · p. 11
LeCTR Algorithm
- LeCTR alternates task-level interaction and learning with updates to the advising critic and each agent’s student and teacher policies.The algorithm exchanges advice, executes task actions, updates local task policies, and stores advising-level transitions.Source: LeCTR Algorithm · p. 11