PaperlayerView exact revision on arXiv

Source-linked AI summary

Adapting LLM Agents with Universal Feedback in Communication

Kuan Wang, Yadong Lu, Michael Santacroce, Yeyun Gong, Chao Zhang, Yelong Shen

arXiv:2310.01444v3cs.CLcs.AI

TL;DR

Existing approaches can use linguistic feedback and non-linguistic rewards only partially. LTC combines them in an iterative, communication-based training framework; across reported benchmarks, it outperforms instruction-tuning baselines, including by 12% on ALFWorld success rate.

  • Problem

    Some settings provide both linguistic feedback and reward signals, but existing methods may use the linguistic data for instruction fine-tuning and rewards only to filter that data.

    Source: §1 · p. 1

  • Method

    LTC uses a universal buffer and iterative exploration and policy updating, with task-specific communication patterns for single-agent and multi-agent settings.

    Source: §1 · pp. 1–2

  • Results

    Across ALFWorld, HotpotQA, and GSM8k, LTC outperforms instruction-tuning baselines, including by 12% on ALFWorld success rate.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors present communication-based iterative learning as promising for adapting LLM agents to new tasks and environments.

    Source: §6 · p. 9

    LTC was evaluated with Llama-7B; computational resource limits prevented training larger models, and LTC underperformed larger-model CoT and CoT-SC on numerical reasoning.

    Source: §4.3 · p. 8

Abstract

from arXiv · show

Recent advances in large language models (LLMs) have demonstrated potential for LLM agents. To facilitate the training for these agents with both linguistic feedback and non-linguistic reward signals, we introduce Learning through Communication (LTC). We design a universal buffer to store all the feedback, and an iterative pipeline to enable an LLM agent to explore and update its policy in an given environment. To optimize agent interactions for task-specific learning with our universal buffer and pipeline, we introduce diverse communication patterns tailored for both single-agent and multi-agent environments. We evaluate the efficacy of our LTC approach on four diverse datasets: ALFWorld (single-agent), HotpotQA (multi-agent collaboration), Chameleon (multi-agent competition), and GSM8k (multi-agent teacher-student). On these data sets, LTC outperforms the supervised instruction fine-tuning baselines by 3.6% to 12%. These results highlight the versatility and efficiency of LTC in facilitating online adaptation for LLM agents.

1 Introduction

Existing approaches may use linguistic feedback while treating non-linguistic rewards only as filters. LTC combines both signal types in an iterative training framework with task-specific communication patterns, and outperforms baselines across the reported benchmarks.

1 Introduction

  • When feedback is heterogeneous, existing methods may use linguistic data for instruction fine-tuning but use reward signals only to filter data, not as a reinforcement-learning objective.Source: §1 · p. 1
  • LTC alternates exploration and policy updating from replay-buffer samples, combining language-modeling and PPO losses to balance language consistency with reward signals.Exploration collects trajectories and rewards; updating uses a sampled subset of the buffer, supporting autonomous adaptation without human supervision.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Three task-specific communication patterns—Single-agent Monologue, Multi-agent Dialogue, and Teacher-student Dialogue—collect linguistic feedback and reward signals from environments or agents.Source: §1 · p. 2
  • 12% on ALFWorld success rate, 5% on HotpotQA EM score, and 3.6% on GSM8k accuracy are LTC’s reported gains over their respective baselines.The baselines are instruction tuning for ALFWorld and HotpotQA, and CoT-Tuning for GSM8k.Source: §1 · p. 2

2 Related Work

Related work covers instruction tuning for general LLM capabilities and a range of LLM-agent methods, including planning, tool use, and collaboration. LTC distinguishes itself by adapting agents through automatically collected exploration data rather than few-shot prompting or pre-collected instruction data.

2.1 Instruction Tuning

  • Instruction tuning improves LLM capabilities and controllability through instruction-data generation, selection, and fine-tuning, while this paper targets adaptation to specific tasks or environments.Source: §2.1 · p. 3

2.2 LLM Agents

  • Prior LLM-agent research spans planning, environment feedback, tool use, and multi-agent collaboration.Source: §2.2 · p. 3
  • LTC collects training data through exploration, alternating interaction and trajectory collection with policy updates rather than relying on few-shot prompts or pre-collected instructions.Source: §2.2 · p. 4, Figure 3 · p. 4

3 Learning Through Communication

LTC alternates exploration, which collects communication trajectories and reward signals, with policy updating from the collected sessions. Its task-specific communication patterns support single-agent and multi-agent interaction, while updating combines language-model and reinforcement objectives.

3 Learning Through Communication

  • LTC alternates environment exploration to collect trajectories and rewards with an updating phase that fine-tunes the agent’s policy.The pipeline repeats these phases across iterations.Source: §3 · p. 4

3.1 Exploration Phase

  • Exploration represents each session as S = (T, M, R), with communication text, a source mask, and reward signals.M identifies text sources, while R records rewards from the system or agents.Source: §3.1 · p. 4
  • Single-agent Monologue collects an agent’s stepwise reasoning, actions, and environment rewards for training during exploration.It is designed for general instruction-following tasks and draws on ReAct-style think-and-act steps.Source: §3.1 · p. 4
  • Multi-agent Dialogue assigns roles such as thinker and actor, with environment rewards based on agent performance or answer correctness.In the HotpotQA example, the thinker analyzes and advises the actor; LTC agents can be trained for both roles.Source: §3.1 · p. 5, Figure 4 · p. 5
  • Teacher-student Dialogue uses teacher feedback and rewards, then poses an analogous question so students practice beyond the original answer.The pattern is designed for complex analytical tasks such as numerical reasoning.Source: §3.1 · p. 5, Figure 4 · p. 5

3.2 Updating phase

  • Updating combines a language-model objective over trajectories with a reinforcement objective that maximizes expected environment or teacher rewards.The combined objective weights the language-model term by β; off-policy PPO optimizes the reinforcement term using agent-generated tokens as actions.Source: §3.2 · pp. 5–6

4 Experiments

Experiments span four agent settings and report gains over instruction-tuning baselines across the evaluated tasks, while LTC underperforms much larger CoT models on GSM8k. The experiments pair dataset-specific communication patterns with an instruction-tuned Llama-7B baseline.

4.1 Datasets

  • ALFWorld, HotpotQA, Chameleon, and GSM8k represent single-agent, collaborative, competitive, and teacher-student settings, respectively, with a communication pattern matched to each.The patterns are monologue for ALFWorld, multi-agent dialogue for HotpotQA and Chameleon, and teacher-student dialogue for GSM8k.Source: §4.1 · p. 6

4.2 Settings

  • Experiments initialize Llama-7B with instruction fine-tuning and compare against prompting methods and tuned Llama-7B baselines; GPT-4 is not used at test time.The instruction-tuned agent serves as the baseline for a fair comparison.Source: §4.2 · p. 7

4.3 Results

  • 91% success rate for LTC exceeds the best instruction-tuning baseline’s 78% on ALFWorld; LTC also reaches 76% on the hardest Pick Two & Place task.Cool and Look each reach 100% success.Source: Table 1 · p. 7, §4.3 · p. 7
  • 5% higher Exact Match (EM) score than the instruction-tuning baseline is reported for LTC on HotpotQA.CoT-SC is 0.2% better than LTC, while ReAct→CoT-SC surpasses LTC by 1.9%.Source: Table 2 · p. 8, §4.3 · p. 8
  • 3.1% higher winning rate against GPT-4 players is reported for LTC over instruction-tuning baselines on Chameleon; LTC teacher-student dialogue beats the GSM8k instruction fine-tuning baseline by 3.6% accuracy.On GSM8k, teacher-student dialogue also surpasses the single-agent monologue baseline.Source: Table 3 · p. 8, §4.3 · p. 8

5 Discussion

The discussion examines prompting efficiency, training behavior, and GPT-4 teacher shortcuts, alongside a GSM8k accuracy table organized by prompting and fine-tuning methods.

5 Discussion

  • Table 4 reports GSM8k accuracy, with prompting methods in the upper block and fine-tuning results in the bottom block.The supplied caption does not specify individual scores or a winning method.Source: Table 4 · p. 8
  • GPT-4 sometimes uses pretraining-based shortcuts as a teacher, whereas LTC-trained LLaMA-7B searches for the second entry instead of shortcutting.The case study contrasts communication-based learning with relying solely on GPT-4-generated data.Source: §5 · pp. 8–9
  • Adding communication-collected data raised performance above 80%, while separating positive and negative samples with PPO produced faster, larger improvements.Accuracy and loss curves are shown in Figures 5 and 6.Source: Figure 5 · p. 9, Figure 6 · p. 9, §5 · p. 9
  • 12.8%, 8.6%, and 10.8% of ICL input tokens were used by LTC agents across the three tasks, respectively.The comparison uses CoT prompts for GSM8k and ReAct prompts for the other two tasks.Source: §5 · p. 9

6 Conclusion

The conclusion presents LTC as communication-based iterative learning for adapting LLM agents, reporting strong performance and efficiency across four tasks. It identifies broader communication patterns and human involvement as future directions.

6 Conclusion

  • LTC performs strongly in success rate and efficiency across ALFWorld, HotpotQA, Chameleon, and GSM8k, consistently outperforming existing LLM-agent and instruction-tuning baselines.The conclusion describes adaptation to new tasks and environments with minimal human effort.Source: §6 · p. 9
  • Three communication modes support interactions with environments and agents such as GPT-4 and humans, whose histories can be organized into PPO training data.The conclusion characterizes this as a closed loop of self-interaction and learning.Source: §6 · p. 9
  • Future work will explore more diverse task-specific communication patterns and include human communication during iterative learning.The authors also plan to open-source the code.Source: §6 · p. 10

A.1 Communication Patterns

The appendix describes communication patterns as a way to collect task-specific exploration data in a unified format.

A.1 Communication Patterns

  • Communication patterns are designed to collect trajectories and reward signals from different types of tasks.Source: §A.1 · p. 15
  • The collected data uses a unified format described in Figure 2.Source: §A.1 · p. 15
  • Three Python-style algorithms demonstrate how the communication patterns collect exploration data.Source: §A.1 · p. 15

A.2 Buffer Structure

The buffer stores interaction data as token sequences with separate text, mask, value, and reward streams, supporting policy updates from multiple communication patterns.

A.2 Buffer Structure

  • Each exploration trail is stored in a replay buffer as five sequences: Sa, Sm, Sv, Sl, and Sr.The buffer data is used in the updating phase.Source: §A.2 · p. 15
  • Sa records valid tokenized system and agent text, including environment descriptions, feedback, actions, reasoning, and hints; invalid generated text is skipped.Source: §A.2 · p. 15
  • Sm masks system tokens from policy and value losses, applies both losses to agent-generated tokens, and masks value loss for other-agent hints.Hints remain supervised by policy loss.Source: §A.2 · pp. 15–16
  • In the Dialogue pattern, GPT-4 supplies thoughts, LLaMA generates actions, and environment responses provide parsed rewards.The algorithm records teacher, student, and system messages with distinct masks.Source: §A.2 · p. 16
  • The teacher-student algorithm has LLaMA answer a question, GPT-4 assess it and generate a similar QA pair, and LLaMA answer that pair for reward evaluation.Source: §A.2 · p. 16
  • Sv stores action-state values from the model's value head, implemented as an auxiliary linear layer with tanh outputs in (−1, 1).The layer follows the second-to-last LlamaDecoderLayer.Source: §A.2 · pp. 16–17
  • Sr stores sparse rewards, usually zero, with +1 or -1 assigned when a task finishes or the current buffer token length overflows.Source: §A.2 · p. 17

A.3 Algorithm of LTC

Algorithm 4 implements LTC as an iterative process that initializes an LLM agent, gathers exploration data, stores it in a replay buffer, and updates the agent with PPO.

A.3 Algorithm of LTC

  • LTC first fine-tunes the pre-trained LLM agent to align its initial configuration with the intended learning tasks.This is followed by iterative exploration and updating.Source: §A.3 · p. 17
  • During exploration, sampled training environments are engaged asynchronously across multiple GPUs to generate trial data.The generated data is synchronized across GPUs and stored in a replay buffer.Source: §A.3 · p. 17
  • During updating, the agent samples from the replay buffer and uses PPO in a distributed data-parallel setting to refine its strategies.Source: §A.3 · p. 17

A.4 Training Loss

LTC combines language-modeling, policy, value, and entropy losses, using reward-bearing generations and masked PPO objectives to train the agent.

A.4 Training Loss

  • The total objective adds language-modeling loss to weighted policy, value, and entropy losses, with β and λ as weighting hyperparameters.The agent samples recent trajectories from the replay buffer after each exploration phase.Source: §A.4 · p. 17
  • The language-modeling loss is cross-entropy on the agent’s generations that receive positive rewards.Source: §A.4 · p. 17
  • The policy loss uses a masked PPO surrogate objective with advantage estimates, while the value loss is a masked mean squared error.The policy mask excludes encoded system messages; the value loss uses a separate mask.Source: §A.4 · pp. 17–18
  • The entropy loss is a small negative factor times policy-distribution entropy, included to encourage sufficient exploration.Its coefficient is 0.01.Source: §A.4 · p. 18

A.5 Implementation detail

The implementation uses asynchronous distributed exploration and compares prompting and tuning baselines; loss ablations and a case study report additional observations about LTC.

A.6 Asynchronously Distributed Generating

  • Asynchronous GPU workers explore open-ended environments until their combined buffers reach a preset size, then synchronize replay buffers for the next update.A barrier holds early-finished workers until the target buffer quantity is reached.Source: §A.6 · p. 19

A.7 Baselines

  • The comparison includes ReAct-Tuning and CoT-Tuning on LLaMA-7B, and test results are from the trained agent rather than GPT-4.The appendix also describes prompting baselines including ReAct, ReAct-IM, CoT, and CoT-SC.Source: §A.7 · p. 19

A.8 Losses

  • With LTC communication, ALFWorld performance quickly surpassed 80%, and separating positive and negative samples with PPO produced faster, greater improvement.Without the communication pattern, improvement from pre-collected instruction data was limited.Source: §A.8 · p. 19
  • In a HotpotQA case study, GPT-4 uses pretrained knowledge to shortcut retrieval, while LTC-trained LLaMA-7B searches for the second entry instead.The authors state that LTC communication provides additional learning benefits compared with relying solely on GPT-4-generated data.Source: §A.8 · p. 20
Loading 2310.01444v3…