PaperlayerView exact revision on arXiv

Source-linked AI summary

Better than Your Teacher: LLM Agents that learn from Privileged AI Feedback

Sanjiban Choudhury, Paloma Sodhi

arXiv:2410.05434v1cs.LGcs.AI

TL;DR

LLM agents lack a mechanism for automatic self-improvement from execution errors, and prompting exceptions is difficult to scale. LEAP iteratively trains agents with feedback from privileged experts whose extra state is hidden at test time. Across three domains, LEAP improves behavior-cloning results and can let weaker students surpass stronger teachers.

  • Problem

    LLMs can struggle to recover from execution errors, while prompting exceptions is difficult to scale and can require intractably long contexts.

    Source: §1 · p. 1

  • Method

    LEAP iteratively fine-tunes LLM agents with expert corrections informed by privileged state available during training but hidden from students at test time.

    Source: §1 · pp. 1–2, §3 · p. 3

  • Results

    Across three domains, LEAP improves behavior-cloning results and lets weaker students surpass stronger teachers; on ALFWorld, Llama3-8B reaches 91.8% versus its teacher's 65.7%.

    Source: §4.1 · p. 5

  • Takeaways & Limitations

    The reported analysis identifies balancing privileged information with student realizability as key to LEAP's success.

    Source: §1 · p. 2

    Generating interaction rollouts is time-consuming, and corrective feedback can produce lengthy outputs that are difficult to balance against feedback effectiveness.

    Source: §6 · p. 10

Abstract

from arXiv · show

While large language models (LLMs) show impressive decision-making abilities, current methods lack a mechanism for automatic self-improvement from errors during task execution. We propose LEAP, an iterative fine-tuning framework that continually improves LLM agents using feedback from AI expert teachers. Our key insight is to equip the expert teachers with a privileged state -- information that is available during training but hidden at test time. This allows even weak experts to provide precise guidance, significantly improving the student agent's performance without access to privileged information at test time. We evaluate LEAP on diverse decision-making benchmarks, including text-based games (ALFWorld), web navigation (WebShop), and interactive coding (Intercode Bash). Our experiments show that LEAP (1) outperforms behavior cloning and ReAct baselines (2) enables weak student models (e.g., Llama3-8B) to exceed the performance of strong teacher models (GPT4-o), and (3) allows weak models to self-improve using privileged versions of themselves. We also provide a theoretical analysis showing that LEAP's success hinges on balancing privileged information with the student's realizability, which we empirically validate. Our code is available at https://leap-llm.github.io

1 INTRODUCTION

LEAP addresses the difficulty of helping LLM agents recover from execution errors by giving training-time expert teachers privileged state information hidden from students at test time. The paper reports improved benchmark performance and a trade-off between privileged information and student realizability.

1 INTRODUCTION

  • LLMs can struggle to recover from test-time errors, while prompting exceptions is difficult to scale and can produce intractably long contexts.Finding a teacher able to correct mistakes from any student-visited state is challenging because recovery can be harder than simply completing the task.Source: §1 · p. 1
  • LEAP gives expert teachers privileged state unavailable to students at test time, using their corrections to teach task performance and recovery from mistakes.The framework iteratively updates the student using corrected reason-and-action labels.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • LEAP balances privileged information against student realizability: unrestricted corrections can be too complex for students, while easier-to-process feedback can be suboptimal.The paper reports using a constrained privileged expert to provide optimal yet realizable feedback.Source: §1 · p. 2
  • The paper evaluates LEAP on ALFWorld, WebShop, and InterCode, and analyzes the trade-off between privileged information and realizability.Its stated contributions include an iterative framework and theoretical and empirical analysis of that trade-off.Source: §1 · p. 2
  • LEAP improves behavior-cloning results on ALFWorld from 65.7% to 91.8%, WebShop from 29.4% to 61.8%, and InterCode Bash from 60.3% to 71.8%.The introduction also reports that weaker student models can surpass stronger teachers.Source: §1 · p. 2

2 PRELIMINARIES

The preliminaries frame decision-making as a partially observable problem in which agents act from interaction history. They motivate interactive imitation learning as an alternative to reinforcement learning for LLM agents.

2 PRELIMINARIES

  • A POMDP models hidden or incomplete environment information through states, actions, observations, transitions, and rewards; agents receive partial observations and rely on history.The history consists of past observations and actions.Source: §2 · pp. 2–3
  • LLM agents predict a natural-language reason and an action from their interaction history, while the added unconstrained reason space makes reinforcement learning impractical.The paper also notes the large history space as a challenge for POMDP reinforcement learning.Source: §2 · p. 3
  • Interactive imitation learning methods such as DAGGER query an expert on student-visited histories, collect corrective reasons and actions, and update the student on accumulated data.DAGGER is described as finding the best policy in the policy class as iterations approach infinity.Source: §2 · p. 3

3 APPROACH

LEAP trains LLM agents through iterative online feedback from a privileged expert. Its approach uses training-only state information while accounting for whether the student's policy can realize the expert's corrections.

3.1 PRIVILEGED EXPERT TEACHER

  • A privileged expert sees the environment's full state, making its decision-making a fully observable MDP, whereas correcting student actions from partial histories can be intractable and suboptimal.The teacher's privileged feedback must not directly reveal privileged information, or the student may be unable to realize the corrected actions.Source: §3.1 · p. 3

3.2 THE LEAP ALGORITHM

  • LEAP starts with supervised fine-tuning on demonstrations, then repeatedly rolls out the current student, obtains privileged corrections on failures, augments the data, and updates the policy.Updates can use aggregated-data SFT or preference optimization such as DPO or KTO; SFT is the default.Source: §3.2 · p. 4, §3.1 · p. 4

3.3 ANALYSIS

  • LEAP's performance bound reflects a trade-off between privileged-expert performance and realizability, alongside the expert's recoverability coefficient and the update's average regret.The analysis defines the average realizability gap between the privileged expert and the best policy over horizon T.Source: §3.3 · pp. 4–5

4 EXPERIMENTS

Across three interactive domains, LEAP improves behavior-cloning policies and students can surpass GPT-4o; experiments also examine privileged-feedback trade-offs, self-improvement, and update methods.

4.1 OVERVIEW OF RESULTS

  • 65.7%→91.8% (ALFWorld), 29.4%→61.8% (WebShop), and 60.3%→71.8% (InterCode): LEAP improves BC, with student policies surpassing GPT-4o on all three.The overview reports teacher comparisons of 91.8% vs. 65.7% on ALFWorld, 61.8% vs. 58.4% on WebShop, and 71.8% vs. 71.3% on InterCode.Source: §4.1 · p. 5, Table 1 · p. 6, Figure 3 · p. 7

4.2 DOMAIN 1: TEXT-BASED GAMES

  • 91.8% out-of-distribution success: ALFWorld’s largest gain came in iteration 1 (65.7%→91.0%), as privileged corrections taught more efficient object search.The report says gains come from learning to search likely locations rather than exhaustively searching shelves or drawers.Source: §4.2 · pp. 5–6, Table 1 · p. 6

4.3 DOMAIN 2: WEB AGENTS

  • 61.8 WebShop score: LEAP’s π2 surpasses GPT-4o’s 58.4, with the largest component advantage in OPT (42.4 vs. 22.1).The training corrections teach when to backtrack from a mismatching product and when to commit to one that fits.Source: Figure 3 · p. 7, §4.3 · pp. 7–8

4.4 DOMAIN 3: INTERACTIVE CODING

  • 71.8% success: the 70B InterCode policy matches or exceeds GPT-4o’s 71.3%, while the 8B policies improve over BC but remain below the teacher.The benchmark uses ground-truth Bash commands as privileged information; the authors note a strong teacher may be difficult to surpass.Source: §4.4 · p. 8, Table 2 · p. 8

4.5 WHAT IS THE TRADE-OFF BETWEEN PRIVILEGED INFORMATION AND REALIZABILITY?

  • Varying privileged-state use produced an optimum: performance rose from π1 to π3, then fell sharply through π5.The experiment varied how much privileged state the expert was prompted to use.Source: §4.5 · p. 8
  • πE1 gives realizable reasoning but a wrong action, whereas πE5 gives the correct action with an unrealizable privileged rationale; πE3 balances both.The πE3 rationale gives a plausible, student-followable reason for the correct action.Source: §4.5 · p. 8

4.6 CAN LEAP BE USED TO SELF-CORRECT A STUDENT?

  • 65.7%→82.1%→91.8%: Llama3-8B self-improves over two ALFWorld iterations, matching the best policy trained with GPT-4o as teacher.The authors caution that self-improvement may require a sufficiently strong base policy and may not hold in every environment.Source: §4.6 · p. 8, Figure 6 · p. 9

4.7 HOW DOES SFT COMPARE TO PREFERENCE OPTIMIZATION?

  • On ALFWorld, SFT improves more than DPO at β=0.1 or 0.01, while KTO reaches 88.1%, close to SFT’s 91.0%.The authors suggest SFT uses correction data more aggressively; KTO settings that emulate SFT-like behavior approach SFT performance.Source: Table 3 · p. 9, §4.7 · p. 9

5 RELATED WORK

Related work studies privileged training information, interactive imitation, and AI feedback, while identifying realizability as a challenge for learners. LEAP applies privileged feedback to LLM decision-making and self-improvement.

5 RELATED WORK

  • Privileged information is used in robotics across navigation, self-driving, manipulation, and locomotion, though expert actions can exceed what a learner can predict [Vapnik et al., 2015].Prior work links this realizability gap to spurious correlations and the latching effect [Bansal et al., 2018; Holtzman et al., 2019; Wang & Sennrich, 2020; Ortega et al., 2021].Source: §5 · p. 10
  • Interactive imitation methods such as DAGGER use corrective actions during student rollouts, but can underperform when partial observability or model capacity prevents asymptotic realizability.The paper positions LEAP as a practical approach to balancing privileged information with realizability for LLM agents.Source: §5 · p. 10
  • Self-correction research uses model-generated revisions, but without external feedback, LLMs often struggle to judge their own reasoning accurately [Huang et al., 2024].Other approaches use model judges, critics, environmental feedback, execution feedback, or tools [Wang et al., 2023; Paul et al., 2023; Chen et al., 2023b; Olausson et al., 2023; Gou et al., 2023].Source: §5 · p. 10
  • LEAP introduces privileged information for LLM decision-making and self-improvement, unlike methods relying on repeated solution generation and evaluation [Pang et al., 2024; Hosseini et al., 2024].Reflexion assumes multiple attempts in the same environment, which LEAP does not.Source: §5 · p. 10

6 LIMITATIONS

LEAP’s training has practical costs: generating interaction rollouts is time-consuming, and expert corrections can become lengthy as trajectories grow.

6 LIMITATIONS

  • Generating rollouts requires environment simulation and complete trajectories, making repeated fine-tuning costly in complex environments requiring multi-step reasoning.The authors identify rollout generation as a time-consuming limitation across multiple fine-tuning iterations.Source: §6 · p. 10
  • Lengthy expert corrections make it challenging to balance feedback effectiveness with verbosity as trajectories grow.The authors identify distilling or summarizing feedback at critical points as future work.Source: §6 · p. 10
  • Distilling and summarizing expert feedback at critical points is proposed as a future direction for addressing its verbosity.This is an author-proposed direction, not a demonstrated solution.Source: §6 · p. 10

REPRODUCIBILITY STATEMENT

The paper describes resources intended to support reproducibility and discusses potential impacts of LEAP across applications. It also flags ethical concerns around autonomous refinement and the need for safeguards and oversight.

REPRODUCIBILITY STATEMENT

  • The reproducibility materials include open-source code, prompts, configurations, training scripts, experiment details, and theoretical derivations and proofs.Setup, hyperparameters, and prompts are in the appendix, with theoretical assumptions and proofs detailed there as well.Source: REPRODUCIBILITY STATEMENT · p. 11

A BROADER IMPACTS

  • The authors describe potential uses in virtual assistants and dialogue agents, where iterative privileged feedback could reduce human intervention and improve efficiency.They also identify education, customer service, and software development as sectors where AI could assist professionals and automate routine tasks.Source: A BROADER IMPACTS · p. 16
  • The authors argue that weaker models outperforming stronger ones could broaden access to advanced models for organizations with limited computational resources.They also identify model transparency and interpretability as areas requiring further research when training relies on privileged feedback.Source: A BROADER IMPACTS · p. 16
  • The paper flags risks of unpredictable behavior and malicious misuse from self-improving AI, calling for ethical guidelines and robust oversight.It identifies trustworthiness and accountability as concerns for increasingly autonomous systems.Source: A BROADER IMPACTS · p. 16

B.1 TRADE-OFF BETWEEN REALIZABILITY AND PRIVILEGED INFORMATION

The analysis frames imitation performance as depending on both expert quality and the realizability gap, which grows as an expert uses more privileged information. LEAP uses a constrained privileged expert to balance these factors, with a practical approximation for its feedback.

B.1 TRADE-OFF BETWEEN REALIZABILITY AND PRIVILEGED INFORMATION

  • The analysis identifies learner performance as depending on privileged-expert performance and the learner–expert realizability gap.The stated policy class is assumed rich enough to contain all history-based policies, and privileged rollouts are available only during training.Source: §B.1 · p. 16
  • More privileged information improves expert performance and lowers its recoverability coefficient, but increases the realizability gap.The analysis therefore treats expert quality and student realizability as competing considerations.Source: §B.1 · p. 17
  • A constrained privileged expert minimizes distance from the privileged expert while remaining within a δ-KL ball of the non-privileged expert.The constraint is intended to trade off performance against realizability.Source: §B.1 · p. 18
  • As δ approaches zero, the constrained expert’s realizability gap decreases while its performance gap from the privileged expert and recoverability coefficient increase.The passage states that a critical δ achieves an optimal trade-off.Source: §B.1 · p. 18

B.2 PRACTICAL APPROXIMATION FOR CONSTRAINED PRIVILEGED EXPERT

  • For a fixed Lagrange multiplier, a closed-form expression motivates a penalty-method approximation to the constrained expert.When expert logits are unavailable, LEAP prompts ask the expert not to reveal privileged information and to give reasons and actions predictable from history alone.Source: §B.2 · p. 19

C.1.1 TRAINING PARAMETERS

The appendix lists LoRA hyperparameters for supervised fine-tuning and preference training, and reports the hardware used for training.

C.1.1 TRAINING PARAMETERS

C.1.2 EVALUATION PARAMETERS

The appendix specifies inference settings for Llama and OpenAI calls, then describes the ALFWorld benchmark, comparison methods, and task-prompt fields. Its single-trial setup excludes approaches that require repeated trials on the same game.

C.1.2 EVALUATION PARAMETERS

  • Llama 3 8B and 70B inference uses temperature 0.3 and a maximum token length of 256.Source: §C.1.2 · p. 20
  • OpenAI calls cover expert corrections, action-reason labeling, and ReAct prompt baselines, with detailed prompts deferred to Appendix D.Source: §C.1.2 · p. 21

C.2 ALFWORLD

  • ALFWorld is a partially observable text-based household game with six task categories, 139 in-distribution games, 134 out-of-distribution games, and a 30-step horizon.Agents interact through text commands; hidden object locations must be inferred from observations.Source: §C.2 · p. 21
  • The ALFWorld comparisons include ReAct with GPT-4o and GPT-4o-mini, a shared instruction prompt, and BUTLER trained using DAGGER.Source: §C.2 · p. 21

1. Task: What is the overall task

  • The task prompt asks for essential objects, critical locations, and an optimal action sequence based on the game log.The single-trial setting rules out methods such as Reflexion that require multiple trials on the same game.Source: §1 · pp. 21–22

C.3 WEBSHOP

This section describes LEAP’s WebShop and InterCode procedures, including privileged-expert corrections and trajectory-grounded guidance while keeping privileged information out of the student’s reasoning. It also covers WebShop evaluation and InterCode prompts, student trajectories, and reasoning annotations for human-labeled actions.

C.3 WEBSHOP

  • WebShop contains 1.18 million products and 12,586 human-provided instructions; evaluation uses 500 test instructions with a 30-step horizon and reports overall score plus ATT, OPT, TYPE, and PRICE.Tasks require agents to fulfill shopping requests through searches and option selections.Source: §C.3 · p. 22
  • The WebShop comparisons include ReAct with GPT-4o and GPT-4o-mini and a behavior-cloning IL baseline, whose separate query-selection and click-selection models constrain the search space.The ReAct comparisons use the same instruction prompt as LEAP.Source: §C.3 · p. 22
  • LEAP LoRA-fine-tunes Llama-3-8B with GPT4-o corrections over four iterations, aggregating corrections from unsuccessful rollouts with the existing demonstrations for supervised fine-tuning.The initial policy uses observation-action demonstrations annotated by GPT4-o; training rollouts cover 12,086 tasks, and the privileged state specifies required attributes, price constraints, the option to select, and an example matching product.Source: §C.3 · p. 22

2. Student agent template:

  • The student-agent prompt generates reason-and-action trajectories for task objectives and is also used to prompt GPT-4 and ReAct baselines.It is used at test time by Llama3 student models.Source: §2 · p. 23

3. Autolabel reasons template:

  • For datasets with human action-labeled demonstrations, the autolabel prompt generates reasoning annotations so students can train to produce both reasons and actions.AlfWorld and WebShop are cited as examples.Source: §3 · p. 23

D.2.1 EXPERT CORRECTIONS TEMPLATE

  • WebShop expert corrections use privileged state to guide improved actions while requiring reasons based only on the student's prior observations and actions, without revealing privileged information.Corrections target selected timesteps judged definitely wrong, and suggested actions must be available among the candidate actions.Source: §D.2.1 · pp. 27–28
  • WebShop experts correct selectively, prioritizing indecision and task completion because matching some criteria earns partial credit, while failing to finish earns zero.The template recommends correcting key moments when the student browses without finishing and says a suboptimal purchase is preferable to no purchase within the time horizon.Source: §D.2.1 · p. 28
  • WebShop student and autolabel templates require candidate-listed actions and reasons grounded in prior observations, with rationales explaining choices under the task’s time horizon.Autolabel reasons are first-person, preserve the original action sequence, and may justify settling for a non-optimal option to finish by clicking buy now.Source: §D.2.1 · pp. 29–30

D.3 INTERCODE

  • InterCode teachers use a privileged optimal SQL example and expected observation to identify mistakes and guide students toward discovering relevant tables without revealing the privileged answer.The expected observation is what the student is evaluated on, and multiple actions can lead to it.Source: §D.3 · pp. 31–32
  • InterCode corrections selectively replace clearly wrong actions with improved reasons and actions based only on the student’s history up to each timestep.The improved reasons should offer general principles, avoid mentioning the original incorrect action, and assume the student followed the original trajectory until that timestep.Source: §D.3 · p. 32
  • The InterCode prompt includes the student trajectory and privileged state, and specifies a corrected action that is blank when is_corrected=False.The output is requested in the specified JSON format.Source: §D.3 · p. 34
Loading 2410.05434v1…