Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization

1 INTRODUCTION

• Most existing language agents are not aligned with environment reward functions, while useful reflection requires identifying mistakes and producing actionable improvements.The introduction describes this as a credit-assignment challenge for a frozen language model.Source: §1 · pp. 1–2
• The paper illustrates uninformative frozen-model reflection with a Teen Titans answer that includes both the original series and its spinoff instead of only Teen Titans Go.The feedback rephrases the failed action sequence as a plan, leading to repetitive incorrect actions in the next trial.Source: Figure 1 · p. 2
• Retroformer agents improve success rates in HotPotQA by 18% with 4 retries, AlfWorld by 36% with 3 retries, and WebShop by 4%.The paper also reports faster learning than Reflexion, which does not use gradient for reasoning and planning.Source: §1 · p. 2
• Retroformer uses policy gradients to fine-tune a retrospective model that refines language-agent prompts from environment feedback across tasks and rewards.The approach learns from failed attempts and assigns credit to actions based on future rewards.Source: §1 · p. 2
• The retrospective model is fine-tuned without accessing the Actor LLM parameters or propagating gradients through it, making Retroformer a plug-in for cloud LLMs such as GPT or Bard.The contribution describes the module as agnostic to the Actor LLM.Source: §1 · p. 2

2 RELATED WORK

• Chain-of-Thought prompts agents to decompose challenging reasoning tasks, while ReAct maps observations to natural-language reasoning and action traces or API calls for environmental interaction.The section describes several applications built on this agent architecture.Source: §2 · p. 3
• Self-Refine supports iterative output refinement but is not tailored to real-world task interaction, while Reflexion uses dynamic memory and self-reflection to improve past actions.The section also describes RAP as using Monte Carlo Tree Search with environment rewards.Source: §2 · p. 3
• Transformer reinforcement-learning methods include online PPO and offline ILQL and DPO, which are used to guide language models toward reward-aligned behaviors.The section places these methods in the broader language-model fine-tuning pipeline.Source: §2 · p. 3

3 NOTATION AND FORMULATION

• The LLM action agent maps prompts to actions, with contextual state information and any memory represented as text in the prompt.The agent is stateless, and its sampled actions make the function random.Source: §3 · p. 3
• The environment transitions between text-represented states based on actions and assigns rewards, with the objective of maximizing cumulative episode returns.Rewards may be sparse, occurring mainly at a terminal state that indicates success or failure.Source: §3 · p. 4
• The retrospective model takes prior states, actions, rewards, and the user prompt as input and produces a new prompt for the LLM.Its parameters are the learnable parameters in the stated optimization setup.Source: §3 · p. 4
• Because only the retrospective model is learnable, treating the frozen LLM as part of the environment makes the objective a regular reinforcement-learning optimization.The formulation preserves the environment reward function while composing it with the LLM transition.Source: §3 · p. 4

4 OUR APPROACH: REINFORCING RETROSPECTIVE LANGUAGE AGENT

• Retroformer combines an actor LLM that generates actions with a retrospective LLM that refines its prompt; the actor is frozen while the retrospective model is locally fine-tunable.The framework assumes the actor's parameters are inaccessible and the retrospective model can be fine-tuned in low-resource settings.Source: §4 · p. 4, Figure 2 · p. 4

4.1 RETROSPECTIVE AGENT ARCHITECTURE

• The retrospective model diagnoses possible causes of failure from the trajectory and persistent memory, then appends concise reflections and plans to the actor prompt for later attempts.The actor's short-term memory is its current trajectory; reflection responses serve as long-term memory.Source: §4.1 · p. 5
• The replay buffer stores reflection instructions, responses, and episode returns across tasks, supporting retrospective-model fine-tuning and learning from both failed and successful reflections.Task-specific trajectory and reflection memory helps exploit prior attempts, while the cross-task buffer supports learning across environments.Source: §4.1 · p. 5

4.2 POLICY GRADIENT OPTIMIZATION

• A reflection response is rated by the change in episode return between consecutive trials: positive changes receive higher ratings, while nonpositive changes receive lower ratings.When possible, reward uses soft matching, such as F1, rather than exact-match binary scoring.Source: §4.1 · p. 5, §4.2 · p. 6
• PPO fine-tunes the retrospective model with response ratings from replay-buffer samples, while a KL-divergence term limits drift from the frozen reference model.The paper describes offline fine-tuning before validation and best-of-n response sampling during online execution.Source: §4.2 · pp. 6–7

5.1.1 ENVIRONMENT

• Across HotPotQA, AlfWorld, and WebShop, experiments evaluate question answering, multi-step decision making, and web browsing, respectively.The study compares ReAct and Reflexion and visualizes agents’ generated text and actions.Source: §5.1.1 · p. 7, §5 · p. 7

5.2 EXPERIMENT SETTINGS

• Experiments use GPT-3 or GPT-4 as frozen actors, fine-tune the retrospective model from LongChat, and report success rates on 100 HotPotQA, 134 AlfWorld, and 100 WebShop validation tasks.ReAct, Reflexion, and Soft Actor-Critic are included as baselines.Source: §5.2 · p. 7

5.3 RESULTS

• 54% success rate in 4 trials exceeded the reported 50% for a prior approach using a larger frozen GPT-3 reflection component.The paper also reports that Retroformer outperforms two strong baselines and improves performance consistently over trials, especially in the first few.Source: §5.3 · p. 8, Figure 4 · p. 8
• Figure 5 illustrates a failure case where frozen-model reflection repeats prior actions, while reinforced reflection identifies a more actionable search direction.The figure caption notes that the lengthy prior-trial observation is abbreviated.Source: §5.3 · p. 8, Figure 5 · p. 9
• In AlfWorld, Retroformer significantly improves over frozen baselines and solves the environment within 3 retries.Performance is slightly better with LoRA rank r = 4, and improvements are mostly from early retries; the retrospective model summarizes failed plans and compares them with task descriptions.Source: §5.3 · pp. 8–9, Figure 6 · p. 9
• In WebShop, Retroformer improves success rate by 4% after 4 retries, with gains more limited than in HotPotQA and AlfWorld.The authors suggest verbal feedback may not be optimal for web browsing, which requires substantial exploration and more precise search queries.Source: §5.3 · p. 9

• Extensive real-world dataset evaluations show Retroformer improves language agents over time in learning speed and final task completion.Source: §6 · p. 9
• Treating the LLM action agent as part of the environment lets policy gradients use diverse reward signals without accessing Actor LLM parameters or propagating gradients through them.Source: §6 · p. 9
• The approach can fine-tune memory, summarization, or actor-prompt components while keeping the rest of the agent architecture fixed.Source: §6 · p. 10

A CHALLENGES

• In text-rich environments, LLM agents can work when states and actions are accurately described in natural language.Source: A CHALLENGES · p. 12
• LLM agents face spurious actions outside the action space, prompt-length limits as trajectories grow, heuristic prompt tuning, and prohibitive direct training of large models.Source: A CHALLENGES · p. 12

B INTUITION

• The framework trains a retrospective language model on user prompts, environmental rewards, and feedback; reinforcement learning optimizes that model instead of the fixed action-generating LLM.Source: B INTUITION · p. 12
• The retrospective model condenses prior experience and feedback into a length-limited prompt, with the original LLM treated as part of the environment because its parameters are fixed.Source: B INTUITION · pp. 12–13

• The experiments use GPT-3 text-davinci-003 as the frozen actor and LongChat-7B-16K as the retrospective model, with actor temperature zero and top-p one.Source: §C.1 · p. 13
• The setup uses LangChain connectors, FastChat for retrospective-model inference, and TRL’s PPO trainer for fine-tuning.Source: §C.1 · p. 13
• The appendix reports 3,383 HotPotQA reflection samples, 523 AlfWorld samples, and 267 WebShop samples; experiments ran on GCP GKE with A100 40GB GPUs.Source: §C.1 · p. 13
• The retrospective model is fine-tuned with 4-bit quantized LoRA adapters; reported trainable-parameter counts are 0.53M or 2.25M, with four training epochs.Source: §C.1 · p. 13
• The offline PPO pipeline has three stages: data collection, reward-model learning, and policy-gradient fine-tuning of the retrospective component.Source: §C.1 · p. 13
• Rollouts are collected across environments; reflection responses are rated by the change in episodic return, and higher-rated responses are accepted over lower-rated ones.Source: §C.1 · p. 14
• The pipeline samples reflection responses for unsuccessful tasks, then rolls out the next episode to obtain returns for rating the responses.Source: §C.1 · p. 14
• RewardTrainer learns to classify accepted versus rejected responses, and PPOTrainer fine-tunes the retrospective model to generate higher-rated reflections.Source: §C.1 · p. 14

• SAC uses 768-dimensional mean-pooled text-action embeddings and online updates to a LongChat-16K action model with LoRA rank 4.Source: §C.2 · p. 14

C.3 REWARD FUNCTION

• HotPotQA uses token-overlap precision and recall after removing stopwords from the generated and ground-truth answers to calculate F1 reward.Precision divides common tokens by generated-answer tokens; recall divides them by ground-truth-answer tokens.Source: §C.3 · p. 14
• AlfWorld rewards each task with a binary success or failure outcome at the end of the episode.Source: §C.3 · p. 14
• WebShop assigns an end-of-episode reward to the chosen product and uses text-matching heuristics to penalize products of the wrong type.The type check distinguishes products that share attributes and options but are different kinds, such as butter and plant-based meat.Source: §C.3 · p. 14

D ADDITIONAL EXPERIMENTS

• The reinforced model separates reflections from new plans and removes irrelevant “Next trial:” text, unlike the frozen model’s interleaved response.The paper presents the structured response in Figure 7 and notes the cleanup by comparison with Figure 5.Source: D ADDITIONAL EXPERIMENTS · p. 15, Figure 7 · p. 15

E.1 ACTOR PROMPT ENGINEERING

• Actor-prompt examples combine the current task and environment context with prior-trial memory, including an AlfWorld example of a repeated-action loop.The prompt examples also show task-specific instructions and demonstrations for HotPotQA and WebShop.Source: §E.1 · pp. 17–18

E.2 RETROSPECTIVE PROMPT ENGINEERING

• Retrospective instructions ask the model to diagnose a failed attempt and propose a concise, actionable plan tied to specific actions.The examples include task-specific instructions for HotPotQA, WebShop, and Docstore, with the AlfWorld example identifying a repeated-action loop.Source: §E.2 · pp. 19–21