PaperlayerView exact revision on arXiv

Source-linked AI summary

The Lighthouse of Language: Enhancing LLM Agents via Critique-Guided Improvement

Ruihan Yang, Fanghua Ye, Jian Li, Siyu Yuan, Yikai Zhang, Zhaopeng Tu, Xiaolong Li, Deqing Yang

arXiv:2503.16024v2cs.CLcs.AI

TL;DR

LLM agents need high-quality feedback that is both informative and usable, but numerical scores provide limited contextual guidance and agents can struggle to apply verbal critiques. CGI pairs an actor with a trained critic and iteratively refines the actor using critiques. Experiments report improvements across three interactive environments, including a small critic outperforming GPT-4 as a critic.

  • Problem

    Numerical feedback can rank candidate actions but provides limited contextual guidance, while agents may struggle to interpret and integrate natural-language critiques.

    Source: §1 · p. 1

  • Method

    CGI pairs an environment-interacting actor with a critic trained to generate actionable evaluations, then uses iterative supervised fine-tuning to improve the actor’s use of critiques.

    Source: §1 · p. 2

  • Results

    Across three interactive environments, CGI substantially improves over baselines, and a small Llama-3-8B critic outperforms GPT-4 as a critic by +29.16%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper concludes that explicit verbal critiques support more flexible action revision and deeper exploration while improving state-of-the-art performance.

    Source: §9 · p. 10

    CGI requires approximately four times the computational time of the baseline, alongside a reported 43.31% gain over No Critique.

    Source: A Limitations · p. 16

Abstract

from arXiv · show

Large language models (LLMs) have recently transformed from text-based assistants to autonomous agents capable of planning, reasoning, and iteratively improving their actions. While numerical reward signals and verifiers can effectively rank candidate actions, they often provide limited contextual guidance. In contrast, natural language feedback better aligns with the generative capabilities of LLMs, providing richer and more actionable suggestions. However, parsing and implementing this feedback effectively can be challenging for LLM-based agents. In this work, we introduce Critique-Guided Improvement (CGI), a novel two-player framework, comprising an actor model that explores an environment and a critic model that generates detailed nature language feedback. By training the critic to produce fine-grained assessments and actionable revisions, and the actor to utilize these critiques, our approach promotes more robust exploration of alternative strategies while avoiding local optima. Experiments in three interactive environments show that CGI outperforms existing baselines by a substantial margin. Notably, even a small critic model surpasses GPT-4 in feedback quality. The resulting actor achieves state-of-the-art performance, demonstrating the power of explicit iterative guidance to enhance decision-making in LLM-based agents.

1 Introduction

CGI addresses the challenge of obtaining and using high-quality feedback for LLM agents by pairing an actor with a critic that generates actionable critiques. Experiments in three interactive environments report improvements over baselines, including gains from both critic feedback and action refinement.

1 Introduction

  • Numerical feedback ranks candidate actions with scalar scores but omits context-specific suggestions, exploration options, and ways to refine suboptimal behavior.Best-of-N generates N candidate actions and uses a trained reward model to select the highest-scoring one.Source: §1 · p. 1
  • Natural-language feedback offers detailed, context-aware recommendations, but self-refinement can produce weak feedback and agents may fail to integrate critiques into their decisions.The passage identifies hallucinations or complex problems as challenges for self-refinement and limited flexibility in adjusting or discarding plans as a utilization challenge.Source: §1 · p. 2
  • CGI pairs an environment-interacting actor with a supervisory critic trained to provide precise evaluations and actionable revisions, then trains the actor to apply critiques through iterative supervised fine-tuning.The framework has Critique Generation and Action Refinement stages.Source: §1 · p. 2
  • A small Llama-3-8B critic outperforms GPT-4 as a critic by +29.16%, while action refinement surpasses state-of-the-art metrics by 26.74%.The paper also reports substantial improvement over baseline approaches in three interactive environments.Source: §1 · p. 2
  • The paper presents CGI as supporting continued improvement on long-horizon tasks and higher scores in fewer steps.Its contributions include a training strategy that iteratively refines actions using natural-language critiques to improve reasoning and adaptability.Source: §1 · p. 2

2 Related Work

Related work distinguishes numerical feedback from natural-language feedback and groups interactive-agent learning methods into prompt-based, training-based, and inference-time sampling approaches. CGI uses a trained critic to assess candidate actions and suggest revisions.

2 Related Work

  • Numerical feedback uses reward models or verifiers to score correctness or preferences, whereas natural-language feedback evaluates actions in denser verbal form.The passage identifies LLM-as-judge and self-refinement or self-critique as ways to generate natural-language feedback.Source: §2 · p. 2
  • Interactive-agent learning methods include prompt-based memory guidance, training-based SFT or DPO, and inference-time sampling such as Best-of-N or Tree-of-Thought.The described training data may come from expert models or exploration strategies such as Monte Carlo Tree Search.Source: §2 · p. 2
  • CGI is presented as an inference-time sampling variation in which a trained critic evaluates candidate actions and suggests revisions for the actor.Its overview depicts the actor generating candidate actions, the critic assessing them and suggesting revisions, and the actor refining its actions.Source: §2 · p. 2, Figure 1 · p. 3

3 Preliminary

The preliminary section models embodied agent tasks as partially observable Markov decision processes and represents interactions as trajectories. It also introduces iterative supervised fine-tuning as exploration followed by learning from reward-filtered correct trajectories.

3 Preliminary

  • An environment task is modeled as a POMDP with instructions, states, available actions, observations, an environment-defined transition function, and a reward function.The actor selects actions from its interaction history and the environment.Source: §3 · p. 3
  • The interaction history contains the instruction and prior action-observation pairs, while a trajectory records the interaction through its final action and observation.The passage writes the history as τt and the trajectory as τ.Source: §3 · p. 3
  • Iterative SFT alternates exploration and learning for K rounds, retaining reward-correct trajectories and using them to fine-tune the actor.The exploration step applies the previous policy across environments; the retained trajectories form Dcorrect. The process is described as exploration and learning [48].Source: §3 · p. 3

4 Methodology

CGI pairs an actor that explores an environment with a critic that evaluates candidate actions and supplies feedback. Critique generation and iterative action refinement train the critic to provide actionable guidance and the actor to use it.

4.1 Overview of the CGI Framework

  • The actor generates candidate actions, the critic critiques them, and the actor uses those critiques to select a refined action for execution.Repeated interaction produces a refined trajectory.Source: §4.1 · p. 4

4.2 Critique Generation

  • The critic is fine-tuned to assess candidate-action alignment with optimal actions and produce structured, step-level critiques combining evaluation and actionable revision.Critiques assess contribution, feasibility, and efficiency, then assign an overall grade and suggest improvements.Source: §4.2 · pp. 4–5

4.3 Action Refinement

  • To address actor policy misalignment, iterative supervised fine-tuning trains the actor to use critiques during exploration and refinement.Only reward-1 trajectories and their critique-action pairs are retained to form correct-trajectory and refinement datasets.Source: §4.3 · p. 5
  • Each learning step combines expert paths and newly collected data to fine-tune the actor, while general datasets are included to improve generalization.The original actor model is fine-tuned rather than the previous iteration’s model to avoid overfitting.Source: §4.3 · pp. 5–6

5 Experiment Settings

The experiments assess the critic and CGI in three interactive environments, using environment-specific test metrics and comparisons with feedback methods and agent baselines.

5.1 Interactive and Agentic Environments

  • WebShop, ScienceWorld, and TextCraft provide shopping, elementary-science, and crafting tasks; evaluation uses 200 test simulations each for WebShop and ScienceWorld and 100 for TextCraft.Average final score is used for WebShop and ScienceWorld, and success rate for TextCraft.Source: §5.1 · p. 6

5.2 Training Settings

  • Both actor and critic use Llama-3-8B-Instruct; training uses 500 WebShop, 350 ScienceWorld, and 374 TextCraft simulations, with three critic-guided interactions and three actor-refinement iterations.Results are reported for the third refinement iteration.Source: §5.2 · p. 6

5.3 Baselines

  • Critic baselines include DGAP, Explicit RM, self-critique, and GPT-4o, while CGI is compared with closed- and open-source models, expert-trajectory agents, Iterative SFT, and Reflexion.The actor is fixed as Llama-3-8B-Instruct for critic comparisons; Iterative SFT and Reflexion results use their third iteration.Source: §5.3 · pp. 6–7

6 Main Results

CGI’s critic and iterative action refinement outperform numerical-feedback and iterative-learning baselines. The results also show that fine-tuned actors can struggle to use critiques, a problem refinement helps mitigate.

6 Main Results

  • For Llama-3-8B, verbal critique improves average performance by 42.89%, versus 5.09% with a discriminator.Critic gains are 46.03% for Llama-3-70B and 10.55% for fine-tuned Llama-3-8B, compared with discriminator gains of 5.49% and 4.42%, respectively.Source: §6 · p. 7
  • In three interactive environments, CGI’s iteratively refined actor outperforms advanced closed-source models and agents trained on expert trajectories; its 8B critic significantly outperforms GPT-4o against the stated Llama-3 baselines.The critic comparison covers Llama-3 (8B and 70B) and an expert-data fine-tuned Llama-3-8B variant.Source: §6 · p. 7, Table 2 · p. 8
  • Fine-tuned actors can use critiques poorly: in ScienceWorld fine-tuned Llama-3-8B scores 55.94% with critique, below vanilla Llama-3-8B and Llama-3-70B.On WebShop, critique guidance lowers fine-tuned Llama-3-8B from 76.12% to 74.68%.Source: §6 · p. 8
  • After three refinement iterations, CGI exceeds the supervised-fine-tuning-only actor by 15.99% and achieves state-of-the-art results.The paper reports that refinement improves critique utilization.Source: §6 · p. 8
  • CGI improves across iterations, whereas Reflexion changes little or degrades and iterative SFT gains mainly in its first iteration.The authors attribute CGI’s sustained improvement to high-quality critiques and better critique use, which support continued sampling of harder-task solutions.Source: §6 · pp. 8–9

7 Further Analysis

The analysis finds that CGI’s critique most often changes actions early in trajectories, while iterative refinement is especially beneficial on harder, longer tasks. Performance also rises with more candidate actions before saturating.

7.1 Qualitative Analysis of CGI

  • After refinement, stage-1 revision ratios rise by 8.02% on ScienceWorld, 17.81% on WebShop, and 9.12% on TextCraft; stage 1 has the most revisions.The analysis divides trajectories into five stages and measures the proportion of actions changed after critique.Source: §7.1 · p. 9
  • On easy ScienceWorld tasks, critique-guided inference improves performance by 59.64%; after three refinement iterations, hard-task performance rises by 28.75%.Task difficulty is grouped by average oracle-agent trajectory length; critique-only gains on harder tasks are +43.95%, less than on easier tasks.Source: §7.1 · p. 9

7.2 Effects of Number of Candidate Actions

  • CGI’s average score rises from 56.89% with one candidate action to 61.72% as candidates increase, then plateaus at M = 7.The average is the mean final score across three environments.Source: §7.2 · p. 10

8 Ablation Study

Removing any of the three actor-training datasets reduces performance, with critique-action pairs making the largest contribution and general instructions supporting generalization.

8 Ablation Study

  • Removing any one of Dcorrect, Drefine, or Dgeneral lowers performance versus the full first-iteration dataset.The ablation evaluates final performance under critic guidance.Source: §8 · p. 10
  • Without Drefine, average performance across three tasks falls from 68.50% to 50.37%, the largest ablation drop.The decline is 5.4% on WebShop, 22% on TextCraft, and 26.94% on ScienceWorld.Source: §8 · p. 10
  • Excluding Dgeneral causes a less severe drop; the authors identify general instructions as important for adapting to the unseen test set.Source: §8 · p. 10

9 Conclusion

CGI separates action proposals from verbal critique to refine LLM agents iteratively; experiments across three environments support its effectiveness, while higher computation and broader impact risks remain relevant considerations.

9 Conclusion

  • Figure 6 ablates Dcorrect, Drefine, and Dgeneral; CGI #Iter1 denotes the actor trained in the first refinement iteration with the full dataset.Source: Figure 6 · p. 10
  • CGI separates an actor that proposes actions from a critic that gives verbal guidance, and experiments across three environments support its effectiveness.The conclusion says explicit verbal critiques enable more flexible action revision and deeper exploration, contributing to improved state-of-the-art performance.Source: §9 · p. 10

A Limitations

  • CGI uses approximately four times the baseline computation, while achieving a 43.31% gain over No Critique.The authors characterize the added computational time as remaining within a reasonable range.Source: A Limitations · p. 16

B Broader Impacts

  • The authors associate CGI with improved task performance in embodied AI and real-world scenarios and say its small critic backbone reduces computational and environmental costs.They also state these features support agents’ alignment with human goals and broader access to feedback mechanisms.Source: B Broader Impacts · p. 16
  • The paper identifies risks that critiques may reproduce training-data biases, expose private reasoning, or facilitate disinformation, social engineering, and unsafe code generation.The authors add that a small critic can lower the cost of replication and misuse.Source: B Broader Impacts · p. 16

C Notations

  • The notation distinguishes Dcorrect trajectories, Drefine critique-action pairs, and Dgeneral conversational data used to maintain language-modeling capability.Dcorrect is collected during each refinement iteration; Drefine trains critique use; Dgeneral helps maintain general capabilities during fine-tuning.Source: C Notations · p. 16

D Instruction Prompt Examples

  • The example critique prompt assesses contribution, feasibility, and efficiency, then gives an overall grade and a suggested revision.The listed grading scale runs from Excellent to Very Poor.Source: D Instruction Prompt Examples · p. 18

E Evaluation Settings

  • The experiments use WebShop, ScienceWorld, and TextCraft, evaluating success rate in WebShop and TextCraft and reward in ScienceWorld.The stated task limits are 10 rounds for WebShop, 30 for ScienceWorld, and 20 for TextCraft.Source: E Evaluation Settings · p. 20, §E.1 · p. 20

E.2 Computational Efficiency Analysis

CGI uses more inference time than No Critique but less than a GPT-4 critic, and the authors report that a third refinement iteration gives the best average performance across tasks.

E.2 Computational Efficiency Analysis

  • CGI takes longer per task than No Critique but reduces inference time relative to using GPT-4 as critic while achieving significant performance improvements.The efficiency evaluation uses Llama-3-8B as the actor across three agentic environments and measures average inference time per task.Source: §E.2 · p. 20

F Implementation Details

  • Both actor and critic use Llama-3-8B-Instruct, and the reported fine-tuning runs used eight NVIDIA A100-40GB GPUs.For action refinement, the authors retrained the base model each time to prevent policy drift.Source: F Implementation Details · p. 21, §F.2 · p. 22

F.1 Training Data

  • Critic training uses GPT-4o-generated expert critiques, with SFT sets of 14K ScienceWorld, 10K WebShop, and 8K TextCraft examples.Source: §F.1 · p. 21
  • Action refinement ran three times, using correct trajectories and critique-action pairs the actor had followed correctly in the prior iteration.The appendix says each iteration’s dataset also includes these two training parts; its sizes are tabulated in Table 4.Source: §F.1 · p. 22, Table 4 · p. 22

F.3 Effect of Iteration Count

  • The third iteration achieves the highest average performance across tasks; iteration four lowers performance in ScienceWorld and TextCraft, with only a +0.67% WebShop improvement.The authors therefore report third-iteration results in the main results section.Source: §F.3 · p. 22, Figure 7 · p. 22

F.4 Effect of Training with Expert Trajectories and Updating the Critic

The authors test whether CGI depends on gold-standard expert trajectories and whether updating the critic after refinement helps; reported results support both robustness to the data substitution and further gains from updating.

F.4 Effect of Training with Expert Trajectories and Updating the Critic

  • The study replaces gold-standard expert trajectories with GPT-4o-generated trajectories and separately retrains the critic using new critiques after action refinement.The retrained critic generates the critique data used for its update, adapting it to the updated actor’s policy.Source: §F.4 · pp. 22–23
  • Updating the critic raises reported performance from 61.44% to 67.02%, while CGI without gold-standard expert data averages 50.71 and outperforms the baseline and GPT-4o data-generating model.The comparison is reported in Table 6.Source: §F.4 · p. 23, Table 6 · p. 23

G.1 Critique Methods

The appendix describes critic and iterative-baseline setups, then illustrates actor failure to follow critiques, critic comparisons, and refinement trajectories in ScienceWorld.

G.1 Critique Methods

  • The critic comparison uses GPT-4 and self-critique for verbal feedback and DGAP cosine similarity to expert data for numerical feedback.The study collected 23K ScienceWorld, 16K WebShop, and 12K TextCraft points for the reward model.Source: §G.1 · p. 23

G.2 Iterative Methods

  • Reflexion adds self-reflection from each iteration to the next system prompt, while Iterative SFT trains on newly collected correct trajectories mixed with ShareGPT data.The appendix reports trajectory counts of 1,676, 1,719, and 1,750 for iterations one through three, respectively.Source: §G.2 · pp. 23–24

H.1 Failure Cases of Fine-tuned Llama3

  • A failure example shows the fine-tuned Llama3 agent disregarding critique and issuing an unsupported “go to bathroom” action.The environment responds that no known action matches that input.Source: §H.1 · p. 24

H.2 Critic-Llama3 Outperforms GPT-4o

  • In two examples, Critic-Llama3 corrects GPT-4o’s approval of a spatially misplaced action and rejection of a more direct goal-directed action.The examples contrast spatial reasoning and goal-directedness assessments.Source: §H.2 · pp. 24–25

H.3 Trajectory Visualization of ScienceWorld

  • In ScienceWorld scene 7, CGI earns higher scores in fewer steps than the base model, and more action refinements correspond to greater efficiency.Figure 8 plots scores from 0 to 100 over time steps from 0 to T.Source: §H.3 · p. 25, Figure 8 · p. 25

Related papers

Loading 2503.16024v2…