PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on the Optimization of Large Language Model-based Agents

Shangheng Du, Jiabao Zhao, Jinxin Shi, Zhentao Xie, Xin Jiang, Yanhong Bai, Liang He

arXiv:2503.12434v2cs.AI

TL;DR

LLM-based agents require optimization specialized for long-term planning, dynamic interaction, and complex decision-making, but existing work lacks a holistic systematic review. This survey organizes agent optimization into parameter-driven and parameter-free approaches, reviews their methods and evaluation resources, and identifies challenges including limited adaptability and nonstandardized evaluation.

  • Problem

    Existing LLM optimization methods do not fully address agent-specific needs such as long-term planning, dynamic environmental interaction, and complex decision-making, while a holistic systematic review is lacking.

  • Method

    The survey categorizes LLM-based agent optimization into parameter-driven fine-tuning, reinforcement learning, and hybrid methods, alongside parameter-free strategies, and reviews datasets, benchmarks, applications, and challenges.

  • Results

    The survey provides a systematic overview of LLM-based agent optimization methods and compares their workflows, evaluation resources, applications, challenges, and future directions.

  • Takeaways & Limitations

    Understanding agent optimization requires considering both parameter updates and behavior-level strategies across trajectory construction, learning, interaction, evaluation, and coordination.

  • Takeaways & Limitations

    Current methods can struggle with sparse rewards, large action spaces, overfitting, computational cost, and poor transfer to unseen domains, while evaluation lacks standardized metrics.

Abstract

from arXiv · show

With the rapid development of Large Language Models (LLMs), LLM-based agents have been widely adopted in various fields, becoming essential for autonomous decision-making and interactive tasks. However, current work typically relies on prompt design or fine-tuning strategies applied to vanilla LLMs, which often leads to limited effectiveness or suboptimal performance in complex agent-related environments. Although LLM optimization techniques can improve model performance across many general tasks, they lack specialized optimization towards critical agent functionalities such as long-term planning, dynamic environmental interaction, and complex decision-making. Although numerous recent studies have explored various strategies to optimize LLM-based agents for complex agent tasks, a systematic review summarizing and comparing these methods from a holistic perspective is still lacking. In this survey, we provide a comprehensive review of LLM-based agent optimization approaches, categorizing them into parameter-driven and parameter-free methods. We first focus on parameter-driven optimization, covering fine-tuning-based optimization, reinforcement learning-based optimization, and hybrid strategies, analyzing key aspects such as trajectory data construction, fine-tuning techniques, reward function design, and optimization algorithms. Additionally, we briefly discuss parameter-free strategies that optimize agent behavior through prompt engineering and external knowledge retrieval. Finally, we summarize the datasets and benchmarks used for evaluation and tuning, review key applications of LLM-based agents, and discuss major challenges and promising future directions. Our repository for related references is available at https://github.com/YoungDubbyDu/LLM-Agent-Optimization.

1 Introduction

This survey addresses the gap in specialized optimization for LLM-based agents, whose autonomous decision-making, long-term planning, and adaptability needs are not fully covered by general LLM techniques. It organizes the field into parameter-driven and parameter-free approaches and situates them within a broader review of frameworks, evaluation, applications, challenges, and future directions.

  • General fine-tuning methods improve instruction following but do not fully address agents’ decision-making, long-term planning, and adaptability challenges.
  • The survey categorizes LLM-based agent optimization into parameter-driven and parameter-free strategies.Parameter-driven methods adjust LLM parameters, while parameter-free methods optimize agent behavior without parameter updates.
  • Parameter-driven optimization covers conventional fine-tuning, reinforcement learning, and hybrid fine-tuning strategies.The review examines trajectory construction, fine-tuning, reward functions, preference alignment, and optimization algorithms.
  • The survey claims to be the first systematic review dedicated specifically to LLM-based agent optimization rather than general LLM optimization or individual agent components.
  • Its scope includes optimization methods for problem-solving and task execution, while excluding work centered on general LLM efficiency, role-playing, or dialogue.The selection focuses on studies since 2022.
  • The paper additionally summarizes open-source frameworks, datasets, evaluation benchmarks, practical applications, challenges, and future directions.

2 Background

This background section places LLM-agent optimization within the progression from classical reinforcement learning to preference-based learning and parameter-efficient LLM fine-tuning. It highlights methods for learning policies, aligning behavior, and adapting pretrained models to task-specific requirements.

  • Value-based reinforcement learning estimates action values, whereas policy-based methods directly optimize policies through reward gradients.PPO is described as improving stability through constrained updates.
  • RLHF incorporates human feedback into policy optimization, while DPO removes the need for explicit reward modeling in preference-based learning.
  • Supervised fine-tuning adapts pretrained LLMs to task-specific requirements using labeled data, with instruction tuning training on instruction-response pairs.
  • Parameter-efficient fine-tuning updates only a small subset of parameters to reduce computational cost while preserving performance.Representative methods include P-Tuning, LoRA, and QLoRA.

3 Parameter-driven Optimization of LLM-based Agents

Parameter-driven optimization focuses on using the model for complex agent tasks through conventional fine-tuning, reinforcement learning, and hybrid parameter-tuning strategies. Conventional fine-tuning centers on constructing and refining task-specific trajectories before adapting the agent, while its effectiveness depends on data quality and is limited by static training objectives.

  • Parameter-driven agent optimization addresses task-specific decision-making and multi-step planning rather than only improving general model capabilities.
  • The survey organizes parameter-driven optimization into conventional fine-tuning, reinforcement learning, and hybrid optimization.
  • 3.1 Conventional Fine-Tuning-based Optimization: Conventional fine-tuning generally constructs high-quality agent trajectories and then adapts LLM parameters through instruction tuning or parameter-efficient methods.Trajectory construction and agent fine-tuning form the two main stages of the workflow.
  • Trajectory Data Construction: Trajectory quality is improved through generation, evaluation and filtering, and the use of low-quality samples before fine-tuning.Filtering may rely on environment feedback, human or rule-based criteria, or model-based evaluation.
  • Trajectory Data Construction: Trajectory construction draws on expert annotations, strong LLM generation, self-exploration through environment interaction, and multi-agent collaboration.These approaches trade off data quality, scalability, and construction complexity.
  • Trajectory-based Fine-Tuning: Standard supervised fine-tuning performs full-parameter adaptation on high-quality data to align pretrained models with task requirements.Behavior cloning is included because it follows the same optimization paradigm.
  • Summary: Conventional fine-tuning can improve task alignment and adaptability to specific objectives, but curated-data dependence, overfitting, static objectives, and absent interactive feedback constrain dynamic generalization.The survey notes that these methods may accumulate errors and struggle with real-time behavioral refinement.
  • Reinforcement Learning-based Optimization: Preference-based reinforcement learning constructs ranked outputs and applies DPO to optimize agents toward desired outputs and task-specific requirements.Preference data can be formed by evaluating diverse trajectories with task success, environment feedback, or other metrics.

4 Parameter-free Optimization of LLM-based Agents

Parameter-free optimization improves LLM-based agent behavior without modifying model parameters, using prompts, feedback, tools, retrieval, experience, and collaboration. The survey organizes these approaches by how they adapt behavior and information flow.

  • Overview: Parameter-free optimization changes agent behavior without modifying LLM parameters, making it suitable for resource-constrained settings.The survey categorizes methods by optimization strategy rather than agent architecture.
  • Experience-based Optimization: Experience-based methods use trajectories, historical data, or accumulated knowledge to refine strategies and support long-term decisions.Memory modules often organize past successes and failures for later retrieval.
  • Feedback-based Optimization: Feedback-based methods use self-reflection, external evaluation, or meta-prompts to trigger iterative behavioral improvement at inference time.The three categories are self-reflection, external feedback, and meta-prompt optimization.
  • Feedback-based Optimization: Self-reflection and external-feedback systems convert outcomes, critiques, or corrections into adjustments that improve strategies, robustness, or tool use.Examples include Reflexion, checker-agent feedback in SAGE, critic models in InteRecAgent, and counterfactual feedback in COPPER.
  • Meta-Prompt Optimization: Meta-prompt optimization updates global instructions from failed trials or task evaluations, while hierarchical methods revise prompts and workflows for multi-step reasoning.MPO additionally trains a meta-planner with SFT and DPO for high-level planning.
  • Tools and Retrieval: Tool-use optimization improves task decomposition, invocation, and error handling, whereas retrieval grounds reasoning in dynamically acquired evidence from heterogeneous sources.Retrieval methods include pipeline optimization, self-reflection, planning support, and domain-specific systems such as PaperQA.
  • Multi-Agent Collaboration: Multi-agent collaborative optimization distributes roles and coordinates information flow so agents can jointly plan and execute complex tasks.Frameworks use role assignment, dynamically constructed networks, and iterative proposal and evaluation to support decomposition.

5 Open-source Frameworks and Stacks

Open-source agent frameworks provide reusable infrastructure for planning, memory, tool invocation, execution control, and collaboration. The survey distinguishes orchestration-centered systems from retrieval-centric and collaboration-centered frameworks.

  • Framework Infrastructure: Open-source frameworks reduce engineering complexity by providing modular abstractions for planning, memory management, tool invocation, and execution control.LangChain and AutoGen are representative systems supporting reusable agent infrastructure.
  • Core Components: Most frameworks share planning, tool-use, memory, and reflection components that support decomposition, external operations, contextual recall, and error identification.These components allow decisions to incorporate prior interactions and enable agents to evaluate outputs.
  • Framework Coverage: The survey selects widely used and actively maintained open-source frameworks while excluding low-code and discontinued platforms.The selection is summarized in Table 8.
  • Framework Categories: Orchestration-centered systems such as LangChain and LangGraph emphasize modular composition and graph-structured control for multi-step workflows.They support chained LLM calls, external tools, and fine-grained state tracking.
  • Framework Categories: LlamaIndex is retrieval-centric with flexible data interfaces, whereas AutoGen, CrewAI, Camel-AI, and AgentScope emphasize role-based interaction and coordinated problem solving.AutoGen supports multi-agent conversation, tool execution, and iterative debugging; CrewAI emphasizes coordinated roles.

6 Evaluation and Datasets for LLM-Based Agents

Agent evaluation combines human and automated assessment across objective, subjective, and interactive settings, supported by diverse task datasets and multi-task benchmarks. The survey also catalogs trajectory datasets for tuning and broad application domains.

  • Evaluation Methodologies: Human evaluation remains necessary for subjective tasks but is costly, difficult to scale, and variable across judgments.It is used for qualities such as creativity, coherence, emotional intelligence, and specialized reasoning.
  • Evaluation Methodologies: Automated evaluation is preferred for scalability and reproducibility, using static metrics, LLM judges, execution feedback, and interactive protocols.Multi-dimensional protocols assess reasoning quality, tool-use reliability, and multi-step decision-making.
  • Evaluation Datasets: Evaluation datasets cover general domains including programming, question answering, and multimodal tasks, with action space distinguishing natural-language and code actions.These categories are summarized in the survey’s dataset tables.
  • Domain Tasks: Domain-specific evaluations measure capabilities in mathematics, QA, code, tool use, web interaction, environments, and multimodal reasoning with task-specific metrics.Examples include EM, F1, Pass@k, execution success, task success, navigation efficiency, reward, and visual-question-answering accuracy.
  • Domain Tasks: Environment-interaction benchmarks assess multi-step adaptation in dynamic physical or virtual settings, while multimodal benchmarks combine image, text, and video information.ScienceWorld, ALFWorld, ALFRED, VQA-V2, A-OKVQA, and ScienceQA are representative datasets.
  • Multi-Task Benchmarks: Multi-task benchmarks combine heterogeneous domains to test generalization across programming, web navigation, tool use, reasoning, embodied interaction, and games.AgentBench, AgentEval, Just-Eval, StreamBench, and AgentBoard use broad task suites and, in some cases, unified ReAct-style formats.
  • Tuning Datasets: Fine-tuning datasets provide trajectories for skills including tool use, web interaction, programming, mathematical reasoning, and embodied decision-making.AgentInstruct, AgentBank, Agent-FLAN, AgentOhana, FireAct, ToRA-CORPUS, AgentTraj, and SMART-Trajectory combine trajectories from diverse sources.

7 Application

LLM-based agents are applied across healthcare, scientific research, embodied intelligence, robotics, and finance. These applications combine domain knowledge, interaction, planning, memory, reflection, and multimodal or financial signals.

  • Healthcare: Healthcare agents span medical question answering, diagnosis, treatment planning, and simulation, progressing from domain fine-tuning to dynamic clinical and multi-agent systems.Simulation frameworks support realistic physician–patient interactions, while systems such as KG4Diagnosis extend multi-agent diagnosis.
  • Science: Scientific agents integrate knowledge retrieval, hypothesis generation, and automated experiment design to support biological and chemical discovery.CellAgent and BioDiscoveryAgent use multi-agent coordination and iterative self-optimization.
  • Science: Scientific workflows remain constrained by high computational costs, long knowledge-retrieval cycles, and complex coordination across experimental stages.The survey identifies efficiency and scalability as needed directions for scientific agent systems.
  • Embodied Intelligence: Embodied and robotic agents combine perception, memory, planning, and multimodal inputs to act in physical or simulated environments.RL-based systems such as SayCan ground LLM reasoning in real-world robotic actions.
  • Finance: Financial agents support market prediction, trading, risk management, and simulation by processing heterogeneous financial signals.Architectures use hierarchical memory and reflective evaluation to address shifting market patterns.

8 Challenges & Future Directions

The survey identifies challenges in adapting optimization algorithms, standardizing evaluation, controlling deployment costs, ensuring safety, and jointly optimizing multi-agent systems. It proposes hybrid optimization, distribution alignment, standardized benchmarks, efficient inference and environments, layered defenses, and joint multi-agent optimization as future directions.

  • Algorithm Adaptability and Efficiency: Current optimization methods struggle to balance algorithmic efficiency with task-specific adaptability because of sparse rewards, large action spaces, overfitting, and domain mismatch.PPO is computationally expensive, while DPO is mainly suited to single-step optimization and can be limited for multi-step interactive tasks.
  • Algorithm Adaptability and Efficiency: Hybrid reinforcement learning and fine-tuning approaches could improve both adaptability and efficiency.The survey also highlights reward-design optimization and efficiency-focused algorithms as ways to reduce computational costs while maintaining adaptability.
  • Algorithm Adaptability and Efficiency: Distribution alignment and domain adaptation could help agents transfer knowledge across tasks and dynamic environments.These techniques are proposed alongside hybrid optimization as potential responses to poor transfer under domain mismatch.
  • Standardized Evaluation Metrics: Evaluation lacks standardized metrics across mathematical reasoning, web navigation, and embodied AI, making fair comparison and optimization assessment difficult.Existing metrics mainly measure task completion rather than the extent or stepwise progress of optimization.
  • Standardized Evaluation Metrics: Unified benchmarks, stepwise progress metrics, and preference-based signals could enable more comprehensive evaluation beyond task completion.The survey presents standardized evaluation frameworks as a direction for fair comparison across diverse agent tasks.
  • Cost and Efficiency Constraints in LLM-Based Agents: Long-horizon and tool-augmented deployment incurs substantial financial, computational, latency, and token costs.Large reasoning models, multi-turn interaction, and extended reasoning chains amplify these deployment constraints.
  • Cost and Efficiency Constraints in LLM-Based Agents: Context optimization, adaptive memory, inference-efficient prompting, and scalable training environments could reduce redundant inference and environment-construction costs.These directions are presented as important for scaling agents to long-horizon and resource-intensive scenarios.
  • Safety: Expanded agent autonomy creates exposure to prompt injection, jailbreaks, and unsafe external actions from manipulated intermediate outputs.The survey identifies multi-agent defenses, filtering, fact-checking, consensus, and human validation as system-level mitigations.

9 Conclusion

The survey concludes by organizing LLM-based agent optimization into parameter-driven and parameter-free approaches, covering methods, data, applications, challenges, and future directions.

  • Conclusion: The survey categorizes LLM-based agent optimization into parameter-driven and parameter-free approaches.It covers fine-tuning, reinforcement learning, hybrid strategies, parameter-free methods, datasets, applications, challenges, and future research directions.
Loading 2503.12434v2…