Symbolic Learning Enables Self-Evolving Agents

Motivation

• Manual decomposition and customization make current language-agent development engineering-centric and difficult to optimize from datasets.This limits robustness and versatility when adapting agents to new tasks or data distributions.Source: §1 · p. 2
• Existing automatic methods often optimize isolated prompts, tools, or nodes, risking local optima rather than improving the complete agent system.The paper contrasts this with jointly optimizing all symbolic components.Source: §1 · p. 2

Agent Symbolic Learning

• Agent symbolic learning maps agent pipelines to neural computational graphs, nodes to layers, and prompts and tools to weights.It applies language-based analogues of loss, back-propagation, gradients, and weight updates.Source: §1 · pp. 2–3
• The framework jointly updates prompts, tools, pipeline structure, and connections, enabling holistic optimization and learning after deployment without requiring ground-truth loss generation.This supports self-evolving agents that learn from experience in the wild.Source: §1 · p. 3

Evaluation

• Proof-of-concept experiments on standard benchmarks and complex agentic tasks show that the framework can optimize prompts, tools, and overall agent pipelines from training data.The authors release the framework’s code and prompts for future data-centric agent-learning research.Source: §1 · p. 3

2.1 Language Models, Prompts, and Language Agents

• Language agents combine LLMs, prompts, tools, and multi-step pipelines to execute tasks beyond simple prompting.Prompts control model behavior, while tools and stacked pipelines support broader applications.Source: §2.1 · p. 3

2.2 From Automated Prompt Engineering to Agent Optimization

• Automated prompt and agent optimization methods include prompt-based approaches and search-based algorithms such as genetic search, DSpy, and GPTSwarm.These methods extend earlier efforts to automate prompt engineering.Source: §2.2 · pp. 3–4
• Existing agent optimizers generally update isolated prompts, tools, nodes, or combinations rather than the entire agent system.The paper links this separation to local-optimum problems for individual components.Source: §2.2 · p. 4
• Search-based methods are limited when task success cannot be expressed numerically and typically lack support for adding nodes or implementing new tools.Software development and creative writing are cited as examples of complex tasks that resist equation-based evaluation.Source: §2.2 · p. 4
• Agent symbolic learning is presented as holistic optimization of prompts, tools, nodes, and their arrangement within agents.LLM-backbone fine-tuning and inter-task transfer are described as orthogonal or complementary directions.Source: §2.2 · p. 4

3.1 Problem Formulation

• Agent symbolic learning maps the agent pipeline to a neural-network computational graph, nodes to layers, and prompts and tools to symbolic weights.The trajectory stores node inputs, outputs, prompts, and tool usage for gradient back-propagation.Source: §3.1 · p. 5
• Language loss evaluates the discrepancy between expected and actual outcomes through textual comments and a prompted numerical score rather than a conventional numerical loss equation.The loss can use ground-truth labels for supervised learning or evaluate outputs and trajectories without labels for unsupervised agent learning.Source: §3.1 · p. 5, §3.2 · p. 6

3.2 Agent Symbolic Learning Procedure

• The procedure records each node execution during a forward pass, computes language loss from the trajectory, and back-propagates language gradients from the last node to the first.Gradients are textual analyses and reflections used to update symbolic components.Source: §3.2 · p. 6
• Language gradients account for downstream node effects, linking local prompt and tool changes to the agent system’s overall success.This chaining is intended to reduce optimization toward isolated local optima.Source: §3.2 · p. 6
• PromptOptimizer edits prompt components, ToolOptimizer improves, deletes, or creates tools, and PipelineOptimizer adds, deletes, or moves pipeline nodes.These symbolic optimizers are implemented as carefully designed prompt pipelines operating in natural-language space.Source: §3.2 · pp. 6–7
• The framework retries illegal code-space updates up to three times, discards persistent errors, rolls back performance-degrading updates, and exposes learning-rate controls.A batched variant aggregates language gradients from separate examples before updating each node.Source: §3.2 · p. 7

4.1.1 Tasks

• The experiments cover HotpotQA, MATH, and HumanEval, plus creative writing and executable software development evaluated with task-specific metrics.Tools are disabled on the standard benchmarks; creative writing uses GPT-4 scores, while software development uses a 1–4 execution-capability score.Source: §4.1.1 · pp. 7–8

4.1.2 Baselines

• Baselines include GPTs, manually configured agents, DSPy, and Agents + AutoPE, with GPT-3.5 and GPT-4 evaluations initialized from the Agents baseline.DSPy is not applicable to complex tasks whose metrics cannot be expressed as equations or code.Source: §4.1.2 · p. 8

4.2 Results

• Agent symbolic learning consistently improves over compared methods on HotpotQA, MATH, and HumanEval, with especially large gains on MATH.The framework is described as more robust than Agents w/ AutoPE and DSPy, which show unstable gains across cases.Source: §4.1.1 · p. 7, §4.2 · p. 8
• The approach significantly outperforms all baselines on software development and creative writing, with larger gaps than on conventional LLM benchmarks.It also exceeds tree-of-thought on creative writing and discovers plan, writing, and revision stages with optimized prompts.Source: §4.2 · pp. 8–9, Table 2 · p. 8, Table 3 · p. 8

4.3 Case Study & Analysis

• A creative-writing case study shows the framework performing prompt engineering and agent-pipeline design in a manner resembling human expert development.Figure 3 presents the optimization dynamics for this case study.Source: §4.3 · p. 9, Figure 3 · p. 9
• Agent initialization has a non-negligible effect on final performance: simple initialization is generally helpful, whereas over-engineered initialization tends to be unstable.The authors also report more significant and stable success on complex real-world tasks than on standard benchmarks.Source: §4.3 · p. 9

• Agent symbolic learning jointly optimizes prompts, tools, and the agent pipeline using language-based loss, gradients, and optimizers.The framework is inspired by standard connectionist learning procedures and enables agents to learn from data after deployment.Source: §5 · p. 9
• Proof-of-concept experiments show that the framework can effectively optimize agents across different task complexities.The authors position it as an early attempt to optimize agents solving complex real-world tasks with sophisticated pipelines.Source: §5 · p. 9

• The Agents framework stacks prompts and tools into complex pipelines and defines the agent system through a configuration file.This representation makes update operations easier for the symbolic optimizers.Source: A Implementation Details · p. 14

• Table 4 presents a prompt template for the language loss function.Source: Table 4 · p. 14
• Table 5 presents a prompt template for gradient back-propagation.Source: Table 5 · p. 15
• Table 6 presents prompt templates for optimizers.Source: Table 6 · p. 16