PaperlayerView exact revision on arXiv

Source-linked AI summary

SCOPE: Prompt Evolution for Enhancing Agent Effectiveness

Zehua Pei, Hui-Ling Zhen, Shixiong Kai, Sinno Jialin Pan, Yunhe Wang, Mingxuan Yuan, Bei Yu

arXiv:2512.15374v2cs.AI

TL;DR

LLM agents often have access to useful context but lack mechanisms for static prompts to manage it, causing corrective and enhancement failures. SCOPE treats context management as online prompt optimization, synthesizing trace-based guidelines and routing them through tactical and strategic memory. On HLE, SCOPE raises task success from 14.23% to 38.64% without human intervention.

  • Problem

    Static prompts do not adapt to dynamic context, producing Corrective and Enhancement Failures that existing task-level methods do not resolve during execution.

  • Method

    SCOPE synthesizes guidelines from execution traces, routes them to tactical or strategic memory, optimizes strategic memory, and evolves parallel prompts from distinct perspectives.

  • Results

    38.64% versus 14.23%: SCOPE more than doubles task success on the expert-level HLE benchmark without human intervention.

  • Takeaways & Limitations

    SCOPE supports step-level adaptation and per-agent prompt optimization by integrating execution-derived guidelines directly into agent prompts.

  • Takeaways & Limitations

    Existing comparison methods operate at task-level granularity, while the supplied discussion does not establish broader deployment limits for SCOPE.

Abstract

from arXiv · show

Large Language Model (LLM) agents are increasingly deployed in environments that generate massive, dynamic contexts. However, a critical bottleneck remains: while agents have access to this context, their static prompts lack the mechanisms to manage it effectively, leading to recurring Corrective and Enhancement failures. To address this capability gap, we introduce Self-evolving Context Optimization via Prompt Evolution (SCOPE). SCOPE frames context management as an \textit{online optimization} problem, synthesizing guidelines from execution traces to automatically evolve the agent's prompt. We propose a Dual-Stream mechanism that routes guidelines between tactical memory (immediate error correction) and strategic memory, which is continuously refined through conflict resolution, subsumption pruning, and consolidation. To maximize strategy coverage, Perspective-Driven Exploration evolves multiple parallel prompts guided by distinct optimization perspectives. Experiments on the HLE benchmark show that SCOPE improves task success rates from 14.23\% to 38.64\% without human intervention. We make our code publicly available at https://github.com/JarvisPei/SCOPE.

1 Introduction

SCOPE addresses a gap in agentic systems: agents receive increasingly complex context, but static prompts do not adapt to manage it. It evolves prompts from execution traces to correct errors, improve strategies, and raise benchmark success.

  • Problem: Static prompts produce Corrective Failures and Enhancement Failures when agents cannot learn from dynamic execution context.Corrective failures involve treating actionable errors as generic alarms, while Enhancement Failures involve persisting with suboptimal strategies without errors.
  • Approach: SCOPE turns context management into automatic online optimization by synthesizing guidelines from execution traces and integrating them into prompts.The framework treats repeated agent execution as an opportunity for prompt evolution rather than relying on manual prompt engineering.
  • Approach: SCOPE routes guidelines through tactical and strategic memory, then uses memory optimization and multiple perspectives to maintain diverse, effective strategies.Tactical memory supports task-specific correction, while strategic memory supports persistent optimization across executions.

2 Observations and Motivation

The paper identifies execution-time context-management failures and argues that existing methods cannot adapt prompts during tasks. SCOPE instead treats prompts as evolvable parameters updated from execution traces.

  • Failure modes: Corrective Failures arise when agents ignore actionable error information, producing repeated tool misuse and sometimes fabricated data.The analysis observed over 70 instances of agents repeating invalid tool actions despite error messages listing valid usage.
  • Failure modes: Enhancement Failures arise when agents miss optimization opportunities, such as continuing single-keyword searches despite context suggesting useful synonyms.Agents can also misinterpret domain concepts when static prompts provide no mechanism for additional verification.
  • System complexity: Long, heterogeneous agent trajectories require optimization tailored to individual agent roles and their distinct failure patterns.Successful GAIA trajectories average 16.4 steps, with complex tasks exceeding 30 turns and specialized agents exhibiting different errors.
  • Existing limitations: Static, offline, and task-level online methods cannot adapt prompts during execution while controlling memory quality and strategy diversity.Existing methods either optimize before deployment, update only between tasks, or append feedback without integrating corrections into instructions.
  • Prompt evolution: SCOPE updates prompts online by synthesizing natural-language guidelines from execution traces and integrating them as evolvable prompt parameters.Because prompt space is discrete, traces provide learning signals for guideline-based rather than gradient-based updates.

3 Methodology

SCOPE evolves agent prompts online from execution traces through guideline synthesis, dual-stream memory routing, memory optimization, and perspective-driven exploration.

  • 3.1 Guideline Synthesis: SCOPE synthesizes candidate guidelines from execution traces, using corrective or enhancement rubrics and best-of-N selection to choose one guideline.Corrective synthesis targets errors, while enhancement synthesis identifies inefficiencies after apparently successful executions.
  • 3.2 Dual-Stream Routing: Guidelines are classified by generality and confidence, then routed to tactical memory for current-task use or strategic memory for persistent cross-task guidance.Tactical guidelines include low-confidence strategic classifications; strategic guidelines require confidence at least c_thresh.
  • 3.2 Dual-Stream Routing: The prompt is updated by combining the initial prompt with strategic and tactical memories: θt+1 = θbase ⊕ Mstrat ⊕ Mtact.Strategic guidelines persist across tasks, whereas tactical guidelines remain valid only for the current task.
  • 3.3 Memory Optimization: Memory optimization prevents strategic-memory growth from diluting attention by resolving conflicts, pruning subsumed guidelines, and consolidating similar ones.The optimizer applies these three steps after adding a new strategic guideline.
  • 3.4 Perspective-Driven Exploration: Perspective-driven exploration evolves K parallel prompts with distinct perspectives and selects the best task result among them.This design supports diverse strategies for different problem types.

4 Experiments

Across three deep-research benchmarks, SCOPE is evaluated against static and prompt-optimization baselines, with component, placement, model-choice, overhead, and domain analyses. It achieves broad performance gains, including more than doubling HLE baseline performance, while adding limited API overhead.

  • 4.2 Overall Performance and Ablation: 38.64% vs. 14.23% on HLE, SCOPE more than doubles the static baseline and establishes state-of-the-art performance across all three benchmarks.Results use Pass@2 over two independent runs per task.
  • 4.2 Overall Performance and Ablation: +10.91% from Perspective-Driven Exploration is the largest ablation gain, exceeding the Guideline Generator’s initial +4.85%.The ablation supports maintaining distinct Efficiency and Thoroughness streams rather than relying on one strategy.
  • 4.3 Guideline Placement: System Prompt vs. User Prompt: 46.06% accuracy results from system-prompt placement on GAIA, despite 227 step-limit cases versus 130 with user-prompt placement.All placements reduce errors by 15–42% relative to baseline; system placement achieves the best accuracy.
  • 4.4 Meta-Agent Model Choice: Nearly identical performance across meta-agent model configurations, within 1.2%, indicates deployment model choice need not determine accuracy.Gemini-2.5-Pro generated 46% more guidelines than GPT-4.1, but selection and optimization filtered for comparable quality.
  • 4.5 SCOPE Enhances Robustness in Long-Horizon Domains: 4.2% more API calls accompanies 152% more prompt tokens and 38.6% fewer completion tokens, with asynchronous synthesis adding zero wall-clock latency per step.On 100 HLE samples, SCOPE scored 36.00 versus 16.00 for the reflection baseline, which used 108% more API calls and added about 10 seconds per step.
  • 4.5 SCOPE Enhances Robustness in Long-Horizon Domains: 43.2% versus 14.9% in Biology/Medicine and 50.3% versus 14.1% in Chemistry show especially large HLE gains in knowledge-intensive domains.On GAIA Level 3, SCOPE reaches 30.8% versus 23.1% for the baseline.
  • 4.5 SCOPE Enhances Robustness in Long-Horizon Domains: +8.2 points on SWE-bench Verified and +17.5 points with GPT-5.4 indicate generalization to coding and stronger frontier models.These additional experiments are reported in the appendix.

5 Analysis

SCOPE’s analysis links agent failures to unexploited execution feedback and examines how corrective, enhancement, and perspective-specific strategies affect behavior. The evidence emphasizes proactive optimization, direct guideline adoption, and complementary perspectives.

  • 5 Analysis: Baseline agents can underperform raw Gemini-2.5-Pro, scoring 14.23% versus 18.76% on HLE and 14.00% versus 19.00% on DeepSearch.The baseline accumulated 1,714 errors across 165 GAIA tasks, illustrating wrapper-induced fragility.
  • 5 Analysis: 15–42% error reductions across placement strategies show that corrective synthesis turns failure traces into real-time debugging guidance.Examples include converting a NameError into a rule to define all variables in code snippets.
  • 5 Analysis: 61% of synthesized guidelines are enhancement guidelines, showing SCOPE primarily codifies successful patterns rather than merely debugging errors.This proactive mode supports continued improvement when error rates are low.
  • 5 Analysis: A subsequent output reproducing a guideline’s recommended phrasing provides evidence that synthesized guidelines are integrated into agent decision-making.The paper calls this direct linguistic transfer language adoption.
  • 5 Analysis: 33.94% intersection between Efficiency and Thoroughness solutions indicates that approximately 23% of solved problems are unique to one perspective.Efficiency performs better on Level 3 tasks, while Thoroughness excels on Level 2.
  • 5 Analysis: Efficiency responds to HTTP 403 errors by failing over quickly, whereas Thoroughness seeks alternate sources, providing complementary retrieval strategies.The distinct responses support coverage of both time-constrained and deep-retrieval tasks.

6 Related Work

Related work provides context through retrieval and compression, offline prompt search, and persistent-memory or feedback-based learning. SCOPE is distinguished by evolving prompts online during execution at step-level granularity.

  • 6 Related Work: RAG and compression methods optimize what context agents receive, whereas SCOPE optimizes how agents process context by evolving prompts.The comparison distinguishes context provision from context-management behavior.
  • 6 Related Work: Offline prompt optimizers such as OPRO, DSPy, and GEPA search before deployment, while SCOPE adapts prompts during execution using step-level feedback.SCOPE complements strong static initializations with online adaptation.
  • 6 Related Work: Memory and feedback methods build strategy libraries or feed corrections into context, but related approaches face task-level granularity and one-for-all limitations in agentic settings.SCOPE instead supports online, per-agent prompt evolution.

7 Conclusion

The paper concludes that SCOPE enables agents to evolve prompts online by learning from execution traces. Experiments on HLE and GAIA show substantial gains over static and existing optimization methods.

  • 7 Conclusion: SCOPE integrates trace-derived guidelines directly into prompts, enabling step-level adaptation and per-agent optimization.The framework treats execution traces as learning signals rather than relying on manually engineered static prompts.
  • 7 Conclusion: SCOPE more than doubles success rates in expert-level domains while outperforming static baselines and existing methods on HLE and GAIA.The conclusion presents prompt evolution as an alternative to static prompt engineering.

A Detailed Analysis of Baseline Failure Modes

Baseline agents often possess relevant information but fail to convert it into correct, efficient, and adaptive actions. The analysis identifies recurring corrective, enhancement, search, context-management, reasoning, and safety failures.

  • Failure Taxonomy: Over 1.5 million execution-trace lines reveal two failure modes: error-triggered Corrective Failures and optimization-opportunity Enhancement Failures.Corrective Failures involve ignoring actionable error feedback, while Enhancement Failures involve persisting with suboptimal behavior despite no explicit error.
  • Corrective Failures: Agents repeatedly ignore explicit tool constraints, including valid identifiers, argument schemas, and environment restrictions, producing error loops.Observed examples include incorrect tool names, hallucinated parameters, and repeated attempts to use forbidden libraries.
  • Enhancement Failures: Agents miss optimization opportunities through rigid search terminology, redundant verification, sequential querying, unnecessary delegation, and excessive verbosity.These patterns consume additional steps and tokens without necessarily improving answer quality.
  • Context Management: Long multi-turn contexts can exceed 100,000 tokens, increasing latency and diluting relevant instructions when active context pruning is absent.The failure is attributed to carrying full interaction histories through extended sessions.
  • Reasoning and Access Failures: Knowledge-level errors and web-access failures persist despite syntactically valid behavior, including statistical misconceptions, repeated blocked-site retries, and failure to pivot sources.These behaviors can lead to incorrect conclusions or task failure even when tools operate normally.
  • Safety Failures: When tools fail or files are inaccessible, agents may fabricate hypothetical data and present confident answers grounded in fiction.This behavior creates a safety risk because the resulting output can appear legitimate despite lacking retrieved evidence.

B.1 Agent Specifications

The appendix specifies the hierarchical agent system, evaluation benchmarks, and SCOPE’s guideline-memory configuration. It also describes domain organization, memory optimization, agent-specific specialization, and step-level prompt updates.

  • Agent Architecture: The baseline hierarchy uses a planning agent plus specialized web-search, analyzer, and browser agents with distinct tools and step limits.The planner orchestrates decomposition, while subordinate agents handle retrieval, analysis, and web interaction.
  • Benchmarks: SCOPE evaluates hierarchical agents on HLE, GAIA, and DeepSearch, covering expert reasoning, multi-tool real-world tasks, and multi-hop information synthesis.The benchmarks include 2,500 HLE questions, 165 GAIA validation tasks, and DeepSearch tasks involving retrieval and cross-document synthesis.
  • Guideline Organization: Learned guidelines are organized into seven domains, including tool usage, data validation, error handling, efficiency, analysis methodology, safety, and general guidance.The domains support organization and subsequent memory optimization.
  • Memory Optimization: When a domain exceeds 10 guidelines, SCOPE resolves conflicts, prunes subsumed rules, and consolidates similar guidance toward a compact set of eight.The optimization target preserves capacity for future learning while maintaining a high-quality memory.
  • Memory Optimization: A worked example reduces 11 efficiency guidelines to 5 while preserving essential guidance and leaving room for continued learning.The example shows the optimization pipeline operating after the domain threshold is exceeded.
  • Agent-Specific and Step-Level Optimization: Per-agent optimization produces specialized guideline sets, while step-level updates enable corrective guidance during execution rather than only after task completion.The browser agent accumulates web-specific efficiency guidance, and more than 40% of analyzed errors occur at step 1.

H.1 Quantitative Behavioral Metrics

Guideline placement changes the balance among accuracy, errors, timeouts, efficiency, and memory stability. System-prompt placement achieves the strongest accuracy, whereas split placement performs worst and user-prompt placement shows greater churn.

  • User Prompt: User-prompt placement produces 1,000 errors and 130 timeouts but can encourage premature termination, verbose compliance, and contexts exceeding 7,600 tokens.Its lower error and timeout counts do not translate into the best accuracy.
  • Efficiency: All guideline placements reduce total steps by 24–33% versus baseline, while hybrid placement achieves the fewest total steps and best accuracy-per-step efficiency.The table caption identifies hybrid as the efficiency leader, whereas system placement leads accuracy.
  • System Prompt: System-prompt guidelines act as implicit background guidance, allowing adaptive exploration with fewer explicit checks and higher error tolerance.The proposed interpretation contrasts constitutional guidance with direct user-prompt instructions.
  • Guideline Dynamics: System placement loads 2,962 strategic guidelines with 1,379 updates, whereas user placement has 1,990 loaded strategic guidelines and 1,648 updates, indicating greater churn.The comparison describes more stable persistent guidance under system placement.
  • Split Placement: Split placement performs worst at 35.76%, with mixed strategic and tactical sources potentially creating conflicting priorities and over-caution.The authors associate the lowest error count with missed opportunities under mixed signals.

I.1 Strategy Patterns by Scenario

SCOPE’s parallel perspectives generate complementary strategies for blocked access, tool failures, and planning complexity. Their ensemble combines efficiency-oriented fast recovery with thoroughness-oriented information recovery and validation.

  • Blocked Access: Efficiency and Thoroughness diverge on blocked access: Efficiency switches agents quickly, while Thoroughness searches alternate sources and recovery tools.The two streams therefore prioritize latency reduction and information recovery, respectively.
  • Tool Failures: For general tool failures, Efficiency favors immediate tool switching, whereas Thoroughness repairs inputs or tries alternative parameters.The strategies encode different responses to the same execution failure.
  • Planning Strategy: Efficiency limits plan complexity, while Thoroughness encourages exhaustive case enumeration and validation before proceeding.The streams therefore trade concise execution paths against comprehensive checking.
  • Global Ensemble: The Global Ensemble outperforms either individual stream by selecting between complementary strategies for heterogeneous task requirements.Efficiency is described as stronger for time-constrained and long-horizon tasks, while Thoroughness suits deep retrieval and careful validation.
  • Meta-Agent Models: Gemini generates 46% more guidelines than GPT-4.1 but achieves only 0.61% better performance, indicating that guideline quality matters more than quantity.Best-of-N selection and memory optimization are described as filtering useful guidelines across model configurations.

J.2 Category Distribution Analysis

Guideline-generation patterns differ substantially across models, but these differences do not translate into meaningful performance or confidence differences across configurations.

  • Category Distribution Analysis: Gemini generates the most guidelines across nearly all semantic categories, including 26 error-handling and 32 efficiency guidelines.It is also the only configuration generating safety-related guidelines, with 2 such guidelines.
  • Confidence Score Distribution: Average confidence scores are nearly identical: GPT-4.1 0.939, Gemini-2.5-Pro 0.940, and Same Model 0.942.The approximately 0.94 scores suggest comparable self-assessed guideline reliability across configurations.
  • Category Distribution Analysis: The results support choosing meta-agent models according to cost or latency rather than accuracy concerns.SCOPE’s selection and optimization mechanisms appear to filter useful guidelines across differing generation patterns.

K.1 Single-Agent Evaluation: SWE-bench Verified

SCOPE generalizes to a single-agent coding system and improves SWE-bench Verified performance, while additional reflection or step budget alone produces much smaller gains. Further evaluations show benefits across stronger models and sensitivity to strategy diversity and guideline routing.

  • Single-Agent Evaluation: SWE-bench Verified: +8.2 points on SWE-bench Verified confirms SCOPE’s gains in a single-agent coding setup with a different architecture.The evaluation used miniswe-agent v2 with GPT-5-mini, requiring only a natural step boundary such as tool-call completion.
  • Single-Agent Evaluation: SWE-bench Verified: Per-step reflection without guideline evolution improves performance by only +0.6, far below SCOPE’s +8.2-point gain.This comparison separates SCOPE’s improvement from the effect of adding extra computation alone.
  • Stronger Frontier Model Evaluation: +17.5 points in accuracy and 56% fewer errors on GAIA show that SCOPE remains effective with GPT-5.4 despite its stronger baseline performance.GPT-5.4 scores 49.80 versus GPT-4.1’s 31.35, yet accumulates 1,083 errors across 165 tasks.
  • Sensitivity Analysis: +10.91% from K = 1 to K = 2 indicates that parallel perspective streams contribute through strategy diversity.By contrast, increasing Best-of-N from N = 2 to N = 3 adds only +0.88% at 50% extra compute, while N = 2 captures most of the benefit over N = 1 (+3.59%).
  • Step Budget Ablation: 50% more steps yields only +0.40% on GAIA and zero on DeepSearch, whereas SCOPE cuts step-limit hits by 45%.Step-limit hits barely change from 255 to 250, indicating repeated errors persist despite a larger budget.
  • Classifier Threshold Sensitivity: A confidence threshold of 0.85 produces the clearest performance peak, while only 320 of 1,570 synthesis attempts are promoted to strategic memory.Lower thresholds promote noisy guidelines, whereas higher thresholds prevent useful guidelines from persisting; 79.2% of rejections are duplicate detection.
Loading 2512.15374v2…