PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Configure Agentic AI Systems

Aditya Taparia, Som Sagar, Ransalu Senanayake

arXiv:2602.11574v3cs.AI

TL;DR

Fixed or hand-tuned agent configurations struggle to adapt resource use across queries in a large combinatorial design space. ARC formulates configuration as an SMDP and learns query-specific choices with a lightweight hierarchical policy. Across reasoning, tool-use, and agentic benchmarks, ARC improves over budget-matched tool-augmented LLMs, including a 31.3% increase in average reasoning accuracy.

  • Problem

    Agentic systems commonly use static heuristics or templates that do not adapt workflows and resources to query difficulty, while the configuration space is combinatorial.

  • Method

    ARC models each query-specific configuration as a temporally extended SMDP option and uses hierarchical policies to select structure and prompts without updating the backbone LLM.

  • Results

    Across reasoning, tool-use, and agentic benchmarks, ARC consistently improves over budget-matched tool-augmented LLMs, increasing average reasoning accuracy by 31.3%.

  • Takeaways & Limitations

    Learning per-query configurations provides an alternative to one-size-fits-all agent designs across reasoning, tool-use, and agentic benchmarks.

  • Takeaways & Limitations

    The reported comparisons constrain base models to match the learned policies’ available per-query or maximum budgets.

Abstract

from arXiv · show

Configuring LLM-based agent systems involves choosing workflows, tools, token budgets, and prompts from a large combinatorial design space, and is typically handled today by fixed templates or hand-tuned heuristics that apply the same configuration regardless of query difficulty, leading to brittle behavior and wasted compute. To address this, we formulate agent configuration as a semi-Markov decision process (SMDP) where each configuration acts as a temporally extended option that determines how an agent system processes a query, and introduce introduce ARC (Agentic Resource & Configuration learner), a lightweight hierarchical policy that dynamically selects query-specific agent configurations. Across reasoning, tool-use, and agentic benchmarks, ARC consistently improves over budget-matched tool-augmented LLMs, increasing average reasoning accuracy by 31.3%, tool-use accuracy by 13.95%, and doubling τ-Bench (Airline) Pass^1 success from 9.0% to 18.0%. These results demonstrate that learning per-query agent configurations is a powerful alternative to "one size fits all" designs.

1 Introduction

ARC addresses the difficulty of configuring agentic systems across a combinatorial design space by learning query-specific configurations. It models configurations as temporally extended SMDP options and uses hierarchical reinforcement learning with a lightweight controller.

  • Agentic performance depends on workflows, tools, information routing, and context management in addition to the underlying LLM.
  • Static or kitchen-sink architectures waste compute and latency by applying heavy resources uniformly, while long contexts can degrade performance.
  • The configuration space becomes combinatorial even for modest systems, making brute-force search and manual tuning impractical.
  • ARC models each configuration as a temporally extended SMDP option because workflows vary in duration and computational cost.
  • ARC uses hierarchical reinforcement learning: a high-level policy selects workflows, tools, and budgets, while a low-level policy composes prompts without updating the backbone LLM.

2 Related Work

The paper connects LLM agents and prompt or workflow optimization with hierarchical reinforcement learning. Its position is that agent configuration can be treated as structured hierarchical control rather than only prompt engineering.

  • Large Language Agents: LLM-based agents extend language models with iterative decision-making, tool use, and multi-step interaction.
  • Prompt and Workflow Optimization: Prompt optimization has expanded from improving static instructions to optimizing workflows and context management.
  • Hierarchical Planning and Control: Hierarchical reinforcement learning decomposes complex tasks into high-level goals and low-level actions to improve exploration in large state-action spaces.
  • Hierarchical Planning and Control: ARC applies hierarchical control to selecting agent architectures, workflows, tools, budgets, and prompts rather than operating within fixed architectural constraints.

3 Methodology

ARC models query-specific agent configuration as an SMDP and decomposes configuration into structural and prompt decisions. It trains these policies with PPO, reward shaping, action masking, and elite-trajectory SFT, with theoretical guarantees for convergence and refinement.

  • Configuration Objective: For each query, ARC selects a configuration containing workflow, tools, token budgets, and prompts, then evaluates correctness against computational cost.
  • SMDP Formulation: Configurations are SMDP options because workflows consume variable numbers of LLM calls and have duration-dependent transitions and rewards.
  • Hierarchical Policy: The structure policy selects workflows, tools, and budgets, while the prompt policy sequentially composes instructions conditioned on that structural choice.
  • Training and Reward Design: PPO trains both policy factors with separate value networks, advantage normalization, entropy regularization, and a reward balancing correctness, duration, tokens, and tool use.
  • Training and Reward Design: The training pipeline stores RL episodes, filters elite high-reward trajectories, and applies SFT to consolidate successful strategies and improve consistency.
  • Action Masking: Action masking reduces the raw option space from 62,208 to 41,904 valid configurations, a 32.6% reduction.
  • Theoretical Guarantees: Theoretical results establish a unique optimal value function for the SMDP and concentration of the final policy on high-reward configurations after SFT.

4 Experiments

Experiments evaluate ARC against fixed, search-based, optimization, and reinforcement-learning baselines across reasoning, tool-use, and agentic benchmarks. ARC generally improves performance, efficiency, transfer across models, and reward, while results also expose domain- and task-specific boundaries.

  • Experimental Setup: ARC is evaluated across six benchmarks spanning reasoning, tool use, and agentic capabilities, against base-model, search, optimization, and RL baselines.The setup uses Qwen 2.5 7B and Gemini 2.5 Flash Lite, with Qwen emphasized in the main text.
  • RQ1: Performance: ARC achieves consistent improvements over baseline methods across the reasoning, tool-use, and agentic benchmark suites.The paper reports the strongest results on most reasoning and tool-use tasks, with τ-Bench results reported separately.
  • RQ1: Performance: 88.6% GSM8K accuracy with Qwen lets ARC outperform GEPA at 83.6% and RL Episodes at 85.2%.On DROP, ARC reaches 63.9% with Qwen and 65.6% with Gemini, improving over the corresponding base models by 27.5 and 19.7%.
  • RQ1: Performance: 18.0% τ-Bench (Airline) Pass^1 doubles the base-model result of 9.0%, while ARC also improves tool-use performance on GAIA over both evaluated base models.GAIA improvements are 4.0 points with Qwen and 1.0 point with Gemini.
  • RQ2: Efficiency: ARC occupies Pareto-optimal regions with higher accuracy at lower token-based API cost than prior methods on GSM8K.The comparison uses a Pareto frontier based on pairwise dominance and evaluates Qwen2.5 7B Instruct costs through OpenRouter rates.
  • RQ3: Transfer and Ablations: Transfer is moderate across reasoning datasets, stronger with overlapping tools, and improves across Qwen model scales; SFT adds 1–3% accuracy and ≈5–35% reward.ARC trained on GSM8K reaches 63.0% on DROP and 57.0% on MedQA, while HotpotQA-to-GAIA transfer reaches 2.0% versus 6.0% in-domain.

5 Conclusion

ARC learns query-adaptive agent configurations through a hierarchical SMDP policy, selecting workflows, tools, budgets, and prompts without updating the backbone LLM. Across benchmarks, it improves performance while supporting diverse workflows and resource-aware execution.

  • 5 Conclusion: ARC models workflows, tools, budgets, and prompts as temporally extended options in an SMDP and selects them through a hierarchical policy.The framework combines masked reinforcement learning with supervised fine-tuning on high-reward trajectories without updating the backbone LLM.
  • 5 Conclusion: ARC improves average reasoning accuracy by 31.3%, tool-use accuracy by 13.95%, and τ-Bench (Airline) Pass^1 from 9.0% to 18.0%.These results are reported across reasoning, tool-use, and agentic benchmarks.
  • 5 Conclusion: ARC supports nine workflows spanning direct inference, reasoning, verification, routing, parallelization, orchestration, optimization, and autonomous agents.The workflows define distinct patterns of LLM calls and agent interactions.

C.2 Proof of Theorem 3.2

The SFT phase concentrates the ARC policy on elite configurations, yielding a formal expected-reward floor based on the elite threshold. This complements empirical evidence that SFT improves performance and reduces variance.

  • C.2 Proof of Theorem 3.2: SFT trains on elite configurations by maximizing their log-likelihood, equivalently minimizing divergence from the empirical elite distribution.Under sufficient capacity, the learned policy matches the empirical elite distribution and assigns no support outside the elite set.
  • C.2 Proof of Theorem 3.2: ARC proposes only configurations successful during training, preventing deployment-time sampling of novel, untested configurations.The support restriction is the basis for the performance guarantee.
  • C.2 Proof of Theorem 3.2: τ defines the expected-performance floor, and in the experiments it is the RL-buffer reward 70th percentile.The resulting policy is guaranteed in expectation to perform at least as well as the top 30% of RL trajectories.
  • C.2 Proof of Theorem 3.2: Adding SFT consistently improves performance and reduces variance across experimental runs.This empirical observation complements the theoretical support and reward guarantees.

D Ablation: Identifying the Best Embedding for State Representation

The embedding ablation evaluates semantic clustering, classification, complexity ranking, and downstream configuration prediction, while comparing representation quality against efficiency. It identifies sentence-t5-base and MetaCLIP-H14 as final candidates.

  • D Ablation: Identifying the Best Embedding for State Representation: The ablation compares 19 text-only, vision-language, and hybrid embedding models in native and projected dimensionality modes.The final candidates are sentence-t5-base and MetaCLIP-H14, whose performance–efficiency trade-off is shown on a Pareto frontier.
  • D Ablation: Identifying the Best Embedding for State Representation: ARI measures whether embeddings cluster semantically similar questions according to dataset and tool-type labels.ARI ranges from −1 to 1, with 1.0 indicating perfect alignment with ground truth.
  • D Ablation: Identifying the Best Embedding for State Representation: Classification accuracy tests whether embeddings distinguish benchmark sources and required tools, averaging cross-validation accuracy across both tasks.The tool classes are calculator, web_search, python, and none.
  • D Ablation: Identifying the Best Embedding for State Representation: Complexity ranking evaluates whether embeddings capture difficulty for budget allocation using Spearman correlation between predicted and true complexity rankings.Higher values indicate better monotonic difficulty ordering.
  • D Ablation: Identifying the Best Embedding for State Representation: Decision prediction most directly evaluates the RL objective by jointly predicting workflow, tool, and compute-budget tier.Its score requires all three predictions to be correct simultaneously and receives the highest overall-score weight.

E Ablation: Identifying the Best Prompt Generator

The prompt-generator ablation evaluates atoms by diversity and quality across five datasets and three agent roles. GPT-5.2 achieves the highest combined score and supplies the prompt-library atoms.

  • E Ablation: Identifying the Best Prompt Generator: Prompt atoms are generated for five datasets by 12 models spanning OpenAI, Anthropic, Meta, Mistral, Google, and Qwen.Each model generates atoms for reasoner, verifier, and answerer roles.
  • E Ablation: Identifying the Best Prompt Generator: Generated atoms are evaluated on diversity and quality, weighted 40% and 60%, respectively.Diversity includes uniqueness, strategy coverage, and semantic diversity; quality includes coherence, specificity, and clarity.
  • E Ablation: Identifying the Best Prompt Generator: GPT-5.2 achieves the highest combined score at 0.487 average across datasets.Its results include coherence 8.79, clarity 5.50, and strategy coverage 0.80.
  • E Ablation: Identifying the Best Prompt Generator: GPT-5.2-generated atoms are therefore used in the prompt library.The selection follows the model-level comparison across the evaluated datasets and metrics.

F Training Details

ARC training progressively learns stable, task-specific configurations by adapting workflow selection and tool usage across datasets. Reward and accuracy improve as the policy concentrates on effective structural patterns.

  • Training Setup: Training uses PPO followed by SFT refinement with dataset-specific structure and prompt policies.The setup includes separate structure and prompt learning rates, reward shaping, and elite-episode refinement.
  • Training Dynamics: Cumulative rewards and validation accuracy increase steadily while reward variance decreases toward stable dataset-specific plateaus.These dynamics indicate consistent progress across GSM8K, DROP, MedQA, HotpotQA, and GAIA.
  • Workflow Selection: The workflow distribution sharpens during training, with datasets favoring distinct dominant workflows or small mixtures of high-value patterns.Examples include Evaluator–Optimizer on GSM8K and Orchestrator–Workers on HotpotQA.
  • Tool Usage: ARC converges to stable, task-specific tool usage rather than indiscriminately invoking tools.HotpotQA and GAIA settle at higher usage levels than primarily textual reasoning tasks such as GSM8K.
  • Workflow Selection: Workflow accuracy varies substantially by dataset, so no single workflow is uniformly optimal.This supports learning query-adaptive configurations instead of relying on a fixed architecture.

G Additional Analysis

Additional analyses show that ARC’s favored workflows combine higher rewards with lower variance, while workflow effectiveness remains strongly dataset-dependent.

  • Workflow Analysis: No workflow dominates across datasets, with complex coordination patterns helping only on some tasks.Multi-step workflows generally outperform Direct execution within benchmarks, but the best choice remains task-dependent.
  • Reward Analysis: Favored workflows such as Reason→Ans and Reason→Verify→Ans achieve higher, less variable reward distributions.Rarely selected patterns show broader, lower-reward distributions.

H Alternative Training Objectives

Alternative training objectives underperform PPO in the reported comparison, with GRPO affected by sparse rewards and DPO requiring more interaction data.

  • GRPO: 81.2% accuracy after 2,000 episodes was achieved by GRPO on GSM8K, versus PPO’s 85.7%.The passage attributes GRPO’s difficulty to group normalization dampening sparse binary correctness signals.
  • Reward Analysis: Reward distributions differ across workflows, with higher-performing workflows showing higher central reward and tighter spread.This provides a distributional view of why some structural patterns are preferred during training.
  • DPO: 79.8% accuracy was achieved by DPO, which required 3× more environment interactions for pairwise preference data.Preference labeling became noisy when configurations had similar rewards, producing unstable training.

I Error Categorization Methodology

ARC’s error methodology classifies failures by checking configuration choices, reasoning, retrieval, and execution, then prioritizes the most actionable matching category.

  • Policy Configuration: The methodology first checks whether workflow, tool, and token-budget choices match query demands.Examples include assigning multi-step problems to Direct workflows or omitting calculators for arithmetic queries.
  • Reasoning Failures: Reasoning failures include wrong operations, missing steps, and omitted critical query constraints.These checks compare prediction structure with ground-truth reasoning requirements.
  • Knowledge and Retrieval: Tool-use failures are separated into retrieval failures and factual errors after tool invocation.The distinction considers whether the system reports unavailable information or confidently contradicts ground truth.
  • Execution Failures: Execution failures cover arithmetic mistakes and extracting the wrong final answer despite a correct approach.The method checks intermediate calculations and whether the correct answer appears before extraction.
  • Prioritization: Errors are assigned in priority order: policy configuration, answer extraction, reasoning, arithmetic, knowledge gap, then unclassified.This ordering surfaces policy failures first because they are considered most actionable for ARC.
  • Examples: A configuration example shows a multi-step logarithm problem receiving Direct workflow and low budget instead of Reason+Verify with high budget.The accompanying examples also illustrate operation, retrieval, and arithmetic errors.
Loading 2602.11574v3…