PaperlayerView exact revision on arXiv

Source-linked AI summary

FAMA: Failure-Aware Meta-Agentic Framework for Open-Source LLMs in Interactive Tool Use Environments

Amir Saeidi, Venkatesh Mishra, Souradeep Mukhopadhyay, Gaowen Liu, Ali Payani, Jayanth Srinivasa, Chitta Baral

arXiv:2604.25135v1cs.CL

TL;DR

Open-source LLM agents often accumulate decision-making errors during long, multi-turn tool-use interactions. FAMA diagnoses dominant failure patterns and selectively composes specialized agents, improving task success by up to 27% across benchmarks.

  • Problem

    Tool-use agents accumulate decision-making errors over long trajectories, while existing training methods rarely target agent-specific failure patterns.

  • Method

    FAMA analyzes baseline failures, then dynamically routes a minimal subset of specialized agents to construct targeted prior context for the tool-use agent.

  • Results

    FAMA consistently improves tool-use accuracy, trajectory reliability, and context efficiency, with task-success gains of up to 27% over open-source baselines across three benchmarks.

  • Takeaways & Limitations

    FAMA supports failure-aware orchestration and targeted context construction as a design principle for dependable, resource-efficient conversational tool-use agents.

  • Takeaways & Limitations

    FAMA’s effectiveness is bounded by its predefined agent pool, and the framework does not automatically discover or synthesize new specialized agents.

Abstract

from arXiv · show

Large Language Models are being increasingly deployed as the decision-making core of autonomous agents capable of effecting change in external environments. Yet, in conversational benchmarks, which simulate real-world customer-centric issue resolution scenarios, these agents frequently fail due to the cascading effects of incorrect decision-making. These challenges are particularly pronounced for open-source LLMs with smaller parameter sizes, limited context windows, and constrained inference budgets, which contribute to increased error accumulation in agentic settings. To tackle these challenges, we present the Failure-Aware Meta-Agentic (FAMA) framework. FAMA operates in two stages: first, it analyzes failure trajectories from baseline agents to identify the most prevalent errors; second, it employs an orchestration mechanism that activates a minimal subset of specialized agents tailored to address these failures by injecting a targeted context for the tool-use agent before the decision-making step. Experiments across open-source LLMs demonstrate performance gains up to 27% across evaluation modes over standard baselines. These results highlight that targeted curation of context through specialized agents to address common failures is a valuable design principle for building reliable, multi-turn tool-use LLM agents that simulate real-world conversational scenarios.

1 Introduction

FAMA addresses accumulating, heterogeneous errors in long-horizon tool-use trajectories by dynamically composing specialized helper agents around open-source LLMs. Across three benchmarks, it improves task success by up to 27% over baselines through targeted context construction.

  • Motivation: Tool-use benchmarks expose accumulating decision-making errors across long trajectories containing heterogeneous context, tool outputs, domain rules, interventions, ambiguous intent, and evolving constraints.Different open-source model backbones exhibit distinct dominant failure modes, especially with limited context windows.
  • Framework: FAMA first analyzes baseline-agent failure trajectories to identify predominant errors and then dynamically constructs prior context for the tool-use agent.The framework is explicitly designed as a two-stage approach for failure-aware agent composition.
  • Framework: A lightweight orchestrator and mitigation agent selectively activate specialized LLM helpers that address identified weaknesses while avoiding excessive context injection.Only necessary agents are enabled, targeting the tool-use agent’s most critical shortcomings.
  • Results: 25% task-success gains in τ-bench, 27% in ACEBench, and 24% in τ-trait are achieved over baselines for open-source model backbones.FAMA is evaluated with open-source backbones ranging from 4–72 billion parameters and improves tool-use accuracy, trajectory reliability, and context efficiency.
  • Results: FAMA improves tool-use performance across τ-bench, τ-trait, and ACEBench through optimized context construction for diverse open-source LLM agents.The paper also analyzes improvements relative to other baseline agents.

2 Related Works

Related work has progressed from isolated, single-turn tool calls toward interactive, multi-turn benchmarks that expose long-horizon reasoning challenges. Parameter-free prompting and orchestration methods, alongside emerging failure-analysis techniques, motivate more systematic evaluation of language agents, especially beyond strong proprietary backbones.

  • LLM-based Tool-Use Benchmarks: Recent benchmarks increasingly evaluate multi-turn, interactive tool use rather than isolated tool calls for single-turn queries.These benchmarks expose long-horizon reasoning challenges for language agents.
  • LLM-based Tool-Use Benchmarks: Structured prompting and modular agent orchestration improve language agents without updating model parameters.The cited approaches include structured prompting and modular agent orchestration.
  • LLM-based Tool-Use Benchmarks: These parameter-free techniques are predominantly evaluated on strong or proprietary LLM backbones with large context windows and robust reasoning capabilities.The passage identifies this evaluation pattern as a limitation of existing work.
  • Failure-Aware Agentic Orchestration: Recent work systematically analyzes failures in multi-agent LLM systems, identifies diverse failure causes, and develops taxonomies and annotation datasets for debugging and evaluation.Other efforts automate failure tracing and attribution to localize error sources.

3 Problem Statement

The problem setting uses realistic, multi-turn tool-use benchmarks to evaluate language-agent decision-making in controlled conversational environments. Both users and customer-service assistants are simulated by LLMs, creating dynamic testbeds for analysis.

  • Benchmark Setting: τ-bench, τ-trait, and ACEBench assess language-agent decision-making in realistic, multi-turn interaction settings.These benchmarks are used to simulate ideal testbeds for tool use in multi-turn conversations.
  • Benchmark Setting: LLMs simulate both the user and customer-service assistant, enabling controlled yet dynamic conversational environments.

4 Method

FAMA is a dynamic framework for identifying failure patterns in open-source model agents and selecting resource-efficient agent configurations to address them. It analyzes baseline failure trajectories, attributes errors, and deploys minimal mitigation subsets that improve performance across evaluated settings.

  • Framework objective: FAMA identifies failure patterns in open-source model agents and determines effective agent sets under resource constraints.The framework is designed for challenging agentic tasks where open-source models exhibit distinct failure modes and limitations.
  • Failure taxonomy: The method organizes failures into four general error categories representing common tool-calling agent failures.The categories include Domain Policy Violation, Incorrect Retrieval from Complex Tool Outputs, and Contextual Misinterpretation and Hallucination.
  • Failure analysis: FAMA uses |E| independent error analysis agents to detect category-specific failure causes from baseline trajectories, producing categorical decisions and supporting rationales.Each analysis agent is responsible for a specific error category e.
  • Failure attribution: An orchestrator combines the analysis outputs with the full user–tool interaction trajectory to attribute failures while accounting for later recovery after environmental feedback.The outputs and rationales are concatenated into a single input for final failure attribution.
  • Mitigation and deployment: A mitigation agent selects an efficient subset A∗⊆A for each identified error category, and aggregated recommendations form the minimal configuration addressing dominant failure modes.The selected subsets are collected across tasks and used to deploy multi-agent frameworks.
  • Empirical outcome: FAMA consistently outperforms IRMA, ReAct, and FC across τ-bench, τ-trait, and ACEBench under different evaluated model settings.The reported comparisons appear in Tables 1–4.

5 Experiments

Experiments evaluate open-source LLM agents and FAMA across tool-calling benchmarks, comparing selective failure-aware orchestration with established baselines. Results show that FAMA improves performance across domains while addressing recurring constraint, memory, and context-window failures.

  • Experimental Setup: Experiments evaluate open-source models exclusively on tool-calling benchmarks, using Qwen3-4B-Instruct, Qwen3-14B, Qwen3-32B, and Qwen2.5-72B-Instruct.Reasoning- or thinking-augmented variants are excluded from the primary model set because their internal reasoning consumes substantial token budget.
  • Baselines and FAMA: FAMA is compared with Function Calling, ReAct, and IRMA, while IRMA’s Self-Reflection extension is excluded as orthogonal to the study’s focus.FAMA selectively enables specialized agents according to dominant failure modes rather than injecting all available agent contexts.
  • Experimental Setup: The evaluation covers τ-bench Airline and Retail, τ-trait Telehealth and Telecom, and single-agent ACEBench tasks in food delivery and telecommunications.The experiments assess methods in dynamic environments where a user-agent simulator interacts with a tool-calling agent.
  • Failure Analysis: All evaluated open-source models struggle with domain constraint violations and contextual misinterpretations, with difficulties increasing in longer conversations.Domain rules may be forgotten over time, and mitigation recommendations strongly favor the Memory and DCE modules.
  • Efficiency: IRMA incurs 50 and 58% overhead and 149.8 and 111.6 seconds of averaged task-completion latency for Qwen3-32B on retail and airline tasks.Additional reasoning or helper agents increase runtime and token costs, worsening context-window constraints in multi-turn settings.

6 Conclusion

FAMA is a dynamic multiagent framework for overcoming open-source language-agent limitations in interactive tool-calling environments. It analyzes dominant failure patterns and uses an orchestrator to select a minimal, task-relevant agent subset to mitigate them.

  • FAMA targets limitations of open-source language agents in interactive environments such as tool calling.The framework is described as a dynamic multiagent approach.
  • FAMA first analyzes agent failures to identify dominant error patterns.
  • FAMA then uses an orchestrator framework to select a minimal, task-relevant subset of agents for error mitigation.

Limitations

FAMA improves performance across benchmarks but remains limited by its reliance on a predefined specialized-agent pool and evaluation in structured conversational environments. Future work should address automatic agent discovery and broader interactive settings.

  • Agent-pool coverage: FAMA’s effectiveness is bounded by the coverage of its predefined pool of specialized agents inherited from the underlying agentic framework.Failure modes not captured by existing agents may remain unaddressed.
  • Future work: The current framework does not address automatic discovery or synthesis of specialized agents.The paper identifies this as an important direction for future work.
  • Evaluation scope: The study primarily evaluates FAMA in structured conversational environments, which do not capture the full spectrum of interactive agent deployments.Future investigation should extend FAMA to embodied, multimodal, and open-ended environments with less well-defined failure taxonomies.

Ethics Statement

The authors used Grammarly and ChatGPT to correct grammatical errors and rephrase sentences.

  • The authors used Grammarly and ChatGPT for grammatical correction and sentence rephrasing.

A More Ablation Studies

Additional ablations show that FAMA’s mitigation-agent-recommended combinations achieve the strongest overall performance, while alternative judgment models identify the same key failures and selections. Subsets of recommended agents can nevertheless remain competitive or improve performance in some cases.

  • Agent Combination Ablations: Alternative agent combinations perform worse than the combination recommended by the mitigation agent in Qwen3-14B Tau-Bench experiments.The tested alternatives combine the memory module with domain-constraint extraction, tool-output reformulation, and tool suggestion agents in different configurations.
  • Agent Combination Ablations: The mitigation-agent-recommended combination achieves the best performance, with the same trend observed across other models.These results support the generalizability of FAMA’s agent-combination selection.
  • Agent Combination Ablations: Using only a subset of recommended agents can produce competitive or improved performance in some cases.The findings indicate that FAMA can remain effective without always activating the full recommended subset.
  • Judgment Model Ablations: GPT-4.1-mini identifies Contextual Misinterpretation and Domain Constraint Violation as primary failures, matching GPT-4o’s decisions in FAMA analysis and agent selection.The repeated procedure evaluates the robustness of failure analysis and agent selection using baseline results.

B Detail of Benchmarks

The benchmarks model multi-turn conversational tool use as partially observable decision-making, covering customer-service, robustness, and function-call correctness settings. They vary in domains, task scales, and evaluation metrics, including pass^k, end-to-end accuracy, and process accuracy.

  • τ-bench: τ-bench evaluates conversational agents in retail and airline customer-service domains using simulated users, policy rules, API tools, and realistic backend state.Its retail setting includes 500 users, 50 products, 1,000 orders, 15 tools, and 115 tasks.
  • τ-trait: τ-trait extends τ-bench with persona-aware user simulation and adds telecom and telehealth to study robustness, personalization, and fairness under behavior shifts.It contains 218 tasks across retail (120), airline (60), telecom (18), and telehealth (20).
  • Evaluation metrics: τ-bench and τ-trait use pass^k, the probability that all k independently sampled outputs successfully complete a task, averaged across tasks.The metric uses n independent trials and c successful trials with reward r = 1.
  • ACEBench: ACEBench benchmarks function-call correctness across English and Chinese everyday-domain tasks using 4,538 APIs spanning 8 domains and 68 sub-domains.The work focuses on 30 multi-turn tasks from a 50-task agent-based collection, discarding 20 multi-step tasks.
  • Evaluation metrics: ACEBench reports end-to-end accuracy for exact attribute alignment and process accuracy for agreement between actual and ideal function-call sequences.Process accuracy is n_m, where m is the ideal sequence length and n is the number of matching steps.

C Analysis of the τ-trait Results

FAMA is evaluated on τ-trait’s Telehealth and Telecom domains, where memory-related failures—especially Incomplete Fulfillment—motivate targeted mitigation and yield robust gains over IRMA across pass^k for k = 1, . . . , 5.

  • Evaluation Scope: FAMA evaluates τ-trait on the Telehealth and Telecom domains, extending prior analysis of the Airline and Retail domains in τ-bench.τ-trait contains four domains: Retail, Airline, Telehealth, and Telecom.
  • Failure Analysis: Domain Constraint Violations and Contextual Misinterpretation remain critical error sources, while memory-related issues are more prominent across τ-trait domains.These error sources are identified by the orchestrator agent.
  • Failure Analysis: Incomplete Fulfillment emerges as the primary cause of task failures, prompting mitigation agents to recommend the Memory module for more models.The Memory module frequently becomes the recommended intervention across models.
  • Failure Mitigation: Using only the Memory module significantly improves open-source model performance and sometimes surpasses combinations that also include the Domain Constraints Extractor agent.The results indicate that adding DCE is not always beneficial when Memory alone addresses the dominant failures.
  • Comparative Results: FAMA consistently outperforms IRMA across Telecom and Telehealth on pass^k, for k = 1, . . . , 5.FC exceeds FAMA in isolated cases but varies substantially by domain and model, unlike FAMA’s consistent cross-domain performance.

D Analysis of the ACEBench Results … G Comparison with Self-Reflection

Across ACEBench and τ-bench, FAMA uses failure analysis to select targeted auxiliary agents, improving tool-use reliability with a minimal Planner–Verifier configuration on ACEBench. The paper also defines failure categories and reports that FAMA consistently outperforms self-reflection in τ-bench.

  • D Analysis of the ACEBench Results: ACEBench evaluates FAMA in a multi-turn setting with an active user agent interacting with a tool-calling agent, against the Base Agent and ReAct.The analysis focuses on ACEBench’s multi-turn agent configuration.
  • D Analysis of the ACEBench Results: All evaluated ACEBench models exhibit substantial Contextual Misinterpretation, for which the mitigation agent consistently recommends Planner and Verifier agents.The failures include incomplete execution, problem-solving misinterpretation, and incorrect environment interactions.
  • D Analysis of the ACEBench Results: FAMA consistently improves ReAct and Base Agent performance over other baselines by adding only two auxiliary agents.The reported improvements appear in Table 3.
  • D.1 Example: Planner and Verifier agents enable successful recovery from login, Wi-Fi, message-deletion, and sender-or-recipient failures, ending with full verification.The trajectory reports “Task Completed Successfully with Full Verification.”
  • E Error Categories and Causes of the Failures: The defined failure categories are Domain Policy Violation, Incorrect Retrieval from Complex Tool Outputs, Contextual Misinterpretation and Hallucination, and Incomplete Fulfillment or Early Stopping.These categories cover policy compliance, extracting state from tool outputs, interpreting intent, and completing multi-part tasks.
  • F Optimal Components of FAMA: For τ-bench and τ-trait, FAMA assumes Memory, DCE, TOR, and TS modules, while ACE-Bench additionally uses Planner and Verifier agents.The ACE-Bench additions address the Base Agent’s inability to generate free-form text.
  • G Comparison with Self-Reflection: FAMA consistently outperforms the Self-Reflection baseline across multiple τ-bench models, indicating that structured failure analysis and targeted mitigation outperform generic reflection and revision.The comparison includes FAMA, ReAct, and Self-Reflection across Tables 9–12.
Loading 2604.25135v1…