xLAM: A Family of Large Action Models to Empower AI Agent Systems

Motivation

• xLAM targets limited open-source choices for agent tasks caused by scarce high-quality datasets and heterogeneous formats that impede transferable learning.Existing datasets also contain incorrect outputs, hallucinated actions, and repeated interaction turns.Source: §1 · p. 2

xLAM Models

• The released xLAM family spans model sizes for different applications, with 1B and 7B models aimed at on-device deployment and 8x7B and 8x22B models for harder tasks.Source: §1 · p. 2

Data Processing

• Data unification and augmentation improve dataset diversity, mitigate overfitting, and significantly improve generalizability across diverse environments.Source: §1 · p. 2

Data Synthesis

• 4 of the top 20 Berkeley Function Calling Leaderboard positions come from the synthetic dataset, including the top-1 spot, while smaller models can match much larger counterparts.Source: §1 · p. 3, Figure 2 · p. 3

Evaluation

• Evaluation across public agent benchmarks shows exceptional performance, while open-sourcing xLAM aims to advance open-source agent models and inform data processing and synthesis.Source: §1 · p. 3

2.1 LLM Agents

• LLM-agent research uses prompting, fine-tuning, and data-processing methods across tasks including question answering, web interaction, tool operation, reasoning, and planning.Source: §2.1 · p. 3
• Many agent frameworks still depend on proprietary models as their core engine, leaving a substantial gap in high-quality open-source models for agent tasks.Source: §2.1 · p. 3

2.2 Agent Benchmarks

• AgentBench, Mint-Bench, and AgentBoard cover code generation, games, web interaction, and reasoning, while ToolBench evaluates multi-turn reasoning and tool use.Source: §2.2 · p. 3
• The Berkeley Function-Calling Leaderboard evaluates function-calling capabilities across varied contexts, contributing to more accessible and standardized agent-model evaluation.Source: §2.2 · p. 3

3.1 Data Unification

• A unified function-calling format organizes tools, instructions, queries, and interaction steps, enabling consistent processing across environments.The modular representation also supports fine-grained augmentation and quality checks, including inspection of hallucinated or erroneous tool calls.Source: §3.1 · p. 4

3.2 Data Augmentation

• Format and instruction augmentation diversify tool ordering, prompt delimiters, instructions, and output representations to reduce overfitting.The pipeline uses shuffled sections and tools, rephrased instructions, and 15 output formats including JSON, XML, YAML, and plain text.Source: §3.2 · pp. 4–5

3.3 Data Quality Verification

• Quality verification combines rule-based checks, LLM judges, and human verification to detect undefined calls, type errors, hallucinated arguments, and weak reasoning.Argument hallucination is checked step by step against the query and prior observations, while reasoning and planning are evaluated at trajectory and thought-step levels.Source: §3.3 · p. 5

3.4 Data Synthesis

• Public agent datasets are limited by static, weakly synthesized, narrow, and often unexecuted data that underrepresents parallel function-calling scenarios.These limitations motivate synthesis from executable APIs and broader coverage of real-world function-calling patterns.Source: §3.4 · p. 5
• APIGen generates verifiable data through format, execution, and semantic checks that filter hallucinations and inaccurate argument values.The synthesis uses over 3,673 APIs across 21 ToolBench categories to produce 60,000 high-quality samples.Source: §3.4 · p. 6

3.5 Data Mixture

• SFT mixes cleaned and augmented agent data, synthetic function-calling data, and general instruction data, while DPO uses model-generated and human-verified preference samples.For xLAM-7b-fc-r and xLAM-1b-fc-r, synthetic function-calling data supplies 50% of training data; general instruction data contributes 20% to 30% for general models.Source: §3.5 · p. 6

4.1 Modeling

• xLAM combines supervised fine-tuning with DPO alignment using a distributed training setup designed for robustness and reproducibility.Training uses shuffled data, process-specific random seeds, and partitioning across devices.Source: §4.1 · p. 6
• xLAM-FC targets simple, multiple, parallel, and parallel-multiple function-calling scenarios while training relevance detection for calls, results, and objectives.The specialized models are designed to handle both individual requests and concurrent function calls.Source: §4.1 · pp. 6–7

4.2 xLAM Model Series

• The xLAM series is built on Mixtral Instruct models and trained with uniformly sampled data to balance performance across multi-turn and function-calling tasks.The series is intended for varied agent use cases rather than a single task category.Source: §4.2 · p. 7
• Specialized xLAM-7b-fc-r and xLAM-1b-fc-r models use DeepSeek-Coder backbones and can be hosted on a single GPU for function-calling workloads.Their smaller sizes target accessible deployment, including simple queries and parallel concurrent requests.Source: §4.2 · p. 7
• The series offers multiple model sizes and specializations to match different computational resources and application requirements.This includes smaller models for accessible deployment and larger models for more demanding agent tasks.Source: §4.2 · p. 7

5.1 Benchmarks

• The benchmark suite covers Webshop, ToolQuery, ToolBench, and BFCL, using Success Rate, Progress Rate, Pass Rate, and function-calling accuracy measures across varied task settings.BFCL v2 includes live real-world scenarios released after model training, and the models were evaluated on that updated benchmark.Source: §5.1 · pp. 7–8

5.2.1 Webshop and ToolQuery

• In Webshop, xLAM-7b-r achieves the highest Success Rate at 0.414, surpassing general LLMs and specialized agent models.The comparison includes GPT-4-0125-preview, GPT-4o-2024-0523, Claude2, AgentOhana-8x7b, and Lemur-70b.Source: §5.2.1 · p. 8, Table 2 · p. 8
• In ToolQuery, xLAM-8x7b-r and xLAM-8x22b-r rank second with a Success Rate of 0.683, exceeding Mixtral-8x7b-inst at 0.167 and Mixtral-8x22b-inst at 0.400.All three xLAM models surpass Mixtral-8x22B-Instruct in the reported comparison.Source: §5.2.1 · pp. 8–9, Table 2 · p. 8
• Under ToolQuery-Unified, xLAM performance remains more consistent than GPT models, while GPT-4o degrades by 42% relative to ToolQuery.The reported explanation is that xLAM training used trajectories adhering to the unified format.Source: §5.2.1 · p. 9, Table 3 · p. 9

5.2.2 ToolBench

• On ToolBench, xLAM models surpass TooLlama-V2 and GPT-3.5-Turbo-0125 across all tested settings and outperform AgentOhana-8x7b on unseen instructions and unseen tools.They achieve performance comparable to GPT-4-0125-preview in the unseen-tools setting.Source: §5.2.2 · p. 9, Table 4 · p. 9

5.2.3 Berkeley Function-Calling Benchmark

• xLAM-8x22b-r achieves the highest overall accuracy at 87.31% on BFCL v2, while xLAM-8x7b-r ranks 6th, outperforming most prominent models including GPT-4o-mini and Claude-3.The leaderboard rank is based on overall accuracy, a weighted average of different evaluation categories, with results dated 09/03/2024.Source: §5.2.3 · p. 9, Table 5 · p. 10

5.3 Ablation Study

• The ablation study examines the effects of data augmentation and data quality verification through cleaning.The supplied figure identifies these two components as the ablation dimensions.Source: Table 2 · p. 8, Figure 3 · p. 11

• xLAM models range from 1B to 8x22B parameters and are trained with a scalable pipeline that unifies, augments, and synthesizes diverse datasets.The models include dense and mixture-of-expert architectures.Source: §6 · p. 11
• Evaluations show that xLAM models consistently perform exceptionally across various benchmarks.Source: §6 · p. 11
• Publicly releasing the xLAM series aims to democratize access to high-performance models for agent tasks and accelerate progress in the field.Source: §6 · p. 11

• The format records each task's identifier, instruction, few-shot examples, user query, and available tools.Tool entries include an API name, description, and parameter schema.Source: A Appendix · p. 15
• Each execution step represents the model's thinking or planning process and its tool calls.Tool calls specify the API name and arguments.Source: A Appendix · p. 15
• The format stores tool arguments, a step identifier, the next observation from execution, and any follow-up user input.Source: A Appendix · p. 15
• Figures 4 and 5 present the unified function-calling format and an example xLAM prompt-output pair, respectively.Source: Figure 4 · p. 15, Figure 5 · p. 16