ModelScope-Agent: Building Your Customizable Agent System with Open-source Large Language Models

Motivation and gap

• Open-source LLMs remain limited in following instructions to use external tools and capturing up-to-date information, while customizable agent systems based on them remain largely unexplored.The introduction contrasts these gaps with agents that rely on closed-source LLMs or focus on particular API types.Source: §1 · p. 1

System design

• ModelScope-Agent uses open-source LLMs as controllers and unifies customizable model training with integration of model APIs and common APIs.Its memory module manages system messages, user history, tool messages, in-context examples, and localized knowledge.Source: §1 · pp. 1–2

Tool-use framework

• The framework combines tool-use data collection, customized training, evaluation, and deployment, including MSAgent-Bench's 598k dialogues and Weighted LM training for API names and parameters.The evaluation framework examines trained models' tool-use abilities in different aspects.Source: §1 · p. 2

Customization

• The library provides diverse model and common APIs by default, supports registering self-defined APIs and automatic API retrieval, and is designed for flexible industry applications.The agent and training framework are documented for usage, construction, and optimization.Source: §1 · p. 2

2 The ModelScope Agent

• Given a human instruction, the selected LLM controller plans tasks, selectively uses tools and memory knowledge, and generates a helpful response.The overall system consists of an open-source LLM controller, a tool-use module, and a memory module.Source: §2 · p. 2

2.1 LLMs as Brain

• LLMs plan and decompose requests, call tools, perform retrieval, and integrate prior information; users can replace supported models or add new ones for training and inference.Model replacement uses model_name and model_config settings.Source: §2.1 · pp. 2–3

2.2 Tool Use

• The tool library supports common and AI-model APIs, user-registered tools, and dense retrieval that selects the top three APIs most relevant to an instruction.Retrieved API schemas are added to contextual prompts before the LLM plans and generates a request for execution.Source: §2.2 · p. 3

2.3 Memory Control

• Memory retrieves localized knowledge and assembles available context, including prompts, API schemas, conversation history, and few-shot examples, for the LLM.Users can customize the knowledge retriever and select context according to query type and the LLM's maximum length.Source: §2.3 · pp. 3–4

2.4 Agent Pipeline

• The agent combines retrieved tool schemas with contextual prompts, asks the LLM to plan and generate API requests, executes selected calls, and repeats as needed before returning a final response.API results are returned to the LLM for further planning; the process stops when no further call is needed.Source: §2.4 · p. 4

3.1 Dataset

• MSAgent-Bench contains 598k dialogues spanning Chinese and English, multiple API categories, multi-turn tool interactions, API-oriented QA, and API-agnostic instructions.The data was collected using ChatGPT-generated instructions, API requests, and answers based on API calling results.Source: §3.1 · p. 4

3.2 Model Training

• The dataset was used to fine-tune multiple open-source LLMs, including LLaMA, Qwen, and ChatPLUG, in multi-round conversation mode.Prompts and answers are concatenated for training.Source: §3.2 · p. 4
• Weighted LM increases emphasis on generating API names and parameters while zeroing the loss for user-prompt and tool-execution tokens.Source: §3.2 · p. 5

4 Evaluation

• MSAgent-Eval combines automatic API-usability evaluation with an Agent Arena that captures human preferences.Source: §4 · p. 5

4.1 Automatic Evaluation Framework

• Automatic evaluation measures API selection with Action EM, response similarity with ROUGE-L, and API-request argument quality with Argument F1.Argument F1 distinguishes half matches—correct argument but wrong value—from full matches with both argument and value correct.Source: §4.1 · p. 5
• Expert annotators prepare diverse instructions, correct API requests, and appropriate responses for automatic evaluation.Users can upload their own annotated test examples for customized scenarios.Source: §4.1 · pp. 5–6

4.2 Human Evaluation with Agent Arena

• The Agent Arena lets users compare two anonymous agents on an instruction using provided APIs, then vote and contribute to ELO-based rankings.Source: §4.2 · p. 6

• ModelScopeGPT connects LLMs to community AI models across NLP, CV, Audio, and Multimodal fields, using API retrieval and knowledge retrieval for model selection and local knowledge access.Source: §5 · p. 6
• Within one month of release, ModelScopeGPT received over 170k requests from 40k user visits.Source: §5 · p. 6
• In an Alibaba Cloud scenario, the agent registered an ECS-renewal API and successfully planned, called it, and informed the user that renewal was complete.The authors describe this test as demonstrating generalization to unseen APIs.Source: §5 · p. 6

• ModelScope-Agent covers flexible system design, data collection, model training, evaluation, and real-world usage examples for open-source LLM-based agents.Source: §6 · p. 6
• The authors position the library as community-driven support for AI-agent learning and best practices in building agent systems.Source: §6 · p. 6
• The conclusion expresses the hope that ModelScope-Agent will help pave the way toward a new era of AI agents.Source: §6 · p. 6

Ethics Statement

• Even after data filtering, customized models may generate biased, fake, or unsafe information; the authors identify safety methods as future work.They also note users can select LLMs and upload clean training data.Source: Ethics Statement · p. 7

A.1 Tool List

• Table 2 reports statistics for the default tool list and gives supported API input languages in parentheses.Source: Table 2 · p. 9

A.2 CustomTool

• Users define a custom tool by specifying its name, description, and parameters according to a predefined schema.The example schema includes a required field.Source: §A.2 · p. 9

B.1 Evaluation Benchmark

• The benchmark includes 10 in-domain APIs and 10 real unseen APIs, with several multi-turn scenarios.Source: §B.1 · p. 9
• Human experts wrote diverse instructions and then annotated the corresponding JSON API requests and answers after API results were obtained.Source: §B.1 · p. 9
• The authors state that they are still in progress and plan to include more APIs in the future.Source: §B.1 · p. 9
• The final test set contains 360 conversations and 2,059 reference snippets: 798 API requests and 1,261 plain-text answers.Source: §B.1 · p. 9

B.2 Evaluation Results

• Qwen had the most favorable reported performance; ChatGPT with two-shot in-context learning was inferior to fine-tuned models, and LLaMA underperformed other fine-tuned models.The authors attribute lower ChatGPT and LLaMA performance partly to Chinese-heavy test cases and limitations in user-intent recognition.Source: §B.2 · p. 9, Table 3 · p. 9

B.3 Weighted LM

• Weighted LM assigns loss weight 0 to user prompts, 1 to API-agnostic assistant text, and 2 to important API-call text such as names and parameters.The authors state that the higher weight can improve API-call generation accuracy.Source: §B.3 · p. 9
• The figure illustrates a text-to-video instruction tested with the same semantic request in English and Chinese.Source: Figure 5 · p. 10
• The figure illustrates an image-chat instruction tested with the same semantic request in English and Chinese.Source: Figure 6 · p. 10

C Cases

• Single-step examples use a tool to generate a video or to chat about an image.Source: C Cases · p. 10
• The multi-step scenario writes promotional copy, reads it, and then generates a video; the Chinese example completes all three steps accurately.Source: C Cases · p. 10, Figure 7 · p. 11
• The multi-turn example uses conversation history to call an API and generate its parameters from prior content.Source: C Cases · p. 10, Figure 8 · p. 11, Figure 9 · p. 11
• The in-domain knowledge QA case requires retrieving domain knowledge and using it in the response.Source: C Cases · p. 11

D Data Collection Procedure

• ChatGPT-simulated user-agent dialogues included synthetic APIs, an iterative API-call loop, and filtering of hallucinated or illegal requests.The collection used prompted ChatGPT instances to act as users, agents, and synthetic APIs.Source: D Data Collection Procedure · p. 12
• Collected-data statistics are summarized in Table 4.Source: D Data Collection Procedure · p. 12, Table 4 · p. 12

E.1 Large Language Models

• Plain-text pretraining leaves LLMs less suited to multimodal tasks and information beyond training data; ModelScope-Agent provides a customizable framework and best practices for open-source LLM tool use.Source: §E.1 · pp. 12–13

E.2 Agent & Tool Learning

• Related work covers closed-source integrations such as Visual-ChatGPT (Wu et al., 2023) and HuggingGPT (Shen et al., 2023), plus open-source learners for neural models or common APIs (ToolAlpaca, Tang et al., 2023; ToolLLaMA, Qin et al., 2023).Source: §E.2 · p. 13
• ModelScope-Agent supports user-customized agent learning for open-source models and both model and common APIs, whereas prior works support only certain specific APIs.Source: §E.2 · p. 13