Source-linked AI summary
TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs
Yaobo Liang, Chenfei Wu, Ting Song, Wenshan Wu, Yan Xia, Yu Liu, Yang Ou, Shuai Lu, Lei Ji, Shaoguang Mao, Yun Wang, Linjun Shou, Ming Gong, Nan Duan
TL;DR
Foundation models remain limited on specialized tasks, while capable domain-specific systems are difficult to access through them. TaskMatrix.AI proposes connecting foundation models with millions of APIs and illustrates this vision through task-completion cases. The paper identifies multimodal foundation-model requirements and API-platform challenges as open issues.
Problem
Foundation models struggle with specialized, execution-sensitive tasks, while existing domain-specific systems are not readily compatible with them.
Method
TaskMatrix.AI connects a foundation model with APIs of specialized models and systems to plan tasks, select tools, generate action code, and execute diversified digital and physical tasks.
Results
Study cases illustrate TaskMatrix.AI’s feasibility for visual tasks, multimodal long-content generation, office automation, and API-composition behavior.
Takeaways & Limitations
The proposed ecosystem is intended to combine foundation models with specialized APIs while supporting lifelong learning, interpretability, and reliable task completion.
Takeaways & Limitations
The vision still requires determining the minimum modality set and training an MCFM that can handle modalities returned by APIs, while maintaining API documentation and quality.
Abstract
from arXiv · showhide
Artificial Intelligence (AI) has made incredible progress recently. On the one hand, advanced foundation models like ChatGPT can offer powerful conversation, in-context learning and code generation abilities on a broad range of open-domain tasks. They can also generate high-level solution outlines for domain-specific tasks based on the common sense knowledge they have acquired. However, they still face difficulties with some specialized tasks because they lack enough domain-specific data during pre-training or they often have errors in their neural network computations on those tasks that need accurate executions. On the other hand, there are also many existing models and systems (symbolic-based or neural-based) that can do some domain-specific tasks very well. However, due to the different implementation or working mechanisms, they are not easily accessible or compatible with foundation models. Therefore, there is a clear and pressing need for a mechanism that can leverage foundation models to propose task solution outlines and then automatically match some of the sub-tasks in the outlines to the off-the-shelf models and systems with special functionalities to complete them. Inspired by this, we introduce TaskMatrix.AI as a new AI ecosystem that connects foundation models with millions of APIs for task completion. Unlike most previous work that aimed to improve a single AI model, TaskMatrix.AI focuses more on using existing foundation models (as a brain-like central system) and APIs of other AI models and systems (as sub-task solvers) to achieve diversified tasks in both digital and physical domains. As a position paper, we will present our vision of how to build such an ecosystem, explain each key component, and use study cases to illustrate both the feasibility of this vision and the main challenges we need to address next.
1 Introduction
Foundation models are powerful but remain limited on specialized, execution-sensitive tasks, while existing domain-specific systems are difficult to connect to them. TaskMatrix.AI proposes an ecosystem linking foundation models with millions of APIs to complete diversified digital and physical tasks.
- Motivation: Foundation models support multimodal understanding and diverse generation but still struggle with accurate calculations and multi-step real-world tasks.Their limitations include insufficient domain-specific data and errors on tasks requiring accurate execution.
- Motivation: Existing symbolic and neural systems can perform specialized tasks well but are not readily available or compatible with foundation models.
- TaskMatrix.AI Vision: TaskMatrix.AI connects foundation models with millions of existing model and system APIs to finish diversified tasks.
- Key Advantages: The ecosystem uses a foundation model to process multimodal inputs and generate API-calling code, while an API platform stores task experts under consistent documentation.
- Key Advantages: Adding APIs with specific functions expands TaskMatrix.AI’s skills, while understandable action codes and API outcomes improve response interpretability.
- Vision: Connecting foundation models to APIs is envisioned to integrate neural and symbolic systems across digital and physical tasks with lifelong learning and reliable capabilities.
2 TaskMatrix.AI Architecture
TaskMatrix.AI combines a multimodal conversational foundation model, an API platform, an API selector, and an action executor. Its architecture plans tasks, retrieves and invokes APIs, incorporates feedback, and supports documentation and model alignment through learning mechanisms.
- Architecture: TaskMatrix.AI’s architecture includes an MCFM, API platform, API selector, and API executor for understanding goals, managing APIs, recommending tools, and executing action codes.The API executor returns intermediate and final execution results.
- Action-Code Generation: The MCFM maps foundation-model parameters, the API platform, user instruction, and conversational context to executable action codes.This procedure is represented as A = MCFM(θ, P, I, C).
- MCFM: The MCFM generates solution outlines from user instructions and conversational context, which users can edit before API selection and code generation.
- API Platform: The API platform stores and manages APIs through a unified documentation schema covering names, parameters, descriptions, examples, and composition instructions.Composition instructions can guide ordering, such as opening a file before editing and saving it afterward.
- API Selector: The API selector searches or uses package-level strategies to retrieve APIs relevant to the task requirements and solution outline.
- Learning and Execution: User feedback supports RLHF for improving MCFM and API-selector behavior, while execution verification checks whether generated codes and outcomes satisfy instructions.Feedback can also guide API developers in improving documentation through instruction, API-call, and feedback triples.
3 Application Scenarios
TaskMatrix.AI is demonstrated across visual creation, multimodal content generation, office and cloud automation, internet and metaverse access, and physical-world interaction. These cases illustrate API composition, task decomposition, and integration with specialized systems.
- Overview: All application cases were implemented in practice across content creation, office automation, cloud services, and physical-world interaction.The paper also discusses internet access, metaverse interaction, and neuro-symbolic integration as additional scenarios.
- Visual Task Completion: TaskMatrix.AI composes multiple visual APIs to handle image generation, questioning, editing, and high-resolution outpainting.The outpainting workflow uses Image Question Answering, Image Captioning, and Replace Objects from Image iteratively to extend an image to 2048 × 4096.
- Multimodal Long Content Generation: For multimodal long content, TaskMatrix.AI plans subtasks, lets users modify the solution outline, generates API action code, and executes it to produce text-and-image content.The approach uses APIs for search and text-to-image generation, and the solution outlines can be reused as templates for similar instructions.
- Office Automation: TaskMatrix.AI decomposes complex PowerPoint instructions into multiple operations and can apply patterns across pages using presentation content and user commands.One example requires 25 APIs, while other examples generate pages from a company list and insert logos based on page titles.
- Cloud Services Utilization: TaskMatrix.AI can understand cloud APIs, recommend appropriate services, and support data management, model training, status checking, and deployment.An Azure example shows users building a personal conversation model with minimal expertise, while detailed documentation supports answering procedural questions.
- More Scenarios: TaskMatrix.AI extends API-mediated interaction to internet services, metaverse agents, robots, IoT devices, and symbolic modules.The paper describes travel planning, booking flights, virtual teammates, smart homes, and delegation to formal reasoning engines or expert systems.
4 Case Study
The PowerPoint case study uses ChatGPT as a multimodal conversational foundation model connected to documented PowerPoint APIs and an action executor. It shows that composition instructions improve API-call ordering and formatting, while feedback can help developers refine API documentation.
- MCFM: ChatGPT serves as the Multimodal Conversational Foundation Model, receiving API documentation, conversational context, and prompts to generate PowerPoint action codes.The case study uses general prompts to align the model with user instructions, defined APIs, and a specified output format.
- Multimodal PowerPoint Processing: PowerPoint content is represented as text, text-box positions, images, and shapes, enabling instructions such as inserting logos and repositioning images.The system parses visual content into structured textual data because ChatGPT cannot directly process images in this setup.
- API Platform: The API platform documents functions, parameters, descriptions, and composition instructions for creating slides, selecting text boxes, inserting text, and navigating slides.Composition instructions specify dependencies such as selecting a title or content text box before editing it.
- Task Decomposition: A complex instruction to generate company slides requires roughly 25 API calls, with the company list obtained from existing PowerPoint content.The study omits separate solution-outline generation because ChatGPT is reported to generate it accurately for this case.
- Composition Instructions: Removing composition instructions produces inconsistent formatting and API ordering, whereas select_title and select_content instructions ensure target selection precedes text insertion.Without these instructions, the model may insert text before selecting a text box and generate long paragraphs without line breaks.
- Feedback to API Developers: ChatGPT-generated feedback from wrong and correct API calls identifies documentation improvements, and the first two feedback items were related to composition instructions.The authors suggest selecting helpful feedback items for API documentation or automating their testing before adding them.
5 Challenges
Fully realizing TaskMatrix.AI requires addressing challenges in multimodal foundation modeling, API infrastructure and calling, security, privacy, and personalization. The paper reports only preliminary validation across several scenarios.
- Preliminary validation has covered several scenarios, but many challenges remain before TaskMatrix.AI’s vision is fully realized.
- Multimodal Conversational Foundation Model: TaskMatrix.AI needs a multimodal conversational foundation model that can reason over varied inputs, plan, and generate high-quality API-execution code.The required modalities include text, image, video, audio, and code.
- API Platform: An API platform must generate usable documentation, assure API quality and dependability, and guide creation of APIs for unmet tasks.The platform is intended to host millions of APIs, whose quality and trustworthiness may differ substantially.
- API Calling: API calling requires selecting suitable APIs and planning complex solutions, including interactive exploration when no solution is immediately available.Selection must connect user intentions with APIs using documentation and previous usage history.
- Security and Privacy: Access to physical and digital systems requires faithful execution of user instructions, secure data transmission, and authorized access to sensitive data.
- Personalization: Personalization must reduce scaling costs while aligning interfaces with individual users from few-shot examples.The passage is truncated after introducing the few-shot alignment challenge.
6 Related Work
Related work has connected foundation models with search APIs, robotics, and other computational tools, while emerging products demonstrate similar API-mediated interaction patterns. TaskMatrix.AI is positioned within this developing research and product landscape.
- Search-API systems such as WebGPT and ReAct, and robotics systems such as PaLM-E and ChatGPT Robotics, apply APIs to specialized information or physical-world tasks.
- API teaching methods include pretraining or fine-tuning on API-use examples, in-context learning, and reinforcement learning from human feedback.
- ACT-1 targets digital-world actions, while LangChain combines language models with other sources of computation or knowledge.
- Visual ChatGPT applies the TaskMatrix.AI idea to visual question answering, generation, and editing, whereas ChatGPT Plugins provide current information, computation, and third-party services.
- Together, these works aim to advance research and products that implement foundation-model interfaces connected to external tools and services.
7 Looking Forward
TaskMatrix.AI is presented as a platform connecting foundation models with existing models and systems through APIs to support diversified tasks. The authors envision this ecosystem contributing to higher productivity and creativity as related technologies advance.
- TaskMatrix.AI connects foundation models with existing systems and models through APIs so people can perform diversified tasks.
- The envisioned ecosystem builds on advances in foundation models, cloud services, robotics, and Internet of Things technologies and infrastructures.
- The authors imagine a future in which this connected platform raises productivity and creativity.