Source-linked AI summary
HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, Yueting Zhuang
TL;DR
HuggingGPT addresses the inability of existing AI models to autonomously solve complicated multimodal tasks by using ChatGPT as a controller over specialized models. It plans and routes subtasks through Hugging Face models, executes them, and integrates their outputs, enabling tasks across modalities and domains while retaining important efficiency and planning limitations.
Problem
Existing LLMs and individual AI models cannot autonomously coordinate the multiple specialized capabilities required for complicated tasks across domains and modalities.
Method
HuggingGPT uses ChatGPT as a controller to plan tasks, select Hugging Face expert models from descriptions, execute subtasks, and summarize their results.
Results
HuggingGPT tackles a wide range of sophisticated AI tasks across language, vision, speech, and other modalities and domains.
Takeaways & Limitations
Language can serve as an interface through which an LLM coordinates specialized models, providing a paradigm for more general AI solutions.
Takeaways & Limitations
HuggingGPT depends heavily on LLM planning, may produce infeasible or non-optimal plans, and incurs increasing response time from multiple LLM interactions.
Abstract
from arXiv · showhide
Solving complicated AI tasks with different domains and modalities is a key step toward artificial general intelligence. While there are numerous AI models available for various domains and modalities, they cannot handle complicated AI tasks autonomously. Considering large language models (LLMs) have exhibited exceptional abilities in language understanding, generation, interaction, and reasoning, we advocate that LLMs could act as a controller to manage existing AI models to solve complicated AI tasks, with language serving as a generic interface to empower this. Based on this philosophy, we present HuggingGPT, an LLM-powered agent that leverages LLMs (e.g., ChatGPT) to connect various AI models in machine learning communities (e.g., Hugging Face) to solve AI tasks. Specifically, we use ChatGPT to conduct task planning when receiving a user request, select models according to their function descriptions available in Hugging Face, execute each subtask with the selected AI model, and summarize the response according to the execution results. By leveraging the strong language capability of ChatGPT and abundant AI models in Hugging Face, HuggingGPT can tackle a wide range of sophisticated AI tasks spanning different modalities and domains and achieve impressive results in language, vision, speech, and other challenging tasks, which paves a new way towards the realization of artificial general intelligence.
1 Introduction
HuggingGPT addresses LLM limitations on multimodal and multi-step AI tasks by using language to connect ChatGPT with specialized models. It plans tasks, selects and executes experts, then integrates their results into responses.
- Current LLMs are strong in language understanding, generation, and reasoning but limited in processing vision and speech and coordinating multiple sub-tasks.
- HuggingGPT uses language as a generic interface for an LLM to plan, schedule, and coordinate external AI models.
- ChatGPT decomposes user requests, selects Hugging Face expert models from their descriptions, executes subtasks, and integrates results into a final response.
- The system is designed to handle complex tasks across multiple modalities and domains by combining an LLM controller with task-specific experts.
- HuggingGPT is presented as a scalable paradigm in which task-specific expert capabilities can be continually incorporated into AI systems.
2 Related Works
HuggingGPT differs from unified multimodal and tool-using approaches by using an LLM as a controller that routes requests to expert models. Its model cooperation mechanism targets tasks across modalities and domains.
- Unlike approaches focused on unified multimodal models or external tool APIs, HuggingGPT organizes cooperation among models through the LLM.
- HuggingGPT uses an LLM to route user requests to expert models, combining language comprehension with specialized model expertise.
- The framework is intended to address AI tasks in any modality or domain through LLM-mediated model cooperation.
3 HuggingGPT
HuggingGPT is a four-stage collaborative system: it plans tasks, selects models, executes inference, and generates responses. Its design uses structured prompts, model descriptions, resource links, and parallel execution to coordinate experts.
- HuggingGPT: HuggingGPT comprises an LLM controller and numerous expert models, with stages for task planning, model selection, task execution, and response generation.
- Task Planning: Task planning decomposes user requests into structured tasks while determining dependencies and execution order.
- Task Planning: The task-planning prompt uses standardized slots for task names, identifiers, dependencies, and arguments, supplemented by demonstrations and chat logs.
- Model Selection: Model selection treats available expert models as choices and uses their descriptions with dynamic in-context assignment to match models to parsed tasks.
- Task Execution: During execution, the system replaces <resource>-task_id placeholders with outputs from prerequisite tasks and runs independent tasks in parallel when possible.
- Response Generation: Response generation summarizes planned tasks, selected models, and inference results, converting structured outputs such as detections into human-language responses.
4 Experiments
The experiments evaluate HuggingGPT’s task planning across single, sequential, and graph requests, using objective and human assessments. Demonstrations and model choice are also examined, with GPT-3.5 outperforming open-source LLMs across evaluated stages.
- Quantitative Evaluation: HuggingGPT is evaluated on single, sequential, and graph task-planning categories using category-specific metrics and GPT-4 assessment for graph plans.Single tasks use F1 and accuracy; sequential tasks use F1 and normalized Edit Distance; graph tasks additionally use a GPT-4 Score.
- Datasets: The evaluation dataset combines GPT-4-generated pseudo-labels with a human-annotated set of 46 complex requests.The pseudo-labeled data cover all three task categories, while expert annotators label selected complex requests.
- Quantitative Evaluation: GPT-3.5 outperforms Alpaca-7b and Vicuna-7b across all evaluated task types, with especially strong advantages on sequential and graph tasks.The paper reports this pattern on GPT-4-annotated datasets.
- Ablation Studies: Increasing the variety of few-shot demonstrations is evaluated as an ablation factor for task planning.The study varies both the number and variety of demonstrations, with Table 7 reporting results for different task-type variety.
- Human Evaluation: Human experts evaluate 130 diverse requests using passing rate, rationality, and success rate across planning, model selection, and response generation.The human evaluation reaches similar conclusions: GPT-3.5 substantially outperforms Alpaca-13b and Vicuna-13b across stages.
5 Limitations
HuggingGPT remains constrained by its reliance on LLM planning, repeated LLM interactions, limited context length, and instability.
- Planning: HuggingGPT cannot ensure that LLM-generated plans are always feasible or optimal because planning heavily relies on the LLM’s capability.The paper identifies improving LLM planning ability as an important direction.
- Efficiency: Multiple LLM interactions throughout the workflow increase response-generation time costs.The efficiency challenge follows from coordinating task automation through a powerful controller.
- Token Lengths: Limited LLM token length makes it difficult to connect numerous models while preserving concise, effective model descriptions.The paper notes that even a 32K-token maximum can remain insufficient for this setting.
- Instability: LLM uncontrollability contributes to instability in the collaborative system.Instability is listed as a separate improvement area in the paper’s limitations discussion.
6 Conclusion
The conclusion presents HuggingGPT as an LLM-controlled system that uses language to connect expert models and coordinate complex AI-task workflows. Its design combines task parsing, model selection, execution, and response integration, while relying on model descriptions and local or cloud endpoints.
- Conclusion: HuggingGPT uses language as an interface between an LLM controller and AI models from machine-learning communities.The LLM exploits language understanding and reasoning to manage external models.
- Workflow: The controller decomposes user intent into subtasks, assigns suitable models from their descriptions, and integrates their outputs into a final response.The workflow is organized around planning, model selection, task execution, and response generation.
- Task Representation: Parsed tasks use slots for the task name, identifier, dependencies, and multimodal arguments.The template includes task, id, dep, and args fields, with text, image, audio, and video inputs under args.
- Inference Endpoints: HuggingGPT can use cloud inference endpoints or local endpoints when models lack cloud availability, are time-consuming, or face network constraints.Local endpoints are faster but cover fewer models.
A.1.4 Task List
The task list covers the supported AI tasks used by HuggingGPT, while GPT-4 Score evaluates whether planned tasks can solve requests with valid logical relationships and order.
- Supported Tasks: HuggingGPT supports 24 AI tasks spanning language, vision, speech, and other modalities.The detailed supported-task list is presented in Table 13.
- GPT-4 Score: GPT-4 Score evaluates a planned task list against the user request, supported task list, solution reachability, and logical task relationships.GPT-4 also provides a rationale for its judgment and receives positive and negative planning demonstrations.
- Evaluation Caveat: GPT-4 Score is not always correct despite high correlation, motivating exploration of more reliable planning metrics.The paper explicitly treats this score as an imperfect evaluation measure.
A.1.6 Human Evaluation
Human experts evaluated HuggingGPT’s task planning and model selection using metrics that assess execution, rationality, and whether final results satisfy user requests.
- 130 diverse user requests were created by sampling 3–5 Hugging Face tasks with different random seeds.Three human experts evaluated the resulting samples.
- Passing Rate measures whether the planned task graph or selected model can be executed successfully.
- Rationality assesses whether generated task sequences or selected tools align rationally with user requests.
- Success Rate verifies whether final results satisfy the user’s request.Three experts annotated the data and their average scores produced the final values.
A.2 Datasets for Task Planning Evaluation
The task-planning evaluation uses 3,497 diverse user requests annotated by GPT-4 and categorized by planning complexity.
- 3,497 diverse user requests were gathered for task-planning evaluation.
- GPT-4 was used to annotate the requests because labeling task plans for every request was heavy.
- The dataset contains 1,450 single-task, 1,917 sequence-task, and 130 graph-task requests.
A.3.1 Case Study on Various Tasks
HuggingGPT is evaluated across multimodal tasks and task forms, illustrating collaboration between an LLM and numerous expert models.
- HuggingGPT is extensively evaluated across diverse multimodal tasks, with selected cases shown in Figures 4 and 5.
- Table 11 is titled “Statistics on datasets for task planning evaluation.”
- The evaluated modalities include language, image, audio, and video.
- The task forms include detection, generation, classification, and question answering.
A.3.2 Case Study on Complex Tasks
HuggingGPT handles complex requests by planning and orchestrating multiple expert models, tracking context and dependencies, and aggregating their results.
- Complex-task orchestration: HuggingGPT uses task planning to organize multiple models when requests contain implicit tasks or require multifaceted information.
- Complex-task orchestration: In multi-round conversations, it tracks contextual state through dialogue-context management and accesses user-referenced resources.
- Complex-task orchestration: For detailed image description, HuggingGPT decomposes the request into five tasks and assigns expert models to gather complementary information.The five tasks are captioning, classification, detection, segmentation, and visual question answering.
- Complex-task orchestration: For requests containing several tasks, HuggingGPT orchestrates multiple expert models and lets the LLM aggregate their inference results.
- Complex-task results: HuggingGPT demonstrates promising performance on various forms of complex tasks through collaboration between an LLM and external expert models.
- Resource dependencies and multimodality: The system recognizes resource dependencies, injects earlier task results into later inputs, and supports parallel or serial audio-video model execution.It also integrates multiple user-input resources for simple reasoning.
B More Discussion about Related Works
HuggingGPT is presented as a specialized autonomous agent for professional AI tasks, using Hugging Face expert models rather than general-purpose tools. Its examples show global planning, model selection, execution, and response generation across language, vision, video, and speech tasks.
- Scenarios: HuggingGPT targets professional AI tasks across vision, language, and speech, while other autonomous agents mainly address daily requests.The comparison frames HuggingGPT as specialized through its use of Hugging Face models.
- Planning: HuggingGPT uses global planning to obtain the entire task queue within one query, unlike agents that iteratively generate the next task.The paper notes that neither planning strategy is judged universally better.
- Pipeline: HuggingGPT connects an LLM controller with numerous expert models through four stages: task planning, model selection, task execution, and response generation.This pipeline is described as supporting task automation and coordination of expert models.
- Case studies: A video-and-dubbing example plans text-to-video and text-to-speech tasks, then produces separate video and audio inference results.The generated files are /videos/5696.mp4 and /audios/3f9e.flac.