Source-linked AI summary
AutoML-GPT: Automatic Machine Learning with GPT
Shujian Zhang, Chengyue Gong, Lemeng Wu, Xingchao Liu, Mingyuan Zhou
TL;DR
Tuning models for unseen private datasets is challenging. AutoML-GPT uses model and data descriptions to prompt an LLM that coordinates expert models through an automated training pipeline. Evaluations report automatic-training capability across multiple AI tasks and effectiveness on unseen-dataset hyperparameter tuning.
Problem
Hyperparameter tuning for unseen private datasets is challenging.
Source: §2.4 · p. 5Method
Results
Evaluations across language, vision, and continual learning demonstrate automatic-training capability and effectiveness in hyperparameter tuning for unseen or new datasets.
Source: §1 · p. 2Takeaways & Limitations
AutoML-GPT is presented as a natural-language interface for tuning machine-learning models across tasks, with use cases in computer vision, question answering, and classification.
Source: §5 · p. 9Future work includes automatically generating model and data cards and extracting task-aware subnetworks from large pretrained models.
Source: §5 · p. 9
Abstract
from arXiv · showhide
AI tasks encompass a wide range of domains and fields. While numerous AI models have been designed for specific tasks and applications, they often require considerable human efforts in finding the right model architecture, optimization algorithm, and hyperparameters. Recent advances in large language models (LLMs) like ChatGPT show remarkable capabilities in various aspects of reasoning, comprehension, and interaction. Consequently, we propose developing task-oriented prompts and automatically utilizing LLMs to automate the training pipeline. To implement this concept, we present the AutoML-GPT, which employs GPT as the bridge to diverse AI models and dynamically trains models with optimized hyperparameters. AutoML-GPT dynamically takes user requests from the model and data cards and composes the corresponding prompt paragraph. Ultimately, with this prompt paragraph, AutoML-GPT will automatically conduct the experiments from data processing to model architecture, hyperparameter tuning, and predicted training log. By leveraging {\ours}'s robust language capabilities and the available AI models, AutoML-GPT can tackle numerous intricate AI tasks across various tasks and datasets. This approach achieves remarkable results in computer vision, natural language processing, and other challenging areas. Extensive experiments and ablation studies demonstrate that our method can be general, effective, and beneficial for many AI tasks.
1 Introduction
AutoML-GPT uses language prompts built from model and data descriptions to connect LLMs with expert models and automate task-specific training. Evaluations span language, vision, and continual learning, including hyperparameter tuning on unseen or new datasets.
1 Introduction
- The pipeline covers data processing, model architecture design, hyperparameter tuning using predicted training logs, and human feedback on experimental data.Users can further interact with the LLM based on generated training logs.Source: §1 · p. 2
- The design supports use of external models on known benchmarks and knowledge transfer to unknown private datasets when only metadata is available.The authors also describe the pipeline as able to absorb task-specific experts and grow its capabilities.Source: §1 · p. 2
- AutoML-GPT combines LLMs with expert models to manage data processing, model architecture design, and task-specific experiments.The system uses language as an interface for LLMs to interact with users and invoke models as needed.Source: §1 · pp. 1–2
- Model-card and data-card descriptions are fused into a fixed-format prompt paragraph for a training pipeline targeting general AI tasks.The model cards describe models, while data cards describe the task-specific data.Source: §1 · p. 2
- Evaluations across language, vision, and continual learning demonstrate AutoML-GPT’s automatic-training capability and effectiveness in tuning hyperparameters for unseen or new datasets.The introduction also reports use cases across multiple AI tasks.Source: §1 · p. 2
2 AutoML-GPT
AutoML-GPT uses data and model information to format prompts for an LLM-controlled training pipeline. The workflow covers data processing, model architecture, hyperparameter tuning, and training-log generation.
2 AutoML-GPT
- AutoML-GPT formats a fixed prompt from data and model information, then processes user requests through data preparation, architecture design, hyperparameter tuning, and training-log generation.The LLM acts as controller and expert models as collaborative executors.Source: §2 · pp. 2–3
2.1 Input Decomposition
- Data cards describe dataset name, type, labels, and default metrics; model cards describe model name, structure, descriptions, and architecture hyperparameters, with optional user metrics or constraints.The system supplies these task specifications to the LLM as high-level instructions.Source: §2.1 · pp. 3–4
2.2 Data Processing
- AutoML-GPT selects data-processing techniques according to the project requirements and data characteristics described in the data card.Examples include image resizing, normalization, and augmentation, or text tokenization and cleanup.Source: §2.2 · p. 4
2.3 Model Architecture
- The system assigns models to tasks using model cards and user requests, while model descriptions specify design components such as layers, activations, and loss functions.The assignment mechanism is intended to support incremental model access and flexibility.Source: §2.3 · p. 4
2.4 Hyperparameter Tuning with Predicted Training Log
- AutoML-GPT predicts training logs for hyperparameter settings and transfers configurations to unseen datasets using similarity between their data-card metadata and existing datasets.The text encoder compares dataset scale, task description, label space, and input/output type; logs can include epoch-level training and validation metrics.Source: §2.4 · pp. 4–5, Figure 4 · p. 5
3 Experiments
AutoML-GPT uses data and model cards to configure a training pipeline across unseen-dataset tuning, object detection, question answering, and classification. The reported cases include improved hyperparameter prediction for a new dataset and strong results on classification.
3 Experiments
- The experiments use GPT-4 and include case studies intended to assess AutoML-GPT from multiple angles.The unseen-dataset test constructs cards from trained datasets, then predicts configurations for a new dataset created from held-out data.Source: §3 · p. 5, §3.1 · p. 5
3.1 Unseen Dataset
- 98% Top 1 accuracy for recommended hyperparameters on the unseen “New” dataset, versus 80% for average randomly selected hyperparameters.Using the suggested settings without additional datasets achieves 82% Top 1 accuracy, better than random selection but below the recommended setting.Source: §3.1 · p. 6
3.2 Object Detection
- For object detection on COCO, AutoML-GPT derives data processing, model architecture, and hyperparameter configurations from data and model cards, then tunes them using predicted training logs.The authors characterize the card-derived prompt as a flexible adaptation pipeline and a complement to work on LLM prompt components.Source: §3.2 · pp. 6–7
3.3 Question Answering
- In the hyperparameter-tuning example, AutoML-GPT’s generated hyperparameters align closely with those provided by DPR, including a learning rate of 10^-5 and maximum epochs of 40.Users can request different evaluation metrics or model architectures, and AutoML-GPT further tunes hyperparameters based on those requests and predicted logs.Source: §3.3 · p. 7
3.4 Classification
- 85.92% accuracy and 0.277 final validation loss are reported for XGBoost on UCI Adult using AutoML-GPT-suggested hyperparameters.The authors describe this as strong performance in classification despite differing inputs and tasks.Source: §3.4 · p. 7, Figure 7 · p. 8
4 Related Work
Related work covers LLM reasoning and GPT-based AI systems. AutoML-GPT is positioned as applying GPT to end-to-end automated training rather than only integrating it as an interaction agent.
Large Language Models
- The paper situates ChatGPT among large language models associated with zero-shot and few-shot learning, citing models including Megatron-turing NLG [Smith et al., 2022], Gopher [Rae et al., 2021], and PaLM [Chowdhery et al., 2022].It highlights ChatGPT’s conversational interaction and explores its zero-shot capability beyond dialogue.Source: §4 · p. 8
Chain of Thought
- Chain-of-thought research includes manually designed reasoning demonstrations and zero-shot approaches such as STaR [Zelikman et al., 2022] and ART [Paranjape et al., 2023].Self-consistency [Wang et al., 2022] and Interactive-Chain-Prompting [Pilault et al., 2023] are also cited as developments in this area.Source: §4 · p. 8
GPT-based Systems
- Prior GPT-based systems integrate GPT into AI systems, while AutoML-GPT focuses on building an end-to-end training pipeline.The discussion names HuggingGPT [Shen et al., 2023], VisualGPT [Wu et al., 2023], and OpenAGI [Ge et al., 2023] as examples of related systems.Source: §4 · pp. 8–9
5 Conclusion
The conclusion presents AutoML-GPT as a system for automatically conducting machine-learning experiments across several task types. It identifies automatic generation of model/data cards and task-aware subnetworks as future work.
5 Conclusion
- The authors report use cases in computer vision, natural question answering, and classification, and describe the method as effective and general.They also report a detailed unseen-dataset use case with additional user interactions.Source: §5 · p. 9
- The authors identify automatic generation of model and data cards for benchmarks and extraction of task-aware subnetworks from large pretrained models as future directions.Source: §5 · p. 9