PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoML

Patara Trirat, Wonyong Jeong, Sung Ju Hwang

arXiv:2410.02958v2cs.LGcs.AIcs.CLcs.MA

TL;DR

Existing AutoML systems can be difficult to configure, while LLM-based frameworks often cover only parts of the pipeline. AutoML-Agent coordinates specialized agents using retrieval-augmented planning and verification; experiments report stronger success rates and downstream task performance across seven ML tasks.

  • Problem

    Existing AutoML systems can require technical expertise, and full-pipeline automation must coordinate data and model processes that can affect one another.

    Source: §1 · p. 1

  • Method

    AutoML-Agent uses retrieval-augmented planning, role-specific task decomposition, prompting-based execution, structured prompt parsing, and multi-stage verification.

    Source: §1 · p. 2

  • Results

    Experiments on seven ML tasks show that AutoML-Agent outperforms existing methods in success rate and downstream task performance.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The experiments support AutoML-Agent as a full-pipeline framework spanning data retrieval through model deployment.

    Source: §5 · p. 9

    The authors note possible security issues from undesirable outputs and privacy concerns associated with external retrieval services and API-based LLMs.

    Source: Impact Statement · p. 10

Abstract

from arXiv · show

Automated machine learning (AutoML) accelerates AI development by automating tasks in the development pipeline, such as optimal model search and hyperparameter tuning. Existing AutoML systems often require technical expertise to set up complex tools, which is in general time-consuming and requires a large amount of human effort. Therefore, recent works have started exploiting large language models (LLM) to lessen such burden and increase the usability of AutoML frameworks via a natural language interface, allowing non-expert users to build their data-driven solutions. These methods, however, are usually designed only for a particular process in the AI development pipeline and do not efficiently use the inherent capacity of the LLMs. This paper proposes AutoML-Agent, a novel multi-agent framework tailored for full-pipeline AutoML, i.e., from data retrieval to model deployment. AutoML-Agent takes user's task descriptions, facilitates collaboration between specialized LLM agents, and delivers deployment-ready models. Unlike existing work, instead of devising a single plan, we introduce a retrieval-augmented planning strategy to enhance exploration to search for more optimal plans. We also decompose each plan into sub-tasks (e.g., data preprocessing and neural network design) each of which is solved by a specialized agent we build via prompting executing in parallel, making the search process more efficient. Moreover, we propose a multi-stage verification to verify executed results and guide the code generation LLM in implementing successful solutions. Extensive experiments on seven downstream tasks using fourteen datasets show that AutoML-Agent achieves a higher success rate in automating the full AutoML process, yielding systems with good performance throughout the diverse domains.

1. Introduction

AutoML-Agent is presented as a multi-agent LLM framework spanning the AI development pipeline, from data retrieval to deployment. Its planning and implementation strategies address the interdependent search and code-generation challenges of full-pipeline AutoML.

1. Introduction

  • Full-pipeline AutoML must coordinate data and model processes because choices in one can affect later processes in the other.Combining outputs from separate frameworks can lead to suboptimal solutions, while search should also minimize computational overhead.Source: §1 · p. 1
  • Interdependencies among pipeline steps enlarge the search space, and different downstream tasks add task-specific requirements.Retrieved dataset types affect preprocessing and network design, which in turn shape hyperparameter choices.Source: §1 · p. 2
  • LLM-generated pipelines may contain incomplete code, incorrect or missing dependencies, or undiscovered bugs, while ambiguous descriptions complicate code generation.The paper identifies requirement analysis and code generation based on disambiguated requirements as needs for accurate implementation.Source: §1 · p. 2
  • AutoML-Agent accepts user task descriptions and coordinates specialized agents to produce a deployment-ready model and inference endpoint.Its retrieval-augmented planning generates multiple plans from retrieved knowledge, then uses role-specific decomposition and prompting-based execution; prompt parsing and multi-stage verification support code-generation instructions.Source: §1 · p. 2
  • The authors report superiority across seven downstream tasks and fourteen datasets.The introduction describes extensive experiments; it does not specify the comparative metrics or values here.Source: §1 · p. 2

2. Related Work

Related work describes traditional AutoML systems as complex to configure and LLM-based frameworks as limited in pipeline coverage or task scope. AutoML-Agent is contrasted with these approaches through retrieval-augmented planning, plan decomposition, prompting-based execution, and verification.

2. Related Work

  • Traditional AutoML systems often have complex configuration procedures and steep learning curves that limit accessibility to non-experts.Only a few cited works support multiple pipeline steps; traditional interfaces can also be time-consuming for experienced users.Source: §2 · pp. 2–3
  • Existing LLM-based frameworks often address a specific pipeline step or downstream task type, while some full-production-pipeline attempts remain task-specific.The paper also notes that some methods overlook LLM capabilities, making search time-consuming for sophisticated planning and verification.Source: §2 · p. 3
  • AutoML-Agent combines retrieval-augmented planning, plan decomposition, prompting-based execution, structure-based prompt parsing, and multi-stage verification.Table 1 is identified as a summary of differences from existing LLM-based frameworks.Source: §2 · p. 3, Table 1 · p. 3

3. AutoML-Agent

AutoML-Agent coordinates specialized LLM agents across a full AutoML pipeline, combining retrieval-augmented plan search, role-specific execution, and verification before producing deployment-ready code.

3. AutoML-Agent

  • The framework spans user-instruction intake, planning, plan execution, verification, and code generation for a deployment-ready model.Failed execution or implementation checks trigger revision steps.Source: §3 · p. 3, §3.2 · p. 4

3.1. Agent Specifications

  • The Agent Manager coordinates specialized agents: Prompt parses instructions, Data analyzes data, Model searches and tunes models, and Operation writes executable code.The framework can use multiple Data or Model agents per task to increase parallelism.Source: §3.1 · pp. 3–4

3.2. Framework Overview

  • The workflow verifies the request, parses it into standardized JSON, generates retrieval-augmented plans, executes them, verifies outcomes, then generates and verifies code.Failures at execution or implementation lead to plan revision.Source: §3.2 · p. 4

3.3. Instruction Data Generation and Prompt Parsing

  • Prompt Agent is instruction-tuned on generated instruction-response pairs to parse user requirements into standardized JSON keys spanning user, problem, dataset, model, knowledge, and service.Those keys support communication among agents and provide context for pipeline generation.Source: §3.3 · pp. 4–5

3.4. Retrieval-Augmented Planning

  • Retrieval-augmented planning uses parsed requirements and retrieved knowledge to generate multiple independent end-to-end plans, broadening exploration while allowing parallelization.Retrieved material includes relevant knowledge and insights gathered through API calls such as web search and paper summaries.Source: §3.4 · p. 5

3.5. Prompting-Based Plan Execution

  • For each plan, role-specific decomposition gives agents smaller subtasks; Data Agent summarizes data work, and Model Agent uses those outcomes for model search and HPO.After verification, the best outcome is translated into code by Operation Agent as the deployment-ready model.Source: §3.5 · pp. 5–6

3.6. Multi-Stage Verification

  • Three verification stages check request adequacy, whether executed pipelines meet user requirements, and whether compiled code meets those requirements.Unsuccessful execution or implementation is documented and informs revised plans; without actual data, the LLM-knowledge fallback ultimately fails implementation verification at runtime.Source: §3.6 · p. 6

4. Experiments

Experiments evaluate AutoML-Agent across seven tasks and five modalities, then compare its results and efficiency with baselines and test framework components and robustness. AutoML-Agent leads on comprehensive score and constraint-aware downstream performance, while additional analyses examine plan count, noise, search speed, and cost.

4.1. Setup

  • Evaluation spans seven tasks across five modalities and 28 generated models, using task-specific metrics and a success-rate scale that includes deployment and constraint conformity.Baselines include handcrafted models, AutoGluon variants, DS-Agent, and zero-shot GPT-3.5 and GPT-4.Source: §4.1 · pp. 6–7

4.2. Main Results

  • AutoML-Agent consistently leads baselines in constraint-aware success rate, averaging 87.1%; all methods score higher in the easier constraint-free setting.The authors suggest retrieved planning knowledge helps agents focus on meeting constraints.Source: §4.2 · p. 7, Figure 4 · p. 8
  • AutoML-Agent leads all baselines on comprehensive score and achieves the best downstream performance across tasks in the constraint-aware setting, especially on complex tasks.General-purpose LLMs also perform relatively well on classical tabular classification and regression tasks.Source: §4.2 · pp. 7–8, Figure 4 · p. 8

4.3. Additional Analysis

  • Ablations show that RAP without decomposition or verification can reduce performance and fail to produce runnable models; decomposition alone still leaves some such failures.The full framework combines all components, including multi-stage verification that feeds back to agents.Source: §4.3 · p. 8
  • Plan count did not significantly affect success rate, but affected NPS and CS; adding plans did not necessarily improve results, so the authors chose three.Similar plans may lead to similar outcomes.Source: §4.3 · p. 8
  • Agents were not highly sensitive to system-prompt phrasing when their roles were clearly defined, and performance remained largely unaffected by injected noisy examples.The noise tests injected irrelevant or low-quality examples before or after insight extraction; the authors attribute resilience to error correction and multi-stage verification.Source: §4.3 · p. 9
  • AutoML-Agent searched about 8x faster than SELA while averaging 0.612 versus SELA’s 0.599; a deployable model search averaged 525 seconds and $0.30.The time and monetary cost figures are per single run using GPT-4o.Source: §4.3 · p. 9, Figure 7 · p. 9

5. Conclusions

AutoML-Agent is a multi-agent LLM framework for full-pipeline AutoML, from data retrieval to model deployment. It combines retrieval-augmented planning and multi-stage verification, and the experiments report higher success rates and downstream task performance.

5. Conclusions

  • AutoML-Agent covers the full pipeline from data retrieval to model deployment using a multi-agent LLM framework.It addresses planning complexity and implementation accuracy with retrieval-augmented planning and multi-stage verification.Source: §5 · p. 9
  • Experiments on seven ML tasks show that AutoML-Agent outperforms existing methods in success rate and downstream task performance.Source: §5 · p. 9
  • Role-specific decomposition and prompting-based execution improve plan execution efficiency.Source: §5 · p. 9

Impact Statement

The paper describes AutoML-Agent as a way to support non-expert users, while noting risks from harmful inputs and external services. Its additional discussions cover motivation, smaller-model challenges, and distinctions from general LLM-agent frameworks.

Impact Statement

  • The paper expects AutoML-Agent to help people with limited AI expertise use AI capabilities, while warning of possible misuse and privacy risks.The authors recommend Docker isolation and careful review of data included in API prompts.Source: Impact Statement · p. 10
  • The authors identify possible misuse and security issues from harmful inputs and recommend Docker isolation; API prompts may also raise privacy concerns.They advise users to review data included in prompts to avoid unintended disclosure.Source: Impact Statement · p. 10

Appendix

  • The appendix outlines discussions, experimental setup, prompts, intermediate and final results, and results with standard deviations.Source: Appendix · p. 13

A.1. Motivation and Necessity

  • The discussion frames AutoML-Agent as supporting researchers and engineers who lack AI expertise, with natural-language task descriptions and deployment-ready models.Examples include evaluating models on a novel dataset and prototyping a lightweight spam detector under constraints such as latency or accuracy.Source: §A.1 · p. 13

A.2. Challenges with Smaller Models

  • Smaller language models, including code-specialized models, failed to generate executable code for tasks requiring extensive planning or interdependent processes in the authors’ early tests.The passage also reports that vanilla GPT-3.5 and GPT-4o struggle with such tasks.Source: §A.2 · p. 13

A.3. Key Distinctions from General LLM-Agent Frameworks

  • The authors distinguish AutoML-Agent from general LLM-agent frameworks through AutoML-specific planning, specialized sub-task agents, and multi-stage verification.They report that general-purpose LLMs using the same backbone underperform specialized methods in success rates and downstream task metrics.Source: §A.3 · p. 14

A.4. Limitations and Future Work

The authors describe limits in code generation for new tasks and for pipelines unlike those evaluated, and identify further retrieval improvements as future work. Extending to reinforcement learning would require domain-specific modules.

A.4. Limitations and Future Work

  • Without skeleton code for entirely new tasks, AutoML-Agent may face increased code-hallucination risk, and code generation quality differs across backbone models.The reported gap includes GPT-4 versus GPT-3.5; reducing dependence on a specific backbone is proposed as future work.Source: §A.4 · p. 14
  • A PaperQA-style scientific querying system is proposed as a future way to improve citation relevance and context-aware prompt generation during planning.The authors suggest this could improve planning decisions.Source: §A.4 · p. 14
  • The experiments mainly cover general supervised and unsupervised settings, limiting code generation for tasks with substantially different pipelines.Reinforcement learning and recommendation systems are named as challenging extensions that would need additional agents for specialized steps.Source: §A.4 · p. 14
  • Extending AutoML-Agent to reinforcement learning would require domain-specific modules for environment interaction, action-space design, and reward processing.The authors envision an RL agent interfacing with an environment while the planner generates procedures such as policy training and evaluation.Source: §A.4 · p. 14

B. Details of Experimental Setup

The experimental setup specifies constraint-free and constraint-aware user instructions and provides a full-pipeline skeleton script. The script covers dataset handling, preprocessing, model training and tuning, evaluation, and deployment.

B. Details of Experimental Setup

B.1. Skeleton Python Script

  • The skeleton script organizes a text-classification pipeline from dataset retrieval and splitting through preprocessing, model selection, training, evaluation, and deployment.It returns processed data, the model, a deployment-ready model and endpoint, and performance and complexity scores.Source: §B.1 · pp. 16–18
  • The script evaluates text classification with Accuracy and F1 and also measures model complexity, including inference time and model size.It stores performance metrics by name and returns both performance and complexity scores.Source: §B.1 · p. 17

B.2. Dataset Descriptions

Table 5 summarizes downstream tasks and benchmark dataset statistics, with descriptions spanning varied datasets and data modalities.

B.2. Dataset Descriptions

  • The benchmark descriptions include classification datasets such as Butterfly, Shopee-IET, Ecommerce Text, Textual Entailment, Banana Quality, and Software Defects.The listed examples range from labeled images and text categories to binary defect prediction.Source: §B.2 · p. 18, Table 5 · p. 19
  • Crop Price has 2,200 samples, while the student-performance dataset has 145 instances and is used for unsupervised clustering rather than classification.Crop features include soil nutrients and weather conditions; the student dataset has 31 integer-type features.Source: §B.2 · p. 19
  • Weather provides 52,603 samples recorded every 10 minutes, and Electricity contains 26,211 hourly consumption samples from 321 customers.Both datasets are supplied without pre-specified splits.Source: §B.2 · p. 19
  • Cora and CiteSeer are citation networks with node, edge, feature, and class statistics and supplied train, validation, and test masks.Cora has 2,708 nodes and 7 classes; CiteSeer has 3,327 nodes and 6 classes.Source: §B.2 · p. 19

B.3. Baselines

The section describes task-specific human-model baselines, AutoGluon, zero-shot GPT models, DS-Agent, and SELA as comparison systems.

B.3. Baselines

  • Human-model baselines draw on benchmark or Kaggle models, including pretrained ResNet-18 for image classification and task-specific models across text, tabular, time-series, and graph tasks.The tabular clustering baseline is manually hyperparameter-tuned KMeans.Source: §B.3 · pp. 19–20
  • AutoGluon is the general AutoML baseline, using its time-series, tabular, and multimodal variants, but it does not cover graph data.The variants are AutoGluon-TS, AutoGluon-Tabular, and AutoGluon-Multimodal.Source: §B.3 · p. 20
  • GPT-3.5 and GPT-4 are evaluated as zero-shot baselines with a prompt requesting complete, runnable full-pipeline code.The prompt supplies the task instruction and a full-pipeline skeleton script.Source: §B.3 · p. 20
  • DS-Agent is modeling-focused and assumes complete data and evaluation code; the study uses its deployment stage, case banks, and Adapter prompt.Its source code for collecting manual human insights during development was unavailable.Source: §B.3 · p. 20
  • SELA uses Monte Carlo Tree Search with LLM agents and iteratively improves results from experimental feedback; the paper describes AutoML-Agent as more efficient with comparable downstream performance.The stated efficiency distinction is that AutoML-Agent bypasses the training process.Source: §B.3 · p. 20

B.4. Evaluation Metrics

This section describes how the system’s executability, task performance, and combined score are evaluated, alongside the agent prompts and procedures used for retrieval-augmented planning, plan execution, model search, and multi-stage verification.

B.4. Evaluation Metrics

  • Success Rate grades whether generated code runs and, at higher levels, provides deployment and meets constraints.Constraint-free grades are 0.00 for non-executable code, 0.50 for a runnable model, and 1.00 for a runnable model with an accessible endpoint; constraint-aware grades add 0.25, 0.75, and 1.00 levels for model execution, partial constraint satisfaction, and full constraint satisfaction, respectively.Source: §B.4 · pp. 20–21
  • NPS normalizes lower-is-better metrics, while a non-runnable model receives NPS 0 because it cannot make predictions.Accuracy and F1-score are reported as raw values; loss-based metrics such as RMSLE use NPS = 1/(1+s), yielding a 0–1 score where higher is better. NPS requires SR > 0.Source: §B.4 · p. 21
  • Comprehensive Score equally weights Success Rate and Normalized Performance Score: CS = 0.5×SR+0.5×NPS.The score evaluates pipeline execution success and downstream task performance simultaneously.Source: §B.4 · p. 21

C.1. Agent Specifications

  • The five agent specifications assign management and JSON parsing to manager roles, with data, modeling, and deployment tasks handled by specialized agents.The manager devises plans; the prompt agent converts requirements to JSON; the data agent retrieves and prepares data; the model agent searches and tunes candidates; and the operation agent implements, trains, deploys, and evaluates the solution.Source: §C.1 · p. 21, §C.1.1 · p. 21, §C.1.2 · p. 21

C.2. Variations of the System Prompt

  • The prompt-design experiment varies system prompts from generic wording to generic instructions and entry-, experience-, and world’s-best-level role descriptions.The appendix identifies these five prompt conditions as the variations used in the experiment described in §4.3.Source: §C.2 · p. 22

C.3.1. KNOWLEDGE RETRIEVAL PROMPT

  • Planning prompts retrieve and summarize task-relevant knowledge from Kaggle Notebooks, Papers with Code, arXiv, and web searches.The prompts ask for useful insights rather than source code, using the user's task, domain, or requirements to guide the summaries.Source: §C.3 · p. 23, §C.3.1 · pp. 23–24

C.3.2. PLANNING PROMPT

  • Planning prompts use retrieved experience to produce actionable end-to-end plans, while revision prompts use failed-plan findings to avoid repeating failures.Plans should be self-contained for AI agents, cover the pipeline from data handling through modeling, and provide instructions rather than code.Source: §C.3.2 · p. 25, §C.3.3 · p. 25

C.4.1. PLAN DECOMPOSITION: DATA AGENT

  • Prompt-based execution decomposes plans into specialized data and model work, and the Data Agent summarizes data retrieval, preprocessing, augmentation, and analysis steps.The data-agent summary should be reproducible and avoid visualization because the agent cannot see it.Source: §C.4 · p. 25, §C.4.1 · p. 26

C.4.2. PLAN DECOMPOSITION: MODEL AGENT

  • The model-agent decomposition prompt directs the agent to find high-performance models, optimize their hyperparameters, analyze dataset characteristics, and select top-k candidates.It frames the agent’s responsibilities around modeling and optimization tasks based on the project plan, user requirements, and data-scientist results.Source: §C.4.2 · p. 26

C.4.4. TRAINING-FREE MODEL SEARCH AND HPO BY MODEL AGENT

  • The training-free search/HPO prompt asks the model agent to explain reproducible steps for finding top-k candidates, specify optimized hyperparameters and values, and report performance and complexity metrics.Requested metrics include parameter count, FLOPs, model size, training time, and inference speed; estimates are allowed when exact values are unknown.Source: §C.4.4 · p. 27

C.5. Prompts for Multi-Stage Verification

  • Multi-stage verification checks intermediate results through request, execution, and implementation verification.The subsection introduces these stages as mechanisms for ensuring correctness between framework steps.Source: §C.5 · p. 27

C.5.1. REQUEST VERIFICATION

  • Request verification checks whether a request concerns ML or AI and whether it provides essential project information, such as the problem and a brief dataset description.The adequacy prompt is designed for non-expert users and accepts essential information without requiring detailed specifications.Source: §C.5.1 · p. 27, §C.5 · p. 27

C.4.3. PSEUDO DATA ANALYSIS BY DATA AGENT

The data-agent prompt specifies how to explain data-side work, while a separate verification prompt checks the implementation against requirements.

C.4.3. PSEUDO DATA ANALYSIS BY DATA AGENT

  • The data-agent prompt requests a reproducible, code-free account of retrieval, preprocessing, augmentation, and data analysis, including tools, rationale, and expected outcomes.It excludes visualization and modeling or training tasks.Source: §C.4.3 · p. 26

Implementation Verification

  • The implementation-verification prompt presents the Python code, execution result, and user requirements, then requires a binary Pass-or-Fail response.The prompt asks the project manager to verify whether code and results satisfy the requirements.Source: Implementation Verification · p. 28

D. Examples of Intermediate Results

The section illustrates intermediate outputs from prompt parsing, retrieval-augmented planning, plan decomposition, and plan execution. Examples include a butterfly image-classification workflow from neural-network selection and training through deployment and user feedback, alongside a plan-decomposition subsection.

D.1. Prompt Parsing

  • Prompt parsing turns banana-quality requests into structured task and dataset fields, while a constrained example retains the requested 0.98 accuracy and candidate model family.The examples distinguish a constraint-free prompt from one specifying XGBoost or LightGBM and an accuracy requirement.Source: §D.1 · pp. 28–29

D.2. Retrieval-Augmented Planning

  • With P = 3, the retrieval-augmented planning example provides actionable butterfly-classification plans covering preprocessing, model choices, training, evaluation, interpretability, and deployment.The plans include transfer learning and alternative architectures, alongside data handling, validation, reproducibility, and deployment steps.Source: §D.2 · pp. 30–31
  • The plan specifies an 80/10/10 train-validation-test split, cross-entropy training with SGD or RMSprop, monitoring classification metrics, and addressing imbalance and tuning with validation and cross-validation.It also recommends early stopping, learning-rate reduction on plateau, and weighted losses or oversampling.Source: §D.2 · p. 33
  • For interpretability and iteration, the plan proposes Grad-CAM or LIME, inspection of misclassified images, reproducible seeds and documentation, and version control.The visualizations are intended to show whether predictions rely on relevant biological features.Source: §D.2 · p. 33
  • The workflow extends to exporting and deploying the model for inference, creating an image-upload demo, and gathering user feedback to guide iterative updates.It suggests formats such as TensorFlow SavedModel or ONNX and optimizing inference for target devices when necessary.Source: §D.2 · pp. 33–34
  • The butterfly-classification plan targets a robust, accurate neural network, starting with fine-tuned pretrained CNNs and exploring alternative architectures, ensembles, and knowledge distillation.It names models including VGG16, MobileNet, and InceptionV3, and suggests combining patch-wise and image-wise classifiers.Source: §D.2 · pp. 32–34

D.3. Plan Decomposition

  • D.3 is introduced as “Example Results from Plan Decomposition,” but this excerpt provides only the subsection heading.Source: §D.3 · p. 34

Data Agent

This section outlines a workflow for retrieving, validating, preprocessing, and analyzing fashion-image data, alongside model candidates and resource-aware comparisons. It also presents clothing-classification pipeline examples, from data loading and model training and evaluation through preparation and deployment.

Data Agent

  • Load image paths and labels from the train/test CSV files, then check image formats, label mappings, dimensions, RGB conversion, and file integrity.The dataset directory is described as containing train and test folders and their label CSVs; the task covers four clothing categories.Source: Data Agent · p. 36
  • Normalize pixel values to [0, 1], resize images to a consistent 224x224 size, and split training data into training and validation sets.The suggested split is 80% training and 20% validation, supporting performance monitoring and overfitting prevention.Source: Data Agent · p. 36
  • Use augmentation pipelines ranging from horizontal flips and normalization to pipelines that also apply random cropping and random erasing.Other listed techniques include rotation, shifting, and zooming; augmentation is intended to increase data diversity and robustness.Source: Data Agent · p. 37
  • Analyze class distribution, image quality, and data patterns or anomalies, then use accuracy and a confusion matrix to examine performance and misclassifications.The plan proposes these checks to identify data issues that could affect model performance.Source: Data Agent · p. 37

Model Agent

  • The Model Agent proposes comparing ResNet18, EfficientNet-B0, and ViT-Small by expected accuracy, parameter/FLOP costs, memory use, and inference latency.The passage presents these as expected characteristics, not measured results.Source: Model Agent · p. 38

E. Showcase of Generated Pipelines

  • The constraint-free showcase generates a four-category clothing-classification pipeline using ResNet50 for training and evaluation, then preparing and deploying it through Gradio.The example code also includes data preprocessing and model preparation for deployment.Source: E. Showcase of Generated Pipelines · pp. 39, 41
  • The constraint-aware showcase prompt requests at least 85% test accuracy, PyTorch implementation, data augmentation, and a confusion matrix; the passage states these as requirements, not achieved outcomes.Source: E. Showcase of Generated Pipelines · p. 41
  • Data preparation stratifies the labels into train, validation, and test sets, applies image transforms, and creates batch-size-32 data loaders.Training transforms include random resized cropping, horizontal flipping, and rotation; evaluation transforms resize and center-crop images.Source: E. Showcase of Generated Pipelines · pp. 42–43
  • Training uses validation loss for early stopping and saves the best model, which is evaluated with accuracy, F1, inference time, and model size.The example sets a 100-epoch maximum and a patience of 10; it also generates a confusion matrix.Source: E. Showcase of Generated Pipelines · pp. 43–44
  • The deployment code loads the saved model, predicts a label for an input image, and exposes prediction through a shareable Gradio interface.The interface accepts a PIL image and returns a label.Source: E. Showcase of Generated Pipelines · p. 44
  • The main function runs the image-classification pipeline from data preparation through model definition, training, evaluation, deployment preparation, and deployment.It uses a pretrained ResNet-18 with a four-output classification layer.Source: E. Showcase of Generated Pipelines · p. 44

F. Detailed Experimental Results

The experiments report performance scores averaged over five runs and examine ablations, hyperparameters, generation costs, search time, and comparisons with recent agent systems.

F. Detailed Experimental Results

Loading 2410.02958v2…