PaperlayerView exact revision on arXiv

Source-linked AI summary

MLCopilot: Unleashing the Power of Large Language Models in Solving Machine Learning Tasks

Lei Zhang, Yuge Zhang, Kan Ren, Dongsheng Li, Yuqing Yang

arXiv:2304.14979v2cs.LGcs.AI

TL;DR

Configuring ML for specific tasks is difficult, and LLMs struggle when given only a task description. MLCopilot retrieves historical task experience and uses LLMs to develop solutions; it produces competitive solutions for novel tasks and achieves the highest normalized accuracy across three trials.

  • Problem

    Configuring ML for specific tasks is labor-intensive, while LLMs given only task descriptions perform no better than random generation.

    Source: §1 · pp. 1–2

  • Method

    MLCopilot canonicalizes historical task data, elicits knowledge, retrieves relevant experiences for a target task, and prompts LLMs to suggest solutions.

    Source: §1 · p. 2

  • Results

    MLCopilot provides reasonable and competitive solutions for novel tasks and achieves the highest normalized accuracy across all three trials.

    Source: §1 · p. 2, §5.2 · p. 7

  • Takeaways & Limitations

    The framework demonstrates LLM use for practical ML tasks involving heterogeneous inputs and intricate reasoning.

    Source: §6 · p. 8

    Performance can degrade when the prompt context window is severely limited.

    Source: §8 · p. 9

Abstract

from arXiv · show

The field of machine learning (ML) has gained widespread adoption, leading to significant demand for adapting ML to specific scenarios, which is yet expensive and non-trivial. The predominant approaches towards the automation of solving ML tasks (e.g., AutoML) are often time-consuming and hard to understand for human developers. In contrast, though human engineers have the incredible ability to understand tasks and reason about solutions, their experience and knowledge are often sparse and difficult to utilize by quantitative approaches. In this paper, we aim to bridge the gap between machine intelligence and human knowledge by introducing a novel framework, which leverages the state-of-the-art large language models to develop ML solutions for novel tasks. We showcase the possibility of extending the capability of LLMs to comprehend structured inputs and perform thorough reasoning for solving novel ML tasks. And we find that, after some dedicated design, the LLM can (i) observe from the existing experiences of ML tasks and (ii) reason effectively to deliver promising results for new tasks. The solution generated can be used directly to achieve high levels of competitiveness. Examples and code available at https://github.com/microsoft/CoML.

1 Introduction

Developing ML solutions for specific tasks demands substantial human effort, and existing automation can be slow and difficult to understand. MLCopilot uses historical task experience with LLMs to generate competitive solutions for novel tasks.

1 Introduction

  • Configuring ML solutions requires extensive human labor, and even the same algorithm may need reconfiguration when task context changes.Transferring successful experience across tasks also calls for high-level reasoning.Source: §1 · p. 1
  • AutoML optimization can require many trials, obscure its behavior, limit human control, and produce solutions whose transferability remains an open problem.The cited discussion names Bayesian optimization as one black-box optimization approach.Source: §1 · p. 1
  • Humans approach new ML tasks by understanding them and drawing on prior experience, motivating a framework that combines machine intelligence with human design patterns.The introduction points to LLMs’ human-like behavior on conversational tasks as a reason to explore this approach.Source: §1 · pp. 1–2
  • LLMs perform no better than random generation from task descriptions alone, while historical data require canonicalization and retrieval to fit limited in-context capacity.The framework also addresses the need for reasoning over knowledge when deriving solutions.Source: §1 · p. 2
  • MLCopilot canonicalizes historical data, elicits knowledge offline, then retrieves relevant task experiences to generate multiple ML solutions online.The framework is designed to provide reasonable and competitive solutions for novel tasks.Source: §1 · p. 2

2 Related Work

Prior work describes LLM strengths in language tasks alongside weaknesses in reasoning and mathematics. AutoML reduces manual effort but faces cost, transfer, and interpretability challenges that related approaches do not jointly resolve.

2.1 Large Language Models

  • LLMs perform well in applications such as chatbots and summarization, but remain limited on tasks requiring reasoning or mathematical skills.Task-automation approaches that decompose work into subtasks do not use experience from other tasks when planning a new one.Source: §2.1 · p. 2

2.2 Machine Learning and AutoML

  • AutoML methods include neural architecture search, meta-learning, and Bayesian optimization, but often require costly trial-and-error and offer limited interpretability or transfer.Prior approaches address some drawbacks, yet related work describes remaining limits in task coverage or search cost.Source: §2.2 · pp. 2–3

3 Preliminaries

ML solutions occupy a space of interrelated design choices, and MLCopilot draws on prior task experience to help develop solutions for new tasks.

3 Preliminaries

  • An ML solution may be a pipeline, configuration, or code snippet, with correlated design choices whose alternatives affect overall performance.The solution is framed as a sample from a complicated solution space.Source: §3 · p. 3
  • MLCopilot uses experience from relevant previous tasks for knowledge elicitation and to develop solutions for a novel task.The paper points to later sections for framework details.Source: §3 · p. 3

4 MLCopilot

MLCopilot tackles the difficulty of exploiting heterogeneous historical ML experience and solving new tasks by using LLMs to reason over experience-derived knowledge. Its offline preparation and online retrieval pipeline supports solutions for novel tasks.

4.1 Overall Framework

  • MLCopilot analyzes canonicalized historical experience offline, then retrieves relevant experience and knowledge online to guide an LLM's solution for a novel task.The framework is designed to address both how to comprehend abundant raw experiences and how to solve tasks using the resulting information.Source: §4.1 · p. 3, Figure 1 · p. 4

4.1.1 Offline Stage: Understanding and Reasoning

  • Historical records pair a task, solution, and evaluated performance; MLCopilot canonicalizes them into an experience pool and elicits knowledge offline for later use.Knowledge elicitation uses sampled experiences and automated post-validation, which tests candidate knowledge on validation tasks and can trigger regeneration.Source: §4.1.1 · p. 4, §4.5 · p. 6

4.1.2 Online Stage: Retrieving and Solving

  • For a new task, MLCopilot retrieves relevant demonstrations and knowledge, then invokes the LLM once to generate a recommended solution.The output is represented as ˜S = LLM(˜T, ˜E, ˜K).Source: §4.1.2 · pp. 4–5

4.2 Task Description in Natural Language

  • Users describe target tasks in natural language, including dataset names, characteristics, and domain constraints, without task-specific meta-features.The authors say this task-type-agnostic input does not require heuristics for designing meta-features.Source: §4.2 · p. 5

4.3 Retrieval

  • Experience retrieval ranks historical tasks by cosine similarity between task-description embeddings and selects the most relevant k as demonstrations.Knowledge retrieval instead matches the new task's solution space.Source: §4.3 · p. 5

4.4 Canonicalization

  • Canonicalization converts heterogeneous ML data into natural language and discretizes continuous values using their distributions and percentile points.The discretization is intended to avoid feeding raw numbers directly to the LLM while minimizing performance loss.Source: §4.4 · p. 5

4.5 Knowledge Elicitation

  • Knowledge elicitation samples experience, asks an LLM to summarize it, and validates candidate knowledge on experience-pool tasks, regenerating it until convergence or an invocation limit.The authors distinguish this process from simple extraction: it reasons over heterogeneous, experience-anchored resources without predefined templates or complicated pipelines.Source: §4.5 · pp. 5–6
  • The resulting natural-language knowledge may serve human practitioners as a cookbook, and the authors released it to encourage community knowledge sharing.The authors describe the released knowledge as a way to share findings and inspire future ML research.Source: §4.5 · p. 6

5 Experiment

Experiments test MLCopilot against traditional and LLM baselines, then examine its components across three ML benchmarks. Results report the highest normalized accuracy across three trials and show that retrieval, discretization, and knowledge use matter.

5.1 Experiment Setup

  • Tests cover HPO-B, PD1, and HyperFD across varied ML tasks, with three ordered solution attempts and metrics averaged over at least five seeds.Each method’s metric@t is its best performance among the first t suggested solutions.Source: §5.1 · p. 6

5.2 Main Results

  • MLCopilot achieved the highest normalized accuracy (nAcc) across all three trials, with especially significant improvement on nAcc@1.Table 2 reports results on HPO-B, PD1, and HyperFD.Source: Table 2 · p. 7, §5.2 · p. 7
  • On PD1, MLCopilot remained best, while LLM-ZS and LLM-FS failed and the Constant baseline nearly outcompeted other baselines.The authors note this casts doubt on the effectiveness of task similarities measured by other baselines and indicates PD1 is more challenging for LLMs.Source: §5.2 · p. 7
  • On HPO-B, LLM-FS achieved performance comparable to most baselines with few demonstrations; HyperFD’s post-cutoff release makes GPT-3.5 memorization of its best solutions unlikely.HyperFD was private and released after GPT-3.5’s knowledge cutoff.Source: §5.2 · p. 7

5.3 Ablation study

  • Text-embedding and meta-feature retrieval both outperformed random retrieval; text embeddings also avoid manual feature design for new task types.The authors describe text embeddings as more promising for varied tasks, while meta-feature design for complex tasks can be non-trivial.Source: §5.3 · pp. 7–8, Table 3 · p. 8
  • Removing knowledge reduced performance, while removing demonstrations caused a complete collapse.The ablation compares retrieving experience and knowledge using nAcc@1 or Rank@1.Source: §5.3 · p. 7, Table 4 · p. 8
  • Discretization mattered: direct continuous values were infeasible, and splits spanning all possible values could perform worse than no discretization; HyperFD was excluded.Split points should be based on statistics of the best solutions.Source: §5.3 · p. 8, Table 6 · p. 8
  • Removing online knowledge significantly decreased final performance, while offline post-validation improved its usefulness, though vague or hallucinated knowledge still occurred.The qualitative study describes knowledge as summarizing past experience and guiding parameter or setting adjustments based on task characteristics.Source: Table 5 · p. 8, §5.3 · p. 8

6 Conclusion

MLCopilot applies LLMs to practical ML tasks, including those with heterogeneous inputs and intricate reasoning.

6 Conclusion

  • The authors present MLCopilot as a way to extend LLM use from text-related tasks to practical ML tasks involving heterogeneous inputs and intricate reasoning.They characterize this as expanding the scope of LLM applications to complex problems.Source: §6 · p. 8

7 Ethical considerations

MLCopilot constrains recommended solutions to the solution space defined by the user. The ethical safeguard therefore depends on that space meeting ethical standards.

7 Ethical considerations

  • Recommended solutions remain within the user-provided solution space, which safeguards against unethical solutions only if that space itself meets ethical standards.The framework's foundational techniques may also be applied beyond strictly defined ML problems, where LLM limitations create potential unpredictability and call for ethical prudence.Source: §7 · p. 9

8 Limitations

The paper discusses potential benchmark leakage, MLCopilot's relationship to AutoML, and conditions that can limit its performance. Its algorithms describe offline knowledge elicitation and online solution generation.

8 Limitations

  • The authors flag possible pretraining leakage on benchmarks such as HPO-B, then report robust performance on HyperFD, a private benchmark published after GPT-3.5's claimed knowledge cutoff.They also report that canonicalizing data significantly enhanced performance, which they offer as evidence that capabilities extend beyond memorization.Source: §8 · p. 9
  • MLCopilot is unlikely to surpass state-of-the-art Bayesian optimization for superior solutions, and the Table 2 evaluation stopped at three suggestions after performance saturated.The authors cite LLM limitations in mathematical computation as the basis for distinguishing MLCopilot from AutoML.Source: §8 · p. 9
  • The authors describe MLCopilot's value as accepting arbitrary task descriptions, using diverse ML experience, and rapidly producing multiple out-of-the-box solutions; they suggest combining it with existing AutoML methods.The paper says performance can degrade when the prompt context window is severely limited, despite robustness tests involving LLM choice and task-description formats.Source: §8 · p. 9

A Algorithms

  • The offline algorithm takes historical data, a round limit, and validation-related inputs to produce an experience pool and selected knowledge; the online algorithm takes a new task description and the two pools to output a solution.The online-stage inputs and output are stated explicitly; the offline algorithm's listed inputs span multiple passages.Source: A Algorithms · p. 12

B.1 Benchmarks

The appendices describe three benchmarks, comparison baselines, post-validation, a PD1 surrogate, and example prompts and prompt-design comparisons.

B.1 Benchmarks

  • HPO-B covers configuration across 16 solution spaces and 101 datasets, PD1 includes 24 neural-network tuning tasks, and HyperFD evaluates face-detector configurations on unseen datasets.The benchmarks use nAcc for HPO-B and PD1; HyperFD provides average precision and rank for every solution on its datasets.Source: §B.1 · pp. 12–13

B.2 Compared Baselines

  • The compared baselines include random and constant portfolios, meta-feature similarity methods, Gaussian-process transfer, joint task-solution prediction, and offline portfolio generation.The descriptions include Random, Constant, ASKL, TST-M, HyperSTAR, and FLAML-Zero.Source: §B.2 · p. 13

B.3 Post-validation

  • Post-validation uses 10% of the training meta-dataset to filter generated knowledge through sampled questions and temperatures, candidate validation, and early stopping.The described loop tracks stagnation to determine when to stop.Source: §B.3 · p. 13

B.4 Building Surrogate Model for PD1

  • For PD1, a two-stage XGBoost surrogate first classifies top-performing solutions, then uses separate regressors for top-performing and all solutions.The benchmark reports normalized accuracy for PD1.Source: §B.1 · p. 13, §B.4 · p. 14

C Prompt Design

  • The prompt appendix provides example HPO-B prompts for offline and online use; Figure 3 varies experience count and demonstrations per task, while Table 9 compares description formats.The supplied passages identify the comparisons but do not state their outcomes.Source: Table 8 · p. 15, Table 7 · p. 15, Figure 3 · p. 16

D.1 Task description in the wild.

Task-description format and prompt length both affect MLCopilot’s performance. Anonymizing task names was particularly harmful, while adding demonstrations generally helped before performance saturated and fluctuated.

D.1 Task description in the wild.

  • Condensed or anonymized descriptions shifted performance, with anonymization particularly harmful; swapping task names improved results on PD1 and HyperFD.The authors conjecture that task names activate relevant LLM neurons and draw on memorized knowledge.Source: §D.1 · p. 16

D.2 Length of prompt.

  • Performance generally improved as prompt tokens increased, then saturated around 1.5k tokens and fluctuated; demonstrating more solutions had higher potential, especially above 3k tokens.The study varied the number of retrieved experiences and the number of solutions demonstrated per task.Source: §D.2 · p. 16

D.3 Choice of LLMs.

MLCopilot was tested with several LLMs beyond the main GPT-3.5 model. It remained competitive across tested models, with results trending higher for stronger and larger models.

D.3 Choice of LLMs.

  • The tested alternatives included GPT-3.5 Turbo, GPT-3 without instruction finetuning, and LLAMA-7B.GPT-3.5 Turbo is described as cost-efficient; LLAMA-7B as an open-source model with a relatively loose GPU-memory requirement.Source: §D.3 · pp. 16–17
  • MLCopilot remained competitive across tested LLMs, with a trend toward better results on stronger and larger models.Table 10 reports performance for MLCopilot equipped with different LLMs.Source: §D.3 · p. 17, Table 10 · p. 17

D.4 Noisy accuracy and faulty canonicalization.

The appendix tests robustness to corrupted historical accuracy and canonicalization, then presents LLM-generated configuration knowledge and evaluation setups for HPO-B and PD1.

D.4 Noisy accuracy and faulty canonicalization.

  • On HPO-B, MLCopilot's top-1 normalized accuracy fell from 81.59 to 78.54 under disturbance, remaining above FLAML at 77.84; similarly disturbing FLAML's input data reduced its score to 73.53.Source: §D.4 · p. 17
  • For the faulty-canonicalization test, 10% of canonicalized data was replaced with random discrete values; robustness was evaluated on HPO-B.Each configuration parameter had a 10% chance of replacement by a random choice among five values.Source: §D.4 · p. 17
  • Faulty canonicalization worsened performance more than perturbed accuracy, although results remained competitive with FLAML.The authors speculate that false canonicalization can mislead LLM reasoning.Source: §D.4 · p. 17, Table 12 · p. 17

E.1 HPO-B

E.2 PD1

Test task: UniRef50, Transformer

The appendix provides task-specific configuration guidance for UniRef50 and WMT15, then describes leave-one-out evaluation and generated guidance for HyperFD tasks.

Test task: UniRef50, Transformer

  • For larger batches and datasets, use higher learning rate and momentum, lower power, and higher lambda; for smaller batches and datasets, reverse these settings.Examples: Fashion MNIST CNN, batch size 256, uses learning rate 0.001, momentum 0.9, power 0.1, and lambda 0.01; CIFAR10 Wide ResNet, batch size 2048, uses 0.01, 0.9, 0.5, and 0.001, respectively. LM1B Transformer, batch size 2048, uses learning rate 0.001, momentum 0.9, power 0.01, and lambda 0.001.Source: Test task: UniRef50, Transformer · p. 25

Test task: WMT15, xformer

E.3 HyperFD

Test task: ANIME

  • For ANIME, the guidance relates crop size and anchor IoU to face counts and location-loss weight to the presence of facial landmarks.Source: Test task: ANIME · p. 26

E.3 HyperFD

Test task: FDDB

  • For FDDB, larger face counts call for larger crops and higher anchor IoU thresholds; facial landmarks call for lower learning rates and Adam or SGD.Source: Test task: FDDB · p. 27

Test task: WIDER FACE

  • WIDER FACE guidance scales crop size, anchor IoU, and negative-to-positive ratio with face counts, and scales location-loss weight with facial landmarks.Source: Test task: WIDER FACE · pp. 28–29
Loading 2304.14979v2…