PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Risks

Alexander Tornede, Difan Deng, Theresa Eimer, Joseph Giovanelli, Aditya Mohan, Tim Ruhkopf, Sarah Segel, Daphne Theodorakopoulos, Tanja Tornede, Henning Wachsmuth, Marius Lindauer

arXiv:2306.08107v3cs.LGcs.CL

TL;DR

AutoML tools are not ready to optimize the full LLM lifecycle, and standard approaches do not transfer straightforwardly to these resource-intensive models. The paper surveys both how AutoML could improve LLMs and how LLMs could enhance AutoML, while identifying integration risks. It concludes that an open-source LLM designed with AutoML goals in mind is a next step.

  • Problem

    Standard AutoML approaches do not transfer straightforwardly to resource-intensive LLMs, while the potential for reciprocal improvement between AutoML and LLMs remains to be explored.

    Source: §1 · p. 2, §2 · p. 3

  • Method

    The paper surveys existing work on AutoML for LLM lifecycle stages and on using LLMs to improve AutoML, and critically assesses integration risks.

    Source: §1 · p. 2

  • Results

    The paper concludes that an open-source LLM designed specifically with AutoML goals in mind is a next step.

    Source: §5 · p. 18

  • Takeaways & Limitations

    The authors see potential for AutoML to improve LLM lifecycle stages and for LLM capabilities to enhance AutoML interaction and technical components.

    Source: §1 · p. 2

    Scaling offline meta-learning and evaluating its resulting policy for LLMs remains an open problem because collecting the required data and evaluations is not trivially feasible.

    Source: §2.2.3 · p. 6

Abstract

from arXiv · show

The fields of both Natural Language Processing (NLP) and Automated Machine Learning (AutoML) have achieved remarkable results over the past years. In NLP, especially Large Language Models (LLMs) have experienced a rapid series of breakthroughs very recently. We envision that the two fields can radically push the boundaries of each other through tight integration. To showcase this vision, we explore the potential of a symbiotic relationship between AutoML and LLMs, shedding light on how they can benefit each other. In particular, we investigate both the opportunities to enhance AutoML approaches with LLMs from different perspectives and the challenges of leveraging AutoML to further improve LLMs. To this end, we survey existing work, and we critically assess risks. We strongly believe that the integration of the two fields has the potential to disrupt both fields, NLP and AutoML. By highlighting conceivable synergies, but also risks, we aim to foster further exploration at the intersection of AutoML and LLMs.

1 Introduction

The paper surveys a two-way integration of AutoML and LLMs: AutoML could improve the LLM lifecycle, while LLMs could enhance AutoML tools. It also identifies barriers and risks to that vision.

1 Introduction

  • The survey examines AutoML for LLM pre-training, fine-tuning, and inference, then considers LLMs for AutoML interaction, configuration, and component replacement.It also critically assesses risks of integration.Source: §1 · p. 2
  • Current AutoML approaches are not ready to holistically optimize the LLM lifecycle because pre-training is computationally demanding and LLM training spans stages with different metrics and learning paradigms.The paper frames these as challenges to lifecycle-wide optimization.Source: §1 · p. 2
  • LLMs could improve AutoML human-machine interaction, ease tool configuration, and enhance internal components through meta-learning from unstructured data.The proposed benefits span user interaction and technical components within AutoML tools.Source: §1 · p. 2
  • Integration risks include inadequate evaluation, catastrophic behavior from hallucinating LLM-powered components, excessive trust in natural-language explanations, and growing resource demands.These are identified as risks rather than reported outcomes.Source: §1 · p. 3

2 AutoML for LLMs

The paper argues that standard AutoML methods do not transfer straightforwardly to LLMs, whose lifecycle poses costly, multi-stage optimization challenges. It outlines five challenges, including architecture search and mismatched objectives and learning paradigms.

2 AutoML for LLMs

  • Standard HPO tools cannot be applied off-the-shelf to very deep, resource-intensive neural networks such as LLMs.The paper situates this among the new challenges that make existing AutoML approaches only partially useful.Source: §2 · p. 3
  • Pre-training is so expensive that only a few, perhaps even a single, full base-model training run may be feasible.This constrains the number of full evaluations available for optimization.Source: §2 · p. 3
  • LLM lifecycle optimization spans pre-training, multiple fine-tuning steps, and inference, but these stages are usually optimized independently rather than jointly.The paper describes holistic optimization across these steps as complex and desirable.Source: §2 · p. 3
  • Finding task-appropriate neural architectures remains difficult, and current NAS methods have not produced ground-breaking architectures.The paper notes that architecture design is still largely a tedious challenge.Source: §2 · p. 3
  • LLM lifecycle stages use different metrics, and pre-training metrics can be noisy, biased, or misleading proxies for downstream performance.AutoML commonly optimizes one learning paradigm at a time, while LLM training uses different paradigms across stages.Source: §2 · p. 3

2.1 Background on LLMs

The LLM lifecycle typically comprises pre-training, fine-tuning, and inference, each with distinct objectives and design decisions. Fine-tuning specializes pretrained models, while inference uses them to generate task outputs.

2.1 Background on LLMs

  • The LLM lifecycle has three stages—pre-training, fine-tuning, and inference—with distinct objectives, hyperparameters, and design decisions.The stages are typically considered separately because their objectives differ.Source: §2.1 · p. 3, Figure 1 · p. 4
  • Pre-training learns general language representations from large unlabeled text corpora, usually as the most expensive training stage.It uses a quasi-unsupervised pretext task; only a few groups and companies can afford to pre-train a base model.Source: §2.1 · p. 3
  • Fine-tuning specializes a pretrained model on domain- or task-specific data, including supervised, instruction, and alignment approaches.Instruction tuning teaches models to follow prompts across tasks; alignment seeks to approximate human value-related objectives such as helpfulness and harmlessness.Source: §2.1 · p. 4
  • At inference, a fine-tuned LLM generates text for tasks such as question answering using its learned knowledge and contextual understanding.Source: §2.1 · p. 5

2.2 Challenge I: Cost of Pre-Training Base Models

The cost of LLM pre-training makes standard black-box AutoML impractical, motivating lower-cost approaches. Existing strategies reduce evaluation cost or adjust training as it proceeds, but each has important scope or scaling constraints.

2.2 Challenge I: Cost of Pre-Training Base Models

  • 100 months—over eight years—would be needed for 100 pre-training runs on a thousand V100 GPUs under the stated example, making black-box AutoML infeasible.The example assumes 10 hyperparameters and at least 10 samples per hyperparameter; a single GPT-3 training was estimated at months on a thousand V100 GPUs.Source: §2.2 · p. 5

2.2.1 Prior-Guided Multi-Fidelity Optimization with Scaling Laws

  • Multi-fidelity optimization can approximate costly evaluations with cheaper runs, while prior-guided methods aim to reduce the number of full training runs.Mallik et al. (2023) achieved strong performance within ten full training runs; multi-fidelity results depend on choosing a fidelity whose rankings correlate with maximum-fidelity rankings.Source: §2.2.1 · p. 5
  • Tuning a smaller model and transferring its configuration can cut cost, but applies only to hyperparameters with stable optima across model scales.Dropout probability and weight decay were empirically found not to transfer well.Source: §2.2.1 · p. 5

2.2.2 Gradient-Based AutoML

  • Gradient-based HPO and cell-level architecture search have so far struggled with robustness and scaling to LLM-sized networks.Source: §2.2.2 · p. 6

2.2.3 Meta-Learning When and How to Adjust Training

  • Meta-learning training adjustments or the learning process itself remains difficult to scale because collecting meta-learning data and evaluating a meta-policy is not trivially feasible for LLMs.Expert checkpointing and adjusting learning rates during training resemble dynamic configuration approaches.Source: §2.2.3 · p. 6

2.3 Challenge II: A Multitude of Different Stages

LLM lifecycle stages have different objectives and metrics, making joint AutoML across the lifecycle too expensive and motivating stage-specific approaches. Pre-training, fine-tuning, and inference each bring distinct optimization decisions and constraints.

2.3 Challenge II: A Multitude of Different Stages

  • Jointly tuning all LLM lifecycle stages is too expensive, and their objectives do not share the same metrics, so they are usually optimized independently.Source: §2.3 · p. 6

2.3.1 Pre-Training

  • Pre-training optimization spans costly choices in data sources, preprocessing, tokenization, architecture, self-supervised strategy, and training settings.These decisions affect encoded knowledge, data quality, and downstream performance.Source: §2.3.1 · p. 6

2.3.2 Fine-Tuning

  • Supervised task-specific fine-tuning can use established AutoML approaches, but may undo alignment, so the two stages are ideally tuned jointly.Source: §2.3.2 · p. 7
  • Instruction-tuning optimization must address labor-intensive data curation, task selection, and cost functions that capture performance across multiple tasks.AutoML can target prompt and inference settings used to extract data, and hyperparameters such as gradient steps, batch size, optimizer, and learning rate.Source: §2.3.2 · p. 7
  • RLHF poses open AutoML challenges around evaluation, training stability, and noisy performance observations that can overfit tuning to a few random seeds.At the discussed point in time, RL tuning for LLMs was even more understudied than AutoRL on traditional control or game benchmarks.Source: §2.3.2 · pp. 7–8
  • RLHF-specific reward-model sizing lacks an established standard, while policy updates often modify only a subset of LLM weights.The subset depends on factors including data-concept complexity, preference scores, and model size; task-specific reward-model sizing and data/parameter trade-offs are potential optimization targets.Source: §2.3.2 · p. 8

2.3.3 Inference

  • Inference optimization must trade off deployment cost against user utility: pruning, reduced precision, output length, and sampling settings can affect these objectives differently.Automated inference optimization is still an open challenge.Source: §2.3.3 · p. 8
  • A cost-based pruning strategy using BlendSearch demonstrates AutoML optimization of LLM inference; distillation offers another target through tunable algorithm hyperparameters.Distillation creates smaller student models intended to perform similarly to teacher models.Source: §2.3.3 · p. 9

2.4 Challenge III: The Multitude of Performance Indicators

Choosing how to measure LLM performance depends on the use case, lifecycle stage, and relevant objectives. The paper highlights multi-objective evaluation and bias as considerations when assessing model quality.

2.4 Challenge III: The Multitude of Performance Indicators

  • LLM quality depends on application-specific metrics, which can include accuracy, inference time, memory, energy, image-text alignment, and adversarial robustness.Multi-objective AutoML can optimize several performance metrics.Source: §2.4 · p. 9
  • During base-model training, the downstream task is unknown, so the final performance metric is not yet known and later-stage capabilities serve as proxies.Source: §2.4 · p. 9
  • Assessing a model also requires considering bias, and AutoML can assist debiasing but human input remains necessary to center fairness values in the training pipeline.AutoML may target the amount and type of generated debiasing data, fine-tuning duration, or the subset of weights tuned.Source: §2.4 · p. 9

2.5 Challenge IV: Combination of Different Learning Paradigms and Modalities

Current AutoML tools are not prepared to jointly optimize LLM training across its learning paradigms and modalities. Such optimization must handle many design choices, multiple performance indicators, and an unknown distribution of downstream tasks.

2.5 Challenge IV: Combination of Different Learning Paradigms and Modalities

  • Jointly optimizing the LLM pipeline entails potentially hundreds of design decisions, several performance indicators, and an unknown distribution of downstream tasks.The stages require separate design and configuration spaces, but the goal is to optimize them jointly.Source: §2.5 · p. 10
  • Possible ways forward include studying interactions among design decisions, finding feasible subsets, developing proxy performance signals, and using multi-objective optimization across downstream tasks.These are proposed directions for making joint optimization tractable.Source: §2.5 · p. 10
  • For multimodal LLMs, modality-specific design options expand the configuration space, and useful fidelity types for efficient AutoML remain uncertain.Potential fidelity choices include data subsets from different modalities and partial training of different components.Source: §2.5 · p. 10

2.6 Challenge V: Determining Neural Architectures for LLMs

Neural architecture choices for LLMs remain difficult to automate: current NAS struggles to discover architectures comparable to human-designed advances, and its NLP benchmark is based on older RNNs rather than transformers. Tailored NAS approaches and parameter-efficient fine-tuning offer promising directions, but LLM search must also account for parallelization.

2.6 Challenge V: Determining Neural Architectures for LLMs

  • NAS aims to reduce manual architecture-design effort, yet current methods have not found innovative state-of-the-art architectures like human-discovered self-attention.The text says NAS may help optimize particular LLM lifecycle stages, but cannot be applied off the shelf without a suitable, potentially adapting search space and computational resources.Source: §2.6 · p. 10
  • Transformer-focused NAS includes language-model search strategies, BERT inter-layer searches, and vision-transformer supernets trained with weight entanglement.The cited approaches use evolution to search for models or supernets.Source: §2.6 · p. 10
  • Fine-tuning can require fewer parameters than pre-training, so selecting which layers or parameter subsets to train may improve downstream performance and parameter efficiency.Options discussed include random parameter subsets, adapter layers, and low-rank adaptation that freezes pre-trained weights and adds a correction term.Source: §2.6 · pp. 10–11
  • First empirical results suggest NAS can support language-model fine-tuning, including task-specific BERT compression and adapter hypernetworks with shared parameters.The survey notes that methods for training hyper- or super-networks may transfer to optimizing LLM fine-tuning.Source: §2.6 · p. 11
  • NAS for LLMs must account for parallelization because it changes architectural and hyperparameter choices that affect stability, convergence speed, and performance.The section therefore identifies parallel awareness as important for NAS approaches.Source: §2.6 · p. 11
  • NAS benchmarks have accelerated algorithm development, but the known NLP benchmark covers about 14k RNN architectures, not modern transformers.The authors identify a reproducible, efficient NAS benchmark for LLMs as an open challenge.Source: §2.6 · p. 11

3 LLMs for AutoML

The survey presents LLMs as potential aids to AutoML at three levels: human interaction, system configuration, and internal components. It describes possible applications ranging from conversational interfaces and textual reports to configuration suggestions and meta-learned replacements, while noting risks and practical constraints.

3.1 Opportunity I: Improving Human-Machine-Interaction with LLMs

  • LLMs could make AutoML easier for non-experts through text-based interaction, while also helping users interpret optimization results in natural language.The section identifies interfaces and textual interpretations as two directions for improving human-machine interaction.Source: §3 · p. 11, §3.1 · p. 12

3.1.1 Opportunity I.a: LLMs as an Interface to AutoML

  • Most AutoML tools target technically knowledgeable ML experts more than domain experts, and conversational assistants could elicit requirements and configure systems accordingly.A chatbot might guide a non-expert through choices such as selecting an appropriate optimization metric, although incorrect configurations remain a risk.Source: §3.1.1 · p. 12

3.1.2 Opportunity I.b: Interpretability of the AutoML Process

  • LLMs could turn optimizer run histories and related results into textual reports, optionally using plots as context and answering follow-up questions.Such reports may make interpretations more user-friendly, though the passage notes that users tend to trust textual explanations highly.Source: §3.1.2 · p. 13

3.2 Opportunity II: LLMs for Configuring AutoML

  • LLMs could suggest search spaces, runtime limits, termination decisions, and multi-fidelity settings for an AutoML task.Search-space choices can trade search speed against the risk of excluding optimal values; fidelity type and budget also influence multi-fidelity performance.Source: §3.2 · p. 14

3.3 Opportunity III: LLMs as Components of AutoML Systems

  • LLMs could replace AutoML subsystems as meta-learned components, including architecture proposal and candidate selection, or performance estimators.Examples include GPT-based architecture suggestion, evolutionary operators, and approaches that learn surrogate or acquisition functions from optimization histories.Source: §3.3 · pp. 14–15
  • Other proposed uses include LLM-assisted feature engineering and zero-shot pipeline suggestions, including systems that simulate AutoML without evaluating a pipeline.A further approach generates code for concrete AutoML tasks iteratively; prompt design and the size of accumulated context can constrain these systems.Source: §3.3 · pp. 14–15

4 Risks

The paper outlines five risks of integrating LLMs and AutoML: prompt-engineering burdens, biased evaluation, false outputs, misplaced user trust, and resource consumption.

4.1 Risk I: Complicated Human-Interaction

  • Reliably extracting LLM knowledge about AutoML still depends on prompt engineering, and automated prompt construction remains an open problem.The authors warn that integration may shift the entry barrier from coding and ML knowledge to prompt-engineering knowledge.Source: §4.1 · p. 16

4.2 Risk II: Evaluation

  • LLMs may have seen public evaluation datasets during training, making results akin to data snooping and strongly biased; no approach guarantees full knowledge removal, so new protocols are needed.The authors discuss evaluating on post-training datasets, but note the challenge of obtaining realistic, freely accessible data that the LLM has not seen.Source: §4.2 · p. 16

4.3 Risk III: False Facts and Misuse

  • Hallucinated LLM outputs can lead to inappropriate AutoML configurations, while running experiments can check replies against observed results.The authors suggest AutoML-specific knowledge graphs, safeguards for degenerate solutions, and predefined defaults as possible protections.Source: §4.3 · pp. 16–17

4.4 Risk IV: Trust and Explanations

  • Laypeople may trust text-based interactions even when the returned model is unsuitable, so systems should surface ambiguities and assumptions and explain how replies were generated.Source: §4.4 · p. 17

4.5 Risk V: Resource Consumption

  • Combining resource-intensive fields could increase consumption, so the authors stress transparency about resources and improving efficiency on both sides.They also note that accumulated AutoML meta-knowledge could potentially yield efficiency gains.Source: §4.5 · p. 17

5 Conclusion

The conclusion identifies concentrated resources for training and maintaining capable LLMs as a major challenge for the AutoML community. It proposes an open-source LLM designed for AutoML as a next step, while emphasizing that the integration remains at an early stage.

5 Conclusion

  • The authors describe LLM–AutoML integration as an early-stage effort with challenges, opportunities, and risks still to address.Source: §5 · p. 17
  • Training and maintaining the most capable LLMs requires immense resources, leaving only a few groups able to provide them and complicating AutoML research, data scrutiny, and safeguards.Source: §5 · p. 17
  • The authors identify developing an open-source LLM with AutoML-specific goals as a next step.Source: §5 · p. 18

Related papers

Loading 2306.08107v3…