PaperlayerView exact revision on arXiv

Source-linked AI summary

Archon: An Architecture Search Framework for Inference-Time Techniques

Jon Saad-Falcon, Adrian Gamarra Lafuente, Shlok Natarajan, Nahum Maru, Hristo Todorov, Etash Guha, E. Kelly Buchanan, Mayee Chen, Neel Guha, Christopher Ré, Azalia Mirhoseini

arXiv:2409.15254v6cs.LGcs.AIcs.CL

TL;DR

The utility and interactions of inference-time techniques across models and tasks, and the large space of possible combinations, remain insufficiently understood. ARCHON automatically designs systems from available models and techniques; its best architectures boost SOTA performance by 15.1% on average across diverse benchmarks, while requiring substantially more time and money than a single model API call.

  • Problem

    The utility and interactions of inference-time techniques across tasks and models remain insufficiently understood, while architecture search spans many choices and no single design maximizes accuracy across tasks.

    Source: §1 · pp. 1–2

  • Method

    ARCHON is a modular framework that automatically designs LLM systems from existing or new inference-time techniques and optimizes for accuracy, latency, or cost.

    Source: §1 · p. 2

  • Results

    ARCHON’s best architectures boost state-of-the-art (SOTA) performance by 15.1% on average across instruction-following, reasoning, and coding benchmarks, surpassing frontier models and prior inference-time architectures.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Practitioners can use ARCHON to search for architectures optimized for their chosen objective, including accuracy, latency, and cost.

    Source: §1 · p. 2

    ARCHON architectures can take 5x more time and money than a single LLM API call because they make multiple successive API calls.

    Source: §4.4 · p. 9

Abstract

from arXiv · show

Inference-time techniques, such as repeated sampling or iterative revisions, are emerging as powerful ways to enhance large-language models (LLMs) at test time. However, best practices for developing systems that combine these techniques remain underdeveloped due to our limited understanding of the utility of each technique across models and tasks, the interactions between them, and the massive search space for combining them. To address these challenges, we introduce Archon, a modular and automated framework for optimizing the process of selecting and combining inference-time techniques and LLMs. Given a compute budget and a set of available LLMs, Archon explores a large design space to discover optimized configurations tailored to target benchmarks. It can design custom or general-purpose architectures that advance the Pareto frontier of accuracy vs. maximum token budget compared to top-performing baselines. Across instruction-following, reasoning, and coding tasks, we show that Archon can leverage additional inference compute budget to design systems that outperform frontier models such as OpenAI's o1, GPT-4o, and Claude 3.5 Sonnet by an average of 15.1%.

1. Introduction

The introduction motivates automated search because inference-time techniques have task-dependent utilities and interactions, and their combinations create a large design space. ARCHON searches that space and reports stronger benchmark performance than frontier models and prior inference-time architectures.

1. Introduction

  • The design challenge is that architecture choices span models, sampling, and candidate combination, with no prevailing architecture that maximizes accuracy across tasks.Prior systems also have limitations in compute utilization and task generalization.Source: §1 · pp. 1–2
  • The authors evaluate existing and proposed inference-time techniques across instruction-following, reasoning, and coding tasks using open- and closed-source models.The techniques include ensembling, fusion, ranking, critiquing, verification, and model-based unit-test generation and evaluation.Source: §1 · p. 2
  • No single inference-time technique dominates across tasks; different approaches work better for different tasks.The evaluated techniques include ensembling, fusion, ranking, critiquing, verification, and model-based unit-test generation and evaluation.Source: §1 · p. 2
  • Generation ensembling combined with critique, verification, and fusion improves response quality beyond the oracle best individual response, particularly for instruction-following and reasoning.The introduction also reports gains from scaling layers and combining multiple inference-time approaches.Source: §1 · p. 2
  • ARCHON is an open-source modular framework that automatically designs systems of inference-time techniques and optimizes for accuracy, latency, or cost.Its architecture search uses Bayesian optimization to traverse potential inference architectures.Source: §1 · pp. 2–3
  • ARCHON’s best architectures boost state-of-the-art (SOTA) performance by 15.1% on average, surpassing frontier models and prior inference-time architectures.They achieve SOTA with 20.0% fewer inference calls, 15.1% fewer input tokens, and 13.5% fewer output tokens than alternative inference-time architectures.Source: §1 · p. 3
  • ARCHON is presented as extensible to new inference-time techniques, models, and tasks through user-friendly interfaces.Source: §1 · p. 3

2. Related Work

Prior inference-time systems often examine limited configurations or manually select a subset of techniques, while optimal choices vary across benchmarks, models, and compute limits. ARCHON instead unifies techniques and automates architecture construction; single-prompt, single-model orchestration frameworks address a different scope.

2. Related Work

  • Prior studies often explore limited configurations, sometimes focusing on specific benchmarks, even though optimal configurations vary with benchmarks, available models, and inference limits.Source: §2 · p. 3
  • Prior systems manually select subsets of techniques, whereas ARCHON unifies available techniques and automates architecture construction for task-specific model and component selection.Source: §2 · p. 3
  • DSPy optimizes a single prompt for one language model using supervised data for tool use, but does not leverage multiple inference-time techniques in parallel or sequentially.Source: §2 · p. 3

3. Inference-Time Techniques for ARCHON

ARCHON organizes inference-time LLM components into layered architectures and searches configurations for target tasks. The section describes component roles and findings, combination patterns, and search methods, including conditions where Bayesian optimization is less effective.

3.1. LLM Components of ARCHON

  • Components generate, rank, fuse, critique, verify, or test candidates; Verifier passes only verified responses onward, while unit-test stages filter candidates that pass all tests.Unit Test Generator produces 5–10 test statements, followed by an Evaluator that ranks responses using test verdicts.Source: §3.1 · pp. 3–4
  • Fusers improved performance by 8.9% on average; Critic added 11.5 percentage points; Ranker improved MT Bench and Arena-Hard-Auto output quality by 10.8% over random selection.Ranker performance was within 2.7% of oracle selection.Source: §3.1 · pp. 3–4
  • Unit-test modules boosted benchmarks requiring more verification by 7.4%; on CodeContests, increased sampling plus tests raised Pass@1 from 17.9 to 29.3% (56%).Verifier improved MixEval, MixEval Hard, and MATH by 8.4%, and was most effective when combined with other techniques.Source: §3.1 · p. 4

3.2. Combining the LLM Components

  • Repeated sampling and added ensemble models increased performance 9.3% and 18.5%, while layering and technique diversity also helped across explored tasks.For reasoning, adding Verifier and unit-test modules with Fuser filtered flawed responses and improved MixEval and CodeContests.Source: §3.2 · pp. 4–5
  • ARCHON processes lists of strings through parallel component layers, with components transforming or filtering candidates; the first string from the final layer is returned.No weights are learned between components, allowing the architecture to be deployed off-the-shelf without tuning; the component orderings were selected through ablations across seven benchmarks and two model classes.Source: §3.2 · p. 5

3.3. Architecture Search Algorithms

  • Bayesian optimization found the best architectures in 96.0% of searches using 88.5% fewer evaluations than greedy and 90.4% fewer than random, but was less effective below 20 calls.Search takes target benchmarks, an inference-call budget, available LLMs, and techniques as inputs; it compares random, greedy, and Bayesian optimization approaches.Source: §3.3 · pp. 6–7

4. Experiments

The experiments compare automatically searched ARCHON architectures with leading models and inference-time systems across instruction-following, reasoning, and coding benchmarks. Results vary by task and model size, while ARCHON’s successive API calls can increase latency and cost.

4. Experiments

  • The experiments assess ARCHON against state-of-the-art LLMs and inference-time architectures for accuracy and compute efficiency, and compare performance across tasks and model sizes.They also examine latency and cost considerations.Source: §4 · p. 7

4.1. Benchmarks and Models

  • The experiments cover instruction-following, reasoning, and coding benchmarks, with architectures searched on a random 20% subset and evaluated on the remaining 80%.The study compares task-specific and general-purpose architectures across open-source, closed-source, and all-source model settings.Source: §4.1 · p. 7

4.2. ARCHON vs. Closed-Source LLMs and Other Inference-Time Architectures

  • 15.1% average improvement over closed-source LMs and 8.4% over open-source inference-time frameworks: closed-source ARCHON leads across the reported benchmarks.The comparison covers MT Bench, Arena-Hard-Auto, MixEval, and MixEval-Hard; Table 1 reports results across open-source, closed-source, and all-source settings.Source: §4.2 · p. 8, Table 1 · p. 8
  • The best general-purpose all-source architecture uses ten initial generators followed by four critique-and-fusion layers that progressively narrow the candidates.Targeted architectures differ by task: instruction-following and reasoning use multiple critique-and-fusion layers, math narrows generations quickly, and coding iterates over one response.Source: §4.2 · p. 8

4.3. ARCHON by Task

  • ARCHON gains vary by task: individualized architectures raise MixEval and MixEval-Hard accuracy by 3.7 and 8.9 percentage points on average, while coding gains are limited without unit tests and more samples.Adding unit-test generation and evaluation with more samples raises GPT-4o CodeContests Pass@1 from 40 to 58 of 140 questions, a 44.3% boost.Source: §4.3 · p. 9

4.4. Discussion

  • ARCHON is most effective with 70B+ models; 7B models improve on the best individual 7B model by 7.5% on average, working better for ranking than for critique or fusion.Successive API calls can take five times more time and money than a single LLM API call.Source: §4.4 · p. 9

A.1. Table of Contents

The appendix contents map the paper’s component analysis, architecture search, comparisons, and benchmark results.

Appendix contents

  • The contents list separate sections for component utilities and interactions, Bayesian architecture optimization, and comparisons with alternative search approaches.They also identify a section comparing architecture algorithms across inference budgets.Source: §A.1 · p. 15
  • The appendix includes a comprehensive benchmark-results section spanning instruction-following, reasoning, and coding tasks.Source: §A.1 · p. 15
  • Additional sections cover tested LLMs, optimized architectures, and performance across inference budgets, model sizes, and costs.Source: §A.1 · p. 15

A.2. ARCHON LLM Components

The appendix points readers to an overview of ARCHON’s inference-time techniques and rules for combining components. It also describes a single-model sampling comparison in which ranking candidates before fusing the top five performs best on two benchmarks.

A.2. ARCHON LLM Components

  • Table 3 is an overview of inference-time techniques, including their definitions, inputs, outputs, costs, and application domains.Source: Table 3 · p. 15
  • On MT Bench and Arena-Hard-Auto, ranking candidates, selecting the top five, and fusing them scores highest among the five tested response-selection approaches.The tested approaches also include oracle selection, random selection, ranking, and fusion; Qwen2 72B Instruct is used for ranking and fusion.Source: Figure 6 · p. 16
  • Table 4 lists the allowed combinations of LLM components used to construct ARCHON.Source: Table 4 · p. 16

A.3. Utilities and Interactions of LLM Components

The subsection evaluates component utility and interactions across instruction-following, reasoning, and coding tasks using 70B+ open-source models.

A.3. Utilities and Interactions of LLM Components

  • The analysis spans instruction-following benchmarks MT Bench, AlpacaEval 2.0, and Arena-Hard-Auto.Source: §A.3 · p. 16
  • Reasoning coverage includes MixEval, MixEval-Hard, and MATH, while coding coverage uses CodeContests.Source: §A.3 · p. 16
  • The ARCHON models in this analysis use a range of 70B+ open-source models.Source: §A.3 · p. 16

A.3.1. GENERATOR

The Generator benefits from additional samples and from ensembling responses across more models. These gains vary by task and selection or fusion mode.

A.3.1. GENERATOR

  • With a limited inference-call budget, additional model samples provide the largest marginal benefit, particularly for coding tasks.Additional sampling also significantly boosts performance overall.Source: §A.3.1 · p. 16
  • Adding models to a task-ordered ensemble yields continual performance increases, while fusion is beneficial across ensemble sizes on MT Bench and Arena-Hard-Auto.Fusing the top five ranked responses scores highest among the five tested response-selection modes.Source: §A.3.1 · p. 16, Figure 7 · p. 17
  • For MT Bench and Arena-Hard-Auto, the ensemble approach scores higher than applying the same techniques to repeated samples from one best-performing model.Models in the open-source 70B+ ensemble are added in task-specific best-to-worst order.Source: Figure 7 · p. 17

A.3.2. FUSER

The Fuser improves accuracy, with larger gains when it receives more candidate responses. Its interactions with other modules and the benefits of additional fusion layers vary across tasks.

A.3.2. FUSER

  • 7.3 points on average: the Fuser with a Ranker for the top five responses improves accuracy over the single-sample best model; the gain over the oracle best candidate is 3.6 points.For a single-generation ensemble of ten 70B+ models, the Fuser alone improves accuracy by 5.2 points on average over the single-generation best model.Source: §A.3.2 · p. 17
  • Across explored benchmarks, added Fuser layers improved performance, but the benefit varied: MixEval gained 1–2 accuracy points, while MT Bench and Alpaca Eval 2.0 gained 2–5 win-rate points with three to four or more layers.The authors attribute the difference to task requirements and report that chat and instruction-following benefit more from iterative revisions and greater final-generation diversity.Source: §A.3.2 · p. 17
  • Adding a Critic improved Fuser performance by 3.1 percentage points on average, and adding a Ranker improved the Ensemble + Critic + Fuser configuration by 4.8 points on average.The Ranker was most effective on style-oriented tasks such as MT Bench and AlpacaEval 2.0.Source: §A.3.2 · p. 17
  • Adding a Verifier improved reasoning-task performance by 3.2 percentage points on average, versus 1.2 points for instruction-following tasks; unit-test generation and evaluation increased coding performance by 10.7 points.Unit-test generation and evaluation added 1.5 points on average for instruction-following and reasoning tasks.Source: §A.3.2 · p. 18

A.3.3. CRITIC

The Critic improves performance across the explored tasks and is especially useful with Fusers and Rankers, where it supplies candidate-response guidance.

A.3.3. CRITIC

  • With a ten-model 70B+ Generator ensemble and Fuser, adding a Critic improved performance by 3.1 percentage points on average across the explored benchmarks.The Critic proved effective for every task explored.Source: §A.3.3 · p. 18
  • With multiple Fuser layers, the Critic provided an average 3.2 percentage-point boost across benchmarks.The Critic helps guide fusion for a single layer and can also be placed between multiple fusion layers.Source: §A.3.3 · p. 18
  • Pairing the Critic with a Ranker improved performance by 5.9 percentage points on average by adding information for comparing candidate responses.Source: §A.3.3 · p. 18

A.3.4. RANKER

The Ranker is most effective on instruction-following tasks and pairs well with the Critic and Fuser, while ranking alone lags behind repeated sampling for complex tasks.

A.3.4. RANKER

  • For MT Bench and Arena-Hard-Auto, ranking improved output quality by 3.8% over random candidate selection and performed within 2.7% of oracle selection.The Ranker was most effective for instruction-following tasks, where pairwise comparisons focus on style and prompt adherence.Source: §A.3.4 · p. 18
  • Pairing the Ranker with a Critic improved performance by 5.9 percentage points on average, while pairing it with a Fuser improved the final response by 3.8 points on average.With a Verifier or Unit Test Generator, performance changed only marginally, by 1–2 points in either direction.Source: §A.3.4 · p. 18
  • Used alone with an ensemble of Generators, the Ranker lagged the ten-sample best-single-model configuration by 3.0 percentage points on average.The authors identify building better Rankers for complex reasoning tasks such as math and coding as a challenge raised also by Brown et al. (2024).Source: §A.3.4 · p. 18

A.3.5. VERIFIER

This section examines the Verifier and unit-test components, Bayesian optimization for searching Archon’s configuration space under different inference-call budgets, and the scope of evaluations and cost reporting across inference budgets, 7B models, API prices, and per-query benchmark costs.

A.3.5. VERIFIER

  • The Verifier’s benefit depended on composition: it lagged the ensemble-and-fuser setup by 1.5 percentage points alone, but augmented Critic and Fuser by 3.7 points on average, especially on reasoning tasks.It was less helpful on instruction-following tasks than on reasoning tasks.Source: §A.3.5 · p. 18, Table 5 · p. 19

A.3.6. UNIT TEST GENERATOR AND EVALUATOR

  • 56% boost in CodeContests Pass@1, from 17.9 to 29.3, followed increased sampling with unit-test generation and evaluation.Unit tests alone with a 70B+ generator ensemble lagged the ensemble-and-fuser setup by 3.1 percentage points on reasoning tasks.Source: §A.3.6 · p. 21
  • Combining unit tests with the Fuser improved performance by 2.1 percentage points across benchmarks, while the ensemble, unit-test, and Fuser combination boosted reasoning performance by 2.5 points on average.For coding, unit tests worked best with the strongest Generator at large sample counts and a final Fuser.Source: §A.3.6 · p. 21

A.4.1. ARCHON SEARCH SPACE AND OBJECTIVE

  • ARCHON’s configuration space includes generator samples (1–5, extended to 1–1000 for CodeContests), fusion layers (1–4), models per fusion layer (2–10 in increments of 2), and optional critic/ranker and verification layers.The critic/ranker toggle applies before each fuser, and verification is toggled before final fusion.Source: §A.4.1 · pp. 24–25

A.4.2. OPTIMIZATION PROCESS

  • Bayesian optimization required 88.5% fewer evaluations than greedy search and 90.4% fewer than random search, finding the best architecture in 96.0% of iterations.It uses expected improvement with a Gaussian-process surrogate and stops at the iteration limit or budget exhaustion.Source: §A.4.2 · p. 25, Figure 10 · p. 26

A.5. Bayes Optimization vs. Alternative Approaches

  • For ARCHON searches below 20 inference calls, Bayesian optimization performed more like greedy or random search; it became more effective for larger, open-ended searches above 20 calls.The study found 230–240 initial testing points effective before allocating further evaluations to configuration search.Source: §A.5 · pp. 26–27

A.7. ARCHON Benchmarks and Results

  • ARCHON architectures consistently outperformed single-call state-of-the-art LLM baselines on complete evaluation benchmarks; general-purpose configurations lagged task-specific ones by 3.2 percentage points on average.The comparison covered custom per-benchmark and single general-purpose architectures.Source: Table 33 · p. 30

A.8. ARCHON LLM Analysis; A.9. ARCHON Architectures

  • Scaling fusion layers alone brought limited benefits, while adding Critic and Ranker improved downstream performance; the all-source architecture used successive critique, ranking, and fusion layers.The all-source configuration was effective across explored benchmarks except CodeContests.Source: Figure 12 · p. 32, Figure 13 · p. 33

A.10. ARCHON by Inference Compute Budget, Model Size, and Cost

  • Table 37 covers Archon under different inference budgets and uses AlpacaEval 2.0 length-controlled win rate (LC WR).Source: Table 37 · p. 34
  • Table 38 covers Archon with 7B open-source models and uses AlpacaEval 2.0 length-controlled win rate (LC WR).The caption says the tested open-source 7B models are from Table 34.Source: Table 38 · p. 34
  • Table 39 lists model API costs as of November 2024.Source: Table 39 · p. 35
  • Table 40 covers Archon costs per query by benchmark.Source: Table 40 · p. 35
Loading 2409.15254v6…