PaperlayerView exact revision on arXiv

Source-linked AI summary

Circuits of LLM Classifiers

Hadi Khalaf, Sajani Vithana, Kunal Talreja, Haewon Jeong, Viveck R. Cadambe, Flavio Calmon

arXiv:2610.30076v1cs.LG

TL;DR

LLM classifiers can give conflicting labels, and choosing which models to query creates an accuracy–cost tradeoff. The paper uses a dynamic program over a learned joint model to choose prompt-specific queries and stopping decisions; across a range of tasks, this approach Pareto-dominates routing and cascading baselines.

  • Problem

    How to combine LLM classifier predictions to improve reliability while deciding which classifiers to query under an accuracy–cost tradeoff remains an open question.

    Source: §1 · p. 2

  • Method

    The paper fits a probabilistic model of labels and classifier outputs, then solves a dynamic program to choose which LLM to query and when to stop for each prompt.

    Source: §1 · p. 3

  • Results

    In a range of classification tasks, the learned-model approach Pareto-dominates routing and cascading baselines in the cost–accuracy tradeoff.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Queries can reveal sample difficulty, allowing circuits to spend more queries on difficult prompts while resolving easier ones quickly.

    Source: §1 · p. 3

    The dynamic program grows exponentially with the number of models, though a small, well-chosen classifier pool may make this less restrictive.

    Source: §5.1 · p. 13

Abstract

from arXiv · show

Language models make it easy to build classifiers by simply prompting a model with the sample to classify. However, their predictions are neither deterministic nor uniformly reliable across samples. How can we combine predictions from different models to improve reliability? We view LLM classifiers as noisy measurement instruments of an underlying target label. Our goal is to maximize average classification accuracy at a given average query cost. To this end, we introduce circuits that organize LLM classifiers into an adaptive sequence of queries and decisions. We then characterize the optimal circuit, obtain the fundamental limits of achievable accuracy, and provide practical insights on realizing the optimal circuit. Across seven classification tasks spanning safety, health, political science, human preference, agent evaluation, decision-making and reasoning, our method improves the cost-accuracy frontier over routing and cascading baselines.

1 Introduction

LLM classifiers can give conflicting labels, so combining them trades query cost against accuracy. The paper models this as a sequential decision problem and uses a dynamic program to choose queries and stopping decisions; a learned version improves the cost–accuracy frontier over routing and cascading baselines.

1 Introduction

  • Repeated evaluations of the same prompt can return conflicting labels, motivating the view of an LLM classifier as a noisy measurement instrument.A classifier is configured by choices including backend model, instruction prompt, grading rubric, and decoding parameters.Source: §1 · p. 1
  • A classification circuit combines classifier outputs through decision rules that can pass results forward, combine them, or select the next query.Routing selects one model, cascades query models sequentially in a fixed order, and majority voting queries several in parallel.Source: §1 · p. 2
  • The optimal circuit is a dynamic-programming solution to a Bayesian sequential decision problem that balances average classification accuracy against average query cost.Its key challenge is valuing an additional query given the measurements already observed.Source: §1 · pp. 2–3
  • A query can reveal both a label and how difficult a prompt is to classify, so disagreement leads the circuit to spend more queries on difficult prompts and resolve easier ones quickly.For a fixed sample, asymptotic error decreases with cost at a rate determined by the most informative classifier per unit cost.Source: §1 · p. 3
  • For a fixed classifier pool, circuit error is bounded by the Bayes error before any query and the Bayes error given all classifier outputs.The latter is the accuracy ceiling for any way of combining that pool.Source: §1 · p. 3
  • A learned two-step approach fits the joint distribution of labels and classifier outputs, then solves a dynamic program to choose which LLM to query and when to stop.Across seven classification tasks, the resulting circuits improve the cost–accuracy frontier over routing and cascading baselines.Source: §1 · p. 3

2 Problem Formulation

The problem formulation treats each LLM configuration as a measurement instrument with a prompt-dependent cost and output distribution. A circuit adaptively chooses which instruments to query and when to stop, then predicts the binary target label from the collected measurements.

2 Problem Formulation

  • A measurement instrument is an LLM configuration characterized by a conditional distribution of outputs given a prompt and a positive query cost.For LLMs, cost is estimated from the input prompt, reasoning and output-token limits, and token prices.Source: §2 · p. 3
  • A classification circuit specifies which instruments to query, how later queries depend on prior measurements, when to stop, and how to predict the target label.The query policy can vary with the prompt; collected measurements feed the prediction rule.Source: §2 · p. 4
  • The framework permits repeated independent queries to the same instrument, although the formal history definition also describes disjoint queried subsets across rounds.The work focuses on binary instrument outputs and a binary target label.Source: §2 · p. 4
  • The formulation assumes a finite pool of instruments and conditionally independent measurements given the prompt.The paper distinguishes within-prompt conditional independence from error correlation measured across prompts.Source: §2 · pp. 4–5

3 Optimizing Classification Circuits

The section formulates circuit design as minimizing classification error plus a weighted query cost, then solves it with sequential query-or-stop decisions. For unseen prompts, a fitted probabilistic model supplies the estimates used by a dynamic program to generate prompt-dependent policies.

3 Optimizing Classification Circuits

  • An optimal circuit can query one instrument at a time, and a dynamic program gives the optimal policy for the expected cost–accuracy objective.The objective weights error by query cost through λ; the optimal policy is a sequence of decisions.Source: §3 · p. 5

3.1 Optimal Circuit Setup

  • At each state, the circuit either stops with the Bayes prediction or queries an unqueried instrument, paying its cost and continuing from the resulting state.Backward induction over the Bellman equations yields which models to query, their order, and when to stop.Source: §3.1 · p. 6

3.2 Generating circuits for unseen prompts: LMCIRCUIT

  • For unseen prompts, LMCIRCUIT fits label-posterior and instrument-output estimates from labeled prompts and measurements, then plugs them into the dynamic program.The same fitted model supports circuits at different cost–accuracy points, and can make a zero-cost initial prediction; it runs completely on CPU.Source: §3.2 · pp. 6–7, Figure 3 · p. 7

4 Analysis of the Optimal Circuit

The analysis characterizes the accuracy–cost limits of circuits and how their error scales with query cost for individual prompts and prompt populations. It also shows how information beyond an existing pool can guide instrument selection, while larger pools make the joint behavior harder to estimate.

4.1 Fundamental Limits of Circuits for a Fixed and Finite Pool

  • At high enough query-cost weight, making no queries is optimal; with zero cost weight, querying the full pool yields the minimum achievable error.The full-pool error is a lower bound for any circuit built from that pool, regardless of the cost weight.Source: §4.1 · pp. 7–8
  • A new instrument’s value depends on its information about the target beyond existing measurements, not its accuracy alone.The conditional-information bound motivates greedy pool construction; on four tasks, greedy selection reached the full-pool frontier with 2–3 instruments and dominated the compared ordering methods.Source: §4.1 · p. 8
  • Larger instrument pools offer more complementary measurements but make the joint distribution needed by the dynamic program harder to estimate.Pool construction therefore balances added information against the statistical cost of estimating joint behavior.Source: §4.1 · p. 8

4.2 Scaling Laws for Classification Circuits

  • The scaling analysis assumes repeatable queries, instruments at least as accurate as random guessing, and label-symmetric response distributions.Even an optimal circuit cannot compensate for a lack of informative measurements; improving instruments on the hardest prompts can change the scaling law.Source: §4.2 · pp. 9–10
  • For a fixed prompt, optimal error decays exponentially with query cost at a rate set by the pool’s maximum information rate.As cost weight approaches zero, nearly all expected cost goes to instruments attaining that rate; prompts with smaller rates need more cost to reach the same error.Source: §4.2 · p. 9
  • Across prompts, exponential error decay on individual prompts need not yield exponential decay in average cost when information rates approach zero.Under the stated tail assumption, α > 1 gives exponential decay, α = 1 gives stretched-exponential decay with −log Eλ ≍ √Cλ, and 0 < α < 1 gives polynomial decay.Source: §4.2 · pp. 9–10

5 Experiments

Across seven tasks, LMCIRCUIT improves the cost–accuracy frontier over routing and cascade baselines, with prompt-dependent query policies and instrument choices. Results also show calibrated decisions, accurate instrument estimates, and fast construction, while the dynamic program’s cost grows exponentially with pool size.

5.1 Results

  • Across seven tasks, LMCIRCUIT attains higher accuracy than every individual instrument and routing and cascade baselines at nearly all budgets.Instrument choices vary by task: mid-cost Gemma models are most informative per unit cost on LMArena, while the cheapest Gemma 4B leads on BeaverTails.Source: §5.1 · p. 10, Figure 4 · p. 11
  • Across seven tasks, LMCIRCUIT is trained in 1–3 minutes, while some routing and cascading baselines require minutes to hours.Its estimation step uses a simple neural network with pretrained prompt embeddings; it does not require token probabilities or training larger models for each instrument.Source: Figure 5 · p. 12, §5.1 · p. 12
  • At the same λ, harder prompts require more calls, and the method’s prior can predict labels without queries, allowing average cost below the cheapest instrument.LMCIRCUIT also produces better-calibrated decisions on unseen data.Source: §5.1 · p. 12
  • On noisy prompts, budget-1 calls improve embedding-prediction accuracy by 5%–44%; on prompts where every call is correct, accuracy matches majority voting within 1.3% while using 38%–86% fewer calls.Prompts are grouped by the fraction of recorded calls returning the correct label.Source: §5.1 · p. 12
  • Across seven tasks, predicted instrument-answer and error correlations are within 4%–17% of measured values, and predicted instrument accuracy is within 1.2%–3.6%.The estimates are evaluated on unseen prompts without test labels.Source: §5.1 · p. 12

5 Experiments

  • The dynamic program grows exponentially with the number of models, although the paper notes that a small, well-chosen classifier pool is often sufficient.Experiments estimate the circuit from training data, vary λ on validation data, and report test accuracy against average query cost across seven tasks.Source: §5 · p. 10, §5.1 · p. 13

5.1 Results

  • For moderately accurate queried models (∼75–80%), the dynamic program selects highly correlated models; for less accurate models (∼63–66%), it selects less correlated models.The paper distinguishes stochastic noise, reducible through repeated or correlated measurements, from systematic noise inherent to instrument training.Source: §5.1 · p. 13

6 Conclusion

The conclusion frames LMCIRCUIT as a decision-theoretic way to allocate costly LLM queries and control the error–cost tradeoff. The related-work discussion connects this approach to sequential information acquisition and distinguishes its learned prompt-dependent model and exact dynamic program from prior methods.

6 Conclusion

  • A lightweight probabilistic model and dynamic program generate prompt-specific circuits that choose what to query and when to stop, with control over the error–cost tradeoff.The conclusion identifies extension to larger pools of general measurement instruments with potential feedback as a challenge.Source: §6 · p. 13

A Related Work

  • Unlike prior sequential-testing work with known observation models, this setting has many possible prompts and requires learning each instrument’s prompt-specific response model.The formulation draws on sequential hypothesis testing.Source: A Related Work · p. 19
  • Unlike cascades that query in a fixed order and may discard earlier outputs, circuits select the next model from observed history and combine outputs into a posterior label belief.The comparison distinguishes cascades from circuits’ adaptive query choice and posterior-based prediction.Source: A Related Work · pp. 19–20
  • The paper’s circuits learn a probabilistic response model and compute an optimal query-and-stop policy by dynamic programming, unlike end-to-end reinforcement-learning policies such as TREACLE.The policy is optimal with respect to the learned model, and its error–cost tradeoff can be analyzed.Source: A Related Work · p. 20
  • Unlike noisy-classifier aggregation that assumes independent errors, the paper’s probabilistic model learns response distributions and prompt embeddings to account for correlations between model outputs.The related work notes that correlated errors remain a hurdle for LLM ensembles (Goel et al., 2025; Kim et al., 2025; Kohli, 2026).Source: A Related Work · p. 21

B Proofs

The supplied proof passages establish that batch queries can be replaced by sequential queries without changing measurements, predictions, or cost. They also state that additional measurements can be ignored, so they cannot reduce the available accuracy.

B Proofs

  • Any batch of queries can be replaced by the same queries in a fixed sequential order, without changing measurements, the final prediction, or total cost.The intermediate responses do not affect subsequent queries in this replacement.Source: B Proofs · p. 21
  • For a fixed prompt, the proof bounds the change in Bayes accuracy from a new measurement by a term involving the change in posterior belief; it then relates that change to conditional mutual information.The passages state that the bound uses the 1-Lipschitz property of the Bayes-accuracy function and Pinsker’s and Jensen’s inequalities.Source: B Proofs · p. 22
  • Additional measurements cannot reduce accuracy because a circuit can always ignore them.Source: B Proofs · p. 22

B.1 Proofs for Fixed-Prompt Scaling

For a fixed prompt, the proof analyzes sequential circuits under symmetric instrument measurements and independent randomness across calls. It bounds the error–cost tradeoff using the most informative instrument per unit cost and characterizes asymptotic query allocation.

B.1 Proofs for Fixed-Prompt Scaling

  • For error probability ε, every circuit requires expected cost at least log(1/ε)/I(z) − O(1), and a sequential likelihood-ratio test attains this bound up to an additive constant.The asymptotic setup allows each instrument to be queried arbitrarily often with independent call randomness.Source: §B.1 · p. 22
  • As λ decreases to zero, the information lower bound and likelihood-ratio construction yield matching leading rates for the optimal fixed-prompt error–cost tradeoff.The likelihood-ratio construction uses an instrument maximizing information per unit cost.Source: §B.1 · pp. 23–24
  • As λ decreases to zero, an optimal circuit’s expected cost allocated to instruments with strictly lower information rates remains O(1), while total expected cost diverges.Source: §B.1 · p. 25

B.2 Proof of the Population Scaling Law

This proof derives population error and cost behavior from fixed-prompt bounds under uniformity conditions. Its scaling results distinguish whether prompts have a finite expected inverse information rate and, under a polynomial information-rate tail, different regimes.

B.2 Proof of the Population Scaling Law

  • Under Assumption 2, uniform fixed-prompt error bounds support the population scaling argument for almost every prompt.The proof also assumes the initial error is bounded away from zero and a highest-information instrument has bounded query cost.Source: §B.2 · p. 26
  • When the expected inverse information rate is finite, the population error satisfies Eλ = Θ(λ), yielding the corresponding population error–cost scaling.The proof uses uniform cost bounds and dominated convergence to pass from fixed prompts to the population.Source: §B.2 · pp. 27–28
  • Under the stated polynomial lower-tail condition, E[1/I(Z)] is finite exactly when α > 1; the proof treats the remaining cases separately.The text says the three resulting regimes are established, but the supplied passages do not expose their full scaling expressions.Source: §B.2 · pp. 29–30

C.1 Data

The appendix describes the seven-dataset evaluation and how the probabilistic model is trained and used to generate circuits. It also lists baseline families and reports calibration results across budgets.

C.1 Data

  • The evaluation covers seven datasets, including agent-plan assessment, health judgments, decision tasks, preference comparisons, and reasoning questions.The appendix also describes BeaverTails safety classification and PolitiCAUSE causal-relation classification.Source: §C.1 · pp. 30–31

C.2 Training the Probabilistic Model

  • The fitted model uses latent prompt types and label and measurement predictions, then supplies probabilities to the dynamic program that generates circuits.Training combines label and measurement losses; validation selects the model and calibrates its prediction heads.Source: §C.2 · pp. 32–33

C.3 Full list of baselines

  • The listed baselines include predictive routers, model cascades, and adaptive-consistency sampling.The named methods include RouteLLM, HybridLLM, Carrot, Cascade Routing, Consistency Cascades, and Adaptive-Consistency.Source: §C.3 · p. 33

C.4.1 Calibration of the circuit’s beliefs.

  • Circuit expected calibration error is at most 2.1% on six tasks and 5.2% on BIG-Bench Hard, versus 4.8%–24.6% for majority voting.The figure also presents Brier score against expected cost and reliability diagrams across budgets.Source: §C.4.1 · p. 33, Figure 6 · p. 34

C.4.2 Joint distribution between instruments

The authors test whether instruments are independent within prompts and assess whether their fitted model recovers instrument dependence and accuracy on unseen prompts. They report close agreement between model-implied and measured quantities across tasks.

C.4.2 Joint distribution between instruments

  • Within-prompt instrument-answer correlations are −0.10 to 0.15, while across-prompt correlations reach 0.69; only one of 20 prompts rejects joint independence.The analysis compares within-prompt correlations and total correlation with a permutation null.Source: §C.4.2 · p. 34, Figure 7 · p. 35
  • The fitted model represents dependence beyond the embedding by conditioning measurements on an unobserved prompt type updated after each measurement.This joint model is used to estimate test-prompt correlations without true test labels.Source: §C.4.2 · pp. 34–35
  • On six tasks, implied answer and error correlations are within 4% of measured values on average; on BIG-Bench Hard, they are 83%–85% of measured values.Predicted instrument accuracy is within 1.2 points on average on five tasks, 2.1 points on AgentJudgeBench, and 3.6 points on BIG-Bench Hard.Source: §C.4.2 · p. 35

C.4.3 How does the circuit allocate its queries?

The analysis examines how prompt reliability and disagreement shape querying, and how the circuit’s first-query allocation relates to estimated information per cost. Example circuits show prompt-specific query paths and stopping decisions.

C.4.3 How does the circuit allocate its queries?

  • Instrument entry order correlates with estimated information per unit cost at 1.00, 0.96, and 0.79 on Arena, BeaverTails, and PolitiCAUSE, respectively.The corresponding correlations for inverse cost are 0.75, 0.89, and 0.93; for accuracy, 0.79, 0.21, and 0.71.Source: §C.4.3 · p. 36, Figure 11 · p. 38
  • On noisy prompts, additional queries improve accuracy by 5–44 percentage points over the embedding prediction; on all-right prompts, budget-1 circuits come within 1.3 points of majority voting on four tasks using 38%–86% fewer calls.On mostly-wrong and all-wrong prompts, further queries to the same pool tend to reinforce the error.Source: §C.4.3 · p. 36, Figure 9 · p. 37

C.5 Circuit Examples

  • Example circuits show the embedding’s initial belief, updated beliefs after instrument answers, and the recorded-answer path through query-or-stop decisions.At stopping, the circuit predicts the label whose belief is above 1/2.Source: §C.5 · p. 37, Figure 13 · p. 40, Figure 14 · p. 40

C.4.3 How does the circuit allocate its queries?

  • At normalized budget 0.2, the circuit makes 1.7–1.9× as many calls when the embedding prediction and instrument majority disagree across Arena, BeaverTails, and PolitiCAUSE.The circuit tends to continue when posterior belief remains near 1/2; agreement typically moves belief away from 1/2 and shortens the circuit.Source: §C.4.3 · p. 36, Figure 10 · p. 38
Loading 2610.30076v1…