PaperlayerView exact revision on arXiv

Source-linked AI summary

When Is Smaller Greener? Right-Sizing Language Models for Agentic AI

Sajid Siraj, Maria Barbati

arXiv:2610.19345v1cs.AIcs.MA

TL;DR

Agentic models are often compared per call, although task success depends on a sequence of steps and failed attempts consume resources. The paper evaluates models per successfully completed task and develops a decision framework; its pilot supports predicted task-length crossovers. The results distinguish short, long, and intermediate task regimes, while the framework's assumptions and energy accounting set limits on their scope.

  • Problem

    Per-call comparisons do not answer which model completes multi-step agent tasks reliably with the least energy, cost, and time.

    Source: §1 · p. 2

  • Method

    The paper builds a decision model and two-stage framework that compare resources per successfully completed task and separate efficiency screening from preference-based choice.

    Source: §8 · p. 31

  • Results

    The pilot supports the predicted structure, with most predicted crossover lengths agreeing with observed ones, while efficient choices vary by task length and type.

    Source: §1 · p. 3, §7 · p. 29

  • Takeaways & Limitations

    Short tasks and freely retryable tasks often favor smaller models, whereas long tasks and especially irreversible tasks can favor larger, more reliable models across resources.

    Source: §7 · p. 29

    The basic model assumes independent step success and alike attempts, takes task length as known, and its energy measures omit hardware manufacture, training, and water use.

    Source: §7.3 · p. 30, §7.4 · p. 30

Abstract

from arXiv · show

AI agents use large language models (LLMs) to plan and act over multiple steps, yet models are usually compared per call rather than per completed task. This paper asks: which model should an agent use to complete tasks reliably while minimising energy, cost and time? We show that per-call comparisons can be misleading for agentic AI. Because a task succeeds only if all required steps succeed, small differences in per-step reliability can produce large differences in overall task success. Failed attempts also consume resources. We therefore evaluate models using energy, cost and time per successfully completed task, treating the completed task as the functional unit. On this basis, we develop a decision model for choosing LLMs in agentic workflows. The model applies to tasks with binary or graded outcomes and to settings with or without retries. We show that each resource has a crossover task length at which a smaller, less reliable model ceases to be the greener, cheaper or faster option. These crossovers divide tasks into three regimes: for short tasks smaller models are preferred, for long tasks larger models are preferred, and only between these extremes do preferences among objectives matter. We embed these results in a two-stage framework that separates efficiency from preference. A Pareto screen first removes models that are never competitive; a weighted decision is required only when a genuine trade-off remains. A pilot study of 15 open-weight models from five developers, ranging from 0.6B to 27B parameters, on two multi-step agent tasks with measured GPU energy supports the predicted structure. The model with the lowest energy per completed task changes with task length and task type, and observed crossover points generally match those predicted from per-step measurements. These findings provide a practical framework for right-sizing language models in agentic AI.

1. Introduction

The paper asks which model lets an agent complete tasks reliably with the least energy, cost, and time. It argues for comparing resources per completed task and proposes a decision framework whose pilot study supports predicted crossover structure.

1. Introduction

  • The selection question is which language model an agent should use to complete tasks reliably with the least energy, cost, and time.The paper notes that deployed agents may run millions of tasks, each involving dozens of calls, and that model choice has reliability and resource consequences.Source: §1 · p. 2
  • Per-call comparisons can mislead because small per-step reliability differences compound across an agent task, and failed attempts still consume resources.A model that uses less energy per call may therefore use more energy per completed task, depending on task length.Source: §1 · p. 2
  • The paper uses the successfully completed task as the comparison unit and builds a decision model covering binary or graded outcomes and reversible or irreversible actions.This adopts the functional-unit framing of life-cycle assessment.Source: §1 · p. 3
  • Each resource has a crossover task length, dividing tasks into three regimes, two of which require no preference information.A preference-free Pareto screen removes models that are worse on all counts; weighted choice is reserved for remaining trade-offs.Source: §1 · p. 3
  • The pilot tests 15 open-weight models from five developers on two agent-task types with measured GPU energy, and most predicted crossover lengths agree with observed ones.The pilot supports the predicted structure.Source: §1 · p. 3
  • The framework addresses AI, operational-research, and sustainability questions by predicting when smaller models suffice, formalizing multi-objective choice, and changing the environmental comparison's functional unit.The paper presents these as contributions to three research communities.Source: §1 · p. 3

2. Literature review

The review finds that research on agent evaluation, model routing, resource accounting, and decision analysis offers relevant components, but has not linked per-task resource measures to model choice across task lengths.

2.1. How AI agents are evaluated

  • Agent evaluation increasingly tests multi-step planning and tool use, but surveys find cost-efficiency, safety, and robustness are rarely measured under realistic conditions.Existing benchmarks measure success across repeated trials or tool-calling reliability, while outcome-only evaluation can hide skipped steps and tool misuse.Source: §2.1 · p. 4
  • Studies show that task success changes with task length and per-step accuracy, but do not examine model choice when energy, cost, and time also matter.Prior work describes declining task success and how earlier errors can affect later steps.Source: §2.1 · p. 4

2.2. How models are chosen today

  • Existing routing methods address quality-cost trade-offs, but generally do not tell decision makers when a smaller model stops being preferable and treat energy or carbon only through cost, if at all.Routing approaches include cascades, learned model selection, and workflow-level allocation.Source: §2.2 · pp. 4–5
  • Multi-criteria model-selection studies use weighted scores for single requests, but neither considers energy or task length.The review contrasts these approaches with the paper's task-length selection question.Source: §2.2 · p. 5

2.3. How the resources of AI inference are counted

  • Most inference-energy studies count per token or request; task-level measures have emerged, but none links per-task resource use to choosing between models.Prior proposals include energy per successful goal and comparing agent resources with its task contribution.Source: §2.3 · p. 5

2.4. Decision-analysis tools

  • The framework draws on multi-criteria decision tools for identifying efficient alternatives and ranking them, alongside reliability theory for tasks requiring every step to succeed.The review also identifies satisficing, ε-constraint, and reference-point methods as relevant ideas.Source: §2.4 · p. 6

2.5. Gaps

  • The review synthesizes four gaps and the paper's responses, framing the unresolved issue as joining task-level resource accounting to model selection.The table is described as summarizing four gaps and how the paper addresses them.Source: §2.5 · p. 6, Table 1 · p. 6

3. Motivation: why per-call comparisons can mislead

The section illustrates why per-call comparison can mislead: failures accumulate across steps and consume resources. It motivates comparing models per successfully completed task and identifies the selection questions the framework must answer.

3. Motivation: why per-call comparisons can mislead

  • A 95%-reliable model succeeds on only about 0.36 of 20-step tasks, versus about 0.82 for a 99%-reliable model, when every step must succeed.If attempts repeat until success, the models require about 2.8 versus 1.2 attempts on average; at 60 steps, the nominally lower-energy-per-call model uses less energy per completed task no longer.Source: §3 · p. 7
  • The functional unit for agentic AI should be one successfully completed task, analogous to relating life-cycle impacts to a quantity of service.The paper positions this unit as a basis for choosing between models, alongside recent goal-level energy accounting and accuracy-cost proposals.Source: §3 · p. 7
  • The framework is designed to identify models that can be ruled out for every task, efficient models at a given length, and when preferences determine the choice.It also asks how the answer changes for irreversible actions or graded outcomes.Source: §3 · p. 7

4. A framework for the selection question

The framework maps model candidates and a task profile to a recommendation through successive feasibility, efficiency, task-level, and preference screens. It separates model and task properties and gives hard constraints priority.

4.1. Overview

  • The framework maps candidates and a task profile to a recommendation, with outputs including the recommended model, efficient set, and any weights used.The framework's stated scope includes a general decision model, special cases, relaxed assumptions, and an illustrative example.Source: §4.1 · p. 8
  • A feasibility screen removes candidates that violate hard constraints, followed by a per-step screen that removes candidates dominated on every per-step measure.The per-step screen's removals can be made once for all tasks in a task family.Source: §4.1 · p. 8
  • The task-level screen identifies efficient candidates per successfully completed task that meet the required chance of completion; preference-based choice runs only if multiple efficient candidates remain.The output records weights when the preference stage is needed, supporting audit of the recommendation.Source: §4.1 · p. 8, Figure 1 · p. 9

4.2. What we measure

  • Model quantities are separated from task-profile properties, and per-step reliability is measured for a task family rather than assumed universal across tasks.Context capacity, safety, and data governance are constraints rather than objectives.Source: §4.2 · p. 8, Table 2 · p. 9
  • Energy values distinguish measurements from estimates, with estimates carrying ranges; all values also carry dates and sources for audit.Closed-model energy may not be directly measurable from provider disclosures.Source: §4.2 · p. 10

4.3. Stage 0: feasibility screen

  • Hard constraints include required context, data-protection deployment rules, minimum safety, provider restrictions, and possible resource ceilings.The feasible set contains candidates that meet every constraint, because efficiency cannot compensate for a model that may not legally process the data.Source: §4.3 · p. 10

4.4. A general model of resources per successful task

The model measures resource use per successfully completed task, accounting for task length, step reliability, retries, and the probability of completion. It then screens feasible models against a completion-probability requirement before comparing resource objectives.

4.4. A general model of resources per successful task

  • A task has L model-call steps, and its outcome may be binary or graded against a decision-maker-set quality floor.In the basic case, steps succeed independently and any failed step spoils the attempt; this assumption is relaxed in Section 4.7.Source: §4.4 · p. 10
  • With at most K attempts, each attempt runs all L steps before its outcome is known, charging failed attempts in full.K = 1 represents irreversible tasks, while K = ∞ represents freely retryable tasks.Source: §4.4 · p. 11
  • Expected resource use per successful task is defined for each resource, and Observation 1 states that it does not depend on the retry limit.The retry limit changes the probability that a task is completed, but not resource efficiency conditional on success.Source: §4.4 · p. 11
  • The decision problem selects a feasible model subject to a required completion probability, with expected successful-output quality optionally treated as an objective.The chance constraint requires completion probability of at least β; for β = 0.9 and K = 3, each attempt must succeed with probability at least 0.54.Source: §4.4 · p. 11

4.5. Three properties of the general model

The model yields a task-length-independent Pareto screen and resource-specific crossover lengths. Together, these results identify when one model dominates, when the other dominates, and when preferences among resources are needed.

4.5. Three properties of the general model

  • A model with at least as high reliability and no greater per-step resource use is dominated for every task length and can be removed once for the task family.The screen depends only on the ordering of per-step measures, not units, scaling, or weights, and serves every task in the family.Source: §4.5 · p. 12
  • Each resource has its own crossover length: the less reliable model’s per-task resource advantage shrinks as task length grows.The log resource ratio is linear in task length with a common slope across energy, cost, and time; starting advantages differ by resource.Source: §4.5 · pp. 12–13
  • Below the earliest crossover the less reliable model dominates, above the latest the more reliable model dominates, and between them neither dominates.In the middle band, the choice depends on the decision maker’s resource preferences.Source: §4.5 · p. 13
  • The preference-dependent middle band disappears when the models’ per-step resource ratios are equal across energy, cost, and time.With m candidate models, the efficient set can change at no more than 3m(m − 1)/2 crossover lengths and stays constant between them.Source: §4.5 · p. 13

4.6. Special cases and variants

The special cases distinguish outcome type and retry availability; variants with graded quality can add quality as an objective. The framework also relaxes assumptions about measured values, reliability over task length, early failure detection, and carbon accounting.

4.6. Special cases and variants

  • With binary outcomes and free retries, the basic efficiency observations apply unchanged.For irreversible binary tasks, the objectives are unchanged, but the feasibility constraint can rule out models beyond a task-length limit.Source: §4.6 · p. 14
  • For irreversible actions with K = 1 and β = 0.9, reliability ρ = 0.95 is feasible only up to 2 steps, while ρ = 0.99 is feasible only up to 10 steps.Beyond those lengths neither model is feasible, indicating the task may need redesign, such as adding a human check before irreversible steps.Source: §4.6 · p. 14
  • With graded outcomes and a quality floor, success means reaching that floor, and quality above it has no extra value.This is a satisficing rule: select the least resource-intensive model that is good enough.Source: §4.6 · p. 14
  • When quality above the floor is also valued, expected quality becomes a fourth objective, traded against resources above a reservation level.The difference between recommendations with and without quality maximisation measures the extra energy, cost, and time paid for quality above the floor.Source: §4.6 · p. 14
  • Adding quality as an objective usually keeps more models efficient and widens the bands where preferences matter.Observation 3 still applies to the three resource objectives.Source: §4.6 · p. 14
  • If raising the quality floor reduces the smaller model’s success probability more, crossover lengths move shorter, making model choice depend on both task length and quality floor.The resulting map has regions where the smaller model dominates, both models are efficient, or the larger model dominates.Source: §4.6 · p. 15

4.7. Relaxing the assumptions

  • When per-step reliability varies with task length, Observation 1 remains unchanged; the three-regime result holds if the smaller-to-larger success-probability ratio falls as length grows.This condition can be checked directly from benchmark data; crossover lines need not remain straight.Source: §4.7 · p. 15
  • Uncertain reliability estimates can shift crossover lengths substantially when models have similar reliability, so the paper reports confidence intervals from benchmark resampling.For early-detected failures, crossover lengths are found numerically and are longer than in the basic case because failed attempts cost less; carbon crossovers equal energy crossovers only when models share location and timing.Source: §4.7 · pp. 15–16

4.8. Stage 2: choosing within a trade-off

Stage 2 ranks models only when the task-level screen leaves a genuine efficient-set trade-off. It uses rescaled resource scores and decision-maker weights, while retaining the completion-probability constraint as a prior safeguard.

4.8. Stage 2: choosing within a trade-off

  • If the task-level screen leaves one model, it is recommended without preferences; if several remain, choosing among them requires knowing the decision maker’s values.Stage 2 makes that choice within the efficient set for the task length.Source: §4.8 · p. 16
  • Within the efficient set, each objective is rescaled from 0 to 1, and the default SAW score combines energy, cost, and time scores using weights that sum to one.The best model on a resource scores 1 and the worst scores 0; weights can be elicited deliberatively or mapped from a task profile.Source: §4.8 · p. 16
  • TOPSIS is computed alongside SAW, and disagreement on the top choice is reported as a signal that the choice is close.Source: §4.8 · p. 16
  • The chance constraint is applied before preference weighting, so assigning all weight to cost cannot promote an unreliable model for an irreversible task.Stage 2 is needed only in preference-dependent bands, where weights can affect the choice.Source: §4.8 · p. 16

4.9. An illustrative example

The illustrative example shows that efficient model choice changes with task length, retry conditions and resource preferences. Right-sizing means choosing among models that meet the completion requirement, not simply choosing the smallest model.

4.9. An illustrative example

  • Table 5’s illustrative candidates are characterised by per-step reliability, energy, cost, time and context; its values are not measurements of named commercial models.Source: Table 5 · p. 17, §4.9 · p. 17
  • With free retries, F starts at half C’s per-step energy, but C’s greater reliability moves their energy crossover to about 17 steps.The efficient set changes with task length at crossover points.Source: §4.9 · p. 17, Table 6 · p. 17
  • For a 3-step freely retried task, F is the only efficient model and uses about 0.11 Wh per successful task versus 1.22 Wh for A, a saving of about 90%.Source: §4.9 · p. 18
  • For a 40-step freely retried task, C uses about 8.1 Wh, 34 cents and 203 seconds per successful task, while B uses about 13.2 Wh, 72 cents and 126 seconds.C suits deployments prioritising carbon or cost, while B suits those prioritising speed; this is the example’s only task requiring Stage 2.Source: §4.9 · p. 18
  • At 40 steps with an irreversible action and required completion probability β = 0.95, only A qualifies, using about 19.6 Wh per successful task—more than twice C.The example describes this as the price of not being able to undo an action.Source: §4.9 · p. 18
  • C’s advantage over B crosses at about 17 steps for time, 64 for energy and 77 for cost, leaving a trade-off band that includes 40 steps.The figure plots C’s resources per successful task relative to B against task length; below zero, C uses less.Source: §4.9 · p. 18, Figure 2 · p. 19

4.10. Right-sizing

  • Right-sizing selects a model meeting the required completion chance that is efficient in energy, cost and time per successful task, then uses stated preferences within a trade-off band.It can therefore select a larger model, as in the irreversible-task example.Source: §4.10 · p. 18, §4.9 · p. 18
  • Whole-service savings depend on task mix and grid carbon intensity, so the paper treats total-savings estimates as bounded scenarios rather than measurements.The framework measures savings as the energy, and hence carbon, difference per successful task against a reference policy.Source: §4.10 · pp. 18–19

5. How can we test the framework?

The section sets out hypotheses and a measurement design for testing the framework’s predictions and decision value. It uses repeated measurements, held-out tasks, varied benchmarks and robustness checks.

5.1. Claims to be tested

  • The tests address whether task success has the structure needed for crossovers, whether predicted crossovers match observed ones, whether framework choices reduce regret, and whether smaller and larger selections yield savings in different tasks.Source: §5.1 · pp. 19–20
  • Regret measures extra energy, cost and time per successful task against the best model in hindsight; comparison rules include fixed-size choices, leaderboard selection, cascades and weighted selection.Source: §5.1 · p. 20
  • H1 underpins the model, while H3 determines its practical value: if H1 fails the stated crossover results do not hold, and if H3 fails per-call rules suffice.Source: §5.1 · p. 20

5.2. What is measured and how

  • Every value is measured on a common harness, and repeated model-task runs provide confidence intervals; held-out tasks test predictions separately from the data used to estimate per-step values.Crossover and regret intervals are obtained by resampling runs.Source: §5.2 · p. 20, Table 7 · p. 20

5.3. Benchmarks and models

  • Benchmarks must vary in task length and permit automatic success checks; τ-bench’s airline domain also includes irreversible actions for V2, while Berkeley function-calling tasks add another family.Graded variants use a smaller set of scored-outcome tasks for illustration.Source: §5.3 · p. 21
  • The main study uses open-weight models of different sizes from one provider under one harness, enabling third-party energy measurement and consistent conditions; closed models later provide a check using public measurements.Source: §5.3 · p. 21

5.4. Robustness

  • Robustness checks vary Stage 2 weights, measured values within confidence intervals and the measurement date to test recommendation stability and rankings of later-released models.Source: §5.4 · p. 21

6. Experiments and results

The pilot tests whether task success declines with length and whether per-step measurements predict resource crossovers, using two multi-step tasks and GPU energy measurements. Across both task types, the most energy-efficient model changes with task length, and crossover predictions generally match observed curves, with important limits on the pilot’s scope and predictive test.

6.1. Pilot study design

  • Runs continue for 150 steps, letting each run supply task outcomes and per-step energy and time across lengths; per-step reliability is estimated from first-failure data.Each model sees the same 15 seeded tasks of each kind at temperature zero; success at length L requires all first L steps to be correct.Source: §6.1 · p. 22
  • The pilot measures 15 open-weight models from five developers on one GPU; energy is summed from GPU-board power readings, while time is server-reported per call.The energy measure excludes the processor, memory, and cooling, making it a lower bound on energy used.Source: §6.1 · p. 22

6.2. Results

  • The per-step screen removed four models for the order task and five for the tools task, including models dominated by smaller or more reliable alternatives.The tools task’s most reliable model errs about once in 80 steps, compared with once in 2,000 steps for the order task.Source: §6.2 · pp. 23–24
  • Success fell with task length for every model and both tasks, while the single-crossing condition held well where model choice mattered.Of 188 comparisons, 129 crossed once and 48 more than once; repeated crossings mainly occurred near ties or in early steps, and 33 still crossed more than once when both curves had at least three successful runs.Source: §6.2 · p. 24
  • 72 of 88 observed energy crossovers fell within the closed-form prediction’s 95% interval; median differences were 9 steps for the order task and 4 for the tools task.Predictions and observations use the same runs, so this tests fit to the data rather than prediction on new data.Source: §6.2 · p. 27, Table 9 · p. 28
  • The least-energy model changes with task length and task type: Llama 3.2 1B is greenest for 1–2 order steps, while Qwen3 8B is greenest beyond 100 steps; the tools-task transition occurs sooner.Only six models completed any 50-step tools task; with free retries, the largest model is never energy-efficient in the order task within 150 steps but becomes greenest in the tools task at about 50–60 steps.Source: Figure 4 · p. 26, §6.2 · p. 27
  • For a 0.9 first-attempt completion requirement, the largest model is the only feasible order-task choice from about 112 to 230 steps, while no model meets the requirement for tools tasks beyond about 9 steps.For such tools tasks, the framework points to changing the task, such as adding human confirmation for irreversible changes.Source: §6.2 · p. 29
  • The pilot uses two synthetic task families, open-weight models up to 27 billion parameters, one GPU, and 15 runs per model and task; energy covers only the GPU board.Its crossover predictions are evaluated on the same runs that produced them, and it does not test H3.Source: §6.2 · p. 29

7. Discussion

The discussion argues that model choice depends predictably on task length and type: smaller models often suit short tasks, larger models can suit long or irreversible tasks, and intermediate cases depend on preferences. It also addresses practical design implications, model and measurement limits, changing market conditions, and how the framework relates to routing.

7. Discussion

  • For short tasks and tasks that can be retried freely, smaller models are often efficient, while larger, more reliable models can use less per completed task on long or irreversible tasks.Between these regimes, the preferred model depends on what the decision maker values.Source: §7 · p. 29

7.1. Is smaller greener?

  • At 20 steps in the order task, a 3B model used about a tenth of the energy of a 27B model per completed task; at 100 steps, a 1.7B model used almost four times as much as an 8B model.In the harder tools task, the 27B model was already the greenest choice from about 50 to 60 steps; per-task measurement can reverse the ranking.Source: §7.1 · pp. 29–30

7.2. What this means for practice

  • For irreversible actions, a chance constraint can rule out every model, so changing the task—for example, adding human confirmation—may be preferable to switching models.The tools task illustrates this boundary for tasks longer than about nine steps; agents may also switch to a larger model when the remaining task passes a crossover length.Source: §7.2 · p. 30

7.3. Limits of the model

  • The basic model assumes independent step outcomes, treats any failed step as spoiling the attempt, assumes attempts are alike, and takes task length as known.Its main results also hold when success is measured directly rather than assumed, provided the single-crossing condition holds; uncertain task length adds uncertainty to crossover estimates.Source: §7.3 · p. 30

7.4. Measuring energy and carbon

  • The energy measures cover model operation but exclude hardware manufacture, model training, and water use, and lower per-task energy does not establish lower total energy because of rebound.For closed models, energy must be estimated and estimates can be wide; the study keeps measured and estimated values separate and carries estimate ranges through to results.Source: §7.4 · p. 30

7.5. A changing market

  • Because models, prices, and hardware change, the efficient set is a snapshot that should be recomputed as market values change.The proposed decision-support service would maintain dated measurements, compute efficient sets, and return recommendations with their weights and sources.Source: §7.5 · p. 31

7.6. Relation to routing

  • The framework complements routing by identifying which models are worth considering, when preferences are needed, and how energy enters the choice.Its efficient sets can constrain a router's choices, while its per-task measures can evaluate routers in energy, cost, and time.Source: §7.6 · p. 31

8. Conclusion and future directions

The conclusion presents a decision framework for selecting models per successfully completed task, separating model efficiency from preference trade-offs. It identifies future work in benchmark testing, model switching, team agents, practitioner preferences, broader resource measurement, and maintaining an open service.

8. Conclusion and future directions

  • The framework screens out models that are worse on every per-step measure, then identifies one crossover length per resource between remaining models, leaving preferences relevant only in one of three task regimes.It covers binary and graded outcomes and tasks whose actions can or cannot be undone.Source: §8 · p. 31
  • Future work includes testing the structure on held-out agentic tasks, extending the model to switching and team agents, and studying practitioners' trade-offs.The paper also proposes broader resource measures and release of an open service with measurement tools and data.Source: §8 · p. 32

Related papers

Loading 2610.19345v1…