PaperlayerView exact revision on arXiv

Source-linked AI summary

ATEST: Adaptive Resource-Aware Test-Time Scaling for LLM Inference

Mobina Kashaniyan, Ali Jannesari

arXiv:2610.25519v1cs.LGcs.AIcs.CLcs.DCcs.PF

TL;DR

Fixed test-time sampling budgets overlook how GPU contention changes the energy cost of each sample. ATEST makes sampling depend on both the expected value of more reasoning and its current GPU cost, with online and offline modes. Across four A100 datasets, ATEST-Online answered every query within the energy budget and achieved the highest or tied-best accuracy, while online control trades latency and throughput on short tasks for robustness under changing load.

  • Problem

    Adaptive TTS methods account for query difficulty or answer stability but treat sample cost as fixed, despite changing GPU load altering energy, latency, and throughput costs.

    Source: §I · p. 1, A. Test-Time Scaling and Adaptive… · p. 3

  • Method

    ATEST jointly considers the value of additional samples and their current GPU cost, using either online stepwise control or offline-calibrated per-query budgets.

    Source: A. Overview · p. 4

  • Results

    Across four A100 datasets, ATEST-Online answered every query within the energy budget and achieved the highest or tied-best accuracy; on high-load GSM8K it achieved 60.0% accuracy with no deferrals.

    Source: §I · p. 2

  • Takeaways & Limitations

    ATEST-Online is the safer choice when GPU load changes, while ATEST-Calibrated remains useful when load is stable and profiles can be reused.

    Source: §VIII · p. 10

    The evaluation used 500 queries per dataset, four datasets, two open models, and two GPU types because available compute constrained the evaluation scope.

    Source: §IX · p. 10

Abstract

from arXiv · show

Test-time scaling (TTS) improves large language model (LLM) reasoning by sampling multiple answers, but fixed sampling budgets ignore both query difficulty and the changing energy cost of inference under GPU contention. We measure up to a 4.1-fold increase in energy for the same four-sample query as GPU load rises. We present ATEST, a resource-aware TTS framework that jointly considers the expected value of additional samples and their current GPU cost. ATEST-Online, its default mode, requires no offline calibration: it incrementally generates samples, estimates the stability of the plurality answer, monitors live GPU conditions, and learns step energy cost online to decide when to stop. ATEST-Calibrated instead selects a per-query budget using offline accuracy and cost models, favoring deployments with stable loads. We evaluate both modes on four reasoning and question-answering datasets using Phi-3-mini on an NVIDIA A100 under controlled varying load, and test transfer to an NVIDIA L40S and Qwen2.5-3B. Across the four A100 datasets, ATEST-Online answers every query without exceeding the energy budget, while achieving the highest or statistically tied accuracy and reducing sampling as load increases. On GSM8K at high load, it achieves 60.0% accuracy with no deferrals, versus 44.0% accuracy and 42.4% deferrals for ATEST-Calibrated. Ablations show that online cost learning drives adaptation to load. Including offline profiling, ATEST-Online uses 42 times less GPU time and 48 times less energy than ATEST-Calibrated for the evaluated workload. Online control trades higher tail latency and lower throughput on short tasks for robustness under varying load; offline calibration remains competitive under stable conditions.

I. INTRODUCTION

TTS can improve answers by sampling multiple responses, but fixed sampling budgets ignore how GPU contention changes each sample’s energy cost. ATEST jointly weighs the value of more samples against their live cost, with an online mode for changing loads and a calibrated mode for stable deployments.

I. INTRODUCTION

  • A fixed four-sample query used up to 4.1× more energy under high GPU load, making sample cost variable rather than fixed.On GSM8K, the same query cost 121.7 J on an idle GPU and 455.6 J on a busy one.Source: §I · pp. 1–2
  • ATEST-Online incrementally samples, tracks vote stability and live GPU conditions, learns step energy cost online, and stops when another step is not worth its cost.It requires no offline profiling.Source: §I · p. 2, Figure 2 · p. 2
  • ATEST-Calibrated profiles each dataset and selects a per-query budget from offline accuracy and cost models, favoring stable-load deployments or short tasks needing low tail latency.Including profiling, ATEST-Online used 42× less GPU time and 48× less energy than ATEST-Calibrated on the same 2,000 queries.Source: §I · p. 2
  • Across four A100 datasets, ATEST-Online answered every query within the energy budget and achieved the highest or tied-best accuracy; on high-load GSM8K it reached 60.0% accuracy with no deferrals.ATEST-Calibrated reached 44.0% accuracy and deferred 42.4% of queries in that GSM8K setting.Source: §I · p. 2
  • The controller treats adaptive sampling as a runtime resource-management decision that balances the usefulness of another sample against its current GPU cost.More samples can improve answers, but also increase GPU work, energy, and waiting time.Source: §I · pp. 1–2

II. BACKGROUND AND RELATED WORK

Adaptive TTS methods adjust sampling using query or reasoning signals, while GPU-serving systems manage execution and resources. ATEST addresses the gap between these approaches by making sampling decisions using both reasoning value and live GPU cost.

A. Test-Time Scaling and Adaptive Sampling

  • Adaptive TTS methods vary the sampling budget using answer agreement, query difficulty, confidence, or reasoning progress, but do not account for the sample’s current GPU cost.The same samples can have different energy, latency, and throughput costs depending on how they run.Source: A. Test-Time Scaling and Adaptive… · p. 3

B. GPU-Aware and Energy-Efficient LLM Serving

  • GPU-serving systems optimize how a given amount of work runs, but do not decide how much reasoning work each query receives.ATEST complements these systems by making the per-query reasoning-work decision.Source: B. GPU-Aware and Energy-Efficient LLM… · p. 3

C. Research Gap

  • ATEST uses live GPU state alongside reasoning signals to decide whether another sample is worthwhile, and ATEST-Online learns cost without per-dataset profiling.The cited motivation reports that offline cost prediction error rises under load, even when live GPU state is added.Source: C. Research Gap · p. 3, Figure 3 · p. 3

III. MOTIVATION

GPU contention makes the energy cost of a fixed sampling budget rise with load, while offline cost models fail to track changing conditions reliably. The motivation therefore calls for a controller that allocates samples using both query uncertainty and current cost.

Motivation

  • With a fixed budget of B = 4, energy per answered query rose to 3.7× the idle value on GSM8K and MATH, and 4.1× on ARC-Challenge at high load.Shared jobs compete for compute, memory bandwidth, and power, and the resulting cost changes with load.Source: §III · p. 3
  • Offline cost-model prediction error grows as GPU load rises; adding live GPU state helps but does not remove the problem.The model’s cost prediction is made before the budget runs.Source: §III · p. 3
  • The controller should give more samples to uncertain queries and use the sample’s current cost rather than its profiled cost.These goals pair reasoning-side value with live resource conditions.Source: §III · p. 4
  • The design goals also include avoiding per-dataset offline calibration, staying within the energy budget, and avoiding deferred queries.ATEST-Online is designed to meet all four goals; ATEST-Calibrated meets the first, partly meets the second, and does not meet the profiling goal.Source: §III · p. 4

IV. PROBLEM FORMULATION

The formulation allocates per-query sampling budgets across a shared-GPU query stream while enforcing a total energy limit. It also treats deferrals as lost service, and distinguishes online stepwise control from one-shot offline budget selection.

IV. PROBLEM FORMULATION

  • For each query in a shared-GPU stream, the controller chooses a sampling budget from {1, 3, 4, 6, 8}, while GPU condition H_t can change the energy cost of that budget.Source: §IV · p. 4
  • The measured energy E_i for each query must sum to no more than the shared total budget E_max.Source: §IV · p. 4
  • A query is admitted only when its predicted cost fits its share of the remaining energy budget; otherwise it is deferred.The share is E_rem/N_rem, where N_rem is the number of queries left.Source: §IV · p. 4
  • A deferred query counts as incorrect, so deferral protects the energy budget but reduces accuracy and service coverage.Source: §IV · p. 4

A. Overview

  • ATEST-Online chooses the budget step by step using generated samples and a cost model learned during execution; ATEST-Calibrated selects the whole budget before sampling using offline models.Source: A. Overview · p. 4
  • ATEST-Online suits changing loads or frequently changing models and datasets, while ATEST-Calibrated can reuse a profile for stable deployments.The calibrated mode is also useful when low tail latency, high throughput on short tasks, or pre-run cost estimates are required.Source: A. Overview · p. 4

B. ATEST-Online

ATEST-Online grows the sample budget incrementally and weighs vote stability against live energy cost, updating its cost model as it runs. It stops when additional sampling is not worthwhile or the answer is already stable.

B. ATEST-Online

  • ATEST-Online starts at B = 1, advances through 1 → 3 → 4 → 6 → 8, and reuses earlier samples while estimating whether the current plurality would persist at Bmax = 8.It estimates stability with a small Monte Carlo simulation of the remaining samples, using current vote counts.Source: B. ATEST-Online · p. 4
  • After each executed step, ATEST-Online uses measured energy and a feature vector describing new-sample count and GPU state to update its online cost model with RLS.The model begins learning from the first query, and measurements after each step let it respond to load changes; a second RLS model predicts latency.Source: B. ATEST-Online · p. 4
  • ATEST-Online continues only when expected stability gain per predicted energy cost meets the threshold and the step fits the remaining budget.It also stops when the answer is stable, further samples are unlikely to resolve disagreement, or Bmax is reached.Source: B. ATEST-Online · p. 5
  • With three votes for 42, ATEST-Online stops at B = 3; with votes 42, 42, and 40, it may add a sample when idle but stop at 42 when busy.The example contrasts a stable vote with a disagreement whose next step can become too costly under load.Source: B. ATEST-Online · p. 5
  • The controller runs on the host CPU, but its main runtime cost is generating samples in several smaller steps rather than one batch.Each step also reads NVML, estimates gain with Monte Carlo, and performs one RLS update.Source: B. ATEST-Online · p. 5

C. ATEST-Calibrated

ATEST-Calibrated profiles dataset-specific accuracy and energy models offline, then uses them to choose a budget per query. Its cost model remains fixed after calibration, so changing real costs can cause overspending or unnecessary deferrals.

C. ATEST-Calibrated

  • For each dataset, ATEST-Calibrated profiles 500 prompts at five budgets and four load levels, fitting fixed accuracy and energy models from 44,000 generated samples.The calibration prompts are separate from evaluation queries; energy is predicted from power and runtime.Source: C. ATEST-Calibrated · p. 5
  • At runtime, ATEST-Calibrated selects the affordable budget upgrade with the best expected accuracy gain per extra joule, deferring a query if even B = 1 does not fit.It then generates the selected number of samples and returns their plurality answer.Source: C. ATEST-Calibrated · p. 5
  • Because its cost model is not updated, ATEST-Calibrated can overspend or reject queries it could have served when actual costs diverge from the model.The mode still reads live GPU state, but its calibration-derived models stay fixed.Source: C. ATEST-Calibrated · p. 5

VI. EXPERIMENTAL METHODOLOGY

The evaluation measures answer quality and system behavior as GPU conditions change, using common models, prompts, generation settings, and sampling budgets across methods.

A. Model, Hardware, and Settings

  • The main evaluation uses Phi-3-mini-4k-instruct in bfloat16 on one NVIDIA A100-SXM4-80GB GPU, with batch size 32 and budgets B = {1, 3, 4, 6, 8}.Transfer tests use an NVIDIA L40S and Qwen2.5-3B-Instruct with the same sampling settings, query sets, and load schedule.Source: A. Model, Hardware, and Settings · p. 5, §VI · p. 5

B. Datasets

  • Four test sets of 500 queries cover GSM8K and MATH long-answer math tasks plus short-answer ARC-Challenge and CommonsenseQA; reported results use test sets.Development and test sets do not overlap, and query sets and controller settings were frozen before results were examined.Source: B. Datasets · p. 6

C. Compared Methods •

  • Comparisons include fixed-B = 4 self-consistency, Adaptive-Cal, ATEST-Calibrated, and default ATEST-Online.Adaptive-Cal is the closest prior work because it models energy; reasoning-only adaptive methods cannot react to GPU load.Source: C. Compared Methods • · p. 6

D. Controlled GPU Load

The methodology creates controlled shared-GPU load and applies a common energy budget and accounting rule across methods. It reports answer quality, energy, budget safety, and system performance.

D. Controlled GPU Load

  • A background injector runs dense FP16 matrix multiplies at idle, low, medium, or high duty settings, with 125 of each 500 queries assigned to each load level in a fixed reused schedule.Fixed-4 utilization measured about 0%, 24%, 54%, and 85%; GSM8K average GPU power rose from 77 W idle to 431 W at high load.Source: D. Controlled GPU Load · p. 6

E. Energy Accounting and Budget

  • Each microbatch is charged GPU occupancy energy from NVML power and runtime, and every method receives the same Emax budget based on reference-condition B = 4 energy.The budget is set separately for each GPU, model, and dataset because sample cost depends on all three.Source: E. Energy Accounting and Budget · p. 6

F. Metrics

  • The evaluation reports accuracy, deferrals, sampling effort, energy per query, joules per correct answer, budget use, p95 latency, throughput, utilization, and power.Overall mean B counts deferred queries as B = 0; per-load mean B averages only answered queries.Source: F. Metrics · p. 6

G. Calibration Cost

ATEST-Online serves every query within budget and achieves the highest or tied-best accuracy across four datasets. The evaluation also examines load response, calibration cost, and whether resource signals drive adaptation.

G. Calibration Cost

  • Offline calibration for the four datasets uses 5.73 GPU-hours and 7.16 MJ before serving begins.The profiling uses 2,000 prompts, 80 settings, and 176,000 generated samples.Source: G. Calibration Cost · p. 6

H. Controlled Ablation

  • The controlled ablation compares three ATEST-Online variants using a non-binding energy budget and a fixed remaining-energy signal.The variants are Full-HW, noHW-State, and ReasoningOnly-Strict.Source: H. Controlled Ablation · p. 6

VII. EVALUATION

  • The evaluation is organized around seven questions spanning load robustness, controller behavior, energy cost, resource-signal effects, runtime overhead, calibration choice, and transfer.Source: §VII · pp. 6–7

A. Q1: Overall Results under Changing Load

B. Q2: Behavior as the Load Rises

  • As load rises, both ATEST modes lower sampling, but on high-load GSM8K ATEST-Calibrated defers 42.4% and reaches 44.0% accuracy versus ATEST-Online’s 60.0%.ATEST-Online lowers its mean budget from idle to high load on all four datasets.Source: B. Q2: Behavior as the Load Rises · p. 7

C. Q3: Energy Cost, Including Profiling

Including profiling, ATEST-Online uses substantially less total energy and GPU time than ATEST-Calibrated for the evaluated workload. A controlled ablation finds that online cost learning is necessary for budgets to follow load, with GPU state further changing decisions.

C. Q3: Energy Cost, Including Profiling

  • Across load levels, both ATEST modes lower their budgets, but ATEST-Calibrated begins deferring at medium or high load on all four datasets.The figure’s top, middle, and bottom panels show accuracy, deferred queries, and mean B, respectively.Source: Figure 4 · p. 8
  • For the same 2,000 queries, ATEST-Calibrated uses about 42× more GPU time and 48× more energy than ATEST-Online when profiling is included.Serving costs are almost the same, but calibrated profiling costs 5.73 GPU-hours and 7.16 MJ for the four datasets.Source: Table IV · p. 8, C. Q3: Energy Cost, Including Profiling · p. 8
  • ATEST-Online is not always the cheapest per correct answer during serving, but after profiling it uses far less total energy on every dataset.ATEST-Calibrated is more efficient on CommonsenseQA during serving, but leaves 4.6% of queries unanswered.Source: C. Q3: Energy Cost, Including Profiling · p. 8

D. Q4: Is the Adaptation Caused by Resource Signals?

  • With budget effects removed, online cost learning is necessary for mean B to follow load, and adding GPU-state input further changes decisions.Without resource feedback, ReasoningOnly-Strict uses the same mean budget, B = 3.12 samples per query, at every load level.Source: Table V · p. 8, D. Q4: Is the Adaptation Caused by… · p. 9

E. Q5: Runtime Overhead and Cost-Model Error

Across changing GPU loads and transfer settings, ATEST-Online remained within budget and answered every query, while offline calibration was competitive under stable load. Online control brings higher tail latency and lower throughput on short tasks, and its cost estimates are noisier there.

E. Q5: Runtime Overhead and Cost-Model Error

  • Online control had higher p95 latency on three datasets and lower throughput on the two multiple-choice datasets, while cost-model error was much higher on short tasks.Energy MAPE was 134% on ARC-Challenge and 112% on CommonsenseQA, but ATEST-Online never exceeded the energy budget; small steps and a safety margin limit the impact of prediction errors.Source: E. Q5: Runtime Overhead and Cost-Model… · p. 9

F. Q6: When Does Offline Calibration Win?

  • With stable GPU load, ATEST-Calibrated matched or beat ATEST-Online in accuracy across all four datasets, and Fixed-4 had the lowest J/correct on three.The no-injected-load run also favored ATEST-Calibrated on GSM8K, while the methods were close on the other datasets.Source: F. Q6: When Does Offline Calibration… · p. 9, Table VIII · p. 10

G. Q7: Does the Behavior Hold on Another GPU and Another Model?

  • Across eight L40S and Qwen2.5-3B settings, ATEST-Online answered every query within budget, was at least as accurate as Fixed-4, and used less energy per correct answer in seven.The exception was Phi-3-mini on L40S CommonsenseQA, where the loose budget let Fixed-4 answer every query and the methods were nearly equal.Source: G. Q7: Does the Behavior Hold on… · pp. 9–10, Table VII · p. 10

VIII. DISCUSSION

The paper recommends choosing between ATEST modes according to load stability and profiling needs, while treating test-time scaling on shared accelerators as runtime resource management. Its evaluation is compute-bounded, and online control costs latency and throughput on short tasks.

VIII. DISCUSSION

  • Stable loads and reusable profiles favor ATEST-Calibrated, while changing loads or frequent model and dataset changes favor ATEST-Online.The modes can also be combined by starting ATEST-Online from a calibrated cost model.Source: §VIII · p. 10
  • An LLM-controlled stopping variant had similar accuracy but exceeded the energy budget on three of four datasets and used extra decision energy.The paper characterizes a small numerical controller as cheaper and easier to keep within a hard budget.Source: §VIII · p. 10

IX. LIMITATIONS AND FUTURE WORK

  • The evaluation is bounded by compute availability and covers 500 queries per dataset, four datasets, two open models, and two GPU types.The paper also identifies short-task runtime overhead and the simple online cost model as areas for future work.Source: §IX · p. 10

X. CONCLUSION

  • Under changing load on four datasets, ATEST-Online answered every query within budget and achieved highest or tied-best accuracy, while offline calibration deferred up to 42.4% at high load.The conclusion reports the same behavior on an L40S and with Qwen2.5-3B, and says online cost learning drives budget adaptation to load.Source: §X · pp. 10–11

Related papers

Loading 2610.25519v1…