Source-linked AI summary
TokenPowerSandbox: Evidence-Gated CPU-First Screening for Energy-Aware LLM Serving
Chenxu Niu
TL;DR
Energy-aware LLM serving needs affordable, trustworthy comparison of configurations across realistic request shapes and service objectives. TokenPowerSandbox gates cheap CPU predictions with target-GPU probes, full-workload verification, and frozen provenance, preserving energy ordering in its tested envelope with 7.35% energy MAPE in no-refit confirmation.
Problem
Energy-aware serving lacks affordable evidence for comparing configurations across realistic workload shapes and deciding when predictions are trustworthy near service-level boundaries.
Method
TokenPowerSandbox combines an interpretable CPU projector, short target-GPU probes, full-workload verification, and hash-tracked frozen provenance with metric-specific evidence gates.
Results
7.35% energy MAPE with ρ=0.933 and 33/36 correct energy orderings demonstrates no-refit preservation of energy decisions within the tested envelope.
Takeaways & Limitations
Cheap predictions are actionable only when evidence kind, scope, chronology, and abstention accompany them and final decisions are independently measured.
Takeaways & Limitations
Evidence is limited to one Qwen2.5-7B model, one H100 80GB, one vLLM release, BF16, a fixed power limit, and bounded protocols without cross-product exploration.
Abstract
from arXiv · showhide
Energy-aware LLM serving requires comparing configurations under realistic request shapes, yet exhaustive target-GPU profiling is costly and a cheap predictor can be dangerously confident outside its measured scope. We present TokenPowerSandbox, an evidence-gated workflow that combines an interpretable CPU-resident projector, short target-GPU probes, full-workload verification, and tamper-evident freeze-before-measurement provenance. On one NVIDIA H100 80GB serving Qwen2.5-7B-Instruct with vLLM, three anchor repeats and six development workloads calibrate workload transfer. The same frozen model is evaluated on a blind holdout and a separately predeclared no-refit confirmation totaling 51 post-freeze runs. Energy MAPE is 6.23% and 7.35%, with Spearman rank correlations of 0.976 and 0.933. However, a predeclared TTFT gate passes at concurrency four (9.27% MAPE) and triggers abstention below four (64.80%), showing why energy accuracy cannot certify latency.
1 Introduction
TokenPowerSandbox frames energy-aware LLM serving as an evidence-gated screening problem: energy depends on request and serving conditions, while cheap predictors can fail near service-level boundaries. Its CPU-first, probe-and-verify workflow freezes predictions and provenance before measurement and explicitly bounds the evidence scope.
- Motivation: Energy depends on token lengths, concurrency, batching, cache policy, precision, and parallelism, so deployment must optimize energy under explicit SLOs.These factors affect power, duration, throughput, TTFT, and TPOT.
- Motivation: Exhaustive target-accelerator measurement is trustworthy but expensive, while low aggregate model error can still misorder candidates near SLO boundaries.Workload models calibrated in one utilization regime may fail sharply in another.
- Method: TokenPowerSandbox separates deterministic CPU Sandbox estimates, short measured Probes, and complete target-workload Verify runs instead of treating CPU predictions as hardware evidence.Retained records include acquisition kind, scope, cost, hashes, software and hardware identity, and chronology.
- Contributions: 51 post-freeze measurements support a CPU-first model calibrated on six development workloads, followed by a frozen blind H100 holdout and predeclared no-refit confirmation.Predictions and experiment order are frozen before the blind campaign.
- Scope: 154 real H100 serving executions establish same-H100 screening for a fixed single-GPU case study, not cross-model, cross-GPU, multi-GPU, or multi-node transfer.The boundaries deliberately isolate the evidence substrate for a later TokenPowerAgent.
2 Problem, Evidence, and Claim Boundary
The section defines an evidence-gated optimization contract that separates requested work from deployable controls and treats full-workload verification as the reference fidelity. Its claim is limited to screening within one measured H100 envelope, detecting failure, and confirming selected candidates independently.
- Problem definition: A serving configuration contains deployable controls, while the fixed workload specification preserves the requested request and token geometry.Separating the workload specification from configuration prevents optimization from silently reducing the requested work.
- Evidence contract: Verify measures GPU energy per 1,000 output tokens and output-token throughput on the full target workload, providing reference fidelity.Sandbox and Probe are earlier evidence stages; Verify is the reference for the frozen configuration study.
- Optimization boundary: The frozen study minimizes verified energy while maximizing throughput; P95 TTFT and P95 TPOT constrain feasibility but are not Pareto objectives.The target set comprises nondominated candidates whose verified metrics satisfy every SLO.
- Evidence contract: Only real hardware execution creates measured or verified evidence; CPU calculations remain simulated or extrapolated regardless of sophistication.Campaign records also track acquisition semantics, outcomes, uncertainty, GPU-hour cost, execution, and provenance.
- Claim boundary: The claim tests cheap screening within one measured H100 envelope, detectable failure, and independent confirmation of candidates selected through the evidence ladder.It does not claim to predict any deployment.
3 TokenPowerSandbox Design
TokenPowerSandbox combines an interpretable CPU-resident projector with measured execution, staged Probe/Verify workloads, and freeze-before-measurement validation. Its design gates predictions by measured scope and separates energy derivation from corrected component metrics.
- Projector: The base projector rejects memory-infeasible configurations, separates prefill and decode work, scales measured anchors by token geometry and batch occupancy, and bounds active power.It preserves the physical identity of the modeled quantities.
- Metric modeling: Energy per 1,000 output tokens, throughput, TTFT, and TPOT are derived with a term-by-term decomposition.Energy is recomputed from corrected power and duration rather than independently fitted; ridge strength is selected per metric using leave-one-workload-out development MAPE.
- Measurement: The executor uses digest-pinned vLLM, fixed generation settings, restricted containers, and DCGM counter reads bracketing only the active request window.It also emits 100-ms telemetry, benchmark output, server metadata, model and token information, and other run artifacts.
- Measurement: Probe submits 64 requests while Verify submits the complete predeclared 256-request workload using the same executor and metrics.Repeats are summarized by median and MAD, with stage labels and manifest hashes attached to summaries.
- Evidence gating: Before each post-development campaign, predictions, identities, schedules, hashes, and manifests are serialized, while validation rejects altered provenance or post-measurement prediction generation.The residual model is restricted to its measured model, runtime, hardware, workload, and bounded context–concurrency envelope; holdout and confirmation data cannot enter fitting.
4 Experimental Methodology
The methodology combines fixed H100 serving conditions with workload-transfer calibration, frozen holdout and scope-confirmation tests, and a separate configuration study using interleaved CPU predictions, probes, and verification. All predictions, identities, and schedules are frozen before measurements, with explicit energy, latency, and SLO gates governing released claims.
- Serving setup: All serving experiments use one NVIDIA H100 80GB GPU at a fixed 700-W limit, with digest-pinned vLLM 0.23.0 serving Qwen2.5-7B-Instruct in BF16.Parallelism degrees remain one, and device energy is measured during the active benchmark window using DCGM cumulative device-energy counter field 156.
- Workload-transfer calibration: Three anchor repeats at 512 input tokens and concurrency eight, plus six development workloads spanning 256–2,048 inputs and concurrency 1–32, fit the residual model.Each development workload is repeated three times; leave-one-workload-out energy MAPE is 10.11%.
- Frozen confirmation: Holdout v2 freezes eight disjoint workloads before 24 cyclically balanced measurements, while scope confirmation v3 adds 27 measurements on a predeclared 3 × 3 input-length/concurrency grid.The primary v3 endpoint is all-workload energy MAPE ≤15%, and TTFT is released at concurrency four only under its stratum-MAPE gate, with abstention below four when sparse TTFT MAPE exceeds 20%.
- Configuration study: The configuration study evaluates 12 frozen candidates plus a predeclared expert baseline under fixed 2,048-input/128-output, concurrency-32 conditions and P95 TTFT/TPOT SLOs.Candidates vary maximum sequences, maximum batched tokens, and chunked prefill; the SLO requires P95 TTFT ≤1,600 ms and P95 TPOT ≤20 ms.
- Configuration acquisition: Each candidate receives three Probe and three Verify repeats, yielding 36 real runs per fidelity, with median Verify records defining the SLO-feasible reference set and Pareto frontier.The cyclically balanced schedule interleaves fidelities without using one to select candidates for the other.
- Post-selection confirmation: After corpus selection, seq32-bt2048-chunk and the expert baseline undergo a new five-seed alternating Verify campaign with no refitting or reselection.The predeclared gate requires five energy pair wins, a positive 95% bootstrap lower bound for mean paired energy savings, lower median paired P95 TTFT, and successful SLO-feasible execution.
5 Results
Frozen CPU-first predictions accurately transfer energy rankings across blind holdout and no-refit confirmation, but latency decomposition fails outside concurrency four, requiring explicit abstention and full verification. The verified configuration improves energy and latency locally, while provenance controls expose analytical-baseline failure and measurement cost.
- Frozen workload studies: 6.23% energy MAPE on holdout v2 improves from 63.01%, with ρ=0.976 and 27/28 correct workload orderings.Median and maximum energy APE are 4.96% and 13.84%; median repeat CV is 0.78%.
- Scope confirmation: 7.35% energy MAPE and ρ=0.933 on no-refit confirmation pass the frozen 15% endpoint with 33/36 correct energy orderings.All 17 individual energy errors remain below 15%, and the same minimum-energy workload is identified as measurement.
- Latency gate and abstention: 9.27% TTFT MAPE at concurrency four passes the 20% threshold, while the sparse concurrency-one-and-two stratum reaches 64.80%, triggering abstention below four.Energy remains below 10% MAPE in every stratum, despite throughput and TPOT errors of 24.02% and 18.25% at k=1.
- Configuration screening: 19.45% energy MAPE and ρ=−0.968 TTFT rank correlation show that the analytical Sandbox alone cannot establish SLO feasibility or frontier recall.Its energy rank is ρ=0.884, but frontier recall is 0%; the 64-request Probe reduces energy and TTFT MAPE to 1.15% and 17.20%, respectively.
- Independent winner confirmation: 1.39% mean paired device-energy saving is confirmed across all five seed-matched pairs, with a 95% interval of [1.19%, 1.59%] and p=0.03125.The locked candidate also increases mean paired throughput by 3.46% and lowers median paired P95-TTFT by 21.43%; the result is local to one model, request shape, H100, and two configurations.
- Evidence-generation cost: 1.117 H100-hours across reported campaigns quantify the offline cost of building controlled evidence before later online policy decisions.This excludes three anchor executions and includes 0.200 Probe and 0.417 Verify hours in the balanced configuration corpus.
6 Discussion
TokenPowerSandbox shows that a sparse, interpretable correction can preserve energy ordering within a tested envelope, but latency support is metric- and concurrency-specific. Its fidelity ladder separates CPU screening, short probes, verification, and evaluator validity from adaptive candidate acquisition.
- What the positive energy result means: A fixed sparse correction reproduces same-H100 energy ordering across the measured context–concurrency envelope without refitting.The blind holdout and separately predeclared no-refit confirmation support reproducibility within the tested envelope.
- What the latency failure teaches: TTFT is supported only at concurrency at least four; elsewhere, direct measurement is required.The low-concurrency failure establishes that scope is metric-specific and has an internal occupancy boundary.
- Why a fidelity ladder is preferable to one predictor: The CPU prior preserves energy rank but misses the only verified Pareto point because its TTFT ordering is reversed.A short measured Probe removes this false negative, while Verify removes remaining false positives; the balanced corpus does not test adaptive acquisition.
- Separation from TokenPowerAgent: TokenPowerSandbox isolates scenarios, evidence acquisition, provenance, metric-specific release, and verification before studying adaptive candidate acquisition.TokenPowerAgent is described as a separate future policy, while the methodology focuses first on evaluator validity.
7 Related Work
Prior work measures inference energy, models workload effects, simulates serving, allocates evaluation budgets, and optimizes energy-aware deployments. TokenPowerSandbox complements these approaches by treating real-GPU measurement, provenance, and release boundaries as the authority for evaluated serving records.
- Inference-energy measurement: Energy-measurement systems quantify inference energy across model, token, hardware, and serving choices within declared device, node, or system boundaries.ML.ENERGY automates comparable measurement, while TokenPowerBench provides declarative workloads and phase-aligned device/node telemetry.
- Inference-energy measurement: TokenPowerSandbox retains real-GPU measurement as authority while adding freeze-before-measurement evidence and release boundaries for measured and predicted serving records.Its synchronized serving window is device-only; host, rack, and cooling energy remain outside the present metric.
- Workload-based energy modeling: Workload-based models already use tokens, analytical prefill/decode estimators, or phase-aware predictors; TokenPowerSandbox instead studies evidence-gated evaluation and provenance.The related work includes heterogeneous-system energy/runtime models, H100-class inference estimation, and predictors spanning models, GPUs, and parallelism.
- Serving simulation and recommendation: Serving simulators, deployment recommenders, and serving systems address simulation, placement, batching, or memory management, whereas TokenPowerSandbox introduces no scheduler.It also refuses to treat one-H100 measurements as communication or queueing evidence for a different topology.
- Multi-fidelity allocation: Hyperband, BOHB, and energy-aware serving optimizers assume an evaluator or informative lower budgets; TokenPowerSandbox audits that premise without proposing adaptive allocation.Future policies can be compared on hash-frozen evaluation summaries without relabeling simulation as measurement or training on final confirmation data.
8 Limitations and Threats to Validity
The evidence is limited to a narrow hardware, model, software, workload, and protocol scope, with important energy and latency conditions untested. Statistical uncertainty, extrapolation, and provenance constraints prevent broader claims about transfer, coverage, or deployment performance.
- Experimental scope: The study covers one Qwen2.5-7B model, one H100 80GB, one vLLM release, BF16, a fixed 700-W limit, and two bounded protocols.The cross-product of serving configurations and workloads is not explored; six development workloads are small for a four-feature residual, and low-concurrency TTFT failure shows numerical range is incomplete scope.
- Measurement scope: DCGM measures device energy only, excluding host, network, cooling, facility, and embodied energy; startup, warmup, deployment amortization, semantic quality, arrivals, production traces, and failures remain untested.Fixed token counts and temperature zero control completed work, while all requests succeeded.
- Statistical validity: 1.39% mean paired device-energy saving was observed in an independent five-pair campaign and should not be extrapolated beyond the tested pair.The five pairs support the reported exact sign-test p-value but not broad effect generalization.
- Statistical validity: Prediction-interval containment cannot be reported as statistical coverage because intervals were not calibrated before post-development campaigns and automatic-release flags remained unset.The campaign validator did not promote the flags after holdout evaluation.
- Extrapolation and provenance: The CPU prior is an out-of-scope analytical extrapolation after knob changes, while multi-GPU communication, placement, H200/B200 adaptation, multi-node validation, and cross-topology claims remain unsupported.Placeholder cross-topology values are disabled for reported same-H100 point estimates; three anchor repeats are retained only in aggregate, while other 151 executions have row-level records.
9 Conclusion
TokenPowerSandbox frames cheap LLM-serving predictions as hash-tracked evidence rather than hardware truth, pairing frozen evaluation with abstention and independent measurement. The report establishes this substrate for TokenPowerAgent without claiming adaptive-agent or multi-GPU results.
- Evaluation and evidence: 6.23% and 7.35% energy MAPE were obtained on a frozen blind single-H100 holdout and separately predeclared no-refit confirmation, with strong rank preservation.A predeclared gate correctly abstained from inaccurate low-concurrency TTFT.
- Evaluation and evidence: The CPU prior preserves energy order across frozen serving configurations but misses the set’s verified frontier.The conclusion reports this limitation without treating the cheap predictor as hardware truth.
- Practical implications: Cheap models are useful only when evidence kind, scope, chronology, and abstention behavior accompany every prediction and final decisions are independently measured.This single-GPU report establishes the substrate for the next TokenPowerAgent stage without claiming adaptive-agent or multi-GPU results in advance.