PaperlayerView exact revision on arXiv

Source-linked AI summary

Tool Waiting and Re-arrival in Compile-Time-Static LLM Serving: Cost Mechanisms and Configuration Selection

Dongkyeom Jang, In-Nea Wang, Junho Jeong

arXiv:2609.34663v1cs.PF

TL;DR

Agentic re-arrivals interact with compile-time-fixed batch buckets and KV slots, but this setting has not been treated systematically. The paper models their execution costs and validates configuration choices on new inputs. At N=8 with substantial baseline reuse loss, increasing batch_size saved 8.25%, while the selected configuration saved 9.72%.

  • Problem

    Irregular tool-driven re-arrivals interact with batch buckets and KV slots fixed at compile time, a setting not treated systematically.

    Source: §I · pp. 1–2

  • Method

    The study measures re-arrival mechanisms, simulates request processing in event order, searches configurations, and validates selected choices on new inputs.

    Source: B. Conclusion · pp. 16–17

  • Results

    At N=8 with substantial reuse loss, increasing batch_size saved 8.25%, while also changing the grid saved 9.72%.

    Source: B. Conclusion · p. 17

  • Takeaways & Limitations

    Compile-time configurations should be selected by diagnosing KV reuse loss and evaluating resulting execution, since padding improvement alone does not guarantee lower cost.

    Source: B. Conclusion · p. 17

    The study used synthetic prompts and one tool call per session, and did not measure reuse loss or interference under repeated calls and accumulating context.

    Source: A. Limitations · p. 15

Abstract

from arXiv · show

In agentic LLM services, a session calls an external tool, waits for it, and re-arrives to continue inference. Statically compiled NPU serving can fix the batch bucket set, the maximum batch size, and the number of KV cache slots at compile time. We define such an environment as a compile-time-static serving substrate and analyze the execution-time cost that tool waiting and re-arrival incur in it. On a single LLM instance, we run synthetic workloads following a measured tool waiting time distribution and compare, on the same inputs, a baseline configuration with settings {1, 2, 4, 8}, 8, and 8 against configurations that change some of them. Because re-arrival times differ across configurations, we build a simulator that replays request processing in time order, select the candidate with the lowest predicted cost among 2,077 configurations, and validate it on new inputs. We identify three mechanisms: discrete batch alignment, KV cache survival, and prefill interference. At a concurrency of 6, absent from the bucket set, tool waiting lowered the padding ratio (0.235 to 0.120) yet increased decode execution time 1.51-fold, so padding alone did not indicate cost. On new inputs at a concurrency of 8, where the baseline reused KV in 9 of 24 re-arrivals, enlarging the maximum batch size alone cut execution cost by 8.25%, and the selected configuration, which also adjusted the bucket set, by 9.72%. Where 17 of 18 re-arrivals were already reused, the effect was 0.59%. Compile-time configurations should thus be selected by diagnosing KV reuse loss and the resulting change in execution.

I. INTRODUCTION

This paper studies how tool waiting and re-arrival affect cost when batch buckets and KV slots are fixed at compile time. It identifies three execution mechanisms and applies them to selecting configurations.

I. INTRODUCTION

  • The intersection of irregular agentic re-arrivals with compile-time-fixed batch buckets and KV slots has not been treated systematically.The study focuses on costs created by this fixed resource structure, distinct from runtime KV retention or eviction policies.Source: §I · pp. 1–2
  • Measurements identify discrete batch alignment, KV-cache survival, and prefill interference as mechanisms through which re-arrivals affect device time and other sessions’ delay.A paired comparison confirmed that the bucket grid causes the padding difference.Source: §I · p. 2
  • The mechanisms interact through the arrival process: resolving reuse failures reduces re-arrival recomputation, while additional prefill can delay background sessions.Tool waiting can improve batch alignment while worsening KV reuse.Source: §I · p. 2
  • The study applies a validated simulator to 2,077 configurations and measures selected configurations on new inputs, finding that changing batch_size captures most of the gain where baseline reuse loss is substantial.When reuse is already retained, adding slots has limited effect; the paper also reports where gains from additional slots stop.Source: §I · p. 2

A. Target System

  • Experiments use one Qwen3-4B instance on four devices of a Rebellions CA25 NPU server.The server’s remaining devices were unused, and measurements target this single instance.Source: A. Target System · p. 2

B. Execution Structure

The serving substrate maps decode requests to compiled batch buckets and reserves KV slots at compile time. Synthetic paired workloads vary tool waiting and re-arrival while controlling planned inputs.

B. Execution Structure

  • Decode steps select a compiled bucket for the actual request count; positions beyond the request count are padding.The baseline uses buckets {1, 2, 4, 8} with batch_size 8.Source: B. Execution Structure · p. 2
  • On the eager path, each KV slot holds one sequence, and the baseline has eight slots with FIFO eviction that is not refreshed on re-access.The block size and number of blocks are fixed at compile time.Source: B. Execution Structure · pp. 2–3

C. Workload and Comparison Design

  • Each synthetic session makes one tool call, then re-arrives with eight added context tokens; surviving prefix KV is reused, otherwise the full re-arrival prompt is recomputed.One waiting interval isolates re-arrival effects from repeated calls and context accumulation.Source: C. Workload and Comparison Design · p. 3
  • Tool waits are sampled from a TraceLab-based distribution, and concurrency N is varied because it controls active-request counts and KV-slot pressure.N values span 3 to 16; 6 and 8 are the simulator’s confirmatory range, while 10 is exploratory.Source: C. Workload and Comparison Design · p. 3
  • Within each replicate, conditions share an input plan, but execution outcomes determine actual completion and re-arrival times.The plan fixes prompts, generation lengths, waiting times, and session composition; it does not fix the times execution produces.Source: C. Workload and Comparison Design · p. 3

D. Measurement Metrics and Experimental Procedure

The study defines device time from decode-step costs and server-side prefill time, then examines how batch alignment affects decode cost. Tool waiting can lower padding while increasing execution time because it changes both bucket use and step count.

D. Measurement Metrics and Experimental Procedure

  • Device time sums measured decode-step cost and server-side prefill elapsed time; it is not pure accelerator occupancy, and comparisons use the same input plan.Padding and recomputation lengthen execution and therefore count as additional cost in this metric.Source: D. Measurement Metrics and… · p. 4
  • The serving stack does not report device time directly, so the study cross-checks a model-based reconstruction against client-observed in-flight intervals and applies a predefined agreement criterion.The client-side channel includes queueing, HTTP, and scheduler overhead; the tolerance is an empirical decision criterion, not a rigorous overhead bound.Source: D. Measurement Metrics and… · p. 4

A. Discrete Batch Alignment

  • A decode step’s cost is largely a fixed cost set by its selected bucket, so splitting work into smaller batches can increase total cost by increasing step count.An additional request adds 0.041 ms on average; measured full-bucket medians range from 9.51 ms for bucket 1 to 12.40 ms for bucket 8.Source: A. Discrete Batch Alignment · p. 4
  • At N=6, the padding ratio with tool waiting was 0.120 versus 0.235 without; at N=8, it was 0.163 versus 0.090.The direction reverses because six is absent from the baseline grid, whereas eight aligns exactly when all sessions are active.Source: A. Discrete Batch Alignment · p. 5
  • Adding bucket 6 reduced the no-waiting padding ratio at N=6 from 0.232 to 0.057 and reversed the padding difference from +0.065 to −0.040.All three replicates showed the same sign reversal, and the intervention changed bucket choice while leaving the no-waiting step distribution unchanged.Source: A. Discrete Batch Alignment · p. 5
  • Across all nine tested conditions, tool waiting increased decode device time, with cost ratios of 1.05–1.78, including five cases with lower padding ratios.At N=6 on the baseline grid, tool waiting produced 2,193 decode steps versus 1,308 without waiting, summed over three replicates.Source: A. Discrete Batch Alignment · p. 5

B. KV Cache Survival

KV reuse depends on whether the prior prompt’s reusable prefix survives until re-arrival lookup; slot pressure and allocation order can turn full reuse into none. In concurrent runs, reuse varies, while a larger-batch intervention improved reuse and reduced recomputation but changed more than slot count alone.

a) Reuse signal and range

  • The reusable range is the 128-token-aligned portion of the previous prompt, excluding generated output; 95 of 168 validation re-arrivals reused it, while 73 had no cached tokens.No partial reuse was observed, and the causes of the 73 failures were not determined case by case.Source: B. KV Cache Survival · pp. 5–6

b) Sequential experiment controlling the number of background requests

  • With 6 background requests, 1,920 tokens were reused; with 7, reuse fell to 0 across re-validation replicates, defining a request-order-specific critical request count.This is not a threshold on concurrency N.Source: B. KV Cache Survival · p. 6

c) Eviction order and KV survival

  • A re-arrival needs a new slot, and FIFO eviction without order refresh can remove the target KV before lookup, producing full reuse or none.The observed eviction count was max(0, m −5) across all 21 runs; survival depends on whether eviction precedes lookup, not the run-wide count alone.Source: B. KV Cache Survival · pp. 6–7

Concurrent execution

  • In concurrent execution, reuse varied across runs: six of eight re-arrivals succeeded in seven repeated runs, versus five in three runs.Earlier transmission rank was associated with successful reuse in five of six condition–replicate combinations, but transmission order was not independently manipulated.Source: B. KV Cache Survival · p. 7

d) Intervention with more KV slots

  • 17,024 more reused tokens matched an exact 17,024-token reduction in re-arrival prefill computation after batch_size rose from 8 to 16.The baseline reused 3 of 8 re-arrivals in each replicate, versus all 8 after the change; bucket 16 was never selected.Source: B. KV Cache Survival · p. 7, Table III · p. 7

e) Additional computation cost of reuse failure

  • For a 2,008-token re-arrival, prefill time rose from about 27 ms with reuse to 359 ms without reuse, about 13.1-fold.The 2,008-token point was used to fit the cost model, so this comparison is not an independent validation.Source: B. KV Cache Survival · p. 7

C. Prefill Interference

An injected prefill can lengthen background sessions’ response intervals because this stack executes prefill and decode exclusively. The section separately models prefill elapsed time and cumulative session waiting, and notes the model’s limited fitting range and one observed underestimation below that range.

a) Exclusive execution and background session delay

  • Prefill and decode do not share an execution step on this stack, so background decode progress is delayed during prefill execution.The injection experiment observed lengthening of background response intervals; exclusive execution is a property of this stack.Source: C. Prefill Interference · pp. 7–8

c) Increase in response arrival intervals

  • For a 2,000-token injected prefill, background response intervals rose from about 12 ms to about 373 ms, with long intervals in all four sessions across all nine injection runs.The delay spike was about 12–15 ms longer than elapsed time, but the excess was not decomposed into a particular execution cost.Source: C. Prefill Interference · p. 8, Table IV · p. 9
  • All 36 background-session cases exceeded the five-times-baseline threshold, but control-run spikes made the existence decision only a partial confirmation.The simultaneity and growth-with-length criteria were satisfied.Source: C. Prefill Interference · p. 8

d) Prefill elapsed-time model

  • The cost model was fitted over 500–6,000 tokens with a maximum absolute residual of 2.4 ms, but below-range application underestimated one 88-token residual computation by about 6 ms.The below-range error was assessed at that single point, and fitting residuals do not establish independent accuracy.Source: C. Prefill Interference · p. 8

e) Cumulative waiting time due to interference

  • Execution cost and background delay are distinct: one prefill’s cost is counted once, while cumulative waiting sums the delays of affected sessions and is not added to device time.When the same sessions wait throughout, Wj = KjTj; using server-side elapsed time or its model for Tj is an approximation.Source: C. Prefill Interference · p. 8

D. Confirmed Range of the Three Mechanisms and Their Coupling through the Arrival Process

The three mechanisms interact through sequential effects, conflicting metric changes, and feedback from execution outcomes to later re-arrivals. These couplings depend on reuse loss, overlap, and the batch grid, so per-mechanism effects cannot simply be added across configurations.

D. Confirmed Range of the Three Mechanisms and Their Coupling through the Arrival Process

  • Batch-grid alignment governs padding, fixed slots and FIFO order govern KV survival, and exclusive execution lets added prefill delay other sessions' decoding.Because these mechanisms share an arrival process, baseline effects cannot be treated as independent fixed contributions when evaluating another configuration.Source: D. Confirmed Range of the Three… · p. 9
  • KV reuse failure adds prefill work that can delay decoding sessions, but the experiments did not directly confirm how adding slots changes background delay.The additional computation is bounded by reuse previously lost; even successful reuse leaves prefill work outside the reused range.Source: D. Confirmed Range of the Three… · p. 9
  • At N=6, tool waiting lowered padding ratio from 0.232 to 0.167 while re-arrival reuse fell from 18/18 to 14/18 on the baseline grid.With bucket 6 added, padding ratio was 0.057 without waiting and 0.098 with it; at N=8 on the baseline grid, waiting reused 11 of 24 versus 7 of 24 without waiting.Source: D. Confirmed Range of the Three… · p. 9
  • Execution changes can alter later completion and re-arrival times, active-request counts, and which KV is evicted, although the observed distribution shifts were not attributed to re-arrival timing alone.The slot intervention also changed the execution limit, so its decode shift cannot be attributed to reuse improvement alone.Source: D. Confirmed Range of the Three… · p. 9

IV. EXECUTION SIMULATOR AND COST PREDICTION BETWEEN CONFIGURATIONS

The paper builds an event-driven simulator to predict cost ratios between configurations by replaying request processing in time order. It models arrivals, KV reuse, prefill, and decode while keeping execution cost distinct from cumulative waiting time.

IV. EXECUTION SIMULATOR AND COST PREDICTION BETWEEN CONFIGURATIONS

  • The event-driven simulator computes each request from arrival to completion in event order and evaluates prediction accuracy for changed-configuration cost ratios, not mechanism-wise cost contributions.It separates reproduction of existing data from prior prediction on new inputs.Source: §IV · p. 10

A. A Simulator That Reproduces the Request Processing

  • Each session plan supplies prompt length, generation length, and tool waiting time; altered completion times therefore change subsequent re-arrival times.Requests enter a queue and begin execution according to running-request count and KV availability.Source: A. A Simulator That Reproduces the… · p. 10
  • KV management uses a fixed slot pool and FIFO eviction, then looks up remaining same-session KV to predict reuse and remaining prefill work.Reuse is computed in 128-token units, and failed reuse yields zero reused tokens.Source: A. A Simulator That Reproduces the… · p. 10
  • Decode selects the smallest bucket that accommodates running requests, generates one token per request per advance, and models step time using bucket cost plus fixed and per-request terms.The first token is produced by prefill, so generation length G requires G−1 decode advances.Source: A. A Simulator That Reproduces the… · p. 10
  • The simulator applies the prefill-time model to remaining work, including work below 500 tokens outside the coefficient-estimation range.That below-range use is an extrapolation.Source: A. A Simulator That Reproduces the… · pp. 10–11
  • Predicted execution cost sums decode and prefill time, excludes idle tool-wait intervals, and is evaluated separately from cumulative waiting time.The configuration comparisons use the execution-cost ratio and do not validate cumulative waiting time.Source: A. A Simulator That Reproduces the… · p. 11

B. Assumptions and Scope of the Simulator

The simulator uses measured coefficients and simplifying assumptions, and does not reproduce every event of the actual scheduler and KV management. Validation separates reproduction of existing data from prediction on new input plans, with specified channel-agreement and prediction-error criteria.

B. Assumptions and Scope of the Simulator

  • The simulator retains earlier cost coefficients without correction to validation results and omits several scheduler, client, and KV-management details, so it represents the main mechanisms rather than every execution event.It also assumes session-index order for simultaneous arrivals and zero client delay before the next request.Source: B. Assumptions and Scope of the… · p. 11

C. Comparison Metrics and Validation Design

  • Validation first checks reproduction on existing data, then records predictions and decision criteria before measuring new input plans.The three configurations share input plans and are measured in three replicates; N=6 and N=8 are confirmatory, while N=10 is exploratory.Source: C. Comparison Metrics and Validation… · p. 11
  • Simulator cost-ratio agreement mainly tests step-distribution and reuse predictions, while a separate measurement channel checks coefficients.The simulator and channel A share cost coefficients; channel B does not.Source: C. Comparison Metrics and Validation… · p. 11
  • Confirmatory prediction requires absolute error at most 0.03 and matching sides of 1 or both ratios within [0.98,1.02].N=10 receives no pass/fail decision.Source: C. Comparison Metrics and Validation… · p. 11

D. Reproduction of Execution Patterns and Cost Prediction by Configuration

The simulator reproduced execution patterns and predicted configuration cost ratios within the validated range, but consistently overestimated savings and did not establish that recomputing re-arrival times improved prediction accuracy. The authors use it to compare configurations while distinguishing validated results from exploratory predictions.

D. Reproduction of Execution Patterns and Cost Prediction by Configuration

  • The four passing cost validations had absolute errors of 0.0058–0.0076, equivalent to prediction errors of 0.58–0.76 percentage points in savings rate.The two initial N=6 comparisons met error and direction criteria but were withheld because their channel-ratio differences exceeded the prerequisite; the new-input re-measurement passed the original requirement.Source: D. Reproduction of Execution Patterns… · p. 12, Table IX · p. 12

E. Direction and Interpretation of Prediction Residuals

  • In every Table IX comparison, the simulator predicted lower cost ratios for changed configurations and overestimated savings, though the direction did not hold for every replicate.At new-input N=6, predicted versus observed baseline reuse was 16/18 versus 17/18; the authors do not attribute the residual to reuse prediction or any single factor.Source: E. Direction and Interpretation of… · p. 12, Table IX · p. 12
  • Across eight comparisons, the full and fixed-arrival simulators were each closer in four, with mean absolute errors of 0.0169 and 0.0172, so an accuracy benefit from recomputing re-arrival times was not confirmed.The full simulator recomputes re-arrival times from changed completion times; the fixed-arrival version uses baseline-run times.Source: E. Direction and Interpretation of… · p. 13
  • The validated simulator jointly modeled request order, batch composition, and KV reuse to predict configuration cost, not cumulative waiting time or the share of total cost explained by those mechanisms.The residual may combine model approximation, measurement and execution variation, and unmodeled mechanisms; a small overall error does not show that each omitted factor is small.Source: E. Direction and Interpretation of… · pp. 12–13

V. Selection of Compile-Time Configurations and Cost Improvement

  • On this stack, increasing batch_size alone restored KV reuse and reduced recomputation prefill where the baseline lost substantial reuse, capturing most of the configuration-change gain.The intervention directly confirmed reuse restoration and reduced prefill work, but did not measure changes in other sessions’ waiting time.Source: §V · p. 13

A. Comparing Configuration Candidates with the Simulator

The simulator searched 2,077 configurations and selected one for new-input measurement, where measured savings varied with baseline KV reuse and the configuration changes. Results support diagnosing reuse loss and request-count/bucket patterns before choosing slots or buckets.

A. Comparing Configuration Candidates with the Simulator

  • Configuration candidates varied batch_size across 8, 10, 12, and 16, with bucket sets containing 1 and batch_size; the search compared predicted decode and prefill cost against the baseline.The 2,077 candidates were scored on search inputs, and the seed-separated evaluation inputs were not used for selection.Source: A. Comparing Configuration Candidates… · p. 13
  • The search selected batch_size=16 with buckets {1, 4, 6, 8, 10, 16}, whose predicted evaluation cost ratio was 0.9066; this was a simulator prediction, not measured savings.The search covered N∈{6, 8, 10}; excluding N=10 changed the selected bucket set, and similar predicted costs did not establish a particular configuration’s superiority.Source: A. Comparing Configuration Candidates… · pp. 13–14

B. Gains from More KV Slots and a Changed Bucket Grid

  • At N=8, more KV slots saved 8.25% by channel A and the configuration that also changed the grid saved 9.72%; the baseline reused 9 of 24 re-arrivals, versus 24 of 24 for both changed configurations.The 17,024-token decrease in re-arrival prefill computation matched the increase in reuse; reduced prefill work, rather than reduced padding, was the main savings path.Source: B. Gains from More KV Slots and a… · p. 14, Table XII · p. 15

C. Priority of Configuration Changes by Reuse Loss

  • At N=6, where the baseline reused 17 of 18 re-arrivals, adding slots saved 0.59%, while also changing the grid raised savings to 2.11%.The additional gain from changing the grid exceeded that from adding slots in this re-measurement; its input plans differed from the N=8 measurement, so the gain difference is not attributed to N alone.Source: C. Priority of Configuration Changes… · p. 14, B. Gains from More KV Slots and a… · p. 14
  • When reuse is mostly retained, further slots can reduce little recomputation; the authors recommend checking additional prefill work and per-step request-count and bucket distributions alongside reuse failures.Failed re-arrivals can differ in token count, prefill cost, and overlap with other requests, so reuse-failure counts alone do not determine the gain.Source: C. Priority of Configuration Changes… · p. 14

D. Conditions Under Which the Gain from Additional KV Slots Is Limited

  • With 16 slots, N=6 and N=8 already reused all re-arrivals and cost barely changed at 24 or 32 slots; at N=10, reuse failures were resolved at 24, with no further improvement at 32.The pooled saturation rule classified adjacent slot increases as saturated, but did not establish that the additional N=10 gain was absent or that 16 slots are universally optimal.Source: D. Conditions under Which the Gain… · pp. 14–15

E. Compile Cost and Memory Usage

Configuration changes carry compile-time and memory costs in addition to execution effects. The authors recommend diagnosing reuse and bucket behavior, searching and validating on the same input plan, and applying the procedure within the study’s workload and system scope.

E. Compile Cost and Memory Usage

  • One compilation took about 7–8 minutes, while the study did not combine compilation and execution into a monetary cost or measure a break-even point.The authors treat compilation cost as distinct from per-run cost.Source: E. Compile Cost and Memory Usage · p. 15, Table XIV · p. 16
  • Each KV slot reserves 1.125 GiB of capacity, or 0.28125 GiB per device across four devices; slot 32 used more memory without clear execution-cost improvement over slot 16 at N=6 and N=8.The effect of added memory on co-located models or instances was not measured.Source: E. Compile Cost and Memory Usage · p. 15, Table XV · p. 16

F. Application Procedure and Scope

  • The application procedure diagnoses baseline reuse failures, added prefill work, and per-step request-count and bucket distributions before comparing simulator predictions and validating the selected configuration on new inputs.It also checks whether added slots still reduce execution cost after reuse failures are resolved, considering memory and compile cost if further gains are small.Source: F. Application Procedure and Scope · p. 15
  • The findings are limited to the measured model and stack, TraceLab-based waiting times, and one re-arrival per session; repeated tool calls, longer contexts, other models, and loads selecting the top bucket may behave differently.The configuration search included exploratory N=10, and excluding that concurrency changed the selected configuration.Source: F. Application Procedure and Scope · p. 15, A. Limitations · pp. 15–16

A. Limitations

  • The study used one model instance and specified software versions with eager attention; it did not measure larger-model cost changes or memory effects on co-located instances.Other attention implementations may use different KV allocation and eviction units, so the observed reuse-termination conditions do not apply directly.Source: A. Limitations · p. 16
  • The cost channels are not direct measurements of pure device execution time, and the simulator’s cost-ratio validation does not validate cumulative waiting time.The effect of restored reuse on other sessions’ waiting time was not directly confirmed; the original search’s per-candidate files were not preserved, though a rerun reproduced its selected configuration and evaluation ratio.Source: A. Limitations · p. 16

B. Conclusion

The conclusion links configuration-dependent cost to interacting execution mechanisms and reports that simulator-guided selection reduced cost most when baseline KV reuse was poor. It also records limits in measurement precision, eviction attribution, and direct confirmation of downstream delay.

B. Conclusion

  • The experiments support links from KV survival to additional prefill and from prefill to other sessions’ decode, but the slot intervention’s reduction in their delay was not directly confirmed.The bucket-grid intervention confirmed that bucket configuration caused the padding difference; the slot intervention increased reuse and reduced prefill computation.Source: B. Conclusion · pp. 16–17
  • At N=8 with large reuse loss, increased batch_size saved 8.25%, and the configuration that also changed the grid saved 9.72%; gains were small when reuse was mostly retained.The simulator predicted confirmatory cost ratios with errors of 0.58–0.76 percentage points.Source: B. Conclusion · p. 17
  • Padding ratio alone does not establish lower execution cost; the study recommends evaluating computation and execution time alongside waiting time and memory use.The method selects compile-time KV-slot and bucket settings for the workload, without excluding runtime KV-management or scheduling improvements.Source: B. Conclusion · p. 17

S1. TOLERANCE SENSITIVITY AND NO-TREATMENT REPEATED MEASUREMENTS

  • In the three-run summed unit used for decisions, the tolerance covered the 95th percentile of channel-ratio differences at both N=6 and N=8.The sensitivity check maintained channel agreement as default τ0 varied from 0.005 to 0.05, but did not validate the whole tolerance rule.Source: S1. TOLERANCE SENSITIVITY AND… · p. 18
  • Common-bucket costs were within between-session variation on recompilation, but the grid-change comparison at bucket 8 exceeded its threshold by 0.01 ms.The recheck’s variation range covered sessions within one day, not variation across days, so invariance of common costs was not fully established.Source: S1. TOLERANCE SENSITIVITY AND… · pp. 18–19

S3. ATTRIBUTION OF THE ONE ADDITIONAL EVICTION IN THE SEQUENTIAL EXPERIMENT

  • The extra eviction matched a dummy-block request from the same KV pool, which was directly observed in 10 runs; the authors do not generalize its critical request count to other pool sizes or compositions.In eviction runs, exactly one evicted slot was never reallocated; the base simulator did not model this path.Source: S3. ATTRIBUTION OF THE ONE ADDITIONAL… · pp. 19–20
  • At N=7, the preregistered criteria did not confirm a padding difference: only two of six blocks were above the band, and the overall ratio remained inside it.The block dispersion also reflects differing input plans, so it is not interpreted as same-input no-treatment variation.Source: S3. ATTRIBUTION OF THE ONE ADDITIONAL… · p. 20

S5. DECOMPOSITION OF CONTRIBUTIONS TO THE PADDING REDUCTION

The section decomposes the padding-ratio reduction from changing the bucket grid at N=6, separating a per-step slot count from a two-factor allocation of the ratio change. The reported reduction is specific to this paired comparison and does not measure device-time savings.

S5. DECOMPOSITION OF CONTRIBUTIONS TO THE PADDING REDUCTION

  • For the paired N=6 comparison, identical step-count distributions meant the grid change affected only n=5 and n=6, switching each from bucket 8 to 6 and removing two padding slots per step.This premise was confirmed for this comparison and is not generalized to other input plans or tool-wait conditions.Source: S5. DECOMPOSITION OF CONTRIBUTIONS TO… · p. 21
  • The two-factor Shapley decomposition allocates the padding-ratio drop across request counts by averaging both application orders; it is not an independent validation of the per-step method.The per-step share uses padding slots removed at each request count and does not account for a changed denominator.Source: S5. DECOMPOSITION OF CONTRIBUTIONS TO… · p. 21
  • Across 1,288 decode steps, the padding ratio decreased from about 0.2324 to 0.0574, a drop of 0.1749263916.The total padding slots fell from 1,525 to 307; the Shapley contributions re-aggregated to the total drop.Source: S5. DECOMPOSITION OF CONTRIBUTIONS TO… · p. 21, Table S7 · p. 21
  • Although 80.9% of the padding-ratio drop occurred at n=6, that share is not interpreted as a share of device-time savings because bucket time costs differ.The n=6 and n=5 step counts were 494 and 115, respectively, with the same two-slot reduction per step.Source: S5. DECOMPOSITION OF CONTRIBUTIONS TO… · p. 21

S6. PER-REPLICATE DECOMPOSITION OF THE VALIDATION COMPARISONS AND SAVINGS RATES

The supplementary analyses clarify how validation cost ratios are aggregated, check selection sensitivity to concurrency and scoring choices, and test the effect of adding the dummy-block eviction rule to the simulator.

S6. PER-REPLICATE DECOMPOSITION OF THE VALIDATION COMPARISONS AND SAVINGS RATES

S8. SENSITIVITY OF THE CONFIGURATION SELECTION TO THE SET OF N AND THE SCORE DEFINITION

S9. CHANGE WHEN THE DUMMY BLOCK RULE IS INCLUDED IN THE SIMULATOR

Loading 2609.34663v1…