PaperlayerView exact revision on arXiv

Source-linked AI summary

Jumping the Line: Exploiting Length Predictions in LLM Scheduling

Yuyang Dai, Rana Shahout, Mahmood Sharif

arXiv:2610.03430v1cs.AI

TL;DR

LLM schedulers use predicted output lengths because actual lengths are unknown when requests arrive. JIL optimizes an adversarial suffix to make a scheduler’s probe underestimate length; attacked requests complete up to 1.53× faster on average, while response utility varies across models and tasks.

  • Problem

    LLM serving schedulers use predicted output lengths because actual response lengths are unknown at request arrival, leaving user-controlled prompts as inputs to the scheduling signal.

    Source: §1 · pp. 1–2

  • Method

    JIL optimizes an adversarial suffix against TRAIL’s lightweight length probe so the scheduler underestimates a request’s output length and assigns it higher priority.

    Source: §1 · p. 2

  • Results

    Attacked requests complete up to 1.53× faster on average, while response utility varies across models and tasks and predicted-length reductions substantially exceed changes in actual output length.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Coarse grouping of length predictions reduces JIL’s scheduling advantage and mitigates benign-request delays.

    Source: §5 · p. 9

    Suffix construction assumes white-box access to the target model and length probe, with model and scheduler parameters fixed.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Efficient request scheduling is increasingly important for reducing completion time in large language model (LLM) serving. Size-based policies such as Shortest Job First prioritize shorter requests, but output lengths are unknown before generation, so practical schedulers rely on predicted lengths. We introduce JIL, an attack on prediction-based LLM schedulers that manipulates the scheduling signal to obtain higher priority and reduce completion time. Using TRAIL as a case study, JIL optimizes an adversarial suffix that causes a lightweight output-length probe to underestimate a request's length. We evaluate JIL on two datasets and four LLMs across varied request profiles and deployment configurations. JIL reduces predicted output lengths by up to 83.4 percent, and adversarial requests complete up to 1.53 times faster on average in end-to-end serving experiments. The reduction in predicted length is substantially larger than the change in actual output length, revealing a mismatch between the scheduler's estimate and the request's realized size. Response utility varies across models and tasks, exposing a trade-off between scheduling advantage and response quality. We also evaluate scheduler-side defenses and find that grouping length predictions into coarse intervals reduces JIL's scheduling advantage and mitigates delays to benign requests.

1 INTRODUCTION

LLM schedulers use predicted output lengths to prioritize requests because actual output size is unknown at arrival. JIL manipulates that prediction with an adversarial suffix, gaining priority while the request can still generate a long response.

Motivation

  • Because output length is unknown when a request arrives, LLM serving systems predict its size from the prompt to guide scheduling.Size-based policies prioritize shorter requests, while FCFS can let long-running requests delay shorter queued requests.Source: §1 · p. 1

Attack surface

  • The attack exploits user control over prompts: the scheduler can predict a short request even when the request still generates a substantially longer response.This predicted-versus-realized size mismatch differs from attacks that increase computational demand.Source: §1 · p. 2

JIL

  • JIL appends an adversarial suffix that makes the scheduler underestimate output length, increasing priority without changing the serving system or other users’ requests.The attack optimizes the scheduler-facing prediction and evaluates whether suffixes preserve response length and quality and transfer across models.Source: §1 · p. 2

Evaluation

  • Across four LLMs and varied request distributions, JIL lowers predicted lengths, moves attacked requests earlier, and yields up to 1.53× faster average completion in end-to-end serving.The prediction reduction is substantially larger than the change in actual output length; response quality varies across models and tasks, and the study also evaluates transfer and scheduler-side defenses.Source: §1 · p. 2

2 RELATED WORK

Prior work uses model-derived estimates to schedule LLM requests and studies adversarial or resource-consumption attacks, but this paper focuses on manipulating scheduler-facing length predictions. It also distinguishes that problem from content-safety defenses and ordinary prediction errors.

2.1 LLM SERVING AND PREDICTION-BASED SCHEDULING

  • TRAIL, LTR, and other systems use predicted request cost or length to approximate size-based scheduling; TRAIL estimates length from target-model representations and uses an SRPT-style scheduler.The paper studies TRAIL because user-controlled input determines a representation that a learned probe maps to a scheduler-used estimate.Source: §2.1 · pp. 2–3

2.2 ADVERSARIAL AND RESOURCE-CONSUMPTION ATTACKS ON LLMS

  • Unlike resource-consumption attacks that increase generated demand, this attack targets the scheduler-facing length predictor to create a mismatch between predicted and actual cost.Prior adversarial suffix work primarily targets generative behavior, while this work tests effects on scheduling position and co-scheduled benign traffic.Source: §2.2 · p. 3

2.3 DEFENSES AND ROBUST SCHEDULING

  • Existing adversarial-suffix defenses target unsafe completions, and robust-scheduling work assumes ordinary prediction errors rather than deliberate input manipulation.The paper examines whether changing request text can mislead a learned length predictor and whether scheduler-side defenses can prevent the resulting priority gain.Source: §2.3 · p. 3

3 METHODOLOGY

The methodology models a scheduler that assigns priority using predicted response length, then optimizes suffixes against that prediction. Evaluation measures prediction changes, serving behavior, and response utility under an explicit attacker capability model.

3.1 SYSTEM AND THREAT MODEL

  • The system predicts response length from pooled transformer hidden states, and requests with smaller predicted service times receive higher priority.The attacker can append a suffix and has white-box access to the target model and length probe during construction; model, probe, policy, and other requests remain fixed.Source: §3.1 · pp. 3–4

3.2 JIL ATTACK

  • JIL searches for a suffix that minimizes the predicted response length of the request with the suffix appended.The attack uses gradients through the model and probe to select and evaluate candidate replacement tokens.Source: §3.2 · p. 4

3.3 EVALUATION CRITERIA

  • Evaluation connects the attack objective to serving behavior and response quality, measuring prediction change, service times and completion positions, and task-specific utility.Utility is judged response quality for open-ended generation and answer accuracy for multiple-choice tasks; output characteristics include repetition and malformed-output rates.Source: §3.3 · pp. 4–5
  • Prediction manipulation is measured by absolute change and per-request relative reduction, with larger relative reduction indicating a stronger decrease from the clean prediction.The absolute change is negative when the suffix lowers predicted length.Source: §3.3 · p. 4
  • Paired clean and attacked executions compare group-mean time to first token and completion latency; speedup above one indicates faster service under attack.Target and benign requests are summarized separately across workload-level ratios.Source: §3.3 · p. 4
  • Completion position tracks each request’s normalized rank in the workload, with negative paired changes indicating movement toward the front of the completion order.Changes are averaged by request group and workload, then summarized across workloads.Source: §3.3 · pp. 4–5
  • Benign-request delays are summarized by the fraction of requests slowed under attack and the 95th percentile of their slowdown factors.The measures describe how widely delays occur and the upper tail of their relative magnitude.Source: §3.3 · p. 5

4 EXPERIMENTS

Experiments test JIL’s prediction manipulation, serving effects, utility, transfer, and defenses across several models and request settings. JIL shifts target scheduling outcomes, while benign-request effects and response utility vary with the evaluation setting.

4.1 EXPERIMENTAL SETUP

  • The evaluation covers four LLMs, open-ended and multiple-choice requests, and paired TRAIL/vLLM serving workloads at 16 and 64 sequence slots.Prediction and utility comparisons use request-level paired bootstrap confidence intervals; scheduling summaries use five paired workloads.Source: §4.1 · pp. 5–6

4.2 RQ1: PREDICTION MANIPULATION

  • On Alpaca, JIL produces large prediction reductions across models; UltraChat reductions vary more by model, and mean predictions fall on both multiple-choice benchmarks for every model.Qwen2.5-7B’s UltraChat median is near zero, with larger reductions concentrated in a subset of requests.Source: §4.2 · p. 6, Figure 1 · p. 6

4.3 RQ2: SCHEDULING IMPACT

  • Targets complete faster in every paired workload at both capacities, moving earlier in completion order as benign requests move later.With 16 slots, target completion speedups are larger on average; request-level benign delays are more frequent and have a larger upper-tail slowdown than with 64 slots.Source: §4.3 · pp. 6–7, Figure 2 · p. 6

4.4 RQ3: OUTPUT LENGTH AND RESPONSE UTILITY

  • Utility losses vary by model and task: judges favor clean UltraChat responses for every model, while Mistral-7B has larger multiple-choice accuracy decreases.Mistral-7B’s mean UltraChat score decrease has a confidence interval spanning zero.Source: Figure 3 · p. 7, §4.4 · p. 8

4.5 RQ4: ATTACK-STRENGTH TRADE-OFF?

  • At 16 slots, all early-stop variants retain target completion speedups above one in every workload, but each has a lower Alpaca reference win rate than the full attack.The 20-token threshold has the smallest mean target speedup; the smallest benign tail slowdown occurs at the intermediate threshold.Source: §4.5 · p. 8, Figure 4 · p. 8

4.6 RQ5: CROSS-MODEL TRANSFERABILITY

  • Prediction manipulation transfers between Llama-3-8B and Mistral-7B in both directions, with larger mean and median reductions when Mistral-7B receives Llama-3-8B suffixes.Transferred suffixes reduce generated output lengths too, but relative output-length decreases are smaller than prediction reductions.Source: §4.6 · pp. 8–9, Table 2 · p. 8

4.7 RQ6: DEFENSES

  • Prediction floors and coarser grouping reduce target completion advantages and benign tail slowdowns, but floor settings shift the frequency of benign delays differently.For grouping, the widest interval minimizes target advantage and tail slowdown, while the intermediate interval minimizes the share of slower benign requests; grouping effects on targets are larger at 16 than 64 slots.Source: §4.7 · p. 9, Table 3 · p. 9

5 CONCLUSION

The conclusion presents JIL as an attack that lowers predicted lengths to gain scheduling priority, while actual response lengths remain substantial. It highlights varying utility effects, cross-model transfer, and scheduler-side mitigations.

5 CONCLUSION

  • Across four LLMs, JIL lowers length predictions while open-ended responses retain substantial length, and paired serving experiments show faster target completion alongside increased benign tail delays under constrained capacity.The conclusion frames sensitivity of learned predictions to user-controlled inputs as a scheduling fairness issue.Source: §5 · pp. 9–10
  • JIL’s utility losses vary across models and tasks, and early stopping retains scheduling gains while further lowering Alpaca win rates.Source: §5 · p. 9
  • Suffixes transfer between Llama-3-8B and Mistral-7B, while prediction floors and coarse grouping reduce target advantages; coarse grouping also reduces benign tail slowdown and delayed-request share.Source: §5 · p. 9

AI USE STATEMENT

The statement describes AI-assisted manuscript review and coding work, alongside author review and responsibility. It reports checks used to validate the experimental pipeline and recorded results.

AI USE STATEMENT

  • AI tools assisted with grammar checking and presentation review, including identifying fragments, dangling words, discontinuities, and formatting inconsistencies.Source: AI USE STATEMENT · p. 10
  • Coding agents assisted with code development, and the authors reviewed the generated code.Source: AI USE STATEMENT · p. 10
  • The authors checked request matching, sample counts, and consistency of recorded measurements with reported results, and take full responsibility for the manuscript.Pipeline validation also included syntax checks, small-scale smoke tests, and inspection of logs and saved outputs.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The ethics statement bounds the study to configured TRAIL/vLLM instances and established benchmark requests. It notes dual-use risks and recommends authorized, controlled follow-up testing.

ETHICS STATEMENT

  • Serving experiments use researcher-configured TRAIL/vLLM instances and requests from established benchmarks, with target and benign requests serving as experimental workload roles.Source: ETHICS STATEMENT · p. 10
  • The attack could give an adversary preferential service and increase other users’ waiting times in shared deployments.Source: ETHICS STATEMENT · p. 10
  • The statement reports evaluation of prediction floors and coarse grouping and calls for follow-up security testing in authorized deployments with controlled workloads.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The reproducibility statement maps the paper’s supplementary material to its experiments and reports a release of implementation details. The supplement proceeds from experimental protocol to prediction, scheduling, utility, and defense analyses.

REPRODUCIBILITY STATEMENT

  • The paper releases implementation details and identifies saved request-level records plus source, checkpoint, and input hashes for tracing reported results.The statement also maps supplementary appendices to transfer, scheduling controls, utility, attack variants, and defenses.Source: REPRODUCIBILITY STATEMENT · p. 10

A EXPERIMENTAL PROTOCOL AND MEASUREMENTS

  • The supplementary experiments move from request construction and response measurements through prediction and transfer, then scheduling controls, utility, attack variants, and defenses.The protocol links prediction reductions to generated lengths and cross-model transfer before examining scheduling and response outcomes.Source: A EXPERIMENTAL PROTOCOL AND MEASUREMENTS · p. 13

A.1 MODELS, PROBES, AND EVALUATION REQUESTS

The appendix specifies the probes, requests, measurement procedures, and workload aggregation used to evaluate JIL. Its experiments report prediction reductions across models and datasets, bidirectional suffix transfer, and discrepancies between clean predictions and generated lengths.

A.1 MODELS, PROBES, AND EVALUATION REQUESTS

  • The evaluation covers Llama-3-8B-Instruct, Qwen2.5-7B-Instruct, Mistral-7B-Instruct-v0.3, and Llama-3.1-70B-Instruct across four datasets and request types.Each model has a probe trained on 2,000 Alpaca responses, validated on 500, and evaluated on 200 Alpaca, 200 UltraChat, 100 ARC-Challenge, and 100 HellaSwag requests.Source: §A.1 · p. 13

A.2 PREDICTION AND RESPONSE MEASUREMENTS

  • Response measurements retain each task’s metric, including answer comparisons for Alpaca and UltraChat and parsed-answer accuracy for ARC-Challenge and HellaSwag.The appendix also defines output diagnostics such as repetition, malformed output, and cap flags.Source: §A.2 · pp. 13–14, Table 4 · p. 13

A.3 WORKLOADS AND STATISTICAL CALCULATION

  • Scheduling uses five matched 25-request workloads at 16- and 64-slot capacities, and summarizes repeated workload-level measurements with 95% t-intervals.The runner records prediction, timing, output-length, and completion-rank data; paired bootstrap intervals are used for prediction and utility.Source: §A.3 · p. 14

B.1 RELATIVE REDUCTION ACROSS REQUESTS

  • Alpaca reductions are large across all four models, while UltraChat medians range from 18.2% to 32.9% except Qwen’s 0.6%.Table 5 summarizes mean, median, and threshold rates over 200 paired requests per model–dataset cell.Source: §B.1 · p. 14, Table 5 · p. 15

B.2 SUFFIX TRANSFER BETWEEN LLAMA AND MISTRAL

  • Llama suffixes reduce Mistral predictions by 58.8% on average, versus 25.2% for Mistral suffixes on Llama; Mistral’s own suffixes yield 74.8% versus 58.8% for transferred suffixes.The transfer comparison uses 200 paired requests in both directions; on Mistral, mean generated lengths are 362.83 tokens for its own suffixes and 345.16 for transferred suffixes.Source: §B.2 · p. 15, Table 6 · p. 15

B.3 CLEAN PREDICTIONS AND GENERATED RESPONSES

  • Clean-probe mean bias ranges from −347.2 to −277.5 tokens on UltraChat and −350.1 to −309.1 on Alpaca, with rank correlations of 0.320–0.718 and 0.153–0.477, respectively.The probes were trained on reference-response lengths, while the comparison uses model-generated responses capped at 512 tokens.Source: §B.3 · p. 15, Table 7 · p. 15

B.4 TRAINING ON MODEL-GENERATED RESPONSE LENGTHS

A probe trained on model-generated response lengths is still affected by JIL, although its clean predictions are substantially more accurate than a constant median predictor. In a fixed group of requests, the clean–attacked prediction gap shrinks as generated prefixes grow.

B.4 TRAINING ON MODEL-GENERATED RESPONSE LENGTHS

  • The generated-length-trained probe achieves 48.76-token test MAE versus 174.32 for the constant training-median predictor, with Spearman correlation 0.911 and bin accuracy 42.6%.The test split contains 500 requests, and 19.6% of test responses reach the 512-token cap.Source: §B.4 · p. 16, Table 8 · p. 16
  • With a 20-token suffix optimized in 30 steps, mean prediction falls from 415.65 to 47.83 tokens, while mean generated length shifts from 426.24 to 346.01 tokens.The mean per-request relative reduction is 86.3%, with a median of 90.4%; the attack also changes predictions from the probe trained on generated lengths.Source: §B.4 · p. 16, Table 8 · p. 16

C.2 EVOLUTION WITHIN A FIXED REQUEST GROUP

  • For the same 141 pairs reaching 256 tokens, the clean-minus-attacked prediction gap decreases from 94.98 to 19.92 tokens on raw inputs and from 89.67 to 18.71 on chat-formatted inputs as prefixes grow.The fixed-probe comparison follows the pairs’ responses at positions 0, 64, 128, and 256.Source: §C.2 · p. 16, §C.1 · p. 16, Figure 5 · p. 17

D.1 COMPLETION UNDER THE PAIRED WORKLOADS

Across paired workloads, attacked targets move forward and complete faster, while benign group-mean effects remain near parity but individual delays are more pronounced at lower capacity. Controlled comparisons separate prediction-driven priority changes from suffix and output-length effects, and the response analyses show task- and model-dependent utility changes.

D.1 COMPLETION UNDER THE PAIRED WORKLOADS

  • Target completion speedups exceed one at both capacities, with a larger mean speedup at constrained capacity; target positions improve while benign positions worsen.Benign group-mean completion speedups are close to one, but delays affect a larger fraction of requests and have a larger upper-tail magnitude at 16 slots.Source: §D.1 · p. 17, Table 9 · p. 17

D.2 TEXT, PREDICTION, OUTPUT LENGTH, AND SCHEDULING POLICY

  • With natural output, targets gain completion advantage under both prediction scheduling and FCFS, while fixed-output speedups move closer to one.The signal and text controls separate scheduler-facing prediction changes from suffix effects on generated responses.Source: §D.2 · p. 17, Table 10 · p. 18, Table 11 · p. 18

D.3 FIXED-OUTPUT SIGNAL COMPARISON WITH FCFS

  • With clean text and 512-token outputs, target position contrasts are negative in four of five workloads at 16 slots and all five at 64 slots.The accompanying figure compares position changes against FCFS and relative completion-speedup distributions.Source: §D.3 · p. 18, Table 12 · p. 18, Figure 6 · p. 19

D.4 CHANGES WITHIN THE BENIGN REQUEST GROUP

  • At 16 and 64 slots, target position contrasts of −3.84 and −3.47 percentage points accompany benign contrasts of +0.96 and +0.87 points.Among benign requests, the earlier clean-order half slows more often: 52% versus 34% at 16 slots and 8% versus 4% at 64 slots.Source: §D.4 · p. 19

E.1 OPEN-ENDED RESPONSES

  • Attacked open-ended response means range from 270.76 to 448.14 tokens on Alpaca and 388.95 to 408.11 on UltraChat; utility changes vary by model and task.Alpaca win-rate changes range from −61.0 to −15.0 percentage points, while UltraChat score changes range from −0.80 to −0.18.Source: §E.1 · pp. 19–20, Table 13 · p. 19, Table 14 · p. 20

E.2 MULTIPLE-CHOICE ANSWERS

  • Mean prediction falls in all eight multiple-choice model–benchmark cells, while accuracy changes range from +1 percentage point to −10 points.The increase is for Qwen on ARC-Challenge and the decrease is for Mistral on ARC-Challenge.Source: §E.2 · p. 20, Table 15 · p. 20

E.3 CONCISE-RESPONSE PROMPT BASELINES

  • Both concise-response instructions achieve 181 wins, or a 90.5% win rate, against dataset references on the same 200 Llama-8B Alpaca requests.Their mean lengths are 344.99 and 314.72 tokens, compared with 454.72 for clean requests and 448.14 for JIL.Source: §E.3 · p. 20, Table 16 · p. 20

F.1 STOPPING AT A PREDICTION-REDUCTION THRESHOLD

  • All early-stop variants retain target completion speedups above one, while their benign upper-tail slowdowns and reference win rates are lower than the full attack’s.The intermediate prediction-reduction threshold has the largest mean speedup among the early-stop variants.Source: §F.1 · p. 21, Table 17 · p. 21

F.2 ADDING A NEXT-TOKEN DISTRIBUTION TERM

The appendix evaluates a next-token-distribution objective, prediction floors, and interval grouping, then documents records supporting reproduction of the experiments.

F.2 ADDING A NEXT-TOKEN DISTRIBUTION TERM

  • The tested objective combines predicted length with a next-token distribution term; each search used a 20-token suffix and ran for 30 steps.Candidate scoring decoded the suffix and tokenized the complete prompt–suffix text; generation used a chat template and 512-token cap.Source: §F.2 · p. 21
  • For λ = 0 and λ = 0.05, mean relative prediction reductions were 85.8%, with mean output lengths of 331.50 and 329.89 tokens, respectively.Clean outputs averaged 426.24 tokens; the four configurations used 200 requests each.Source: §F.2 · p. 21, Table 18 · p. 22
  • For λ = 0.5 and λ = 5, mean predictions were 153.21 tokens versus 104.64 clean, with mean per-request reductions of −74.9%.Mean generated length was 415.91 tokens in both configurations.Source: §F.2 · p. 21, Table 18 · p. 22
  • Lower-weight prediction pools accelerated targets and moved them forward at both capacities, while higher-weight pools had mean target completion speedups below one and positive mean position changes.The paired scheduler tests used 16 and 64 slots; Table 19 reports completion speedups, normalized positions, and benign tail slowdowns.Source: §F.2 · pp. 21–22, Table 19 · p. 22
  • Table 20 tracks first-token service and the fraction of requests with slower completion, complementing the group-average completion outcomes.TTFT speedups describe the beginning of service; completion-delay fractions show how frequently total response time increases.Source: §F.2 · p. 22, Table 20 · p. 22

F.3 APPLYING A MINIMUM SCHEDULER PREDICTION

  • At 16 slots, increasing the prediction floor lowered target completion speedup and position gain, while benign upper-tail slowdown also decreased.At 64 slots, target completion speedups were closer across floor settings; floors of 80 and 120 tokens were tested.Source: §F.3 · pp. 22–23, Table 21 · p. 23

F.4 GROUPING PREDICTIONS INTO TOKEN INTERVALS

  • At 16 slots, wider prediction intervals reduced target completion speedup and forward position change; the 128-token interval had the smallest target speedup and benign tail slowdown.At 64 slots, target completion speedups remained similar across widths; tested widths were 32, 64, and 128 tokens.Source: §F.4 · p. 23, Table 23 · p. 24
  • At 16 slots, the 64-token interval had the smallest fraction of slower benign requests, while the 128-token interval had the smallest benign tail slowdown.Table 24 reports first-token speedups and slower-request fractions across the repeated workloads.Source: §F.4 · p. 23, Table 24 · p. 24

Related papers

Loading 2610.03430v1…