PaperlayerView exact revision on arXiv

Source-linked AI summary

Reasoning Language Model Inference Serving Unveiled: An Empirical Study

Qi Li, Junpan Wu, Xiang Liu, Yuxin Wang, Zeyu Li, Zhenheng Tang, Yuhan Chen, Shaohuai Shi, Xiaowen Chu

arXiv:2510.18672v1cs.LGcs.AI

TL;DR

RLLM serving behavior remains underexplored despite serving engines being designed for traditional LLMs. This paper studies RLLM serving and optimization techniques, finding distinct behaviors and mixed optimization effects, with real-world workload results aligned with its findings.

  • Problem

    Serving engines were initially designed for traditional LLMs, and whether RLLMs have distinct serving behaviors remains largely unexplored.

    Source: §1 · p. 2

  • Method

    The authors assess RLLM serving, benchmark it against LLMs, evaluate serving optimizations, and test findings under Gamma-modeled real-world workloads.

    Source: §1 · p. 2

  • Results

    RLLMs differ in serving behavior from LLMs; quantization and speculative decoding improve efficiency with small accuracy compromises, while caching techniques can degrade accuracy or performance for small RLLMs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Optimization effects are not uniform for small RLLMs, and real-world workload evaluation aligns with the reported serving findings.

    Source: §1 · p. 2

    Evaluation is constrained by limited ecosystem support and hardware resources, with broader hardware coverage left for future work.

    Source: B Limitation · p. 18

Abstract

from arXiv · show

The reasoning large language model (RLLM) has been proven competitive in solving complex reasoning tasks such as mathematics, coding, compared to general LLM. However, the serving performance and behavior of RLLM remains unexplored, which may undermine the deployment and utilization of RLLM in real-world scenario. To close this gap, in this paper, we conduct a comprehensive study of RLLM service. We first perform a pilot study on comparing the serving performance between RLLM and traditional LLM and reveal that there are several distinct differences regarding serving behavior: (1) significant memory usage and fluctuations; (2) straggler requests; (3) adaptive running time; (4) domain preference. Then we further investigate whether existing inference optimization techniques are valid for RLLM. Our main takeaways are that model quantization methods and speculative decoding can improve service system efficiency with small compromise to RLLM accuracy, while prefix caching, KV cache quantization may even degrade accuracy or serving performance for small RLLM. Lastly, we conduct evaluation under real world workload modeled by Gamma distribution to verify our findings. Empirical results of real world workload evaluation across different dataset are aligned with our main findings regarding RLLM serving. We hope our work can provide the research community and industry with insights to advance RLLM inference serving.

1 Introduction

This paper studies whether RLLM serving differs from traditional LLM serving, then evaluates serving optimizations and real-world workloads. It reports distinct serving behaviors and finds that optimization effects vary for small RLLMs.

1 Introduction

  • Serving behavior remains largely unexplored for RLLMs, although current engines were designed for traditional LLMs and direct transfer of their optimizations may be suboptimal.The paper identifies efficient private deployment on limited GPU resources as a motivation.Source: §1 · p. 2
  • RLLMs show fluctuating, high KV-cache use, straggler-driven long-tail runtimes, adaptive runtimes, and stronger math but comparable knowledge-task performance versus LLMs.These findings are reported for batched requests.Source: §1 · p. 2
  • Model quantization and speculative decoding improve serving efficiency with small accuracy compromises, while prefix caching and KV-cache quantization can degrade accuracy or performance for small RLLMs.Evaluation under Gamma-modeled real-world workloads across datasets aligned with the paper’s main serving findings.Source: §1 · p. 2
  • The authors introduce ASU to assess accuracy, provider-side, and user-side metrics, and ASU-Perf to benchmark RLLM serving.They position the work as the first dissection of RLLM serving performance.Source: §1 · p. 2

2 Preliminaries

The preliminaries distinguish reasoning-focused models from standard LLMs and outline how LLM serving works. They also define common measures of serving performance.

2 Preliminaries

  • RLLMs target complex multi-step reasoning, often using long chain-of-thought prompting to generate explicit step-by-step reasoning pathways.The passage contrasts these models with standard LLMs that may give more direct answers.Source: §2 · pp. 2–3
  • LLM serving processes prompts during a compute-intensive prefill stage, then generates tokens iteratively during a memory-intensive decoding stage.Continuous batching and paged KV-cache management are described as serving-system approaches to throughput and memory use.Source: §2 · p. 3
  • Serving metrics include TTFT, end-to-end latency, intertoken latency, tokens per second, and requests per second.These capture response timing and system throughput from different perspectives.Source: §2 · p. 3

3 Experimental Settings

The experiments compare general LLMs with their long-CoT-tuned RLLM counterparts across reasoning and knowledge datasets. The ASU framework evaluates accuracy alongside provider- and user-side serving performance.

3.1 Setups

  • Four model scales are evaluated using general LLMs and their long-CoT-tuned RLLM counterparts.The models range from 7B to 70B parameters.Source: §3.1 · p. 3
  • Evaluation uses GSM8K, MATH-500, and AIME-2024 for different math difficulty levels, plus GPQA for knowledge reasoning.The paper describes these as easy, medium, hardest, and knowledge-reasoning evaluations, respectively.Source: §3.1 · p. 3
  • The evaluation uses vLLM and SGLang inference engines and the authors’ ASU-Perf benchmarking suite.ASU-Perf is used throughout the evaluation.Source: §3.1 · p. 3

3.2 The ASU Assessment Framework

  • ASU combines dataset accuracy, provider-side TPS, and user-side TTFVT and end-to-end request running time.TTFVT is used because the framework assumes RLLM reasoning tokens are invisible to users.Source: §3.2 · p. 4

4 Pilot Investigations: Serving LLM v.s. RLLM

Across eight models and multiple inference engines, the pilot found that RLLM serving differs from LLM serving, with task type and difficulty shaping performance and runtime.

Main Results: Results with Different Token Budget

  • For most datasets, token budgets of 4,096–8,192 provide sufficiently good RLLM performance, while larger budgets reduce performance on GPQA and AIME-2024.The evaluation varied budgets from 0.5K to 20K for 14B and 32B RLLMs across datasets.Source: Figure 1 · p. 4, §4 · p. 4

Main Results: Results with Different Batch Size

  • Larger batches reduce RLLM processing time and improve TPS without changing accuracy, but increase average TTFVT.The comparison held the experimental setting constant while varying batch size.Source: §4 · p. 4

Serving Performance and Behaviors

  • RLLMs outperform same-scale LLMs on math, match them on knowledge tasks, and show greater cache fluctuation, stragglers, and adaptive runtime.The comparison covers serving traces as well as accuracy across task domains.Source: §4 · pp. 4, 6, Figure 2 · p. 5

Main Findings for RLLM Serving Characteristics

  • RLLM memory utilization ranged from 3% to 70% in extreme cases, versus typically below 3% KV-cache usage for traditional LLMs.Long reasoning chains increase KV-cache demand; the engine discards each request's cache when that request completes.Source: §4 · pp. 5–6
  • Batch-arriving requests with uneven difficulty leave a few slow stragglers running, extending batch completion and reducing system throughput.The remaining requests run until they finish reasoning or reach the token budget.Source: §4 · pp. 5–6
  • RLLM runtime varies across datasets and tracks task difficulty, while LLM runtime scales approximately linearly with dataset size despite task-difficulty differences.LLMs show smaller runtime differences across datasets and less sensitivity to task difficulty.Source: §4 · pp. 5–6

5 Observations on RLLM Serving Optimization

The section evaluates serving optimizations while requiring that RLLM accuracy be preserved as much as possible. Outcomes depend on the technique and model size: some improve efficiency, while others reduce efficiency or accuracy for smaller models.

5.1 Is model weight quantization methods effective in boosting RLLM serving?

  • GPTQ-IN4 and FP8 preserve 7B RLLM performance on most datasets with about 3% degradation or better, while GPTQ drops 15–25% on challenging AIME24 tasks.AWQ and L4 preserve performance across datasets but nearly double E2E time and halve throughput.Source: §5.1 · p. 7

5.2 Could KV Cache Quantization Lead to Better RLLM Serving Performance?

  • KV cache quantization accelerates 14B RLLMs with comparable performance, but nearly eliminates 7B performance despite improving metrics such as TTFVT and TPS.The 14B and 32B models retain performance with minimal degradation.Source: §5.2 · p. 7

5.3 Is Prefix Caching Useful for Contributing Efficient RLLM Serving?

  • Prefix caching improves runtime speed and serving metrics for RLLMs sized 14B and above without compromising performance, but increases latency for 7B models.The evaluation compares RLLMs with prefix caching enabled and disabled.Source: §5.3 · pp. 7–8, Figure 5 · p. 8

5.4 Does Speculative Decoding Help to Improve RLLM Serving Performance?

  • Speculative decoding improves RLLM running time across model scales without benchmark-performance degradation, but significantly reduces throughput and worsens TTFVT.The evaluation uses n-gram matching with vLLM.Source: §5.4 · p. 8

6 Applying to Real World Workload

The study tests RLLM serving under streaming, stochastic, bursty workloads modeled with a Gamma distribution. Under these workloads, the serving behaviors observed in the earlier section remain valid across datasets.

6 Applying to Real World Workload

  • RLLMs have higher average KV-cache usage than LLMs, with utilization near 100% for long periods; the resulting queueing can prolong turnaround time.The authors attribute persistent high utilization to accumulated straggler requests.Source: §6 · p. 8, Figure 6 · p. 9
  • Figure 7 reports inference-engine running-request counts for 14B models under a real-world workload; the authors say the reported phenomena hold consistently across different datasets.Source: Figure 7 · p. 9, §6 · p. 9

7 Related Work

The related work reviews research on reasoning models and methods for efficient LLM inference and serving.

7 Related Work

  • Recent RLLMs use test-time scaling to produce long chain-of-thought reasoning, with work targeting complex reasoning tasks such as mathematics and programming.The passage describes long reasoning, exploration, backtracking, and reflection as characteristics of RLLM reasoning.Source: §7 · p. 9

LLM Inference and Serving.

  • LLM serving optimization work spans parameter-memory optimization, request scheduling, dynamic-memory optimization, efficient decoding, system optimization, and model or algorithm optimization.Examples include quantization, KV-cache methods, speculative decoding, and prefill-decoding disaggregation.Source: LLM Inference and Serving · p. 9

8 Conclusion

The paper studies RLLM serving behavior and performance, evaluates existing inference optimizations, and tests its findings under Gamma-distributed real-world workloads.

8 Conclusion

  • RLLMs differ from traditional LLMs in serving behavior, and Gamma-workload evaluation aligns with the paper’s main findings.The study also investigates whether existing inference optimization techniques are valid for RLLM serving.Source: §8 · p. 9

9 Reproducibility Statement

The paper lists resources and experimental settings for reproducibility, states how LLMs were used, and outlines limitations and broader context for RLLM serving.

9 Reproducibility Statement

  • Experiments used publicly accessible datasets (introduced in G.1, with download websites listed in F) and multiple runs on NVIDIA Tesla RTX4090-24GB, RTX A6000-48GB, and NVIDIA A100-PCIE-40GB GPUs, using Python 3.11 and PyTorch 2.5.Model and hyperparameter details are provided in F.Source: §9 · p. 10

A Use of LLMs Statement

  • The authors say LLMs were used only for grammar checking and language polishing after drafting, not for the research process.Source: A Use of LLMs Statement · p. 18

B Limitation

  • Evaluation is limited by current serving-engine support and hardware resources, with broader engine and hardware coverage left for future work.Source: B Limitation · p. 18

C Boarder Impact

  • The authors frame their study as offering perspectives to advance efficient RLLM serving and help democratize access to cutting-edge RLLMs.Source: C Boarder Impact · p. 18

D.1 Reasoning Large Language Models

  • RLLMs use test-time scaling with long chain-of-thought reasoning; the passage describes exploration, backtracking, reflection, and “aha” moments.It also reports LIMO improving AIME performance from 6.5% to 57.1% using 817 curated training samples.Source: §D.1 · p. 18

D.2 LLM Inference and Serving

  • LLM serving methods span memory optimization, scheduling, KV-cache management, decoding, system design, and model or algorithm optimization.Examples include weight quantization, speculative decoding, prefill-decode disaggregation, and Flash Attention.Source: §D.2 · pp. 18–19

D.3 LLM Evaluation

The related work surveys LLM evaluation and serving, while noting that RLLM serving support and targeted optimizations remain limited. It also reviews serving techniques for memory use and GPU utilization.

D.3 LLM Evaluation

  • Prior evaluation work covers holistic language-model assessment, model reports, quantization, knowledge editing, and serving evaluation.Source: §D.3 · p. 19

D.4 Ecosystem Support for RLLM Serving.

  • Current serving systems offer limited RLLM support, and some techniques, including Eagle speculative decoding, lack RLLM compatibility.The authors call for collaboration to address issues identified in their study.Source: §D.4 · p. 19

E An Introduction to LLM Serving

E.1 Serving Performance

  • Serving techniques include iteration-level continuous batching, prefill-decode disaggregation, and paged attention to address GPU utilization and memory waste.The passage illustrates over-allocation with a request using 8 tokens but reserving 2080 token spaces and producing 80 tokens.Source: §E.1 · p. 20

E.2 Serving Metric

The appendix defines serving metrics and reports evaluation details and results for disaggregated prefilling and quantization. It also points to supplementary pilot-study and optimization results.

E.2 Serving Metric

  • The metric overview covers throughput, TTFT, TTFVT, TBT, KV-cache utilization, TPS, RPS, model initialization latency, and end-to-end latency.TTFVT measures time until a token is displayed; RLLM reasoning can make it much larger than TTFT.Source: §E.2 · pp. 20–21

G.1 Details Evaluation Datasets

  • Evaluation uses GSM8K, MATH500, AIME24, and GPQA Diamond, on servers with RTX A6000, RTX 4090, or A100 GPUs.The paper reports use of vLLM 0.8.1 and SGLang 0.4.6.post1.Source: §G.1 · p. 24, §G.2 · p. 24, §G.3 · p. 24

H.1 Can Disaggregated Prefilling Improve RLLM Serving Performance ?

  • Under 1P1D, PD-disaggregation declines RLLM serving metrics versus mixed PD; decoding is the bottleneck, prefill devices are largely idle, and KV-cache transfer adds communication overhead.Source: Table 1 · p. 25, §H.1 · p. 25

I.2 Main Results for Pilot Study

  • The materials report token-budget variation across different datasets and provide full RLLM-versus-LLM serving comparisons for 7B, 14B, 32B, and 70B models in Tables 2–9.Source: §I.2 · pp. 25–26, Figure 8 · p. 26

J.3 Prefix Caching

K Extended Results for Real World Benchmarking

The extended real-world benchmark figures track KV-cache usage and the number of running requests across datasets for 7B and 32B models.

K Extended Results for Real World Benchmarking

  • Figure 12 plots KV-cache usage for 7B models under real-world workloads across datasets.Source: Figure 12 · p. 42
  • Figure 13 plots KV-cache usage for 32B models under real-world workloads across datasets.Source: Figure 13 · p. 42
  • Figure 14 tracks the number of running requests in the inference engine for 7B models under real-world workloads.Source: Figure 14 · p. 43
  • Figure 15 tracks the number of running requests in the inference engine for 32B models under real-world workloads.Source: Figure 15 · p. 43

Related papers

Loading 2510.18672v1…