PaperlayerView exact revision on arXiv

Source-linked AI summary

Sustained 70B-Class AWQ Inference on a Single NVIDIA L20: Throughput, Stability, Energy, and Quality Characterization

Yin Li

arXiv:2609.05420v1cs.DC

TL;DR

70B-class serving is commonly associated with larger or multi-GPU systems, motivating a test of whether one 48GB L20 can sustain a useful quantized workload. The report measures Qwen2.5-72B-Instruct-AWQ with vLLM and AWQ Marlin under fixed workloads, finding a stable, throughput-oriented operating point while leaving quality-retention and broader serving claims unresolved.

  • Problem

    The report addresses limited empirical evidence about what one NVIDIA L20 can sustain when serving a 70B-class AWQ model under fixed, reproducible workloads.

  • Method

    The study empirically characterizes Qwen2.5-72B-Instruct-AWQ on one L20 using vLLM’s AWQ Marlin path, fixed-shape throughput tests, repeated runs, a 24-hour soak, energy estimates, and absolute quality checks.

  • Results

    A single NVIDIA L20 repeatedly and stably served the 70B-class AWQ model as a throughput-oriented endpoint under the tested fixed-shape workload.

  • Takeaways & Limitations

    The evidence supports using this configuration as a narrowly scoped throughput-oriented 70B-class endpoint under the tested workload.

  • Takeaways & Limitations

    The evidence does not support a low-latency chat endpoint or high-concurrency long-context claim.

Abstract

from arXiv · show

Serving 70B-class open-weight language models is usually associated with 80GB accelerators, tensor-parallel multi-GPU systems, or vendor-managed inference profiles. This technical report evaluates whether a single NVIDIA L20 48GB GPU can sustain a useful 70B-class quantized serving workload. We measure Qwen2.5-72B-Instruct-AWQ served with vLLM 0.8.5.post1 and AWQ Marlin on one L20. Under a fixed workload of approximately 512 input tokens and 256 output tokens, a 24-hour concurrency-10 soak completed 36,740/36,740 requests with no request failures and no vLLM CUDA OOM, traceback, or killed-process signatures. The system sustained 108.84 output tokens/s, with p95 time-to-first-token of 6.61s and p95 end-to-end latency of 23.54s. GPU-board power sampled through nvidia-smi produced an estimated 7.92 kWh over the run, corresponding to 0.330 output tokens/J and 1.008 total tokens/J. Repeated fixed-shape runs at concurrency 1, 4, 8, and 16 completed 12/12 runs successfully; the concurrency-16 condition averaged 127.22 +/- 12.68 output tokens/s over three runs. The same AWQ endpoint also produced absolute quality scores of 0.8130 on MMLU, 0.8309 on CMMLU, and 0.8082 on GSM8K, plus 80/80 MT-Bench answer generations and a 60-item 8K LongBench subset. The evidence supports a narrow claim: a carefully configured single L20 can serve Qwen2.5-72B-Instruct-AWQ as a throughput-oriented 70B-class endpoint under the tested fixed-shape workload. It does not prove lossless AWQ quality retention, low-latency interactive serving, broad production SLA coverage, or equivalence to a BF16/FP16 baseline.

1 Executive Summary

This report empirically characterizes a single-L20, 70B-class AWQ serving configuration, combining operational measurements with explicit boundaries on what the evidence does not establish.

  • A single NVIDIA L20 can repeatedly and stably serve a 70B-class AWQ model with short-context, throughput-oriented batching.
  • The contribution is a measurement package covering serving configuration, throughput, variance, 24-hour stability, energy, quality checks, and quality-retention blockers.
  • The supported operating point uses approximately 512 input and 256 output tokens, 24-hour stability at concurrency 10, and repeated short-run throughput near 127 output tokens/s at concurrency 16.
  • Non-claims: The evidence does not establish AWQ losslessness, low-latency chat serving, generalization across models or prompts, official LongBench status, or an MT-Bench judge score.

3 Introduction

The report asks what one NVIDIA L20 can sustain for a 70B-class AWQ model under fixed, reproducible workloads. It answers through empirical systems characterization rather than a new model, quantization method, or serving algorithm.

  • 70B-class models are useful and self-hostable, but serving discussions commonly assume 80GB GPUs, multi-GPU tensor parallelism, or vendor-managed profiles.
  • A 48GB GPU can fit 4-bit 70B weights, but sustained serving also depends on KV cache, runtime buffers, scheduling, batching, kernels, thermal behavior, and tail latency.
  • The central question is what one NVIDIA L20 can actually sustain when serving a 70B-class AWQ model under fixed, reproducible workloads.
  • The study characterizes a constrained deployment point with day-long stability, confidence intervals, energy estimates, and quality checks.
  • The evaluation includes a 24-hour concurrency-10 soak, repeated fixed-shape runs at concurrency 1, 4, 8, and 16, and absolute quality measurements across several benchmarks.

4 Background

The report frames single-L20 70B serving as a constrained systems problem involving memory, batching, kernels, workload shape, and long-run measurement. It uses a short-context AWQ Marlin configuration and fixed-shape tests to characterize throughput, stability, energy, and confidence intervals rather than propose a new algorithm.

  • Background: 4-bit weights can fit a 70B-class model on a 48GB GPU, but KV cache, runtime buffers, scheduling state, tokenization, and fragmentation also consume capacity.Increasing context length or concurrency can move the same deployment from stable to out-of-memory.
  • Background: AWQ Marlin combines vLLM serving with an optimized mixed-precision FP16-by-INT4 kernel, making the checkpoint-and-kernel configuration materially relevant.AWQ protects salient channels identified through activation statistics, while Marlin provides the optimized inference path.
  • Background: The primary service uses a 1024-token context limit, 48 maximum sequences, 4096 maximum batched tokens, 98% GPU-memory utilization, eager execution, and one unit of swap space.The report identifies this as a short-context throughput profile; separate 4096- and 8192-context checks were not the main throughput condition.
  • Background: The fixed-shape benchmark uses approximately 527 server-side prompt tokens, 256 output tokens, streaming responses, and 128 unique prompts.The workload also records 498 raw tokenizer prompt tokens and reuses the same shape for concurrency experiments.
  • Background: The 24-hour soak runs at concurrency 10, while repeated short runs test concurrencies 1, 4, 8, and 16 with three repeats per condition.Repeated-run confidence intervals use run-level summaries with n = 3 per concurrency.
  • Background: The measurements are repo-native fixed-shape serving results, not an MLPerf submission with LoadGen, a Poisson server scenario, audited latency constraints, or an official quality target.Energy is estimated by trapezoidal integration of sampled GPU-board power rather than wall-power measurement.

6 Serving Results

The serving evaluation shows a throughput peak at concurrency 16, successful repeated fixed-shape runs, and stable 24-hour operation at concurrency 10. Energy is reported as a GPU-board estimate rather than a full-system measurement.

  • Fixed-Shape Screening Sweep: Concurrency 16 is the highest-throughput screened point, while concurrency 24 regresses and sharply increases tail latency.The sweep uses approximately 512 input tokens and 256 output tokens.
  • Repeated Fixed-Shape Runs: All twelve repeated fixed-shape runs completed successfully with no vLLM OOM or error signatures.The repeated conditions were concurrency 1, 4, 8, and 16, with three runs per condition.
  • Repeated Fixed-Shape Runs: 127.22 output tokens/s was the repeated concurrency-16 mean, close to the earlier 127.70 output tokens/s result.Concurrency 8 was especially tight at 93.26 ± 0.23 output tokens/s, while concurrency 16 showed wider variance.
  • Twenty-Four-Hour Soak: The 24-hour concurrency-10 soak completed normally with zero CUDA OOM, OutOfMemory, Traceback, ERROR, or Killed signatures.The day-long result provides sustained-stability evidence under the exact tested workload.
  • Twenty-Four-Hour Soak: 108.84 output tokens/s was the 24-hour throughput, matching the short concurrency-10 screening result of 108.70 output tokens/s.This agreement supports the stability of the measured throughput under the tested workload.
  • GPU-Board Energy: GPU-board energy is estimated from sampled power using trapezoidal integration and excludes wall-power components such as CPU, memory, fans, and cooling.The estimate is intended for regression testing and rough deployment planning, not full data-center accounting.

7 Quality Characterization

The quality characterization documents absolute AWQ results while leaving quality-retention and MT-Bench judge comparisons unresolved. The reported LongBench measurements are a strict 60-sample subset, and the local BF16/FP16 baseline is infeasible on one L20.

  • Absolute Quality Measurements: Absolute AWQ quality measurements are reported, but the report does not establish lossless AWQ quality retention.The measurements document the tested candidate and can detect severe quality collapse, but no BF16/FP16 comparison is included.
  • LongBench Subset: The 60 generated LongBench samples received task-specific scores of 38.67% on multifieldqa en, 30.53% on hotpotqa, 12.90% on multi news, 12.08% on gov report, 2.76% on lcc, and 0.00% on passage count.These stricter task-metric results remain a subset result rather than an official leaderboard run.
  • Quality Evaluation Boundaries: The three MMLU failures were context-limit validation failures caused by prompts exceeding the 1024-token short-context profile, not CUDA OOMs.The failure mode is tied to the configured service profile.
  • Blocked Comparisons: A local 72B BF16/FP16 baseline is infeasible on one L20 because the weights require roughly 144GB before KV cache and runtime overhead, versus about 46GB of GPU memory.The report also avoids self-judged MT-Bench scoring with the AWQ candidate.
  • Blocked Comparisons: MT-Bench currently reports answer generation only, not a judge score, and LongBench is a 60-item 8K subset rather than an official leaderboard run.These constraints limit the scope of the quality characterization.

8 Long-Context Capacity Checks

The long-context checks establish single-concurrency capacity at 4K and 8K without OOM, but they do not evaluate high-throughput long-context serving. A separate service profile, workload matrix, and soak are required for that claim.

  • Long-Context Capacity Checks: The 4K and 8K checks show that the model can handle longer prompts at concurrency 1 without OOM.These are capacity checks rather than high-throughput serving results.
  • Long-Context Capacity Checks: High-concurrency long-context serving remains unsupported because it requires a separate service profile, workload matrix, and soak.The fixed-shape throughput findings do not establish this broader operating condition.

9 External Positioning

The report positions its single-L20 AWQ result as a workload-specific comparison rather than a general hardware ranking. Repeated-run evidence supports stable throughput, while the profile remains unsuitable for low-latency interactive serving.

  • Comparative positioning: This single-L20 AWQ Marlin configuration is competitive with published single-GPU figures under the matched 512-input/256-output/10-concurrent workload.The comparison is directional because model, quantization, kernel, scheduler, sequence lengths, concurrency, sampling, and measurement method differ across systems.
  • Comparative positioning: Qwen official speed references use different operating points and should not be directly compared with the report’s aggregate concurrent-serving throughput.The cited official context concerns batch-1 and long-output measurements rather than the aggregate workloads studied here.
  • Comparative positioning: NVIDIA NIM lists optimized Qwen2.5 72B Instruct FP8 L20 profiles using 4 or 8 L20 GPUs, whereas this report evaluates a single-L20 AWQ Marlin deployment.The report therefore presents a different deployment point from the conservative FP8 profile matrix.
  • Operational boundary: At concurrency 16, repeated runs averaged 11.91s p95 TTFT and 34.95s p95 end-to-end latency for fixed 256-token outputs.The authors characterize this profile as useful for batch generation, asynchronous agents, offline summarization, and internal processing, but not sub-second interactive applications.
  • Operational boundary: The repeated-run campaign completed all runs successfully, with concurrency 16 showing wider throughput variance and concurrency 8 remaining highly stable.The 24-hour soak supplies separate evidence of long-run operational stability at concurrency 10.
  • Future comparisons: The report identifies BF16/FP16 quality retention, judge scoring, runtime and quantization ablations, and broader model replication as priority additions.These additions would address comparative evidence rather than merely extend the existing deployment characterization.

11 Reporting Checklist Against Strong Technical Reports

The report audits itself against stronger technical-report practices and concludes that its central deployment claim is credible but its comparative evidence remains incomplete. The main gaps concern controlled baselines, ablations, judge scoring, and broader replication.

  • Reporting standard: The report compares its evidence package with technical reports that combine central claims, broad evaluations, failure analysis, and release artifacts.The cited examples emphasize capability and limitation evaluation, broad suites, architecture and efficiency reporting, and benchmark standards.
  • Self-audit: Table 11 audits the report against common expectations for high-quality technical and benchmark reports.Its stated purpose is to prevent overclaiming and make remaining work explicit.
  • Assessment: The report is credible as a narrow technical report but is not yet a top-tier research paper because its strongest missing evidence is comparative.The stated gaps are controlled full-precision retention, official judge scoring, runtime and quantization ablations, and broader replication.

12 Threats to Validity

The report’s validity is bounded by its single model and quantization path, fixed short-context workload, incomplete quality baseline, limited statistical sampling, and GPU-board-only energy accounting. Implementation artifacts document the experiments but do not remove these scope constraints.

  • Model and serving path: The main results use only Qwen2.5-72B-Instruct-AWQ with AWQ Marlin, so other models, quantization formats, and kernels may behave differently.This limits generalization beyond the tested model and serving path.
  • Workload validity: The primary workload fixes approximately 512 input tokens and 256 output tokens, unlike traffic with variable lengths, retries, tool calls, or burstiness.Results therefore characterize a controlled workload rather than arbitrary production traffic.
  • Context-length scope: The highest-throughput results use max model len=1024 and should not be generalized to high-concurrency 8K, 16K, or 32K serving.Separate 4096- and 8192-context single-concurrency checks do not change the main throughput profile.
  • Quality validity: Absolute AWQ quality scores do not establish retention against full precision because no matched BF16/FP16 baseline is available.The report also generated 80/80 MT-Bench answers without obtaining an external judge score.
  • Energy scope: Reported energy is GPU-board energy sampled through nvidia-smi rather than whole-system wall energy.This constrains interpretation of the energy measurements for end-to-end deployment accounting.
  • Statistical scope: Three repeated runs per condition can detect gross instability but do not provide broad statistical confidence across days, hosts, driver versions, and ambient conditions.The report’s command and artifact paths document the repeated-run and soak procedures used for this evaluation.

14 Conclusion

The report concludes that one NVIDIA L20 can stably serve Qwen2.5-72B-Instruct-AWQ for a controlled, short-context throughput workload. It presents this as a narrow deployment result, not evidence for quality retention, interactive latency, broad production coverage, or full-precision equivalence.

  • Conclusion: 108.84 output tokens/s over a 24-hour concurrency-10 soak, 12/12 successful repeated runs, and 127.22 ± 12.68 output tokens/s at concurrency 16 support the deployment result.Together, these measurements combine long-run stability with repeated fixed-shape throughput evidence.
  • Conclusion: The supported claim is that single-L20 70B-class AWQ serving is feasible and stable for a controlled throughput-oriented workload.Stronger claims require baseline and ablation experiments covering quality, latency, context length, and production generality.
Loading 2609.05420v1…