PaperlayerView exact revision on arXiv

Source-linked AI summary

How Fast Can Reward Models Score? A Systems Study of C++ and PyTorch Inference Runtimes for RLHF

Venkata Naga Sai Vishnu Rohit Pulipaka, Anish Katta, Deva Rohit Reddy Peddireddy

arXiv:2607.19712v1cs.LG

TL;DR

Reward-model scoring is repeatedly invoked in RLHF, yet alternative inference backends are rarely evaluated systematically. This paper benchmarks a native C++ ONNX Runtime engine against PyTorch and FastAPI on CPU and GPU, finding decisive CPU gains but a GPU advantage for torch.compile.

  • Problem

    RLHF reward-model serving lacks systematic evidence comparing inference backends beyond default PyTorch eager mode and torch.compile, despite scoring’s repeated impact on training time.

  • Method

    The paper benchmarks a native C++ ONNX Runtime reward-scoring engine against PyTorch eager mode, torch.compile, and FastAPI on CPU and GPU using independent repeated launches.

  • Results

    On CPU, the C++ engine beat every PyTorch baseline; on GPU, it beat PyTorch eager mode and FastAPI, while torch.compile led.

  • Takeaways & Limitations

    The study identifies ONNX Runtime rather than C++ as the CPU speedup source and length-aware batching as a throughput lever on GPU.

  • Takeaways & Limitations

    The benchmarks measure inference in isolation on one development machine, so translation to end-to-end RLHF training time and other hardware remains unmeasured.

Abstract

from arXiv · show

In RLHF pipelines, reward scoring blocks policy updates. Slow scoring bottlenecks the entire loop, since no update runs until every rollout gets a score. And yet most setups just default to PyTorch eager mode or torch.compile, no one checks if that's actually fastest. Scoring itself is small. Rollout generation eats far more of a typical RLHF step. But scoring and generation fight over the same CPU and GPU resources, so a faster scoring engine doesn't shrink step time on its own. It mainly frees up capacity generation can use instead. We built a native C++ inference engine on ONNX Runtime. First step: confirm correctness. Output matched the PyTorch reference to 5.7 x 10^-6 on CPU and 4.2 x 10^-3 on GPU, close enough to trust. Then we tested it against PyTorch eager mode, torch.compile, and FastAPI, on both CPU and GPU. CPU was decisive. Our engine beat every baseline, confidence intervals didn't even overlap. GPU gave a different view: we beat PyTorch and FastAPI, but torch.compile came out ahead. Further testing traced the speedup to ONNX Runtime itself, not C++ as a language. And batching strategy mattered more than either the language or the runtime choice, more than we expected. The results are from repeated, independent runs, since single runs just aren't reliable enough to trust.

1 Introduction

Reward-model inference runs at every RLHF training step, making its latency a direct contributor to multi-day training time, yet most pipelines rely on PyTorch defaults. This paper evaluates a custom C++ engine against those defaults and finds hardware- and deployment-dependent performance differences.

  • The study builds a custom C++ reward-model inference engine, verifies mathematical parity with PyTorch, and benchmarks it against PyTorch eager mode and torch.compile.The engine uses ONNX Runtime, a production-serving approach whose value for RLHF had limited prior data.
  • On CPU, the ONNX Runtime C++ engine consistently outpaced every PyTorch baseline, while on GPU torch.compile beat it at the median.On GPU, the C++ engine still beat plain PyTorch and FastAPI, and the torch.compile advantage was statistically significant.
  • Fixed-length padding reduced throughput by 5 to 8 times on CPU and 3.5 to 4 times on GPU compared with grouping similar-length requests.Length-aware grouping recovered the loss only on GPU because the CPU environment lacked meaningful batch parallelism.
  • Independent process launches were used for every metric because substantial single-machine run-to-run noise could falsely reverse system rankings.The methodology was intended to prevent unreliable conclusions from single runs.
  • Reward-model scoring occurs at every training step before policy updates, so per-call latency directly increases RLHF training wall-clock time.The scoring step processes massive batches of candidate responses and must finish before the policy update.

2 Related Work

Prior RLHF work has emphasized training mechanics and generation, while reward-model execution speed has received little attention. This study situates its findings within inference optimization and performance-measurement research, showing that execution mode—not programming language—is the key performance divide.

  • RLHF Systems Gap: Reward-model execution speed has largely been ignored in RLHF literature, despite generation commonly being identified as the dominant step cost.Existing work focuses on sampling, policy stability, reward hacking, and actor generation rather than scoring.
  • Inference Optimization: ONNX graph runtimes, torch.compile, and serving frameworks address inference overhead, compilation, batching, and scheduling, primarily for production or generative workloads.The related systems literature largely targets production inference or high-concurrency generation rather than reward scoring.
  • Study Contribution: The true performance divide is graph execution versus eager mode, with isolation tests showing that gains come entirely from execution mode rather than C++ versus Python.The same ONNX session produced gains when run through Python instead of C++, attributing the improvement to the execution mode.
  • Measurement Methodology: Means and confidence intervals across independent process launches replace single-run measurements to control variables that can falsely favor slower systems.The methodology follows prior performance-measurement work on confounders such as memory layout and OS scheduling.

3 Methodology · 3.1 Engine and Models · 3.2 Correctness Validation

The methodology introduced a native C++ reward-scoring engine built around ONNX Runtime and validated it against PyTorch before benchmarking. Validation showed small CPU and larger-but-expected GPU score differences caused by floating-point accumulation order.

  • 3.1 Engine and Models: The C++ engine uses ONNX Runtime for tokenization, batching, and postprocessing previously handled in Python.This design shifts the surrounding inference workflow from Python into the native engine.
  • 3.1 Engine and Models: Two transformer encoder reward models tested whether results generalized beyond one architecture.The models were OpenAssistant’s DeBERTa v3 large reward model and an Electra large discriminator reward model.
  • 3.1 Engine and Models: OpenAssistant’s DeBERTa v3 large reward model served as the main target, with an Electra large discriminator reward model as a second check.Both models were built on the BERT pretraining approach.
  • 3.2 Correctness Validation: The C++ engine was validated against a PyTorch reference model using identical inputs and direct reward-score comparisons.Correctness was checked before any speed comparison.
  • 3.2 Correctness Validation: 5.7e-6 was the largest absolute CPU score difference, with a 3.8e-6 mean difference.These figures compare C++ engine outputs with the PyTorch reference model.
  • 3.2 Correctness Validation: 4.2e-3 was the largest absolute GPU score difference, with a 1.9e-3 average difference.The larger GPU gap was attributed to CUDA kernels accumulating floating-point operations in a different order than CPU kernels.

3.3 Baselines · 3.4 Statistical Method

The study compares the C++ engine with three PyTorch execution paths and FastAPI, using repeated independent launches rather than single timed runs. Latency summaries aggregate per-row p50 and p95 statistics within launches, then report means and confidence intervals across launches.

  • 3.3 Baselines: The baselines are PyTorch eager mode, torch.compile, and FastAPI wrapping the PyTorch model.Eager mode represents the default RLHF path, torch.compile fuses operations into optimized kernels, and FastAPI represents HTTP-based serving.
  • 3.3 Baselines: FastAPI serves as a realistic serving-layer baseline because RLHF infrastructure often calls reward models over HTTP rather than in process.
  • 3.4 Statistical Method: Single timed runs are considered unreliable because process noise, OS scheduling, and CPU turbo scaling can alter apparent system speed.
  • 3.4 Statistical Method: Each system was launched as a fresh, independent process multiple times, with five repeats for the C++ engine on CPU and GPU.The passage also states that all three PyTorch GPU baselines received five repeats.
  • 3.4 Statistical Method: Means, standard deviations, and confidence intervals were computed using Python’s statistics module with fixed small-sample t critical values.A supplementary Welch’s t-test was also run for each headline comparison and reported alongside confidence intervals in Section 4.1.
  • 3.4 Statistical Method: Within each launch, every evaluation row was scored once, and per-row latencies were summarized by p50 and p95.
  • 3.4 Statistical Method: Across independent launches, reported values are means and 95% confidence intervals of launch summaries, not pooled row-level percentiles.This yields a mean of medians for p50 and a mean of 95th percentiles for p95.

3.5 Dataset

Evaluation used 60 fixed-seed samples from Anthropic’s hh-rlhf dataset, selected to span response lengths. Across five launches, observed noise was small relative to the reported system differences, supporting the reliability of the comparisons.

  • Dataset: The evaluation used 60 fixed-seed rows from Anthropic’s hh-rlhf dataset, covering a range of response lengths and shared prompts and responses across benchmarks.No formal a priori power analysis was performed; the scale was reasoned about informally and then checked empirically.
  • Dataset: The C++ engine’s CPU p50 varied by about 9% of its mean across five launches, with a 30.7 ms standard deviation around a 335.9 ms mean.The reported variation was compared against system differences to assess whether launch-to-launch noise could explain the results.
  • Dataset: The nearest CPU-baseline gap was roughly 246 ms, about 8× the C++ engine’s launch-to-launch noise.This supported the conclusion that observed system differences were larger than per-launch variability.

3.6 Batching and Concurrency

The study examined how batching and concurrency affect inference across multiple runtimes, models, devices, datasets, and serving configurations. Batching compared naive and length-bucketed strategies, while concurrency tested shared versus per-thread engine instances under controlled repeated measurements.

  • Batching and Concurrency: The batching sweep was repeated on Electra and an independently sampled 150-row dataset to test robustness to model choice and dataset size.The GPU sweep covered the C++ engine only.
  • Batching and Concurrency: Concurrency tested request levels 2, 4, and 8 with either one shared C++ engine instance or one instance per thread.Tests used DeBERTa and Electra on CPU and GPU, and also evaluated the shared-instance setup over real HTTP with FastAPI; results were means with 95 percent confidence intervals across 5 runs.

3.7 Hardware and Limitations · 3.8 Software Versions and Precision

The study’s hardware results are limited to one development machine, CPU, and NVIDIA GPU. All benchmarks used fp32 and ONNX Runtime 1.26.0 with identically pinned execution providers across C++ and Python.

  • 3.7 Hardware and Limitations: All benchmarks ran on one development machine, one CPU, and one NVIDIA GPU.No second machine with different hardware was available to test generalization.
  • 3.7 Hardware and Limitations: The hardware evaluation therefore does not establish performance generalization across different machines.The study explicitly avoids implying a broader hardware claim than tested.
  • 3.8 Software Versions and Precision: Every benchmark ran in fp32, with no half-precision or mixed-precision path in the codebase.This applies to both CPU and GPU benchmarks.
  • 3.8 Software Versions and Precision: Both the C++ engine and Python ONNX Runtime baseline used ONNX Runtime 1.26.0.The CPU used CPUExecutionProvider, while the GPU used CUDAExecutionProvider.
  • 3.8 Software Versions and Precision: The C++ and Python environments pinned ONNX Runtime identically through CMake FetchContent and pyproject.toml.This controlled the runtime release across implementations.
  • 3.8 Software Versions and Precision: Identical ONNX Runtime releases prevented differing versions from confounding C++ versus Python parity checks.The passage specifies this as the purpose of matching the environments.

4 Results

The C++ ONNX Runtime engine decisively outperformed PyTorch baselines on CPU, while torch.compile was fastest on GPU. Batching strategy and resource contention materially shaped throughput, and reruns supported the main findings while exposing one unverified comparison.

  • Latency comparison: 335.9 ms: the C++ engine beat all three CPU baselines by 1.7–1.9x, with non-overlapping confidence intervals and p < .001.The conservative confidence-interval comparison still showed at least a 1.4x margin.
  • Latency comparison: 19.0 ms: torch.compile beat the C++ engine’s 27.4 ms GPU median, while the C++ engine beat plain PyTorch and FastAPI.At p95, torch.compile remained ahead at 25.6 ms versus 116.2 ms, so static ONNX Runtime did not reduce tail latency.
  • Runtime and implementation ablations: 349 ms: Python calling the same ONNX Runtime session nearly matched C++ at 335.9 ms, showing the CPU advantage came from the runtime rather than the language.The shared ONNX Runtime session, model, and library produced essentially tied confidence intervals.
  • Runtime and implementation ablations: 3.8 times faster: the native C++ tokenizer took 64.3 microseconds versus Python AutoTokenizer’s 245.8 microseconds, while zero-copy and preallocation showed no effect.Reruns placed baseline, zero-copy, and preallocation in the same range: 228.7–238.6 ms on CPU and 16.5–18.6 ms on GPU.
  • Batching and throughput: 5 to 8x: naive CPU batching reduced throughput relative to batch size 1, whereas bucketing improved GPU performance but never exceeded CPU’s 3.10 rows per second.Independent reruns reproduced the device-specific pattern on Electra and a 150-row dataset.
  • Concurrency and robustness: 11 percent: shared-instance throughput increased at most from concurrency 2 to 8, while multi-instance execution degraded through thread-pool oversubscription.The authors identify length-aware batching as the only tested throughput lever on this CPU and 6 GiB GPU; other hardware remains untested.

5 Discussion

The discussion recommends architecture- and workload-specific choices rather than a blanket preference for C++ or torch.compile. It emphasizes that ONNX Runtime, length-aware batching, and repeated independent measurements determine practical performance, while GPU torch.compile and recompilation risk require particular caution.

  • Architecture-specific recommendations: Python calling the identical ONNX Runtime session matches C++ on CPU, leaving C++’s measurable advantage mainly to faster tokenization.Exporting to ONNX and serving through Python captures almost all of the CPU speedup; tokenization is only a tiny fraction of total latency.
  • Architecture-specific recommendations: Torch.compile beats the dedicated ONNX Runtime engine on GPU at the median and also wins p95 by a wider margin.The median result is supported by strict confidence intervals rather than noise, though dynamic shapes may incur recompilation costs.
  • Architecture-specific recommendations: A fixed 60 row shape distribution cannot rule out recompilation risk from unseen extreme sequence lengths, which can trigger a massive latency spike.The repeated launches in this study did not exercise this cache-miss scenario.
  • RLHF pipeline context: Actor generation accounts for upwards of 85 percent of total RLHF step time in the cited DeepSpeed Chat baseline, so isolated reward-model ratios do not directly measure step-time share.The paper does not directly measure scoring relative to policy rollout generation in a real training step.
  • Batching and scaling: Padding every request to the longest batch member costs 5 to 8 times CPU throughput and 3.5 to 4 times GPU throughput versus not batching at all.Sorting or bucketing by length before batching is presented as the practical remedy.
  • Batching and scaling: Separate engine instances add redundant memory and CUDA-context overhead without parallel gain, while length-aware batching is the only effective throughput lever on the tested CPU and 6 GiB GPU.A shared instance already serializes cleanly, whereas independent instances fail when VRAM is exhausted.

6 Limitations

The study’s limitations center on narrow hardware coverage, incomplete validation of concurrency and process isolation, excluded deployment baselines, and inference-only evaluation rather than end-to-end RLHF training. These constraints limit how broadly the findings and their training-time impact can be generalized.

  • Hardware and environment: All benchmarks used one development machine with one CPU and one 6 GiB NVIDIA GPU, without cross-checks on other hardware or operating systems.An independently sampled 150-row dataset was the closest substitute for additional environments.
  • Concurrency validation: 4–5 independent GPU sessions marked the observed memory ceiling, but concurrency levels 5, 6, and 7 were tested only once.All three levels failed with identical out-of-memory errors below the tested concurrency of 8, making the boundary a quick check rather than a rigorous result.
  • Process isolation: Multiple engine instances ran as separate threads in one process, so true memory isolation through separate operating-system processes was not tested.The study expects hardware contention findings to remain unchanged, but process-level isolation remains unvalidated.
  • Excluded baselines: Triton Inference Server was excluded because the available scaffold used a hardcoded latency-and-reward formula rather than a real server.A valid comparison would require a running Triton server with the exported ONNX model, Docker Desktop, and large image downloads.
  • End-to-end impact: The paper benchmarks isolated inference rather than wall-clock throughput in an end-to-end RLHF training loop.The training-time benefit of faster reward scoring depends on the fraction of each training step occupied by reward scoring.

7 Conclusion

The conclusion finds an unambiguous CPU advantage for the native engine, a mixed GPU result, and substantial effects from batching strategy. It also cautions that component-level scoring benchmarks do not establish whole-step RLHF impact and emphasizes repeated independent runs with confidence intervals.

  • Runtime comparison: On CPU, the C++ engine beats every tested PyTorch baseline with non-overlapping confidence intervals, and the speedup comes from ONNX Runtime rather than PyTorch eager mode.The conclusion isolates the runtime as the source of the advantage, not the C++ language itself.
  • Runtime comparison: On GPU, the C++ engine beats PyTorch eager mode and FastAPI, but torch.compile leads at both the median and 95th percentile.The GPU result was confirmed using the same confidence-interval standard as the other results.
  • Scope and limitations: Reward-scoring ratios do not measure its share of a real RLHF step, while outside evidence suggests rollout generation can occupy most of wall-clock time.The findings therefore apply to the measured scoring component rather than necessarily identifying the dominant training-loop bottleneck.
  • Serving and batching: Naive batch padding hurts throughput on both CPU and GPU, while length-aware scheduling recovers that loss only on GPU because CPU lacks batch parallelism.The conclusion identifies batching strategy as a practical serving consideration beyond runtime selection.
  • Serving and batching: Concurrent requests do not scale throughput on this hardware: shared-engine concurrency stays flat, separate engine instances worsen throughput, and batching is the only effective scaling method.This finding holds for both concurrency approaches described in the conclusion.
  • Benchmarking methodology: Independent process launches, summarized with means and confidence intervals, made the results reliable despite single timed runs being untrustworthy.The authors present this measurement discipline as a transferable lesson for benchmarking RLHF infrastructure.
Loading 2607.19712v1…