PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentReplay: Token-Wise Trace Replay Is Essential for Fair Serving System Performance Benchmarking

Zaifeng Pan, Michael Wang, Chris Wu, Zhengding Hu, Xinwei Qiang, Zhongkai Yu, Yufei Ding

arXiv:2609.32283v1cs.DC

TL;DR

Agent-serving benchmarks can confuse execution improvements with changes in the workload because outputs and tool interactions alter later requests. AgentReplay records and replays token histories and other workload properties while leaving serving decisions configurable. Its experiments show that token-wise replay controls workload variation that greedy decoding and length-wise replay cannot avoid, supporting fairer serving comparisons.

  • Problem

    Different outputs and tool interactions can change subsequent agent requests, making serving improvements difficult to distinguish from workload variation.

    Source: §1 · p. 1

  • Method

    AgentReplay records agent traces and forces recorded output tokens during normal autoregressive computation, with optional expert-selection and tool-duration replay.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    AgentReplay’s token-wise replay eliminates workload variation that greedy decoding and length-wise replay cannot avoid, enabling fairer comparisons across serving configurations.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    A capable-model trace provides a repeatable long-horizon workload for a compatible smaller target model.

    Source: §1 · p. 2, §6 · p. 9

    Fixed-trace comparisons measure recorded execution cost, not output quality or behavioral effects, and their coverage depends on diverse, representative traces.

    Source: §6 · p. 9

Abstract

from arXiv · show

LLM-based agents execute multi-turn workflows with interleaved model inference and tool calls, making efficient serving increasingly important. However, evaluating serving optimizations is challenging because identical tasks can produce different execution trajectories. Changes in generated tokens can alter subsequent prompts, tool calls, and reasoning turns, making it difficult to distinguish system improvements from workload variation. Greedy decoding does not guarantee identical outputs, while replaying only sequence lengths loses token information that affects prefix caching and mixture-of-experts (MoE) routing. To address these problems, we propose AgentReplay, a configurable trace record-and-replay framework for agent serving. AgentReplay records input/output tokens, expert selections, request dependencies, and tool durations. During replay, it forces the recorded output tokens while performing normal autoregressive computation, with optional controls for MoE expert selection and tool delays. This allows different systems to execute the same recorded workload while retaining their own batching, scheduling, and parallelization decisions. We further separate trajectory generation from performance evaluation, enabling compatible smaller models to replay long-horizon traces collected with more capable models. Our experiments show that token-wise replay in AgentReplay effectively eliminates workload variation that greedy decoding and length-wise replay cannot avoid, enabling fairer performance comparisons across serving configurations.

1 INTRODUCTION

Agent-serving benchmarks can confound system performance with workload changes because model outputs and tool timing alter later requests. AgentReplay records and replays agent behavior to compare configurations on a controlled workload, including long-horizon traces run by smaller models.

1 INTRODUCTION

  • Different generated tokens or tool durations can change subsequent prompts, tool calls, and turns, making apparent speedups difficult to attribute to serving optimizations.The same initial task does not guarantee the same realized workload.Source: §1 · pp. 1–2
  • Greedy decoding can vary across serving configurations, while length-wise replay misses token content that affects prefix-cache reuse and MoE expert selection.Matching sequence lengths alone therefore does not ensure comparable execution work.Source: §1 · pp. 1–2
  • AgentReplay records agent traces and forces recorded output tokens during normal autoregressive computation, with optional MoE-routing and tool-duration controls.The replay driver follows request dependencies while leaving batching, scheduling, and completion times to the serving system.Source: §1 · p. 2
  • Compatible smaller models can replay long-horizon trajectories collected with more capable models, separating trajectory generation from performance measurement.This reduces resource requirements without requiring the smaller model to solve the original tasks.Source: §1 · p. 2
  • AgentReplay’s token-wise replay eliminates workload variation that greedy decoding and length-wise replay cannot avoid, supporting fairer serving comparisons.The comparison preserves recorded token sequences rather than only initial tasks or sequence lengths.Source: §1 · p. 2

2 BACKGROUND

Agent workloads consist of dependent, intermittent model requests separated by tool or environment interactions. A serving benchmark should control workload properties that affect the optimization under study while leaving serving mechanisms free to change.

Agent workloads

  • Each agent turn sends accumulated context to an LLM, and its output or a tool observation shapes later turns, creating stateful and intermittent requests.An execution graph captures dependencies among LLM and tool invocations, including concurrent branches.Source: §2 · p. 2

LLM serving

  • Serving engines queue requests, form iteration-level batches, execute model kernels under configured parallelism, and manage parameters and KV caches across memory tiers.Performance depends on decisions across these serving layers.Source: §2 · pp. 2–3, Figure 1 · p. 3

Content and timing affect serving work

  • Token content affects prefix-cache reuse and MoE routing, while tool duration changes later request arrivals and memory pressure.These workload properties matter even when evaluating optimizations focused on LLM serving.Source: §2 · p. 3

Benchmarking objective

  • A controlled benchmark fixes relevant trace properties while allowing batching, scheduling, cache placement, kernels, communication, and physical expert placement to vary.Live-agent evaluation remains necessary for task success and behavioral effects; length-wise replay is sufficient only when the mechanism is insensitive to token content and omitted properties.Source: §2 · pp. 3–4

3 AGENTREPLAY

AgentReplay records agent executions and replays their dependencies and workload properties while leaving serving-system decisions configurable. It supports token, optional expert-routing, and tool-duration replay, as well as long-horizon evaluation on compatible smaller models.

3 AGENTREPLAY

  • The recorder captures real LLM and tool invocations, and the replay driver uses the trace to evaluate serving configurations with independently selectable controls.The controls preserve token sequences, logical expert selections, and tool delays; model computation, scheduling, and expert placement remain configurable.Source: §3 · p. 4, Figure 2 · p. 4

3.1 TRACE RECORDING

  • A trace stores invocation dependencies, input and output tokens, optional logical expert selections, and tool durations, with dependencies obtained from the application or orchestration runtime.Recorded inputs use token IDs, and replay executes the recorded graph rather than asking the agent to choose actions again.Source: §3.1 · p. 4

3.2 MULTI-LEVEL TRACE REPLAY

  • Token-wise replay commits each recorded output token before updating subsequent inputs and token-dependent state, while retaining normal autoregressive computation.Optional router replay substitutes recorded logical expert IDs before mapping and dispatch; duration replay releases dependent requests after recorded tool delays, not at fixed absolute timestamps.Source: §3.2 · p. 5

3.3 LONG-HORIZON EVALUATION WITH SMALLER MODELS

  • A capable model can generate long trajectories for a compatible smaller model to execute through normal forward passes, exposing context growth and token reuse it may not reach live.When tokenizers or serialization differ, the recorded messages and observations are retokenized for the target, subject to its context limit.Source: §3.3 · p. 5

4 EXPERIMENTS

The experiments show that greedy decoding can produce variable agent workloads, while token-wise replay stabilizes repeated measurement. Length-wise replay can change MoE expert traffic and execution costs, and a smaller model can replay a completed long-horizon trace with repeatable timing.

4 EXPERIMENTS

  • The experiments use Qwen3.6-35B-A3B on Astropy and SymPy; workload variability is summarized by CV across complete runs, treating each run as one sample.Cumulative input tokens include context resubmitted across turns, and cumulative time adds tool durations.Source: §4 · p. 6

4.1 DOES GREEDY DECODING PRESERVE THE WORKLOAD?

  • Across 50 unchanged greedy-decoding runs, SymPy workload CVs reach 28.66% for turns, 142.21% for cumulative input tokens, and 103.49% for output tokens.Thirty-one SymPy runs entered unproductive cycles, while none of the Astropy runs did.Source: §4.1 · p. 6
  • Token-and-tool trace replay reduces cumulative-time CV from 25.58% to 0.66% on Astropy and from 114.93% to 1.30% on SymPy.These reductions measure repeatability, not speedup, because greedy runs and replay may contain different realized work.Source: §4.1 · p. 6

4.2 DOES LENGTH-WISE REPLAY PRESERVE MOE PERFORMANCE?

  • Across 40 layers, the top 32 length-wise experts average 77.7% of prefill and 56.8% of decode assignments, versus 13.8% and 17.1% under token-wise replay.Figure 4 uses the same length-wise frequency ordering for both replay modes, making the traffic shift across paired experts visible.Source: §4.2 · pp. 6–7, Figure 4 · p. 7
  • Length-wise replay overestimates EP prefill cost while underestimating TP decode and offloaded execution costs relative to token-wise replay.At batch 32, the full EP prefill phase takes 1.18× as long; TP decode takes 0.84× the reference, and offloading length-wise hot experts yields 0.39× prefill and 0.63× decode times.Source: §4.2 · p. 7, Figure 5 · p. 8, Figure 6 · p. 8

4.3 CAN A SMALLER MODEL REPLAY A CAPABLE MODEL’S WORKLOAD?

  • Cross-model replay retains all 132 turns and tool calls, with cumulative-time CV 0.45% versus 114.93% for natural execution.The comparison demonstrates repeatable execution of one completed trace, not autonomous task success or speedup; broader traces are needed to assess representativeness.Source: §4.3 · pp. 8–9

5 RELATED WORK

Related work includes reproducibility controls, agent-trace tools, and recorded MoE routing. AgentReplay focuses on controlled serving comparisons and cross-model replay of long-horizon workloads.

Nondeterminism in LLM inference

  • Batch-invariant kernels improve output reproducibility, but constrain kernels, parallelization, and routing choices; AgentReplay fixes selected workload properties while retaining each system’s native execution path.Greedy decoding can still diverge when serving configurations change numerical results (Yuan et al., 2026; He & Thinking Machines Lab, 2025).Source: §5 · p. 9

Agent traces and replay

  • TraceLab characterizes real-world coding-agent workloads, while AgentRR reuses successful traces to guide later behavior; neither defines the identical serving workload targeted by AgentReplay (Zhu et al., 2026; Feng et al., 2025).AgentReplay also differs from concurrent XPerf by studying route controls and capable-model trajectories replayed on compatible smaller models (Wang et al., 2026).Source: §5 · p. 9

MoE router replay

  • Prior MoE systems record and enforce expert selections for training consistency, debugging, or profiling, whereas AgentReplay applies routing controls to comparative serving benchmarks (Ma et al., 2025).Source: §5 · p. 9

6 DISCUSSION AND CONCLUSION

AgentReplay is presented as a controlled execution benchmark, not a substitute for evaluating output quality or behavior-changing optimizations. Its findings support token-wise replay for controlled comparisons, while trace representativeness and excluded speculative-decoding behavior bound its scope.

6 DISCUSSION AND CONCLUSION

  • Fixed-trace comparisons measure recorded execution cost, not whether an optimization preserves task quality; speculative decoding also requires separate treatment.Coverage depends on collecting diverse, representative traces.Source: §6 · p. 9
  • The conclusion reports that token-wise replay controls token histories, optional MoE assignments, and tool delays while retaining model execution and system scheduling freedom.Experiments support it as a basis for controlled serving comparisons.Source: §6 · p. 9

AI USE STATEMENT

The authors report using generative AI tools across manuscript editing, literature review, experimental protocol development, and figure-generation code, while taking responsibility for the claims and results.

AI USE STATEMENT

  • The authors used generative AI tools for manuscript editing, literature review, experimental protocol development, and figure-generation code, and remain responsible for accuracy.Source: AI USE STATEMENT · p. 10

Related papers

Loading 2609.32283v1…