Ready Cohorts: Bounding GPU Opportunity and Avoiding Host Round Trips in LLM-Agent Control

Josef Liyanjun Chen

Source-linked AI summary

Ready Cohorts: Bounding GPU Opportunity and Avoiding Host Round Trips in LLM-Agent Control

Josef Liyanjun Chen

arXiv:2608.12123v1Aug 12, 2026cs.DCcs.AIcs.OS

TL;DR

The paper asks when deterministic LLM-agent control transitions can be grouped profitably on GPUs and how keeping intermediate decisions on device changes execution. It formalizes deadline-feasible cohort supply and evaluates device-resident decisions, finding gains under both gates but requiring finite online measurement for deployment benefit.

Problem
The paper asks when deterministic agent transitions expose enough same-group work for profitable GPU execution and when intermediate decisions can avoid host-mediated control.
Method
The paper formalizes ready-cohort packing from trace events and separately compares host-mediated versus device-resident binary decisions across GPU placements.
Results
Exact sliding-deadline packing raises eligible share from 30.19% to 43.00%, while device-resident decisions are faster in all 36 tested configurations.
Takeaways & Limitations
GPU agent control has two measurable gates: deadline-feasible cohort supply and observation placement, with online measurements needed to assess achieved share and service-level benefit.
Takeaways & Limitations
The trace and mechanism studies cannot be multiplied because they use different models, and the offline optimum assumes future knowledge, zero service time, and unlimited capacity.

Abstract

from arXiv · show

LLM-agent services repeatedly execute small deterministic transitions between model and tool calls: route an outcome, update state, and emit the next effect. We ask when this control path exposes enough concurrent work for GPU execution, and what changes when a GPU-computed route decision remains on device. We formalize the ready-cohort boundary using fixed-partition share F, exact offline share P*, local upper bound U, and online achieved share A. Under zero service time, unlimited capacity, and equal relative launch deadlines, a specialized dynamic program computes P* exactly. In a stationary Poisson replay of one pinned 851-session public trace panel, the primary condition at 100,000 target active sessions, K=256, and a 50 ms launch deadline gives F=30.19%, P*=43.00%, and U=45.85%. Exact packing recovers 81.83% of the opportunity lost at fixed window boundaries. The outcome-derived route key is a conditioning proxy, not proof of executable identity. A separate mechanism study keeps a GPU-computed binary decision on device instead of returning four bytes to the host and redispatching. Across four named GPU placements, the device-resident path is faster in all 36 configurations; within-placement row-median ratios range from 1.19x to 2.39x. Across both admissible mechanisms, all 14,557,440 tested batched invocations match a separately implemented host oracle. A fixed nested device graph that removes no host decision is slower in all 60 configurations across five placements. Together, the studies establish two measurable gates for GPU agent control: deadline-feasible cohort supply and observation placement. A joined finite online runtime is required to measure A, CPU displacement, and service-level benefit.

1 Introduction · 2 Execution and semantic scope

The paper frames GPU acceleration for agent control around two gates: enough semantically compatible transitions must become ready before launch deadlines, and GPU-computed decisions should remain off the host when possible. It studies this boundary through trace-based cohort packing and mechanism experiments, while limiting the execution unit to pure deterministic state transitions and treating route-key similarity as a proxy rather than proof of fusion eligibility.

1 Introduction: Agent runtimes continuously execute small deterministic transitions between model and tool calls, including state updates, policy checks, route selection, and effect emission.These transitions are individually small but become a datacenter concern under many concurrent sessions.
1 Introduction: GPU execution requires cohorts large enough to cross the hardware/runtime threshold, share executable semantics, and meet launch deadlines.Host-visible decisions also add copy, synchronization, branching, and redispatch overhead; these dimensions define the ready-cohort boundary.
1 Introduction: The workload experiment compares fixed-window eligibility with exact offline packing and a local upper bound, while the mechanism experiment changes where one GPU-computed decision is observed.An earlier device-launch design serves as a negative control, and the experiments remain numerically separate because K = 256 is swept rather than measured for the resident-policy study.
1 Introduction: The framework separates hardware threshold K, fixed-partition share F, exact offline share P⋆, local bound U, and online achieved share A, with each quantity assigned a defined inference boundary.Under zero service time, unlimited capacity, and equal relative launch deadlines, a specialized dynamic program computes P⋆ exactly.
2.1 Unit of execution: A deterministic control transition reads typed state and an available event, then produces typed state, a route, and possibly an effect descriptor.Model inference, tool execution, and privileged external-effect commits remain outside the execution unit.
2.2 Compatibility and exactness: Outcome-derived route keys are conditioning proxies: they omit state-machine node, schema, arguments, policy context, and multi-tool identities, so they do not prove semantic fusion admissibility.Coarser event classes and pooled routes are diagnostics rather than evidence that compatible implementations can process events together without changing semantics.
2.2 Compatibility and exactness: Mechanism-study correctness requires exact equality of every declared state field and the full decision sequence against a separately implemented host oracle.This criterion does not establish correct recovery, distributed ordering, or real tool effects.
2.3 Launch deadlines, not completion SLOs: Every trace event has a latest admissible launch time, while the mathematical model sets kernel service and post-launch queueing to zero.Thus, the 50 ms primary value is a scheduler waiting bound, not a 50 ms completion service-level objective; finite service is deferred to the proposed online system.

3 The ready-cohort model

The ready-cohort model defines deadline-feasible, route-homogeneous batches and distinguishes fixed-partition eligibility from offline opportunity, local overlap, and finite-runtime acceleration. It also limits interpretation of route labels and grouping because trace labels do not prove implementation equivalence or admissible fusion.

A route label is not proof of executable identity, and finer trace grouping can fragment cohorts without making coarse fusion admissible.Pooled kernels may alter control flow, memory access, actions, or numerical trajectory.
A feasible batch requires |B| ≥ Kr, a common route ri = r, and an overlapping launch time τ satisfying ti ≤ τ ≤ di for every event.The model permits batches larger than Kr and imposes no maximum batch size, service time, or device-capacity constraint.
F measures the exact share schedulable under a frozen, non-overlapping time partition, not a universal ceiling because deadlines can join events across boundaries.Fixed-partition eligibility is defined by each window-and-route bucket meeting its threshold.
P ⋆ is the future-informed maximum assigned share of a feasible offline schedule, so it measures opportunity rather than online performance.The offline schedule assigns each event at most once.
U is only a local upper bound: it may count incompatible opportunities that reuse events and therefore need not correspond to an achievable schedule.Every feasible batch contributes events with sufficient same-route overlap at its launch time.
A is the deadline-accelerated share achieved by a specified finite online runtime, using the same route, threshold, deadline, event set, and denominator as P ⋆.Queueing, service, fallback, capacity, and late, missing, failed, or fallback events can reduce A relative to offline opportunity.

4 Exact packing under equal relative deadlines

Under zero service time, unlimited simultaneous capacity, and equal relative deadlines, an exact dynamic program computes the maximum-cardinality offline packing by launching batches at event deadlines. Its implementation is exact on an integer-nanosecond clock, while the reported frozen evaluator can be quadratic in the worst case without changing the optimum.

Deadline launch times: An optimal offline schedule launches every batch at an event deadline under zero service time and unlimited simultaneous capacity.This structural result does not extend unchanged to arbitrary deadlines or finite service.
Contiguous block form: Equal relative deadlines permit an optimum of disjoint contiguous blocks in sorted release order for each route.Crossing assignments can be exchanged, and intervening feasible events can be added to form contiguous blocks.
Dynamic program: O(N log N) evaluation is achievable with grouped sorting, route slices, a moving left pointer, and a monotone deque.The frozen evaluator instead scans the full group array once per route and uses binary search, with bound O(NR + Σ_r n_r log n_r).
Exact clock: The exact clock rounds releases and relative deadlines to integer nanoseconds and uses inclusive comparisons without floating tolerance.The implementation agrees with subset brute force on tiny instances, including route-specific thresholds and adversarial boundary cases.
Role in this paper: The trace oracle selects the maximum number of ordered points partitionable into clusters of at least K points and diameter at most δ.This is the fixed-radius decision form of one-dimensional r-gathering with outliers, used here only as an exact trace oracle.

5 Trace study

The trace study uses a fixed-panel subset of the public Exgentic agent-trace dataset to evaluate ready-cohort packing under a prospectively frozen replay design. Its route key is only a conditioning proxy, and the replay is a controlled load model rather than a production-arrival model.

Data source: The source is the complete tau2_airline, tau2_retail, and tau2_telecom subset of the public Exgentic agent-trace dataset.All 19 retained shard hashes match commit-resolved URLs.
Interpretation limits: The construction treats model completion as the point when a control transition could become ready, but route keys do not prove executable compatibility.Recorded harnesses are not claimed to implement the studied transition, and their span durations are not treated as model service times.
Experimental design: The exact-packing experiment spans C ∈ {1,000, 10,000, 100,000}, δ ∈ {10, 25, 50, 100, 250} ms, three grouping modes, K ∈ {32, 64, 128, 256}, and three replay seeds.The design yields 180 design cells and 540 cell-seed rows, with every generated event retained.
Primary cell: The primary cell uses C = 100,000, route-key grouping, K = 256, and δ = 50 ms.The three seeds quantify Monte Carlo variation under one fixed panel and arrival model, not independent deployment traces; therefore no population p-value or confidence interval is reported.

6 Trace-conditioned opportunity

Trace-conditioned opportunity depends on deadline-feasible route-key cohorts, not nominal swarm size alone: exact packing raises admitted share above fixed windows but remains below the local bound. The workload budget is characterized by F, P* and P*−F, while online systems must measure achieved share A.

Primary opportunity: 43.00% exact sliding-deadline share exceeds 30.19% fixed-window share by 12.81 percentage points and remains 2.85 points below the 45.85% local bound.The exact share ranges from 42.43% to 43.41% across three seeds.
Route-key conditioning: P* is zero for every tested deadline when C ≤ 10,000 under route-key grouping at K = 256.At C = 100,000, P* remains zero at 10 and 25 ms, showing that nominal swarm size does not ensure a profitable short-budget cohort.
Hardware threshold: At C = 100,000 and 50 ms, exact shares are 66.8%, 66.0%, 48.4%, and 43.0% for K = 32, 64, 128, and 256, respectively.At C = 10,000 and 50 ms, P* is 22.2% for K = 32 but 0.0% for K = 64.
Online workload budget: F is the fixed-window baseline, P* is the maximum attainable legal-scheduler share, and P*−F is headroom for improved packing.An online runtime must measure route-specific K and report achieved share A.

7 Device-resident decision study

The study isolates GPU-resident placement of binary decisions over resident state, comparing host round trips with device tail-launching across a frozen mechanism grid. It separately defines a no-decision oracle floor and validates admissible mechanisms against an independent host implementation.

Mechanisms: Device-resident execution keeps predicate evaluation and route selection on the GPU, tail-launching per-epoch path graphs without exposing decisions to the host.The host-roundtrip baseline copies a four-byte predicate result to pinned host memory, synchronizes, and launches the selected route graph from the host.
Mechanisms: The no-decision floor replays the oracle route sequence as one graph while omitting predicate and selection work, so it is not an admissible online scheduler.Its final state is checked independently, but its decision sequence is not an independent observation.
Experimental design: The full grid spans N ∈{256, 2048, 16384} agents and H ∈{2, 8, 32} epochs, with five warmups, three calibration samples, and 30 measured rows per mechanism-cell.One batch count, calibrated from the fastest mechanism, is applied to all mechanisms in each cell; each row runs for at least 100 ms of aggregate timed work subject to a frozen cap.
Scope and limitations: Performance effects are reported by named placement, with 30 rows reducing technical timing noise rather than providing independent hardware replications.The reported placement scope includes a local GTX 1660 Ti, Modal L4, RunPod L4, and Lambda H100 SXM5; replication counts are L4 n = 2, H100 n = 1, and GTX n = 1.
Validation: Correctness requires every final state field and the complete epoch-by-epoch decision trace to match a separately written host implementation; failures block performance interpretation.The independent host implementation computes both route functions and the predicate without calling device transition functions.

8 Mechanism results

Device residency improves the measured control path over a host round trip while preserving exact agreement with the host oracle. A nested device launcher, by contrast, adds overhead when it removes no host decision; joined-runtime effects are deferred to Section 9.

Admissible mechanisms: 14,557,440 tested batched invocations across the two admissible mechanisms are field-exact and decision-exact against the host oracle.All 3,240 measured rows pass the frozen status, timing, duration, source, provider, and correctness gates.
Device-resident path: 1.19× to 2.39×: the device-resident path has a lower within-placement median than the host round trip in all 36 placement-cells.At N = 256, H = 32, named-placement ratios are 1.71× on the local GTX 1660 Ti, 2.39× on the Modal L4, 2.06× on the RunPod L4, and 1.84× on the Lambda H100.
Device-resident path: 258 to 309 µs versus 467 to 625 µs: at the primary cell, the device path is faster by 194 to 363 µs per 32-epoch cohort invocation.The device path remains 6.60× to 8.17× slower than the oracle floor because predicate, selector, and graph overhead remains large beside the synthetic route bodies.
Nested device launch: 1.07× to 1.99×: a fixed child graph launched from a GPU kernel is slower across 5 named placements and all 60 cells, while removing no host decision.All 12,000 rows are field-exact; this negative control isolates device residency from device launch.
Scope of comparison: Cold graph construction, ingress, effect egress, validation, and a tuned CPU route body enter the joined runtime evaluation in Section 9, not this timed comparison.The mechanism comparison isolates one control-path choice.

9 Joining the evidence

The two studies establish a staged test for GPU agent control: deadline-feasible cohort supply and observation placement must both pass before a route transition moves from CPU to GPU. The evidence defines a finite-capacity online runtime and named measurements, but does not yet provide a service-level acceleration estimate.

Staged systems test: Two gates govern GPU route placement: cohort supply must provide enough same-group work before deadlines, and observation placement must avoid host-copy and redispatch overhead.Failure at either gate is a concrete reason to keep the transition on the CPU.
Evidence limits: The current studies cannot be multiplied into a service-level acceleration estimate because they use different sweeps, cohort sizes, and route bodies.The trace study also does not measure whether 32 consecutive control epochs execute without intervening observations.
Online runtime: A finite-capacity online route compactor should accept typed completion events, queue verified executable routes by deadline, launch above the safe-suffix threshold K_r, and fall back when batching is infeasible.The runtime target includes A and R_A alongside invocation P99, deadline misses, CPU core-seconds, exact trajectories, utility, network bytes, cost, and shared-GPU TTFT/TPOT guardrails.
Online runtime: A credible runtime evaluation requires z_i=1 only for exact, deadline-feasible accelerated events; late, missing, failed, and fallback events remain in E with z_i=0.This makes offline recovery above fixed windows, when P⋆>F, a direct runtime target.
Deployment architecture: A regional route service is the strongest architecture because aggregation directly attacks cohort supply, while ingress, state movement, effect egress, and noninterference determine whether savings survive deployment.A dedicated control GPU is the weakest starting point because utilization and CPU displacement are not established; external lifecycle effects remain under privileged CPU or DPU authority.

10 Related work · 11 Scope of inference

The paper positions route-key-conditioned post-event transitions at the interface between agent serving, GPU control, and batching theory, adding observation placement as a second axis. Its evidence is bounded by narrow trace, mechanism, timing, and algorithmic scopes, so deployment benefit remains to be measured.

10 Related work: Prior GPU systems map agent state and control to devices, whereas these measurements ask whether traces supply deadline-feasible grouped work and what matched host observation costs.Related work includes FLAME GPU 2, CUDA Graphs, GPUOS, MPK, and Event Tensor.
10 Related work: Agent-serving systems study CPU–GPU crossings, orchestration, memory, state, tools, scheduling, and placement; batching research studies arrivals, deadlines, occupancy, families, release times, and clustering.The cited literature spans production characterization, Agora, ThunderAgent, Parrot, tool scheduling, tokenization, CPU-induced slowdowns, real-time batching, and r-gathering.
10 Related work: The paper places route-key-conditioned post-event transitions above a hardware crossover at the interface of prior serving systems and batching theory, with observation placement as a second axis.A cohort may exist while a host round trip still dominates a short control chain.
11 Scope of inference: The trace threshold is a swept candidate, the mechanism’s H = 32 horizon is absent from the trace model, and the result sets cannot be multiplied.The offline optimum assumes future knowledge, zero service time, unlimited capacity, no upper batch size, and launch rather than completion deadlines; safe-suffix extrapolation remains limited.
11 Scope of inference: 851 sessions come from one panel spanning three related customer-service domains under stationary Poisson arrivals, so seeds measure variation under that panel rather than a workload population.Bursts, correlated releases, and other corpora can move the boundary; the outcome-derived route key is a conditioning proxy rather than executable identity.
11 Scope of inference: Resident-policy-001 makes one global binary decision over a regular synthetic state array, while real route services still require compaction, variable bodies, effects, recovery, and CPU fallback.Setup, reset, copying, and validation are outside timing; the four placements confound hardware, provider, host, image, driver, and region.
11 Scope of inference: The mechanism metric is batch-average cohort-horizon wall time, not invocation P99 or end-to-end task time.The study does not establish CPU core-seconds, energy, cost, model throughput, TTFT, TPOT, task utility, or external-effect reliability.
11 Scope of inference: The dynamic program is a specialized exact instrument within mature batching, clustering, and scheduling literatures, without an exhaustive algorithmic priority claim.A tuned CPU implementation, resource use, energy, cost, throughput, latency, utility, and reliability remain deployment measurements.

12 Reproducibility · 13 Conclusion

The paper releases reproducible, hash-bound artifacts and scripts, while concluding that GPU agent control requires both deadline-feasible ready-cohort supply and device-resident observation. Exact offline packing and mechanism results define measurable gates for validating an online route compactor.

12 Reproducibility: Public code and processed-evidence materials are released through the project repository and Hugging Face mirror.The release materials are available at github.com/josefchen/ready-cohorts and the processed-evidence mirror is hosted on Hugging Face.
12 Reproducibility: Separate raw, processed, analytical, preregistration, and output artifacts preserve prior runs while assigning new identities to new executions.Analysis does not overwrite prior raw measurements.
12 Reproducibility: Hash-validated scripts generate the manuscript’s numerical macros and primary tables, checking trace dependencies, resident-policy contrasts, native-dispatch contrasts, and key invariants.The generator validates all 19 local Parquet shards and checks primary-cell counts, boundary invariants, trace-gate flags, and directional inequalities.
12 Reproducibility: Dataset and conversion commits bind the trace, while SHA-256 checks match all 19 local shards to the manifest and commit-resolved remote bytes.Derived features retain timestamps, counts, lengths, route labels, and public identifiers, excluding prompt text, tool arguments, and tool results.
13 Conclusion: 30.19% to 43.00%: exact sliding-deadline packing recovers 81.83% of the opportunity lost by fixed windows.The conclusion frames GPU agent control as requiring enough same-group events before launch deadlines and notes that cohort supply can collapse to zero.
13 Conclusion: 1.19× to 2.39×: keeping the tested binary decision on device reduces cohort-horizon wall time in all 36 cells across four named placements.A nested device graph removing no host decision loses in all 60 cells across five placements, so device launch alone is insufficient.
13 Conclusion: Achieved share A relative to F and P ⋆, raw P99, CPU core-seconds, exact effects, task utility, cost, and shared-inference guardrails are the decisive online measurements.The GPU design is rejected if it cannot convert offline headroom into online work or displace CPU without harming model service.

A Proof and evaluator scope … C Protocol deviations

The paper specifies a binary-program reference formulation for arbitrary deadlines and uses an equal-deadline evaluator that returns one witness without quadratic candidate assignment. Its evidence and protocol scope limit population, end-to-end, and native-dispatch conclusions to the retained designs and placements.

A Proof and evaluator scope: Candidate launch times may be restricted to unique deadlines under zero-service and unlimited-capacity assumptions.This restriction applies to arbitrary deadlines under the stated assumptions.
A.1 Binary-program reference formulation: The reference formulation assigns events to deadlines and indicates route-specific batches using binary variables.It defines xi_τ for event assignments, y_rτ for batches, and M_rτ for route-r event intervals containing τ.
A.1 Binary-program reference formulation: The evaluator does not minimize batch count or wait among maximum-cardinality schedules.Instead, the equal-deadline evaluator avoids quadratic candidate assignment and returns one witness.
B Evidence inventory: Population and end-to-end claims require the joined confirmation design.Table 6 organizes evidence layers, outer units, and admissible inference.
C Protocol deviations: The retained cloud protocol included Modal plus one of RunPod or Lambda, with both external placements retained.The additional placement was a disclosed descriptive scope expansion.
C Protocol deviations: Horizon-ratio monotonicity is exploratory because its operational test was not fully specified in the preregistration.A correctness and engineering smoke preceded source freeze, and the smoke and full local run shared one GPU UUID.
C Protocol deviations: The retained native-dispatch calibration contains five named placements: one H100, three L4s, and one GTX.The proposed layout of two fresh placements per provider/SKU and at least six H100 placements was not completed.
C Protocol deviations: The native-dispatch calibration supports only the descriptive negative result reported here.The retained layout does not support the originally proposed nuisance-estimation design.