PaperlayerView exact revision on arXiv

Source-linked AI summary

Can Agent Harnesses and Inference Engines Hear Each Other? The HEAR Protocol for Agentic LLM Serving

Jiaqi Zhao, Haodong Chen, Jitai Hao, Wei Zhao, Jinghao Pang, Qiang Huang, Jun Yu

arXiv:2610.06597v1cs.AI

TL;DR

Agentic serving decisions span harness workflow information and engine runtime state, but existing interfaces do not systematically connect these views. HEAR standardizes their bidirectional exchanges and supports cache-aware coordination and role-specific execution configuration. Across the evaluated benchmarks, these approaches report serving speedups without observed task-quality degradation.

  • Problem

    Agent harnesses know workflow dependencies, context lifecycles, and objectives, while inference engines observe queues, KV-cache state, resource pressure, and capabilities, but existing interfaces do not systematically connect these views.

    Source: §1 · p. 2

  • Method

    HEAR standardizes bidirectional harness–engine messages and separates protocol semantics from policies, which it instantiates for cache-aware runtime coordination and role-specific inference configuration.

    Source: §1 · p. 2

  • Results

    1.61× batch speedup on SCBench, with 1.23× and 2.45× end-to-end speedups on BrowseComp-Plus and DeepResearchBench, respectively, without observed task-quality degradation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results show that harness intent and engine feedback are complementary, workload-dependent signals for execution coordination.

    Source: §1 · p. 3

    The method improves inference efficiency but does not address the safety or reliability of underlying model outputs.

    Source: Ethics statement · p. 11

Abstract

from arXiv · show

LLM agents increasingly execute complex workflows involving multi-turn reasoning, tool use, and parallel agents. Efficient serving requires decisions that span two layers with complementary information: the agent harness understands workflow dependencies, context lifecycles, and execution objectives, whereas the inference engine observes request queues, KV-cache state, resource pressure, and execution capabilities. Existing interfaces do not systematically connect these views, limiting workflow-aware execution. HEAR, a bidirectional Harness--Engine Pairing protocol for agentic LLM serving. HEAR standardizes how the harness communicates workflow intent and execution requirements and how the engine returns runtime state, capabilities, and outcomes. By separating protocol semantics from optimization policies, HEAR supports diverse coordination strategies without changing workflow or model semantics. We instantiate HEAR for online cache-aware runtime coordination and workload-aware execution-mode selection for agent roles. Across four conversational and research-agent benchmarks under memory-constrained, concurrent serving, HEAR achieves a $1.61\times$ batch speedup and reduces median time-to-first-token by $2.23\times$ on SCBench. Mooncake shows that workflow intent and live engine state provide complementary benefits across load regimes. On BrowseComp-Plus and DeepResearchBench, workload-specific configurations yield $1.23\times$ and $2.45\times$ end-to-end speedups, respectively, without observed task-quality degradation. These results establish HEAR as a reusable coordination substrate for efficient agentic LLM serving.

1 Introduction

Agent workflows require harness and engine coordination because the two layers hold complementary information about workflow intent and runtime state. HEAR standardizes their exchanges and supports cache-aware runtime coordination and role-specific inference configuration, with reported efficiency gains across conversational and research-agent benchmarks.

1 Introduction

  • The coordination gap arises because the harness knows dependencies, context lifecycles, and objectives, while the engine sees cache residency, queues, resource pressure, and execution modes.Without shared signals, engines may evict soon-to-be-reused KV state and harnesses may schedule cold work before cache-resident requests.Source: §1 · p. 2, Figure 1 · p. 2
  • HEAR defines a bidirectional protocol for workflow intent, execution requirements, engine state, capabilities, and outcomes, while leaving optimization policies separate from protocol semantics.It coordinates serving of existing workflow requests without changing task-selection logic or model semantics.Source: §1 · p. 2
  • Two policies use the same interaction contract: online cache-aware coordination chooses which ready request runs next, while role-specific configuration selects how workflow roles execute.The latter maps role and workload characteristics to engine-supported modes using measured efficiency, capacity, and quality profiles.Source: §1 · p. 2
  • 1.61× batch speedup and 2.23× median time-to-first-token reduction on SCBench, plus 1.23× and 2.45× end-to-end speedups on BrowseComp-Plus and DeepResearchBench, without observed quality degradation.The evaluation spans four conversational and research-agent benchmarks under memory-constrained, concurrent serving.Source: §1 · pp. 2–3
  • Mooncake shows that future-reuse intent improves cache retention, but must be conditioned on live engine state to remain beneficial under overload.Together, the results identify harness intent and engine feedback as complementary, workload-dependent signals for execution coordination.Source: §1 · pp. 2–3

2 Related Work

Prior systems address agent orchestration, inference efficiency, and cross-layer coordination, but HEAR provides a shared interaction contract between an existing harness and engine. It standardizes cross-layer message semantics without prescribing policies or adding a runtime tier.

Agent Harnesses and Workflow Orchestration

  • Agent harnesses manage control flow, model calls, context, and tools, but lack direct access to inference-engine queues, batching, KV-cache residency, and resource pressure.HEAR conveys workflow intent and requirements to the engine and returns engine state and execution outcomes to the harness.Source: §2 · p. 3

LLM Inference Systems

  • Inference systems provide batching and KV-cache mechanisms, but do not define how workflow semantics reach the engine or runtime state and outcomes return to the harness.HEAR complements rather than replaces their schedulers, cache managers, and execution modes.Source: §2 · p. 3

Cross-Layer Coordination for Agentic Serving

  • Cross-layer systems combine workflow structure with runtime feedback, but their signals are coupled to particular policies or execution substrates.Examples include dependency exposure, history-based prioritization, workflow-aware retention and prefetching, and cache-pressure concurrency control.Source: §2 · p. 3
  • HEAR standardizes interaction between an existing harness and engine while leaving policies’ decision rules intact and adding neither a runtime tier nor relocated application logic.It associates descriptions and controls with requests and contexts, and returns engine state, capabilities, and outcomes.Source: §2 · p. 3

3 The HEAR Protocol

HEAR is a bidirectional execution-coordination protocol that connects harness workflow semantics with inference-engine state and outcomes. It supplies reusable message meanings and interaction rules while leaving policy decisions to the coordinating system.

3 The HEAR Protocol

  • HEAR defines the meaning and handling of harness–engine exchanges, while policies decide what information to exchange and what actions to take.Repeated exchanges can coordinate a dynamic workflow as workflow progress and engine conditions evolve.Source: §3 · p. 4

3.1 Scope and Execution Model

  • The harness manages roles, dependencies, readiness, and context lifecycles, while the engine manages execution, queues, KV caches, and serving resources.HEAR links each exchange to a request, context version, or serving instance without changing task selection, relaxing dependencies, or modifying model semantics.Source: §3.1 · p. 4

3.2 Message Categories

  • Harness-to-engine messages convey workflow facts or predictions and request actions; engine-to-harness messages report conditions, capabilities, and how requests were handled.The four categories are Execution Description and Intent, Execution Requirements and Control, State and Capabilities, and Execution Outcomes.Source: §3.2 · pp. 4–5, Table 1 · p. 5

3.3 Interaction Semantics

  • HEAR distinguishes descriptions from commands, preferences from requirements, observations from guarantees, and operation acceptance from completion.For example, reported KV residency can guide scheduling without reserving prefixes, and accepting a preparation request does not mean its effect is realized.Source: §3.3 · pp. 5–6

3.4 Policy Integration

  • Policies can combine workflow information with the latest relevant engine update to select an execution arrangement or configuration.Without engine feedback, the decision reduces to harness-local information; adapters can expose existing policies’ required descriptions, observations, and controls through HEAR.Source: §3.4 · pp. 6–7

4 Experiments

The experiments evaluate cache-aware coordination and role-specific inference configuration across conversational and research-agent workloads. Results show that workload-specific policies improve efficiency, while the preferred policy varies with load and workload.

4.1 Experimental Setup

  • The evaluation combines controlled closed-loop SCBench with a production-derived Mooncake trace, plus BrowseComp-Plus and DeepResearchBench in a common LangGraph-based harness.Within each setting, methods use identical models, inputs, prompts, workflow and tool budgets, and hardware; policies and parameters are frozen before evaluation.Source: §4.1 · p. 7, §4.2 · p. 8

4.2 Cache-Aware Runtime Coordination

  • Cache-aware coordination uses live cache and queue state with harness intent and requirements to control request ordering and KV retention.Cache and queue reports are State and Capabilities; priorities are preferences, while waiting protection is a mandatory requirement.Source: §4.2 · p. 7
  • Cache-Aware+Guard-40 improves P50, P95, and maximum TTFT over FCFS while retaining a 1.61× batch speedup on SCBench.Waiting protection constrains repeated deferral of cold requests; raising the threshold to 60 seconds favors reuse and batch completion but weakens tail-latency protection.Source: §4.2 · p. 8, Table 3 · p. 8
  • At 50% Mooncake load, Session-Aware raises reuse from 19.5% to 31.9%; at 75% the combined policy leads session time, throughput, and reuse, while 100% splits metrics.At 100% load, Cache-Aware improves P50, session latency, and throughput, while Session-Aware improves tail latency and reuse.Source: §4.2 · p. 8, Figure 3 · p. 8, Table 4 · p. 9

4.3 Capability-Aware Execution Configuration

  • 1.23× on BrowseComp-Plus and 2.45× on DeepResearchBench are the end-to-end speedups of selected configurations, with task quality preserved.BrowseComp-Plus uses vLLM–H2O and preserves Judge accuracy (46.63% vs. 46.15%); DeepResearchBench uses OmniKV–H2O and preserves RACE (41.1% vs. 40.6%).Source: §4.3 · pp. 9–10, Table 5 · p. 10
  • Reader/sub-agent H2O cuts mean latency 5.60× on BrowseComp-Plus and 7.22× on DeepResearchBench; the preferred Main-side mode changes with workload.BrowseComp-Plus favors vLLM for Main requests, whereas DeepResearchBench favors OmniKV for its longer decoding workload.Source: §4.3 · p. 10, Table 6 · p. 10
  • Neither selected mapping is optimal for both workloads: universal OmniKV–H2O increases BrowseComp-Plus wall time by 7.8%, while universal vLLM–H2O increases DeepResearchBench makespan by 14.0%.Aggregate efficiency and P95 latency can favor different mappings, so selection must account for the communicated execution objective.Source: §4.3 · p. 10

4.4 Cross-Policy Takeaway

  • Across both policy families, decisions adapt to the workload while the HEAR interface and workflow semantics remain unchanged.Cache coordination uses dynamic state and intent; capability-aware configuration uses stable engine profiles, workload descriptions, and objectives.Source: §4.4 · p. 10

5 Conclusions

HEAR connects harness workflow intent and requirements with engine state, capabilities, and outcomes while preserving workflow and model semantics. The paper reports efficiency gains across cache-aware coordination and role-specific configurations, and identifies boundaries around deployment metadata and model-output safety.

5 Conclusions

  • HEAR provides a reusable, policy-independent coordination substrate, with reported speedups on SCBench, BrowseComp-Plus, and DeepResearchBench and no observed task-quality degradation on the latter two benchmarks.The conclusion reports a 1.61× batch speedup on SCBench and 1.23× and 2.45× end-to-end speedups on BrowseComp-Plus and DeepResearchBench, respectively.Source: §5 · p. 11

AI use statement

  • The authors used generative AI for conceptual, methodological, coding, writing, translation, literature-review, formatting, and figure-preparation assistance, but not to generate datasets or experimental results.The authors state that they determined the research questions, system design, experimental decisions, and conclusions, and manually reviewed AI-assisted content.Source: AI use statement · p. 11

Ethics statement

  • The protocol exchanges execution and resource metadata, so practical deployments should enforce access control, tenant isolation, and bounded waiting.The method improves inference efficiency but does not address the safety or reliability of underlying model outputs.Source: Ethics statement · p. 11

Reproducibility statement

  • The main text specifies the protocol, policy instantiations, workloads, baselines, metrics, and settings; Appendix A provides implementation details and execution-mode configurations for reproduction.Source: Reproducibility statement · p. 11

A.1 Protocol and Selector Implementation

The appendix describes capability-aware role-to-mode selection and cache-aware runtime coordination, along with their evaluation settings and protocol costs. Role mappings are calibrated and verified before formal evaluation, while cache policies use replayed workloads and live engine state.

A.1 Protocol and Selector Implementation

  • The selector uses role, workflow, workload, concurrency, and objective inputs alongside engine-supported modes and development profiles, then verifies and freezes an accepted mapping before formal evaluation.Formal-evaluation results are not used during ranking or verification.Source: §A.1 · pp. 14–15

A.2 Cache-Aware Runtime Coordination Settings

  • Formal cache-policy comparisons replay the same recorded trajectories; SCBench arrivals and think times are simulated, while Mooncake replays production timing and request information at three session-load levels.Experiments hold model, cache capacity, trajectory, and other serving parameters constant except for the coordination policy being compared.Source: §A.2 · p. 15
  • Cache-Aware prioritizes requests with reusable prefixes, while Guard protects requests after they exceed a waiting threshold so later cache-hot requests cannot keep bypassing them.Requests are classified as hot when reusable prefixes cover at least 50% of the prompt; hot requests are ordered by decreasing reusable-prefix length.Source: §A.2 · pp. 15–16
  • Session-Aware estimates a session’s return probability from completed conversations and combines it with recency for KV-cache retention priority, which decays as idle time increases.The estimate does not use future information from the current replay.Source: §A.2 · p. 16

A.3 Protocol Cost Accounting

  • HEAR adds control-plane metadata processing rather than model-side computation; the online path updates state, handles controls, and reports outcomes, while role-to-mode mappings are loaded once and looked up at dispatch.The reported end-to-end results include the online control-plane path.Source: §A.3 · p. 16

A.4 Role-Specific Models and Execution Modes

  • Role-specific runs use GLM-4.7-Flash with Main and Reader/sub-agent inference on separate H100 GPUs; DeepResearchBench configurations also apply a FIFO prefill gate to SnapKV sub-agent requests.The gate limits in-flight prefill workload because SnapKV does not sparsify prefill and long requests reserve space for their full prompts until prefill completes.Source: §A.4 · p. 17, vLLM · p. 18

B Additional Results

The runtime traces show how coordination policies affect execution order, KV restoration, and queueing. Guard limits prolonged waiting on SCBench and reduces queue buildup on Mooncake without materially changing the overall KV-occupancy pattern.

B.1 Runtime Behavior of Cache-Aware Runtime Coordination

  • Under SCBench replay, FCFS queueing extends the execution horizon, while Cache-Aware concentrates service earlier and Guard limits prolonged waiting from repeated bypass.All configurations replay the same session-arrival and think-time trajectory, so user-gap patterns are shared across methods.Source: §B.1 · p. 18, Figure 4 · p. 19
  • Across the traces, policy differences are reported in service timing, KV restoration, and waiting-request buildup, while SCBench user gaps and Mooncake GPU KV occupancy remain similar across policies.Source: §B.1 · p. 18, Figure 4 · p. 19, Figure 5 · p. 19
  • On Mooncake replay, session-aware variants restore substantially more KV than cache-aware scheduling, while coordinated policies reduce peak waiting requests versus FCFS and Guard further limits queue buildup.GPU KV occupancy follows a similar trajectory across policies, and Guard does not materially change the overall occupancy pattern.Source: §B.1 · p. 18, Figure 5 · p. 19

B.2 Complete Six-Arm Results of Role-specific inference configuration

The selected configurations minimize aggregate completion time on their respective benchmarks without observed task-quality degradation. The results also show that DeepResearchBench sparse configurations vary more in execution efficiency than in RACE.

B.2 Complete Six-Arm Results of Role-specific inference configuration

  • BrowseComp-Plus results report aggregate efficiency and per-question latency in Panel (a), and task quality and completion outcomes in Panel (b).Throughput is measured in questions per hour and end-to-end latency in minutes; C, I, U, and F denote correct, incorrect, unresolved, and run-failure questions.Source: §B.2 · p. 19, Table 10 · p. 20
  • DeepResearchBench results report makespan, throughput, and per-instance latency in Panel (a), and RACE scores, quality dimensions, and completion outcomes in Panel (b).Quality scores are multiplied by 100; N is the number of reports evaluated by the RACE judge, and Comp., Instr., and Read. denote comprehensiveness, instruction following, and readability.Source: §B.2 · p. 19, Table 11 · p. 20
  • Among the configurations evaluated, development-selected vLLM–H2O and OmniKV–H2O achieve the lowest aggregate completion time on BrowseComp-Plus and DeepResearchBench, respectively, without observed degradation in task quality.Source: §B.2 · p. 19
  • DeepResearchBench sparse configurations differ substantially more in execution efficiency than in RACE.Source: §B.2 · p. 20

B.3 Role-Level Workload and Execution Analysis

BrowseComp-Plus and DeepResearchBench share a high-level role organization but have substantially different workload shapes.

B.3 Role-Level Workload and Execution Analysis

  • The two benchmarks share the same high-level role organization but expose substantially different workload shapes.Source: §B.3 · p. 20
  • Table 12 reports role-level workloads under each benchmark’s selected configuration, normalized per question for BrowseComp-Plus and per instance for DeepResearchBench.Source: §B.3 · p. 20

B.4 Quality Evaluation

The quality evaluation uses benchmark-specific evaluators and coverage, then compares selected configurations with all-vLLM baselines using paired cohorts.

B.4 Quality Evaluation

  • BrowseComp-Plus answer quality is evaluated over all 208 questions per arm, with unresolved questions and run failures retained in the accuracy denominator.Table 14 reports outcome distributions and Wilson 95% confidence intervals; Table 15 reports retrieval and citation quality.Source: §B.4 · p. 20, Table 14 · p. 21, Table 15 · p. 21
  • DeepResearchBench reports evaluation coverage and four RACE dimensions because task errors and empty reports reduce valid articles available for judging.Generated reports are evaluated with the official reference-free RACE evaluator; Table 16 reports quality and coverage.Source: §B.4 · p. 21, Table 16 · p. 22
  • Paired quality comparisons use each benchmark’s selected configuration against its all-vLLM baseline, with 208 BrowseComp-Plus questions and 89 common valid-score DeepResearchBench instances.Confidence intervals are percentile bootstrap intervals over paired questions or instances; Table 17 reports the comparison.Source: §B.4 · p. 21, Table 17 · p. 22
  • BrowseComp-Plus outputs are judged by Qwen3-32B across RTX PRO 6000 and H100 environments; re-evaluating all arms in one environment would remove this difference.DeepResearchBench uses the vendored official RACE evaluator with gpt-5.4-mini, and provider errors from the initial high-concurrency judge pass are re-evaluated at concurrency four.Source: §B.4 · p. 21
Loading 2610.06597v1…