PaperlayerView exact revision on arXiv

Source-linked AI summary

The Cost of a Hop: Benchmarking NLIP and A2A

Ranjan Sinha, Anindita Das, Ashika Anand Babu, Hari Palleti

arXiv:2610.04053v1cs.AI

TL;DR

Agent protocols are increasingly important as distributed LLM agents communicate across systems, but NLIP lacked controlled performance evaluation and protocol latency had not been decomposed by phase. This study compares NLIP and A2A across environments and finds a consistent lightweight-stage advantage for NLIP, with magnitude dependent on hardware.

  • Problem

    NLIP had not appeared in a controlled protocol-performance study, and prior benchmarks had not decomposed where protocol latency is spent.

    Source: §XI · p. 9

  • Method

    The study compares NLIP and A2A across multiple environments, measuring latency by phase and evaluating A2A-SDK with connection caching.

    Source: §IV · p. 5, C. Connection-Caching Condition · p. 7, §XI · p. 9

  • Results

    NLIP is 8.4–9.6× faster than A2A-SDK on the lightweight stage in two environments and about 4× faster in a third, with advantage magnitude varying by hardware.

    Source: §VII · p. 8, §XI · p. 9

  • Takeaways & Limitations

    The paper recommends NLIP for latency-sensitive, coordination-heavy systems and A2A for governance-critical or coarse-grained tasks, with a hybrid deployment when both are needed.

    Source: §XI · p. 10

    The evaluation covers one sequential workflow and 19 distinct queries, so it does not establish behavior under concurrent load or across other workflow types.

    Source: §IX · p. 9

Abstract

from arXiv · show

Autonomous agents built on Large Language Models (LLMs) need standardized protocols to interoperate across systems. Several now exist (A2A, MCP, ACP, ANP, NLIP), but the Natural Language Interaction Protocol (NLIP) has not appeared in any controlled performance study, and no work has measured where an agent protocol's latency is spent. We compare NLIP and the Agent-to-Agent (A2A) protocol empirically, decomposing latency into message creation, connection, and send phases across three independent hardware environments. For lightweight coordination, NLIP is 8.4-9.6x faster than the baseline A2A SDK implementation on two environments and about 4x on a third; the direction of the advantage is consistent, its magnitude depends on the hardware. The advantage is stage-specific: for the end-to-end pipeline, where LLM inference dominates, the protocols are near parity. The difference comes almost entirely from connection setup. To test A2A at its best, we also ran A2A SDK with connection caching enabled; caching narrows its gap with NLIP by a hardware-dependent amount, from 2.75x on one machine to near-parity on faster hardware, where at scale a cache-optimized A2A-SDK matches NLIP. We report these as measured conditions without a single causal account of the residual send-phase cost. Against the more optimized Python-A2A, NLIP leads by about 4x on the same stage. We close with a protocol-selection guide keyed to workload characteristics.

I. INTRODUCTION

The paper examines protocol latency as an increasingly important systems cost in distributed, multi-agent workflows. It compares NLIP and A2A in a controlled study that decomposes latency by phase and spans multiple environments.

I. INTRODUCTION

  • At data-center scale, a request may fan out into many fine-grained agent hops, making per-interaction protocol overhead a contributor to aggregate latency and resource consumption.The paper frames protocol efficiency as a big-data systems concern as well as an AI concern.Source: §I · p. 1

A. Background and Motivation

  • NLIP emphasizes natural-language-first, semantically rich exchanges, whereas A2A uses structured, capability-driven communication with explicit delegation and discovery.The paper presents them as contrasting approaches at the agent-to-agent interaction layer.Source: §I · p. 1, A. Background and Motivation · p. 2

I. INTRODUCTION

  • The study provides the first controlled latency comparison including NLIP, decomposes costs by phase, tests connection caching, and validates results across three machines.It also presents a workload-keyed protocol-selection matrix.Source: §I · p. 1

A. Background and Motivation

  • ProtocolBench compares other agent protocols on task-level outcomes, while this study adds NLIP, decomposes latency by phase, and compares implementations and a caching condition.The paper distinguishes transport-layer costs from work that reduces message overhead through topology or pruning strategies.Source: A. Background and Motivation · p. 2

II. PROTOCOL OVERVIEW

The overview situates NLIP and A2A as different approaches to agent communication. NLIP prioritizes semantic continuity and flexible content, while A2A emphasizes structured, explicit exchanges and enterprise features.

A. Natural Language Interaction Protocol (NLIP)

  • NLIP uses natural language as its primary interaction medium and supports agent-to-agent and human-to-agent communication.It runs over REST and HTTPS with JSON transport, and its envelope can declare text, structured JSON, tokens, or binary content.Source: A. Natural Language Interaction… · p. 2
  • NLIP combines natural-language contracts, persistent semantic context, stateless REST, multimodal content, and hot-swappable contracts without schema-version negotiation.Its design favors semantic continuity over compact messages.Source: A. Natural Language Interaction… · p. 2

B. Agent-to-Agent Protocol (A2A)

  • A2A uses JSON-RPC 2.0 and Agent Cards for structured exchanges and capability discovery, with stateful sessions, streaming, explicit errors, and enterprise security features.Its self-contained messages simplify behavioral reasoning but require context reconstruction at each boundary.Source: B. Agent-to-Agent Protocol (A2A) · p. 2

III. EXPERIMENTAL METHODOLOGY

The methodology compares protocol implementations under controlled, functionally equivalent workflows. It measures two text-carrying agent transitions and separates message creation, connection, sending, and total latency.

A. Workflow Design

  • The three-stage workflow combines transcription receipt, LLM-based subreddit identification, and lightweight Reddit search, spanning substantial processing and coordination overhead.Transcription latency is external; the search stage represents lightweight tool invocation where protocol overhead can dominate the work.Source: A. Workflow Design · pp. 2–3
  • Only the two text-carrying inter-agent transitions are measured; the external audio-ingestion hop and its transport cost are excluded.The measured transitions run over NLIP, A2A-SDK, or Python-A2A.Source: A. Workflow Design · p. 3

B. Experimental Setup

  • Experiments span three Apple-silicon environments with differing query sets, while within-machine comparisons hold host, network, LLM provider, and API endpoints fixed.The machines are not hardware-matched, and all runs are sequential; cross-machine conclusions concern direction and rough effect magnitude, not absolute latency.Source: B. Experimental Setup · p. 3
  • The evaluation compares official NLIP, A2A-SDK, and Python-A2A implementations, adding A2A-SDK connection caching as an optimization condition.The reported implementation versions were the latest stable releases available at evaluation time.Source: B. Experimental Setup · p. 3
  • The analysis reports confidence intervals, effect sizes, and latency percentiles, retaining outliers and treating percentiles as sequential run-to-run variability rather than loaded tail behavior.The design also compares two implementations of the same protocol and a connection-optimized baseline.Source: B. Experimental Setup · p. 3

III. EXPERIMENTAL METHODOLOGY

  • The benchmark uses symmetric instrumentation and timestamps at phase boundaries to measure message creation, connection, send, and total latency.This decomposition is intended to separate protocol overhead from network variability and locate differences by phase.Source: §III · p. 2, B. Experimental Setup · p. 3

IV. RESULTS

NLIP is faster than A2A implementations across the measured environments, with its largest advantage in lightweight coordination. End-to-end latency is near parity when LLM inference dominates, and the phase decomposition places the main difference in connection setup.

IV. RESULTS

  • 9.6× to 4.2×: NLIP’s Subreddits-stage lead over A2A-SDK narrows from Machine B to Machine C, while the ordering remains unchanged.The same Dataset-19 workload ran on both machines; the change is in A2A-SDK’s connection phase.Source: Table I · p. 4, §IV · p. 4
  • Sequential-run percentiles show NLIP below both A2A implementations at every Subreddits-stage percentile, but do not characterize behavior under load.For the end-to-end pipeline, the implementations cluster because LLM inference dominates.Source: §IV · p. 4, Table II · p. 5
  • End-to-end latencies cluster near parity when LLM inference dominates, although NLIP is fastest in absolute terms across the 500–1000 ms range.Protocol overhead is a small fraction of total latency in this compute-heavy pipeline.Source: §IV · p. 4, Figure 2 · p. 5
  • NLIP runs at 5–10 ms on the Subreddits stage, versus 60–70 ms for A2A-SDK and 25–35 ms for Python-A2A.The ranking holds for every query in both datasets; Python-A2A’s intermediate performance reduces but does not remove A2A coordination overhead.Source: §IV · p. 5
  • The phase decomposition places the performance difference in connection setup; message creation is negligible, so serialization and formatting do not drive it.The reported phase metrics are message creation, connection, and send.Source: Figure 1 · p. 4, §IV · p. 5

A. Scale Test (Dataset-200, Machine C)

At 200 queries on Machine C, NLIP retains its phase profile and leads default A2A-SDK, while cached A2A-SDK reaches near-parity. Dataset-200 also increases query volume, variety, and length, so the scale test changes both volume and complexity.

A. Scale Test (Dataset-200, Machine C)

V. COMPARATIVE ANALYSIS

The measurements indicate that implementation quality influences observed latency to a degree comparable to protocol design.

  • Implementation quality influences observed latency to a degree comparable to protocol design.Source: §V · p. 6

A. Protocol Architecture Differences

The measured latency difference is concentrated in connection establishment: message creation is negligible, while A2A-SDK connection costs vary substantially across machines. Multi-hop scaling of the measured per-hop gap remains a projection, not a measured result.

A. Protocol Architecture Differences

  • On Machine B, A2A-SDK connection latency is 62.71 ms versus NLIP’s 2.56 ms; on Machine C, it is 25.87 ms versus 2.64 ms.These are 24.5× and 9.8× gaps, respectively, on the same Dataset-19 workload; NLIP’s connection cost stays near 2.6 ms while Python-A2A sits between the two.Source: A. Protocol Architecture Differences · p. 6
  • Message creation takes 0.06–0.15 ms across implementations, so serialization and formatting contribute little to the measured gap.On the tabulated evidence, the divergence comes from connection establishment rather than message processing.Source: A. Protocol Architecture Differences · p. 6
  • On Machine B, NLIP’s 2.56 ms connection overhead is 60 ms lower per hop than A2A-SDK’s 62.71 ms.This is a single-hop, single-machine result; extrapolation to multi-hop chains is a projection, not a measured claim.Source: A. Protocol Architecture Differences · pp. 6–7
  • On Machine B, A2A-SDK’s 62.71 ms connection overhead alone exceeds NLIP’s full 7.18 ms execution time by 8.7× on the Subreddits agent.Source: A. Protocol Architecture Differences · p. 7

B. Implementation Quality Impact

Observed latency depends on implementation and configuration as well as protocol, and the importance of protocol overhead depends on operation granularity. The caching send-phase increase remains unexplained, and sequential runs do not establish behavior under concurrent stress.

B. Implementation Quality Impact

  • Python-A2A is 2.0–2.2× faster than A2A-SDK on the lightweight stage despite following the same protocol.The authors caution that saying “A2A is slower than NLIP” conflates protocol-level and implementation-level factors; Python-A2A performs comparably to NLIP on end-to-end workflows.Source: B. Implementation Quality Impact · p. 7

C. Connection-Caching Condition

  • With caching on Machine B, A2A-SDK totals 19.93 ms and NLIP’s uncached total is 7.24 ms, a 2.75× gap; on Machine C, the gap narrows to 1.27×.Caching reduces connection latency below 1 ms on both machines, but send latency rises to 18.07 ms on B and only 7.82 ms on C; the cause is unresolved and the configuration is not documented well enough for independent reproduction.Source: C. Connection-Caching Condition · p. 7

D. Operation Granularity as Performance Determinant

  • For 700–1000 ms end-to-end workflows, protocol overhead is negligible; for 5–10 ms lightweight operations, it becomes dominant.The selection rule follows operation characteristics: frequent fine-grained coordination favors NLIP, while coarse-grained tasks amortize protocol overhead.Source: D. Operation Granularity as… · p. 7

E. Percentile Variability Under Sequential Execution

  • Because runs were sequential rather than concurrent, percentile spread reflects system and network jitter, not load behavior.The study makes no claims about tail latency or concurrent stress; dedicated stress testing is left to future work.Source: E. Percentile Variability Under… · p. 7
  • Python-A2A still trails NLIP by 4.1–4.3× on the lightweight stage, while cached A2A-SDK reaches parity on the fastest machine.Implementation and configuration can narrow the gap, sometimes to zero, without changing the general picture that NLIP is fastest by default.Source: E. Percentile Variability Under… · p. 7

F. Synthesis: A Multifactor View

For lightweight operations, protocol architecture dominates latency, while operation granularity determines whether protocol overhead matters; connection establishment is the clearest measured divider. Rankings recur across tested datasets and machines, but the scope and causal detail remain bounded.

F. Synthesis: A Multifactor View

  • NLIP leads A2A-SDK by 8.4–9.6× and Python-A2A by 4.1–4.3× on the lightweight stage, while caching narrows A2A-SDK’s gap to parity at scale on the fastest machine.Implementation quality is secondary, worth roughly 2× within the A2A family; protocol overhead dominates work of a few milliseconds but fades into work of hundreds, and load was not characterized.Source: F. Synthesis: A Multifactor View · p. 7

VI. MECHANISMS BEHIND THE CONNECTION GAP

  • A2A-SDK’s connection latency is 24.5× NLIP’s on Machine B and 9.8× on Machine C; separately, it is 62.71 ms versus Python-A2A’s 7.36 ms.Instrumentation measures aggregate connection latency, not its sub-components; the proposed decomposition remains unverified.Source: §VI · pp. 7–8
  • Parsing-cost reduction is only an interpretation, and compounding the Machine B connection gap across longer chains is a projection from a measured two-stage workflow.The instrumentation does not isolate parsing cost, and multi-hop scaling was not measured.Source: §VI · p. 8

A. Statistical Robustness

  • Rankings held across both datasets, and Subreddit-stage confidence intervals did not overlap; the main patterns recurred across tested environments.No interval conclusions are drawn for tail percentiles because the design was sequential.Source: A. Statistical Robustness · p. 8
  • The central claim rests on one lightweight stage in one workflow; 19 queries are modest, and absolute latencies across machines are not directly comparable.Hardware was not matched across machines, and analytical or batch workflows need further testing.Source: A. Statistical Robustness · p. 8

VII. DISCUSSION AND DESIGN TRADE-OFFS

Connection-establishment cost and coordination strategy, rather than message syntax, explain the reported efficiency pattern. The paper relates protocol choice to operation granularity and recommends matching the protocol to latency and governance needs.

VII. DISCUSSION AND DESIGN TRADE-OFFS

  • Protocol efficiency comes primarily from connection-establishment cost and coordination strategy, not message syntax; caching narrows A2A-SDK’s disadvantage to parity at scale on the fastest machine.A2A’s small messages coexist with substantial per-interaction connection overhead; NLIP uses a stateless, low-overhead connection model.Source: §VII · p. 8
  • For operations completing in <50 ms, protocol overhead dominates: NLIP’s lightweight-stage advantage over A2A-SDK is 8.4–9.6×; above 500 ms, implementations cluster more closely at the median.The fixed overhead is amortized as per-call work grows.Source: §VII · p. 8
  • A2A’s deterministic execution, governance, authorization, and enterprise RPC compatibility may matter regardless of its performance characteristics.These requirements can make A2A suitable even where latency is not its advantage.Source: §VII · p. 8

VIII. PROTOCOL SELECTION GUIDELINES

  • Favor NLIP for latency-sensitive coordination and A2A for governance or strict-schema needs; a hybrid user-facing NLIP/backend A2A deployment is proposed but untested.Python-A2A is already near parity with NLIP on end-to-end workflows, so the performance gap may narrow as implementations mature.Source: §VIII · pp. 8–9
  • The paper does not report Cohen’s d because paired per-query data across both datasets was not assembled.Effect-size magnitudes are described qualitatively.Source: §VIII · p. 9

IX. LIMITATIONS AND FUTURE WORK

The study’s conclusions are bounded by its workflow and sequential measurements, while its conclusion links protocol choice to workload and governance needs. It also describes a planned release of materials for reproducing the benchmark.

IX. LIMITATIONS AND FUTURE WORK

  • The evaluation covers one workflow and 19 queries, with sequential execution and a single lightweight coordination stage; concurrent behavior and broader workflow generalization remain untested.The paper also notes that TTFT, ITL, cold-start components, added NLIP security overhead, and long-horizon context-window effects were not measured.Source: §IX · p. 9

X. REPRODUCIBILITY

  • On acceptance, the authors plan to release the benchmark harness, implementations, query sets, raw timing logs, and analysis scripts for recomputing results.The planned repository will record library versions, cached-condition configuration, and machine specifications.Source: §X · p. 9

XI. CONCLUSION

  • 8.4–9.6×: NLIP beat A2A-SDK on the lightweight stage in two environments; cache-optimized A2A-SDK reached parity at scale on the fastest machine.The direction of NLIP’s default advantage was consistent across three environments, though its magnitude varied with hardware.Source: §XI · p. 9
  • Python-A2A is about twice as fast as A2A-SDK yet trails NLIP by default; the ordering suggests, but does not conclusively establish, a protocol-level effect.The study did not test concurrent load or longer agent chains, which it says are needed before the results guide production choice.Source: §XI · p. 10
  • Choose NLIP for latency-sensitive, coordination-heavy systems and A2A for governance-critical or coarse-grained tasks; the proposed hybrid deployment was not tested.The proposed arrangement puts NLIP at user-facing layers and A2A at the backend.Source: §XI · p. 10

Related papers

Loading 2610.04053v1…