AI-NativeBench: An Open-Source White-Box Agentic Benchmark Suite for AI-Native Systems

• AI-Native systems combine probabilistic agentic behavior with standardized MCP and A2A interoperability, creating opaque failures beyond traditional deterministic engineering.Agentic services support planning, reasoning, and tool execution, but can also hallucinate or enter reasoning loops.Source: §1 · p. 2
• Black-box benchmarks measure outcomes or action sequences but cannot distinguish framework errors, protocol mismatches, and model hallucinations.This leaves system-level engineering characteristics and internal bottlenecks unobserved.Source: §1 · p. 2
• AI-NativeBench provides an open-source, application-centric white-box suite spanning eight applications, three domains, MCP and A2A protocols, and architectures from monolithic agents to heterogeneous multi-agent systems.Its trace-first methodology treats agentic spans as first-class citizens through OpenTelemetry, enabling attribution of latency and errors across monolithic, MCP, and A2A variants.Source: §1 · pp. 2–3
• Across seven LLMs and 21 application variants, lightweight models often adhere to rigid protocols better than flagship models, while reasoning models may bypass tools despite producing deeper insights.This parameter paradox and content-process divergence challenge the assumption that greater model capability guarantees stronger workflow adherence.Source: §1 · p. 2
• LLM inference consumes 86.9%–99.9% of execution time, making protocol overhead secondary in most high-capability workflows and shifting latency attention to critical-path stragglers.The study also finds failed workflows consume more resources because agents exhaust retry budgets on doomed workflows, while A2A synchronization incurs context redundancy.Source: §1 · p. 3
• The benchmark and dataset support a shift from measuring raw model capability toward diagnosing reliable, cost-effective AI-Native systems.The paper is positioned as a reproducible foundation for evaluating correctness, performance, and economics.Source: §1 · p. 3

2.1 From Cloud-Native to AI-Native

• AI-Native architecture replaces deterministic microservices with autonomous agentic services, using MCP for tool communication and A2A for inter-agent coordination.End-to-end functionality emerges from dynamic standardized interactions, introducing protocol-level failure modes and bottlenecks.Source: §2.1 · p. 4

2.2 AI-Native Application Observability

• Metrics summarize what happened but do not explain why, while pure traces are semantically blind and trajectories are performance-blind.A fused trace is therefore proposed to measure semantic behavior and performance overheads together.Source: §2.2 · pp. 4–5
• AI-Native observability must combine semantic trajectories with technical traces because trajectories expose decision failures while traces expose latency bottlenecks.The unified trace can reveal both Agent B’s skipped tool call and Agent C’s Hotel_API() bottleneck, including 6s and 4.9s durations.Source: §2.2 · p. 5

2.3 Related Work

• Existing agent benchmarks remain task-centric and black-box, lacking realistic MCP/A2A architectures and quantitative visibility into component-level performance and reliability.The identified gap motivates application-centric white-box benchmarking grounded in mature cloud-native tracing principles.Source: §2.3 · pp. 6–7

3 AI-Native System Benchmark Suite

• AI-NativeBench comprises eight applications across three domains, spanning one to five agents and up to nine tools.The suite supports controlled comparisons across lightweight utilities and larger engineering-intensive systems.Source: §3 · p. 7
• Overall, the suite supports systematic studies of model selection, architectural and protocol trade-offs, and cost–performance trade-offs in realistic AI-Native deployments.These comparisons are enabled by modular applications instantiated under multiple architectural variants.Source: §3 · p. 7

3.1 Design Principles

• The benchmark keeps topology complexity bounded, reflecting production systems where most execution trajectories have fewer than 10 steps and Book Writer reaches five agents and nine tools.This design avoids artificial complexity while retaining focused multi-agent collaboration.Source: §3 · p. 7, §3.1 · p. 8
• Unified traces combine semantic trajectories with technical execution data to diagnose deterministic and stochastic failures during agentic execution.OpenTelemetry instruments agent services and MCP/A2A interactions by default.Source: §3 · p. 7, §3.1 · p. 8
• Architectural heterogeneity enables controlled comparisons among native framework deployments, MCP tool invocation, A2A communication, and heterogeneous multi-framework systems.Researchers can vary agents, models, protocols, and frameworks to study performance, reliability, and latency trade-offs.Source: §3 · p. 7, §3.1 · p. 8

3.2 Communication & Collaboration Task

• Email Responder compares native and MCP-based two-agent email processing, isolating protocol-induced overhead from agent logic.The MCP variant decouples Gmail access through standardized external services.Source: §3.2.1 · p. 9

3.3 Software & Data Engineering

• Software and data engineering workloads progress from MCP-based Markdown validation to multi-agent code execution, heterogeneous landing-page generation, and governed Text-to-SQL retries.SQL Assistant uses a four-agent workflow with compliance checking, revision feedback, A2A handoffs, and SQLite execution.Source: §3.3.1 · p. 10, §3.3.2 · p. 10, §3.3.3 · p. 10

3.4 Content Generation

• Content-generation workloads scale from iterative social-media correction to hierarchical book production with parallel chapter processing and five-agent quality assurance.Social Media Manager uses an Analyze → Generate → Review → Retry loop, while Book Writer applies map-reduce-style chapter execution.Source: §3.4.2 · p. 11, §3.4.1 · p. 11, §3.4 · p. 11

4 Evaluation Methodology

• AI-NativeBench uses unified OpenTelemetry instrumentation to inspect internal execution states and diagnose engineering behavior beyond black-box outcome scores.The benchmark treats agents as complex distributed systems rather than model inference endpoints.Source: §4 · p. 11
• The evaluation asks how model capability affects workflow adherence, latency anatomy, and token costs, including whether intelligence produces outcome-process divergence.These questions cover behavioral correctness, performance overhead, and economic efficiency.Source: §4 · pp. 11–12

4.1 RQ1: Behavioral Correctness

• Behavioral correctness combines verified outcome scores for deterministic applications with pass rates for open-ended workflows that complete without exceptions and satisfy structural constraints.Outcome scores measure verified assertions, while pass rate emphasizes operational survivability and engineering robustness.Source: §4.1 · p. 12
• Retry rate measures execution friction because self-correction indicates difficulty producing correct actions in one shot and increases latency and token costs.The metric records the percentage of runs triggering self-correction at least once.Source: §4.1 · pp. 12–13
• The trace-first pipeline constructs a deterministic golden trace from orchestration configuration and task requirements, excluding stochastic reasoning and allowing equivalent unconstrained tool orders.The reference path specifies required agent spans and tool signatures while avoiding redundant calls and extra loops.Source: §4.1 · pp. 12–13
• Exact-match and any-order trace comparisons expose alignment gaps, while tool precision and recall identify unnecessary calls and omitted required tools.These metrics compare the actual trace with the golden trace and evaluate correctness and completeness of tool usage.Source: §4.1 · pp. 13–14

4.2 RQ2: Performance Overhead

• End-to-end latency is analyzed by agent role to distinguish computationally intensive agents from inefficient coordination steps.Distributed traces provide per-agent latency contributions to the overall response time.Source: §4.2 · p. 14
• Non-LLM latency isolates infrastructure cost from token generation and is decomposed into server, A2A, framework, and tool components.The components represent hosting overhead, protocol serialization and network delays, agent-library logic, and external-tool execution.Source: §4.2 · p. 14
• The benchmark compares MCP, A2A, and H-A2A with the same task and model, using paired latency differences and trace-level per-agent deltas to attribute protocol effects.Runs sharing the same input provide the basis for paired comparisons.Source: §4.2 · p. 14

4.3 RQ3: Token Economics

• Token usage is analyzed through retry penalties and failure efficiency because pay-per-token execution makes performance-per-cost as important as accuracy.The analysis conditions token consumption on operational outcomes.Source: §4.3 · p. 14
• Retry penalty is measured by comparing successful runs with retries against successful runs without retries to quantify self-healing resources.This isolates the additional token expenditure required to recover from intermediate errors.Source: §4.3 · p. 14
• Failure efficiency examines whether failed runs stop early or exhaust context windows and retry budgets, identifying economic waste without value generation.The results support decisions about budget-aware circuit breakers and recovery-budget trade-offs.Source: §4.3 · p. 14

4.4.1 Testbed Environment.

• Experiments run on isolated Huawei Cloud ECS instances with 16 vCPUs, 64 GiB memory, Ubuntu 24.04.2, and a single-job policy.The isolation policy prevents co-located workloads from interfering with CPU or memory bandwidth.Source: §4.4.1 · p. 15
• Applications use Python 3.10, version-locked dependencies, CrewAI, LangGraph, AutoGen, MCP, A2A, and unified OpenTelemetry-compatible tracing.The observability stack captures both LLM generations and tool invocations.Source: §4.4.2 · p. 15

4.4.3 Models Evaluated.

• Seven proprietary and open-source general-purpose or reasoning-enhanced LLMs are evaluated through a unified interface with identical prompts, instructions, and decoding settings.The models include GPT-5, GPT-4o-mini, DeepSeek-V3.1, DeepSeek-R1, Gemini-2.5-flash variants, and Qwen3-235b.Source: §4.4.3 · p. 15

4.4.4 Workload.

• Workloads use diverse authentic data sources, ground-truth outputs where available, and typically balanced datasets of N=60 to manage multi-agent execution cost.The applications span communication, software and data engineering, and content generation.Source: §4.4.4 · p. 15

5.1.1 Impact of Parameter Paradox.

• GPT-4o-mini nearly matches GPT-5 in task completion while achieving higher workflow alignment and ground-truth quality, exemplifying the parameter paradox.GPT-4o-mini records pass_rate ≈0.99 versus GPT-5’s ≈1.00, but any_order_match 0.64 versus 0.35 and average_score 0.67 versus 0.55.Source: §5.1.1 · pp. 17–18
• GPT-5 offers higher tool-calling precision but lower recall, making it effective for compact deterministic tasks while risking omitted actions in verification-intensive workflows.Its precision is 0.94 versus 0.85, while recall is 0.90 versus 0.95; structured Game Builder tasks reach exact_match 1.00.Source: §5.1.1 · p. 18

5.1.2 Impact of Reasoning Mode.

• Reasoning mode creates content-process divergence: DeepSeek-R1 improves decision depth but reduces adherence and stability by internalizing protocol interactions.DeepSeek-R1’s any_order_match falls to 0.31 versus 0.65 for DeepSeek-V3.1, precision to 0.80 versus 0.87, and retry_rate rises to 0.22 versus 0.11.Source: §5.1.2 · p. 19
• Gemini reasoning improves validation adherence and recall but increases false positives and retries, reducing average_score from 0.14 to 0.08.In Markdown Validator, exact_match rises 0.13 → 0.26, any_order_match 0.13 → 0.33, and recall 0.73 → 0.88, while retry_rate rises 0.06 → 0.21.Source: §5.1.2 · p. 19

5.1.3 Impact of Architectural Decoupling.

• MCP preserves behavioral consistency and reduces retries through strict tool contracts, whereas A2A lowers alignment despite stable completion by fragmenting semantic continuity.MCP slightly raises pass_rate from 0.99 to 1.00 and reduces retry_rate from 0.04 to 0.03; A2A lowers any_order_match from 0.29 to 0.25 while recall rises from 0.81 to 0.84.Source: §5.1.3 · p. 20
• Architectural decoupling is workload-dependent: isolation helps lightweight models on noise-sensitive tasks but disrupts dependency-heavy workflows by severing execution context.GPT-4o-mini improves SQL Assistant alignment from 0.12 to 0.53, while DeepSeek-R1’s Landing Page Generator average_score collapses from 0.58 to 0.08 when moving to A2A.Source: §5.1.3 · pp. 20–21
• H-A2A increases outcome quality while collapsing procedural alignment, producing functionally effective but opaque execution paths.Across applications, average_score rises from 0.31 to 0.45 while any_order_match falls from 0.25 to 0.19; in SQL Assistant, Gemini-2.5-flash reaches average_score 0.55 with any_order_match 0.00.Source: §5.1.3 · p. 21

5.2.1 Application-Level Breakdown.

• DeepSeek-R1 averages 589.60 s versus 191.58 s for DeepSeek-V3.1, revealing a roughly 3× reasoning-model slowdown and a long-tail latency problem.GPT-5 provides a 45.52 s global mean, while DeepSeek-R1 is also approximately 24× slower than GPT-5 in Email Responder.Source: §5.2.1 · pp. 22–23
• LLM computation consumes 86.9%–99.9% of execution time, leaving protocol overhead statistically secondary in most configurations.DeepSeek-R1 reaches 99.99% of latency in H-A2A Recruitment Assistant and Landing Page Generator variants.Source: §5.2.1 · pp. 23, 25

5.2.2 Agent-Level Breakdown.

• A single straggler agent often determines end-to-end latency, so trace-based optimization should target dominant nodes rather than uniformly improving all agents.Stragglers consume up to 88.2% of execution time; the Content Editor accounts for nearly 74% in Landing Page Generator.Source: §5.2.2 · pp. 25–27
• H-A2A reduces mean latency by 6.5%–31.3% across projects by limiting synchronized reasoning loops among homogeneous agents.Recruitment Assistant falls by 156.82 s, while Qwen3-235b decreases from 1476.89 s to 973.11 s in the heterogeneous setup.Source: §5.2.2 · p. 27
• Heterogeneity is not universally beneficial: GPT-5 incurs latency penalties of 33.7% in SQL Assistant and 30.3% in Landing Page Generator.Context fragmentation becomes overhead when frontier-model reasoning is already efficient.Source: §5.2.2 · p. 27

5.2.3 Infrastructure-Level Breakdown.

• Infrastructure costs invert across architectures: A2A is protocol-dominant, while H-A2A is tool-dominant.A2A protocol overhead averages ≈66.51% of non-LLM latency, whereas H-A2A tool execution accounts for ≈57.05% versus 33.62% for A2A protocol costs.Source: §5.2.3 · p. 28
• CrewAI imposes the highest framework overhead at ≈64.8%, compared with 25.9% for LangGraph and 9.3% for AutoGen, creating a latency floor for lightweight models.This cost is masked by slow reasoning models but matters in high-frequency control loops using models such as GPT-4o-mini.Source: §5.2.3 · p. 28

5.3.1 Cost of Self-Healing.

• Retry success averages about 83k tokens versus 48k for direct success, an approximately 70% increase.The premium grows when self-correction loops are triggered.Source: §5.3.1 · p. 29
• Self-healing can multiply token usage, reaching 19.9× for Markdown Validator and 15.4× for Email Responder under Gemini-2.5-flash-nothinking.SQL Assistant reaches 3.1× under H-A2A with DeepSeek-R1, while Social Media Manager reaches 3.6× under A2A.Source: §5.3.1 · p. 29

5.3.2 Expensive Failure Pattern.

• Recursive failure workflows inflate token consumption by roughly 195%–738%, with Social Media Manager reaching 337k tokens for DeepSeek-R1 in H-A2A.The system may exhaust retry budgets without guaranteeing recovery.Source: §5.3.2 · pp. 29–30
• A Social Media Manager failure consumes 36,884 tokens across five generation-and-critique iterations versus 5,355 tokens for direct success.Repeated verifier rejection drives the recursive cost accumulation.Source: §5.3.2 · p. 29, Figure 28 · p. 30
• Linear workflows can exhibit fail-fast behavior, with failures costing 44.6% less than direct success for DeepSeek-R1 in Landing Page Generator A2A.Gemini-2.5-flash-nothinking failures in the MCP variant cost 85.2% less than direct success.Source: §5.3.2 · p. 30

5.3.3 Protocol Cost of A2A.

• A2A generally increases token use despite limited infrastructure latency, with Gemini-2.5-flash rising about 70% in SQL Assistant and DeepSeek-V3.1 rising 81.6% in Landing Page Generator.The increases are associated with repeated transmission of complete schemas and other semantic context.Source: §5.3.3 · p. 30
• A2A’s context redundancy overhead exceeds 80% in some settings because isolated agents repeatedly receive restated state information.This token inflation arises from preserving semantic consistency across distributed agents.Source: §5.3.3 · pp. 30–31

6.1.1 Implication 1: Capability-Aware Orchestration for AI-Native Systems.

• Adaptive orchestration should pair adherence-strong models with capability-dense specialists instead of assigning models statically.GPT-4o-mini can coordinate protocol-faithful execution, while GPT-5 and DeepSeek-R1 handle complex leaf tasks.Source: §6.1.1 · pp. 31–32, §6.1 · p. 31
• Reasoning mode should be governed at runtime because it can increase latency, token consumption, and the risk of costly failures.A complexity classifier is proposed to activate reasoning only when correctness depends on it.Source: §6.1.1 · p. 32
• Protocol selection should adapt to workflow requirements, using MCP for bounded reliable tasks and A2A where autonomous exploration is preferable.Tracing should also enforce tool-use checks so correct outputs produced through incorrect sequences are rejected.Source: §6.1.1 · p. 33

6.1.2 Implication 2: Straggler-Centric Performance Engineering.

• Inference consumes more than 99% of execution time, shifting performance engineering from network optimization toward profiling and accelerating critical-path stragglers.Tracing should expose agent and tool spans to support interventions such as model swapping, context pruning, or parallelization.Source: §6.1.2 · pp. 33–34
• Distributed micro-agent architectures can improve fault isolation because inference costs overshadow network latency, but context redundancy remains a financial constraint.Architectural boundaries should therefore follow logical domains and token budgets rather than physical proximity.Source: §6.1.2 · p. 33
• Heavyweight orchestration frameworks can add substantial serialization delay, so latency-sensitive systems should favor lightweight control planes and heterogeneous topologies.CrewAI is contrasted with LangGraph, while mixed-framework designs may disrupt synchronized reasoning loops.Source: §6.1.2 · p. 34

6.1.3 Implication 3: Budget-Aware Governance for AI-Native Systems.

• AI-Native resilience should replace mechanical retries with resource-adaptive planning, semantic circuit breakers, and economic fail-fast policies.These mechanisms limit token expenditure by tracking budget state, verifying intermediate logic, and restarting when information gain stalls.Source: §6.1.3 · pp. 34–35

6.2 Threats to Validity

• The findings may not generalize cleanly because LLM behavior is nondeterministic, model APIs can change silently, and benchmarks may underrepresent specialized domains.The study mitigates drift by locking versions where possible, but notes limited coverage of legal and medical counseling.Source: §6.2 · p. 35

• AI-NativeBench addresses the insufficiency of black-box benchmarks by evaluating probabilistic agentic services through distributed traces and MCP/A2A protocols.The benchmark targets system behavior that final outcomes alone cannot explain.Source: §7 · p. 35
• The study finds a parameter paradox, pervasive inference dominance that makes network overhead secondary, and self-healing mechanisms that can multiply costs.These findings challenge assumptions about model scale, performance bottlenecks, and resilience in AI-Native systems.Source: §7 · p. 35
• The paper advocates shifting evaluation from raw reasoning capability toward system-level engineering characteristics using agentic spans as first-class trace entities.The benchmark and dataset are presented as an open-source foundation for further AI-Native systems research.Source: §7 · p. 36