PaperlayerView exact revision on arXiv

Source-linked AI summary

A Baseline Assessment for the Agentic Web Technology Stack

Vagner Figueredo de Santana, Diogo Cortiz

arXiv:2610.13703v1cs.AIcs.ETcs.SE

TL;DR

Fragmented, fast-evolving agent technologies leave limited assessment of how protocols, frameworks, and libraries work together in an Agentic Web stack. The paper presents MASTIF and a baseline assessment; results indicate that suitable stacks differ across metrics and tasks, while the assessment is bounded by the models, judges, frameworks, tasks, and protocol testing setup considered.

  • Problem

    Fragmented protocols, frameworks, and tools make it difficult to assess how technologies work together in Agentic Web solutions.

    Source: §1 · p. 2

  • Method

    The paper presents MASTIF for empirical Agentic Web technology-stack assessment and formalizes efficiency and social-value measures.

    Source: §1 · p. 3, §3.2 · p. 8, §3.4 · p. 9

  • Results

    No single Agentic Web stack suffices for all metrics; Qwen2.5-7B, SemanticKernel, and LangChain are identified as interesting starting points for Mind2Web-like tasks.

    Source: §7 · p. 27

  • Takeaways & Limitations

    Technology stacks should be selected and tested for the specific context, tasks, and target metrics, with costs controlled.

    Source: §7 · p. 27

    The assessment is limited by the models, judge, frameworks, and tasks considered, and protocol testing covers message exchange rather than full implementations.

    Source: §7.4 · pp. 28–29

Abstract

from arXiv · show

The fast pace in which protocols, frameworks, and technologies are being proposed for LLM-based agents results in a complex scenario for anyone aiming to create state-of-the-art AI solutions. Without an assessment on how different protocols, frameworks, and libraries work together to compose an Agentic Web solution, developers and organizations face different challenges (e.g., skilling, governance, infrastructure) even before becoming part of the Agentic Web ecosystem. In this context, we propose a system for performing Agentic Web evaluations called MASTIF (Multi-Agent System TestIng Framework) and present a baseline assessment of Agentic Web technologies, including 3 LLMs, 3 protocols, and 4 agent implementation frameworks, against 300 Mind2Web benchmark tasks. Results suggest that selecting a task-specific technology stack may result in gains of task completion rate, efficiency score, and social value stability scores; and reductions in verbosity and time to complete tasks. With the presented results, we expect to present a baseline in terms of metrics and results for this complex and fast evolving scenario for those aiming at designing, assessing, and developing multi-agent systems.

1 Introduction

LLM-based agents extend AI agents with natural-language capabilities and can plan, act, and coordinate in the Agentic Web. Because agent technologies are fragmented and fast-moving, this paper introduces MASTIF and an empirical baseline assessment.

1 Introduction

  • LLM-based agents may model environments, learn their operation, or use complex orchestration for difficult tasks without a single correct answer.The paper uses “LLM-based agents” and “agents” interchangeably.Source: §1 · p. 1
  • The Agentic Web adds an agent-agent interaction layer in which autonomous agents can plan, coordinate, and execute goal-directed tasks alongside human-Web interactions.Agents may also act as Web users, interfaces, or agents in the physical world.Source: §1 · p. 2
  • Fragmented protocols, frameworks, and tools make it difficult for developers and organizations to onboard agentic systems or keep technologies working together.The paper frames its baseline assessment through questions about task completion, efficiency, Web tool-calling, and social-value stability.Source: §1 · p. 2
  • The paper contributes MASTIF, a system for empirical assessment, and a baseline assessment of the Agentic Web technology stack.The paper evaluates technologies in the context of the challenges posed by the fast-evolving ecosystem.Source: §1 · p. 3

2 Related Work

Related work surveys agent capabilities, architectures, and communication protocols, alongside research on benchmarks and multi-agent evaluations. The included protocols address different needs, from connecting tools and data to agent interoperability and payments.

2.1 Types of Agents

  • Agent capabilities range from reflexive action to environment modeling, goal-directed planning, utility-based choice, and learning from feedback or experience.Simple reflex agents lack memory and internal models; model-based reflex agents update an environment representation but lack explicit goal-driven planning.Source: §2.1.1 · p. 3, §2.1.2 · p. 3, §2.1.3 · p. 3

2.2 Architecture

  • AGNTCY proposes an Internet of Agents framework spanning discovery, identity and access control, messaging, observability, and protocol integration.Its discovery component uses OASF to help agents discover and understand other agents’ capabilities.Source: §2.2.1 · p. 4

2.3 Protocols

  • MCP is a context protocol connecting data sources with LLM-powered tools, while A2A supports agent communication and collaboration across platforms and frameworks.MCP uses clients, servers, and JSON-RPC; A2A uses JSON-RPC for multi-agent scenarios.Source: §2.3.1 · p. 4, §2.3.2 · p. 4

2.3.3 Agent Communication Protocol (ACP).

  • ACP provides framework- and infrastructure-independent agent interoperability through a standardized RESTful API supporting modalities, asynchronous communication, streaming, discovery, and long tasks.Its message-based design is agnostic to internal implementations.Source: §2.3.3 · pp. 4–5

2.3.4 Agent Network Protocol (ANP).

  • ANP is a peer-to-peer protocol for agent connections and collaboration, whereas Agora translates natural language into structured protocols for heterogeneous decentralized agents.ANP is described as an open, secure collaboration network; Agora maps a user’s task to protocols required by agents.Source: §2.3.4 · p. 5, §2.3.5 · p. 5

2.3.5 Agora Protocol.

  • AP2 targets compliant agent-led payments across platforms and can extend A2A and MCP.Its Intent Mandate and Cart Mandate support comparison of the initial human-led task with the product or service purchased by an agent.Source: §2.3.6 · p. 5

2.3 Protocols

  • A cited adoption roadmap sequences MCP for tool access, ACP for multimodal messaging, A2A for collaborative execution, and ANP for decentralized marketplaces.The authors of the roadmap argue that “no single protocol suffices across all contexts” [22].Source: §2.3.6 · p. 5

2.4 Agentic Frameworks

The paper selects four Python frameworks for comparing multi-agent system implementation. It reviews Web-task benchmarks and evaluation practices, including challenges in assessing servers and managing long conversations.

2.4 Agentic Frameworks

  • The study selects LangChain, CrewAI, LangGraph, and Semantic Kernel as popular, LLM-agnostic Python frameworks with built-in tools and varied workflow paradigms.The paper focuses on Python because it is described as the most popular language.Source: §2.4 · p. 5

2.5 Benchmarks for Web Tasks

  • WebArena covers 812 long-horizon tasks, WebShop provides over 12,000 product-search instructions, and Mind2Web encompasses over 2,000 tasks across 137 websites and 31 domains.These benchmarks address different Web-task settings, from routine Internet tasks to simulated shopping and generalist Web agents.Source: §2.5 · pp. 5–6

2.6 MAS Evaluations

  • MCP server evaluations report differences in handling large contexts and prompting patterns, unknown client or model identity, and 755 tool-name collisions for “search.”For multi-agent evaluation, LLM-as-a-judge is described as effective and scalable when test cases have clear answers; long conversations may require compression and memory mechanisms.Source: §2.6 · p. 6
  • Prompting guidance for multi-agent systems emphasizes delegation, scaling effort to query complexity, tool selection, and parallel tool calling.The cited guidance also recommends starting broad and then narrowing the research process.Source: §2.6 · p. 6

3 Problem Definition

The section formalizes how technology stacks are evaluated through task completion, efficiency, tool use, and social-value stability. It defines task completion at the task level and efficiency as a profile spanning completion, convergence time, and verbosity.

3 Problem Definition

  • The framework evaluates technology stacks through task completion, efficiency, tool-use success, and social-value stability.The metrics are intended to support ranking stacks and identifying promising Agentic Web technology directions.Source: §3 · p. 7, §3.2 · p. 8, §3.3 · p. 8

3.1 Task Completion

  • Task completion checks whether an output satisfies task instructions, using an LLM judge or human assessment, and averages completion across tasks.The success predicate is evaluated for each task under a technology stack.Source: §3.1 · p. 7

3.2 Efficiency

  • Efficiency is a profile of task completion, convergence time, and verbosity, with verbosity counting reasoning-trace and output tokens per task.Convergence time runs from the orchestrator receiving the prompt until the system produces its final output.Source: §3.2 · p. 8

3.3 Tool Use

  • Tool-use success rate measures the share of tool calls whose identifiers are available and whose execution succeeds.Calls that are unsuccessful or hallucinated count as failures.Source: §3.3 · p. 8

3.4 Social Values

  • Social-value stability tracks how an agent’s social values change from the prompt through internal reasoning to its final output.A classifier assigns social-value labels to prompt sentences, and the stack-level score is measured across the test set.Source: §3.4 · p. 9

4 Experiment Setup

The experiment compares agent technology stacks using selected models, protocols, frameworks, and a stratified sample of Mind2Web tasks. MASTIF runs the evaluations and logs metrics, while the analysis compares stacks across task completion, efficiency, tool use, and social-value stability.

4.1.1 Protocols.

  • The protocol comparison includes MCP for routing tool calls and A2A for agent-to-agent interaction, alongside a standard condition without protocol directives.The study notes that ACP was being merged with A2A when the protocols were selected.Source: §4.1.1 · p. 9

4.1.2 Frameworks.

  • The selected frameworks are LangGraph, LangChain, CrewAI, and Semantic Kernel, chosen for compatibility and integration with the studied protocols.The frameworks are described as LLM-agnostic.Source: §4.1.2 · p. 9

4.1.3 Benchmark.

  • The benchmark uses a stratified sample of 300 Mind2Web tasks spanning websites and domains, including e-commerce, travel planning, and enterprise knowledge assistance.Mind2Web was selected for assessing generalist web agents on real-world websites.Source: §4.1.3 · p. 10

4.1.4 Large language Models.

  • The study selects Mistral-7B-Instruct-v0.1, Qwen2.5-7B, and Llama-4-Scout-17B-16E-Instruct, with gpt-4o-mini judging responses.The judge is from a different model family to prevent preference leakage while controlling costs.Source: §4.1.4 · p. 10

4.1.5 MAS Test Framework.

  • MASTIF supports multiple tasks, prompt templates, inference providers, models, and implementation frameworks, and logs latency, token use, reasoning steps, and other metrics.The framework also includes built-in and custom tools for web interaction needed by Mind2Web tasks.Source: §4.1.5 · pp. 10–11, Figure 1 · p. 10

4.2 Experiment Design and Procedure

  • The experiment evaluates 10,800 executions from 300 of 2,350 Mind2Web tasks across three models, four frameworks, and three protocols.Stratification was used to include different domains and task complexities; the workflow has navigating, observing, interacting, and final reporting nodes.Source: §4.2 · pp. 12–13, Table 1 · p. 13

4.3 Result Analysis

  • The analysis compares task completion, convergence time, verbosity, efficiency, tool-use success, and social-value stability across stacks.It plans normality testing followed by ANOVA or Kruskal-Wallis and Dunn’s post-hoc tests, as appropriate.Source: §4.3 · p. 13

5 Results

Across the tested metrics, different technology stacks performed best for different objectives: Qwen2.5-7B with standard messages and LangChain is a promising all-around efficiency combination, while other stacks suit specific trade-offs.

5.1 Task Completion Rate

  • Qwen2.5-7B (mean rank=6095.99), standard (mean rank=5898.66), and SemanticKernel (mean rank=6039.33) were the best-performing stack components for task completion.The reported result is under the test conditions studied.Source: §5.1 · p. 14, Figure 2 · p. 16

5.2.1 Convergence Time.

  • Qwen2.5-7B (mean ranks=4974.57), MCP (mean rank=4874.13), and LangChain (mean rank=4510.52) were the best-performing components for lowest convergence time.The figure groups convergence time in seconds by model, protocol, and framework.Source: §5.2.1 · p. 17, Figure 3 · p. 17

5.2.2 Verbosity.

  • Qwen2.5-7B (mean ranks=3713.37), standard (mean rank=4011.82), and LangChain (mean rank=5113.11) were the best-performing components for lowest verbosity.The figure groups verbosity in tokens by model, protocol, and framework.Source: §5.2.2 · p. 18, Figure 4 · p. 18

5.2.3 Efficiency.

  • Qwen2.5-7B, standard, and LangChain are a promising combination for high task completion with low convergence time and verbosity.The Pareto analysis identifies other combinations for different goals, including Qwen2.5-7B with A2A and LangChain or SemanticKernel for balancing low convergence time and verbosity.Source: §5.2.3 · p. 18, Figure 5 · p. 20

5.3 Tool Use

  • Only Qwen2.5-7B externalized tool use; under that condition, MCP (mean rank=1856.70) and SemanticKernel (mean rank=1856.14) promoted tool use and obtained the best results.Other models' internal answers or unexternalized tool use affected the analysis across combinations.Source: §5.3 · p. 21, Figure 6 · p. 21

5.4 Social Values

  • Mistral-7B (mean rank=6100.76), A2A (mean rank=5460.16), and SemanticKernel (mean rank=5778.08) were the best-performing components for social-value stability.Social values were more stable in reasoning steps than between input tasks and final answers; reported reasoning-step values include trust, integrity, safety, and accountability.Source: §5.4 · pp. 22–23, Figure 7 · p. 23, Figure 8 · p. 24

6 Discussion

Across the evaluated conditions, the best-performing stack varied by metric: Qwen2.5-7B with LangChain and standard conditions led task completion and efficiency, while other stacks led tool use and social value stability.

6.1 Task Completion

  • Qwen2.5-7B, LangChain, and standard conditions represented the best-performing baseline for task completion under the studied conditions.Task completion was also impacted under Qwen2.5-7B and MCP conditions.Source: Table 6 · p. 26, §6.1 · p. 26

6.2 Efficiency

  • Qwen2.5-7B, LangChain, and standard conditions offered the best balance of task completion, low convergence time, and low verbosity.The Pareto assessment also identified Qwen2.5-7B with Semantic Kernel under standard conditions for better task completion, and A2A combinations with Semantic Kernel or LangChain for low convergence time and verbosity.Source: Table 6 · p. 26, §6.2 · p. 26
  • Llama-4-Scout-17B-16E with Semantic Kernel was an efficiency outlier, taking longer to converge while using fewer tokens than other conditions.Mistral-7B with standard conditions and LangChain or LangGraph combined low convergence time with average task completion.Source: §6.2 · p. 26

6.3 Tool Use

  • Within Qwen2.5-7B conditions, Semantic Kernel and MCP used the provided tools more, and MCP had the best tool-use success rates.Tool use was externalized only by Qwen2.5-7B, so these results do not compare models at the stack level.Source: §6.3 · p. 27

6.4 Social Value

  • Mistral-7B, A2A, and Semantic Kernel were more stable in processing, reasoning, and answering tasks.Semantic Kernel obtained the best-performing social value stability scores.Source: Table 6 · p. 26, §6.4 · p. 27

7 Conclusion

The conclusion argues that no single Agentic Web stack suits every metric and recommends systematic, context-sensitive assessment. It also identifies practical testing constraints and outlines extensions to MASTIF studies.

7 Conclusion

  • No single Agentic Web technology stack suffices for all metrics, so contexts, goal metrics, and tasks need systematic consideration.Qwen2.5-7B with Semantic Kernel and LangChain are suggested as starting points for tasks similar to Mind2Web.Source: §7 · p. 27
  • MASTIF proposes assessing social value stability across prompts, reasoning steps, and final answers; Semantic Kernel obtained the best-performing stability scores.The proposed Jaccard Pairwise score supported analysis from task input to output.Source: §7 · p. 27

7.1 Lessons Learned

  • MAS development and testing are complicated by multiple failure points, low observability, nondeterminism, and fragile tool use.Guidance-like tool descriptions prompted tool use in the experiments but increased input-token burden, potentially affecting token cost.Source: §7.1 · p. 28
  • Experiments combining models, protocols, and frameworks can produce large logs, so running each model separately can ease log and result assessment.MASTIF handles several interruption types, including quota errors and inference-provider issues.Source: §7.1 · p. 28

7.2 Reproducibility

  • The experiment is reproducible using the available Mind2Web dataset, open-source MASTIF, and the experiment YAML with parameters, models, prompts, and frameworks.The authors state these materials can support reproduction or expansion of the experiment.Source: §7.2 · p. 28

7.3 Ethics Statement

  • The study used the open Mind2Web dataset, with tasks stratified to cover different tasks, complexities, domains, and Web contexts, and involved no human participants.Source: §7.3 · p. 28

7.4 Limitations

  • The assessment is limited by its numbers of models, judges, frameworks, and tasks; testing protocol message exchange did not evaluate full implementations.Adding tasks, models, or judges increases execution volume and cost; representative services with pre-built servers for all studied protocols were not available to the authors.Source: §7.4 · pp. 28–29

7.5 Next Steps

  • Next steps include varying MAS setups, tools, max_steps, temperature, and frameworks, and testing technologies such as WebMCP and servers supporting full MCP and A2A exchanges.Source: §7.5 · p. 29

A Agentic Frameworks

The appendix table lists LLM-agnostic agent frameworks in Python surveyed in the study.

A Agentic Frameworks

B Benchmarks

The appendix table lists benchmarks related to LLM-based agents surveyed in the study.

B Benchmarks

  • Table 8 lists benchmarks related to LLM-based agents surveyed in this study.Source: B Benchmarks · p. 32

C Judge Model Rubric

The judge rubric scores agent responses on task understanding, adherence, and completion, with dimension-specific anchors ranging from 0.0 to 1.0. It requires a JSON-only evaluation with a score and rationale for each dimension.

C Judge Model Rubric

  • The rubric evaluates task understanding, task adherence, and task completion, assigning each a score from 0.0 to 1.0 with a brief rationale.Understanding concerns goals, constraints, and required output; adherence concerns on-task reasoning; completion concerns whether the final response enables success.Source: C Judge Model Rubric · p. 32
  • Understanding scores range from 1.0 for capturing all goals, constraints, and outputs to 0.5 for missing key requirements.Intermediate anchors are 0.7 for capturing the main goal while missing minor constraints or details.Source: C Judge Model Rubric · p. 32
  • Adherence scores range from 1.0 for fully relevant steps to 0.5 for mixed relevance with important off-track reasoning.The 0.7 anchor allows minor irrelevant or redundant steps.Source: C Judge Model Rubric · p. 33
  • Completion scores range from 1.0 for correct, actionable completion to 0.0 for failure or an incorrect result.The 0.7 anchor describes likely completion with minor gaps or inefficiencies.Source: C Judge Model Rubric · p. 33
  • The judge must return only a JSON object with task_understanding, task_adherence, and task_completion objects, each containing score and rationale.The prompt prohibits extra text, markdown, or explanations outside the JSON object.Source: C Judge Model Rubric · p. 33

D Generative AI Usage Statement

The manuscript states that GenAI was not used in its preparation, but was used after writing for an initial review and recommendations before submission.

D Generative AI Usage Statement

  • GenAI was not used to prepare the manuscript; after writing, it generated an initial review and recommended improvements before submission.The review prompt requested reduced-format bullet feedback on the attached PDF against a CFP.Source: D Generative AI Usage Statement · p. 33
  • The initial review was prompted to produce a bullet list of feedback using the CFP criteria.The prompt caption identifies this as the review prompt used before submission.Source: D Generative AI Usage Statement · p. 33
Loading 2610.13703v1…