PaperlayerView exact revision on arXiv

Source-linked AI summary

An Empirical Study of Testing Practices in Open Source AI Agent Frameworks and Agentic Applications

Mohammed Mehedi Hasan, Hao Li, Emad Fallahzadeh, Gopi Krishnan Rajbahadur, Bram Adams, Ahmed E. Hassan

arXiv:2509.19185v3cs.SEcs.ET

TL;DR

Benchmarks evaluate FM-based agents on tasks, but there is limited understanding of how developers test their internal correctness during development. We analyze testing practices across 39 frameworks and 439 agentic applications and map them to canonical components. The findings show reliance on adapted traditional tests and a pronounced testing blind spot for prompts.

  • Problem

    Task benchmarks provide task-level evaluations, but evidence about how developers verify internal correctness of FM-based agents during development is limited.

    Source: §1 · pp. 2–3

  • Method

    We conduct an empirical study of unit-testing practices in 39 agent frameworks and 439 agentic applications and map observed components to a canonical architecture.

    Source: §1 · p. 4, §10 · p. 49

  • Results

    Resource Artifacts account for 29.7% of framework tests and 40.1% of application tests, while Trigger prompts appear in around 1% of test functions.

    Source: §1 · p. 4

  • Takeaways & Limitations

    Practitioners predominantly adapt traditional testing strategies, while low use of domain-specific methods such as DeepEval indicates a gap in awareness and usability.

    Source: §10 · p. 49

    LLM-as-a-Judge testing incurs token overhead and remains probabilistic and bias-prone, requiring periodic human oversight.

    Source: §8.1.1 · p. 42

Abstract

from arXiv · show

Foundation model (FM)-based AI agents are rapidly gaining adoption across diverse domains, but their inherent non-determinism and non-reproducibility pose testing and quality assurance challenges. While recent benchmarks provide task-level evaluations, there is limited understanding of how developers verify the internal correctness of these agents during development. To address this gap, we conduct the first large-scale empirical study of testing practices in the AI agent ecosystem, analyzing 39 open-source agent frameworks and 439 agentic applications. We identify ten distinct testing patterns and find that novel, agent-specific methods like DeepEval are seldom used (around 1%), while traditional patterns like negative and membership testing are widely adapted to manage FM uncertainty. By mapping these patterns to canonical architectural components of agent frameworks and agentic applications, we uncover a fundamental inversion of testing effort: deterministic components like Resource Artifacts (tools) and Coordination Artifacts (workflows) consume over 70% of testing effort, while the FM-based Plan Body receives less than 5%. Crucially, this reveals a critical blind spot, as the Trigger component (prompts) remains neglected, appearing in around 1% of all tests. Our findings offer the first empirical testing baseline in FM-based agent frameworks and agentic applications, revealing a rational but incomplete adaptation to non-determinism. To address it, framework developers should improve support for novel testing methods, application developers must adopt prompt regression testing, and researchers should explore barriers to adoption. Strengthening these practices is vital for building more robust and dependable AI agents.

1 Introduction

FM-based agents combine model-driven behavior with tools, planning, and memory, but task benchmarks do not establish how developers verify internal correctness during development. This study examines testing practices and maps them to a canonical architecture, finding both adapted traditional verification and an imbalance in component-level attention.

1 Introduction

  • Benchmark success is an incomplete proxy for real-world reliability, while systematic evidence about how practitioners test FM-based agent frameworks and applications is lacking.The study distinguishes task-level evaluation from software testing, which verifies intended behavior as code evolves and exposes newly introduced faults.Source: §1 · pp. 2–3
  • Rapid architectural change makes durable testing strategies difficult, motivating the mapping of evolving concrete components to a stable canonical model.The study uses that model to assess how testing patterns relate to underlying components.Source: §1 · pp. 3–4
  • We analyze unit-testing practices in 39 agent frameworks and 439 agentic applications, then map observed components to a conceptual architecture to identify component-specific patterns.The study addresses two questions: which testing practices practitioners use and how testing effort maps to agent architecture.Source: §1 · pp. 3–4
  • Practitioners adapt familiar patterns such as membership, mock assertion, and negative testing to handle FM non-determinism, while novel methods such as DeepEval and Hyperparameter Control see very low adoption.The paper presents this split as a strategy for avoiding the fragility of strict assertions.Source: §1 · pp. 3–4
  • Testing concentrates on deterministic infrastructure: Resource Artifacts account for 29.7% of framework tests and 40.1% of application tests, while Trigger prompts appear in around 1% of test functions.The study identifies 13 canonical components receiving unit-testing attention and describes the prompt interface as a critical blind spot.Source: §1 · p. 4
  • The study contributes a curated dataset, a taxonomy of 10 testing patterns, a comparative baseline, a 13-component canonical catalog, and a component-wise mapping of testing patterns.The catalog extends the JaCaMo-based model with a Registry component.Source: §1 · p. 5

2 A Motivational Example

The HomeBot example follows a prototype into production, where model changes and new integrations expose testing challenges. Its familiar tests provide little guidance for handling non-deterministic behavior or isolating component-level failures.

Phase 2: The Unseen Decay

  • After deployment, the same user input can produce materially different actions, and an unannounced FM update changes prompt interpretation without an automated regression suite to detect semantic drift.The example contrasts silently turning on lights with asking whether to turn on a reading lamp; both are conversationally plausible but differ in system action.Source: §2 · pp. 5–6

Phase 3: The Component Explosion

  • Adding RAG and an MCP utility API creates distinct testing questions about retrieval relevance and whether a failure comes from the FM or a changed server schema.A keyword match can miss a manual passage using a synonym, while the utility API is a black box.Source: §2 · p. 6

Phase 4: The Testing Gap

  • Alex finds familiar mocking and component tests difficult to apply when FM behavior is non-deterministic and components evolve independently.The framework and application test suites instead present fragmented ad hoc scripts and coarse-grained end-to-end tests, with no clear consensus on practices or component priorities.Source: §2 · p. 6

3 Background

The background establishes software-testing concepts, describes the two-layer agent ecosystem, and adopts a canonical architecture for analyzing test targets across changing implementations.

3.1 Foundations of Software Testing

  • Software tests exercise a Subject Under Test through test functions that initialize state, execute a command, and check outcomes against requirements.The Arrange-Act-Assert convention structures this process, while test doubles can isolate the SUT from external dependencies.Source: §3.1 · p. 7

3.2 The Agent Ecosystem

  • Agent frameworks provide reusable infrastructure, whereas agentic applications implement domain-specific workflows, prompts, and tool compositions on top of them.The two layers expose different execution abstractions, so the nature and granularity of their unit tests differ.Source: §3.2 · pp. 7–8

3.3 Canonical Agent Architecture

  • The canonical architecture groups agent internals, environmental artifacts and signals, and organizational coordination structures as a stable reference model.The Agent dimension includes internal state and control constructs; Environment covers resources and observable signals; Organization covers roles and interaction mechanisms.Source: §3.3 · p. 8
  • The model adds Registries to account for dynamic discovery and invocation of tools or agents across system boundaries.This shared vocabulary supports identifying test targets independently of framework-specific abstractions.Source: §3.3 · p. 8

4 Related Work

Prior work studies testing patterns in traditional software and machine learning, while empirical evidence about quality assurance in agent systems remains limited. This study addresses that gap with a large-scale investigation of unit testing in open-source agent frameworks and applications.

4.1 Testing in OSS Ecosystem

  • Traditional software-testing research covers patterns such as Arrange-Act-Assert, test doubles, parameterized tests, and assertion strategies, but their translation to FM-based agents remains unclear.Prior work reports AAA in 77% of Java unit tests and parameterized tests running 30 times faster than the traditional approach.Source: §4.1 · pp. 8–9

4.2 Testing in ML Ecosystem

  • ML testing research addresses learned behavior, data pipelines, and properties such as fairness and robustness, but has not established how these patterns adapt to composite FM-agent architectures.Prior work includes Oracle Approximation and Value Range Analysis; studies have also identified nine high-level patterns for 16 ML-specific properties.Source: §4.2 · pp. 9–10

4.3 Evaluating Agent Frameworks and Agentic Systems

  • Agent evaluations commonly use task benchmarks, but benchmarks alone do not resolve practical reliability concerns, and comprehensive research on unit-testing practices in agent frameworks and applications is absent.The study addresses this gap through a large-scale empirical investigation of open-source systems.Source: §4.3 · p. 10

5 Methodology

The study applies a three-phase empirical workflow to collect and analyze test functions from AI agent frameworks and applications, using repository filtering, AST-based mining, stratified sampling, and card sorting. Independent and consensus coding identifies testing patterns, labels systems under test, maps them to agent-architecture components, and assesses coding agreement.

5.1 Collecting Open-Source AI Agent Frameworks’ Test Functions

  • The researchers curated 39 Python-based frameworks through repository searches, contributor and test-file filters, and manual verification against agent-support criteria.The framework search returned 107 repositories; two failed manual checks, while the Python and test-file filters left 41 candidates before verification.Source: §5.1.1 · p. 11, §5.1.2 · pp. 12–13, §5.1.3 · p. 13

5.1.4 Mining Test Functions

  • An AST-based parser extracted 15,964 framework test functions from 1,981 test files using test-file and test-function naming conventions.A function qualified when it appeared in a test_*.py file and its name began with test_.Source: §5.1.4 · pp. 13–14

5.1.5 Stratified Random Sampling

  • The framework analysis used a proportional, repository-stratified sample of 377 functions, calculated with Cochran’s formula at 95% confidence and a 0.05 margin of error.The calculation used N = 15,964, Z = 1.96, and p = 0.5.Source: §5.1.5 · p. 14

5.2.1 GitHub Search

  • Application discovery searched framework-specific imports for 33 of the 39 frameworks; six without importable SDKs were excluded from code-based identification.The researchers derived Python import statements from documentation, source code, and examples, then used them in GitHub Code Search.Source: §5.2.1 · p. 16

5.2.2 Filtering Toy Repositories

  • 439 agentic applications remained after filtering 7,397 repositories; their median size was 30 KLoC, and sampled projects covered diverse application domains.The domain review manually inspected 50 applications and identified categories including assistants, software, medicine, crypto, and project management.Source: §5.2.2 · p. 18, Figure 2 · p. 19

5.2.3 Mining and Deduplicating Test Functions

  • After removing copied framework tests through SHA-256 matching, the application dataset contained 110,891 unique test functions.Application test hashes were compared with hashes from each relevant framework, and matches were excluded.Source: §5.2.3 · pp. 19–20

5.2.4 Stratified Random Sampling

  • The application analysis sampled 382 valid functions proportionally across repositories from an initial Cochran-based sample of approximately 383.One sampled function with an empty implementation was excluded; the top 10 repositories accounted for approximately 70% of all functions, motivating stratification.Source: §5.2.4 · p. 20

5.3 Card Sorting

  • Hybrid card sorting identified recurring testing patterns, while closed card sorting mapped extracted subjects under test to core agent components.The analysis focused on test-code setup and verification strategies, including mocking, patching, parameterization, and custom assertions.Source: §5.3 · p. 21, §5.3.1 · pp. 21–22

5.3.1 Hybrid Card Sorting for Testing Patterns

  • Two authors independently sorted 15% of test functions from each dataset, using multi-label annotations because a function could exhibit multiple testing patterns.Five foundational studies supplied examples but did not form a closed framework, so raters could identify novel patterns; they examined test code, the SUT, and README documentation.Source: §5.3.1 · p. 23
  • Raters aligned category naming and granularity without discussing individual sorting results, then finalized a consensus taxonomy, re-sorted the sample, and used Jaccard similarity for multi-label agreement.They chose Jaccard because traditional agreement coefficients assume one categorical label per item.Source: §5.3.1 · pp. 23–24
  • Jaccard agreement was 0.88 for agent framework patterns and 0.85 for agentic applications; MASI-based Krippendorff’s α was 0.80 for agent frameworks and 0.75 for agentic applications.Jaccard measured label-set overlap, while MASI-based Krippendorff’s α adjusted for chance agreement and partial label matches.Source: §5.3.1 · p. 24

5.3.2 Extracting SUTs

  • Two authors independently labeled SUTs in a 15% sample of test functions from both datasets, recording each function’s repository, name, and corresponding SUT.Because test code often lacked explicit SUT names, raters used the test’s functional purpose and domain context to classify it.Source: §5.3.2 · pp. 24–25
  • After third-rater adjudication, Cohen’s Kappa for SUT labeling rose from 0.52 to 0.87, indicating strong consistency in the finalized taxonomy.The first author then labeled the remaining test functions using the consensus definitions.Source: §5.3.2 · p. 25

5.3.3 Closed Card Sorting for Components

  • The first and second authors used closed card sorting to map identified SUTs to architectural components drawn from the study’s framework and application model.For example, they mapped embeddings to Resource Artifacts.Source: §5.3.3 · p. 25

6 RQ1: What testing practices are commonly adopted by practitioners to evaluate open-source AI agent frameworks and agentic applications?

The study catalogs how developers structure tests and verify agent behavior despite model uncertainty. Practitioners mostly adapt familiar testing patterns, while specialized methods remain rarely used.

6.1 Approach

  • The study uses hybrid card-sorting to identify test-structure and verification patterns, then compares their prevalence in popular and remaining repositories.The popularity comparison uses Fisher’s exact test with Benjamini–Hochberg false-discovery-rate adjustment.Source: §6.1 · p. 26

6.2 Findings

  • Three structural and seven verification patterns were identified, with eight of the ten patterns previously reported in software engineering or ML testing.Parameterized testing is the most adopted structural pattern: 28.7% of framework tests and 26.1% of application tests, versus 9% reported in traditional software engineering.Source: Figure 4 · p. 26, §6.2 · pp. 27, 30, Table 6 · p. 28
  • Tests set model temperature to zero or a very low value when reproducible outputs are needed, although higher temperatures support more diverse generation.Hyperparameter control is described as a pattern targeting the model’s stochastic behavior.Source: §6.2 · pp. 28–29, Figure 6 · p. 30
  • DeepEval and Hyperparameter Control appear in around 1% of tests, while practitioners more commonly adapt familiar patterns to address non-determinism.DeepEval occurs in 1.1% of application test functions; Hyperparameter Control occurs in 0.5% of framework tests and 0.8% of application tests.Source: Figure 4 · p. 26, §6.2 · pp. 29, 30, 32
  • Strict assertion-based testing appears in 72–82% of tests, but flexible verification accompanies it in 40.2% of framework tests and 36.7% of application tests.Assertion-based testing is used 3–4 times more often than the next most common pattern; flexible additions include membership, mock, and negative testing.Source: §6.2 · pp. 30–31, Figure 7 · p. 31
  • Only Mock Assertion differs significantly in top-10 frameworks: 22.9% versus 9.1% in other repositories; no pattern significantly dominates among top-10 applications.The framework comparison reports p = 0.0004 and adjusted p = 0.0049 for Mock Assertion.Source: §6.2 · p. 31, Table 7 · p. 32

7 RQ2: How do these testing practices map to the architectural components of the agent frameworks and agentic applications?

The study maps tested subjects to a canonical agent architecture to examine how testing effort and patterns vary across components. The distribution favors deterministic infrastructure, while prompt testing is rare.

7.2.1 Component Mapping With SUT

  • The study identifies 35 tested subjects and maps them to 13 canonical agent components, with one to seven subjects under each component.The mapping uses manually identified subjects under test and a closed card-sorting process.Source: §7.1 · pp. 32–33, §7.2.1 · p. 33, Table 8 · p. 33

7.2.2 Component-wise Test Frequency

  • Around 1% of test functions address Triggers, the prompt component, despite prompts’ dependence in agent behavior and their potential to degrade as remote models evolve.The paper identifies the near-absence of prompt regression testing as a risk for unexpected functionality decay.Source: §7.2.2 · pp. 36–37, Figure 9 · p. 38
  • Resource Artifacts account for 29.7% of framework tests and 39.2% of application tests, compared with 4.8% and 5.0% for the Plan Body.The study also reports Resource Artifact shares of 27.8% and 35.2% of test-code lines, versus Plan Body shares of 4.7% and 5.2%.Source: §7.2.2 · p. 36, Figure 9 · p. 38
  • Frameworks and applications test the same four most-frequent components, which together account for 65% and 75.6% of their test functions, respectively.The four are Resource Artifacts, Coordination Artifacts, Observable Property, and Constitutive Entity; Context and Group are seldom tested.Source: §7.2.2 · p. 38, Figure 9 · p. 38

7.2.3 Component Wise Testing Patterns

  • Assertion-Based and Parameterized Testing recur across components, while frameworks and applications use different patterns for 69% of canonical components.Frameworks emphasize general robustness; applications favor context-specific correctness with more adaptive patterns.Source: §7.2.3 · pp. 38–39, Figure 10 · p. 39
  • For Coordination Artifacts, frameworks use more negative, snapshot, and mock assertions, while applications use Membership Testing about 2.5 times more often and omit Snapshot Assertion.This contrast corresponds to framework attention to intermediate workflow states and application checks of final outcomes.Source: §7.2.3 · pp. 39–40, Figure 10 · p. 39
  • In Constitutive Entities, frameworks use Parameterized Testing and Mock Assertion more often, while applications use Test Double 1.4 times more often.The reported application rates for Parameterized Testing and Mock Assertion are 2.2x and 3.5x lower, respectively; frameworks emphasize edge cases and applications isolated interactions.Source: Figure 10 · p. 39, §7.2.3 · p. 40
  • For Boundary Artifacts, Assertion-Based Testing falls from 84.6% in frameworks to 55.6% in applications, where Membership Testing rises nearly 2.9 times.Both groups use Test Double; frameworks additionally use Mock Assertion and Snapshot Testing more, while applications favor lighter, context-specific checks.Source: Figure 10 · p. 39, §7.2.3 · p. 40

8 Implications

The implications call for testing practices that distinguish deterministic infrastructure from non-deterministic agent behavior. They recommend stronger framework support, prompt regression at the application layer, and research into adoption barriers and agent-specific testing guidance.

8.1.1 For Agent Framework Developers

  • Frameworks should provide semantic verification for variable but valid agent outputs, though LLM judges add token, cost, latency, bias, and oversight burdens.The proposed mitigations include smaller specialized judges, spot-checking, asynchronous evaluation, and bias management.Source: §8.1.1 · pp. 41–42
  • Framework developers should define testing contracts that certify general robustness and let applications focus on their own logic.The proposed contract could reduce duplicated testing across framework and application layers.Source: §8.1.1 · pp. 42–43
  • Because non-deterministic components are harder to control and observe, developers prioritize deterministic artifacts that are more feasible to verify under current constraints.The authors also connect this pattern to developers’ acceptance of LLM non-determinism and difficulty integrating agents into deterministic CI/CD pipelines.Source: §8.1.1 · pp. 43–44

8.1.2 For Agentic Application Developers

  • Prompt regression suites can test curated prompts and semantic outputs against updated FM versions, addressing the finding that prompts appear in around 1% of tests.The recommendation responds to provider model updates that may make previously successful prompts fail silently.Source: §8.1.2 · p. 44
  • Setting temperature to 0 can make FM outputs more reproducible while developers debug workflows and tools, after which they can raise it to test realistic variability.Hyperparameter Control appears in 0.5% of framework tests and 0% of application tests.Source: §8.1.2 · p. 44
  • Application developers should rely on framework robustness guarantees and prioritize custom tools, external connectors, and application-specific workflows.Frameworks and applications both test Resource Artifacts, Coordination Artifacts, Observable Property, and Constitutive Entity, sometimes duplicating effort for different purposes.Source: §8.1.2 · pp. 44–45

8.2 Implications for Researchers

  • Researchers should investigate why DeepEval and Hyperparameter Control remain rarely adopted, using practitioner surveys and interviews to examine possible barriers.The passages identify awareness, tool maturity, perceived complexity, and traditional testing culture as questions for future study.Source: §8.2 · p. 45
  • The authors characterize agent testing as hybrid: deterministic infrastructure practices are retained, LLM evaluation and oracle strategies are adapted, and some classical ML methods need component-specific reformulation.They call for a Theory of Agent Testing that links canonical components to suitable strategies; the recommendations are synthesized from SE, ML, and LLM application sources.Source: §8.2 · pp. 45, 46, 48

9 Threats to Validity

The study identifies limits in repository discovery, application identification, and the scope of its analysis. Its construct measures also depend on component mapping, cross-domain comparison choices, and a particular test-file naming convention.

9.1 External Threats

  • GitHub keyword search may miss relevant frameworks, although manual domain-expert verification was used to reduce omission risk.LangGraph was added after investigation, while the authors cannot guarantee the framework list is exhaustive.Source: §9.1 · p. 48
  • Import-statement searching may exclude applications using dynamic imports or obscured wrappers and may include projects that use frameworks only for minor non-agentic utilities.Source: §9.1 · p. 48
  • The study documents testing practices rather than empirically measuring how component variations or failures affect agent behavior.The effects of prompt variation are outside its scope.Source: §9.1 · p. 48

9.2 Construct Threats

  • Mapping test subjects to canonical components involved possible rater bias; two independent raters used closed card sorting, resolved disagreements through discussion, and achieved κ = 0.83.Source: §9.2 · pp. 48–49
  • Cross-domain pattern comparisons may be affected by inconsistent granularity, which the authors addressed by normalizing proportions and aggregating semantically similar sub-patterns.Source: §9.2 · p. 49
  • Using the test_* filename convention may exclude repositories with other test naming schemes or tests embedded in production code, so findings are scoped to explicit prefix-based test files.Source: §9.2 · p. 49

10 Conclusion

Across 39 agent frameworks and 439 agentic applications, testing reflects a pragmatic but incomplete adaptation to non-determinism and architectural change. Traditional strategies predominate, while prompt testing remains a critical blind spot.

10 Conclusion

  • Across 39 frameworks and 439 applications, testing practices show a pragmatic but incomplete adaptation to non-determinism and evolving architectural complexity.The study presents this as the first large-scale empirical evidence of practitioners’ testing practices in FM-based AI agents.Source: §10 · p. 49
  • 40.2% of framework and 36.7% of application test functions combine assertion-based testing with negative testing and membership assertions, while DeepEval appears in only 1.1% of application test functions.The study maps patterns to 13 canonical agent-architecture components and finds the Trigger component is covered by only around 1% of test functions.Source: §10 · p. 49
  • The Trigger component is the prompt interface governing the FM, and its near-total neglect exposes the ecosystem to semantic drift, performance decay, and silent failures.The Trigger is covered by only around 1% of test functions.Source: §10 · pp. 49–50
  • Reliable AI agents require augmenting established engineering principles with targeted new methodologies, rather than discarding them.The study offers an empirical baseline and highlights where testing effort must be rebalanced.Source: §10 · p. 50

Declaration

The declarations report no human or animal participants, provide study-material availability and competing-interest statements, and include author contributions. The appendix example sets temperature to zero in a hyperparameter-control test.

Declaration

A.1.1 Hyperparameter Control

  • The Hyperparameter Control example sets temperature to 0 while testing concurrent API calls through an agent instructed to report success or error based on tool output.The test asserts that the result contains no “error.”Source: §A.1.1 · p. 60

A.1.2 Parameterized Testing

The appendix illustrates parameterized testing across models, a mocked MCP connection failure, and an equality assertion for a memory object.

A.1.2 Parameterized Testing

  • Parameterized Testing passes different LLM models to the same test function, with the example using GPT-2 and OPT-350M.The test checks whether model-specific keyword arguments can be passed.Source: §A.1.2 · p. 60

A.1.3 Test Double

  • The Test Double example mocks an MCP server connection failure and verifies that the resulting ConnectionError contains the expected message.The test patches the MCP client and raises a connection error from the mock.Source: §A.1.3 · pp. 60–61

A.2.1 Assertion Based Testing

  • Assertion Based Testing verifies equality between a memory object's name and “MyMemory.”The listing caption identifies equality verification as the pattern.Source: §A.2.1 · p. 61

A.2.2 DeepEval

The appendix illustrates DeepEval’s probabilistic assertion alongside membership, mock, negative, and snapshot testing examples.

A.2.2 DeepEval

  • DeepEval is used for a probabilistic assertion that retrieved resume-agent output contains the expected company name.The example constructs a message from the job description and evaluates the chat outcome.Source: §A.2.2 · p. 61

A.2.3 Membership Testing

  • Membership Testing checks whether a string or pattern appears in a result, with examples verifying guardrail outcomes and message content.The examples check both a Boolean result and whether a guardrail-limit phrase is present.Source: §A.2.3 · p. 62

A.2.4 Mock Assertion

  • Mock Assertion verifies that a mocked PII-detection method is called when processing a request containing personal information.The example also checks that the agent returns no response.Source: §A.2.4 · p. 62

A.2.5 Negative Test

  • The Negative Test example expects a LookupError for Mermaid code with an incorrect start node.The example uses pytest.raises to check the error.Source: §A.2.5 · p. 62

A.2.6 Snapshot Testing

  • Snapshot Testing compares markdown slash-command output against stored snapshots, including a case with no markdown output.The example also checks the snapshot for markdown output from the last question.Source: §A.2.6 · pp. 62–63

A.2.7 Value Range Analysis

Value Range Analysis verifies whether output falls within a predefined range. The accompanying test snippets assert properties of sanitized filenames.

A.2.7 Value Range Analysis

  • Value Range Analysis checks whether output is within a predefined range.The passage identifies this as the purpose of the analysis.Source: §A.2.7 · p. 63
  • The test function is named test_path_traversal_characters.Source: §A.2.7 · p. 63
  • The path-traversal test asserts that the sanitized value starts with “passwd_” and does not exceed MAX_FILENAME_LENGTH.These are the two asserted conditions shown in the snippets.Source: §A.2.7 · p. 63

B Canonical Agent Architecture

The study maps observed agent-system components to JaCaMo’s Agent, Environment, and Organization architecture, supplementing it with registries. The section defines these components and relates them to modern agent systems.

B Canonical Agent Architecture

  • The component map uses JaCaMo’s Agent, Environment, and Organization dimensions and adds registries as a newer architectural concept.The mapping is intended to provide an analytical foundation beyond implementation-specific details.Source: B Canonical Agent Architecture · p. 63, §B.14 · p. 65
  • The Belief Base holds an agent’s information about itself and its environment, corresponding in FM-based agents to memory systems, vector stores, and other persistent data sources.The beliefs may be incomplete or incorrect.Source: §B.1 · p. 63
  • A Goal is a desired world state, and a Plan is an action course composed of trigger, context, and body; FM-based agents use prompts as triggers and may generate plan bodies dynamically.Context supplies just-in-time information for selecting among plans; the body comprises actions or reasoning steps.Source: §B.2 · p. 63, §B.3 · p. 63, §B.4 · p. 64
  • Actions include messages exchanged among agents or with users, internal computations without external effects, and operations that affect the external environment.Source: §B.9 · p. 64, §B.7 · p. 64, §B.8 · p. 64
  • Resource Artifacts are reusable environment entities, Coordination Artifacts synchronize multi-agent interactions, and Boundary Artifacts connect the system to outside users or software.Examples include API wrappers, message queues, and graphical user interfaces.Source: §B.10 · p. 64, §B.11 · p. 64, §B.12 · p. 65
  • Observable Properties expose artifact state as data, while Signals notify observing agents of significant changes; modern examples include logs, metrics, and return values.Source: §B.13 · p. 65
  • A Registry catalogs agents or tools for publishing, discovery, and reuse; MCP registries support standardized tool discovery and invocation across frameworks.Source: §B.14 · p. 65

Related papers

Loading 2509.19185v3…