PaperlayerView exact revision on arXiv

Source-linked AI summary

Architectural Design Decisions in AI Agent Harnesses

Hu Wei

arXiv:2604.18071v1cs.AI

TL;DR

Comparative evidence about the architecture of reusable non-LLM infrastructure around agent systems remains limited. This study investigates 70 publicly available projects and identifies recurring dimensions and structured decision combinations.

  • Problem

    Comparative evidence remains limited on which infrastructure-level design decisions recur across agent projects and how they bundle.

    Source: §1.1 · p. 2

  • Method

    A protocol-guided, source-grounded empirical study of 70 publicly available projects uses descriptive summaries, co-occurrence analysis, and pattern synthesis.

    Source: §1.1 · p. 2, §3.1 · p. 7

  • Results

    Five recurring dimensions form structured combinations, including richer subagent coordination with more explicit context handling and stronger execution environments with more structured safety controls.

    Source: §8.1 · p. 34

  • Takeaways & Limitations

    The study offers grounded guidance for framework designers, selectors, and researchers.

    Source: §1.1 · p. 2

    Co-occurrence analysis is descriptive and cannot establish causal direction, and the corpus covers publicly available projects rather than fully closed industrial harnesses.

    Source: §3.9 · p. 12, §7.3 · p. 33

Abstract

from arXiv · show

AI agent systems increasingly rely on reusable non-LLM engineering infrastructure that packages tool mediation, context handling, delegation, safety control, and orchestration. Yet the architectural design decisions in this surrounding infrastructure remain understudied. This paper presents a protocol-guided, source-grounded empirical study of 70 publicly available agent-system projects, addressing three questions: which design-decision dimensions recur across projects, which co-occurrences structure those decisions, and which typical architectural patterns emerge. Methodologically, we contribute a transparent investigation procedure for analyzing heterogeneous agent-system corpora through source-code and technical-material reading. Empirically, we identify five recurring design dimensions (subagent architecture, context management, tool systems, safety mechanisms, and orchestration) and find that the corpus favors file-persistent, hybrid, and hierarchical context strategies; registry-oriented tool systems remain dominant while MCP- and plugin-oriented extensions are emerging; and intermediate isolation is common but high-assurance audit is rare. Cross-project co-occurrence analysis reveals that deeper coordination pairs with more explicit context services, stronger execution environments with more structured governance, and formalized tool-registration boundaries with broader ecosystem ambitions. We synthesize five recurring architectural patterns spanning lightweight tools, balanced CLI frameworks, multi-agent orchestrators, enterprise systems, and scenario-verticalized projects. The result provides an evidence-based account of architectural regularities in agent-system engineering, with grounded guidance for framework designers, selectors, and researchers.

1 Introduction

The paper addresses the lack of comparative evidence about how agent systems’ surrounding engineering infrastructure is architected. It studies 70 publicly available projects to identify recurring decisions, their relationships, and typical architectural patterns.

1.1 Motivation

  • Prior work leaves limited evidence about recurring infrastructure-level decisions and their bundles, limiting empirical guidance for framework developers and selectors.Researchers also lack a shared vocabulary for comparing these architectures.Source: §1.1 · p. 2

1.2 Research Questions

  • The study asks which design dimensions recur, how decisions co-occur, and how projects’ complexity and positioning relate to typical patterns.These questions progress from identifying the design space to analyzing relationships and informing comparison.Source: §1.2 · pp. 2–3

1.3 Contributions

  • The authors characterize five recurring dimensions—subagent architecture, context management, tool systems, safety, and orchestration—using a transparent, protocol-guided investigation procedure.The procedure uses explicit evidence trails and comparable coding logic.Source: §1.3 · p. 3
  • Cross-project analysis identifies recurring decision bundles and reports that several simplifying assumptions do not hold across the corpus.The assumptions include that language determines architecture, use case fixes complexity, and capability growth implies governance maturity.Source: §1.3 · p. 3
  • The authors synthesize typical patterns linking architectural choices with projects’ complexity profiles, governance commitments, and product positioning.They present these patterns as a comparative framework for explanation, development, and selection.Source: §1.3 · p. 3
  • The paper provides a documented corpus boundary, search and screening protocol, and operational codebook to support auditing and extending the comparison.The paper then moves from related work and method to design dimensions, co-occurrences, patterns, and validity considerations.Source: §1.3 · p. 3, §1.4 · pp. 3–4

2 Related Work

Prior research has examined agent capabilities, frameworks, and safety, but offers less comparative evidence about how reusable execution infrastructure is assembled. This paper applies design-space analysis to that infrastructure across existing projects.

2.1 AI Agent Systems

  • Agent research has studied reasoning and action [22] and long-term memory [11], but these capability findings do not compare how reusable infrastructures operationalize them.The paper positions its focus as the surrounding engineering layer, rather than agent algorithms and behaviors.Source: §2.1 · pp. 4–5

2.2 Agent Frameworks and Platforms

  • Application frameworks such as LangChain and CrewAI emphasize development abstractions, while infrastructure-focused projects address execution and governance; comparative accounts of their architectural bundles remain missing.Examples of infrastructure-focused projects include OpenHands [20] and SWE-agent [23].Source: §2.2 · pp. 5–6
  • Capability benchmarks assess what agents can do, while safety studies address risks, leaving execution-substrate choices and their combinations less explained.The cited benchmarks include AgentBench [2], SWE-Bench [24], and WebArena [28]; safety work includes studies of MCP security [17].Source: §2.2 · pp. 5–6

2.3 Software Architecture Design Space Analysis

  • Design-space analysis treats recurring architectural decisions and their relationships as a structured space; this study applies that perspective retrospectively to built frameworks, not one pending design.The authors aim to identify recurring, comparable decisions without claiming an exhaustive ontology or a new universal category.Source: §2.3 · pp. 6–7
  • Prior agent-risk work often focuses on behavior or evaluation, while this study examines how infrastructure decisions recur and combine across public implementations.Its scope complements empirical studies of MCP servers and safety audits [17].Source: §2.3 · pp. 6–7

3 Methodology

The study uses a qualitative, cross-sectional design to describe observable architectural decisions in publicly available agent-system projects. Its fixed protocol grounds investigation primarily in source artifacts, with public technical materials used when full source visibility is unavailable.

3.2 Project Selection

  • The 70-project corpus combines 67 open-source repositories with three public-evidence comparison cases and was assembled for analytic coverage, not an exhaustive census.Projects were selected for inspectable infrastructure beyond a single prompt-to-API path; discovery used repository search, related work, manual review, and snowballing.Source: §3.2 · pp. 7–8

3.3 Analysis Dimensions

  • Five focal dimensions—subagent architecture, context management, tool systems, safety mechanisms, and orchestration—were retained for recurring, comparable evidence relevant to the research questions.Subagent architecture captures role decomposition and delegation, while orchestration captures temporal task control; either can occur without the other.Source: §3.3 · pp. 8–9

3.4 Data Collection Process

  • Project investigations followed SOPs across 14 architectural modules, with LLM agents assisting navigation and evidence surfacing while humans verified artifacts and made final coding judgments.The process combined vertical project investigation and horizontal cross-project synthesis; a 15-project audit achieved 94% initial field-level agreement before consensus coding.Source: §3.4 · p. 10

3.5 Analysis Methods

  • Three complementary analysis methods address the paper’s research questions.The methods are descriptive statistics, co-occurrence analysis, and pattern synthesis.Source: §3.5 · p. 11

3.6 Descriptive Statistics

  • For RQ1, the study reports counts and percentages for categorical options to identify the most and least common design decisions.Decisions are encoded as presence/absence observations or categorical labels rather than a single numeric ranking.Source: §3.3 · p. 9, §3.6 · p. 12

3.7 Co-occurrence Analysis

  • For RQ2, support, confidence, and lift describe recurring decision co-occurrences, not statistical significance or causal relationships.Only relationships judged architecturally interpretable and supported by visible implementation evidence are reported.Source: §3.7 · p. 12

3.8 Pattern Synthesis

  • For RQ3, recurring decision bundles are synthesized into typical patterns across projects with different complexity, implementation scope, and positioning.Representative projects anchor pattern descriptions; the aim is interpretive synthesis rather than a leaderboard.Source: §3.8 · p. 12

3.9 Limitations

  • Interpretation is bounded by a mostly open-source, time-specific corpus, non-exhaustive discovery, judgment in borderline sampled inspections, and five dimensions that do not exhaust the design space.The study notes that fully closed industrial harnesses may differ, project positions can shift, and additional concerns may emerge as the ecosystem matures.Source: §3.9 · pp. 12–13

4 The Agent Harness Design Space

Section 4 maps recurring design choices across five dimensions using marginal distributions across projects, establishing the one-dimension-at-a-time picture for the co-occurrence analysis that follows.

4.1 Subagent Architecture Decisions

  • Subagent patterns range from no spawning (30.0%) and tool-based delegation (17.1%) to orchestrator-worker (18.6%) and recursive hierarchies (12.9%).Source inspection also found explicit depth limits or recursion-prevention mechanisms in several projects that appeared deeply recursive.Source: Table 4 · p. 13, §4.1 · p. 14

4.2 Context Management Decisions

  • File-persistent, hybrid, and hierarchical context designs anchor the corpus, while purely prompt-window-oriented systems are rare.Context management also varies in compression, persistence scope, and token budgeting; explicit budgets can trigger compression before prompts become overloaded.Source: §4.2 · pp. 14–15, Table 5 · p. 15

4.3 Tool System Decisions

  • Explicit registries are modal (34.3%), while MCP (14.3%) and plugin-oriented (10.0%) systems remain minority patterns.The paper links formalized registration and discovery to broader extension ambitions and the need for clearer execution boundaries and governance.Source: §4.3 · pp. 16–17

4.4 Security and Isolation Decisions

  • Intermediate isolation is common, but high-assurance audit is rare, and some projects expose broad execution power without strong accountability.Process separation accounts for 45%; 40% report no audit and 5% use tamper-evident trails.Source: Table 7 · p. 17, Table 8 · p. 17, §4.4 · p. 18

4.5 Orchestration Decisions

  • Imperative and ReAct-style control remain the ecosystem’s default center of gravity, while a sizable minority externalize workflow structure or shift to event-driven coordination for repeatable pipelines, collaborative execution, or persistent operating contexts.Imperative workflows account for 45%, event-driven workflows 30%, ReAct-style workflows 50%, and Plan-and-Execute workflows 35%.Source: §4.5 · pp. 18, 20

5 Design Decision Co-occurrences

The co-occurrence analysis identifies recurring bundles among coordination, context, execution safety, and tool registration, while also showing that language, use case, and capability do not determine architecture. These relationships are descriptive rather than causal, and the patterns are not mechanically determined.

5.1 Co-occurrence Analysis Method

  • The analysis uses descriptive support, confidence, and lift to examine coded-decision pairs, prioritizing recurring and architecturally interpretable relationships rather than inferential tests.The measures do not establish causal order, statistical significance, sampling generalization, or framework quality; interpretation also draws on project-level source inspection.Source: §5.1 · pp. 20–21

5.2 Key Co-occurrences

  • Subagent complexity co-occurs with richer context (support 0.73, lift 1.8); E-pattern memory scores average 4.1 and F-pattern scores 4.2, versus 2.8 without subagents.File persistence occurs in 85% of E-pattern and 78% of F-pattern cases, compared with 20% of A-pattern cases.Source: §5.2 · p. 23
  • Container-isolated projects implement policy engines in 100% of cases, versus 23% without container isolation; security scores are 4.5 for container, 3.2 for process, and 2.1 for no sandbox.The corpus associates stronger isolation with broader approval logic and more explicit audit trails.Source: §5.2 · pp. 23–24
  • MCP-first projects average 4.62 on tool discovery, compared with 3.86 for registry-centered and 2.81 for minimalist systems; formalized extension boundaries are associated with broader ecosystem ambitions.The authors caution that the score difference does not prove MCP causes broader ecosystem ambition.Source: §5.2 · p. 24

5.3 Important Non-Co-occurrences

  • Language and use case do not determine architectural pattern: advanced subagent patterns span language families, while general-purpose, coding, and research projects vary in design complexity.Advanced subagent patterns range from 40% of TypeScript projects to 57% of Rust projects; Python and Go are at 42% and 43%.Source: §5.3 · pp. 24–25
  • Capability growth does not guarantee safety maturity: some projects expose substantial execution power with modest oversight, while others invest in safety despite narrower capabilities.The result is an uneven safety landscape, not a single maturity trajectory.Source: §5.3 · p. 25

5.4 Interpreting the Co-occurrence Structure

  • Across the corpus, reportable relationships form intelligible bundles, but the analysis is interpretive and does not identify directional causal mechanisms.Coordination often accompanies explicit state management, execution choices accompany governance, and extension choices associate with broader ecosystem positioning.Source: §5.4 · pp. 25–26

6 Typical Design Patterns

The section synthesizes five recurring architectural patterns as descriptive bundles, not maturity levels or quality tiers. The patterns span lightweight tools, balanced CLI frameworks, multi-agent orchestrators, enterprise systems, and scenario-verticalized or research-oriented projects.

6.1 Pattern Identification Criteria

  • Pattern identification re-read project reports for recurring bundles, grouped similar projects into provisional archetypes, tested them against explicit criteria, and assigned dominant patterns while retaining hybrid cases.The patterns are interpretive centers of gravity, not statistically inferred latent classes or sharp boundaries.Source: §6.1 · pp. 26–27

Pattern 1: Lightweight Tool

  • Lightweight tools use simple, narrow architectures—often single-agent, with basic persistence, hard-coded tools, and little or no sandboxing—favoring fast iteration and deployment over long-running coordination and control.They suit narrowly scoped tasks but become strained as workflows lengthen, tool surfaces expand, or decomposition matters.Source: Pattern 1: Lightweight Tool · p. 28

Pattern 2: Balanced CLI Framework

  • Balanced CLI frameworks combine file-based persistence, structured or MCP-first tools, and process-level controls for repeatable developer workflows without the deeper governance of larger platforms.Their tradeoff is practical extensibility and repeatability without deeper coordination and governance.Source: Pattern 2: Balanced CLI Framework · pp. 28–29

Pattern 3: Multi-Agent Orchestrator

  • Multi-agent orchestrators invest in task decomposition and coordination, commonly pairing orchestrator-worker or recursive agents with hierarchical or hybrid memory and structured safety controls.This expands problem-solving reach but raises context-management, safety-design, and comprehensibility costs; it fits when decomposition provides real benefits.Source: Pattern 3: Multi-Agent Orchestrator · p. 29

Pattern 4: Enterprise Full-Featured

  • Enterprise systems bundle recursive or event-driven agents, layered memory, MCP and plugin extensibility, and controls such as container isolation, policy engines, and audit trails.The bundle serves production, security-sensitive, and compliance-oriented settings, with substantial infrastructure and governance costs.Source: Pattern 4: Enterprise Full-Featured · p. 30

Pattern 5: Scenario-Verticalized or Research-Oriented

  • Scenario-verticalized or research-oriented projects optimize a narrow objective, developing a relevant capability while other infrastructure may remain minimal; they trade generality, robustness, or governance completeness for specialization.This pattern suits exploration or vertical specialization, but is less suitable when systems must generalize or provide strong operational guarantees.Source: Pattern 5: Scenario-Verticalized or… · pp. 30–31

7 Discussion

The discussion turns recurring architectural bundles into guidance for designers and selectors, relates the design-space approach to prior research, and identifies study limitations and future research directions.

7.1 Implications for Designers and Selectors

  • Designers should choose an intended complexity envelope and coherent architectural bundles, while selectors should match a framework’s commitments to operational needs rather than abstract sophistication.The corpus shows different product positions align with distinct bundles, and does not converge on a single dominant recipe.Source: §7.1 · pp. 31–32

7.2 Relation to Prior Work and Broader Implications

  • The study shifts analysis from agent behavior and individual framework examples to comparative structure in reusable infrastructure, making architectural diversity empirically analyzable.It also offers descriptive conjectures about coordination and context, execution power and governance, and formal tool registration and ecosystem ambition; these are not causal laws.Source: §7.2 · pp. 32–33
  • Recurring tensions between extensibility and control, coordination and manageable context, and execution capability and safety help explain why the corpus has no single best architecture.The discussion presents architectural pluralism as partly arising from competing infrastructural commitments.Source: §7.2 · p. 33

7.3 Threats to Validity and Limitations

  • The study’s scope favors source-visible projects and its five dimensions are not an exhaustive ontology; coding involves interpretation, co-occurrences are non-causal, and findings are a snapshot.The corpus is broad but not exhaustive, and chance-corrected agreement was not reported for the full corpus-wide matrix.Source: §7.3 · p. 33

7.4 Future Research Directions

  • Future work could track framework evolution, expand project-level measurement, evaluate architectural outcomes, and standardize codebooks and extraction procedures.Outcome comparisons could examine maintainability, extensibility, safety incidents, ecosystem growth, or user adoption.Source: §7.4 · pp. 33–34

8 Conclusion

The paper compares architectural decisions across 70 publicly available agent-system projects using a protocol-guided, source-grounded study. It finds recurring dimensions and bundles, contributes a comparative research package, and points to extensions for future work.

8 Conclusion

  • The study analyzes 70 publicly available projects to identify recurring decisions, their co-occurrences, and patterns associated with project complexity and positioning.“Agent harness” is a shorthand for the surrounding engineering layer, not a strict category definition.Source: §8 · p. 34

8.1 Research Summary

  • Five focal dimensions recur, and richer subagent coordination is paired with more explicit context handling while stronger execution environments are paired with more structured safety controls.Projects occupy distinct, internally coherent regions rather than converging on one architecture; the account is not a ranking or maturity ladder.Source: §8.1 · p. 34

8.2 Contribution Recap

  • The paper contributes a transparent investigation procedure, a structured account of co-occurrences and non-co-occurrences, typical patterns and analytic conjectures, and an auditable comparative research package.The package specifies a corpus boundary, search and screening protocol, and operational codebook.Source: §8.2 · pp. 34–35

8.3 Future Work

  • Future work can track frameworks over time, improve project-level measurement, test the conjectures on broader or focused corpora, and develop decision support and architectural benchmarks.The study frames empirical characterization as a step toward cumulative and comparative research on agent-systems engineering.Source: §8.3 · p. 35

A.1 Project Selection and Data Collection

Projects were selected using public evidence, meaningful agent implementation, recent activity, and sufficient architectural scale.

A.2 Project Selection and Data Collection

  • Eligible projects needed publicly available implementation evidence, such as source code, detailed technical documentation, or public papers.Other publicly circulating source-visible materials could also qualify.Source: §A.1 · p. 38
  • Projects had to provide meaningful agent implementation and surrounding infrastructure beyond an API wrapper or prompt template.Source: §A.1 · p. 38
  • Projects needed active development or significant updates within the prior 12 months and more than approximately 500 lines of code or equivalent architectural footprint.The scale criterion also applied to public-evidence comparison cases.Source: §A.1 · p. 38

A.2 Project Metadata Summary

The corpus overview identifies a project corpus of 70.

A.2 Project Metadata Summary

A.3 Complete Project List with Public Evidence Links

The appendix provides a complete project list with public evidence links, covering a varied set of agent-system projects.

A.3 Complete Project List with Public Evidence Links

  • Table 16 is the complete project list with public evidence links.The entries include MASFactory, angel-claw, AstrBot, and AutoResearchClaw.Source: Table 16 · p. 38, §A.3 · pp. 38–39
  • Other listed projects include Codex, OpenHands, OpenClaw, Gemini CLI, and Qwen Code.These entries appear alongside projects such as Cline, Goose, and Langroid.Source: §A.3 · pp. 39–41
  • The roster also includes CoPaw, docker-agent, EvoScientist, PraisonAI, and zeroclaw.The list further includes projects such as agentpool, deepagents, and deer-flow.Source: §A.3 · pp. 38, 39, 41

A.4 Official Products from Major AI Labs

The appendix identifies six major AI-lab products in the corpus, clarifies potentially confusing project names, and describes the codebook and evidence-coding approach.

A.4 Official Products from Major AI Labs

  • Six projects (8.6%) are official or first-party products from major AI companies.The list includes Claude Code, Codex, Gemini CLI, Kimi CLI, Qwen Code, and Mistral Vibe.Source: §A.4 · p. 42

A.5 Notes on Project Names

  • The corpus labels Anthropic’s source-visible leaked snapshot as claude-code-src and analyzes it alongside public product materials.Anthropic’s Claude Code is closed-source; other similarly named entries are community projects.Source: §A.4 · p. 42, §A.5 · p. 42

B Operational Codebook

  • The operational codebook makes the study’s focal variables auditable and easier to extend, without claiming an exhaustive architectural ontology.The codebook is presented in Table 17.Source: B Operational Codebook · p. 42

B.1 Coding Principles

  • Project records were coded through source-first inspection of implementation artifacts, with confidence notes used when category boundaries were ambiguous.The records retain coded values, supporting artifacts, and notes for partial or interpretive evidence.Source: §B.1 · p. 42

B.2 Focal Variables and Operational Categories

The appendix documents the operational codebook, evidence trail, and repeatable discovery and screening protocol used to construct a cross-sectional corpus.

B.2 Focal Variables and Operational Categories

  • Table 17 presents the operational codebook for the study’s five focal dimensions.Source: Table 17 · p. 43

B.3 Evidence Trail

  • Each project record retains coded values, supporting implementation artifacts, and notes where evidence is partial or interpretive.The full corpus list enables later work to revisit cases using the same coding frame.Source: §B.3 · p. 43

C Search and Selection Protocol

  • The search protocol prioritizes broad analytic coverage and repeatable screening rather than reconstructing a closed universe of projects.The study describes project discovery as protocol-driven rather than ad hoc.Source: C Search and Selection Protocol · pp. 43–44

C.1 Discovery Channels

  • Candidate projects were gathered through public repository search, related-work and documentation reading, manual review, and snowball expansion.Source: §C.1 · p. 44

C.2 Keyword Families

  • Keyword search was supplemented by tracing projects cited in comparisons, technical posts, benchmark discussions, and repository references.Source: §C.2 · p. 44

C.3 Starting Points for Snowball Expansion

  • Snowballing used well-discussed frameworks and execution substrates as practical entry points, not as a canonical benchmark set.Examples include LangChain, AutoGPT, CrewAI, OpenHands, OpenClaw, Claude Code CLI, and docker-agent.Source: §C.3 · p. 44

C.4 Screening Logic

  • Projects had to provide inspectable evidence, meaningful execution-substrate functionality, sufficient architectural scale, and observable structure for coding.The study excluded prompt collections, thin wrappers, benchmark-only artifacts, small demonstrations, and products without enough public evidence.Source: §C.4 · p. 45

C.5 Corpus Freeze and Boundary

  • The project list was frozen on 23 March 2026 as a cross-sectional snapshot of the ecosystem visible by that date.Source: §C.5 · p. 45

C.6 Why This Is Not PRISMA

The study uses a repeatable, broad-coverage protocol for an empirical analysis of a changing ecosystem, not a formal systematic literature review or a complete census.

C.6 Why This Is Not PRISMA

  • The study is an empirical analysis of a rapidly changing implementation ecosystem, not a formal systematic literature review.Source: §C.6 · p. 45
  • Its protocol prioritizes cross-project architectural coverage, repeatable screening, and auditable project records over exhaustive repository reconstruction.Source: §C.6 · p. 45
  • The corpus characterizes recurring architectural structure but does not claim a complete census of all Agent frameworks ever released.Source: §C.6 · p. 45
Loading 2604.18071v1…