PaperlayerView exact revision on arXiv

Source-linked AI summary

SemaClaw: A Step Towards General-Purpose Personal AI Agents through Harness Engineering

Ningyan Zhu, Huacan Wang, Jie Zhou, Feiyu Chen, Shuo Zhang, Ge Chen, Chen Liu, Jiarou Wu, Wangyi Chen, Xiaofeng Mou, Yi Xu

arXiv:2604.11548v1cs.AI

TL;DR

Personal AI agents need an open harness architecture that addresses reliable orchestration, runtime safety, and durable knowledge beyond model capability alone. SemaClaw provides a layered multi-agent framework with DAG Teams, PermissionBridge, multi-tier context management, and a wiki knowledge infrastructure. The paper positions harness design as an independent source of agent performance and reports a 13.7 percentage point Terminal Bench 2.0 completion gain from harness changes alone.

  • Problem

    Personal agents must perform complex actions safely and retain useful knowledge across sessions, but open-source frameworks have not systematically addressed orchestration, authorization, and memory together.

  • Method

    SemaClaw combines a two-layer harness architecture with DAG-based orchestration, runtime permission checkpoints, three-layer context management, and a wiki-based personal knowledge infrastructure.

  • Results

    13.7 percentage points: harness configuration raised Terminal Bench 2.0 task completion from 52.8% to 66.5% with the model held constant.

  • Takeaways & Limitations

    The paper argues that reliability, safety, and long-term usefulness are primarily engineering properties of the harness around the model.

  • Takeaways & Limitations

    SemaClaw’s current scope centers on CLI and Web UI, lacks full upstream extension alignment, and keeps wiki retrieval separate from the agent’s memory search pipeline.

Abstract

from arXiv · show

The rise of OpenClaw in early 2026 marks the moment when millions of users began deploying personal AI agents into their daily lives, delegating tasks ranging from travel planning to multi-step research. This scale of adoption signals that two parallel arcs of development have reached an inflection point. First is a paradigm shift in AI engineering, evolving from prompt and context engineering to harness engineering-designing the complete infrastructure necessary to transform unconstrained agents into controllable, auditable, and production-reliable systems. As model capabilities converge, this harness layer is becoming the primary site of architectural differentiation. Second is the evolution of human-agent interaction from discrete tasks toward a persistent, contextually aware collaborative relationship, which demands open, trustworthy and extensible harness infrastructure. We present SemaClaw, an open-source multi-agent application framework that addresses these shifts by taking a step towards general-purpose personal AI agents through harness engineering. Our primary contributions include a DAG-based two-phase hybrid agent team orchestration method, a PermissionBridge behavioral safety system, a three-tier context management architecture, and an agentic wiki skill for automated personal knowledge base construction.

1 Introduction

SemaClaw frames personal-agent reliability, safety, and long-term usefulness as harness-engineering problems involving orchestration, runtime authorization, memory, and extensibility. It presents an open-source layered framework combining these mechanisms with scheduled execution and a wiki-based knowledge infrastructure.

  • The engineering gap: Open-source personal agents require system-level harness design because model capability alone does not ensure reliable multi-step execution, safety boundaries, or cross-session knowledge accumulation.The paper identifies orchestration, behavioral safety, and structured long-term memory as a joint engineering gap.
  • Framework architecture: SemaClaw separates a reusable event-driven agent runtime from an application harness providing channels, memory extensions, team coordination, and plugins.The two-layer architecture uses sema-code-core for context lifecycle, tool orchestration, and multi-tenant isolation, with semaclaw layered above it.
  • Orchestration: DAG Teams combines LLM-generated task graphs with deterministic execution to support dynamic decomposition, explicit dependencies, observability, and localized failure recovery.The approach targets tasks involving hierarchical decomposition, partial ordering, intermediate dependencies, and recovery from failure.
  • Context and memory: The three-layer context architecture unifies working context, retrieval-based long-term memory, and persistent per-agent persona partitioning.History compaction preserves active constraints, open decisions, and completed-task state while discarding exploratory transcript detail.
  • Behavioral safety: PermissionBridge makes authorization a runtime primitive by requiring explicit approval for high-risk actions and supporting agent-initiated clarification requests.This addresses operations such as file modification, external API invocation, and code execution whose paths may be dynamically determined.
  • Extensibility and knowledge: The framework adds four task-execution modes and a wiki-based knowledge layer so task-derived information can remain user-owned, organized, and retrievable across future sessions.The scheduled system ranges from notifications and scripts to agents and hybrid script-plus-agent execution; the wiki is shared through CLI and Web UI over common Markdown files.

2 Technical Foundations

The ReAct loop interleaves reasoning, tool actions, and observations, but production agents require additional runtime support for persistence, context lifecycle management, and tenant isolation.

  • ReAct execution: ReAct cycles through Thought, Action, and Observation until the agent completes the task or cannot proceed.Observations update subsequent reasoning and enable self-correction when outcomes contradict expectations.
  • ReAct execution: Structured function calling makes tool interfaces typed, machine-parseable, and more reliable than natural-language tool descriptions.MCP standardizes external tool integration across frameworks and providers.
  • Production requirements: State persistence requires memory outside the context window so agents can retain knowledge, task continuity, and user preferences across sessions.A stateless ReAct loop begins each new task with fresh context.
  • Production requirements: Context lifecycle management is needed because accumulating Thought–Action–Observation transcripts can overflow the window or dilute attention.History compaction semantically compresses prior reasoning into a condensed summary.
  • Production requirements: Multi-tenant isolation requires runtime-level separation of session state, memory, and tool state in concurrent deployments.Cross-tenant leakage is both a correctness and security failure.
  • Production requirements: Together, persistence, lifecycle management, and isolation define the engineering gap between the ReAct abstraction and a production-grade agent runtime.The loop structure alone does not address these deployment problems.

2.2 Context Management: Architecting the Agent’s Cognitive Input

Context management treats an agent’s cognitive input as a governed resource spanning working memory, external memory, and structured injection. The architecture preserves relevant information while controlling noise, but compaction introduces silent corruption risks.

  • Context architecture: Agent context comprises conversation state, tool observations, retrieved memory, and stable identity or task constraints available during reasoning.Context quality depends on the structure of these cognitive inputs, not merely the model’s reasoning capability.
  • Working memory: Working memory is finite and degrades when low-value transcript content dilutes attention before the context window overflows.Hard truncation can discard earlier constraints while retaining recent, low-value observations.
  • Working memory: History compaction replaces raw Thought–Action–Observation transcripts with structured summaries preserving active constraints, open decisions, and completed-task state.It prioritizes high-density continuation state over a verbatim record of exploratory reasoning.
  • Working memory: Compaction errors can silently drop constraints or misrepresent completed subtasks, corrupting the working state that later reasoning depends on.The original history cannot be recovered once discarded, making these errors asymmetric and compounding.
  • External memory: External memory persists prior decisions, preferences, and task history outside the context window and retrieves them selectively for inferred task states.This differs from ordinary document retrieval because queries are inferred from agent activity rather than issued directly by users.
  • Structured Context Injection: Structured context injection loads workspace-scoped files at deliberate lifecycle boundaries, while avoiding both missing background and irrelevant context pollution.Examples include SOUL.md, MEMORY.md, RULES.md, CLAUDE.md, and Plan.md, depending on the framework.

2.3 Plugin Ecosystems: Four Layers of Agent Capability Extension

Agent capability extension is organized into four complementary layers that balance generality with focus: tools act externally, subagents delegate reasoning, skills load capabilities progressively, and hooks govern execution.

  • Design framework: A layered extension architecture balances broad capability access against the reasoning burden of irrelevant options.The four mechanisms operate at different abstraction levels and address distinct engineering concerns.
  • MCP tools: MCP tools standardize how external tools are declared, discovered, and invoked, extending the agent’s available actions.Large tool surfaces can consume context and motivate coarser-grained skills.
  • Subagents: Subagents extend reasoning by using independent personas, prompts, workflows, and tool access tailored to specialized subtasks.Their isolated contexts prevent orchestrator context from growing proportionally with task complexity.
  • Skills: Skills package prompts, context documents, and tool references, then progressively inject them on demand rather than loading them at startup.Lazy loading keeps irrelevant capabilities out of context and allows new skills without runtime redeployment.
  • Hooks: Hooks provide lifecycle-event insertion points through which the harness can inspect, log, modify, block, or trigger actions during execution.They operate at the Harness Engineering layer rather than extending what the agent can do or know.
  • Design framework: The four mechanisms are complementary, and conflating their roles makes agent systems harder to reason about, test, and maintain.The framework assigns each extension mechanism a distinct concern and execution layer.

2.4 Multi-Agent Orchestration: From Call Chains to Collaborative Teams

Multi-agent orchestration spans stateless handoffs, explicit graphs, and dynamic runtime coordination, each trading flexibility against predictability or debuggability. SemaClaw explores composing dynamic task decomposition with deterministic graph execution through a two-phase approach.

  • A single agent’s context window remains a fundamental constraint when tasks require concurrent expertise, parallel workstreams, or cognitive load beyond one window.
  • Stateless Swarm: Stateless swarms coordinate through handoffs without a central coordinator or persistent shared state, favoring simplicity and composability for short, well-defined chains.They lose accumulated reasoning context across handoffs and provide no shared state for recovery when chains break.
  • Explicit Graph Structure: Explicit graph orchestration defines nodes and control-flow edges before execution, localizing failures and enabling independent testing and human approval checkpoints.Its structure is well-suited to stable workflows where correctness and auditability outweigh flexibility.
  • Explicit Graph Structure: Explicit graphs cannot naturally express task structures whose subtasks, dependencies, or agent assignments emerge only at runtime.Design-time predictability therefore imposes a pre-specification burden and may require over-specified branches or task restructuring.
  • Dynamic Orchestration: Dynamic orchestration adapts decomposition and dispatch to evolving task state, but its opaque reasoning complicates diagnosis, reproducibility, and failure containment.Worker context isolation limits propagation but does not resolve opacity in the orchestrator’s own reasoning.
  • Two-Phase Hybrid Orchestration: SemaClaw’s proposed two-phase direction has an LLM infer an inspectable dependency graph before a deterministic executor runs it, combining dynamic structure with graph-based failure isolation.The paper identifies overhead and plan reliability as open questions for this composition.

3 SemaClaw: Design and Implementation

SemaClaw is an open-source multi-agent application framework inspired by OpenClaw. It extends that foundation with architectural refinements and mechanisms for runtime decoupling, persistent personas, structured context, and additional capabilities.

  • SemaClaw is an open-source multi-agent application framework built on sema-code-core and inspired by OpenClaw’s channel-based deployment and multi-group management patterns.
  • SemaClaw’s contribution lies in architectural refinements and new mechanisms built on OpenClaw’s conceptual blueprint.These include a decoupled runtime architecture, persistent-persona-based agent teams, structured context management, and additional capabilities.

3.1 Layered Architecture: Decoupling Core from Claw

SemaClaw separates a reusable event-driven runtime from an application harness through two independently released projects with a strict dependency boundary. This separation supports independent evolution, reuse, and layered extension.

  • SemaClaw’s two-layer architecture places the reusable sema-code-core runtime beneath the semaclaw application harness.The runtime supplies execution and lifecycle foundations, while the application layer adds channels, routing, memory, plugins, and agent coordination.
  • Runtime Boundary: sema-code-core exposes agent lifecycle transitions through a uniform typed event-facade interface rather than direct access to internal execution state.Events cover session initialization, tool invocation, compaction, context updates, and session termination.
  • Runtime Boundary: The facade enforces runtime–application separation and allows any application that understands the event protocol to compose with the runtime without modifying it.
  • Design Rationale: Separating the projects lets runtime capabilities and application orchestration evolve independently because they respond to different engineering pressures.Channel integration can change without changing the execution loop, while history-compaction improvements need not alter message routing.
  • Design Rationale: The independent runtime makes solutions for tenant isolation, context lifecycle management, and tool orchestration reusable across application domains and deployment channels.
  • Design Rationale: The decoupled design creates extension surfaces at both runtime and application levels while preserving the event-facade contract.

3.2 Context Management in Practice

SemaClaw implements context management through independently governed working memory, external persistent memory, and persona-partitioned structured injections. Hybrid retrieval, compaction safeguards, and workspace switching support continuity across execution and sessions.

  • SemaClaw instantiates three context sources as compacted in-context history, persistent hybrid-retrieval memory, and persona-partitioned soul and workspace directories.
  • Working Memory: Working memory is owned by sema-code-core, which manages history and compaction while the application layer interacts through events rather than direct reads or writes.
  • Working Memory: Compaction activates at 75% of configured context length with an 8,000-token overhead buffer, emitting events that report pre- and post-compression token counts, ratio, and summary text.
  • Working Memory: Compaction quality is less consistent for general conversational agents because the prompt was developed primarily for code-centric workflows.
  • External Memory: External memory persists information beyond session-scoped working memory through MEMORY.md and dated daily logs, with session-export JSON not yet indexed by the current pipeline.
  • External Memory: Hybrid retrieval combines FTS5 BM25 and vector similarity, then falls back from Vector+FTS5 to FTS5 alone and token keyword scanning when vector search is unavailable or low quality.
  • External Memory: The memory_search MCP tool lets the agent choose when memory is relevant and control the query, result count, and source scope during execution.

3.3 Human-in-the-Loop: The PermissionBridge

PermissionBridge makes human oversight a native runtime capability for authorization and clarification, pausing execution until user responses arrive. Its tiered policy preserves usability by pre-authorizing internal tools while gating external actions.

  • Coordination architecture: PermissionBridge mediates between the agent runtime and human approval interface as a globally scoped coordination layer.It preserves session continuity while execution waits for authorization or clarification.
  • Coordination architecture: Tool permission requests pause execution, serialize the proposed invocation and rationale, and resume, deny, or modify the action after user response.Users can approve, deny, or modify the proposed tool call.
  • Coordination architecture: Agent-initiated questions use the same boundary and bridge protocol, returning the user’s clarification directly into the next reasoning step.The question response is integrated as if it were a tool result.
  • Permission tiering: Internal tools are pre-authorized, whereas external tools require explicit per-invocation consent by default.The boundary applies least privilege at the tool layer, covering external MCP servers, filesystem operations, and outbound APIs.
  • Permission tiering: A single shared bridge multiplexes concurrent approval requests by unique identifiers and routes each response to the correct waiting execution context.Approval requests can remain pending across concurrent agent sessions.
  • Trust as a design primitive: Native runtime integration and clear tiering make human oversight a practical path rather than an exceptional interruption to autonomy.The design treats structural constraints as more reliable than conventions and incrementally extends the trusted action space.

3.4 Plugin Ecosystem in Practice

SemaClaw’s plugin ecosystem extends the framework through four layers: MCP tools, subagents, hooks, and skills. These layers combine built-in operational capabilities, delegated agent work, lifecycle intervention, and user-managed extensions.

  • Architecture: The plugin ecosystem instantiates SemaClaw’s four-layer extension model across each extension mechanism.The layers are MCP tools, subagents, hooks, and skills.
  • MCP tool layer: Built-in MCP servers provide memory, workspace, dispatch, scheduling, and outbound messaging, while external servers extend these capabilities under the permission policy.Built-in servers are pre-authorized; user-installed external servers are subject to approval requirements.
  • Subagent layer: The subagent layer lets orchestrators delegate tasks to named team agents using natural-language prompts, with routing based on persona alignment.The interface is prompt-defined rather than explicitly addressed in the dispatch request.
  • Hooks layer: Lifecycle hooks expose task start, tool invocation, session compaction, and completion boundaries for external observation or intervention.External systems can interact without modifying the runtime.
  • Skills layer: Users can browse, install, remove, and toggle skills through the CLI or Web UI, with changes taking effect without an agent restart.Both interfaces expose skill descriptions and activation state.

3.5 Agent Teams: Dynamic Orchestration on Persistent Personas

SemaClaw combines persistent agent identities with DAG-based orchestration: an orchestrator makes a flexible routing and dependency plan, then deterministic runtime components execute it. This separation supports specialization, inspectability, and localized task coordination.

  • Dynamic orchestration: Figure 5 separates LLM-driven task decomposition into a dependency DAG from deterministic DispatchBridge execution by persistent worker agents.The architecture targets flexibility in planning and debuggability in multi-agent workflows.
  • Persistent personas: Team agents are persistent filesystem-level identities whose persona, memory, logs, and workspace survive restarts and accumulate over operation.Each agent is registered under a folder namespace spanning the system stack.
  • Persistent personas: The orchestrator’s LLM selects an agent name using roster, goal, persona, and memory context, while resolveAgent performs deterministic case-insensitive exact matching.Semantic judgment occurs in natural-language reasoning; binding is a lookup with no vector similarity.
  • Persistent personas: Persistent memory lets long-running workers develop specialization depth that stateless swarm participants cannot replicate.Accumulated task history is stored under each agent’s folder and retrieval index.
  • DAG declaration: The orchestrator emits the full task structure before worker invocation, with each task specifying an agent, prompt, and dependency list.The declaration forms an explicit DAG and is validated before execution proceeds.
  • Deterministic execution: After declaration, a deterministic scheduler governs execution with inspectable state, dispatching tasks when dependencies reach terminal states.DispatchBridge processes pending work and handles completion, error, and timeout states.
  • Deterministic execution: Workers receive the parent goal, prerequisite results, and peer-task statuses in augmented prompts, keeping each execution self-contained.This provides task context without requiring mid-task queries to the orchestrator.
  • Runtime coordination: Dispatch coordination uses persistent state and locking, while completion events immediately unblock downstream tasks and preserve workspace continuity for consecutive assignments.Startup recovery and heartbeat mechanisms address interrupted parents and long-running coordination.

3.6 A Four-Mode Scheduled Task System

SemaClaw matches scheduled work to four execution modes spanning deterministic notifications and scripts, full agentic reasoning, and hybrid data preparation. The taxonomy makes complexity, cost, latency, and reliability trade-offs explicit at configuration time.

  • Motivation: Scheduled tasks range from simple reminders to autonomous multi-step executions, so routing every job through a full agentic pipeline can waste compute and tokens.A single lightweight automation path would miss use cases requiring judgment and adaptation.
  • Execution taxonomy: SemaClaw defines four execution modes matched to distinct scheduled-task complexity profiles.The modes span pure notification, pure script, pure agent, and hybrid script-plus-agent execution.
  • Pure notification: Pure notification sends a pre-authored message at a scheduled time without model invocation, consuming no tokens and completing with minimal latency.It suits reminders and recurring announcements whose content is known in advance.
  • Pure script: Pure script runs deterministic code outside the agent runtime without context, model calls, or permission overhead.It suits fully specifiable data collection, file processing, and API polling tasks.
  • Pure agent: Pure agent runs a full agentic execution with pre-authored prompts, tools, and memory for tasks requiring judgment, synthesis, or adaptation.Examples include weekly summaries, status reports, and research tasks.
  • Hybrid execution: Hybrid script-plus-agent gathers and preprocesses data deterministically before passing structured context to an agent for interpretation and synthesis.This separates expensive data gathering from the reasoning step.
  • Design principle: Matching execution mode to task complexity makes trade-offs explicit and avoids unnecessary cost, latency, and reliability losses.Operators select the appropriate execution primitive during configuration.

3.7 Wiki-Based Personal Knowledge Infrastructure: A User-Owned Knowledge Layer

SemaClaw externalizes task-derived understanding into a user-owned, topic-organized wiki rather than leaving it in transient logs. Markdown files, curator workflows, separate retrieval, and direct human editing make the corpus shared infrastructure for agent and user learning.

  • Motivation: The wiki preserves distilled task knowledge in a user-owned corpus organized by topic rather than time.It addresses the loss of conclusions and structured understanding that can occur when logs are compacted or rolled off.
  • Storage: The corpus consists of human-readable Markdown files with YAML frontmatter stored directly on the user’s local filesystem.This avoids a database or proprietary index while keeping entries inspectable, movable, syncable, and editable independently of the running agent.
  • Construction: A small CLI supports inspection, categorization, saving, and organization, with inbox/ providing a fallback when classification is uncertain.The agent decides whether content fits an existing category, creates a new category, or defers categorization.
  • Retrieval: Knowledge-layer search operates over deliberately preserved content and remains distinct from memory_search over conversation transcripts and automatic logs.The former returns self-contained knowledge entries, whereas the latter returns slices of dialogue.
  • Human–agent loop: The web interface is a direct view of the same files, allowing users to browse, edit, and reorganize the corpus without intermediate state or synchronization.Agent and user therefore interact with one shared, editable knowledge substrate.
  • Vibe learning: Vibe learning turns completed tasks into structured, retrievable records that support both the agent’s future grounding and the user’s intellectual capital.The paper frames this as knowledge accumulation that compounds through joint authorship.

4 Open Questions and Future Directions

The paper treats SemaClaw’s design as a set of unresolved engineering bets rather than a finished architecture. Its open questions concern persistent identities, harness–model complementarity, privacy and ownership, and the limits of harness substitution.

  • Framing: SemaClaw presents five design questions as working bets that should be stress-tested through further community experience.The authors explicitly characterize the system as functional but unfinished.
  • Orchestration architecture: Persistent personas preserve task-specific history, while DAG Teams combine dynamic planning with dispatch to registered worker identities.The resulting hybrid uses dynamic planning over a static roster.
  • Orchestration architecture: Identity drift can invalidate role assumptions, and novel task types may force imperfect routing because agents cannot be composed on demand.These limitations arise from relying on persistent identities and roster summaries for dispatch.
  • Orchestration architecture: The choice between virtual and persistent agents remains an architectural commitment whose appropriateness SemaClaw has not settled.The authors remain open to whether this decision was bound at the right level.
  • Harness–model complementarity: Harnesses can substitute for some model capabilities by supplying retrieved knowledge, decomposition, and domain context, but not for all novel reasoning.The cited Terminal Bench result reports a 13.7 percentage point completion gain from harness changes alone.
  • Harness–model complementarity: SemaClaw assumes a capable mid-tier model may reach parity with an unsupported frontier model on many structured personal-productivity tasks, but lacks rigorous validation of this claim.The authors identify the LangChain data as their closest external evidence.
  • Memory ownership and privacy: Local storage keeps accumulated memory on user-controlled infrastructure without mandatory cloud sync or telemetry, but does not make the data secure by itself.The paper also notes unresolved legal and technical questions around ownership of the resulting knowledge profile.

5 Conclusion

SemaClaw argues that reliable, safe, and useful personal agents depend primarily on a layered harness around the model. It presents an open, composable architecture and identifies concrete implementation limits and unresolved field-wide questions.

  • Core argument: SemaClaw’s central claim is that reliability, safety, and long-term usefulness are primarily engineering properties of the surrounding harness.The architecture is presented as layered, with mutually reinforcing design decisions.
  • Architecture: The system separates a reusable runtime foundation from the application layer, allowing execution, tool orchestration, and isolation to evolve independently.The application layer adds channels, routing, and persona management.
  • Architecture: Three-tier context management supplies identity, cross-session recall, and task-specific context, while PermissionBridge gates execution at the tool boundary.PermissionBridge applies least privilege through human-in-the-loop control.
  • System properties: The layered harness bounds agent authority, embeds human oversight in execution, and remains open and composable across personal and small-team deployments.The same mechanisms are intended to accumulate value over time.
  • Persistent collaboration: SemaClaw supports persistent collaboration through trustworthy delegation and a memory substrate expressed as wiki-based personal knowledge infrastructure.PermissionBridge governs expanding action authority, while context and the wiki support relational continuity.
  • Contributions: Its contributions include an open harness architecture, operational primitives for orchestration and safety, and a vocabulary for discussing harness-engineering decisions.The listed primitives include DAG-driven dispatch, PermissionBridge, scheduled tasks, and stateful plugins.
  • Limitations: Current limitations include incomplete channel integrations, incomplete alignment with upstream extension conventions, and no cross-source retrieval between wiki content and session memory.The authors frame these scope boundaries as a roadmap for subsequent development.
  • Looking forward: SemaClaw offers a practical reference and starting point rather than a final architecture for personal AI agents.The paper positions its value in exposing explicit engineering choices for community evaluation and improvement.
Loading 2604.11548v1…