PaperlayerView exact revision on arXiv

Source-linked AI summary

Raven: The Harness of Harnesses for Composable Agentic Intelligence

EverMind AI

arXiv:2609.33439v1cs.AIcs.CLcs.GTcs.MAcs.NE

TL;DR

Manual harness design is difficult to scale, and domain-specific assumptions complicate composing agents across workflows. Raven composes executable model–harness pairs through a Host Agent and reuses experience across tasks. It leads the reported MAOB planning comparisons, while the theory gives conditions for composition to expand reliable task coverage.

  • Problem

    Manual harness development is difficult to scale across models and domains, while domain-specific assumptions make it challenging to match subtasks to suitable agents and establish compatible handoffs.

    Source: §1 · p. 4

  • Method

    Raven treats executable model–harness pairs as composable units, coordinates registered agents through task dependencies, and supports adaptation and reuse through harness evolution, persistent memory, and Skill Forge.

    Source: §1 · p. 4

  • Results

    Raven ranks first on all four MAOB planning metrics under both tested backbones, with Exact Match gains of 10.4 and 10.5 percentage points over the strongest baseline.

    Source: §9 · p. 59

  • Takeaways & Limitations

    Under a shared resource budget, composition can expand reliable task coverage beyond individual agents when complementary capabilities, compatible handoffs, and bounded planning and execution errors meet sufficient conditions.

    Source: §1 · p. 5

    The theory does not establish strict set inclusion or improved average performance; empirical comparisons require strong individual-agent baselines with comparable models, tools, information, and total cost.

    Source: §2.5 · p. 12

Abstract

from arXiv · show

As large language models advance, AI agents are moving beyond isolated, domain-specific tasks toward long-horizon, cross-domain workflows. This transition exposes two challenges: increasing harness complexity makes manual design difficult to scale, while tighter coupling to specific domains limits the generality of a single harness. The central question thus shifts from how to engineer a stronger harness for one domain to how to autonomously construct specialized harnesses, improve them through experience, and orchestrate them across domains. We introduce Raven, \emph{The Harness of Harnesses}, an open-source multi-agent ecosystem that automatically constructs and evolves modular harnesses for specific models and domains, treating each executable model--harness pair as a composable unit of intelligence. To support an \emph{All-Domain Collaboration Network}, its Host Agent decomposes goals, matches subtasks to specialized agents, coordinates execution dependencies, and integrates results, while a host archive and EverOS preserve experience across tasks and Skill Forge makes that experience available as reusable procedures. Our theory establishes sufficient conditions for such composition to expand reliable task coverage beyond that of the available individual agents under a shared resource budget. On complex and long-horizon tasks, Raven significantly outperforms the state-of-the-art agent systems, pushing the frontier of composable agentic intelligence.

1 Introduction

Complex goals require specialized work whose outputs must support later stages, while increasing harness complexity and domain-specific assumptions make manual design and cross-agent composition difficult. Raven treats executable model–harness pairs as composable units and combines orchestration, harness adaptation, and experience reuse.

1 Introduction

  • Complex workflows need distinct tools and completion criteria, but harness assumptions and coordination costs make compatible subtask assignment and handoffs challenging.The paper frames the central problem as constructing and improving specialized harnesses while making their capabilities composable across domains.Source: §1 · p. 4
  • Raven connects native and third-party agents through execution adapters and a shared interface, while the Host Agent assigns dependent subtasks and the runtime schedules ready nodes and records handoff artifacts.The architecture also supports modular harness evolution, persistent memory, and skill reuse.Source: §1 · pp. 4–5, Figure 2 · p. 5
  • Raven adapts modular harness policies through diagnosis, candidate generation, and evaluation, while archives, optional EverOS access, and Skill Forge support experience and procedure reuse.The evolution process evaluates candidate harnesses with a frozen task model.Source: §1 · p. 4
  • Under a shared resource budget, complementary capabilities, compatible handoffs, and bounded planning and execution errors can let composition solve tasks individual agents cannot reliably solve alone.The analysis accounts for planning and coordination costs alongside worker execution.Source: §1 · p. 5
  • Raven ranks first among compared systems on all four MAOB graph metrics under both tested backbones, with Exact Match gains of 10.4 and 10.5 percentage points over the strongest baseline.Evaluations also report that a curated skill library improves Raven on three benchmarks, with larger gains than OpenClaw on two.Source: §1 · p. 5

2 Theory

This section states sufficient conditions under which composing harnessed agents can extend reliable coverage to new tasks and task families under a shared resource budget, including typed execution contracts, deterministic soundness, and probabilistic reliability bounds. These guarantees depend on the stated premises and do not imply strict capability inclusion, average-performance gains, or unconditional reliability.

2.1 Harnessed Agents and Task-Relative Capabilities

  • Capability is task-relative under a shared cost budget: each harnessed model is a stochastic policy over permitted histories, and success is judged by the task’s correctness criterion.The system budget includes planning, workers, handoffs, verification, and retries; nontermination counts as unsuccessful, and objective correctness is distinct from a completion declaration.Source: §2.1 · pp. 6–7

2.2 Resource Limits and the Scope of Composition

  • Finite information budgets limit guarantees: fewer black-box queries than target positions cannot guarantee finding every target, and limitations of individual agents do not by themselves establish complementary strengths.The passage also cautions that capability profiles support conclusions only for the evaluated systems and intended task distributions.Source: §2.2 · p. 7

2.3 Host-Mediated Harness Composition

  • A compatible, well-typed plan preserves assertions across operations, so if every operation succeeds within its allocation, deterministic composition soundness guarantees a correct delivered record within budget B.The proof proceeds in the plan’s serial order; parallel execution requires a certified linearization preserving observations, records, state effects, acceptance, and cost.Source: §2.3 · pp. 8–9

2.4 Reliability of Composed Execution

  • For a valid plan whose operations realize their contracts in the actual conditional execution contexts, composed success is bounded below by one minus the accumulated local error bound, without requiring independent errors.An isolated benchmark average does not establish the required relationship between local experiments and actual execution.Source: §2.4 · pp. 9–10, Then the conditional success… · p. 10

Then the conditional success probability satisfies

  • Host-level success is at least (1 − ¯ϵ(t; B))(1 − ηH(t; B)) when valid-plan probability is at least 1 − ηH and the local error bounds have a uniform envelope ¯ϵ.The bound requires compatible plans and providers that realize their contracts in selected contexts; additional nodes can reduce subtask difficulty while increasing handoff cost and error.Source: Then the conditional success… · pp. 10–11

2.5 Capabilities Beyond Individual Agents

  • Composition can exceed individual capability when a task satisfies the system’s reliability hypotheses and its success probability clears the individual-agent upper bound.Local complementarity alone is insufficient: a separate upper bound on complete single-agent performance is needed, and the comparison uses the same task and budget.Source: §2.5 · p. 11
  • 1 versus at most 1/2: a three-node parity plan succeeds with probability 1 within budget B, while every standalone agent’s success is bounded by 1/2.The construction uses agents that can read U or W separately and a synthesis agent that computes parity; its guarantee assumes finite allocations for calls, handoffs, delivery, and planning whose total is at most B.Source: §2.5 · p. 12
  • The capability result establishes new-task coverage, not strict set inclusion or improved average performance; preserving the full union additionally requires feasible single-agent paths and a reliability-preserving selector.The passage says empirical comparisons require strong individual-agent baselines with comparable models, tools, initial information, and total cost.Source: §2.5 · p. 12

2.6 Reliable Task-Family Coverage and Implications for Raven

  • Appendix A.5 defines a typed grammar for primitive, sequential, and fork–join tasks and constructs compatible finite DAGs by structural induction, with fork, join, and artifact transfer costed explicitly.The construction proves plan existence; compiler-based specialization yields valid planning from correct input binding and compilation, while the general result also allows other valid plans.Source: §2.6 · p. 12
  • Uniform bounds ηH(t; B) ≤ η⋆ and ϵ̄(t; B) ≤ ϵ⋆ with η⋆ + (1 − η⋆)ϵ⋆ ≤ δ guarantee pSH(t; B) ≥ 1 − δ for every task in T⋆.This establishes reliable task-family coverage under Proposition 1’s hypotheses, not an unconditional guarantee.Source: §2.6 · p. 12
  • Bounded expanded node and edge counts yield uniform resource and error envelopes, but per-task budgets can meet the reliability target without one finite budget covering the whole family.The analysis does not distinguish an orchestrated system from a single program simulating it with the same interfaces and resources.Source: §2.6 · p. 12
  • The theory assigns Raven’s orchestration, agents, memory, skill construction, and self-evolution distinct analytical roles, but their effects on capability, cost, and conditional reliability require evaluation.Evaluation should pair graph-agreement measures with semantic contract checks, end-to-end outcomes, cost accounting, and failure attribution; average success rates alone do not establish conditional reliability premises.Source: §2.6 · pp. 12–13

3 Multi-Agent Collaboration

This section describes Raven’s multi-agent collaboration mechanisms, including typed dependency graphs, validated execution, artifact handoffs, host-mediated recovery, and bounded, provenance-aware memory sharing. It distinguishes task artifacts from retained experience and notes that the memory layer’s performance effect is unevaluated, while memory context alone does not establish the theory’s semantic guarantees.

3 Multi-Agent Collaboration

  • Raven combines built-in specialists and independently developed agents through execution adapters, coordinating their differing tools and control policies with typed dependency graphs and recorded artifact handoffs.The graph is checked before workers start, and each handoff passes through a recorded artifact.Source: §3 · p. 13

3.1 Task Decomposition and Agent Orchestration

  • The host plans backward from requested deliverables into a dependency graph, while staged orchestration guidance and pre-dispatch checks constrain graph construction and execution.Validation checks format, structure, capabilities, agent status, and environment before dispatch, but does not determine whether assignments are semantically adequate.Source: §3.1 · pp. 14–15

3.2 Host-Mediated Graph Execution

  • Ready nodes run under a shared concurrency limit, and an independent judge evaluates outputs before successors can consume them; suspended work can be continued, abandoned, or replanned by the host.Worker clarification requests are answered from conversation context or memory when possible, with unresolved questions forwarded to the user.Source: §3.2 · pp. 16–18
  • Completion verdicts do not certify objective correctness, and structural scheduling alone establishes no cost advantage; evaluation must account for planning, verification, retries, clarification, and handoffs.If the judge fails or times out, the availability fallback treats a normally returned call as accomplished; domain tests and final artifact inspection provide independent evidence.Source: §3.2 · pp. 16–18

3.3 Multi-Agent Memory Flow

  • File-backed records preserve prompts, outputs, and transcripts, while dependency-scoped memory paths let file-capable agents inspect relevant predecessor experience without eagerly copying it into prompts.Artifact completion releases dependencies independently of asynchronous memory recording.Source: §3.3 · pp. 19–20
  • Completed nodes and agent instance handles persist across runs, allowing later graphs or delegations to reuse recorded outputs and continue sessions instead of repeating work.Only earlier nodes completed with recorded outputs are eligible for reuse; failed, skipped, cancelled, and running nodes are not.Source: §3.3 · p. 20
  • Across sessions, task-aware retrieval and default application and project scopes enable bounded context transfer, while dependency-scoped provenance and experience reuse do not establish improved final-task success.The passage distinguishes graph agreement from evaluation of final-task success.Source: §3.3 · p. 20

3.4 Shared Group Memory

  • Shared group memory uses the host’s broader view of run reports and later user messages to assess agent contributions and inform future orchestration and worker invocations.A round is either a finished graph run or a single delegation; the completion judge evaluates nodes, while group memory assesses agent contributions across the round.Source: §3.4 · p. 20
  • The layer stores host-owned round summaries and outcomes plus per-agent verdicts, then retrieves bounded experience before planning and worker execution.Workers receive selected records through prefetch while retaining ordinary access to their own libraries; access to personal user memory and other owners’ agent libraries is excluded.Source: §3.4 · pp. 20–21, Figure 7 · p. 21
  • Memory-enabled execution requires retrieval and context-producing operations to be explicit in an annotated expanded plan, because supplying a memory block alone does not establish semantic guarantees.The expanded plan accounts for retrieval costs and recorded outputs while preserving the theory’s assembly restriction.Source: §3.4 · p. 22
  • At round closure, the host records deterministic outcomes and per-agent close verdicts, then can revise outcomes with user feedback on the next turn.Execution-structure labels such as consumed or terminal do not establish output quality or user acceptance; feedback verdicts are appended alongside close verdicts.Source: §3.4 · pp. 22–23
  • The append-only host ledger preserves authoritative assessment text and provenance, but host verdicts may be incorrect and group memory is disabled by default and unevaluated.Its effect on orchestration and task performance remains to be established.Source: §3.4 · p. 23

4 Agent Harness Self-Evolution

Raven evolves a frozen-model harness through bounded policy edits, diagnosis-driven candidate search, and evaluation, retaining candidates in a semantic bank.

4.1 Modular Harness Design

  • Four strategy interfaces expose editable decisions around a frozen model, while a shared shell handles dispatch, accounting, and bounded execution control.The Memory, Planning, Capability, and Action roles govern context, turn guidance, tool exposure, and response or completion judgments.Source: Figure 8 · p. 24, §4.1 · pp. 24–25

4.2 Diagnosis-Driven Harness Evolution

  • Failure diagnoses connect observed trace events to modifiable mechanisms, and the Evolver proposes patches without updating the frozen Task Agent’s weights.Development and deployment costs are measured separately; the utility objective has no implicit cost penalty.Source: §4.2 · p. 26

4.3 Harness Gene Bank and Recombination

  • The semantic gene bank stores complete evaluated harnesses by edit category and diagnosed pathology, enabling synthesis or recombination while requiring combined candidates to be evaluated.Candidates can enter a new or weaker archive cell without becoming the next parent; evolution gains may be unstable, transfer only partially to held-out tasks, and depend on the executing model.Source: §4.3 · pp. 26–27

4.4 Gated Harness Screening and Selection

  • Screening rejects invalid or inactive candidates before testing paired gain, then confirms survivors on the full training set before updating the archive.The selected harness is frozen before held-out testing, whose outcomes do not revise the search; repeated adaptive training screening provides no family-wise significance guarantee.Source: §4.4 · pp. 27–29

5 Memory System

EverOS structures interaction history into episodes, consolidates them into scenes and profiles, retrieves query-relevant episodes and facts, and keeps user context distinct from reusable agent procedures.

5.1 Episodic Trace Formation

  • A semantic-boundary detector groups contiguous messages into segments, preserving context for episode formation while buffering messages until a later boundary or explicit flush.Episodes provide narratives and atomic facts; source metadata links representations to their origins, and agent memory can retain tool calls and results.Source: §5.1 · p. 30

5.2 Semantic Consolidation

  • Online clustering groups episode embeddings into time-constrained MemScenes, which select source evidence for profile updates; optional reflection can merge episodes and deprecate superseded records.A recurring topic can occupy multiple scenes when occurrences fall outside the allowed time gap.Source: §5.2 · pp. 30–32

5.3 Reconstructive Retrieval

  • HYBRID retrieval searches episodes lexically and densely, uses reciprocal rank fusion to order expansion, then ranks episode and fact candidates by calibrated relevance within a bounded set.Selected facts return under their parent episodes, restoring context; RRF sets expansion order while calibrated scores determine final relevance.Source: §5.3 · pp. 32–33, Figure 10 · p. 32

5.4 User Context Assembly

  • For ordinary turns, Raven combines backend user recall with a host archive, selecting profile sections by lexical overlap and including all Notes sections.A profile’s adapter score of 1.0 is a placement convention, not a relevance estimate or confidence value.Source: §5.4 · p. 33

5.5 Agent Experience and Cross-Session Reuse

  • The agent memory track turns execution cases into reusable skills, while Markdown records persist memory separately from operational state and a derived retrieval index.The host can read worker-session records, and resulting memory artifacts can be made available to downstream graph nodes.Source: §5.5 · p. 34

6 Skill Forge

Skill Forge makes experience reusable by curating and retrieving task-appropriate skills, and by turning stored execution cases and clustered experience into owner-scoped procedures through bounded, validated updates. Its catalog spans SkillHub, local, and EverOS skills; accepted updates or high-confidence skills alone do not demonstrate task improvement.

6.1 Curated Skill Catalog

  • SkillCorpus builds SkillHub’s catalog through parsing, filtering, deduplication, quality and safety review, and release admission; its reference snapshot contains 96,401 active skills.The count describes a fixed corpus snapshot and may differ from the live SkillHub catalog.Source: §6 · p. 34, §6.1 · p. 35

6.2 Task-Aware Skill Retrieval and Selection

  • The router rewrites or skips the task query, fuses local, memory, and SkillHub results, then applies policy checks and an LLM gate that considers available tools and specialists.If the gate call or response parsing fails, a separate ranked fallback is used; resource resolution occurs after selection.Source: §6.2 · pp. 36–37

6.3 Experience-Driven Skill Self-Evolution

  • EverOS skips low-quality cases, clusters eligible cases by task intent, and uses a structured LLM call to propose additions or updates to existing skills.Cases below θskip or deployments without embedding capability produce no update; update prompts distinguish failure from success by case quality.Source: §6.3 · pp. 38–39
  • Emitted revisions are stored as owner-scoped EverOS Markdown skills with case lineage for later retrieval, while skills with confidence below 0.1 are not automatically retired.Cascade indexing later exposes revisions to the memory source; optional maturity scoring is skipped by default.Source: §6.3 · pp. 39–40
  • Skill updates cluster intents, select bounded catalog skills and supporting cases, validate proposed operations, and write accepted records under sanitized owner-scoped names.Updates are skipped when embeddings are unavailable or quality falls below θskip; malformed top-level output errors before writes, and records below 0.1 confidence are still emitted and retrievable.Source: §6.3 · p. 40
  • Catalog quality, skill confidence, and measured task outcome are distinct; Section 7.7 evaluates the fixed catalog’s task-performance effect with the model and harness held fixed.Neither a structurally accepted update nor a high-confidence skill establishes improvement.Source: §6.3 · p. 40

7 Evaluation

Raven’s evaluation examines orchestration, harness adaptation, and specialist workflows across planning quality, held-out task performance, and procedural reuse. It reports findings across research, repository coding, visual design, and unattended computing, including resource conditions, workflow mechanisms, and limits on gains.

7.1.1 The Multi-Agent Orchestration Benchmark

  • MAOB pairs 140 occupational-task requests with reviewed reference DAGs over four specialists, covering all 11 domain subsets containing at least two domains.The benchmark spans 137 occupations; 98 tasks are purely serial and 42 allow parallel execution. Its coverage is exhaustive only for combinations within the declared roster, not all occupational scenarios or tool configurations.Source: §7.1.1 · pp. 41–42

7.1.2 Metrics

  • MAOB reports four separate domain-level metrics—Node F1, Edge F1, POA, and Exact Match—to distinguish specialist selection, dependencies, precedence, and whole-plan agreement.The benchmark normalizes dependency edges through transitive reduction, so redundant transitive edges are not penalized; POA is interpreted alongside Node F1.Source: §7.1.2 · pp. 43–44

7.1.3 Experimental Setup

  • The orchestration comparison tests Raven, Claude Code, and Hermes Agent on matched inputs and backbones, scoring submitted plans before workers execute.The evaluation uses Qwen3.8-27B and DeepSeek-V4-Flash-0731 over the same 140 tasks; all systems receive identical request text and delegation instructions, with domain descriptions supplied through their native tool schemas.Source: §7.1.3 · p. 44

7.1.4 Results

  • Raven outperforms both orchestration baselines on all four metrics with each backbone, including 0.923 Node F1 versus 0.776 for the strongest Qwen3.8-27B baseline.With DeepSeek-V4-Flash-0731, Raven reaches 0.963 Node F1 and 0.897 Edge F1, versus strongest-baseline scores of 0.854 and 0.813. Exact Match rates are 0.711 versus 0.607 with Qwen3.8-27B and 0.867 versus 0.762 with DeepSeek-V4-Flash-0731. Edge F1 remains lower than Node F1 for every system under both backbones.Source: Figure 13 · p. 44, §7.1.4 · pp. 44–45

7.2 Harness Self-Evolution

  • Harness evolution improves held-out Pass@1 on all seven benchmarks with a frozen Qwen3.6-27B backbone.The largest gains are 15.4 percentage points on AppWorld, 13.9 on BrowseComp+, and 13.7 on LiveCodeBench; six comparisons pass the source’s paired-gain criterion, while SWE-bench Verified improves but does not pass it on its 26-task test split. The evolved harness is selected on training tasks and assessed on held-out test tasks.Source: §7.2 · p. 45, Figure 14 · p. 46

7.3 Raven-Research: Evidence-Grounded Deep Research

  • Raven-Research achieves the highest pooled accuracy across all three shared backbones, exceeding the strongest baseline by 6.6, 3.3, and 7.6 percentage points, respectively.With DeepSeek-V4-Flash, it scores 69.3% on BrowseComp and 60.0% on Humanity’s Last Exam; all three harnesses exceed 86% on FRAMES, while DeepSeek-Harness leads Raven-Research by 3.4 percentage points on xBench-DeepSearch. Raven-Research nevertheless has the strongest aggregate performance across the combined task set.Source: §7.3 · p. 46, Table 4 · p. 47
  • Five of six same-backbone paired accuracy comparisons for Raven-Research have exact McNemar p < 0.02; the remaining comparison has a positive 3.3-percentage-point difference with p = 0.21.The remaining comparison is against DeepSeek-Harness with Qwen3.5-397B-A17B; the numerical gain is positive across all six comparisons, with statistical support in five.Source: Figure 15 · p. 47, §7.3 · p. 48
  • With DeepSeek-V4-Flash, Raven-Research reaches 0.0242 USD mean cost per question, between the two locally evaluated baselines, and outperforms commercial services scoring 50.3% to 63.2% accuracy.It uses fewer input tokens and more output tokens than both local baselines; the commercial services use their own models and search stacks with provider-specific cost accounting.Source: Figure 15 · p. 47, §7.3 · p. 48

7.4 Raven-Code: Repository-Aware Software Engineering

  • Raven-Code leads six of eight coding settings; on SWE-Refactor it beats leaderboard entries by 9.5 points with DeepSeek-V4-Flash and 3.0 with GPT-5.6 Luna.On SWE-bench Pro it resolves 15 more tasks than Claude Code with the same Qwen3.8-27B backbone; on Verified its advantages are 0.6 points with DeepSeek-V4-Flash and 1.2 with Claude Opus 4.8. On WorkBuddy-Code it beats three local baselines by 1.4–4.0 points with DeepSeek-V4-Flash, while other backbones remain close to the strongest alternatives. Its repository-aware harness supplies project instructions and verifies edits through current-file checks, syntax checks, and tests.Source: §7.4 · pp. 48–49, Figure 16 · p. 49
  • Raven-Code with Claude Opus 5 reaches DataAgentBench Pass@1 0.8762 and Pass@5 0.9097, with the highest Pass@1 among entries on August 24, 2026.The benchmark covers 54 natural-language questions over 12 datasets stored across PostgreSQL, MongoDB, SQLite, and DuckDB.Source: §7.4 · p. 49, Figure 17 · p. 50

7.5 Raven-Design: Render-and-Inspect Visual Design

  • Raven-Design scores 80.2 versus Claude Code’s 78.3 with Claude Opus 5, and 72.9 versus 52.4 with GPT-5.6 Luna on PresentBench.Both Raven-Design scores exceed every public leaderboard entry shown. It also leads in all six visual-task benchmark and backbone settings, by 0.3–2.8 points over the strongest alternative; the largest individual-baseline gap is 15.5 points on GPT-5.6 Luna Dashboard tasks. Its workflow renders outputs for inspection before delivery.Source: §7.5 · pp. 49–51, Figure 18 · p. 50, Figure 19 · p. 51
  • Seven showcase runs demonstrate Raven-Design composing research, code, and verification into decks, posters, and an interactive website.The four decks use distinct subject-specific visual styles while maintaining internal consistency; the task graphs also show specialists contributing to intermediate and final artifact stages. These are qualitative examples without human preference ratings.Source: §7.5 · pp. 51–53, Figure 20 · p. 52, Table 5 · p. 52

7.6 Raven-Oncall: Unattended Run-and-Watch Automation

  • On autoresearch, Raven-Oncall reaches 1.038 BPB versus Claude Code’s 1.053 under Claude Opus 5, in similar runtimes of 4.90 versus 4.99 hours.With DeepSeek-V4.1-Flash it reaches 1.037 BPB in 3.74 hours at a reported cost of 0.97 USD. Each persistent campaign records its target, measurements, permitted actions, and budget; decisions and final reports are gated by recorded evidence.Source: Figure 21 · p. 53, §7.6 · p. 53
  • On AI4S, Raven-Oncall solves 14 of 17 scientific tasks with Claude Opus 5, versus Claude Code’s 11, while reducing average runtime, token use, and cost.With DeepSeek-V4-Flash it solves 7 tasks at an average cost of 0.10 USD. The benchmark spans 17 sustained scientific and technical tasks, and success is judged against task-specific reference values and evidence withheld from the agent.Source: §7.6 · pp. 53–54, Figure 22 · p. 54

7.7 Skill Retrieval and Reuse

  • Skill retrieval improves all 12 harness–backbone–benchmark comparisons; pooled gains are +7.5 ± 2.3 SkillsBench points, +1.51 ± 0.49 GDPval points, and +2.79 ± 0.70 QwenClawBench points.The fixed SkillCorpus library and identical precomputed skill selections let Raven and OpenClaw be compared under the same procedural input. Raven gains more than OpenClaw on SkillsBench and QwenClawBench with both backbones; with Qwen3.5-397B-A17B it reaches 22.6% SkillsBench Pass@1.Source: §7.7 · pp. 54–55, Figure 23 · p. 55
  • Trace inspection links Raven’s stronger skill use to executing, verifying, and fixing work; skill gains also rise with coverage, while GDPval cell gains are not statistically distinguishable from zero.On SkillsBench, replacing trained retrieval with off-the-shelf models lowers Pass@1 from 22.6% to 13.8%, and replacing the curated catalog with a raw crawl lowers it to 14.9%; both remain above the 9.2% no-skill baseline. Pooled gains rise from 2.2 points for low-coverage tasks to 25.1 for tasks scoring at least 0.75. SkillRouter comparison: on the Hard tier, Hit@1 is 0.760 versus 0.720. Mathematics and Finance & Economics show near-zero gains in one tested cell.Source: §7.7 · pp. 55–56

8 Related Work

Raven’s related-work discussion positions its explicit orchestration contracts, evolvable harnesses, scoped memory, and task-aware skill reuse as complementary mechanisms for composing heterogeneous agents.

8.1 Multi-Agent Collaboration and Orchestration

  • Prior work finds that collaboration gains depend on task structure and architecture, since complementary agents may add coverage while redundant messages and handoffs consume budget.This motivates evaluating collaboration-plan structure separately from final artifact quality [38].Source: §8.1 · p. 56
  • Raven’s host can select heterogeneous executors at invocation time and validate typed DAG dependencies and artifact handoffs before execution.The runtime checks capabilities and records artifacts for later nodes, separating planning from execution.Source: §8.1 · pp. 56–57

8.2 Agent Harnesses and Self-Evolution

  • Raven factors harnesses into four strategy interfaces and freezes the selected harness before service, distinguishing its approach from prompt-only optimization, workflow synthesis, and unrestricted self-editing.The interfaces are Memory, Planning, Capability, and Action; candidate changes are screened through diagnosis and gated evaluation.Source: §8.2 · p. 57

8.3 Long-Term Memory for Agents

  • EverOS separates user profiles and episodes from agent cases and experience-derived skills, while Raven keeps retrieved memories distinct from declared artifact dependencies.The host controls which memories are visible at each invocation, and retrieved text is marked as untrusted context.Source: §8.3 · p. 58

8.4 Skill Acquisition, Retrieval, and Evolution

  • Raven selects across local, memory-derived, and SkillHub procedures with provenance, bounded retrieval, and compatibility checks, while updates are proposed separately for later indexing.This keeps retrieval available while skill revisions are pending.Source: §8.4 · p. 59

9 Conclusion and Future Work

Raven combines agent composition, harness evolution, memory, and skill reuse; the conclusion reports component evaluations and outlines a future network across devices. The accompanying appendix bounds what composition can establish under finite budgets and specified execution assumptions.

9 Conclusion and Future Work

  • Raven’s theory gives sufficient conditions for composition to expand reliable task coverage beyond individual agents under a shared budget, while evaluation reports gains across planning, specialist, harness, and skill capabilities.On MAOB, Raven ranks first on four planning metrics under both tested backbones, with Exact Match gains of 10.4 and 10.5 percentage points over the strongest baseline.Source: §9 · p. 59
  • The proposed future network would connect Raven instances across devices to exchange capabilities, delegate tasks, transfer context, and synchronize ongoing work.The envisioned network would support continuity across interfaces and be evaluated for task success, user continuity, and coordination cost.Source: §9 · p. 59

A.1 Notation and Scope

  • The appendix fixes system versions and an input law; conditional probability arguments condition only on events with positive probability.Its countable encoding convention makes this conditioning unambiguous without probability densities.Source: §A.1 · p. 65

A.2 Operation Records and Execution Semantics

  • The execution semantics use immutable ledger records and serially schedule accepted operations, with final delivery and its cost included in the last operation.Applying the analysis to concurrent execution requires a measurable serial representation preserving inputs, outputs, transitions, acceptance, terminal records, and charged cost.Source: §A.2 · pp. 65–66

A.3 A Finite-Budget Information Limit

  • With fewer black-box queries than target positions, some target’s success probability is at most nqry/Npos; an unevaluated final guess changes the bound to min{1, (nqry + 1)/Npos}.The argument does not rule out advantages from structured priors or exhaustive search with enough queries.Source: §A.3 · pp. 66–67

A.4 From Local Experiments to Workflow Reliability

The reliability analysis connects local executor guarantees to workflow success through successful-prefix conditions, context-law consistency, and feasible eligible assignments.

Local-to-workflow consistency

  • A local provider’s error bound carries into a workflow only when its invocation law matches the actual conditional entry law for each positive-probability successful prefix.The resulting local bound is conditioned on prior correctness as well as the visible prefix.Source: §A.4 · p. 67

Consistency assumptions

  • If planning or earlier outcomes change the conditional law of omitted variables, the configuration must include that information or the outcome model must be refined.Multiplying marginal success probabilities is generally invalid; the analysis instead conditions on prior correctness and uses actual marginal failure rates for a union bound.Source: §A.4 · p. 67

Resource feasibility

  • The resource envelope gives a sufficient design-time condition for feasibility, while a large sum of error bounds weakens the lower bound without proving actual performance falls.The stated bound accounts for bounded plans and handoffs under a shared budget.Source: §A.4 · p. 68

Eligible assignments

  • An eligible executor assignment supports the workflow reliability bound only when it is also compatible and feasible, transfers realize their contracts, and context conditions hold.Nonempty eligible sets alone do not certify context membership, and the proposition does not rule out other ways an individual agent could solve the complete task.Source: §A.4 · p. 68

A.5 A Typed Task Grammar and Its Compilation

The appendix defines a restricted typed task grammar and a compilation procedure, then establishes conditions under which compiled plans preserve task relations and support coverage guarantees. Parallel branches require state separation, and reliability bounds additionally depend on invocation-law assumptions.

A Typed Task Grammar and Its Compilation

  • The grammar builds typed expressions from primitives using guarded sequence and fork–join constructors, with preconditions, postconditions, executor assignments, and state footprints.The restricted task grammar is fixed before observing host success; sequence wiring uses a declared typed projection or serialization.Source: §A.5 · p. 68
  • Parallel branches require a read-only shared state component and disjoint mutable components; concurrent execution additionally requires trace preservation.Nested partitions cannot expose a sibling’s mutable state.Source: §A.5 · pp. 69, 71
  • Compilation makes transfers explicit: sequences hand off a transformed first-child output, while fork–join plans distribute branch inputs and return ordered outputs to the join.Added transfers preserve mutable task state on successful transitions.Source: §A.5 · pp. 68, 70
  • Proposition 4 establishes that serial compilation yields a finite compatible plan whose successful execution establishes the declared input/output relation.Successful prefixes enable the next operation, including within compiled fragments.Source: §A.5 · p. 70
  • Node, handoff, cost, and conditional-error allocations are composed through the plan, with sequence handoffs counted explicitly.A primitive contributes one node and no handoffs; repeated occurrences may have different allocations.Source: §A.5 · p. 71
  • Reliability bounds require contract realizations to cover every invocation law induced by compiled graphs, including inputs assembled from earlier outputs.Typing and structural induction alone do not establish this distributional premise.Source: §A.5 · p. 71
  • The task family bounds expanded nodes and handoffs while fixing task specifications and input laws independently of observed agent success.Correct binding preserves the reference input and task-state projection throughout planning.Source: §A.5 · pp. 69, 71

Coverage through correct compilation

  • Under the stated planning, provider-realization, and kernel-consistency conditions, Corollary 3 establishes coverage through correct compilation for tasks in the bounded graph-size family.The family may contain infinitely many tasks, and the construction is a sufficient way to meet the coverage conditions.Source: §A.5 · pp. 71–72

B Method Implementation Details

This appendix records interfaces, implementation defaults, and protocol provenance for Sections 3–6; experimental configurations are described in Appendix C.

B Method Implementation Details

B.1 Notation Across Method Modules

The appendix directs readers to notation tables and specifies where graph and prompt interface contracts are defined.

B.1 Notation Across Method Modules

  • Table 6 collects theory notation, while Tables 7 and 8 define method-specific objects; local indices and type-distinguishing typography are scoped to their stated subsections.Source: §B.1 · p. 73, Table 7 · p. 73, Table 8 · p. 74

B.2 Graph and Prompt Contracts

  • Table 9 specifies graph and model interface fields, while graph admission and skill-response handling are defined by Table 1 and Eq. (73).The case-to-skill operation format is defined in Section 6.3.Source: §B.2 · p. 73, Table 9 · p. 74

B.3 Implementation Defaults

The appendix collects implementation defaults, harness-evolution evidence, and the source prior used for skill curation, while noting that published results lack complete per-run settings.

B.3 Implementation Defaults

  • EverOS supplies memory and case-to-skill defaults, while Raven supplies graph-execution, recall-adaptation, and skill-routing defaults.SkillHub detail checks reject blocked skills and incompatible external-home-path references; missing or malformed retrieved detail scores do not trigger the safety-threshold rejection.Source: §B.3 · pp. 74–75, Table 10 · p. 75

B.4 Harness Evolution Protocol and Evidence

  • The seven reported benchmark results reproduce HarnessBank v2 results under a frozen Qwen3.6-27B task model, Claude Opus 4.8 evolver, and three attempts per task.The source does not provide a complete per-run manifest, including each run’s screening subset and proposal order or all specified settings and retry limits.Source: §B.4 · p. 75

B.5 Source Prior for Skill Curation

  • SkillCorpus’s source prior uses a shrinkage estimate with K0 = 10 and µ0 = 0.685, and the settings were fixed before downstream evaluation.For an empty source, the track score reduces to µ0; the source reports that benchmark outcomes were not used to optimize these settings.Source: §B.5 · pp. 75–76

C Evaluation Details

The evaluation appendix supplements the main results with scoring conventions and evaluation details, including reproduced results for harness self-evolution. It also specifies how orchestration predictions and metric averages are handled.

C Evaluation Details

  • The appendix records scoring conventions, evaluation configurations, grading procedures, cost accounting, and departures from official benchmark protocols.Its tables reproduce harness-evolution, skill-library, and skill-ablation results from their cited sources.Source: C Evaluation Details · p. 77

C.1 Multi-Agent Orchestration

  • Domain folding can create projection conflicts, leaving Edge F1, POA, and Exact Match unavailable; cyclic predictions without that flag receive zero scores under specified conventions.Metric means omit unavailable values, so different metrics can have different denominators.Source: §C.1 · p. 77
  • The orchestration scorer selects among annotated references by the highest lexicographic tuple of Node F1, Edge F1, and POA.Within the selected reference, Edge F1 uses the best accepted reduced edge set, while POA and Exact Match use annotated accepted pair relations.Source: §C.1 · p. 77

C.2 Harness Self-Evolution

  • Table 11 lists the values plotted in Figure 14 for held-out Pass@1 before and after harness self-evolution with frozen Qwen3.6-27B.The table reports scores and percentage-point gains as reproduced from HarnessBank; the supplied passages do not give the values.Source: §C.2 · p. 77, Table 11 · p. 77

C.3 Raven-Research

The Raven-Research evaluation combines a mixed-source research benchmark with specified grading, system configurations, and budgets. Its comparisons are observational, and the reported scores may be affected by benchmark exposure.

C.3 Raven-Research

  • DeepResearch Mixed combines BrowseComp, FRAMES, Humanity’s Last Exam, and xBench-DeepSearch release 2510, with Humanity’s Last Exam evaluated in its text-only exact-match setting.Source: §C.3 · p. 77
  • An LLM judge grades short answers for semantic equivalence, unanswered questions count as incorrect, and regrading changed accuracy by approximately 1.3 percentage points.Token counts are per-question means; cost accounting differs across the DeepSeek-V4-Flash group and commercial services.Source: §C.3 · p. 78
  • Raven-Research used up to 150 tool iterations and per-question limits of 90, 120, or 200 minutes, while systems retained different turn limits, time limits, and context policies.The three commercial services ran end to end with their own models and search; each system answered each question once between August 10 and August 18, 2026.Source: §C.3 · p. 78
  • Because harness limits and context policies differ, the results are observational and do not isolate individual harness components; benchmark exposure may inflate absolute BrowseComp and Humanity’s Last Exam scores.The same questions were used during research-flow development, and evaluated systems did not block access to public benchmark copies.Source: §C.3 · p. 78

C.4 Raven-Code

The appendix documents task-specific evaluation procedures for Raven-Code, Raven-Design, and Raven-Oncall. These procedures vary in benchmark protocols, execution conditions, and grading methods.

C.4 Raven-Code

  • Raven-Code and its baselines were run through AgentEval, generally with one attempt per task, and infrastructure failures counted as unsolved.The supplied evaluation details span SWE-bench Pro and Verified, WorkBuddy-Code, SWE-Refactor, and DataAgentBench, with distinct network, attempt, scoring, and model conditions.Source: §C.4 · pp. 78–79
  • DataAgentBench used five trials per query with hints enabled, while its Claude Opus 5 entry also used Claude Haiku 4.5 and Claude Sonnet 5 for semantic mapping.The entry therefore counts as a tuned-prompt system under the leaderboard categories and had not been submitted to the maintainers.Source: §C.4 · p. 79

C.5 Raven-Design

  • Raven-Design evaluations used one attempt per task in constrained containers, with a deck pipeline that fixed slide coordinates and typography before the agent filled layout slots.Source: §C.5 · p. 79
  • PresentBench scores exclude failed generations, and not every Raven-Design or Claude Code configuration covered the full task set; other design evaluations used internal grading that differs from official protocols.For ArtifactsBench and GDPval, an internal grader assessed rendered deliverables; benchmark task subsets and grading procedures are described in the appendix.Source: §C.5 · p. 79

C.6 Raven-Oncall

  • AI4AI restricted Raven-Oncall to changing the training script, fixed the data pipeline and evaluation function, and capped each training run at 300 seconds within a 5-GPU-hour campaign.Reported BPB averages several seeds; runtime covers the full campaign, including work outside training runs.Source: §C.6 · pp. 79–80
  • AI4S covers 17 scientific and operational tasks, with task-specific budgets and manual judgments against rubrics containing reference values and evidence withheld from agents.All systems received the same budgets and machines; Claude Code's simulation and batch-inference prompts included machine addresses and job commands that Raven-Oncall selected itself.Source: §C.6 · p. 80

C.7 Skill Retrieval and Reuse

The Skill Retrieval and Reuse appendix describes benchmark scoring, retrieval protocols, and tables reporting skill comparisons and an ablation. The supplied passages define the comparisons but do not provide their numerical outcomes.

C.7 Skill Retrieval and Reuse

  • SkillsBench reports binary pass rate over 87 tasks, counting four tasks that fail to build in the evaluation environment as failures under both conditions.Absolute baseline scores are not directly comparable with leaderboard values under other configurations, but within-cell comparisons retain matching inference settings.Source: §C.7 · p. 80
  • The retrieval evaluation follows SkillRouter on 75 core SkillsBench tasks, and Hit@1 measures whether the top-ranked skill belongs to the task’s designated ground-truth skill set.Table 13 reports a single-run ablation of one component at a time on Raven × Qwen3.5-397B-A17B, measured against the no-skill baseline.Source: §C.7 · p. 80, Table 13 · p. 81
  • Table 12 compares three-run means with and without SkillCorpus skills within each harness–backbone cell, and reports task-clustered pooled gain with a standard error and z-score.SkillsBench reports Pass@1, while GDPval and QwenClawBench report mean rewards multiplied by 100.Source: §C.7 · p. 80, Table 12 · p. 81

Related papers

Loading 2609.33439v1…