PaperlayerView exact revision on arXiv

Source-linked AI summary

PatchBoard: Schema-Grounded State Mutation for Reliable and Auditable LLM Multi-Agent Collaboration

Shuyu Zhang, Yaqi Shi, Jiarui Zhang, Yanxiao Zhao, Lu Wang

arXiv:2605.29313v2cs.CL

TL;DR

Natural-language dialogue and loosely structured shared memory make intermediate state difficult to validate, attribute, and audit. PatchBoard replaces dialogue with schema-grounded JSON Patch mutations, validated transactionally by a deterministic kernel; across 630 matched ALFWorld episodes, it reports higher success and lower tokens per successful task than LangGraph and Flock.

  • Problem

    Natural-language dialogue and loosely structured shared memory make intermediate state difficult to validate, attribute, and audit.

    Source: §1 · p. 1

  • Method

    PatchBoard uses an Architect-defined task schema and workflow with a deterministic kernel that validates role-authorized JSON Patch mutations before transactional commit.

    Source: §1 · p. 2

  • Results

    84.6% success and 45.5k tokens per successful task, versus LangGraph’s 30.8% and 368.3k and Flock’s 61.6% and 64.2k, across 630 matched ALFWorld episodes.

    Source: §1 · p. 2

  • Takeaways & Limitations

    PatchBoard makes collaboration a sequence of schema-grounded, role-authorized, replayable mutations, providing a tighter runtime boundary for reliable and auditable coordination.

    Source: §6 · p. 8

    Structural validation does not guarantee semantic correctness: schema-valid claims may still be false, incomplete, or unsupported.

    Source: Limitations · p. 9

Abstract

from arXiv · show

LLM multi-agent systems often coordinate through natural-language dialogue or loosely structured shared memory, making intermediate state difficult to validate, attribute, and audit. We introduce PatchBoard, a schema-grounded collaboration architecture that replaces inter-agent dialogue with validated JSON Patch mutations over a shared structured state. An Architect agent constructs a task-specific schema and workflow rules, while a deterministic kernel validates each proposed state mutation against schema constraints, role-specific write contracts, and runtime invariants before committing it transactionally. On 630 matched ALFWorld episodes, PatchBoard achieves an 84.6% success rate, compared with 30.8% for LangGraph and 61.6% for Flock, while reducing tokens per successful task to 45.5k, compared with 368.3k and 64.2k, respectively.

1 Introduction

Natural-language coordination can leave long-running agents with ambiguous, unverified shared state, while existing structure and formatting controls do not fully define what may be committed. PatchBoard uses schema-grounded state mutations and a deterministic validating kernel; on matched ALFWorld episodes, it outperforms LangGraph and Flock in success and tokens per successful task.

1 Introduction

  • Natural-language histories mix task facts with metadiscussion and repairs, making committed state unclear and allowing unverified information to affect later agents.The introduction notes that polluted shared context can influence subsequent reasoning, tool calls, and environment actions.Source: §1 · p. 1
  • Workflow, code, and blackboard approaches leave runtime questions about trusting generated control logic or whether shared-state updates are valid, authorized, fresh, and safe to commit.Structured output controls formatting but does not itself establish a boundary between a model suggestion and committed state.Source: §1 · pp. 1–2
  • 84.6% success and 45.5k tokens per successful task, versus LangGraph’s 30.8% and 368.3k and Flock’s 61.6% and 64.2k, on 630 matched ALFWorld episodes.The reported cost metric is tokens per successful task.Source: §1 · p. 2
  • PatchBoard replaces open-ended inter-agent dialogue with restricted JSON Patch mutations over shared structured state, making collaboration explicit, typed, and auditable.An Architect defines the task schema, worker contracts, context budgets, and workflow rules.Source: §1 · p. 2
  • A deterministic kernel validates proposed updates against schema and role-specific write constraints, commits accepted patches transactionally, and records replayable logs.The prototype evaluation also includes blackboard controls, ablations, sensitivity analyses, fault injection, and a diagnostic QA study.Source: §1 · p. 2

2 Related Work

Prior work explores agent communication, structured workflows, shared memory, and constrained outputs, but leaves open how shared state should be authorized and audited at runtime. PatchBoard positions schema-grounded, role-authorized, replayable mutations as a tighter coordination boundary.

LLM multi-agent coordination

  • LLM multi-agent systems use role-specialized agents and explicit interaction protocols, while coordination failures, ambiguous handoffs, and redundant communication remain challenges.Examples include conversational infrastructures and role-based workflows for software development.Source: §2 · p. 2

Structured workflows and verification-aware orchestration

  • Workflow graphs, state machines, LM programs, and automated planning move coordination toward explicit control structures.These approaches make execution more explicit than free-form chat.Source: §2 · p. 2

Shared memory, blackboards, and agent memory

  • Blackboards and agent memory support shared or persistent state, but leave open how state is updated, authorized, and audited during long-horizon collaboration.Source: §2 · p. 2

Structured outputs, transactions, and semantic verification

  • Structured generation reduces formatting errors, while transactional runtime guarantees and evidence-grounded QA address related control and semantic-support concerns.These strands connect output constraints, transaction guarantees, and semantic verification.Source: §2 · pp. 2–3

Related-work synthesis

  • PatchBoard advances these lines by treating collaboration as schema-grounded, role-authorized, replayable state mutations with a tighter runtime boundary.Source: §2 · p. 3

3 Methodology

PatchBoard coordinates workers through validated mutations to shared structured state, with an Architect defining the task blueprint and a deterministic kernel controlling runtime updates. Budgeted views and logged transitions provide explicit constraints on what workers can see and change.

3.1 Method Overview

  • The kernel accepts a worker patch only after applying it to a temporary state and checking schema constraints and runtime invariants.Accepted patches commit transactionally; rejected patches are logged without changing committed state.Source: §3.1 · p. 3

3.2 Architect and Task Blueprint

  • The Architect creates a blueprint specifying the task schema, worker roles, context budgets, and workflow rules before workers run.The kernel checks the blueprint against a meta-schema and rejects structurally invalid blueprints before execution.Source: §3.2 · p. 3

3.3 Schema-Grounded Patch Interface

  • Workers use a restricted, schema-grounded JSON Patch interface, with path-level write permissions and limited operations for adding or replacing fields.Destructive removal is disabled by default and available only to privileged roles.Source: §3.3 · p. 4

3.4 Deterministic Kernel

  • The deterministic kernel validates worker outputs, commits accepted patches, logs transactions, and schedules future workers from events emitted by committed updates.It can also respond to repeated invalid patches, no-op edits, exhausted budgets, or repeated state hashes by stopping or redirecting execution.Source: §3.4 · pp. 4–5

3.5 Budgeted Context Views

  • The kernel builds each worker a bounded view from its read contract and context budget, prioritizing active fields and compressing lower-priority collections when needed.Workers can request typed expansions; view hashes allow calls to be traced to the state slice they received.Source: §3.5 · p. 5

3.6 Structural Properties

  • Conditioned on a valid blueprint and correct kernel, schema checks reject malformed or unauthorized updates, while transaction records make accepted trajectories attributable and replayable.The logs preserve worker identity, viewed-state hashes, patches, rejection reasons, and resulting-state hashes.Source: §3.6 · pp. 5–6

4 Experimental Setup

The evaluation tests task success, normalized cost, and structural behavior using matched ALFWorld episodes, a HotpotQA diagnostic, and controlled fault injection. The setup uses common execution conditions and reports confidence intervals for its main metrics.

4.1 Evaluation Goals

  • The evaluation asks whether schema-grounded mutation improves task success and normalized cost, whether shared memory alone explains gains, and which components contribute.Fault injection and diagnostic QA assess structural state validity separately from semantic support.Source: §4.1 · p. 6

4.2 Benchmarks and Systems

  • ALFWorld uses 126 matched gamefiles across six task types and five seeds each, for 630 episodes per system, with LangGraph and Flock as baselines.The secondary HotpotQA diagnostic uses 240 matched validation examples to assess evidence-grounded claim propagation; Figure 2 summarizes ALFWorld success, steps, and tokens per success under matched seeds and gamefiles.Source: §4.2 · p. 6, Figure 2 · p. 7

4.3 Metrics and Estimation

  • ALFWorld metrics include success, environment steps, total tokens, and tokens per success; HotpotQA metrics include accuracy, unsupported claims, evidence coverage, and verified claim precision.The study reports 95% confidence intervals, using Wilson intervals for proportions and paired bootstrap intervals for continuous measures.Source: §4.3 · p. 6

4.4 Fault Injection

  • Fault injection tests five fault types, with 200 instances of each per system, covering invalid JSON, bad paths or types, unauthorized writes, false claims, and cycle halts.The faults test both contamination by malformed updates and whether repeated no-op or oscillatory trajectories are stopped.Source: §4.4 · pp. 6–7

5 Results and Analysis

PatchBoard outperforms the compared systems on matched ALFWorld episodes, while blackboard controls and component analyses examine what contributes to its success and cost profile. Sensitivity and fault-injection results further probe context budgets, schema source, and structural reliability.

5.1 Main ALFWorld Results

  • 533/630 episodes were solved by PatchBoard, versus 194/630 for LangGraph and 388/630 for Flock; PatchBoard also had the lowest tokens per success.The comparison uses matched ALFWorld episodes.Source: §5.1 · p. 7

5.2 Blackboard Controls

  • Both blackboard controls solve fewer episodes than PatchBoard; the plain blackboard costs more per success, while the structured JSON blackboard narrows the cost gap but still trails in success.The results suggest transactional validation and write contracts add gains beyond shared memory.Source: §5.2 · p. 7, Figure 3 · p. 7

5.3 Ablation Study

  • Removing the patch/schema interface and bounded context views produces the largest ablation drops; removing the patch/schema interface more than doubles tokens per success.Removing context slicing produces a comparable success drop, while other component effects are smaller on this evaluation.Source: §5.3 · p. 7, Figure 4 · p. 8

5.4 Sensitivity Analyses

  • The smallest tested context budget has the best success-cost profile; increasing the budget does not improve success and raises normalized cost.The authors interpret this as support for bounded views, since exposing more state can add irrelevant context without improving local decision quality.Source: §5.4 · pp. 7–8, Figure 5 · p. 7
  • Generated task-specific schemas outperform fixed schemas in both success and normalized cost, supporting Architect-built blueprints in this setting.The results also indicate that schema construction quality affects system reliability.Source: §5.4 · pp. 7–8, Figure 6 · p. 8

5.5 Fault Isolation and Termination

  • PatchBoard shows zero observed contamination for invalid JSON, bad paths or types, and unauthorized writes, but schema-valid false claims remain possible; its high cycle-halt rate stops most repeated no-op or oscillatory trajectories.Fault injection covered 200 injections per fault type.Source: §5.5 · p. 8, Table 1 · p. 8

6 Conclusion

PatchBoard replaces open-ended inter-agent dialogue with validated mutations to shared structured state, managed by an Architect and deterministic kernel. On matched ALFWorld episodes it achieves the strongest reported success-cost profile, while analyses identify structural reliability as distinct from semantic correctness.

6 Conclusion

  • Across 630 matched ALFWorld episodes, PatchBoard solves 84.6% of tasks and reduces tokens per successful task to 45.5k, the strongest success-cost profile among compared systems.Blackboard controls, ablations, and sensitivity analyses indicate gains from the patch/schema interface, bounded context views, and transactional validation beyond shared memory alone.Source: §6 · p. 8
  • An Architect defines task schemas, workflow rules, worker contracts, and context budgets; a deterministic kernel validates and transactionally commits mutations and records replayable logs.PatchBoard replaces open-ended dialogue with validated JSON Patch mutations over shared structured state.Source: §6 · p. 8
  • Schema-grounded mutation improves structural validity and auditability, but factual correctness still depends on evidence selection, verifier design, and task-specific semantic checks.The diagnostic HotpotQA study clarifies this boundary.Source: §6 · p. 8

Limitations

PatchBoard provides structural reliability, not a guarantee of semantic correctness, and its strongest evidence comes from ALFWorld. Its usefulness also depends on blueprint quality and task setting, with engineering overhead that matters when auditability and state integrity are not central requirements.

Limitations

  • Schema-valid claims may still be false, incomplete, or unsupported, so structural validation alone does not guarantee semantic correctness.Factual tasks may require stronger evidence retrieval, verifier design, and human review for high-impact decisions.Source: Limitations · p. 9
  • PatchBoard depends on blueprint quality and task setting; sparse schemas may block progress, while permissive schemas weaken role isolation.Its strongest evidence is from ALFWorld, where task state and success conditions are clearly defined; open-ended settings may require richer schemas and more complex validation.Source: Limitations · p. 9
  • Schema construction, context slicing, patch validation, and transaction logging add engineering overhead, making PatchBoard most suitable when auditability, attribution, and state integrity are central.Source: Limitations · p. 9

Ethical considerations

PatchBoard’s kernel conditionally guarantees deterministic handling of proposed updates, while ethical deployment still requires safeguards against false confidence and sensitive-log exposure.

Ethical considerations

  • High-impact state transitions require evidence, provenance, and human review; transaction logs require access controls, retention policies, and redaction.The paper warns that auditable transitions can create false confidence.Source: Ethical considerations · p. 9
  • The evaluation uses public benchmarks, tools, and model APIs, and collects no new human-subject data.Source: Ethical considerations · p. 9

A PatchBoard Kernel Pseudocode

  • Validation, commit, logging, scheduling, and circuit decisions are deterministic conditional on an accepted blueprint and proposed worker patches.Source: A PatchBoard Kernel Pseudocode · p. 11
  • Only candidate patches that pass syntax, authorization, patch-application, post-state-schema, and registered-invariant checks change committed state; rejected patches are logged.Source: A PatchBoard Kernel Pseudocode · p. 11
  • After each proposal, circuit policy handles repeated invalid edits, no-op loops, or short state cycles deterministically.Source: A PatchBoard Kernel Pseudocode · p. 11

B Implementation and Reproducibility Details

The appendix specifies the LangGraph and Flock baseline workflows and identifies the run-configuration details recorded for reproducibility.

B.1 Baseline System Settings

  • LangGraph uses a supervisor with planner, action, and critic subagents, coordinating ALFWorld turns through tool-mediated messages rather than validated patch transactions.Source: §B.1 · p. 12
  • Flock uses a planner-and-executor workflow: the planner publishes a plan artifact, and the executor returns an admissible action.It receives the same observations, action interface, model, decoding configuration, step budget, and timeout as the other systems.Source: §B.1 · p. 12

B.2 Run Configuration

  • Table 3 records the model backend, runtime hardware, benchmark scale, execution limits, and estimation settings used for the reported results.The appendix supplies concrete settings for reproducibility.Source: §B.2 · p. 12

C Diagnostic HotpotQA Results

The HotpotQA diagnostic marks a boundary: PatchBoard’s structural validation does not establish factual support, and the systems show similar answer accuracy.

C Diagnostic HotpotQA Results

D Running Example and Token Cost Breakdown

A representative ALFWorld trajectory shows repeated actor calls as the largest token cost, alongside a small schema-and-context share; its trace illustrates validated updates and logged repair.

D Running Example and Token Cost Breakdown

  • 26.1k tokens, or 64.6% of the 40.3k total, went to actor worker calls in the representative clean-and-place trajectory.Actor calls recur across environment turns and condition on observations, admissible actions, recent state, and the task objective.Source: D Running Example and Token Cost… · p. 13
  • Architect blueprint generation used 2.8k tokens, planner calls 6.2k, verifier calls 2.4k, and repair/retry calls 2.3k.Source: D Running Example and Token Cost… · p. 13
  • Schema and context overhead was 0.6k tokens, or 1.5% of the total, including schema fragments, bounded state views, patch instructions, and state handles.Source: D Running Example and Token Cost… · p. 13
  • The Architect’s blueprint defines the schema, worker contracts, workflow rules, and context budgets; the planner then proposes a patch checked for authorization and schema validity.Source: D Running Example and Token Cost… · p. 13
  • An inadmissible placement attempt is logged, while its repair hint is accepted separately; the retry navigates to the table before placement completes the task.Source: D Running Example and Token Cost… · p. 13

Related papers

Loading 2605.29313v2…