Maat: Independent Deterministic Contract-Based Governance for Multi-Agent LLM Workflows

1 Introduction

• Multi-agent decomposition supports parallelism and specialization but adds interfaces where errors can enter and propagate; prior work documents 14 failure modes [1].The cited taxonomy groups failures into system-design, inter-agent-misalignment, and task-verification categories.Source: §1 · p. 2
• A simple model gives end-to-end reliability R ≈ p^n for n independent stages, while the paper cautions that real errors may be correlated or amplified downstream.The model is presented as intuition for how small local error rates can matter as workflow depth grows.Source: §1 · p. 2
• Maat asks whether deterministic checks can prevent contract-detectable handoff defects without relying on another language model to judge them.Learned or LLM-based evaluators have probabilistic verdicts and may share failure modes with the systems they assess.Source: §1 · p. 2
• Maat formalizes deterministic contract validation for handoffs and reports a six-domain evaluation that separates end-to-end correctness from protective early termination.The paper also identifies configuration validity as a reliability requirement and treats cross-model effectiveness as a hypothesis, not an established result.Source: §1 · p. 2

2 Problem Setting: Why Handoffs Matter

• Wrong values, omitted fields, or identity inconsistencies can become authoritative downstream state, turning discount, patient-identity, or insurance errors into consequential outputs.The paper motivates checking seams between agents because downstream agents reuse upstream state, which can amplify an early defect rather than merely accumulate failures.Source: §2 · pp. 2–3
• Determinism does not ensure correctness, and Maat detects only defects observable as encoded handoff or contract violations.A payload may be structurally valid yet violate a business rule; a misconfigured deterministic rule can also produce a reproducibly wrong verdict.Source: §2 · p. 3

Failure taxonomies and attribution

• Maat checks whether a transition satisfies an explicit contract before the receiving agent consumes it, unlike work focused on locating failure origins [1], [6], [7], [10].The cited work includes empirical taxonomies, automated error generation, prefill-stage signals, and hypothesis verification.Source: §3 · p. 3

Trajectory anomalies and guardrails

• Trajectory-anomaly and learned-guardrail approaches broaden detectable behavior beyond explicit contracts, while Maat’s fixed configuration produces reproducible verdicts [8], [9], [11].The comparison is scoped to a fixed engine version, configuration, anchor, and handoff.Source: §3 · p. 3

Propagation-aware safety

• Propagation-aware methods model error dependencies or claim evolution, whereas Maat blocks explicit structural and contract violations at observable interfaces [3], [4], [12], [13].Maat does not model semantic belief dynamics.Source: §3 · p. 3

Architecture and shared state

• Architecture and shared-state studies support a systems view in which reliability depends on external coordination state as well as individual model capability [5], [15].The cited work reports reduced trace-level error amplification with centralized verification and emphasizes durable shared state, lineage, and reproducibility in another setting.Source: §3 · p. 3

4.1 Public system model and disclosure boundary

• Maat compares selected handoffs with a versioned anchor describing authorized workflow state, and Figure 1 places that check before downstream consumption without a language model in validation.The paper specifies the validator’s observable contract rather than disclosing its internal evaluation order or rule-compilation procedure.Source: §4.1 · pp. 3–4, Figure 1 · p. 4

4.2 Seven gates and requirement primitives

• The seven gates cover plan, handoff, depth, role, health, circuit-breaker, and entity-consistency checks; requirement primitives encode domain rules.Structural validity differs from contract validity: a well-formed handoff may still violate a requirement, and Maat does not claim to detect every hallucination.Source: §4.2 · p. 4

4.3 Framework integration artifact: CrewAI

• The CrewAI artifact demonstrates integration at a framework boundary, with configured anchors enabling business-rule checks that shape-only checks cannot provide.Figure 2 distinguishes the public gate layer from requirement primitives; the artifact is an integration demonstration, not another benchmark arm.Source: §4.3 · p. 5, Figure 2 · p. 5

4.4 Configuration as part of the system

• Two benchmark episodes exposed configuration failures, so the evaluated system includes the engine, anchor, and configuration state, which should be tested and versioned.The paper recommends testing declared rules on representative live-shaped payloads and treating semantically irrelevant reordering as unchanged business state.Source: §4.4 · p. 5

5.1 Research questions

• The four research questions address contract correctness, model-call cost when defective chains stop, intercepted versus uncovered failures, and sensitivity to anchor and rule configuration.Source: §5.1 · p. 6

5.2 Three-arm design

• Off, warn, and intervene modes respectively leave workflows unguarded, log findings while continuing, or block or redirect the chain; corrected scoring credits halts only for verified defects and specific findings.Failure profiles corrupt authoritative data or context to study propagation, rather than asking agents to make an error.Source: §5.2 · p. 6

5.3 Domain workflows

• Six workflows comprise 522 trials, with agent counts following task decomposition rather than a fixed synthetic team size.Source: §5.3 · p. 6, Table 1 · p. 7

5.4 What each benchmark is intended to stress

• The six benchmarks stress different handoff seams, including identity and allergy state, parallel branches, scope amendments, cross-client contamination, and contract-dense adjudication.Hospital triage is a controlled safety-relevant simulation, not a clinical evaluation; the benchmarks are not claims of production readiness.Source: §5.4 · p. 7

5.5 Result versioning

• The public evidence uses a later repository snapshot that revised the insurance row and documented 11 false-positive findings; v2 added a scorer audit and review of every governed-arm halt.The paper treats configuration state as part of the measured system and says it must be versioned with results.Source: §5.5 · p. 7
• 5.92 (off) vs. 5.83 (intervene), or −1.4%, on the full insurance grid; the paired subset’s +19.1% excludes both deny profiles.The paired subset should not be read as a whole-benchmark effect.Source: Table 2 · p. 8, §5.5 · p. 8

6.2 Representative contract catches

• Examples include cross-marketplace price drift, tax misclassification, payout inflation, claimant-identity drift, data-residency violations, and revenue mis-reporting.These checks detect contract or state-consistency violations, not open-domain factual errors lacking a corresponding anchor rule.Source: §6.2 · pp. 8–9

6.1 End-to-end correctness and cost

• Five of six workflows improved on the paired subset and software development was flat; counting false-alarm halts as failed work puts the governed arm below baseline in four workflows.The B2B result includes genuine but un-injected arithmetic errors, and the hospital result rests on only nine pairs.Source: §6.1 · p. 8, Figure 4 · p. 9
• Cost savings occur when an early blocking defect skips downstream calls, not as an intrinsic reduction in validation cost.Enterprise discovery was approximately cost-neutral; false-alarm halts can save cost by stopping correct work.Source: §6.1 · p. 8, Figure 5 · p. 9

6.3 Configuration correction and false positives

• The halt audit found 35 false alarms among 94 governed-arm halts, caused by validator defects including provenance handling, empty-answer checks, and scenario-value mismatches.Determinism does not ensure correctness; anchor construction and scenario-conditioned expected values must themselves be tested.Source: §6.3 · pp. 10–11

6.4 Behavioral gates

• Targeted behavioral-gate tests reported 0% false trips on a healthy pipeline and 100% trips under engineered chronic failure, but do not validate every gate equally across domain workflows.Survival monitoring is calibration-dependent rather than governed by a universal threshold.Source: §6.4 · p. 11

7 Operational Scope: Where Maat Is and Is Not a Good Fit

• Maat fits workflows with stable, inspectable authority and costly or regulated consequences downstream of a violating handoff.Examples include insurance policies, financial authorizations, data-residency requirements, clinical protocols, and service-level obligations.Source: §7 · p. 11
• Maat is less compelling when downstream agents provide independent redundancy, no stable contract exists, or failures are purely semantic.Software-development agents often re-derive requirements, while a false explanation may pass if it violates no encoded state.Source: §7 · p. 11

7.1 Deployment posture

• Warn mode records findings while workflows continue, making it a suitable starting posture for observing blocks during anchor calibration.Intervene mode halts or routes execution to recovery; only intervention can prevent later model calls and create measured downstream cost savings.Source: §7.1 · p. 11
• Routing blocked handoffs to human review can support oversight requirements, but the validator alone does not constitute EU AI Act compliance.Human review is presented as an alternative to continuing the chain or silently repairing the payload.Source: §7.1 · pp. 11–12

8.1 What the results support

• When a defect is a deterministic predicate over the handoff, tracked state, and correctly configured contract, inline validation can prevent it reaching later agents.The supported cases include stable quantities and identities, explicit eligibility rules, limits, and jurisdictional constraints.Source: §8.1 · p. 12
• Contract gating can stop some trajectories before downstream effects, complementing attribution work that explains failed trajectories after the fact [6] [7] [10].This aligns with architecture-level evidence that validation bottlenecks can reduce error propagation [5]; the functions are not substitutes.Source: §8.1 · p. 12

8.2 What the results do not support

• The evaluation does not establish general hallucination detection or model-independent utility, lacks per-trial distributions for inferential claims, and uses controlled simulations rather than deployments.The experiments use a single base model and do not reproduce prospective hospital, insurer, or marketplace settings.Source: §8.2 · p. 12
• The false-positive revision shows that an audit-friendly deterministic layer can still be wrong when its specification is wrong.The authors argue specification errors remain inspectable, testable, and versionable, but are not cost-free.Source: §8.2 · p. 12

8.3 Model-dependence hypothesis

• The authors propose, but do not demonstrate, that fixed-contract verdicts are model-invariant while defect manifestation rates depend on the model.They propose paired cross-model tests holding workflow, seeds, anchor, injections, and validator version fixed while changing the base model and measuring manifestation, conditional recall, false positives, and cost.Source: §8.3 · p. 12

• Cross-model generalization remains open because the domain results use a single base model.Source: §9 · p. 13
• The structured synthetic benchmarks do not reproduce the full organizational, legal, data-quality, and latency conditions of production deployments.Source: §9 · p. 13
• Corrected paired estimates rest on small samples for B2B (17) and hospital (9), and the B2B anchor supports meaningful checks for only one of five customer seeds.Source: §9 · p. 13
• Validator accuracy depends on the anchor and rule bindings, with documented insurance false positives demonstrating configuration risk.Source: §9 · p. 13
• Public evidence is strongest for handoff completeness, requirement primitives, entity consistency, validation depth, survival calibration, and circuit breaking, not every declared gate.Source: §9 · p. 13
• Without complete per-trial distributions, the published aggregate means and profile outcomes do not support p-values, confidence intervals, or significance claims.Source: §9 · p. 13
• Reported dollar costs are benchmark-specific because API prices and token use vary; skipped downstream inference after early termination is the portable mechanism.Source: §9 · p. 13

• The public repository provides benchmark designs, per-benchmark summaries, gate analyses, and consolidated tables.Source: §10 · p. 13
• A CrewAI integration artifact demonstrates a local structural analyzer, handoff connector, verdict replay, and minimal business-rule demo.Source: §10 · p. 13
• The artifact does not expose the proprietary conductor or complete configuration methodology, so it cannot reproduce every end-to-end conductor decision from source alone.The disclosure distinguishes reproducibility of the open analyzer and demo from reproducibility of the full implementation.Source: §10 · p. 13

11 Conclusion

• Corrected results show improvements where halts are attributable to real defects in five of six workflows on the paired subset.A hand review found that 37% of version 1 halts were false alarms caused by validator defects; the originally reported gains on every workflow do not survive attribution-aware scoring.Source: §11 · pp. 13–14
• Anchor construction, path resolution, expected-value semantics, and entity normalization must be validated and versioned with the result.The benchmark history shows that determinism does not remove specification risk; Maat is an external deterministic boundary only for behavior expressible as contracts and state invariants.Source: §11 · p. 14

A Mapping to Selected MAST Failure Modes

• Table 5 presents a conceptual mapping of selected MAST failure modes to Maat mechanisms, not a claim that all modes are fully solved.The paper omits frequencies because the latest MAST revision differs from percentages in earlier internal drafts.Source: A Mapping to Selected MAST Failure Modes · p. 15, Table 5 · p. 15

B Benchmark Interpretation Checklist

• A non-firing validator can reflect an untested defect, pre-emption, model self-correction, misconfiguration, or a correctly silent rule on a valid handoff.This taxonomy prevents “no alert” from being conflated with “validated correct.”Source: B Benchmark Interpretation Checklist · pp. 15–16

C Caught Scenarios and Handoff Locations

• Trace records identify the validated producer output, downstream consumer, handoff, and finding mechanism; an earlier verdict may pre-empt the intended check.A non-firing intended rule may also reflect self-correction or misconfiguration, and the representative scenarios are trace-level provenance rather than separate benchmark results.Source: C Caught Scenarios and Handoff Locations · p. 16, Table 6 · p. 16