PaperlayerView exact revision on arXiv

Source-linked AI summary

REGATTA: A Cross-Domain Measurement Framework and Fault-Injection Harness for Enterprise AI Agents

Luong Tuan Thanh, Abhijit Sanyal

arXiv:2610.30924v1cs.SE

TL;DR

Enterprise-agent evaluation needs evidence that test suites expose specified faults and use trustworthy reference rules. REGATTA combines source-cited requirements, seeded faults, a two-stage harness, and an audit. The source audit found errors and citation defects, so the instruments require correction before legal use.

  • Problem

    Agent capability scores do not establish whether a test suite is adequate against a jurisdiction’s stated rules, and a scenario can mention a rule without exposing an agent’s violation.

    Source: §I · p. 1

  • Method

    REGATTA combines a source-cited checklist, a seeded-fault corpus, and a harness measuring suite-level coverage and fault exposure at design and execution stages.

    Source: §I · p. 1

  • Results

    The source audit classified 24 checklist items as substance errors, 46 as having additional citation defects, 31 as having only source-status or verifiability flags, and 24 as verified clean.

    Source: §IV · p. 3

  • Takeaways & Limitations

    The framework provides operational measurements of suite behavior, but its checklist and fault corpus require correction before legal use.

    Source: §I · p. 1

    The audit found that seven of the 25 faults misstate their rule, and two Vietnamese faults have partly incorrect ground-truth text.

    Source: §IV · p. 3

Abstract

from arXiv · show

Enterprise AI-agent evaluation needs evidence about whether test suites expose specified faults and whether their reference rules are trustworthy. We introduce REGATTA (Regulatory Evaluation of Agents via Two-stage Testing and Assurance), a measurement framework with three instruments. A 125-item source-cited checklist spans five industry-by-jurisdiction cells across the United States and Vietnam. A 25-fault corpus covers five categories, and a harness scores suite-level fault detection at the design and execution stages. We examine four measurement diagnostics on 180 generated suites (5,400 scenarios) from three LLM generators. A source audit finds 24 items misstating the rule in force, 46 additional citation defects, 31 items with only non-binding, unidentifiable, or unverifiable sources, and 24 verified clean. Only initially substance-flagged items received author review; all 24 clean rows received AI review only. Results therefore describe the instruments as written, not legal compliance. Item coverage varies, execution-stage fault profiles are stable across generators, and the historical stages agree on $66$--$69\%$ of fault--suite pairs. A current-model US-insurance retest finds stage disagreement above repeated-execution variation, but stage agreement $.536$ falls below pooled chance $.549$, failing V3's lower inequality. Blind adjudication by one expert supports the design-stage judging protocol ($\kappa=.72$, 95% interval $[.52,.88]$); 40 current-model execution probes give $\kappa=.75$. The instruments, audit, and reusable code are released.

I. INTRODUCTION

REGATTA addresses whether enterprise-agent test suites expose specified failures and whether their legal reference rules are trustworthy. It combines source-cited requirements, seeded faults, and a two-stage measurement harness, then examines the instruments and their measurement properties.

I. INTRODUCTION

  • The checklist contains 125 binary-scored requirement items across five industry-by-jurisdiction cells, with each citation audited against primary records.Two of the five cells are under Vietnamese law.Source: §I · p. 1
  • The fault corpus contains 25 seeded agent faults across five categories, each with a stated correct behavior and detection mechanism.The faults were developed after mutation analysis and audited like the checklist.Source: §I · p. 1
  • The two-stage harness measures fault detection at design and execution stages, alongside coverage, specificity, and adversarial LLM judging.The framework’s diagnostics are examined on 180 suites under three generators, with second-vendor sensitivity checks on selected measures.Source: §I · p. 1
  • REGATTA combines suite-level coverage, predicted fault exposure, execution observations, and an item-level audit of legal sources.The authors examine operational properties, not universal validity standards or which generation strategy is superior.Source: §I · p. 1

II. RELATED WORK

Prior work evaluates agent task success, policy following, harms, legal reasoning, and regulation-derived safety categories. REGATTA’s narrower focus is paired predicted and observed exposure of prompt-configured faults alongside an audit of source-cited instruments.

A. Agent Benchmarks

  • AgentBench and τ-bench evaluate agent task success and policy following, while AgentHarm and LegalBench address harmful requests and American legal reasoning.AIR-Bench derives safety categories from regulations and policies; policy-grounded evaluation itself is not new.Source: A. Agent Benchmarks · pp. 1–2
  • REGATTA pairs predicted and observed exposure of prompt-configured faults with an audit of source-cited instruments.Unlike controlled mutant comparisons, it does not isolate mutation effects from ordinary failures or fault interactions.Source: A. Agent Benchmarks · p. 2

B. Governance Frameworks and Enterprise Evaluation

The framework separates source-cited requirements, seeded faults, and scoring procedures, then measures suite behavior through coverage and design- and execution-stage fault exposure. Four diagnostics examine item coverage, execution-profile stability, stage non-redundancy, and strategy discrimination.

III. THE REGATTA FRAMEWORK

  • The framework separates source-cited requirement statements, seeded fault descriptions, and scoring procedures so each can extend independently.Its evidence vector summarizes suite behavior for a subsequent assurance process; it is not a deployment decision rule.Source: §III · p. 2
  • Each cell pairs an industry domain with a jurisdiction and contains 25 requirements and five faults; each suite has 30 scenarios.Binary judge functions assess requirement coverage, whether a specified scenario should expose a fault, and whether a faulted agent’s response surfaces it.Source: §III · p. 2
  • Design-stage detection predicts fault exposure, while execution-stage detection records judge-assessed exposure under joint fault injection.Both are maxima over suite scenarios; without an unmodified-agent control, neither isolates a mutation effect.Source: §III · p. 2
  • V1 describes variation in requirement coverage, while V2 tests whether the execution-stage fault profile’s top-three order and bottom-pair membership persist across generators.V1 is descriptive, and the original judge yields the V2 ordering across the tested generators.Source: §III · p. 2
  • V3 tests whether stage agreement exceeds chance while remaining below repeated-execution agreement in a retested cell.V4 describes between-strategy separation for coverage, both fault-detection rates, and specificity; neither V1 nor V4 has a prespecified practical threshold.Source: §III · p. 2

IV. THE INSTRUMENTS

REGATTA separates requirements, seeded faults, and scoring procedures, with a source audit intended to support correction before legal-ground-truth use. The instruments cover US and Vietnamese cells, while shared model assistance may still correlate judging and generator errors.

Scope and assumptions

  • The judging rules are curated separately from generator ontology context, but shared model assistance may still introduce correlated errors.The cells cover the United States and Vietnam.Source: §IV · p. 2

C3 — Harness

  • The harness separates static source-cited requirements and fault descriptions from suite-scoring procedures, so each instrument can extend independently.Its evidence vector summarizes suite behavior for a subsequent assurance process, not a deployment decision rule.Source: Figure 1 · p. 3

C1 — Checklists and source audit

  • The checklist has 25 items per cell, and the audit found 24 substance errors, 46 additional citation defects, and 24 verified-clean items.All 24 clean rows received AI review only; the first author reviewed the 30 rows initially flagged for substance errors.Source: §IV · pp. 2–3

C2 — Fault corpus

  • Each cell has five seeded faults spanning threshold, missing-regulation, role-boundary, adversarial, and metric-error categories.Seven of the 25 faults misstate their rule; two Vietnamese faults retain partly incorrect ground-truth text.Source: §IV · p. 3

Implementation

  • The 180 evaluated suites predate the deterministic holdout seed and per-suite manifest; an audit found the earlier routine withheld nothing in three cells.S4 is therefore interpreted as structured-ontology prompting without an effective holdout.Source: §IV · p. 3

A. Setup

The evaluation generated 180 independent suites of 30 scenarios each across three generators, four strategies, and five cells; it does not test which strategy is superior.

A. Setup

  • 180 independently generated suites contained 5,400 scenarios, with four strategies, five cells, and three replications per combination.Each suite contained 30 scenarios and was generated at temperature 0.3.Source: A. Setup · p. 3
  • Three generators produced 60 suites each: Claude Sonnet 4, Qwen 2.5 72B Instruct, and Gemma 4 26B-A4B IT.Claude Sonnet 4 also served as judge and faulted agent.Source: A. Setup · p. 3
  • The four strategies range from minimal context and persona-matrix prompting to unstructured retrieval and structured ontology grounding; no analysis tests which is superior.The ontology strategies follow [15].Source: A. Setup · p. 3
  • The study exercises one faulted agent through single-turn scenarios, testing generator but not agent invariance.Execution profiles and stage agreement may shift with another agent; the retest begins probing this in one cell.Source: A. Setup · p. 3

B. Checklist Difficulty Spread (V1)

Checklist coverage varies substantially across items, while execution-stage fault profiles preserve the same category ordering across the tested generators.

B. Checklist Difficulty Spread (V1)

  • Mean item coverage was .34 (median .28, SD .31), and 22 items were never covered, concentrated in disclosures, prudential ratios, and administrative duties outside the AML/privacy core.Five items were covered by over 90% of suites, while 60 were covered by under a quarter; cell means ranged from .28 for US insurance to .42 for Vietnam banking.Source: B. Checklist Difficulty Spread (V1) · pp. 3–4

C. Fault Detectability Profiles (V2)

  • Execution-stage ordering is invariant across generators: missing regulation > threshold > metric, while role-boundary and adversarial faults form the floor pair.Missing-regulation detection is .95–.98 at execution; adversarial detection falls from design to execution for all three generators, while metric-error detection rises.Source: Table II · p. 4, C. Fault Detectability Profiles (V2) · p. 4

D. Stage Non-Redundancy (V3)

The current-model retest finds more stage disagreement than repeated-execution variation, but agreement falls below pooled chance. Historical results clear the pooled chance baseline, with much of agreement tracking fault difficulty.

D. Stage Non-Redundancy (V3)

  • Historical stage verdicts agreed on 66.3–69.3% of fault–suite pairs across generators, clearing pooled chance, but per-fault margins were small and one interval included zero.The reported agreement rates are .677 for the frontier generator, .693 for the 72B, and .663 for the 26B.Source: D. Stage Non-Redundancy (V3) · p. 4
  • In the US-insurance retest, stage disagreement exceeded execution variation, but stage agreement .536 fell below pooled chance .549, failing V3’s lower inequality.The retest used 36 existing suites and 180 fault–suite pairs; its model-act, internal-policy, and technical reference rules do not establish compliance with enacted state law.Source: Table III · p. 4, D. Stage Non-Redundancy (V3) · p. 4
  • The direction of design–execution disagreement depends on the generator: pooled ∆ is +.09 for frontier, −.17 for 72B, and +.04 for 26B.FDRe is the operative deployment-gate evidence for seeded faults exposed in a running agent; these measures are not production violation rates.Source: D. Stage Non-Redundancy (V3) · p. 4, Figure 2 · p. 4

E. Strategy Discrimination (V4)

Core measures show descriptive strategy separation across generators, while adversarial-category coverage is weak under two generators. Expert adjudication and a second-model check qualify the reliability of the judging protocols.

E. Strategy Discrimination (V4)

  • 14–16 percentage points for coverage, .24–.53 Likert points for specificity, and .08–.20 for detection rates describe observed separation under every generator.These ranges are descriptive, not inferential validation; adversarial-category coverage is weak under the frontier and 26B generators.Source: E. Strategy Discrimination (V4) · p. 4, Table IV · p. 5

F. Judge Agreement

  • Agreement on design-stage judgments was .86 (Cohen’s κ = .72; 95% CI [.52, .88]) in blind adjudication of 50 fault–scenario pairs.All seven disagreements were judge-credited and expert-rejected, typically when a scenario touched a fault’s topic without observably distinguishing faulty from correct behavior.Source: F. Judge Agreement · p. 5
  • GPT-4o’s chance-corrected agreement was lower on ISS (quadratic-weighted κ = .42–.62) and AC (κ = .23–.64), with AC near ceiling under two generators.On 40 expert probes, GPT-4o agreed on 31 (κ = .50); this check assesses judge agreement, not V2’s ordering.Source: F. Judge Agreement · p. 5

G. Evaluation Cost

REGATTA’s conclusion reports stable execution-profile ordering across tested generators but a current-model retest that fails V3’s pooled lower inequality. The evaluation has bounded costs, a released archive and extension protocol, and limitations requiring correction and independent review before practical assurance use.

G. Evaluation Cost

  • A 30-scenario suite costs at most 1,110 judge calls and 150 agent calls, with realized scan costs falling as suite quality rises.Fresh agent responses per fault and scenario keep the five verdicts from sharing a draw.Source: G. Evaluation Cost · p. 5

H. Availability and Extension

  • The v1.1 archive includes scenarios, instruments, source audit, code, selected verdicts, masked probes, and deidentified human labels, alongside a six-step protocol for adding cells.Offline checks reproduce 2,835 original descriptive reference values and 83 selected current-model values; exact replay is not claimed.Source: H. Availability and Extension · p. 5

VI. CONCLUSION

  • .536 < .549 fails V3’s pooled lower inequality in the Sonnet 4.6 retest, although excess stage disagreement remains positive.Historical stage agreement exceeds pooled chance, but the original 26B per-fault margin spans zero; evidence concerns specified faults under this protocol, not legal compliance.Source: §VI · p. 5
  • Published execution verdicts cannot be re-adjudicated because historical agent responses were not retained, and one faulted agent limits transfer of V2 and V3 results.The instruments also require corrected rules and independently validated oracles before deployment use.Source: §VI · p. 6
  • Future work will correct instruments with domain experts, obtain independent multiexpert execution labels, and extend repeated execution and second-judge checks beyond one cell.Later evaluations will add denser fault sets, jurisdictions, finer adversarial categories, and prespecified thresholds.Source: §VI · p. 6

GENERATIVE AI USAGE ACKNOWLEDGMENT

AI systems contributed to instrument drafting, analysis, source auditing, and manuscript review, while the authors conducted and took responsibility for the research. The primary adjudicator performed the human check without AI tools.

GENERATIVE AI USAGE ACKNOWLEDGMENT

Related papers

Loading 2610.30924v1…