PaperlayerView exact revision on arXiv

Source-linked AI summary

When Agent Governance Helps

Michael Ray Johnson, Linda Naimi

arXiv:2609.05531v1cs.CLcs.AIcs.CYcs.MA

TL;DR

The paper addresses the lack of an integrative specification for governed autotelic agent organizations and limited evidence on whether governance improves task behavior. It synthesizes GAMPO from 321 sources and probes its prompt-layer instantiation on a healthcare benchmark; results show capacity-, model-, and domain-specific effects, with case-grounded advisory gains in some tasks but not care-management.

  • Problem

    There is no integrative framework for designing, operating, and evaluating governed autotelic agent organizations, and evidence is lacking on whether governance scaffolding improves task behavior.

    Source: §1 · p. 4, §2 · p. 5

  • Method

    The paper synthesizes 321 sources into GAMPO and probes a prompt-layer instantiation on a long-horizon healthcare benchmark.

    Source: §5.6 · p. 20, §8 · p. 32

  • Results

    Governance effects were capacity-, model-, and domain-specific; the answer-blind advisory raised prior-authorization from 40% to 84% under best-of-five self-consistency, while care-management met a content-quality wall.

    Source: Summary of Contributions · p. 3

  • Takeaways & Limitations

    The findings support case-grounded specifications over a uniform procedure for some frontier tasks, but do not show a uniform benefit across healthcare domains.

    Source: Summary of Contributions · p. 3

    Results are exploratory, with single-trial designs and per-cell samples of 5–25; the authors call for causal testing and replication beyond healthcare.

    Source: Summary of Contributions · p. 3, §7 · p. 32

Abstract

from arXiv · show

No specification says how a governed autotelic AI agent organization, where agents pursue self-generated goals inside guardrails, should be designed and evaluated. We answer in two parts. First, we synthesize the Governed Autotelic Multi-Agent Product Organization (GAMPO) framework from a document-based qualitative evidence synthesis of 321 sources, integrating agency, agile, platform, and governance theory into a runnable specification. Second, we probe a prompt-layer instantiation of GAMPO on CHI-Bench, a long-horizon healthcare benchmark, across open and frontier models. The result is a boundary condition: governance benefit is gated by a model's spare capacity and is domain- and model-specific. On capacity-constrained open models the full procedure yields no reliable benefit, whereas a single "verify your writes" sentence doubles task success (pass@1 2/20 to 4/20). At the frontier the same scaffold lifts prior-authorization 24% to 40% but nets zero on another model, a gap traced to a stable recommendation-override disposition. A second result refines the first: replacing the generic procedure with an answer-blind, per-task definition-of-done, keyed only to the case's own policy and published standards, never the hidden key, raises prior-authorization to 84% under best-of-five self-consistency (68% single-attempt, confirmed by a held-out board) and utilization-management to 44%, while care-management meets a subjective content-quality wall. The contribution is a named, auditable framework and capability-gated evidence that governance should be sized to spare capacity, and that at the frontier a case-grounded specification beats a uniform procedure. Findings are exploratory: partial instantiation, small per-cell samples (n = 5-25), and single trials.

Summary of Contributions

The paper presents GAMPO, a framework synthesized from 321 sources, and evidence that governance effects depend on model capacity, task domain, and governance form. Its results favor case-grounded specifications for some frontier tasks, while cautioning that the findings are exploratory.

Summary of Contributions

  • GAMPO is a named framework synthesized from 321 appraised sources, and its governance procedure helps only when a model has spare reasoning capacity.On weaker models, the full procedure was neutral or occasionally harmful; the one-sentence verification habit helped consistently across runs.Source: Summary of Contributions · p. 3
  • The authors recommend sizing governance to spare capacity and treating these small-sample associational findings as design heuristics, not causal guarantees.For capacity-constrained deployments, they favor cheap, action-grounded checks over long procedural scaffolds.Source: Summary of Contributions · p. 3
  • Prior-authorization pass@1 rose from 24% to 40% on Opus 4.8 and from 12% to 32% on Gemini 3.1 Pro and Kimi 2.7-code under the generic procedure.The procedure was neutral-to-harmful outside prior-authorization and near zero on some frontier models.Source: Summary of Contributions · p. 3
  • The full procedure’s pooled fractional reward was 0.393, versus 0.431 unscaffolded; a one-sentence verification prompt yielded 0.590 and doubled pass@1 from 2/20 to 4/20.The gains are from the verification prompt, not the full framework.Source: Summary of Contributions · p. 3
  • 40% to 84% pass@1 under best-of-five self-consistency for prior-authorization; utilization-management reaches 44%, while care-management stalls at a subjective-judge content-quality wall.The advisory’s strict single-attempt prior-authorization result was 68%, confirmed by a held-out board.Source: Summary of Contributions · p. 3
  • Results are exploratory: the advisory evaluations used single-draw or best-of-five designs, with per-cell samples of 5–25.Pass@1 confidence intervals are wide and overlap, so rankings are indicative.Source: Summary of Contributions · pp. 3–4

1. Introduction

The introduction frames the need to design and evaluate governed AI-agent organizations that pursue bounded, self-generated goals. It presents GAMPO and a benchmark probe as complementary responses to gaps in specification and evidence.

1. Introduction

  • An autotelic agent generates, scores, and pursues its own bounded goals within governance constraints, extending ideas of intrinsic motivation.The organization-scale setting is a team of language-model agents planning, writing, testing, and shipping code with little human supervision.Source: §1 · p. 4
  • The paper identifies two gaps: no integrative framework for designing governed autotelic agent organizations, and insufficient evidence on whether governance scaffolding improves agent behavior.It argues that a specification without a hard-task probe and a benchmark result without a specification are each incomplete.Source: §1 · p. 4
  • The paper combines a named framework, capability-gated benchmark evidence, a decision rule and autonomy ladder, and a QES-plus-prompt-layer probe.The roadmap places the synthesis and framework before the benchmark results and discussion.Source: §1 · p. 4

2. Background and Literature Synthesis

The literature synthesis appraises a broad source base and combines prior findings about workflow-dependent agent performance and hybrid deployment with governance theory. It identifies the need for an auditable, runnable specification and an empirical test of its effects.

2. Background and Literature Synthesis

  • The qualitative evidence synthesis screened 1,847 records and included 321 sources, appraising them with instruments matched to source type and recording decisions in a public evidence ledger.Searches covered academic databases, preprints, practitioner sources, standards bodies, and vendor documentation through Q1 2026.Source: §2 · p. 5
  • Prior evidence reports workflow-dependent AI coding gains, including experienced developers working about 19% slower on mature codebases with AI assistance (Becker et al., 2025), and roughly 70% of effective deployments keeping humans involved from the start.The synthesis presents these findings as reasons to avoid context-free productivity claims.Source: §2 · p. 5
  • The identified gap is an auditable, tool-agnostic specification for agent-organization artifacts, non-delegable actions, and self-evaluation, alongside an empirical test of its effect on task success.The gap concerns the population, measure, and model conditions for governance scaffolding.Source: §2 · p. 5
  • GAMPO integrates six traditions, including autotelic agency, knowledge taxonomy, Disciplined Agile, platform theory, sociotechnical governance, and a process-coverage map.Its conceptual model predicts an outcome vector from five design constructs, moderated by way-of-working fit and economic routing.Source: §2 · p. 5

3. Research Questions and Propositions

The research questions ask how to design and evaluate a governed autotelic agent organization and what synthesized evidence supports its architecture, process, and evaluation. The propositions guide the framework, while the benchmark probes the operational kernel of one proposition.

3. Research Questions and Propositions

  • The primary question asks how to design, govern, and evaluate an autotelic multi-agent organization that starts lean, pursues bounded goals, and produces auditable evidence of usefulness, safety, trust, and business value.Source: §3 · p. 5
  • Secondary questions address governance and architecture, economic and process efficiency, and methods for producing auditable, longitudinal, tool-agnostic evaluation evidence.Source: §3 · p. 7
  • The synthesis yields ten propositions; the benchmark probes P1’s claim that repository-based governance scaffolding is associated with reduced scope drift, hallucinated implementation, and ungoverned autonomy.Other load-bearing propositions concern leaner role topologies, nine non-delegable boundaries, and evidence-ledger artifacts that survive model and tool substitution.Source: §3 · p. 7

4. Methods

Methods pair a document-based qualitative evidence synthesis and conceptual design-science artifact with a controlled paired benchmark probe of GAMPO as prompt-layer scaffolding.

Design

  • The study paired interpretive synthesis under a pragmatist paradigm with a controlled model-evaluation study.The synthesis produced the GAMPO specification; the empirical probe instantiated it as a prompt-layer scaffold.Source: §4 · p. 7

Phase 1: evidence synthesis

  • 321 sources were synthesized across twelve literature streams, with a 10% subsample blindly re-appraised and appraisal coverage spanning AMSTAR 2, JBI, CASP, and AACODS.The corpus included 74.8% peer-reviewed sources.Source: §4 · p. 7

Phase 2: conceptual artifact

  • GAMPO was specified as a runnable repository artifact and conceptually evaluated against its evidence base, without a testbed or users.Design choices supported by fewer than two independent sources were flagged as provisional.Source: §4 · p. 7

Empirical probe: benchmark setting

  • CHI-Bench evaluates agents driving clinical cases to terminal status across provider prior-authorization, payer utilization-management, and care-management tasks using a high-fidelity simulator exposing 20 apps through 87 tools.A 1,290-document operations handbook guided agents; each task used weighted checks of actual post-run system state, yielding fractional reward in [0,1] and binary pass@1.Source: §4 · pp. 7, 9

Measures, models, and analysis

  • The paired benchmark compared an unscaffolded baseline with GAMPO and targeted prompt variants, including a single-sentence write-verification prompt and an answer-blind advisory pipeline.The advisory definition of done used the case’s visible policy and chart, never the hidden key; strict single-attempt pass@1 and best-of-five self-consistency were reported separately.Source: §4 · p. 9

Instantiation scope

  • The probe instruments GAMPO’s prompt-layer kernel rather than the full framework, leaving several components, including human-approval gates and five-layer evaluation, untested.Of 26 components, 8 were instrumented and 4 partially instrumented.Source: §4 · p. 9, Figure 5 · p. 11

5. Results

The results examine how GAMPO and a case-grounded, answer-blind definition of done perform across prior-authorization, utilization-management, and care-management, including limits related to content judgments and simulator uncertainty. They also report constrained-model verification results and ablation evidence on the contribution of generic governance versus case-specific facts.

5.2 Finding 1: At the frontier governance helps where the model has traction, but the benefit is gated by spare model capacity

  • 37.3% overall (28/75 tasks) was the filed Opus 4.8 full-GAMPO score; paired success rose from 24% to 40% on prior-authorization and 8% to 16% on utilization-management.The six-task care-management subset rose from 2/6 to 4/6, but the benchmark’s 95% intervals were wide and overlapping at n = 25 per domain, so domain rankings are indicative rather than definitive.Source: §5.2 · pp. 12, 14, Table 1 · p. 14

5.1 The framework (RQ1–RQ3)

  • Governance gains were gated by spare capacity: the full framework gave gpt-oss-120b no reliable pooled benefit, and constrained models showed neutral, harmful, or floor-level outcomes.Below the capability floor effects were approximately zero; eligibility also required tool-calling and loop convergence, while active rather than total parameters tracked capacity regime. The synthesis-derived framework specifies machine-readable governance artifacts, claim routing, non-delegation boundaries, and a five-layer evaluation; these are design propositions, not measured effects.Source: §5.1 · p. 12, §5.2 · pp. 14–15

5.3 Finding 2: A one-sentence verify prompt dominates the full framework

  • On a 35B model across PA and UM, verify-base raised pooled fractional reward from 0.431 to 0.590 and doubled pass@1 from 2/20 to 4/20, outperforming the full framework (pooled reward 0.393).The pass@1 doubling was a descriptive signal, not a powered effect (paired sign test p approximately 0.75, n = 20); gampo-lite recovered reward from 0.393 to 0.478.Source: §5.3 · pp. 15, 17

5.4 Finding 3: No single model leads every domain

  • Under the identical procedure, different frontier models led prior-authorization at 40% and utilization-management at 36%; no single model topped all domains, and domain difficulty rankings were model-dependent.The prior-authorization benefit replicated for Gemini 3.1 Pro and Kimi 2.7-code, with pass@1 rising from 12% to 32% for each.Source: §5.4 · pp. 17–18

5.5 Finding 4: Residual gaps are judgment ceilings, partly repairable by prompt architecture

  • A judgment ceiling, not missing instructions, drove many residual failures: one model consistently followed a tool’s submit recommendation despite disqualifying evidence, while another consistently overrode it.Rewriting mandate language to treat deny, do-not-submit, gather-more, and escalate as equally valid outcomes changed one resistant task from 0.000 to 1.000; merely permitting override did not.Source: §5.5 · pp. 18, 20

5.6 Finding 5: At the frontier the effective governance is an answer-blind, per-task definition-of-done, not the generic procedure, which roughly doubles the prior-authorization result

  • 84% prior-authorization and 44% utilization-management pass@1 under best-of-five answer-blind self-consistency, versus 40% and 16% with generic GAMPO; held-out single-run results were 68% and 39.2%.The task-level definition of done uses only visible case policy and published standards, never hidden benchmark information; care-management reached 40% single-draw pass@1, below the generic procedure’s 56%, with subjective judging and simulator variation limiting interpretation. The answer-blind advisory also raised Gemini 3.6 Flash prior-authorization from 12% to 32% in single-attempt runs, while Opus 5 scored 54.7% overall across the three domains.Source: §5.6 · pp. 20, 22, 23, Figure 11 · p. 22
  • Prior-authorization rose from 24% baseline to 40% with the procedure and 68% with the single-attempt advisory; its best-of-five ceiling was 84%.The 68% single-attempt result was confirmed on a held-out board; 84% is a self-consistency ceiling, not a single-attempt score.Source: Table 5 · p. 24, §5.6 · pp. 24–25
  • Utilization-management reached 39.2% single-attempt and 44% under best-of-five selection, versus 8% baseline and 16% with the procedure.The 39.2% figure is the confirmed mean binary pass over five draws per task, not a single-draw score.Source: Table 5 · p. 24, §5.6 · pp. 24–25
  • Care-management advisory scored 40%, matching its 40% baseline but below the procedure’s 56% on a different simulator, so that comparison is exploratory.The benchmark’s pinned simulator was retired, the advisory used its successor, and member-consent variance confounds single-draw movement.Source: §5.6 · pp. 24–25, §5.7 · pp. 25–26

5.7 Robustness and sensitivity

  • Single-trial results were variable: a re-roll flipped 2 of 5 borderline tasks, and frontier absolute scores differed 40% versus 32% despite matching +16-point paired lifts.Care-management’s initial local floor was a dead-simulator artifact; corrected small-sample lifts failed to replicate on two newer models, and harness calibration limits absolute cross-model comparisons.Source: §5.7 · pp. 25–26
  • On held-out prior-authorization tasks, the frozen advisory scored 68% versus 48% without per-task facts and 24% baseline, supporting generalization of generic governance.Per-task facts added five recovered tasks while costing two through redistribution; that directional increment was not statistically resolved at n = 25.Source: §5.7 · p. 26
  • The advisory’s prior-authorization result reproduced at 72% on Opus 5, while utilization-management reached 36% and procedure-only care-management reached 56%.The Opus 5 sign-off commit lever was rejected under the zero-regression rule: it recovered one task but regressed another, leaving net zero.Source: §5.6 · p. 24, §5.7 · p. 26

6. Discussion

The discussion interprets governance as a capacity-dependent intervention, not a uniform good, and translates the findings into implementation and organizational implications. The benchmark probes only a prompt-layer kernel, while the broader framework and runtime extend beyond what was tested.

Interpretation

  • The spare-capacity account is an explanatory hypothesis, not an established causal mechanism: governance instructions may compete with task work when reasoning capacity is scarce.The discussion links this account to constrained-model performance and to judgment dispositions that scaffolding alone did not change.Source: §6 · p. 26

Implications for executives

  • Governance should be sized to spare capacity, with minimal action-grounded verification for constrained models and case-grounded specifications for frontier models where they have traction.The authors characterize these as design heuristics based on small samples, not causal guarantees.Source: §6 · p. 27

From specification to runnable implementation

  • The runtime fully realizes its reasoning and governance core, but its healthcare integration surfaces are simulations and live CI/CD and production observability remain stubbed.The empirical results still bound only the prompt-layer kernel, not an end-to-end runtime evaluation.Source: §6 · p. 27

Implications across people, process, technology, and economics

  • The companion readiness catalog reports that GAMPO fully instantiates seven of thirteen patterns, simulates five, and leaves one out of scope.The integration and continuous-monitoring surfaces that remain simulated leave a minimum viable architecture for live regulated deployment absent.Source: §6 · p. 28

From specification to runnable implementation

  • The Opus 4.8 prior-authorization progression reaches 84% pass@1 through measured lifts attributed to accuracy-oriented patterns, while readiness patterns do not move pass@1.Chart-fidelity grounding contributed +20 board points (+5 net paired after redistribution); the final sixteen-point increment used best-of-five self-consistency, not a single-attempt metric.Source: §6 · p. 28

Implications for academics

  • The study offers QES plus an instrumented prompt-layer probe as a way to produce an auditable framework and test its claims, while identifying spare capacity and recommendation override as hypotheses for further study.The people and economic implications in the companion readiness lens are reasoned from results rather than directly measured.Source: §6 · pp. 27–28

Boundaries

  • The empirical findings cover a prompt-layer kernel on one healthcare benchmark and should not be generalized beyond long-horizon, policy-rich tool-use tasks.Broader propositions remain synthesis-derived.Source: §6 · p. 28

7. Limitations and Future Research

The authors bound the benchmark evidence to a partial prompt-layer instantiation and exploratory trials, then outline work to evaluate the full runtime and strengthen the evidence. The mapped runtime has a complete reasoning and governance core but simulated integration surfaces.

Limitations

  • The probe instruments 8 of 26 framework components, with 4 more partial, so its results evaluate the prompt-layer kernel rather than the full framework.The ledger, coercive policy-as-code, human-approval gates, and five-layer evaluation are among the components not exercised.Source: §7 · p. 28
  • The runtime audit maps a fully instantiated reasoning and governance core alongside simulated healthcare integration, while live EHR and payer connectivity, CI/CD, and production observability are deferred.The pattern mapping reports seven of thirteen patterns fully instantiated, five simulated, and one out of scope.Source: Figure 12 · p. 29, Figure 13 · p. 30
  • Benchmark findings are exploratory: per-cell samples range from n = 5–25, there is one trial per task, and open-model and frontier arms used different harnesses.The verify-base advantage is descriptively large but not significant under a conservative paired test; absolute single-run levels vary across draws.Source: §7 · p. 32

Future work

  • Future work calls for end-to-end runtime evaluation under load, live FHIR and payer connections, and larger, multi-trial paired tests on fresh tasks and additional domains.The answer-blind advisory’s held-out evaluation is to be extended beyond the public 25 tasks and into the other two domains.Source: §7 · p. 32
  • Further studies should manipulate capacity directly, test non-healthcare benchmarks and other model families, and check harness-invariance with an alternative agentic framework.The authors also call for an approve-side regression to bound over-denial risk before deployment.Source: §7 · p. 32

8. Conclusion

The paper synthesizes 321 appraised sources into GAMPO, a runnable governance specification, and probes its prompt-layer kernel on a long-horizon healthcare benchmark. The results make governance benefit capacity-, model-, and domain-specific, with a case-grounded specification improving prior-authorization more than a uniform procedure.

Conclusion

  • GAMPO is an auditable framework synthesized from 321 appraised sources, with a runnable specification and a runtime whose reasoning and governance core is implemented.The benchmark probe exercises only the prompt-layer kernel; integration surfaces in the runtime are answer-blind simulations.Source: §8 · p. 32
  • Governance benefit is model- and domain-specific and gated by spare capacity: on capacity-constrained models, one “verify your writes” sentence wins (pooled fractional 0.431 to 0.590; pass@1 doubled), while the full framework is at best neutral.Source: §8 · p. 33
  • Replacing the uniform procedure with an answer-blind, case-grounded definition-of-done raises prior-authorization to 84% under best-of-five self-consistency, while care-management meets a subjective content-quality wall.The prior-authorization result is 68% single-attempt on a held-out board; the paper’s takeaway is to size governance to spare capacity and use case-grounded specifications at the frontier.Source: §8 · p. 33

Disclosures

The paper identifies its conceptual framework and evidence base as dissertation-derived, while the reported benchmark study is post-proposal work. It reports no human subjects and declares no competing interests.

Source

  • The conceptual framework and evidence base derive from the author’s doctoral dissertation proposal, but the empirical benchmark study is post-proposal work.Source: Disclosures · p. 33

Ethics

  • The study involved no human subjects and is described as non-human-subjects research under 45 CFR 46.Source: Disclosures · p. 33

Data and code availability

  • The GAMPO agent submission and evaluation artifacts are available on the CHI-Bench leaderboard, and the author declares no competing interests.CHI-Bench is identified as third-party work (Chen et al., 2026a).Source: Disclosures · p. 33
Loading 2609.05531v1…