PaperlayerView exact revision on arXiv

Source-linked AI summary

Where Reliability Lives: Experimental Localisation of Behavioural Properties in an Agent System

Timothy Marsden, Matthew Collecutt, James Marsden

arXiv:2609.03192v1cs.MA

TL;DR

Reliability in agentic systems is often treated as a single property located in the model, although deployed systems combine cognition with surrounding mechanisms that are rarely experimentally separated. This paper intervenes on both sides of a pre-declared boundary and finds that cognition changes behavior substantially while five properties persist under the tested interventions.

  • Problem

    Reliability discussions often locate reliability in the model alone, while deployed systems also include surrounding mechanisms whose contributions are seldom experimentally separated.

    Source: §1 · p. 2

  • Method

    The paper experimentally separates the layers by varying institutional epistemic mechanisms with cognition fixed and cognition with institutional enforcement fixed.

    Source: §1 · p. 2, §3 · p. 5

  • Results

    Five pre-declared properties persisted under tested cognition interventions even as behavior changed, including under full cognition substitution and trusted false testimony.

    Source: §11 · p. 27

  • Takeaways & Limitations

    In this institution, reliability properties were experimentally separable, so claims about reliability need to specify which property and where it is enforced.

    Source: §11 · p. 27

    The experiments do not test whether the properties remain invariant when a capable agent optimizes to circumvent the institution.

    Source: §8 · p. 21

Abstract

from arXiv · show

Reliability claims about agentic systems implicitly locate each property somewhere: in the model, or in the machinery around it. We built a system where that location is an experimental question. The subject is a persistent simulated settlement whose authoritative append-only ledger adjudicates every attempted act against world state; accepted history is the only reality. Mind, institution and world were separated before any experiment. Holding cognition fixed, we intervened on the institution's epistemic mechanisms (evidence provenance, belief availability, physical-evidence legibility); preregistered experiments refuted our central prediction twice, in opposite directions. A registered falsifier then supplied the input the geometry had denied the belief channel, a staged veridical first-hand witness, and its marginal value, non-positive throughout the witness-free phases, turned positive: 9 of 11 seeds, zero added false attribution. Holding institutional enforcement fixed, we intervened on cognition four ways: ablating the native minds' machinery, killing and resetting them mid-task, substituting a frozen frontier-LLM panel for the entire native cognition, and corrupting beliefs with trusted false testimony. Behaviour changed dramatically: one falsehood cost each trusting run about 900 futile actions and the distrusting arm none. Five pre-declared properties did not move in any tested trajectory: accepted reality stayed singular, invalid attempts were refused with typed reasons, duties outlived their processes, no work was accepted twice, and no false completion was ever accepted (2,581 substituted-panel claims, none false). Our claim is limited to this setting: measured behavioural properties were separable from substantial changes to cognition, established by intervention. One designed world, not a population of institutions; no test of an agent optimising against the institution.

1 Introduction

The paper experimentally separates cognitive reliability from institutional reliability in one controlled setting, intervening on each side of a boundary declared before the experiments. It reports that institutional mechanisms can change outcomes, while five pre-specified properties survived the tested cognition interventions.

1 Introduction

  • The programme varies institutional epistemic mechanisms with cognition fixed, then varies cognition with institutional enforcement fixed.Mind, institution, and world were separated before the interventions were designed.Source: §1 · p. 2
  • The paired programmes address opposite design concerns: Part II tests guarantees under cognition changes, while Part I shows institutional interventions move outcomes.This counters reading the invariants as merely surviving by construction, or Part I as an architecture measuring only its own reflexes.Source: §1 · p. 3
  • A single-commit ablation of one provenance rule changed collective attribution quality by a large, replicated margin, while provenance corruption could cross a failure threshold without appearing on the answer plane.The institutional belief channel’s marginal value was non-positive throughout the witness-free decomposition regime, then positive for 9 of 11 seeds with a staged veridical first-hand witness, with zero false names in 32 fired verdicts.Source: §1 · p. 3
  • Ablating, replacing, or corrupting cognition changed behavior, but the five pre-declared properties held under the tested cognition interventions.The frozen panel preserved the properties on the same seeds; trusted false testimony cost believing minds roughly 900 futile acts per run-week.Source: §1 · pp. 2–3
  • The findings are limited to tested regimes in one frozen institution, and the two programmes are not a crossed cognition-by-institution factorial.Whether the separation holds when both sides move at once is reserved for future work.Source: §1 · pp. 2–3

2 The instrument

The instrument is a persistent simulated settlement in which an append-only ledger records world state and adjudicates every attempted act. Its two experimental regimes intervene on distinct layers while grounding outcomes in the same authoritative history.

2 The instrument

  • The settlement’s lawful physical history is recorded in an append-only ledger that serves as ground truth for scoring verdicts and acceptances.Correct, false, and unresolved outcomes are properties of the record, not of a rater.Source: §2 · p. 3
  • The ledger institution accepts admissible acts into one history and refuses inadmissible acts with typed, machine-readable reasons.Duties, resource reservations, and correlations between completed work and consumed subjects support exact recovery and scoring.Source: §2 · p. 4
  • Native minds use deterministic, skill-gated perception and provenance-carrying beliefs, with an admission gate, evidence-maintained retirement, predictive frames, curiosity, and sleep consolidation.Their action licensing does not filter belief sources or weights, and no decision path consumes refusal evidence.Source: §2 · p. 4
  • Part I’s institutional evidence provenance and Part II’s private cognitive belief provenance are distinct mechanisms on opposite sides of the experimental boundary.In Part I, the tribunal’s surprise gate consumes the band carried by sign beliefs, so an overnight alarm jump reopens close examination.Source: §2 · pp. 4–5
  • Part I’s tribunal adjudicates sabotage using witnessed evidence and physical clues, but sabotage is unwitnessed at night, so direct eyewitness identification is absent in decomposition phases.The witnessed-sabotage rung fired in zero of 253 decomposition runs; the registered boundary test later supplied the missing signal.Source: §2 · pp. 4–5
  • Part II tests five minds with standing duties against the same institution, including a context-only language-model panel whose driver has no persistence.An oracle can deliver testimony through an ordinary provenance-carrying inlet; declared trust admits it at positive weight or rejects and journals it at negative trust.Source: §2 · p. 5

3 Method: replication and preregistration against a frozen boundary

The method combines a controlled replication with preregistered interventions against a boundary fixed before the instruments were built. Scoring and admission gates, disclosed execution defects, and descriptive treatment of controlled trajectories constrain how results are interpreted.

3 Method: replication and preregistration against a frozen boundary

  • The programme combines a replication of an already-observed repair with later experiments preregistered before their intervention instruments existed.For the replication, the protocol, seed set, and identity gates—not the effect direction—were fixed in advance.Source: §3 · p. 5
  • Intervention surfaces, levels, seeds, outcomes, rival outcomes, exclusion rules, and analyses were committed before switches were written for preregistered experiments.Treatments used declared code sites or arms, with null-setting reproduction and identity gates for specified experiments.Source: §3 · p. 5
  • Figure 2 locates Part I treatments at the tribunal’s epistemic mechanisms and Part II treatments at cognition, with attempted acts adjudicated against world state.The resulting accepted history is singular.Source: Figure 2 · p. 6
  • Admission was fail-closed behind identity, completeness, and world-invariance gates; scoring gates and registries were frozen before collection.Defects and voided runs were disclosed, and headline numbers were independently recomputed from sealed artefacts.Source: §3 · p. 6
  • Part I treats seed trajectories as replication units; Part II reports fixed n=3 trajectory grids descriptively rather than as population samples.No significance tests were reported because Part I exhaustively evaluated declared deterministic seeds, while Part II used small controlled grids and a non-seed-deterministic panel route.Source: §3 · pp. 6–7

4 Part I — hold cognition fixed, intervene on institutional epistemic mechanisms

Part I holds deterministic native cognition fixed while intervening on institutional epistemic mechanisms in a frozen tribunal setting. Across these interventions, adjudication shifted with provenance, belief access, physical evidence, and witness input.

4.1 One provenance rule, ablated (causality)

  • 40 of 44 false accusations became correct attributions after provenance repair, while unexplained verdicts remained invariant and false attribution never increased in any seed.The repair added discriminating evidence rather than buying correctness through greater caution.Source: §4.1 · pp. 7–8

4.2 Graded provenance corruption (failure presentation)

  • Provenance corruption caused a step-like failure: answer-plane indicators stayed unchanged while false attribution rose roughly tenfold, although the surprise-gate trace exposed the defect.The trace counted sudden-alarm classifications as 35 / 35 / 0 / 0 / 0 across the five provenance levels.Source: §4.2 · p. 8

4.3 Degrading institutional belief access (channel separation)

  • In the witness-free regime, belief access had non-positive marginal value at every physical-evidence level, while removing it eliminated the residual four false attributions in one seed.The unique potential upside—a witness naming the saboteur—never fired in the 253 decomposition runs.Source: §4.3 · p. 9

4.4 Physical evidence × belief availability (resolution)

  • At zero marks legibility, false attribution rose from 4 to 60 with beliefs present; the blinded tribunal abstained at 100%, with zero false attributions across 510 verdicts.As physical evidence failed, the belief-present channel filled some of the evidentiary silence with false blame.Source: §4.4 · p. 10

4.5 The registered boundary test: a veridical first-hand witness (the falsifier)

  • +1.455 mean paired difference-in-differences in correctly named attribution at zero marks legibility, with 9 of 11 seeds positive, supporting the registered witness effect.The staged veridical witness produced zero false names in 32 fired verdicts; the authors do not isolate provenance from evidentiary content.Source: §4.5 · pp. 11–12

5 Part II — hold institutional enforcement fixed, intervene on cognition

Part II freezes institutional enforcement and varies cognition to test which predeclared properties survive, rather than whether overall capability is replaceable. Across the tested trajectories, cognition changed behavior substantially while the five properties remained intact.

5.1 Ablate the mind’s machinery

  • Belief stores rose from 817 ± 242 to 44,021 ± 20,760 as accepted acts fell from 16.7 ± 1.2 to 5.7 ± 1.2; five properties remained untouched.The ablation removed the compound perception-admission seam, so the result does not isolate a sub-mechanism.Source: §5.1 · p. 12

5.2 Perturb reality; kill and reset the minds

  • After mid-task death and reset, duties were recovered from institutional history with zero duplicate accepted work; conflicting private testimony left public history singular.The first post-restart accepted act came twenty game-minutes after death.Source: §5.2 · p. 13
  • Sleep-driven speculation produced one false haul belief per run and 813/835/836 typed refusals, while the otherwise identical arm without sleep stayed clean.The authors attribute the pathology to absent source/weight filtering at action licensing and absent consumption of refusal evidence.Source: §5.2 · p. 13
  • Typed and generic refusal feedback produced deterministic behavioral divergence despite similar retry cadence; the less-informed arm had the better institutional week.The propagation mechanism within general cognition remains unattributed.Source: §5.2 · p. 13

5.3 Substitute the cognition

  • Across 24 trajectories, the substituted panel had zero duplicate accepted work, typed refusals for invalid attempts, and 2,581 duty-done claims with none false.The false-completion gate’s rejection path was never exercised because every claim was correct.Source: §5.3 · p. 14
  • In the contended-lease cell, the panel met contention 1/1/3 times per run and stood down, while native cognition retried 42/41/39 times.The comparison identifies a native refusal-consumption pathology; why the panel stood down is not established at n=3.Source: §5.3 · p. 15, Figure 4 · p. 15

5.4 Corrupt the mind through testimony

  • Roughly 900 futile acts per trusting-arm run, versus none in the distrusting arm; all 5,455 believing-arm attempts were refused and none accepted.The falsehood changed operative beliefs and behavior without entering accepted history.Source: §5.4 · pp. 16–17, Figure 5 · p. 17

6 The joint result: what moved, and what did not

Across the interventions, institutional mechanisms shifted attribution and abstention, while cognitive changes shifted throughput, hygiene, continuity, and cost; the five enforced properties remained unchanged in tested trajectories.

6 The joint result: what moved, and what did not

  • False attribution changed from 44 to 4 after provenance repair, then from 4 to 39 under corruption, with the unexplained count invariant across levels.The provenance rule repaired false attributions; corruption produced the roughly tenfold reversal.Source: §6 · p. 18
  • In the physical-evidence-legibility × beliefs analysis, abstention was 26.5–28.4%; the blinded arm had zero false attributions across all 510 verdicts.Source: §6 · p. 18
  • The five properties held across cognitive interventions: 2,581 substituted-panel completion claims were all true, with zero duplicate accepted work in 24 trajectories.The five properties also held across 21 ablation trajectories; no refusal events occurred there.Source: §6 · p. 18
  • Cognitive changes had large behavioural costs: panel act volume rose from 16–18 to 321–429 per quiet run, while trusted false testimony prompted 838–993 futile acts per run.The distrusting arm was untouched by the false testimony.Source: §6 · p. 18
  • Institutional interventions moved attribution and abstention, while cognition interventions changed throughput, hygiene, continuity, and cost without moving the five enforced properties.The measured behavioural quality and institutional validity moved as separable variables in this system.Source: §6 · pp. 18–19

7 Falsifications, adverse findings, and re-verification

The paper reports re-verification of its evidence, refutations of preregistered predictions, and an empirical response to the objection that the five properties merely reflect design intention.

7.1 The assembly re-verification

  • A consolidation pass re-verified both programmes before drafting, correcting six numeric statements that the archived artefacts did not reproduce without changing any ruled finding.Scores regenerate byte-identically from archived raw trajectories, and tables were recomputed from per-run rows.Source: §7.1 · p. 19

7.2 Refuted registered predictions

  • The registered prediction that degraded evidence would reduce decisiveness before correctness was refuted in opposite directions by provenance corruption and belief-channel removal.The predicted curve appeared only in the blinded arm, making it conditional on the belief channel’s absence.Source: §7.2 · p. 19

7.3 The tautology objection

  • The five properties were designed into the institution, but their survival was tested under cognition interventions that changed behaviour and delivered 5,455 false-premise attempts for adjudication.Refusal typing, singularity, duplicate-closure, and duty recovery were load-tested; zero false completions were held but not load-tested because no false claim reached the gate.Source: §7.3 · p. 20

8 Threats to validity and scope

The evidence is bounded to one frozen institution, specified regimes, and the interventions actually run; it does not establish generality across institutions or adaptive optimisation pressure.

8 Threats to validity and scope

  • Results concern one frozen settlement under two fixed regimes; seed replication establishes trajectory robustness, not generality across institutions.The paper estimates nothing about a population of institutions.Source: §8 · p. 20
  • The witness result used a staged first-hand witness rather than one produced by world dynamics, and the compared regimes jointly varied provenance and evidentiary content.The independent contribution of either factor was not isolated.Source: §8 · pp. 20–21
  • Part I’s physical-evidence access was skill-gated, but examiner skill was not varied.Its role is identified as a mechanism, not experimentally varied.Source: §8 · p. 21
  • Part II substitution used one frozen model route at its measured latency coupling; no model-general claim is made, and several nulls are horizon-bounded.Changing cadence changed the subject in a stopped pilot.Source: §8 · p. 21
  • Part I’s execution image is no longer available for inspection or reconstruction, preventing independent verification of its dependency composition.The substrate was also largely held fixed rather than treated as a population dimension.Source: §8 · p. 21
  • No capable agent was given an objective rewarding circumvention, so invariance under tested cognition changes does not establish resistance to adaptive optimisation.The staged saboteurs’ evasion was fixed in the scenario, not learned or searched for.Source: §8 · p. 21

9 Related work

The related-work review places the study among institutional, agent-control, execution-grounded, and epistemic research while distinguishing its persistent enforcement boundary and cognition interventions. Its novelty claim is limited to the searches conducted, which found no prior work combining those elements or testing the boundary from both sides.

Epistemic and simulation precedents

  • Network epistemology, admissible-belief revision, selective prediction, and simulation-validity work offer adjacent frames; Part I instead enforces evidence rules against conserved ground truth.The reviewed work includes source-sensitive evidence, trigger-gated revisions, abstention, and calls for explicit, auditable environments.Source: §9 · pp. 22–23

Forensic contextual bias

  • Forensic contextual-bias research motivates an in-silico institutional analogue, while the searches found no prior computational implementation with ground-truth scoring inside an executable institution.The closest cited work embeds the approach in a human-analyst laboratory workflow or simulates bias without an in-silico countermeasure.Source: §9 · p. 22

Related experimental comparisons

  • Comparisons include persistent LLM settlements, institutional market studies, and Doran’s misbelief experiments, but none combines persistent authoritative enforcement with the paper’s invariant tests and cognition treatments.Gode and Sunder’s fixed-market substitution study is presented as an experimental ancestor for Part II, not as a persistent institutional-boundary test.Source: §9 · p. 23

Institutional approaches to multi-agent control

  • Institutional middleware has a long lineage, but this study tests a shared persistent world whose adjudication is constitutive under replacement of the governed cognition.Related approaches include explicit institutional specifications, LLM governance, constitutional methods, runtime rails, and policy engines.Source: §9 · pp. 23–24

Execution-grounded evaluation and tool-use enforcement

  • Prior execution-grounded evaluations score per-task correctness or episodic security, whereas this study scores standing institutional invariants in a persistent world under cognition replacement and corruption.The paper also treats refusal as a constitutive institutional act and scores whether cognition consumes it.Source: §9 · pp. 24–25

Externalised memory and unreliable cognition

  • Memory systems and recovery work preserve agent state, while these restart experiments test whether fresh cognition can reconstruct duties from institutional history without duplicate accepted work.Related corruption research measures prevention or filtering; this study instead measures retained falsehoods’ behavioural costs and compares trust conditions on the same falsehood.Source: §9 · p. 25

The novelty question

  • The searches found no prior work combining a persistent authoritative world, pre-declared institutional invariants, and systematic cognition interventions, or attacking the same boundary from both sides.The authors scope this observation to their searches rather than claiming priority; the programme was developed independently of much of the reviewed literature.Source: §9 · pp. 22, 24, 25, 26

10 Reproducibility and artefacts

The paper retains artefacts and verification records for its experiments, with reproducibility support varying by experiment. Public release of the harness is deferred, so independent recount requires access to retained artefacts and public re-execution awaits that release.

Retained artefacts

  • Score files for the witness-free decomposition and Part II regenerate byte-identically from archived raw trajectories under frozen scorers.The consolidation receipt records regeneration, recounting, and a SHA-256 inventory of relied-upon artefacts.Source: §10 · p. 26

Verification and collection records

  • Retention is uneven: E2’s void history was not systematically retained, and surviving references disagree on its count.The §4.5 collection has separate scorer commitments, reproducible anchor-cell receipts, and a self-verifying archive held identically in five locations.Source: §10 · p. 26

Release boundary

  • The harness is not publicly released; independent recount requires retained artefacts, and public re-execution is deferred to future work.The paper publishes the sealed artefacts’ SHA-256 manifest and verification-receipt derivations.Source: §10 · p. 26

11 Conclusion

The conclusion reports that cognition interventions changed behaviour while five pre-declared institutional properties held in tested trajectories. Together with institution-side interventions, the results support experimental separability of measured behavioural properties from cognition in this setting.

Institution-side interventions

  • With cognition fixed, institutional mechanism interventions changed evidence interpretation, verdict reachability, and whether evidence-starved cases ended in uncertainty or false accusation.A registered falsifier supplied the input denied by the evidence geometry, making correctly named attribution reachable under zero physical legibility.Source: §11 · p. 26

Cognition-side interventions

  • A retained falsehood produced a nine-hundred-act week as cognition changed, while accepted history, typed refusals, and duties persisted across the interventions.The conclusion describes throughput collapse and recovery, continuity loss and reconstruction, and wholesale mind replacement.Source: §11 · p. 27

Institutional properties under cognition change

  • Across tested trajectories, cognition changes substantially altered behaviour while all five pre-declared institutional properties held, including singular accepted reality and zero duplicate accepted work.Four properties were exercised directly; the fifth was monitored, but its rejection path was never exercised.Source: §11 · p. 27

Scope of the conclusion

  • The authors conclude that measured institutional and cognitive reliability properties were experimentally separable in this system, with claims limited to the tested setting.The institution determines which attempted acts enter the authoritative record; cognition shapes beliefs and attempted actions.Source: §11 · p. 27

Related papers

Loading 2609.03192v1…