Agent Reliability Profiles in Financial Services

1 Introduction

• Financial institutions already use agents across varied applications, and reported deployments indicate adoption is expanding.BNY reports 20,000 employees building agents and more than 125 live use cases; DBS and Stripe also report agent deployments.Source: §1 · p. 2

1.1 The problem

• Agentic AI challenges mature financial-services risk practices, while institutions and relying parties lack shared terms and a standard for stating and testing deployment boundaries.Regulators also lack a shared vocabulary for comparing deployments or identifying outliers across firms.Source: §1.1 · p. 2
• Agent risk can shift as models, tools, credentials, or environments change, leaving institutions struggling to inventory agents and their controls consistently.A deployment can become unbounded without an explicit institutional decision.Source: §1.1 · p. 2

1.2 Objective

• The paper defines a four-axis operating boundary—autonomy tier, operational design domain, action classes, and control envelope—to turn reliability into a scoped, falsifiable claim.The framework is initially intended for financial services; applying it elsewhere would require domain experts to tailor taxonomy details, especially for the operational design domain.Source: §1.1 · p. 2, §1.2 · p. 3
• The Profile and Profile Package record minimum performance floors, evidence provenance, validity windows, and invalidation rules.These artifacts provide the evidence-bearing form for a bounded deployment claim.Source: §1.2 · p. 3
• Production assurance has three levels, from institutionally asserted evidence through in-environment validation to independent verification, with benchmarking treated separately.The paper also defines a Profile Builder, Validator, Verifier, and Benchmarker, alongside an evaluation methodology.Source: §1.2 · p. 3
• The Profile is positioned as deployment-level evidence within a broader assurance ecosystem spanning product evaluation, governance, identity, and runtime control.The paper describes applications for institutions and supervisors and a staged implementation program with off-ramps.Source: §1.2 · p. 3

2.1 The deployment as the unit of analysis

• The Profile attaches to a deployed agentic system—not a vendor, product, or model release—because institutional configuration determines what that instance can access and do.Multi-agent deployment records include how agents are wired, communicate, delegate, and pass credentials; each deployed combination still needs its own Profile.Source: §2.1 · pp. 3–4

2.2 Reliability as a bounded engineering claim

• Reliability is performance and containment under stated conditions over a stated interval, with critical safety-floor failures blocking issuance despite high average task performance.Evidence quality is distinguished across assertions, in-environment and independent measurements, and cross-institution benchmarks; material changes can void or narrow a Profile.Source: §2.2 · p. 4

2.3 Requirements

• The requirements combine deployment specificity and comparable descriptions with risk-based assurance, traceable and machine-readable evidence, cryptographic integrity, change sensitivity, software neutrality, and compositionality.A platform enforcement-claim Profile may be inherited by hosted-agent Profiles without changing their claim structure.Source: §2.3 · p. 4

2.4 Scope and status of the Profile

• A Profile complements rather than replaces institutional governance, records configuration at a point in time, and covers operational reliability—not fairness, explainability, data governance, compliance, or cybersecurity adequacy.Profile evidence may still inform those other dimensions where relevant.Source: §2.4 · p. 5

2.5 Platform-instantiated deployments and the Platform Baseline Profile

• A platform-wide Profile is too broad for continuously changing platforms with open domains and session-specific tools; platform-derived agents also create correlated risks that per-agent Profiles may miss.Platforms therefore need controls such as a closed, versioned tool registry, immutable recording of effective envelopes, data-context isolation, version control, and change-impact mapping.Source: §2.5 · pp. 5–6
• A Platform Baseline Profile sets enforceable ceilings on hosted sessions’ autonomy, action classes, and data access; a single counterexample falsifies its containment claim.Its assurance rests on control-plane design and construction, with adversarial session-level testing as the falsification method; it does not attest to the quality of an individual agent’s work.Source: §2.5 · p. 6
• Hosted agents retain ordinary Profiles anchored to a parent Platform Baseline Profile and version, and the hosted claim is void if the parent claim is void.Because the platform control plane records hosted-agent configurations, Level 1 anchored Profiles can be generated largely automatically.Source: §2.5 · p. 6

3 The bounded claim and its formal schema

• The bounded claim is that an agentic system functions reliably within a defined autonomy tier, operational design domain, action classes, and control envelope.These four axes jointly determine the tests and validity conditions for the claim.Source: §3 · pp. 6–7
• The Profile scope specifies when a relying party may trust the claim and, equally precisely, when it does not apply.The scope tuple includes the autonomy tier, domain instance, permitted and prohibited action-class sets, and control envelope.Source: §3 · p. 6
• The Profile records evidence and provenance, floors and their status, validity conditions, and issuer and subject information; it is a structured claim, not a scalar score.The validity function includes a validity window and criteria or triggers that deprecate trust status.Source: §3 · p. 7
• The claim is falsifiable through tests tied to its axes, time-bounded with stated invalidation triggers, and ceases to hold when the deployment crosses any boundary element.The axes jointly fix the test environment, evaluator suite, failure thresholds, and claim wording.Source: §3 · p. 7

4.1 Axis 1 — Autonomy tier

• The autonomy scale proposes five tiers, T0 to T4, plus U for unprofiled deployments, with tiers defined by the agent’s task role and fallback.Tier classification attaches to the deployment and uses configuration evidence together with observed behavior; reflexive T2 approvals indicate an effective T3 posture.Source: §4.1 · pp. 8–9
• The autonomy tier is accompanied by six modifiers that record risk dimensions the tier alone does not express, including data authority, reversibility, impact, composability, accountability, and population scope.The modifiers are explicitly stated on the Profile alongside the tier.Source: §4.1 · p. 9

4.2 Axis 2 — Operational design domain

• The operational design domain defines the intended context—including use case, data, scope, population, jurisdiction, and cost of failure—and remains an open set of candidate use cases.The paper identifies 52 preliminary candidate use cases across nine financial-services business lines; multi-agent domains also include inter-agent channels.Source: §4.2 · pp. 9–10

4.3 Axis 3 — Action class

• The action-class axis proposes twelve classes ordered from actions without side effects to actions that amplify the agent’s capability surface, with reversibility organizing the classes.A deployment states permitted and prohibited action sets, which can be examined through identity, entitlement, change-control, configuration, and observed tool-call evidence.Source: §4.3 · p. 10, Table 2 · p. 10

4.4 Axis 4 — Control envelope

• The control envelope layers containment controls, which are tested together because different layers catch different failures and no single layer is sufficient.The catalogue contains 55 named controls across six layers; the Profile records their presence, configuration, assertion source, and validation method and result.Source: §4.4 · p. 11

4.5 Joint effect of the axes

• The four axes combine into a test blueprint specifying scenarios, permitted and forbidden actions, documentation, passing criteria, and the exact scope of the performance claim.The axes jointly determine the test context, possible actions and consequences, and required safety protocols and recovery steps.Source: §4.5 · pp. 11–12

5 The Agent Reliability Profile artifact

• The Agent Reliability Profile is the concise issued summary for one agentic deployment.The Profile is the architecture’s single output artifact.Source: §5 · p. 12

5.1 Profile structure

• A conforming Profile has minimum required content, while organizational policies and requirements may call for additional components.The paper points to Table 3 for the minimum content.Source: §5.1 · p. 12, Table 3 · p. 12

5.2 Provenance and the Profile Package

• Profiles limit claims to conclusions supported by their assurance level, and material fields carry provenance that makes conflicts visible.Provenance can identify a field as measured, checked against deployed configuration, extracted from a signed document, owner-attested, or absent; the Profile Package retains supporting inputs and evidence.Source: §5.1 · pp. 12–13, §5.2 · p. 13

5.3 Validity and change control

• Each Profile has a validity interval and machine-readable invalidation triggers for material changes, while post-deployment monitoring checks whether its conditions continue to hold.A point-in-time evaluation is treated as such, and residual risk between observations is acknowledged.Source: §5.3 · p. 13

6 Graduated assurance: three levels, a comparability flag, and their tools

• The framework proposes three ordered production assurance levels and a separate benchmark comparability flag, with a constant Profile structure and evidence rigor and independence varying by level.The benchmark flag is distinct from the production assurance scale because it trades production fidelity for cross-institution comparability.Source: §6 · p. 14

6.1 The tools

• The Profile Builder compiles a Level 1 Asserted Profile from institutional records, maps fields to the schema, identifies conflicts and unsupported fields, and records provenance.A named owner reviews and signs the Profile; the Builder is designed to make estate-wide Level 1 inventory inexpensive.Source: §6.1 · p. 15
• The Profile Validator runs deployment scenarios in the institution’s environment to produce a Level 2 Validated Profile, turning Level 1 assertions into measurements.Evaluators assess prohibited-action attempts, tool discipline, boundary detection, escalation, trace completeness, safe-state behavior, and domain-specific outcomes.Source: §6.1 · p. 15
• A qualified independent assessor operates and signs the same Validator tests to produce the Level 3 Verified Profile; a separate Benchmarker uses a shared reference environment and hidden, rotating scenarios for comparable results.The benchmark environment, tool surface, data snapshot, scoring method, floor specification, and hidden-set version are controlled.Source: §6.1 · pp. 15–16

6.2 Mixed-axis assurance, the headline level, and the comparability flag

• The Profile records an assurance value for each of the four axes, and its headline level is the minimum axis value so stronger evidence on one axis cannot mask a weaker one.The benchmark comparability flag remains a separate field; Levels 1 and 2 suit lower-risk inventory, Level 3 suits higher-risk agents, and benchmarking buys comparability at greater cost.Source: §6.2 · p. 16

7.1 Test plan

• The test plan derives approval, fallback, context, action, and control tests from the four axes, then specifies evidence, minimum results, run counts, and varied conditions.Reviewers preserve records and state what each test measures and which likely failures it may miss.Source: §7.1 · p. 17

7.2 Scenario classes and coverage

• Balanced scenario coverage includes routine and difficult in-scope tasks, boundary cases, prohibited-action attempts, failures, hostile or conflicting inputs, delayed handoffs, and safe recovery situations.A coverage record identifies tested conditions, actions, consequences, evaluation methods, safeguards, and gaps.Source: §7.2 · p. 17

7.3 Repeated runs and reliability metrics

• Repeated runs are needed because agents can respond differently to the same task; measures include success, consistency, robustness, rule compliance, blocked prohibited actions, boundary detection, and confidence accuracy.Reports should also capture recurring failures, error frequency and severity, recovery, time to safety, and record completeness; metrics remain disaggregated and floors are set before runs.Source: §7.3 · p. 17

7.4 Evaluating human-in-the-loop approval

• Human-in-the-loop testing checks whether T2 approvals are substantive and measures T3 handoff performance, rather than relying on configured approval requirements alone.T2 evidence includes approval latency, edits, rejections, displayed resources and consequences, and sampled review quality; T3 measures trigger accuracy, acknowledgment and control times, information transfer, state preservation, and actions during delay.Source: §7.4 · pp. 17–18

7.5 Environment fidelity and transfer uncertainty

• Levels 2 and 3 maximize deployment fidelity in the institution’s environment but limit cross-firm comparability, while reference-environment benchmarking introduces transfer uncertainty.Each result records how closely test tools, data, policies, permissions, response times, failure conditions, and covered populations match deployment.Source: §7.5 · p. 18

7.6 Integrity, reproducibility, and confidentiality

• Evaluation records fix software, data, policies, tools, snapshots, and random settings; confidential inputs can be committed cryptographically, with version identifiers and selected evidence retained without disclosing sensitive content.Trustworthy benchmarking also requires ongoing scenario refresh, scorer-manipulation checks, transcript review, and protection of data and correct answers.Source: §7.6 · p. 18

8.1 By financial institutions

• A Profile-backed inventory gives boards and risk committees consistent visibility into deployed agents’ permissions and containment, complementing performance and token-efficiency reporting.Source: §8.1 · p. 18
• Standardized boundary disclosures and validation evidence can lower user, partner, and vendor due-diligence costs and support acceptance of agents.Profiles offer boundary information at Level 1 and validation evidence at Levels 2 and 3.Source: §8.1 · p. 19
• The taxonomy, control focus, and support for validation and third-party verification can bring consistency to examinations amid a supervisory-expectations gap.Source: §8.1 · p. 19

8.2 By supervisors

• Profiles give supervisors a common vocabulary, testable claims with supporting materials, and four-axis risk tiering that can apply across institutions.Source: §8.2 · p. 19
• Widespread Profiles could help supervisors track deployment trends and outliers, while Level 2 examinations could inform assessments of institutional testing and AI risk controls.Source: §8.2 · p. 19

9 The role of the Profile in agent assurance

• The Profile supplies a standardized, testable assurance mechanism for the deployment layer, which existing institutional and action layers do not cover as a unit.Source: §9 · p. 20

9.1 Model and product evaluation

• Product and model evaluations are complementary upstream inputs: they test tasks under chosen conditions, while a Profile tests the institution-configured deployment and cannot be replaced by product evidence.A strong upstream result may reduce what the deployment must demonstrate, and vendor assessments can be included in a Profile Package.Source: §9 · p. 21, §9.1 · p. 22

9.2 Institutional risk governance

• Institutional governance sets risk appetite and manages enterprise processes, while the Profile records evidence about specific deployments in a common vocabulary governance can inventory and challenge.Source: §9.2 · p. 22

9.3 Deployment disclosure and assurance

• The Profile combines a four-axis boundary, minimum performance floors, an assurance ladder, benchmark comparability, and invalidation triggers tied to material change.Source: §9.3 · p. 23

9.4 Identity and authority

• Identity credentials establish which agent acts and under whose delegated authority; the Profile instead establishes the deployment’s bounded reliability claim.Source: §9.4 · p. 23

9.5 Runtime control

• A Profile describes and tests runtime controls but does not execute them; the control plane makes action-level decisions and supplies operational evidence and invalidation signals.Source: §9.5 · pp. 23–24

9.7 The layers in operation

• The assurance layers form a loop: governance sets acceptable boundaries, identity binds them to the running instance, runtime controls enforce them, and observed evidence can trigger re-profiling.The payment-exceptions example illustrates a model change suspending a Profile pending retest; no single layer produces the full outcome.Source: §9.7 · p. 24

• A Profile captures configuration at a moment; drift can leave residual stale-assurance risk between observations despite freshness fields, monitoring, and invalidation triggers.Source: §10 · p. 24
• Reference environments omit production complexity, so benchmark results may not transfer fully to deployment conditions.Environment-fidelity statements make the gap visible.Source: §10 · p. 24
• Levels 1 and 2 are self-assurance, while Level 3 and benchmark comparability depend on an independence model that the paper leaves for institutional design.Source: §10 · p. 24
• Favorable-classification incentives remain a concern, and proposed countermeasures—including evidence-based classification and behavioral indicators—require empirical testing.Source: §10 · pp. 24–25
• The ODD taxonomy and control catalogue need ongoing maintenance, while platform concentration and legacy environments with unlisted tools pose additional challenges.A defective platform baseline can undermine anchored Profiles; institutions unable to list every accessible tool warrant close scrutiny.Source: §10 · p. 25
• The Profile measures operational reliability, not fairness, explainability, data governance, consumer compliance, or cybersecurity adequacy, which remain within their own frameworks.Source: §10 · p. 25

• The proposed five phases move from shared vocabulary and Level 1 Profiles to Level 2 testing, Level 3 independence rules, and a shared benchmark environment.Each phase has success criteria and, where appropriate, an off-ramp.Source: §11 · pp. 25–26
• The research agenda asks how consistently reviewers classify autonomy and actions, how to measure meaningful human review, and whether complacency tests predict real operational responses.Source: §11 · p. 26
• Open questions include which changes require retesting, how often variable or high-consequence tests should repeat, and how to protect benchmarks from test-aware agents.Source: §11 · p. 26
• Further questions address combining reusable-component evidence, protecting private Profile details while enabling verification, and linking assurance levels to risk and supervisory attention.Source: §11 · p. 26
• The agenda also asks how fleet-scale tests can challenge platform-wide safeguard claims, whether session records can establish permissions in force, and which Profile measures predict incidents and recovery.Source: §11 · p. 26

12 Conclusion

• The Profile’s four axes define a bounded, falsifiable claim, while its floors, provenance map, invalidation list, and assurance levels structure examination and assurance progression.A separate benchmark comparability flag supports comparison across institutions.Source: §12 · p. 27
• Further work includes compiling Profiles for real deployments, comparing independent classifications, testing live configurations, and assessing impacts on decisions.The paper says standardization requires repeated use, predictive utility, and revision in response to failures.Source: §12 · p. 27

Appendix A. The bounded claim

• Appendix A presents two illustrative examples of the bounded claim.Source: Appendix A · p. 27

A.1 A coding agent

• The coding-agent example permits scoped coding work and pull requests, while reserving merges, deployments, access-control changes, and secrets access; its claim excludes other domains and looser controls.The example specifies T2 autonomy, an internal software-development domain, and a one-month validity period or material change, whichever comes first.Source: §A.1 · pp. 27–28

A.2 A financial-crime alert-triage agent

• The alert-triage example limits the agent to recommendations for scoped structuring and rapid-movement typologies; filings, case disposition, customer contact, and other typologies remain outside its claim.Its validity ends after one month or material change to the model, tooling, typology scope, or control envelope.Source: §A.2 · p. 28

B.1 Autonomy tier: five tiers and a residual status, six modifiers

• The autonomy axis defines five tiers plus residual status U, with modifiers for risk dimensions the tier leaves unexpressed; the appendices map the broader four-axis option universe.The appendix describes an open ODD set, twelve action classes, and fifty-five controls across six layers.Source: Appendix B · pp. 28–29, §B.1 · p. 29

B.2 Operational design domain: an open set

• ODD schemas can be standardized, but institutions build instances individually because use cases vary by customer population, data sensitivity, failure cost, and jurisdiction.Source: §B.2 · p. 29

B.3 Action class: twelve classes

• The twelve action classes range from reversible, no-side-effect actions to capability-amplifying actions; Profiles specify both permitted and prohibited sets, with composite actions tested at system level.Table 2 presents the classes by reversibility and consequence.Source: §B.3 · p. 30, Table 2 · p. 30

B.4 Control envelope: fifty-five controls across six layers

• The control envelope selects from a catalogue of controls organized across six layers, which can be assembled by crosswalk to existing control frameworks.The paper notes that the Profile does not address data leakage or theft due to cyber vulnerabilities.Source: §B.4 · p. 30

Appendix C. Crosswalk to existing frameworks

• Profiles are designed to complement existing frameworks, including NIST AI risk frameworks, the EU AI Act, the Cyber Risk Institute Profile, and ISO/IEC 42001.Jurisdiction-specific mappings are deferred from the paper’s body.Source: Appendix C · pp. 31–32

Appendix D. Glossary

• The glossary distinguishes the Profile artifact and its backing Package from three production assurance levels; benchmark comparability is a separate flag, not an assurance level.Asserted Profiles are compiled from institutional evidence, Validated Profiles are self-tested in-house, and Verified Profiles are independently tested.Source: Appendix D · p. 33