PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Bounded Autonomy in Regulated Agentic AI: A Diagnostic Harness with Constitutional Rewards, Escalation Labels, and Runtime Governance

Dipankar Sarkar

arXiv:2609.37501v1cs.CLcs.AIcs.LG

TL;DR

Regulated, high-stakes agentic settings make wrong answers costly, while common steering techniques do not transfer well. RegLLM makes escalation measurable and exposes substantial variation across nominally identical smoke-pilot runs.

  • Problem

    In regulated, high-stakes domains, wrong agent answers can cause fines, harm, or missed obligations, and dominant steering techniques do not transfer well.

    Source: §1 · p. 1

  • Method

    RegLLM combines should-escalate labels, a shared domain constitution, AI-judge scores, and runtime supervision to evaluate, train, and govern regulated agents.

    Source: §1 · p. 2, §7 · p. 6

  • Results

    Two nominally identical GPU pilots on the same evaluation split produced materially different metrics, and the same DPO adapter’s apparent effect on escalation recall flipped between runs.

    Source: §1 · p. 2, §7 · p. 6

  • Takeaways & Limitations

    At n=8 and 30 DPO steps on 10–20 preferences, noise can swamp adapter effects, so single-run smoke results should not be over-interpreted.

    Source: §6 · p. 6

    Should-escalate labels were template-generated rather than expert-annotated, and the work lacks an inter-rater agreement protocol.

    Source: Limitations · p. 7

Abstract

from arXiv · show

We propose RegLLM, a diagnostic harness for bounded autonomy in regulated agentic workflows. It instruments six trustworthiness signals: citation validity, source grounding, schema compliance, escalation correctness, constitutional alignment, and unsafe-action rate. Signals are distinguished by their source of supervision: programmatic verifiers, task-level escalation labels, or AI-judge scores. A deterministic runtime supervisor blocks ungrounded answers and forces escalation, logging interventions. The same domain constitution informs evaluation, training rewards, and serving guardrails. Task-level should-escalate labels make the act-versus-defer decision a measurable training signal. We demonstrate the harness at smoke scale. An offline reference run (n=12) lifts escalation recall from 0 to 0.67 and reduces unsafe-action rate from 0.33 to 0.08 when governance is enabled. Two single-GPU Qwen2.5-3B LoRA/DPO pilots (n=8, same seed and evaluation split) expose substantial variation: nominally identical RL-base configurations yield task success of 0.25 versus 0.12 and escalation recall of 1.0 versus 0.5. An answer-quality adapter changes recall from 1.0 to 0.5 in Run A, but from 0.5 to 1.0 in Run B. An escalation-aware variant produces no measurable change in Run B. These small pilots do not establish reliable adapter effects or production readiness. Their contribution is diagnostic: configuration variance can overwhelm apparent tuning effects on bounded-autonomy metrics, motivating larger evaluation sets and repeated runs.

1 Introduction

The paper frames regulated agents as a high-stakes setting where prevailing steering techniques transfer poorly and the answer-versus-defer decision lacks a principled learned basis. RegLLM makes escalation a labeled training signal and combines a shared constitution with runtime governance; its pilots expose substantial variation at smoke scale.

1 Introduction

  • Regulated agents face high costs for wrong answers, while dominant steering techniques do not transfer well to these settings.The central unresolved workflow decision is when an agent should answer versus defer to a qualified human.Source: §1 · p. 1
  • Task-level should_escalate labels let the agent’s act-versus-defer decision become a learnable objective rather than a hand-coded threshold.The reward checks the terminal decision against the label, so correct escalation contributes to the trajectory score.Source: §1 · pp. 1–2
  • A single domain constitution serves as both a hybrid-reward term and a runtime bounded-autonomy policy across evaluation, training, and serving.Runtime rules block ungrounded answers, force escalation on out-of-scope triggers, and log interventions.Source: §1 · p. 2
  • Two nominally identical GPU pilots on the same evaluation split produce materially different metrics, while the same DPO adapter shifts escalation recall in opposite directions.The harness makes the limits of common smoke-pilot sample sizes empirically visible.Source: §1 · p. 2

2 Related Work

Prior work offers verifiable-reward methods, preference- and judge-based alignment, guardrails, domain-specific financial models, and auditing practices. The paper positions RegLLM around governed autonomous action and the regulated-domain decision of when to defer.

2 Related Work

3 The Harness

The harness combines a domain constitution, a tool-using compliance agent, distinct trustworthiness signal sources, hybrid rewards, preference training, and deterministic runtime governance. Its components share trajectory-level evidence and an auditable intervention trail.

3.1 Pluggable Domain Constitution

  • The worked example encodes 16 UK FCA principles, while a new regulated domain can supply its own principles without changing reward or governance machinery.A constitution comprises named principles with evaluation criteria and critique prompts.Source: §3.1 · p. 3

3.2 The Compliance Agent

  • The ReAct-style agent uses RETRIEVE, CITE, ANSWER, and ESCALATE, recording each step, cited provisions, and terminal decision for reward computation and audit..Source: §3.2 · p. 3

3.3 Three Categories of Signal

  • Trustworthiness signals are separated by supervision source: programmatic checks, task-level escalation labels, and constitutional-alignment scores from an LLM judge.The audit log records the signal category supporting each intervention; task labels and judge scores have their own reliability dependencies.Source: §3.3 · p. 3

3.4 Hybrid Reward

  • The hybrid reward combines verifiable and constitutional terms with a hard guardrail-violation penalty, making escalation correctness part of the verifiable signal.The constitutional term uses sampled principles scored by an AI judge; the verifiable term averages programmatic sub-signals.Source: §3.4 · pp. 3–4

3.5 Training

  • Rollouts are ranked by reward to form preference pairs for DPO training with LoRA; a stretch configuration uses GRPO with the same reward.Source: §3.5 · p. 4

3.6 Runtime Governance and Bounded Autonomy

  • At inference, an external deterministic layer blocks answers without cited provisions, forces escalation on specified competence or uncertainty triggers, and appends steps and interventions to an audit log.The cited guardrail principles are P9 and P10–P12.Source: §3.6 · p. 4

4 Implementation and Reproducibility

The implementation separates inference, training, and orchestration while providing schemas, metric specifications, pseudocode, and sample tasks as evaluation artefacts. It is designed to run training on a commodity single GPU and clean up rented infrastructure after failures.

4 Implementation and Reproducibility

  • Inference uses an open-weight LLM behind an OpenAI-compatible HTTP API, while training uses a single rented 24 GB-class GPU and S3-compatible object storage for inputs and checkpoints.The framework includes a Constitution interface, RegulatoryEnvironment, ReAct agent, hybrid reward, policy wrapper, and GovernedAgent.Source: §4 · p. 4
  • The orchestrator guarantees instance and volume teardown, and secrets are kept environment-only.The teardown design ensures a failure does not leave a paid GPU running.Source: §4 · p. 4
  • Ancillary materials provide task and constitution schemas, metric specifications, pseudocode, and sample tasks; reported numbers are regeneratable from these artefacts.Per-configuration metrics include escalation precision and recall and unsafe-action rate, alongside a principle-keyed audit log.Source: §4 · p. 4
  • Adapting to another regulated domain requires its principles, a parser for corpus and task labels, and a grounding oracle; reward weights and models are configurable.The example grounding oracle is lexical/structural, with NLI or a rule engine identified as alternatives.Source: §4 · p. 4

5 Experimental Protocol and Findings

The offline reference run shows governance improving escalation and safety metrics, while two GPU pilots expose substantial run-to-run variation that prevents reliable attribution of adapter effects.

Protocol

  • The held-out corpus uses UK FCA Handbook modules PRIN, COBS, SYSC, and CONC as both the provision corpus and grounding oracle.The diagnostic suite distinguishes exercised diagnostics from proposed follow-ups.Source: §5 · pp. 4–5

Task labels and data

  • The 12-task offline and 8-task GPU evaluation splits use author-generated labels from task templates, with an in-scope/out-of-scope ratio set by escalation_fraction=0.25.The labels were not expert-annotated, and the paper reports no inter-rater agreement.Source: §5 · pp. 4–5

Reference harness (offline)

  • Governance raised offline escalation recall from 0 to 0.67 and citation validity to 1.0, while cutting unsafe-action rate from 0.33 to 0.08.This was a held-out n=12 run using a lexical baseline policy and heuristic judge.Source: Table 2 · p. 5, §5 · p. 5

GPU pilots: variance is the principal finding

  • RL-base escalation recall fell from 1.00 to 0.50 and task success from 0.25 to 0.12 between GPU runs; the AQ adapter’s recall effect flipped direction.The EA adapter produced no measurable change in Run B.Source: Table 3 · p. 5, §5 · p. 5
  • The two pilots shared code commit, evaluation split, seed, model identifier, and policy hyperparameters, but GPU-host and software-version differences remain plausible sources of observed variance.Container image digests, package microversions, and deterministic CUDA flags were not pinned or enabled.Source: §5 · p. 5

Experimental findings

  • Across both GPU pilots, governance logged zero interventions; the no-citation gate was inert because answered turns cited provisions and missed escalations rarely reached terminal answers.A competence-boundary trigger would require a calibrated inference-time judge or stronger base-model behavior.Source: §5 · pp. 5–6

6 Discussion

At smoke-pilot scale, run variance swallows hypothesized adapter effects, so single-run directions should not be over-interpreted. The discussion also distinguishes label-dependent escalation evaluation from runtime competence-boundary governance.

What the variance finding is and is not

  • At n=8 and 30 DPO steps on 10–20 preferences, bounded-autonomy metric noise swallows the adapter effects the pilots aim to characterize.The finding is methodological, not a refutation of DPO or escalation-aware preference data.Source: §6 · p. 6

Two intentionally separate question types

  • Should-escalate correctness depends on task-level labels, which typically require expert annotation in regulated domains and may be contested.Treating these labels as programmatic understates annotation effort.Source: §6 · p. 6

Runtime governance needs a competence-boundary component

  • Zero interventions show that citation-only gates are insufficient: answered turns can cite provisions while missed escalations never reach a terminal answer for inspection.Competence-boundary detection should be evaluated as a first-class runtime component and may be calibrated against should-escalate labels.Source: §6 · p. 6

When to reach for this

  • The framework targets teams with a written rulebook, a way to identify out-of-scope queries, and a small open base model to fine-tune; it is not turnkey.Practitioners remain responsible for the constitution, corpus, and escalation labels.Source: §6 · p. 6

7 Conclusion

RegLLM makes act-versus-defer decisions a task-labeled training signal and uses one domain constitution in rewards and runtime policy. Its smoke-scale pilots reveal run variance that overwhelms apparent adapter effects, supporting larger evaluations and repeated runs.

Conclusion

  • RegLLM treats each task’s should-escalate label as a verifiable training signal for the act-versus-defer decision.The domain constitution also serves as both a hybrid-reward term and runtime bounded-autonomy policy.Source: §7 · p. 6
  • Two nominally identical GPU runs produced materially different RL-base metrics, while the AQ adapter’s apparent effect on escalation recall flipped direction.The escalation-aware variant produced no measurable change in the second run.Source: §7 · p. 6
  • The paper concludes that responsible evaluation requires larger samples and more seeds than current preference-tuning pipelines budget for.The harness makes this requirement empirically visible at smoke-pilot scale.Source: §7 · p. 6

Limitations

The evidence is a diagnostic smoke-scale demonstration, not a benchmark, bounded by small, noisy pilots and limitations in labels, judging, runtime governance, and domain coverage.

Limitations

  • The evidence is intentionally small-scale: the offline run uses n=12, while the GPU pilot uses n=8, one seed, one base model, and one regulated domain.Aggregate metrics are noisy, and absolute numbers should not be compared with production benchmarks.Source: Limitations · p. 7
  • Escalation correctness depends on reliable should_escalate labels, but these labels are template-generated, lack an inter-rater protocol, and are not validated for production-scale use.In regulated domains, labels typically require expert annotation and may be contested or shift with regulatory change.Source: Limitations · p. 7
  • The constitutional-alignment score depends on a model judge, while the worked example uses lexical overlap for grounding and a heuristic judge for reproducibility.Expert calibration, judge-consistency checks, and adversarial probing are not done here.Source: Limitations · p. 7
  • Runtime governance was inert in both GPU pilots, where cited answers passed the no-citation gate and missed-escalation trajectories did not reach a terminal answer.The proposed competence-boundary fix requires a calibrated inference-time judge or richer base-model behaviour than the smoke pilot provides.Source: Limitations · p. 7
  • Identical seeds, evaluation splits, base models, and policy hyperparameters still yielded materially different RL-base metrics, while the AQ adapter’s apparent recall effect flipped direction.The escalation-aware variant produced no measurable change; the contrast is not evidence of the adapters’ true behaviour and motivates multi-seed, larger-n evaluation.Source: Limitations · p. 7
  • Only the FCA example was instantiated and evaluated; other regulated domains require their own principles, corpus, grounding oracle, and escalation labels.Those domain-specific components are described as non-trivial.Source: Limitations · p. 7

Ethics Statement

The harness is intended to keep agents within human-set limits and defer matters outside their competence to qualified professionals. It is evaluation tooling and a smoke-scale prototype, not a deployable compliance system.

Ethics Statement

  • The harness is designed to keep agents within regulator-set limits and defer matters outside their competence to qualified human professionals.It uses publicly available UK FCA Handbook text and an open-weight policy model, with no personal or sensitive data.Source: Ethics Statement · p. 7
  • The work provides evaluation tooling and a smoke-scale prototype, not a deployable compliance system.The paper states that regulated-domain escalation labels require expert annotation, which this work does not provide.Source: Ethics Statement · p. 7
  • The authors interpret the negative result as indicating that answer-quality preference pipelines alone are insufficient for regulated autonomy.They state that responsible deployment requires additional labelling and runtime hooks.Source: Ethics Statement · p. 7

Related papers

Loading 2609.37501v1…