PaperlayerView exact revision on arXiv

Source-linked AI summary

Regret Dominates Surprise: Design-Time Requirements Engineering for Agentic-AI Safety

Nuwayyir Almohammadi, Rami Bahsoon, Tao Chen

arXiv:2609.05692v1cs.SE

TL;DR

GORE supports obstacle analysis but lacks a systematic design-time way to evaluate tactic choices under epistemic uncertainty, including when autonomy should yield to reflection or human control. MS-RGR extends GORE with surprise and regret signals and a design-time gate for selecting resolution tactics. The paper reports near-zero silent failures versus a 96% sensor-only baseline and risk detection 17.5× faster, while noting that its AGENTHARM gate was applied post-hoc to execution traces.

  • Problem

    GORE handles known obstacles but lacks a systematic design-time way to evaluate tactic quality under epistemic uncertainty, including the autonomy/reasoning boundary.

    Source: §II · p. 2, §III · p. 3

  • Method

    MS-RGR extends GORE with design-time regret thresholds and a gate that combines surprise with regret to select resolution tactics.

    Source: §I · p. 2, D. Step 3: Define Regret Thresholds as… · p. 4, F. Step 5: Regret-Aware Selection of… · p. 5

  • Results

    Near-zero silent failures versus a 96% sensor-only baseline; risk detection is 17.5× faster.

    Source: §I · p. 2

  • Takeaways & Limitations

    MS-RGR provides a verifiable complement to GORE for preserving the autonomy boundary under silent failure.

    Source: §X · p. 11

    The DRI gate was applied post-hoc to recorded traces, so results evidence log-based computability rather than authentic design-time integration effects.

    Source: 4) Limitations · p. 10

Abstract

from arXiv · show

Requirements engineers for agentic-AI domains face challenges in evaluating, specifying, and operationalizing safe autonomy. Mainstream frameworks, such as Goal-Oriented Requirements Engineering (GORE), lack mechanisms to systematically address these challenges under epistemic uncertainty. We contribute an approach that builds on GORE to model and simulate safe autonomy in agentic-AI systems. We introduce a novel Regret-Dominance Mechanism (MS-RGR) to operationalize safe autonomy. MS-RGR uses two signals: epistemic surprise (novelty detection) and cognitive regret (evaluative risk) to address the trilemma problem: should the agent operate in routine autonomy, undergo reflective reasoning, or escalate to human? We instantiate MS-RGR in elderly care monitoring and autonomous driving. A 100-seed stochastic simulation shows MS-RGR reduces silent failures to near-zero and detects risk approximately 17.5 times faster than a sensor-only baseline, remaining formally traceable via LTL safety properties. A retrospective proxy instantiation applying the DRI gate post-hoc over execution traces from 208 AGENTHARM scenarios across seven LLMs shows the gate improves harmful-task refusal only for models with strong baseline safety (over 80% pre-gate refusal, e.g., 84.1% to 90.9%), indicating MS-RGR amplifies rather than substitutes for model-level safety training. We discuss threats to validity, positioning MS-RGR as initial feasibility evidence for design-time safety constraints in agentic-AI requirements engineering.

I. INTRODUCTION

Agentic-AI autonomy creates requirements-engineering challenges because systems must act safely amid unknowns and epistemic uncertainty. MS-RGR combines surprise and regret to make autonomy, reflection, and human escalation design-time safety choices.

I. INTRODUCTION

  • Agentic systems can act autonomously in high-stakes settings, while existing RE frameworks lack systematic ways to evaluate decisions under epistemic uncertainty.The stated trilemma is whether to continue routine autonomy, reason reflectively, or hand control to a human.Source: §I · p. 1, §II · p. 2
  • MS-RGR combines surprise as a signal of novelty with regret as evaluative risk to govern routine, reflective, or human-supported autonomy.The approach anchors these signals in GORE so thresholds remain traceable to stakeholder requirements.Source: §I · pp. 1–2, §II · p. 2
  • Requirements engineers can embed deterministic resolution tactics at design time, including goal weakening for moderate risk and goal sacrifice for critical risk.Critical-risk escalation replaces the software agent with a human caregiver; these are explicit engineering boundary decisions, not autonomous runtime model choices.Source: §I · p. 2
  • Near-zero silent failures versus a 96% sensor-only baseline; risk detection is 17.5× faster.The introduction reports these as MS-RGR outcomes.Source: §I · p. 2

II. BACKGROUND AND MOTIVATION

  • The gate distinguishes benign anomalies, which can be monitored, from high-surprise, high-regret hazards that warrant immediate escalation.Surprise alone risks unnecessary interruptions, while regret alone is reactive and post-hoc.Source: §II · p. 2

III. RELATED WORK AND PROBLEM FRAMING

Prior requirements-engineering approaches provide obstacle analysis or sensor-driven monitoring, but leave the autonomy boundary under epistemic uncertainty without an explicit design-time evaluative criterion. MS-RGR extends GORE with a log-based regret threshold and a gate for selecting routine execution, reflection, or human escalation.

III. RELATED WORK AND PROBLEM FRAMING

  • KAOS obstacle analysis and sensor-driven MAPE-K monitoring lack an explicit design-time evaluative boundary for the autonomy/reasoning trilemma.MS-RGR makes the boundary computable from interaction logs independently of sensor state.Source: §III · p. 3

IV. A MULTI-SIGNAL FRAMEWORK FOR REGRET-AWARE GOAL RESOLUTION IN AGENTIC-AI (MS-RGR)

  • MS-RGR extends GORE obstacle analysis with cognitive-regret thresholds that constrain which resolution tactics are admissible.Its five-step process models goals and tactics, identifies regret-prone situations, defines thresholds, triggers assessment, and selects tactics.Source: A. Overview · p. 3, §IV · p. 3

A. Overview

  • The gate assigns routine execution when DRI < τ and SE = 0, reflective reasoning when DRI ≥ τ, and human escalation when DRI ≥ τcrit.The predicates are computable from interaction logs before deployment and do not require sensor confirmation.Source: A. Overview · p. 3
  • The runtime mechanism combines environmental surprise with execution logs in the DRI, then routes the agent among routine, reflective, and human modes.It operationalizes design-time requirements as an executable MAPE-K loop [9].Source: Figure 2 · p. 4
  • TBR is 0% across all 100 seeds under sensor freeze for the critical human-escalation predicate.This is the reported RQ2 test of the third design-time predicate.Source: A. Overview · p. 4
  • The KAOS model traces root goal G0 through obstacles O1 and O2 to resolution tactics RT5–RT7, alongside a MAPE-K feedback loop.The obstacles represent silent sensor failure and trust-bankruptcy risk.Source: A. Overview · p. 4, Figure 1 · p. 4

B. Step 1: Model Goals and Resolution Tactics

The method models goals and obstacles, identifies situations where a feasible alternative could have produced a better outcome, and turns regret thresholds into safety requirements. The DRI combines persistence, user sensitivity, and a trust buffer to distinguish routine, adaptive, and safety-enforcing responses.

B. Step 1: Model Goals and Resolution Tactics

  • GORE/KAOS captures system goals,; for each goal, it identifies potential obstacles and defines multiple resolution tactics, following established obstacle-refinement practices in GORE,.Source: B. Step 1: Model Goals and Resolution… · p. 4

C. Step 2: Identify Regret-Prone Situations

  • Designers identify regret-prone situations by asking how a system could meet its functional goal yet fail ethically or otherwise have had a better alternative.The analysis considers context, the chosen action and actual outcome, plus a feasible alternative and its better outcome.Source: C. Step 2: Identify Regret-Prone… · p. 4

D. Step 3: Define Regret Thresholds as Safety Requirements

  • The DRI uses exponentially growing prompt persistence and a trust buffer bounded below at 0.05; its user-sensitivity score weights latency, refusal, and abandonment rates.The defaults are α = 0.5, β = 0.3, γ = 0.2, and η = 0.05; the buffer encodes near-ruin loss aversion, with DRI approaching 1.0 as the buffer approaches its minimum.Source: D. Step 3: Define Regret Thresholds as… · p. 4
  • In the medication example, DRI is 0.27 after two unanswered prompts and 0.56 after five, moving from continued System 1 execution to the Medium zone that primes System 2.The example uses L = 3.0, R = 0.2, A = 0.1, and Tbuffer values of 0.8 and 0.6 at the two points.Source: D. Step 3: Define Regret Thresholds as… · p. 4
  • High regret begins at DRI ≥ 0.6, where safety-enforcing tactics are mandated; 0.4 ≤ DRI < 0.6 calls for adaptive tactics, while DRI < 0.4 is routine-safe.The default τ = 0.6 is the High-zone boundary; τ = 0.4 and τ = 0.8 bracket it for life-critical and low-stakes variants, respectively, and τ = 0.8 was identified as unsafe.Source: D. Step 3: Define Regret Thresholds as… · p. 4

E. Step 4: Detect Epistemic Novelty and Initiate Multi-Signal Reflection

Runtime novelty triggers reflective reassessment, while design-time regret thresholds retain decision authority. The gate computes DRI, routes to admissible tactics, and updates the trust buffer; its tactic choices remain traceable to the goal–obstacle model.

E. Step 4: Detect Epistemic Novelty and Initiate Multi-Signal Reflection

F. Step 5: Regret-Aware Selection of Resolution Tactics

  • When surprise is absent, the gate still fires at DRI(t) ≥ τcrit because regret accumulates from interaction logs independently of the frozen sensor.Every tactic selection traces to the design-time goal–obstacle model, making the routine/reflective switch formally auditable before deployment,.Source: F. Step 5: Regret-Aware Selection of… · p. 5
  • The method lists five global LTL safety properties and assigns chosen tactics residual-risk labels M0 (Normal), M1 (Cautious), or M2 (Interrupt).Table I contains the global safety properties.Source: F. Step 5: Regret-Aware Selection of… · p. 5, Table I · p. 5

G. Step 6: Feedback for Model Evolution

The feedback step uses persistent surprise to refine the goal–obstacle model, while the DRI plot tests zone assignment across threshold settings. The following section summarizes regret-calibrated resolution tactics.

G. Step 6: Feedback for Model Evolution

  • Persistent surprise signals prompt KAOS obstacle elaboration rather than runtime decision inputs, with engineers refining obstacles, decomposing them, or substituting tactics.Figure 3 shows DRI accumulation across four τ configurations.Source: G. Step 6: Feedback for Model Evolution · p. 5
  • Three τ configurations preserve correct zone assignment across all seeds, whereas τ = 0.8 intrudes into RT5–7 and is linked to OCR = 99% failure; ordinal zone ordering persists across the tested λ and η ranges.Figure 3 plots DRI across 100 timesteps for 100 seeds, with mean and 10th–90th percentile bands; the parameter ranges are λ ∈ {0.10–0.25} and η ∈ {0.03–0.10}.Source: Figure 3 · p. 5

H. Regret-Calibrated Resolution Tactics

I. Handling Undetected Surprise Signals

The method addresses undetected surprise with DRI-based caution, then instantiates the goal, obstacle, and tactic model in elderly care and autonomous driving. The evaluation uses goal-driven stochastic simulation for pre-deployment what-if analysis.

I. Handling Undetected Surprise Signals

  • When surprise is undetected despite genuine risk, high DRI triggers M1-Cautious escalation, while low DRI maintains M0-Normal and logs the case for future elaboration.The text identifies this as a conservative response even when epistemic monitoring fails, with the S3 scenario confirmed in Section VII.Source: §I · pp. 5–6

V. METHOD INSTANTIATION: ELDERLY CARE

  • The elderly-care instantiation covers mobility assistance, medication adherence, and emergency response, with KAOS goals for safe mobility, medication compliance, and timely response.Potential obstacles are identified for each goal; Table III instantiates the domain’s tactics.Source: §V · p. 6, A. Goals and Obstacles · p. 6, Table III · p. 6

B. Regret-Prone Situations

  • In the mobility example, RT1 is selected as a resident stands, creating fall risk; the outcome is assessed as High regret, for which RT4 caregiver notification would have been preferable.The DRI trace is referenced in Example 1, Section IV-C.Source: B. Regret-Prone Situations · p. 6

C. Runtime Obstacle Handling

  • At runtime, high DRI escalates to RT3–RT7 (M1 or M2) with or without surprise, while low DRI maintains RT0–RT1 (M0) regardless of surprise..Source: C. Runtime Obstacle Handling · p. 6

VI. METHOD INSTANTIATION: AUTONOMOUS DRIVING

  • In the autonomous-driving passenger-comfort scenario, DRI = 0.64 ≥ τ = 0.6 during fast cornering routes to RT6, goal weakening, while preserving partial goal satisfaction.The driving instantiation re-instantiates DRI variables over navigation signals while using the same gate logic and RT0–RT7 structure as elderly care.Source: §VI · p. 6

VII. EVALUATION

  • The feasibility study uses goal-driven stochastic simulation to evaluate MS-RGR, with DRI as a quality variable computed from standard execution logs.The simulator supports requirements engineers’ pre-deployment what-if analyses of safety thresholds and resolution tactics.Source: §VII · p. 6

A. Experiment Setup

The experiment uses stochastic simulation and theory-removal conditions to evaluate operability, robustness, traceability, and what-if utility.

A. Experiment Setup

  • 100 independent seeds each run for 100 discrete timesteps, with friction and surprise injected as independent probabilistic events.The simulation sets Pr(friction) = 0.7 and Pr(surprise) = 0.2.Source: A. Experiment Setup · p. 6
  • Five research questions assess DRI operability, safe tactic selection under sensor freeze, LTL traceability, theory contributions, and graduated trilemma coverage.The questions are organized as What-If Analysis I for RQ1–RQ3 and an ablation study for RQ4.Source: A. Experiment Setup · p. 6
  • The comparisons include Full MS-RGR, fixed-trust and sensor-only baselines, plus ablations removing Prospect Theory, Reactance Theory, the dual-process gate, or satisficing.The sensor-only baseline uses surprise-threshold gating without DRI; the dual-process ablation removes the τ gate and always escalates to System 2.Source: A. Experiment Setup · p. 7

B. Case Study I: Goal-Driven Stochastic Simulation

The stochastic simulation tests DRI ordering, robustness under sensor freeze, and formal traceability across τ configurations. Full MS-RGR reduces simulated silent failures and detects risk faster than the sensor-only baseline.

B. Case Study I: Goal-Driven Stochastic Simulation

  • The first case study applies stochastic simulation to a goal-driven system, with RQ1–RQ3 evaluated through What-If Analysis I.The RQ4 evaluation is conducted separately as an ablation study.Source: B. Case Study I: Goal-Driven… · p. 7

1) RQ1 – Operability:

  • 100% OCR across 100 seeds in Strict, Standard, and Relaxed configurations confirms correctly ordered regret levels with zero ordering inversions.In Lenient mode (τ = 0.8), OCR was 99%.Source: 1) RQ1 – Operability · p. 7

2) RQ2 – Robustness:

  • 0% versus 96% TBR under S3 sensor-freeze scenarios, while Full MS-RGR detects risk in 4.1 versus 72.0 timesteps under the Sensor-Only baseline.The TBR comparison is reported for 100 seeds; the risk-detection comparison is 17.5× faster for Full MS-RGR.Source: Table V · p. 7, 2) RQ2 – Robustness · p. 7

3) RQ3 – Traceability:

4) RQ4 – What-If Utility:

The what-if analyses assess the behavioral theories and structural properties behind MS-RGR, then examine sensitivity to τ. Their results support the combined mechanism while identifying a calibration boundary and an evaluation limitation.

4) RQ4 – What-If Utility:

  • 33.3% activation, 6.0-step TTI, zero false alarms, and 0.480 intervention cost make Condition A uniquely best across all four metrics (p = 0.001 for each test).Figure 5 reports that Full MS-RGR uniquely minimizes all four metrics simultaneously.Source: Figure 5 · p. 8, 4) RQ4 – What-If Utility · p. 8
  • Removing Prospect Theory raises ACT% from 33.3% to 52.5%; removing Reactance delays TTI from 6.0 to 8.5 steps; removing the dual-process gate raises FAR% to 100%; removing satisficing raises IC by 30%.The reported ablation results attribute these changes to omitted trust recovery, slower linear persistence, unconditional escalation, and skipping goal weakening, respectively.Source: 4) RQ4 – What-If Utility · p. 8

5) RQ5 – RE Coverage of the routine/reflective/escalation trilemma:

  • KAOS design-time thresholds, three ordinal DRI zones, and LTL properties provide graduated, auditable responses; removing the dual-process gate collapses behavior to binary escalation with FAR = 100%.The ablation discussion links the zone structure to differentiated handling of transitional cases.Source: 5) RQ5 – RE Coverage of the… · p. 8
  • FAR% does not measure ground-truth safety, and activations at DRI ≥ τ that prove safe under reflection remain an unquantified false-alarm category.The passage says full characterization requires ground-truth activation labels.Source: 5) RQ5 – RE Coverage of the… · p. 8

6) Sensitivity Analysis:

C. Case Study II: Agentic-AI Safety via AgentHarm

The AGENTHARM case study applies the DRI gate post-hoc to execution traces from harmful-task scenarios across seven models. Refusal gains concentrate in models with strong baseline safety, while category-level effects vary.

1) Evaluation Methodology:

2) Results by Harm Category:

C. Case Study II: Agentic-AI Safety via AgentHarm

  • The evaluation compares refusal rates before and after gate application across seven models.The models were evaluated over the 208 scenarios, using tool-call grading for one model and keyword detection for the others.Source: 3) Cross-Model Comparison · p. 10, Table VII · p. 10

3) Cross-Model Comparison:

  • Meaningful gains occurred only above 80% baseline refusal: Claude Haiku 4.5 rose from 84.1% to 90.9% (+6.7 pp), GPT-OSS-120B from 95.5% to 100% (+4.5 pp), and five other models changed by ≤0.6 pp.The authors position the gate as amplifying, not substituting for, model-level safety training.Source: Table VII · p. 10, 3) Cross-Model Comparison · p. 10

4) Limitations:

The paper reports feasibility benefits from MS-RGR while identifying limits in proxy evaluation, simulation validity, and generalizability. It frames further validation and deployment as future work.

4) Limitations:

  • Free-model evaluation used keyword detection on plain chat rather than AGENTHARM’s tool-call grading, and the DRI gate was applied post-hoc rather than embedded before fine-tuning.Capability denials were counted as compliance, understating true safety; the authors say the results establish log-based computability, not the effect of authentic design-time integration.Source: 4) Limitations · p. 10

A. Mitigation of Spurious Adaptations

  • MS-RGR adaptation reduces spurious interruptions by triggering only when epistemic mismatch coincides with a violation of design-time safety thresholds.Full empirical investigation across all 44 public test scenarios is identified as immediate future work.Source: A. Mitigation of Spurious Adaptations · p. 10

B. Traceable Governance under Uncertainty

  • Ordinal Low, Medium, and High regret levels let designers specify safety concerns as qualitative constraints interpretable across the system lifecycle.The paper describes this as shifting control from runtime intelligence to verifiable engineering intent, including under unknown unknowns.Source: B. Traceable Governance under… · p. 10

IX. THREATS TO VALIDITY

  • Threats include unvalidated human-trust alignment and domain generalization, plus synthetic traces and deployment-specific simulation probabilities not yet validated in the studied contexts.The authors describe simulation results as feasibility evidence pending real-world deployment.Source: §IX · p. 10
  • Tactic selection is traceable to the GORE/KAOS requirements model, while structured mapping tables and ordinal thresholds support reproducible regret-level assignment.Empirical validation of DRI against human-subject trust data remains future work.Source: §IX · p. 10

X. CONCLUSION AND FUTURE WORK

  • The conclusion says the log-based DRI threshold preserves the autonomy boundary under silent failure and identifies multi-agent architectures and real-time surprise monitoring as future work.Source: §X · p. 11
Loading 2609.05692v1…