A Local Agentic AI Framework to Generate Edge Case Scenarios for Safety Assurance

1. Introduction

• Safety-critical autonomous systems require through-life assurance despite reduced human oversight and open-world conditions that can include underspecified or unknown environments.Novel inputs and unexpected situations complicate arguments that systems meet operational, safety, and regulatory standards throughout their operational lifetime.Source: §1 · pp. 1–2

1.1. Problem Statement

• Manual generation of realistic edge cases is infeasible across vast, nonlinear parameter spaces, while Monte Carlo sampling struggles with dimensionality and does not target edge cases.Edge cases are statistical outliers near ODD boundaries; identifying triggering conditions can expose functional gaps and inform risk mitigation.Source: §1.1 · p. 2

1.2. Proposed Solution

• The proposed solution uses language models to generate and test edge-case scenarios for autonomous systems in complex environments.The framework converts safety requirements into structured JSON scenarios, runs them in simulation, and analyses simulation metrics as traceable safety-assurance evidence.Source: §1.2 · pp. 2–3, Figure 1 · p. 3

1.3. Key Contributions •

• The contributions combine standards-to-JSON mapping, iterative edge-case generation, simulation validation reporting higher edge-case yield and hazard discovery than baselines, safety-case evidence, and consumer-laptop feasibility.The paper positions the framework as a contribution to safety assurance and describes the remaining sections covering architecture, empirical analyses, results, limitations, and future work.Source: §1.3 · p. 3

2. Regulatory Background & Related Literature

• Scenario-based testing supports system-level assurance, but achieving it requires an automated, compute-efficient way to specify and test the system, environment, tasks, and expected behavior.The common workflow generates scenarios, simulates them, and analyzes the results.Source: §2 · p. 4

2.1. Safety Assurance Standards and Frameworks

• System-level safety assurance requires comprehensive testing across the ODD, while conventional standards and current AI validation leave gaps for high-dimensional, underspecified environments and local edge-case failures.SOTIF distinguishes known/unsafe boundary tests from unknown/unsafe edge cases; PAS 8800 requires synthetic edge-case scenarios but leaves implementation mechanisms open.Source: §2.1 · p. 4
• STAMP/STPA, SACE, and scenario-based testing offer safety-assurance routes, but scenario coverage can be broad, application-specific, or weakly guided for edge-case testing.STAMP/STPA may generate many scenarios and bespoke modelling templates; SACE provides little guidance on simulation-based scenario coverage.Source: §2.1 · pp. 4–5, §2 · p. 4

2.2. Robotic Simulation & Virtual Testing

• Simulation provides a controlled way to test generated scenarios without risking people, property, or the environment, and Unity-3D can represent complex dynamic environments.Even low-fidelity simulation can reveal navigation bugs, while higher-fidelity game engines can support realistic environments and real-time LiDAR simulation.Source: §2.2 · p. 5

2.3. AI-Driven Scenario Generation

• Scenario generation can target edge cases, but existing methods range from random or model-intensive techniques to evolutionary search and reinforcement learning with configuration or training demands.Monte Carlo search is not edge-case targeted; statistical modelling can be difficult, evolutionary methods require configuration and trial-and-error, and reinforcement learning requires extensive training and reward specification.Source: §2.3 · pp. 5–6
• Prior LM scenario pipelines may depend on datasets, manual intervention, or user-specified critical scenarios, motivating automated, compute-efficient generation from safety standards to executable scenarios.The proposed framework is fully automated, incorporates feedback, and does not require prior knowledge of errors or functional gaps.Source: §2.3 · p. 6

3. System Architecture & Local Agentic Closed Loop

• The closed loop generates and validates scenarios, runs them in simulation, uses evaluation metrics to choose exploitation or exploration, and retains scenarios and results as traceable safety-assurance evidence.Post-processing relates scenario descriptions and metric scores to SOTIF triggering conditions and concrete, reproducible evidence artifacts.Source: §3 · pp. 6–7

3.1. Agentic AI Closed Loop Overview

• A generator LM uses ODD knowledge bases and a task description to draft scenarios; an independent reflection LM critiques them so the generator can revise and validate them.The knowledge bases comprise environment description, system description, and system requirements.Source: §3.1 · pp. 7–8

3.2. Local Agentic Architecture and Optimisation

• The laptop-oriented design reserves software engineering for orchestration and decision logic, while structured JSON reduces communication overhead between LMs and simulation.JSON encoding allows the closed loop to run alongside Unity-3D simulation on a single-GPU laptop.Source: §3.2 · p. 8
• Scenarios are generated in batches of four, then validated and repaired before simulation to reduce local-model JSON errors and avoid wasting compute on invalid inputs.Pydantic checks JSON and data types and can catch unexpected fields; the authors report four as the largest error-free batch size for their setup.Source: §3.2 · p. 8

3.2.1. Exploitation

• When simulation metrics indicate functional gaps, the framework prompts the LM to rewrite the current scenario into new, similar scenarios.The rewrite prompt requires strictly valid JSON following the scenario interface.Source: §3.2.1 · p. 9

3.2.2. Exploration

• Exploration incorporates scenario history to request novel edge cases; a short FIFO history addresses smaller models’ context limits and repetition, while temperature balances creativity against rule-following.The authors selected temperature 0.7 through inspection of generated sets and note that excessive temperature can introduce hallucinations; they also use separate LMs for generation and reflection to reduce observed self-critique confirmation bias.Source: §3.2.2 · p. 9

3.3. Unity-3D Simulation Environment

• The simulator tests a robot navigating numbered tiles in order while avoiding fixed and random obstacles, with JSON scenarios and evaluation metrics supported by the simulation interface.The robot uses seven-beam LiDAR and episodes end on success, timeout, or visiting the wrong tile.Source: §3.3 · pp. 9–10

3.3.1. Evals

• Simulation metrics determine whether to exploit or explore: low success, more than 24 crashes, or visiting fewer than 75% of tiles triggers exploitation.The decision logic uses successScore <= 45, crash_count > 24, or avg_ratio_tiles_visited < 0.75.Source: §3.3.1 · p. 10

4. Safety Requirement to JSON Schema Mapping

• GSN decomposes the system-level zero-collision claim into navigation verification goals and lower-level evidence requirements for scenario generation and evaluation.The evidence for those goals includes evaluation metrics, simulation failures and collisions, identified SOTIF area 3-to-area 2 edge cases, and a hazard discovery report.Source: §4 · p. 11, Figure 5 · p. 11, Figure 6 · p. 12

4.1. JSON Schema Specification

• The JSON schema encodes scenario spatial layouts with a name, text description, and start location, and is intended to be agnostic to the LM, simulator, task, and domain.The simulation also returns evaluation metrics as JSON, which can be parsed and verified with a variable Pydantic specification.Source: §4.1 · pp. 12–13

4.2. Constraint Validation

• The domain ontology combines environment, system, and requirement descriptions with the task description as prompt knowledge bases that guide scenario generation and constrain outputs to the ODD and schema.Removing environment or system descriptions led to scenarios violating specified obstacle-scale or start-location rules; reflection and refinement validate scenarios against the knowledge bases.Source: §4.2 · p. 13

4.3. SOTIF Trigger Mapping

• A scenario is a SOTIF triggering condition when simulation exposes a safety-constraint violation at any time; otherwise it is classified as known safe.The guarantees can include minimum obstacle clearance, limits on incorrect tile visits, or path-completion time bounds.Source: §4.3 · p. 13
• Unity metrics and scenario descriptions identify triggering conditions and flag area 3 unknown/unsafe scenarios for safety engineers to convert into area 2 known/unsafe cases and mitigate.The JSON scenario and its text description form a reproducible artifact, while metric and simulation evidence supports traceable safety-case claims.Source: §4.3 · pp. 13–14

5. Experimental Setup & Benchmarking

• The framework is presented as simulator- and use-case-agnostic, with a Unity-3D Sorter arena used here to test the trained robotic agent.The lightweight JSON schemas are intended to adapt to more complex systems, environments, and tasks.Source: §5 · p. 14, §5.1 · p. 14

5.1.1. Brains

• The tested navigation brain uses deep reinforcement learning to learn actions from rewards and penalties, exposing policy gaps through scenario-based testing.The policy is trained with PPO, which limits large changes from the previous policy.Source: §5.1.1 · pp. 14–15

5.1.2. Incremental Curriculum Learning

• Incremental curriculum learning advances from simple tasks to more complex ones, transferring the current policy and continuing each lesson until a metric baseline is exceeded.The training begins with up to two tiles and increases the number by two per lesson, up to 20.Source: §5.1.2 · p. 15

5.1.3. Training

• The PPO brain was trained for 15 million epochs using eight parallel Unity-3D instances and curriculum lessons with reward-based stopping criteria.Training scenarios used 1–20 tiles and five movable obstacles in randomized grid layouts.Source: §5.1.3 · p. 15, Figure 7 · p. 16

5.2. Scenario-Based Testing

• Each scenario-generation approach produced 512 scenarios, each run for 50 episodes, for 25,600 episodes per approach to account for stochastic policy actions and environmental randomness.The scenarios were tested against coverage, robustness, accuracy, and navigation functional gaps.Source: §5.2 · p. 15

5.2.1. Baselines for Comparison

• The study compares the agentic approach with two baseline scenario-generation approaches and evaluates their scenario-generation performance using a set of metrics.The baselines are described as inspired by those in the authors’ previous paper.Source: §5.2.1 · p. 15

5.3. Evaluation Metrics

• The evaluation metrics assess coverage, complexity, diversity, difficulty, and hazard frequency, using measures such as success rate, reward, tile visitation, collisions, path length, and action entropy.Spatial heatmaps and distributions of obstacles, tiles, positions, and values are used to examine coverage.Source: §5.3 · p. 16

6. Results

• The evaluation combines Unity metrics with scenario spatial layouts to compare Random_1, Random_5, and Agentic generation techniques.Table 1 presents the baseline approaches, and Table 2 lists the evaluation metrics used in the closed-loop analyses.Source: §6 · p. 16, Table 1 · p. 17, Table 2 · p. 17

6.1. Failure Discovery Efficiency

• Success, failure, reward, and regret charts assess how well each method generates critical scenarios that expose functional gaps.The figures include success ratios, success-versus-timeout-versus-wrong-tile ratios, cumulative reward, and regret measured as average path length divided by ideal path length.Source: §6.1 · p. 18, Figure 8 · p. 18, Figure 10 · p. 18

6.1.1. Spatial Heatmaps

• Tile visitation, collisions, path length, and action-entropy charts assess scenario difficulty, complexity, diversity, and action exposure.The inter-quartile ranges indicate scenario diversity, while obstacle and tile heatmaps and distributions show coverage and completeness.Source: §6.1 · p. 18, §6.1.1 · p. 18, Figure 12 · p. 19

6.2. Qualitative Analysis

• Dense obstacle clusters near the agent’s start and progressively challenging environments characterize low-reward critical scenarios, while strategic route-blocking obstacles characterize a higher-reward, less challenging scenario.The low-reward cases also show high path length, multiple collisions, timeouts, and wrong-tile visits; the latter often coincide with the agent becoming stuck.Source: §6.2 · p. 19

7.1.1. Functional Gaps

• Agentic generation yielded fewer successful runs and generally higher final-reward quartiles than random methods, but random methods also found occasional severe failures and had similar regret.The authors note that random discovery of edge cases occurs by chance.Source: §7.1.1 · pp. 20–22
• Agentic scenarios showed higher tile-visiting failure and collision rates, broader path-length spread, and skewed action choices, exposing gaps missed by random scenarios.The action-distribution results indicate skewed policy choices, while transition entropy does not show more repetitive actions than Random_1.Source: §7.1.1 · pp. 22–23

7.1.2. Spatial Analyses

• Agentic generation targets a subset rather than random’s coverage, often arranging obstacles in geometric patterns and selecting the maximum permitted counts of 10 obstacles and 20 tiles.The reported patterns include circles, T-junctions, blocking lines and arcs, and diamonds.Source: Figure 17 · p. 20, Figure 18 · p. 21, Figure 19 · p. 21
• Agentic scenarios have higher minimum ideal path length and wider interquartile spread than Random, indicating greater complexity and diversity.Figure 22 presents ideal path length across scenarios for each generation method.Source: §7.1.2 · p. 23, Figure 22 · p. 23

7.2. Safety Assurance Case Artefact Synthesis

• Clustering scenario descriptions with simulation metrics links triggering conditions to functional gaps and supports GSN evidence, moving discovered unknown-unsafe cases into known-unsafe SOTIF area 2.The hazard discovery report maps the spatial layouts of θJS ON1 and θJS ON2 to triggering conditions and gaps exposed by simulation metrics.Source: §7.2 · p. 23, Table 3 · p. 24

7.3. Limitations

• Smaller language models sometimes generated scenarios with more than 10 obstacles despite prompt limits, motivating validation guardrails rather than longer prompts.The proposed handling is to regenerate a scenario or randomly select 10 obstacles during Pydantic validation.Source: §7.3 · p. 24
• The prototype’s exploitation step sometimes revised critical spatial layouts into scenarios that were easy to solve, and the authors call for investigation of reward results.They observed this behavior in a visual cross-section of Unity-3D scenarios.Source: §7.3 · p. 24
• Evaluation remains bounded by simulation’s sim-to-real gap and lacks an absolute coverage metric; scenario range is assessed using metric spreads and visual inspection.The authors acknowledge that Unity-3D does not create a full digital twin.Source: §7.3 · pp. 24–25

8. Conclusion & Future Work

• The framework addresses the challenge of verifying safety across the ODD, particularly whether unusual spatial layouts trigger navigation failures and under what conditions.The authors say these findings can inform navigation-model robustness improvements or runtime failsafe mitigations.Source: §8 · p. 25
• The local, closed-loop framework runs scenario generation, simulation, evaluation, and feedback on a standard laptop, avoiding cloud costs, latency, service outages, and privacy issues.Its JSON input and evaluation schemas are described as simulator-independent; Unity-3D was used here.Source: §8 · p. 25
• Clustering evaluation metrics with scenario descriptions can identify triggering conditions linked to functional gaps and produce traceable evidence for safety assurance.The authors describe converting ISO 21448 area 3 unknown/unsafe scenarios to area 2 known/unsafe when a generated scenario violates safety boundaries.Source: §8 · p. 25

8.1. Future Work

• Scenario-based testing revealed functional gaps in the RL navigation policy, including agents that become terminally stuck and time out or push obstacles aside.The authors propose using generated scenarios for curriculum learning and retesting updated policies.Source: §8.1 · p. 26
• Replay logs were not collected because of disk-space limits; the authors propose collecting them to investigate agent and environment states and actions preceding failures.They expect these logs to support forensic analysis alongside thematic analysis.Source: §8.1 · p. 26
• Additional metrics and detailed logs could support collision categorisation and reveal conditions such as failures in Unity-3D Night Mode or getting stuck around particular obstacle shapes.The authors describe this feedback as part of deciding whether to explore or exploit.Source: §8.1 · p. 26
• The current Unity-3D Sorter setting has static obstacles and waypoints; extending the framework to dynamic obstacles and movable waypoints would require JSON trajectory specifications.That extension would allow generation of reactive moving hazards alongside static layouts.Source: §8.1 · p. 26
• The authors plan to transfer generated JSON layouts to physical hardware-in-the-loop testbeds and investigate schema needs and the sim-to-real gap.Source: §8.1 · p. 26

• The work was supported by the Centre for Assuring Autonomy, a partnership between Lloyd’s Register Foundation and the University of York.Source: Funding Declaration · p. 26

• The authors report no relevant financial or non-financial interests to disclose.Source: Competing Interests · p. 26

• All data used in the paper were generated using the described agentic framework.Source: Data Availability · p. 26

Appendix A.

• Scenarios are encoded across JSON, TypeScript, and Pydantic specifications, with fields for the agent, tiles, and obstacles.The specifications include defaults such as a room radius of 20 and a maximum of 5000 episode steps.Source: Appendix A · pp. 31, 34, 35
• The domain ontology combines environment, system, and requirements descriptions in the contextual prompt, alongside the task description.Requirements specify 1–20 uniquely indexed tiles, 1–10 non-overlapping obstacles, and an agent start at x=0.0, z=0.0.Source: Appendix A · pp. 31, 35
• The Unity arena randomizes 1–10 obstacles and tile values from 1 to 20, while the single agent must visit tiles in ascending order and avoid collisions.Observations include seven LiDAR rays, position and orientation, and tile features; actions provide discrete movement and rotation.Source: Appendix A · p. 31
• Incremental PPO curriculum settings list lesson values including 2, 4, 16, and 18, with reward thresholds specified for each shown lesson.The configuration also specifies 100-step minimum lesson lengths and reward-based completion criteria.Source: Appendix A · p. 33
• The task prompt requests four diverse edge-case testing scenarios in valid JSON, with exact coordinates, varied obstacle layouts and tile orders, and an explanation of each challenge to the agent.It also asks the model to vary the number and oneHotIDs of tiles while following the supplied YAML and TypeScript interface.Source: Appendix A · p. 36
• Evaluation outputs include success, timeout, wrong-tile, episode-length, regret, tile-coverage, reward, distribution, transition, diagnostic, and obstacle-collision measures.Each scenario is evaluated in Unity-3D over 50 repetitions.Source: Appendix A · pp. 31, 36