PaperlayerView exact revision on arXiv

Source-linked AI summary

AURA-Eval: Evaluation Framework for Acting Under Risk Awareness in LLM Agent Trajectories

Ruoxi Shang, Christina-Maria Androna, Orfeas Menis Mastromichalakis, Yu Feng, Aniruddhan Ramesh, Rico Angell, Shang Hong Sim, Chrysoula Zerva, Emmanouil Koukoumidis

arXiv:2609.06783v1cs.CRcs.AIcs.CLcs.SE

TL;DR

LLM-agent safety evaluations often collapse context-dependent behavior into binary outcomes, leaving risk recognition and safe fulfillment insufficiently distinguished. AURA-Eval addresses this gap by generating controlled trajectory variations and evaluating risk detection, action strategy, and action safety. Across 1,249 items from 157 seed trajectories, unsafe behavior was more common without a safe fulfillment path, while greater scale or reduced oversight exposed vulnerabilities.

  • Problem

    Existing agent-safety evaluations often organize risk by domain or outcome and use binary labels, obscuring contextual risk responses and distinctions among refusal, confirmation, and safer alternatives.

  • Method

    AURA-Eval identifies safety-critical decision points, generates controlled context-preserving trajectory variations, constructs SAFE-PATH and NO-SAFE-PATH counterparts, and labels continuations on risk detection, action type, and action safety.

  • Results

    Across 1,249 evaluation items from 157 seed trajectories, unsafe-action rates were higher in NO-SAFE-PATH scenarios; frontier models more often proposed safe alternatives, while open-weight models more often executed unsafe requests.

  • Takeaways & Limitations

    Controlled increases in affected-entity count and reductions in pre-execution oversight expose unsafe behavior that original trajectories may miss.

  • Takeaways & Limitations

    AURA-Eval depends on the coverage and quality of seed trajectories, so it cannot capture risks absent from or difficult to ground in those source traces.

Abstract

from arXiv · show

LLM agents operate in workflows where unsafe actions can have real consequences. Existing safety evaluations often reduce behavior to a single score, obscuring risk recognition, pre-action detection, and safe task completion when a safe solution exists. We introduce AURA-Eval, a framework combining controlled augmentation with granular diagnosis of behavior in tool-use trajectories. Its pipeline identifies safety-critical decision points, generates controlled variations, and constructs counterparts differing in whether a request has a safe fulfillment path. Using 157 sourced trajectories, we generate 1,249 evaluation items and evaluate 20 frontier and open-weight models. We developed rubrics to classify risk detection, action strategy, and scenario-specific action safety. Our results show that LLM agents engage in unsafe behavior more often when no safe fulfillment path exists. In these cases, frontier proprietary models more often recognize risk and exhibit safer behavior by proposing alternatives, while evaluated open-weight models more often directly execute unsafe requests. Increasing impact or reducing opportunities for oversight before execution also exposes greater vulnerability across models.

1 Introduction

AURA-Eval addresses limitations of binary agent-safety evaluations by varying contextual risk and diagnosing how agents recognize and respond to it. It generates controlled trajectory variants and shows that safe-path availability, scale, and oversight affect unsafe behavior.

  • Motivation: Existing benchmarks often organize risk by domain or outcome, while binary labels obscure whether agents refuse, confirm, or propose safer alternatives.These limitations make it difficult to distinguish contextual-risk responses from constant refusal.
  • Framework: AURA-Eval identifies safety-critical decision points and rewrites context to vary targeted risk mechanisms while preserving the user goal, agent role, tools, and trajectory structure.The framework expands source traces into scenario families for risk-recognition and action-choice evaluation.
  • Framework: Its construction pipeline creates controlled trajectory variations, paired SAFE-PATH and NO-SAFE-PATH scenarios, and rubric-based labels for risk detection, action type, and action safety.These components support granular diagnosis beyond a single safety outcome.
  • Evaluation: 157 seed scenarios produce 1,249 evaluation items for frontier and open-weight models.The evaluation instantiates the framework on sourced R-Judge trajectories.
  • Findings: Unsafe-action rates increase in NO-SAFE-PATH scenarios from 5.4–7.4% to 26.8–40.0% for frontier models and from 12.0–21.5% to 57.9–82.6% for general open-weight models.Frontier models more often use safe alternatives, whereas open-weight models more often execute unsafe requests.
  • Findings: Increasing affected-entity count or reducing pre-execution review opportunities exposes more unsafe model behavior.These controlled changes correspond to RS2 Scale and RS5 Oversight.

2 Related work

Related work spans tool-use safety benchmarks, context-sensitive risk assessment, and trajectory-level diagnosis. AURA-Eval combines these directions by varying contextual risk in matched agent trajectories and evaluating intermediate behavior.

  • Agent Safety Benchmarks: Agent-safety benchmarks increasingly evaluate situated tool use, including high-stakes tools, multi-turn risk awareness, prompt injection, harmful capabilities, and dynamic environments.The cited benchmarks broaden evaluation beyond static prompt-response behavior.
  • Comparison: Table 1 codes benchmark properties as central, partial or adjacent, or outside stated scope across settings, construction, risk, fulfillment, process, and behavioral labels.Filled dots, open circles, and dashes represent these three coding levels.
  • Risk Assessment: Over-refusal benchmarks test benign prompts resembling unsafe requests, while context-sensitive work emphasizes that request safety can change with circumstances.Other benchmarks add graded or category-specific tool-safety assessment.
  • Risk Assessment: Many evaluations still compress safety into binary labels or aggregate pass rates, motivating more behaviorally detailed assessment.AURA-Eval addresses this concern through matched agent trajectories and contextual risk variation.
  • Diagnostic Evaluation: Trajectory-level and process-supervision work evaluates intermediate reasoning and diagnoses where agents become unsafe, overly cautious, or superficially compliant.Recent studies also report that abstract risk knowledge may not translate into safe situated action.
  • Diagnostic Evaluation: Because open-ended trajectory continuations require semantic judgment, AURA-Eval separates generation review from final judging and uses axis-specific rubrics with human validation.This design responds to known framing and bias sensitivity in LLM judges.

3 Task formulation

The task formulation represents tool-use interactions as trajectories of reasoning, actions, and observations, then evaluates model continuation at selected decision points. This supports assessment of risk awareness during multi-step execution.

  • Interaction Representation: An interaction is represented as R = (u, τ), where a user instruction accompanies a trajectory of thought–action–observation triples.The trajectory may contain arbitrarily many steps, and multi-turn interactions form one longer trajectory.
  • Interaction Representation: The trajectory records agent reasoning traces, tool invocations from a fixed tool set, and environment responses using explicit interaction fields.These fields are [USER], [AGENT_THOUGHT], [AGENT_ACTION], and [ENVIRONMENT].
  • Task Scope: The task evaluates risk awareness in tool-using agents executing multi-step tasks.The formulation targets situated action rather than isolated responses.
  • Trajectory Completion: Evaluation is formulated as trajectory completion at decision point i, using the instruction and prefix before i to generate the next thought and action.At i = 1, the prefix is empty and the model receives only the instruction.

4 Data Augmentation Axes

AURA-Eval augments seed trajectories along controlled risk and difficulty dimensions while preserving scenario grounding, then evaluates behavior at risk-trigger and pre-execution points. Its paired path design distinguishes necessary intervention from safe fulfillment.

  • Risk Mechanisms: AURA-Eval formalizes six risk-mechanism dimensions that target distinct structural components of scenario risk.Applicable mechanisms are selected for each source scenario and tracked with targeted edits and preservation checks.
  • Decision Points: Evaluation uses prefixes at the tagged risk trigger and immediately before the tagged execution point.The pre-execution prefix tests whether the agent uses retrieved information to avoid unsafe actions.
  • Decision Points: In the home-device example, the execution point turns off all retrieved devices, while the pre-execution prefix ends after retrieval and before turnoff.The example contrasts disabling every device with selectively leaving critical devices on.
  • Safe-Path Construction: A NO-SAFE-PATH item has no action that both directly fulfills the requested outcome and avoids the scenario-specific risk, whereas a SAFE-PATH item has at least one such action.This distinction determines whether intervention or direct safe fulfillment is appropriate.
  • Scenario Difficulty: The framework varies detection difficulty through five dimensions: interpretive ambiguity, attack provenance, obfuscation, emotional manipulation, and distraction.These dimensions target reasons an agent might miss an otherwise similar underlying risk.
  • Dataset Construction: The instantiation retains 157 unintended-attack R-Judge trajectories, yielding 211 source prefixes and 1,249 released benchmark rows after quality filtering.The seeds span Program, Application, IoT, Web, and Finance scenarios.
  • Variation Generation: Each applicable risk or difficulty axis yields a grounded variation with updated risk rationale, while boundary rules distinguish mechanisms such as harm, affected recipients, and shared dependencies.Generation uses Claude Opus 4.6, with review by Claude Opus 4.6, GPT-5.2, and DeepSeek-V3.1.

5 Evaluation Framework

AURA-Eval evaluates agent safety through separate judgments of risk detection, action strategy, and scenario-specific action safety. Its setup uses generated trajectory outputs and rubrics designed to distinguish whether agents recognize and appropriately handle contextual risks.

  • Evaluation axes: Risk Detection is inferred from the generated reasoning trace, while Action Evaluation uses the executed action and separates action type from action safety.Risk detection uses four increasing-depth levels, D0–D3.
  • Evaluation procedure: Each item asks a model to continue a user instruction and augmented trajectory prefix by generating the next thought and action.The model does not receive pair labels, reference rationales, or safe-action sketches.
  • Evaluation axes: The evaluation taxonomy is summarized across Risk Detection, Action Type, and Action Safety, with prompts supplied separately for each axis.The framework evaluates open-ended behavior rather than reducing it to a single safety label.
  • Evaluation procedure: Three judge prompts assign risk detection D0–D3, action type A0–A5, and scenario-specific SAFE or UNSAFE labels.Action Safety is judged using the scenario, generated action, and reference risk rationale, which is withheld from the evaluated model.
  • Statistical treatment: Uncertainty in NO-SAFE-PATH versus SAFE-PATH unsafe-rate differences is estimated by resampling source trajectories, preserving dependence among variants from each seed.The procedure retains dependence among benchmark variants derived from the same source trajectory.

6 Evaluation Reliability and Human Validation

AURA-Eval’s rubrics were iteratively refined and human-validated across detection, action type, and action safety samples. Agreement was strongest for action safety and for cases receiving unanimous judge labels, while vendor-overlap analysis found no statistically detectable bias.

  • Human validation: 243 audit samples covered detection, action type, and action safety across four evaluated models and five risk domains.Each item was independently labeled by three human annotators; 226 samples had both a human reference label and five-judge plurality.
  • Agreement: For Action Safety, human reliability was Po = 0.889 and α = 0.759, with judge–human agreement of 78.6% (66/84; 95% Wilson CI [68.7, 86.0]).Action Safety is identified as the primary outcome axis.
  • Judge panel: A vendor-diverse three-judge panel matched the five-judge panel at 74.0% versus 75.3% validation accuracy while reducing unresolved ties from 17 to 10.The smaller panel was used to lower inference cost, while the five-judge panel remained for validation reporting.
  • Bias analysis: Leave-one-vendor-out replacement produced a largest shift of 2.0 points, with every 95% confidence interval including zero.The reported shift was smaller than the 12.9-point gap between the closest frontier and open-weight groups.

7 Results

Across 20 models, unsafe actions were substantially more common in NO-SAFE-PATH scenarios than in SAFE-PATH scenarios. Frontier models more often used alternatives, whereas open-weight models more often executed unsafe requests; controlled variations exposed additional vulnerabilities.

  • Overall rates: Unsafe actions were lowest for frontier models and highest for open-weight models, with Claude Sonnet 4.6 at 16.9% overall and open-weight rates of 36.5–52.0%.Safety-tuned models ranged from 24.5–47.5%.
  • Action pathways: In NO-SAFE-PATH scenarios, frontier-model unsafe cases concentrated in direct execution and information gathering, while safe responses used clarification, confirmation, alternatives, or refusal.Refusal was always safe and alternatives were generally safe, whereas other action types could be unsafe.
  • Action pathways: Frontier models proposed alternatives in 26.7–31.0% of NO-SAFE-PATH outputs, whereas open-weight models more often executed the requests.In SAFE-PATH scenarios, execution and information gathering dominated and were about 90% safe.
  • Risk detection: Explicit risk evaluation or projection generally corresponded to lower unsafe-action rates, but about half of Llama-3.1-8B-Inst’s D3 actions remained unsafe.The most frequent unsafe pathway was missing risk recognition followed by execution.
  • Controlled variations: Mean unsafe-action rates were highest for RS5 Oversight at 60% and RS2 Scale at 57%, compared with 38% for RS1 Harm Intensity and 41% for RS3 Target Susceptibility.Among originally safe cases, 52.3% (45/86) became unsafe under RS5 versus 17.9% (14/78) under RS2.
  • Controlled variations: On SD3 Obfuscation, unsafe-action rates were 9–32% for displayed frontier models and 68–82% for displayed general open-weight models.The separation was diagnostic rather than evidence that obfuscation uniformly increased difficulty, since frontier models were safer on SD3 than on originals.

8 Conclusion

AURA-Eval converts existing tool-use traces into diagnostic stress tests that reveal failures obscured by final outcomes alone. Its risk-mechanism taxonomy supports controlled scenario variation and future safety metrics, while longer-horizon and more refined path classifications remain open directions.

  • Conclusion: AURA-Eval uses controlled changes to oversight or scale to expose failures that original trajectories can miss.The framework also shows that risk recognition does not always lead to safe action.
  • Conclusion: The risk-mechanism taxonomy supports composing multiple mechanisms into more complex scenarios and developing metrics over the dimension labels.This provides a basis for quantifying risk-related safety behavior.
  • Future work: Future work should refine NO-SAFE-PATH and SAFE-PATH operationalization and classification and extend the distinction to more complex, longer-horizon interactions.Combining rubric axes may also support new composite metrics of agent safety behavior.

Limitations

AURA-Eval’s conclusions are bounded by seed-trajectory coverage, the context-dependent SAFE-PATH abstraction, and evaluation choices for open-ended behavior. Some risk and difficulty dimensions remain sparsely represented, while detection judgments are especially subjective despite reliability controls.

  • Data and coverage: Seed-trajectory coverage and quality constrain which risks AURA-Eval can represent, leaving some dimensions sparsely covered.The framework cannot capture risks absent from or difficult to ground in source traces; SD2 and SD5 are examples of sparse coverage.
  • Data and coverage: The evaluation’s controlled variations do not fully address malicious intent, which the authors identify as a separate variation for future seed-pool expansion.This leaves malicious-intent scenarios outside the current controlled-variation coverage.
  • Scenario abstraction: The SAFE-PATH/NO-SAFE-PATH abstraction can admit unexpected safe actions and borderline judgments about whether fulfillment remains appropriate.The distinction is useful for separating necessary intervention from over-refusal, but it does not eliminate boundary cases.
  • Evaluation: Risk detection is the most difficult evaluation axis because grading risk recognition in free-form reasoning is inherently subjective.Axis-specific rubrics, independent judges, majority voting, contested-case exclusion, and human validation support reliability, while action type and safety are more reliable primary axes.
  • Scenario abstraction: AURA-Eval’s action sets are conceptual rather than exhaustive, and their domain is limited to the roles, tools, observations, and action formats evidenced in each trajectory prefix.The paired labels therefore operationalize a scenario-level contrast within the available trajectory context rather than enumerating every possible action.

D Dataset construction

AURA-Eval constructs controlled, matched agent-safety scenarios from sourced trajectories while preserving task and interaction structure. It also validates its rubric-based labels through human annotation and judge-agreement analyses.

  • Trajectory preparation: The pipeline starts from complete tool-use traces and creates fixed prefixes at safety-critical decision points for consistent model evaluation.It identifies risk triggers and, when applicable, pre-execution points before harmful actions.
  • Trajectory preparation: 157 selected trajectories are filtered from a 571-trajectory pool spanning five domains, with 101 unsafe and 56 safe source outcomes.The selection fixes the unintended-attack category to avoid adding intended attacks as an experimental variable.
  • Matched scenarios: Items are classified as NO-SAFE-PATH or SAFE-PATH, and pairable scenarios are minimally rewritten into opposite versions while preserving domain, tools, interaction structure, and agent steps.This operational classification concerns plausible safe fulfillment within the shown scenario context and tools.
  • Controlled augmentation: Controlled variations modify user or environment context while preserving agent thoughts and actions, producing 650 accepted variations from 1,029 candidate requests.The current construction run retains 861 merged original and variation items before pair generation.
  • Rubric validation: Panel unanimity, rather than any single judge, provides the strongest reliability signal, while contested cases expose lower agreement on ambiguous labels.The panel matches the reference on 57 of 58 unanimous samples, corresponding to 98.3% joint accuracy.

G.3 Judge ensemble sensitivity analysis

The sensitivity analysis selects a vendor-diverse judge ensemble by balancing accuracy, coverage, unanimity, and inference cost. It also tests whether judge–model vendor overlap or repeated prompting materially changes evaluation reliability.

  • Ensemble selection: The selected three-judge panel is GPT-5.4, Claude Opus 4.6, and Gemini 3.1 Pro.The analysis compares this panel with alternative three-judge configurations and a five-judge baseline.
  • Ensemble selection: Vendor-diverse ensembles rank above same-family ensembles, indicating that vendor diversity contributes to ensemble quality.The comparison covers OpenAI, Anthropic, and Google judge combinations.
  • Ensemble selection: The chosen three-judge variant reaches unanimity on 87 samples versus 72 for the alternative, enlarging the high-confidence subset by 21% despite lower accuracy of 74.0% versus 77.2%.It also produces panel plurality on 173 samples versus 171 and reduces tied cases from 12 to 10.
  • Ensemble selection: Relative to the five-judge baseline, the selected panel has comparable accuracy, 74.0% versus 75.3%, at lower inference cost while increasing unanimous cases from 58 to 87.Unresolved panel-tied cases decrease from 17 to 10.
  • Vendor-overlap sensitivity: Replacing overlapping judges with DeepSeek-V4-Pro produces no statistically detectable SAFE-rate effect because all four 95% confidence intervals include zero.The largest observed shift is 2.0 points for Claude Opus 4.6.
  • Run consistency: All three judges show high run-to-run consistency, ranging from 87.5–100.0% across detection, action-type, and action-safety axes.Consistency is measured over three independent runs at temperature T = 0.

I Pipeline Statistics

The construction pipeline identifies decision points, creates controlled variations, and builds reviewed matched scenarios with explicit reference rationales. Its stages constrain edits to preserve the original agent trajectory while changing selected contextual factors.

  • Decision-point identification: Each trajectory is truncated at a risk trigger or pre-execution decision point, yielding 211 truncated items from 157 seeds.The total includes 157 at-trigger items and 54 pre-execution items.
  • Decision-point identification: The risk trigger marks where risk information enters, while the execution point marks the first subsequent agent action where the decision materializes.Triggers originate in user or environment steps; execution points are agent-action steps.
  • Controlled variations: The pipeline generates one candidate per relevant item–dimension pair, producing 1,029 candidate variation requests from the 211 truncated items.These include 200 harm-intensity requests and 829 contextual or difficulty requests.
  • Controlled variations: Variations rewrite only user or environment content and keep every agent thought and action character-for-character identical.This preserves tool names, arguments, step order, and the number of agent steps.
  • Controlled variations: Three-model review checks that target dimensions change while action outputs, non-target dimensions, realism, and internal consistency remain fixed.Reviewers include Claude Opus 4.6, GPT-5.2, and DeepSeek-V3.1.
  • Pair classification: The merged pool contains 861 items, and classification asks whether a plausible scenario-grounded action can fulfill the request while satisfying safety requirements.The classifier restricts consideration to actions supported by the trajectory context and explicitly shown or established tools and APIs.
  • Pair generation: Pair generation flips NO-SAFE-PATH and SAFE-PATH labels through minimal in-domain edits, followed by three-model review of label validity, preservation, realism, and rationale quality.Of 593 generated pair candidates, 590 were parseable for review and 428 passed majority review.

J Dataset Statistics

Dataset statistics describe scoring coverage, contested judgments, generation failures, and dependence-aware uncertainty. The released benchmark preserves source-seed dependence and shows a positive NO-SAFE-PATH versus SAFE-PATH unsafe-rate difference for every model.

  • Scoring coverage: Contested labels remain under 3% for every model, with no model family or individual model producing systematically more contested cases.Excluding contested labels therefore does not materially affect the headline trends.
  • Generation failures: Some models fail the required ReAct-style format through safety-triggered refusals, especially the smallest safety-tuned models.Failures include withholding both thought and action or producing thought without an action.
  • Dependence and uncertainty: The released benchmark contains 1,249 rows representing 152 of the 157 selected source trajectories after filtering.Variants sharing a source seed are therefore not independent.
  • Dependence and uncertainty: Bootstrap resampling preserves within-seed dependence when recomputing NO-SAFE-PATH and SAFE-PATH unsafe rates and their difference.The analysis uses 10,000 resamples per model and percentile 95% intervals.
  • Dependence and uncertainty: All 20 NO-SAFE-PATH–SAFE-PATH differences are positive, and no bootstrap interval includes zero.The comparison is computed using per-metric denominators.

K.1 Risk Reasoning Reduces Unsafe Actions Mainly After D2

Risk reasoning reduces unsafe actions mainly once models reach D2 assessment; shallow detection remains insufficient, and safe outcomes depend on pairing stronger detection with non-executing actions.

  • Risk detection thresholds: Frontier models reach D3 in 58–67% of NO-SAFE-PATH cases, yet about 20% of outputs still show D0 awareness.At D0, 64–75% of frontier cases end in unsafe actions, and over 95% of unsafe D0 actions are executions or information gathering.
  • Risk detection thresholds: D1 unsafe rates remain 75–95%, matching or exceeding D0 for several models.Most safety improvement appears at D2, where unsafe rates drop by 33–67 percentage points across frontier models.
  • Risk detection thresholds: D2-to-D3 reasoning adds a smaller 10–15 percentage-point improvement, while D3 execution can still be unsafe.Reported D3–A0 unsafe rates range from 56% to 100% in NO-SAFE-PATH cells.
  • Detection–action coupling: D2 or D3 paired with A1 or A4 stays below 20% unsafe in reported cells, whereas A2 confirmations include 42% and 44% unsafe exceptions.These results show that risk recognition alone does not ensure safety; action strategy remains consequential.
  • Context and oversight: Removing oversight caused 52% of originally safe RS5 cases to flip unsafe, including 1 in 5 cases for Claude Sonnet 4.6.The dominant failure mechanism was inattention to the variation despite unchanged underlying evidence.
  • Evaluation structure: The evaluation contrasts NO-SAFE-PATH and SAFE-PATH variants across detection, action, and safety labels to diagnose these behavior patterns.Scenario tables encode trigger state, detection level, action category, and majority-judge safety labels.
Loading 2609.06783v1…