PaperlayerView exact revision on arXiv

Source-linked AI summary

Safe Actions Alone Do Not Ensure Safe Agents: Identifying Unfulfilled Obligations with Guard Models

Youwei Feng, Yitong Zhang, Yuetong Liu, Jia Li

arXiv:2610.11773v1cs.AI

TL;DR

Forbidden-action identification alone may miss safety-critical required actions left undone, and the paper reports no existing benchmark for evaluating obligation identification. The authors introduce ObligationBench, a 240-instance benchmark with 120 positive and 120 negative cases, reviewed by human experts to ensure accurate and complete ground truth. ObligationGuard achieves 57.52% recall and 21.67% exact match, surpassing all evaluated models on both metrics; existing models reach at most 48.97% recall and 10.00% exact match.

  • Problem

    Forbidden-action identification alone may miss safety-critical required actions left undone, and the paper reports no existing benchmark for evaluating obligation identification.

    Source: §1 · p. 1

  • Method

    The authors introduce ObligationBench, a 240-instance benchmark with 120 positive and 120 negative cases, reviewed by human experts to ensure accurate and complete ground truth.

    Source: §1 · p. 2

  • Results

    ObligationGuard achieves 57.52% recall and 21.67% exact match, surpassing all evaluated models on both metrics; existing models reach at most 48.97% recall and 10.00% exact match.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The authors call for obligation identification to be incorporated into the design and evaluation of future guard models to improve agent safety.

    Source: §8 · p. 18

    The choice of evaluated models may limit how broadly the findings apply, despite evaluation across 14 models and additional backbone fine-tuning.

    Source: §7.2 · p. 18

Abstract

from arXiv · show

Guard models are increasingly used to safeguard LLM-based agents, primarily by identifying actions that agents are forbidden to perform. However, identifying forbidden actions alone is insufficient to ensure agent safety. In this paper, we argue that agent safety also depends on identifying required yet unperformed safety-critical actions, which we call obligations. Our preliminary study on a popular benchmark for evaluating safety shows that 56.92% of GLM-5.3 trajectories contain unfulfilled obligations, compared with only 30.00% containing forbidden actions. This finding reveals unfulfilled obligations as a major and previously overlooked source of safety risk. However, to our knowledge, no existing benchmark evaluates whether guard models can identify these obligations. To close this gap, we introduce ObligationBench, the first benchmark for evaluating the capability of obligation identification, comprising 240 expert-validated trajectories covering issue resolution, feature development, and terminal operations. Our evaluation of 14 representative models reveals substantial limitations: the highest recall and exact-match rate are only 48.97% and 10.00%, respectively. To address these limitations, we develop ObligationGuard using 40,000 synthetic training examples. ObligationGuard achieves 57.52% recall and an exact-match rate of 21.67%, surpassing all evaluated models on both metrics. We call on the community to incorporate obligation identification into the design and evaluation of future guard models to improve agent safety.

1 Introduction

Identifying forbidden actions alone misses required but unperformed safety-critical actions, which the paper calls obligations. The authors introduce a benchmark and guard model to evaluate and address this gap.

1 Introduction

  • On SusVibes, 56.92% of GLM-5.3 trajectories contained unfulfilled obligations, compared with 30.00% containing forbidden actions.The paper argues that safety-critical actions may be required even when an agent performs no forbidden action.Source: §1 · p. 1
  • ObligationBench is the first benchmark for obligation identification, with 240 expert-reviewed trajectories spanning issue resolution, feature development, and terminal operations.It includes 120 positive and 120 negative cases, with varied agents, safety risks, and expert-corrected annotations.Source: §1 · p. 2
  • ObligationGuard uses a two-stage synthesis process that defines obligation sets in scenario templates before generating labeled trajectories.The authors identify costly trajectory collection and human annotation as barriers to scaling training data.Source: §1 · p. 2
  • Existing models reach at most 48.97% recall and 10.00% exact match, while performance worsens when obligations arise earlier or more coexist.The evaluation covers 14 representative models.Source: §1 · p. 2
  • 57.52% recall and a 21.67% exact-match rate let ObligationGuard surpass all evaluated models on both metrics.The model is trained on 40,000 synthetic examples generated through scenario planning and trajectory synthesis.Source: §1 · pp. 2–3

2 Related Work and Our Motivation

LLM agents act through iterative action–observation trajectories, while prior agent-safety work and guard models focus largely on harmful or forbidden behavior. The section motivates obligations by showing how missing safety-critical actions can leave a trajectory unsafe and reports their frequency on SusVibes.

2.1 LLM-based Agents

  • LLM agents iteratively choose actions using the task and trajectory, receive environment observations, and extend the trajectory until execution ends.Each interaction adds an action and its resulting observation.Source: §2.1 · p. 3

2.2 Agent Safety and Guard Models

  • Prior safety work examines harmful requests, malicious instructions, and coding vulnerabilities, while guard models monitor behavior and assess trajectory safety against tasks.Some guards also guide actions before execution.Source: §2.2 · pp. 3–4

2.3 Our Motivation

  • A persistent-login agent passed functional tests but omitted expired-session rejection, leaving an old cookie usable despite performing no forbidden action.The missing expiration check is an example of an obligation.Source: §2.3 · p. 4, Figure 2 · p. 4
  • The SusVibes study analyzed DeepSeek-V4.1-Flash and GLM-5.3 trajectories, labeling unsafe executions for forbidden actions, unfulfilled obligations, or both.The analysis examined executions that failed safety tests.Source: §2.3 · pp. 4–5
  • For GLM-5.3, unfulfilled obligations occur in 56.92% of executions versus 30.00% containing forbidden actions, showing the gap in forbidden-action-only monitoring.The rates are percentages of evaluated executions.Source: Table 1 · p. 5, §2.3 · p. 5
  • The findings motivate ObligationBench for evaluating obligation identification and ObligationGuard for improving that capability.The authors frame both as ways to help agents complete tasks more safely.Source: §2.3 · p. 5

3 ObligationBench

ObligationBench evaluates guard models’ identification of unfulfilled obligations using expert-validated agent trajectories. It spans multiple task domains and safety categories, and its construction combines trajectory generation, annotation, and human review.

3 ObligationBench

  • ObligationBench is the first benchmark designed to evaluate whether guard models identify agents’ unfulfilled obligations.It provides annotations of obligations remaining unfulfilled at termination.Source: §3 · p. 5

3.2 Benchmark Statistics

  • 240 expert-validated instances contain 339 unfulfilled obligations, averaging 2.83 obligations per positive instance.The benchmark is evenly split between 120 positive and 120 negative cases.Source: Figure 3 · p. 6, §3.2 · p. 6

3.1 Benchmark Design

  • Positive instances contain at least one unfulfilled obligation, while paired negative instances contain none, testing identification and avoidance of false alarms.Unnecessary interventions in obligation-free cases may reduce agent utility.Source: §3.1 · p. 6
  • The benchmark covers issue resolution, feature development, and terminal operation using tasks from three established benchmarks.These domains represent repository-level issue resolution, feature implementation, and open-ended terminal interactions.Source: §3.1 · p. 6

3.2 Benchmark Statistics

  • The 339 obligations span five safety categories, with no single category dominating the benchmark.Data Exposure accounts for 93 obligations (27.43%); Asset Tampering for 86 (25.37%); Identity Forgery for 88 (25.96%); Unauthorized Access for 53 (15.63%); and Others for 19 (5.60%).Source: §3.2 · pp. 6–7

3.3 Benchmark Construction

  • Benchmark construction collects real-task agent trajectories, filters for potential obligations, annotates them, and subjects annotations to human review.Two experienced software engineers independently review and correct annotations; disagreements are discussed and ambiguous cases discarded.Source: Figure 4 · p. 7, §3.3 · pp. 7–8

3.2 Benchmark Statistics

  • 80% of positive instances contain multiple obligations, testing whether models recover complete sets of coexisting obligations.Positive instances range from one to five obligations.Source: §3.2 · p. 7

3.3 Benchmark Construction

  • For each retained positive task, the authors obtain a matched negative by rerunning it until a trajectory has no unfulfilled obligations.This produces 240 expert-validated instances: 120 positive cases and 120 hard negatives.Source: §3.3 · p. 8

4 ObligationGuard

ObligationGuard is trained on synthetic task-and-trajectory examples because real trajectories and complete obligation annotations are costly to obtain. The authors plan scenarios and obligation sets, synthesize consistent trajectories, then fine-tune a language model.

4.1 Data Synthesis

  • The authors use a two-stage synthesis approach: plan tasks with ground-truth obligation sets, then generate trajectories consistent with those plans.This addresses the cost and scalability challenges of collecting and annotating real executions.Source: §4.1 · p. 8
  • Scenario planning produces 10,000 templates recording each task, its scenario and safety requirements, and its ground-truth obligation set.An empty planned set means no obligations should remain unfulfilled at termination.Source: §4.1 · p. 8, Figure 5 · p. 9
  • Trajectory synthesis retains 40,000 examples whose generated trajectories leave exactly the planned obligations unfulfilled at termination.The planned obligation sets serve as ground-truth labels, and incoherent generations are rejected.Source: §4.1 · p. 8

4.2 Model Training

  • ObligationGuard is produced by full-parameter supervised fine-tuning of Qwen3-8B on the synthetic trajectories, using each task and complete trajectory as input.The target is the ground-truth obligation set for that task and trajectory.Source: §4.2 · p. 8

5 Experimental Setup

The experiments assess existing and proposed models’ obligation identification, factors affecting performance, ObligationGuard’s design choices, and effects on agent safety. The setup compares 14 models, reports several complementary metrics, and specifies evaluation and training procedures.

5.1 Research Questions

  • The experiments address four questions: existing-model capability, performance factors, ObligationGuard’s effectiveness, and whether stronger identification improves agent safety.The safety question is evaluated by integrating guards into a coding agent and measuring functional correctness and safety on SusVibes.Source: §5.1 · p. 9

5.2 Evaluated Models

  • The evaluation compares ObligationGuard with 14 existing models, including general-purpose LLMs and guard models across model families and sizes.Qwen3-8B is included as ObligationGuard’s backbone.Source: §5.2 · p. 10

5.3 Metrics

  • The evaluation reports precision, recall, and exact match on positive instances, plus classification accuracy separately for positive and negative instances.Exact match requires the predicted obligation set to be accurate and complete; classification accuracy tests whether any obligations remain.Source: §5.3 · p. 10

5.4 Implementation Details

  • Models are run once per instance with the same prompt and deterministic decoding when supported; ObligationGuard is trained for two epochs with AdamW.Training uses a learning rate of 1 × 10^-5 and batch size 32; training and evaluation contexts have a 32,768-token limit.Source: §5.4 · p. 11

6 Experimental Results

ObligationGuard improves obligation identification on ObligationBench and raises secure task completion on SusVibes while largely preserving functional correctness. The experiments also show where identification degrades and how synthetic-data design affects performance.

6.1 RQ1: How effectively can existing models identify obligations?

  • 57.52% Recall and 21.67% EM let ObligationGuard surpass all evaluated models, exceeding the best existing Recall by 8.55 percentage points and more than doubling the best existing EM.Existing models’ best Recall and EM are 48.97% and 10.00%; the three existing guard models reach at most 5.60% Recall and 0.83% EM.Source: §6.1 · p. 11, Table 2 · p. 12

6.2 RQ2: What factors affect models’ performance in obligation identification?

  • As post-creation actions increase from at most four to more than sixteen, Recall falls from 42.82% to 25.48%; as coexisting obligations rise from one to five, Recall falls from 58.33% to 23.56%.EM also declines across both factors: 12.41% to 3.87% with greater action distance, and 30.83% to 0.00% with more coexisting obligations.Source: Figure 6 · p. 12, §6.2 · p. 13

6.3 RQ3: What contributes to the effectiveness of ObligationGuard?

  • Fine-tuning benefits other model sizes and a second family: Qwen3-4B Recall rises from 10.91% to 43.66%, while Llama-3.1-8B-Instruct Recall rises from 0.59% to 49.85%.Fine-tuned Qwen3-4B exceeds the original Qwen3-14B on all three metrics, and fine-tuned Llama-3.1-8B-Instruct exceeds the best existing Recall and EM from RQ1.Source: §6.3 · p. 14
  • Increasing synthetic training data from 5K to 40K raises Precision from 70.06% to 82.98%, Recall from 32.45% to 57.52%, and EM from 5.00% to 21.67%; templates further raise Qwen3-8B to 82.98% Precision, 57.52% Recall, and 21.67% EM.With the same generator, data size, and fine-tuning settings, template-guided synthesis more than doubles EM versus direct synthesis by specifying requirements and obligations before trajectory generation.Source: §6.3 · pp. 14–15, Figure 7 · p. 14, Figure 8 · p. 15

6.4 RQ4: Does stronger obligation identification improve agent safety?

  • 15.1% SecPass with ObligationGuard versus 6.5% without guidance, while FuncPass changes only from 27.4% to 26.9%.The strongest existing guard model in this experiment reaches 12.4% SecPass; ObligationGuard guidance chiefly converts initially functional but unsafe solutions into safe ones.Source: Table 4 · p. 16, §6.4 · p. 16, Table 5 · p. 16
  • Recall on ObligationBench is strongly associated with SecPass improvement on SusVibes across six guard models (Spearman’s ρ=0.94; Pearson’s r=0.97).ObligationGuard’s guidance also addressed a persistent-login failure by prompting server-side rejection of expired sessions.Source: §6.4 · pp. 16–17, Figure 9 · p. 17

7 Discussion

Manual review supports the reliability of the LLM’s semantic-matching judgments on a balanced sample. The discussion also identifies limits involving annotations, benchmark coverage, practical relevance, and generalizability.

7.1 Reliability of LLM Judgment

  • 95.0% overall agreement was found in a balanced manual review of 400 obligation pairs judged matched or unmatched by the LLM.Reviewers confirmed 98.0% of matched judgments and 92.0% of unmatched judgments.Source: §7.1 · p. 17

7.2 Threats to Validity

  • The authors identify threats from annotation reliability, benchmark diversity and realism, practical relevance, and the choice of evaluated models.Mitigations include expert review and discarding ambiguous cases, real execution trajectories across selected domains, SusVibes guidance tests, and evaluation across multiple model families and sizes.Source: §7.2 · pp. 17–18

8 Conclusion

The paper concludes that identifying forbidden actions alone is insufficient for agent safety because unfulfilled obligations are a major safety risk. It introduces ObligationBench and ObligationGuard, reporting improved obligation identification and safer task completion while largely preserving functional completion.

8 Conclusion

  • ObligationGuard improves obligation identification and helps agents complete tasks more safely while largely preserving functional completion.The paper encourages incorporating obligation identification into future guard-model design and evaluation.Source: §8 · p. 18
  • The paper introduces ObligationBench, the first benchmark for obligation identification.Source: §8 · p. 18
  • The evaluation finds that 14 existing models struggle to identify accurate and complete obligation sets.Source: §8 · p. 18
  • The paper develops ObligationGuard using synthetic training data generated through scenario planning and trajectory synthesis.Source: §8 · p. 18
  • The paper argues that unfulfilled obligations are a major source of safety risk, so identifying forbidden actions alone is insufficient.Source: §8 · p. 18
Loading 2610.11773v1…