PaperlayerView exact revision on arXiv

Source-linked AI summary

Positive-Unlabeled Learning for Agent Safety False Alarm Auditing

Xichen Yan, Chongyang Gao, Kezhen Chen, Guangyi Zhang, Jiaqi Wu, Lixu Wang

arXiv:2610.02925v1cs.AI

TL;DR

Conservative safety monitors generate many false alarms, while native scores overlap and verified-safe non-alarms may poorly represent the safe alarms needing review. The paper proposes a two-stage PU ranking framework to prioritize alarms, improving results over PU baselines and recovering more false alarms than PULDA at a 5% review budget.

  • Problem

    Conservative monitors produce many safe alarms, while native scores overlap and verified-safe non-alarms may poorly represent safe alarms needing review.

    Source: §1 · p. 2

  • Method

    A two-stage PU ranking framework adapts verified-safe supervision toward alarms, distills consistent orderings across PU classifiers, and refines rankings using reference support and alarm relations.

    Source: §1 · p. 3

  • Results

    A macro AUPRC of 0.6444 outperforms eight evaluated PU baselines by 5.27–16.98 percentage points, and 33.3% more false alarms than PULDA are recovered at a 5% auditing budget.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The ranking prioritizes human review without certifying alarms safe or changing the original monitor decisions.

    Source: ETHICS STATEMENT · p. 10

    During fitting, safety labels are observed only for verified-safe non-alarms; labels and false-alarm proportions for the alarm cohorts are unknown.

    Source: §2 · p. 4

Abstract

from arXiv · show

Safety monitors help safeguard language-model agents interacting with external tools and environments, but conservative monitoring can generate many false alarms, consuming extensive review resources and weakening trust in alerts. Because false and genuine alarms often remain interleaved in native monitor scores, obtaining a reliable cutoff still requires substantial manual verification. In practice, a small set of verified-safe non-alarmed trajectories may be available while alarms remain unlabeled, naturally casting false-alarm auditing as a positive-unlabeled (PU) ranking problem. The key challenge is monitor-induced selection, since observed safe references are accepted by the monitor, while the hidden safe alarms of interest are precisely those it incorrectly flags, making the observed positives poorly representative of the positives to be recovered. To address this challenge, we propose a two-stage framework in which Trust-aware PU Supervision adapts safe references toward the alarm domain and protects plausible false alarms from excessive negative pressure, while Reliability-gated Rank Distillation consolidates consistent ordering preferences from multiple PU reference models into a single student. Consensus-guided Structural Refinement then improves the student ranking using hierarchical safe-reference support, alarm relations, and predicted reference consensus. The framework requires no alarm safety labels for fitting and leaves the underlying monitor unchanged. Across mainstream safety monitors, our method achieves a macro AUPRC of $0.6444$, outperforming eight evaluated PU baselines by 5.27--16.98 absolute percentage points; compared with PULDA, the strongest evaluated PU baseline, it recovers 33.3% more false alarms at a 5% review budget.

1 INTRODUCTION

Agent safety monitors assess trajectories from tool-using interactions, but conservative monitoring produces many false alarms that overlap genuine alarms in native score order. The paper formulates false-alarm auditing as PU ranking under monitor-induced selection and proposes a two-stage framework to prioritize alarms for review.

1 INTRODUCTION

  • Language-model agents use external tools and environments, and their execution trajectories provide the basis for monitors that detect unsafe behavior.Trajectories record intermediate actions, observations, and tool interactions.Source: §1 · p. 1
  • Across mainstream monitors, 39.4%–57.9% of triggered alarms correspond to safe trajectories, reflecting the cost of sensitivity-oriented monitoring.The paper notes that false warnings can erode trust and require human and computational effort to investigate.Source: §1 · p. 2
  • False and true alarms overlap in native monitor-score order, so moving a cutoff cannot reliably separate them under a limited review budget.Figure 1 shows the interleaving across four dataset–monitor settings; all 510 alarms are shown.Source: §1 · p. 2, Figure 1 · p. 2
  • Verified-safe non-alarms provide observed positives, but hidden safe alarms are monitor-rejected positives, so the reference set may not represent the alarms to recover.Directly learning from the observed positives risks reproducing the monitor’s existing decision pattern.Source: §1 · p. 2
  • The proposed two-stage PU ranking framework adapts safe references toward the alarm pool, distills consistent orderings from multiple PU classifiers, and refines the queue using reference support and alarm relations.The refinement also uses reference agreement.Source: §1 · p. 3
  • A macro AUPRC of 0.6444 outperforms eight evaluated PU baselines by 5.27–16.98 percentage points, and the method recovers 33.3% more false alarms than PULDA at a 5% auditing budget.These results are reported across mainstream safety monitors.Source: §1 · p. 3

2 PRELIMINARIES

The preliminaries position false-alarm auditing as a complementary task to safety judgment and define its data and review objective. The setup uses verified-safe non-alarms as observed positives while alarm labels remain unknown during fitting.

2 PRELIMINARIES

  • Prior safety-monitoring work focuses on safety judgments and interventions, whereas this paper prioritizes likely false alarms for human review after alarms are raised.Source: §2 · p. 3
  • The setting connects PU methods for risk estimation, distribution alignment, and supervision to monitor-induced selection, where safe references and hidden safe alarms are selected by opposite monitor decisions.The paper situates its approach alongside work on nonrandom positive selection and PU AUC optimization.Source: §2 · p. 3
  • The observed data comprise verified-safe non-alarmed trajectories, unlabeled training alarms, and a separate target alarm cohort, with frozen representations, task features, and isolation-group identifiers available.Source: §2 · p. 4
  • Safety is the positive class; only the verified-safe non-alarmed set has observed safety labels during fitting, while alarm labels and false-alarm proportions are unknown.Source: §2 · p. 4
  • The auditor orders target alarms for human review to maximize false alarms within budget B, using target features but reserving target safety labels for evaluation after the order is fixed.The ordering sets review priority while preserving all original alarm decisions.Source: §2 · p. 4

3 METHOD

The method ranks alarms for review with a two-stage PU auditor: Stage I adapts safe references and consolidates reference orderings, while Stage II refines a frozen student ranking using cohort structure and predicted agreement.

3.1 OVERVIEW

  • Stage I learns one student, then Stage II freezes it and refines the review queue; neither stage uses target safety labels for fitting.The final queue uses no test-time reference ensemble.Source: §3.1 · p. 4, Figure 2 · p. 4

3.2 STAGE I: RELIABLE PU RANKER LEARNING

  • Stage I first transfers trust-aware supervision from verified-safe non-alarms toward the alarm domain, then distills reference auditors’ ordering preferences into one student.The auditors share a PU objective that combines distribution alignment and group-structured representation mixing.Source: §3.1 · p. 4, §3.2 · p. 4, Figure 2 · p. 4

3.2.1 TRUST-AWARE PU SUPERVISION

  • Trust-aware supervision moves safe-reference comparison scores toward the monitor’s alarm-selection boundary and softens negative pressure on plausible false alarms.The protection mass is based on TN-supported safe mass among alarms, not an estimate of global false-positive prevalence.Source: §3.2.1 · p. 5

3.2.2 RELIABILITY-GATED RANK DISTILLATION

  • Reliability-gated distillation transfers median normalized ranks from frozen reference auditors, weighting reference-preferred pair directions and downweighting nearly tied positions.A cohort-level gate reduces distillation influence when transfer is weaker or reference allocation overlap is lower.Source: §3.2.2 · pp. 5–6

3.3 STAGE II: CONSENSUS-GUIDED STRUCTURAL REFINEMENT

  • Stage II freezes the student, combines hierarchical safe-reference support with supported relations among alarms, and uses predicted reference agreement to scale each alarm’s final adjustment.When local TN support is sparse, the hierarchy can borrow support from a broader ancestor.Source: §3.3 · p. 6

4 EXPERIMENTS

External validation on Agent-SafetyBench shows higher AUPRC and Top-5% false-alarm recovery than PULDA. The experiments also compare against direct LLM re-judgment and assess the framework’s components, robustness, and deployment cost.

4.1 EXPERIMENTAL SETUP

  • Each of four dataset–monitor settings uses 40 verified-safe non-alarms and unlabeled training alarms; labels stay sealed until rankings are fixed and monitors remain frozen.There are 510 target alarms in total.Source: §4.1 · p. 7

4.4 DIRECT LLM RE-JUDGMENT

  • 64.44 versus 57.78 AUPRC, with 4.67, 9.00, and 15.00 more false alarms recovered at 5%, 10%, and 20% budgets than zero-shot Qwen2.5-7B-Instruct.The comparison tests cohort-level PU supervision against independent semantic re-judgment of each alarm.Source: §4.4 · p. 7

4.2 RANKING PERFORMANCE AND REVIEW YIELD

  • Table 1 places two ATBench settings in its upper portion and four-setting macro mean±std over seeds, with summed false-positive yield, below.The macro results use seeds {13, 37, 73}.Source: Table 1 · p. 8

4.3 LARGE-SCALE EXTERNAL VALIDATION

  • AUPRC rises from 24.39 to 28.72 and Top-5% recovery from 66 to 94 false alarms (+42.4%) against PULDA on Agent-SafetyBench.The evaluation covers 4,054 held-out alarms from 1,400 cases and two frozen monitors.Source: §4.3 · p. 7, Table 2 · p. 8

4.5 ABLATION AND ROBUSTNESS

  • AUPRC rises from 58.42 to 60.35, 61.92, and 64.44 as trust-aware supervision, rank distillation, and structural refinement are added.Each component improves all three ranking metrics at every stage.Source: §4.5 · p. 8, Table 3 · p. 8
  • Across a 0.5–8× prior-mass range, AUPRC remains 61.52–64.44 and Top-5% recovery 21.67–22.67; increasing references from 10 to 80 raises AUPRC from 59.28 to 66.45.At the default 40-reference setting, K = 3 captures nearly all observed benefit from increasing the reference bank.Source: §4.5 · p. 9, Figure 4 · p. 9

4.6 DEPLOYMENT COST

  • The default K = 3 configuration takes 162.86 seconds of offline fitting with cached features on one NVIDIA H200; deployment uses a single student scorer.The final auditing pipeline does not use reference-model ensembling.Source: §4.6 · p. 9

5 CONCLUSION

The framework addresses monitor-induced selection in false-alarm auditing with two-stage PU ranking and further uses hierarchical safe-reference support and alarm relations. It improves macro AUPRC over eight PU baselines and recovers more false alarms than PULDA at a 5% review budget.

5 CONCLUSION

  • 33.3% more false alarms are recovered than with PULDA at a 5% review budget.The method also improves macro AUPRC over eight PU baselines by 5.27–16.98 absolute percentage points.Source: §5 · p. 9
  • The two-stage PU ranking framework adapts verified-safe supervision toward the alarm domain, stabilizes preferences across PU models, and uses hierarchical safe-reference support and alarm relations.It addresses monitor-induced selection, where verified-safe non-alarms may poorly represent hidden safe alarms.Source: §5 · p. 9
  • The framework requires no alarm safety labels for fitting and leaves monitor decisions unchanged.Source: §5 · p. 9

AI USE STATEMENT

Generative AI tools assisted with language editing, manuscript organization, and presentation feedback. The authors reviewed and verified the research and take responsibility for the final content.

AI USE STATEMENT

  • Generative AI tools assisted with language editing, manuscript organization, and feedback on presentation.Source: AI USE STATEMENT · p. 10
  • The authors reviewed and verified all research decisions, experimental designs, implementations, analyses, and reported results.Source: AI USE STATEMENT · p. 10
  • The authors take full responsibility for the final content.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The work prioritizes alarms for human review and does not certify highly ranked alarms as safe. Deployment retains the original monitor decisions and keeps reviewers involved, while respecting trace privacy and access conditions.

ETHICS STATEMENT

  • The method prioritizes alarms for human review rather than automatically suppressing them.Source: ETHICS STATEMENT · p. 10
  • A high-ranked alarm is not certified safe, and original monitor decisions remain in force.Source: ETHICS STATEMENT · p. 10
  • Deployment should keep reviewers in the loop and respect privacy and access conditions for agent traces.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The main text and appendix specify the method, evaluation protocol, and fixed training configuration. The repository provides implementation and regeneration materials, and benchmark data and checkpoints follow their original distribution licenses.

REPRODUCIBILITY STATEMENT

A TRAINING DEFINITIONS AND IMPLEMENTATION

Training adapts verified-safe references toward the alarm domain, limits negative pressure on plausible false alarms, and distills PU-model rankings into a student. A frozen-student refinement then uses hierarchical safe-reference support, alarm relations, and reference consensus to adjust the review queue.

A TRAINING DEFINITIONS AND IMPLEMENTATION

A.2 TN TRANSFER AND SOFT PROTECTION

  • A role classifier and role-boundary proximity transfer verified-safe TN references toward the alarm domain, while soft protection redistributes comparison pressure without labeling alarms.Transfer quantities are fixed before PU fitting; task-conditioned comparisons preserve task features.Source: §A.2 · p. 13

A.3 PU OBJECTIVE AND REFERENCE FITS

  • Scorers share a PU objective with alignment and group-structured mixing terms, enabling reference fits from verified TNs and unlabeled alarms.Cross-role pairs use group-structured mixing; same-role pairs use standard interpolation.Source: §A.3 · p. 14

A.4 REFERENCE RANKS AND STUDENT DISTILLATION

  • Independent reference models’ frozen outputs become normalized ranks whose coordinatewise median guides fixed allocations and student distillation.Rank operators use ascending ranks with midpoint tie handling; the reference bank supplies relative ordering information throughout student optimization.Source: §A.1 · p. 13, §A.4 · p. 14

A.6 TN-SUPPORTED MASS AND SENSITIVITY

  • The training-only TN-supported-mass statistic is a support measure, not an estimate of global false-alarm prevalence.It is computed from held-out role-classifier scores across three isolation-group folds before target labels are accessed.Source: §A.6 · p. 15, Table 6 · p. 15
  • AUPRC 61.52–64.44 and Top-5% recovery 21.67–22.67 persist across the evaluated support-mass sweep, above strongest-baseline operating points of 59.17 and 17.00, respectively.The sweep evaluates {0.5, 1, 2, 4, 8} times the training-only TN-supported mass.Source: §A.6 · p. 15, Table 7 · p. 15

B.1 HIERARCHICAL TN SUPPORT

  • Structural refinement is a frozen-student readout that uses a root–parent–leaf hierarchy, shrinking weak local TN support toward the nearest reliable ancestor.Reliability depends only on training support and geometry; comparing local, flat, and hierarchical views isolates the additional hierarchical support.Source: B STRUCTURAL QUEUE REFINEMENT · p. 16, §B.1 · p. 16

B.2 CONSENSUS-SUPPORTED RELATIONS

  • Cross-group alarm relations are retained only when root and a more specific support view agree, then predicted reference consensus scales the final displacement.The student remains frozen during refinement; a ridge head predicts relative consensus from standardized frozen monitor representations using reference ranks only.Source: §B.2 · p. 16, §B.3 · p. 16

C DATA PROVENANCE AND EXPERIMENTAL CONTROLS

The experiments evaluate the method against semantic re-judgment and controls for selection, representation, prior mass, and ensembling. They also report compute, deployment conditions, and sensitivity to TN and reference-bank sizes.

C.1 BENCHMARKS AND MONITOR IDENTIFIERS

C.2 COMPUTATIONAL COST

  • The default K = 3 configuration requires 162.86 seconds of offline cohort-level fitting on one NVIDIA H200 with cached monitor and task features.Deployment uses one student scorer and stored cohort statistics, without reference-model ensembling.Source: §C.2 · p. 17, Table 9 · p. 17

C.3 LOCAL LLM-AS-A-JUDGE CONTROL

  • 64.44 macro AUPRC versus 57.78 for direct re-judgment; the method recovers 4.67, 9.00, and 15.00 additional false alarms at 5%, 10%, and 20% review budgets.The zero-shot Qwen2.5-7B-Instruct baseline receives the safety policy, task instruction, and serialized trajectory, but no target labels or verified-safe references.Source: §C.3 · p. 17

C.4 DEPLOYMENT PROTOCOL

  • The method is designed for periodic batch auditing, with cohort statistics constructed by the same fixed procedure for each audit batch.Prediction outputs are fixed before target labels are accessed.Source: §C.4 · p. 17

D.1 SELECTION AND REPRESENTATION CONTROLS

  • The complete method outperforms both selection-modeling controls and deterministic representation baselines based on proximity to verified TNs.The controls test whether modeling monitor selection alone or using TN proximity can account for the ranking.Source: §D.1 · p. 17, Table 10 · p. 18

D.2 PRIOR-MASS SENSITIVITY

  • The method remains above the strongest baseline throughout the complete evaluated prior-mass range.The sensitivity sweep is reported in Table 7 and visualized in Figure 4(b,e).Source: §D.2 · p. 17

D.3 ENSEMBLE CONTROL

  • With three-model logit averaging, PULDA reaches 60.24 AUPRC; the single-reference variant reaches 62.35, and full K = 3 reaches 64.44 with one student scorer at inference.The improvement extends beyond inference-time ensembling; native outputs are consistently archived at the 20% cutoff, while some full-score entries are unavailable.Source: §D.3 · pp. 17–18, Table 11 · p. 18

D.4 TN AND REFERENCE-BANK SENSITIVITY

  • AUPRC rises from 59.28 with 10 verified TNs to 66.45 with 80; at 40 TNs, increasing K from 1 to 3 raises AUPRC from 62.35 to 64.44.K = 3 captures nearly all of the observed reference-bank benefit.Source: §D.4 · p. 18

E AGENT-SAFETYBENCH EXTERNAL VALIDATION

External validation evaluates the method on held-out Agent-SafetyBench alarms across two frozen monitors. It reports improved recovery over PULDA across cohort sizes and risk categories.

Protocol

  • The held-out evaluation contains 4,054 alarms, including 638 false alarms, across three trajectory-generation seeds and two frozen monitors.The training split supplies 40 verified-safe non-alarms, and settings and predictions are fixed before evaluation labels are accessed.Source: E AGENT-SAFETYBENCH EXTERNAL VALIDATION · p. 18

Full results

Risk-category results

Scaling and uncertainty

  • As nested cohorts grow from 1,034 to 4,054 alarms, the AUPRC improvement over PULDA increases from 3.07 to 4.33 points.Uncertainty is estimated using 2,000 case-level bootstrap resamples.Source: E AGENT-SAFETYBENCH EXTERNAL VALIDATION · p. 19

Related papers

Loading 2610.02925v1…