PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond F1: Evaluating Coverage and Failure Recovery in AI Model Security Scanners

Qianlong Lan, Vinothini Pandurangan, Anuj Kaul, Indranil Sanyal

arXiv:2608.27424v1cs.CRcs.AI

TL;DR

Conventional scanner metrics can miss whether a tool produces any usable security judgment. This paper evaluates three scanners with a controlled family-level benchmark that separates availability from conditional correctness and examines overlap and recovery. ModelAudit provides the broadest labeled-family availability, ModelScan is perfectly accurate when definitive, and supplementary scanners recover ModelScan’s incomplete malicious cases without adding unique true-positive families.

  • Problem

    Conventional evaluation metrics characterize scanner correctness only when a usable security judgment is available, leaving judgment availability as a separate evaluation dimension.

  • Method

    The study applies a family-aware methodology to a controlled synthetic Pickle/PyTorch benchmark, separating outcome states and analyzing scanner overlap and failure recovery.

  • Results

    ModelAudit produced definitive judgments for 100% of labeled families, ModelScan for 49.6%, and ModelScan achieved 100% conditional F1 when definitive.

  • Takeaways & Limitations

    Judgment accuracy should be evaluated separately from availability, while supplementary scanners can provide recovery without necessarily expanding primary detection coverage.

  • Takeaways & Limitations

    The synthetic, structured corpus cannot substitute for real-world malware, adaptive adversaries, or deployment-prevalence estimates.

Abstract

from arXiv · show

Static scanners are increasingly used to identify executable or otherwise unsafe content in machine- learning artifacts, yet conventional evaluation metrics characterize only cases where a scanner yields a usable security judgment. We evaluate ModelScan, ModelAudit, and Fickling using a controlled, artifact-backed benchmark on a synthetic corpus of 170 Pickle and PyTorch focused artifacts across 145 specimen families, 135 of which have binary security ground truth and 10 of which are intentionally malformed without labels. We explicitly distinguish non-N/A coverage, analysis completion, definitive security decisions, non-security findings, and unsupported outcomes. On labeled families, ModelAudit produced definitive security decisions for all 135 families (100%), Fickling for 110 (81.5%), and ModelScan for 67 (49.6%). Conditional on making a definitive judgment, ModelScan achieved 100% precision, recall, and F1. Fickling identified no unique true- positive families beyond those found by the combination of ModelAudit and ModelScan. Furthermore, for the 48 malicious families where ModelScan failed to complete its analysis, both ModelAudit and Fickling generated detections consistent with ground truth. These findings underscore the need to separate judgment accuracy from judgment availability, as well as incremental detection coverage from tool-level redundancy.

1 Introduction

The paper argues that scanner evaluation must distinguish whether a usable security judgment is produced from whether that judgment is correct. It introduces a family-aware benchmark separating availability, correctness, and cross-scanner coverage or recovery.

  • Motivation: Scanner quality has separate availability and conditional-correctness dimensions.Standard conditional metrics characterize correctness only when a usable judgment exists, while incomplete and unsupported outcomes remain operationally meaningful.
  • Benchmark design: The benchmark evaluates ModelScan, ModelAudit, and Fickling using 170 Pickle/PyTorch-focused artifacts organized into 145 specimen families.The methodology is controlled and artifact-backed, with family-level analysis and explicit treatment of scanner outcomes.
  • Evaluation framework: The evaluation separates non-N/A coverage, analysis completion, definitive security decisions, non-security findings, and execution failures.These categories support corpus-wide and labeled-family decision coverage alongside conventional conditional detection metrics.
  • Evaluation framework: Figure 1 organizes the framework around judgment availability, conditional correctness, and cross-scanner composition.The composition layer considers whether multiple scanners add detection coverage or recover decisions after failures.

2 Research Questions

The study asks how scanners differ in coverage, completion, and conditional accuracy, and whether specialized or heterogeneous scanners add unique detections, recovery, or operational value. It also examines latency and robustness to filename and extension changes.

  • RQ1: Coverage and availability: RQ1 compares observed non-N/A coverage, analysis completion, and definitive security-judgment availability.It asks whether scanners can produce usable security decisions, not merely whether their outputs are correct when available.
  • RQ2: Conditional accuracy: RQ2 examines whether conditional precision, recall, and F1 adequately characterize effectiveness when incomplete and unsupported outcomes are reported separately.The question directly tests the distinction between conditional accuracy and practical coverage.
  • RQ3: Incremental coverage: RQ3 tests whether Fickling adds unique family-level true-positive or false-positive coverage beyond ModelAudit plus ModelScan.The comparison targets incremental value from specialized Pickle analysis.
  • RQ4: Recovery: RQ4 asks whether heterogeneous scanners can provide a ground-truth-consistent decision when one scanner returns an incomplete result.This frames cross-scanner use as operational recovery rather than only detection aggregation.
  • RQ5: Operational characteristics: RQ5 compares execution latency and decision stability under evaluated filename and extension transformations.The robustness test varies declarations including .txt, .py, .bin, and omitted extensions while keeping underlying bytes unchanged.

3 Background and System Context

The benchmark models scanner behavior in a production-oriented asynchronous scanning layer while isolating it from deployment infrastructure and downstream execution controls.

  • System context: The primary experiments isolate asynchronous scanner workers from deployment infrastructure and commercial platform integrations.The setup characterizes isolated scanner behavior rather than an end-to-end ingestion service or downstream execution sandbox.

4 Methodology

The methodology uses a frozen, synthetic Pickle/PyTorch corpus with family-level ground truth and explicit outcome categories. It reports availability and conditional metrics separately, while also measuring recovery, latency, and rename robustness.

  • Corpus and unit of analysis: 170 artifacts are organized into 145 specimen families, which serve as the primary unit for security-effectiveness analysis.Intentional variants are grouped with their underlying family to avoid counting byte-identical copies as independent security observations.
  • Corpus and ground truth: The corpus contains 70 malicious, 65 benign, and 10 malformed families with intentionally unknown ground truth.The malicious set includes unsafe-deserialization and polyglot serialization cases; malformed families are excluded from confusion-matrix calculations.
  • Corpus and ground truth: The benchmark focuses on Pickle/PyTorch serialization, comprising 130 Pickle, 30 PyTorch checkpoint, and 10 Pickle-container artifacts.Standalone SafeTensors, ONNX, and GGUF formats are excluded.
  • Corpus and ground truth: Ground truth is assigned from the frozen corpus manifest at the specimen-family level, with transformed variants retaining their parent labels.Scanner observations are adjudicated against reviewed frozen labels rather than used to infer ground truth.
  • Outcome adjudication: Scanner observations are classified into detected, clean, not applicable, scan incomplete, scanner error, and non-security finding states.A nonzero process exit is not automatically a failure when valid detections are emitted.
  • Outcome adjudication: The benchmark distinguishes specimen maliciousness from deserialization capability when interpreting security findings.A conservative warning on benign custom serialization can be security-relevant yet count as a false positive under specimen-level ground truth.
  • Metrics: Observed non-N/A coverage measures explicit non-applicability absence, whereas completion requires a result rather than an incomplete or error state.Non-N/A status does not itself demonstrate successful format support.
  • Metrics: CDR measures corpus-wide definitive judgments, while LDR measures labeled-family decisions and excludes ten unknown-ground-truth families.Neither metric converts incomplete, unsupported, error, or non-security outcomes into false negatives.

5 Results

The results separate scanner judgment availability from conditional security accuracy and characterize benign-family warnings under specimen-level ground truth. They show that incomplete outcomes materially affect practical coverage, while scanner warnings on benign custom serialization can reflect conservative security semantics rather than arbitrary mistakes.

  • Observed Coverage, Completion, and Decision Availability: Table 1 distinguishes non-N/A coverage, completion, and definitive judgments eligible for TP/FP/TN/FN classification.Completion is reported both conditional on non-N/A dispositions and relative to the full corpus.
  • Conditional Accuracy Can Obscure Incompleteness: 100% conditional precision, recall, and F1 did not prevent ModelScan from producing definitive judgments for only 67/135 labeled families (49.6%).Its corpus-wide rate was 67/145 (46.2%) when the ten unknown malformed families were included in the operational denominator.
  • Benign-Specimen Warning Characterization: ModelAudit generated 60 false-positive families, including all 50 benign controls and 10 of 15 benign checkpoints, yielding specificity of 5/65 (7.7%).These results characterize alert quality under specimen-level ground truth.
  • Benign-Specimen Warning Characterization: Fickling produced 50 false-positive families from benign controls, with specificity of 0/50 and conditional FPR of 100% among definitive benign judgments.The 15 benign checkpoint families were not applicable to Fickling.
  • Benign-Specimen Warning Characterization: ModelScan produced no false positives among definitive benign judgments, classifying 45 benign families as true negatives while leaving 20 benign families incomplete.Incomplete benign families were excluded from its conditional FPR denominator.
  • Benign-Specimen Warning Characterization: ModelAudit and Fickling issued warnings for frozen benign custom serialization, so false positives reflect disagreement with specimen-level ground truth rather than necessarily useless warnings.Conservative identification of potentially executable deserialization constructs can remain security-relevant for benign specimens.

5.4 Fickling Adds Redundancy Without Incremental Family Coverage

Fickling adds failure-recovery redundancy but no incremental family-level detection beyond ModelAudit and ModelScan in this benchmark. Union results also depend on false-positive inheritance and fixed operational conditions.

  • Incremental coverage: 0 unique true-positive and 0 unique false-positive families are added by Fickling beyond the ModelAudit–ModelScan union.Fickling produces 60 true-positive and 50 false-positive families, all already represented in the other scanners’ unions.
  • Failure recovery: 48/48 malicious families incomplete under ModelScan are detected by both ModelAudit and Fickling.ModelAudit alone achieves the same malicious-family recovery as the three-scanner configuration.
  • False-positive boundary: 20 benign ModelScan-incomplete families generate false-positive warnings from ModelAudit, while Fickling also warns on applicable benign controls.Thus, successful malicious recovery does not imply correct recovery for all incomplete decisions.
  • Union policy: The ensemble uses detection union, flagging a family when any configured scanner flags it rather than applying majority voting or consensus.
  • Operational context: Under this frozen corpus, adding ModelScan or Fickling to ModelAudit leaves both the malicious true-positive set and ModelAudit’s false-positive set unchanged.The measurements use fixed scanner versions, timeouts, offline execution, and two workers; hardware metadata was not retained.

6 Discussion

The discussion separates conditional detection quality from judgment availability and distinguishes incremental coverage from failure-mode redundancy. ModelAudit provides broad availability and malicious-family coverage, while its specimen-level false-positive burden creates a policy tradeoff.

  • Core finding: Conditional detection quality and judgment availability are fundamentally different operational properties.ModelScan has perfect conditional results across 67 definitive judgments but definitive decisions for only 49.6% of labeled families.
  • Scanner trade-offs: 100% labeled-family decision coverage and all 70 malicious-family detections are achieved by ModelAudit, alongside a 92.3% benign-family false-positive rate.The false-positive rate creates a substantial operational alert burden under specimen-level ground truth.
  • Ground-truth semantics: Conservative warnings for benign custom serializations expose a policy tradeoff between blocking structurally hazardous constructs and allowing known-benign custom serializations.These warnings are false positives relative to frozen specimen labels but reflect the scanners’ capability-oriented behavior.
  • Redundancy: Fickling detects all 48 malicious families in the ModelScan-incomplete subset without expanding family-level true-positive coverage beyond ModelAudit plus ModelScan.Supplementary scanners can therefore provide fallback decision availability under tool failures.
  • Implications: Deployment policies should consider marginal detection yields, failure recovery, false-positive overhead, format applicability, and operational cost.Evaluation should also separate conservative capability-flagging metrics from specimen-level maliciousness benchmarks when their semantics diverge.

7 Limitations and Safety

The benchmark’s conclusions are bounded by its controlled, synthetic Pickle/PyTorch scope and by intentionally unlabeled malformed families. Fixed snapshots, limited robustness testing, absent hardware telemetry, and the distinction between structural capability and specimen maliciousness further constrain interpretation.

  • Scope: The benchmark uses 145 families and 170 artifacts focused specifically on Pickle and PyTorch serialization formats.It is not a population-level estimate of scanner performance across arbitrary production environments.
  • Corpus limitations: Synthetic deterministic artifacts support reproducibility and strict ground-truth control but cannot substitute for real-world malware, adaptive adversaries, or unseen zero-day exploits.The structured class distribution also means conditional precision should not be interpreted as expected production positive predictive value.
  • Ground truth: 10 malformed families intentionally lack binary security labels and are excluded from TP, FP, TN, and FN calculations.The methodology reports corpus-wide and labeled-family coverage rather than treating these artifacts as valid ground-truth cases.
  • Evaluation semantics: Specimen-level false-positive labels may classify actionable capability warnings as false positives, so deployment-specific operational utility requires dedicated policy studies.
  • Additional boundaries: The rename experiment covers 15 scanner-level comparisons, Fickling’s zero-incremental result is corpus-specific, and static scanning does not guarantee runtime security or absence of weight-space backdoors.Hardware telemetry was not recorded, further limiting portability of operational findings.

8 Reproducibility

The benchmark is reproducible through a public repository and a frozen, cryptographically tracked artifact bundle. It preserves manifests, execution records, scanner outputs, adjudications, and generation and metric tooling for independent inspection.

  • Public materials: The public repository provides the frozen benchmark bundle and experimental materials required to inspect and reproduce the reported results.
  • Provenance: The frozen bundle records corpus manifests, SHA-256 checksums, scanner versions, 510 execution records, raw outputs, adjudications, scanner matrices, recovery analyses, rename evaluations, and computed metrics.The execution matrix contains no duplicate IDs or missing or unexpected scanner–artifact combinations.
  • Integrity: All 510 adjudication records have confirmed review status, while the corpus manifest and adjudication ledger are frozen and cryptographically hashed.Primary results and summary metrics are derived mechanically from these frozen artifacts.
  • Reproduction tooling: The repository includes corpus-generation tooling, scanner adapters, benchmark configuration, and metric-generation scripts.Reproduction should use the frozen versions and integrity metadata because scanner dependencies and upstream software may evolve.

9 Related Work

Prior work addresses model-artifact security through static inspection, safe loading, and operational evaluation, while recognizing that serialization scanning covers only part of the broader supply-chain threat landscape.

  • Security Context: Python and PyTorch documentation warn that untrusted Pickle deserialization can enable arbitrary code execution, motivating artifact scanning and restricted loading.Model repositories deploy scanning as a supply-chain defense layer.
  • Security Scanners: ModelScan, Fickling, and ModelAudit inspect unsafe calls, Pickle bytecode, and embedded file-structure risks, respectively.PickleBall instead enforces safe-loading policies during deserialization.
  • Evaluation Perspective: Operational studies show that detection counts alone miss false-positive burden and the trade-off between reduced false negatives and increased flagged code when tools are combined.This benchmark applies that operational perspective to machine-learning artifact scanners.
  • Scope Boundary: Static serialization scanners cannot establish the absence of neural weight-space backdoors or data-poisoning artifacts, which remain outside this benchmark’s scope.Executable serialization payloads are treated as one vector within the broader model supply-chain threat landscape.

10 Conclusion

The benchmark shows that conditional detection accuracy does not capture whether model-artifact scanners can produce usable judgments. It further finds that supplementary scanners can provide failure-mode redundancy without increasing unique family-level detection coverage, motivating broader future evaluation.

  • 10 Conclusion: 100% conditional F1 for ModelScan corresponds to definitive judgments for only 49.6% (67/135) of labeled families.ModelAudit judged all 135 labeled families and detected all 70 malicious families, while falsely flagging 60 of 65 benign families.
  • 10 Conclusion: All 48 malicious families with incomplete ModelScan analysis were detected by Fickling, providing failure-mode redundancy without expanding overall family-level detection coverage.Fickling contributed no unique true-positive detections beyond the combination of ModelAudit and ModelScan.
  • 10 Conclusion: Scanner evaluation should separate judgment correctness from judgment availability and explicitly characterize false-positive warning semantics.Future work targets natural malicious artifacts, adversarial evasion, broader serialization ecosystems, longitudinal versions, and deployment policies.
Loading 2608.27424v1…