PaperlayerView exact revision on arXiv

Source-linked AI summary

VX-One: Typed, Calibrated Decision Heads for Real-Time Perimeter Triage - An empirical study of unknown-attack honesty, cascade gray zones, and calibration under drift

Juan S. Santillana

arXiv:2610.30090v1cs.CRcs.CLcs.LGcs.NI

TL;DR

Perimeter triage needs fast, calibrated decisions that remain honest on unseen events, but incumbent approaches miss at least one of these requirements. VX-One packages typed decision heads and evaluates them against calibrated baselines and in a two-phase signature-engine cascade. Its advantages are conditional: the paper reports stronger detection robustness and attribution honesty in some unsaturated settings, alongside calibration and transfer failures.

  • Problem

    Perimeter mitigation needs line-of-fire decisions that are fast, calibrated for thresholding, and honest on events absent from training, requirements incumbent technology classes do not all meet.

    Source: §1 · p. 1

  • Method

    VX-One uses a shared model with calibrated gate, closed-set choice including unknown, and ordinal-score heads, and evaluates the pipeline with public-data baselines and a signature-engine cascade.

    Source: §1 · p. 1

  • Results

    Typed decisions help detection robustness and attribution honesty under drift or unknown families, while trees win attribution in saturable benchmarks and calibration collapses under drift.

    Source: §1 · p. 2, §5.7 · p. 7

  • Takeaways & Limitations

    The measured boundary is conditional rather than a uniform win: typed decisions have value in adversarial or drifting regimes, but not in saturable benchmarks.

    Source: §5.7 · p. 7

    LOAO measures generalization across benchmark families, not robustness against real-world novel attacks, and the authors caution against interpreting it as zero-day detection.

    Source: §6 · p. 8

Abstract

from arXiv · show

Perimeter triage faces a latency/semantics trade-off: signature engines are fast and format-safe but blind to unseen attacks, while generative models are semantic but slow and produce unparseable output. In late 2026 a class of commercial System One decision models (TypeSafe Jev; Microsoft-Decision-1) began selling typed decision primitives - binary gates, closed-set choices, ordinal scores - as product APIs, with no academic treatment. We present VX-One: a small multi-head network (100K-10M parameters) whose outputs are these primitives - a calibrated gate, a choice over a closed candidate set that always includes an explicit unknown option, and an ordinal score, trained multi-task, temperature-calibrated per head (~0.7-1.7 ms p50 on CPU). We evaluate it on UNSW-NB15, CIC-IDS2017, EMBER2024 against calibrated XGBoost, Random-Forest, MLP, Kitsune, and measure a two-phase cascade (Suricata with 52,674 Emerging-Threats rules -> VX-One) over five days of CIC-IDS2017. Findings: (i) VX-One wins detection (AUROC/FPR@95%TPR) on UNSW-NB15 and under temporal drift on CIC-IDS2017 (0.711 vs 0.637), but XGBoost wins multi-class attribution on all three benchmarks; (ii) leave-one-family-out gate detects never-seen families on 9/9 UNSW folds yet drops below chance on CIC recon family, and abstention threshold transfers poorly across benchmarks; (iii) on payload-less reconstructions signature engines resolve ~0.1% of flows - gray zone ~99.9% of traffic, recovered by Phase 2 (combined recall >=0.978); (iv) calibration collapses under temporal drift (ECE 0.04->0.40) for every method while ranking partially survives. We provide a measured account of where typed calibrated decisions help perimeter triage, and where they fail.

1 Introduction

Perimeter triage needs fast, calibrated decisions that remain honest on unseen events, while incumbent systems each miss part of that requirement. VX-One packages typed decision heads and evaluates where they help—and fail—against baselines and in a signature-engine cascade.

Motivation

  • Deterministic engines are fast and format-safe but structurally blind to zero-day and low-and-slow attacks, with rule growth difficult for operations teams to maintain.Text-based models bring semantics but can be too heavy for line-of-fire decisions; generative agents add latency and output-format fragility.Source: §1 · p. 1
  • Classical ML performs well at tabular attribution, but treats every event as a membership test and can confidently misattribute families absent from training.XGBoost wins the paper’s attribution benchmarks.Source: §1 · p. 1

Research gap

  • A commercial class of typed decision APIs began shipping in October 2026, but the paper reports no peer-reviewed academic treatment or security-domain assessment of that class.The authors build the primitives themselves and evaluate the pipeline on public data with anti-strawman baselines.Source: §1 · p. 1

Contributions

  • The system uses a shared encoder with a calibrated gate, a closed-set choice that includes unknown, and an ordinal score, trained and calibrated per head.The evaluation includes unknown-family honesty, a Suricata-to-VX-One cascade, and measured negative results.Source: §1 · p. 1
  • VX-One’s measured advantage is conditional: typed decisions help detection robustness and attribution honesty under drift or unknown families, but trees win attribution in saturable benchmarks.The paper reports calibration collapse under drift, non-portable abstention thresholds, and a measurable accuracy–honesty trade-off.Source: §1 · p. 2

2 Related Work

Related work situates VX-One among typed decision APIs, single-pass models, calibration and abstention methods, signature engines, and network-traffic ML. The paper distinguishes its contribution as typed calibrated outputs and their security-domain evaluation.

System One decision models

  • System One decision models package binary gates, closed-set choices, and ordinal scores with calibrated option probabilities and single-pass scoring; the cited vendor benchmarks remain industry claims.The paper describes its work as the first academic instantiation and security-domain evaluation of the triad.Source: §2 · p. 2

Single-pass inference

  • Single-pass inference means producing the full output in one forward pass, but BERT-family classifiers also do so; VX-One’s differentiator is typed calibrated outputs, not single-pass inference alone.The paper places the Viola–Jones cascade as an ancestor of early-exit decision layers.Source: §2 · p. 2

Calibration and abstention

  • The calibration protocol draws on temperature scaling [14] and other probability transforms, while abstention follows reject-option and selective-classification work [11].The paper notes that temporal splits violate the exchangeability condition needed for distribution-free coverage claims [4].Source: §2 · p. 2

Signature engines and their limits

  • Deterministic engines including Snort, Zeek, YARA, and Suricata anchor the signature tier; the experiments use stratified and temporal splits with audited labels.Prior audits document systematic CICIDS2017 labeling errors.Source: §2 · p. 2

ML on network traffic and malware

  • Kitsune [25] is adapted from packet-level unsupervised detection to flows, while calibrated boosted trees [5] are the attribution baseline VX-One loses to on all three benchmarks.The paper also identifies Viola–Jones as an ancestor of its second-phase-only-when-needed cascade.Source: §2 · p. 2

3 VX-One Design

VX-One maps benchmark telemetry into a shared representation and three typed heads, then uses calibration and controller rules to produce decisions. The design keeps probability composition within single heads and makes operational thresholds explicit cost statements.

3.1 State object

  • Telemetry is converted to a flat state with dataset-specific feature dimensions and, where available, protocol categorical fields; preprocessing imputes, transforms, scales, and clips features.The feature stage includes a row flag for non-finite values and is reproducible from frozen configurations.Source: §3.1 · p. 2

3.2 Trunk and typed heads

  • A shared MLP representation feeds a calibrated gate, a closed-set choice containing unknown, and a CORN ordinal score over benign, low, high, and critical severity.Unknown is a trained class for EMBER2024; for UNSW and CIC it is implemented as decision-time abstention.Source: §3.2 · p. 2

3.3 Training formulation

  • Training balances tasks with a modified homoscedastic weighting scheme and de-tilts choice logits before calibration so the recovered posterior remains population-correct.The gate uses unweighted binary cross-entropy because weighting shifts its optimal logit by an offset temperature calibration cannot undo.Source: §3.3 · p. 3

3.4 Decision controller semantics

  • The controller uses single-head gates and composes boolean decisions rather than probabilities across heads, because calibrated heads can yield a poorly calibrated joint policy.The default 0.5 thresholds encode particular cost assumptions; operators with different cost ratios can use the stated cost-sensitive mapping.Source: §3.4 · p. 3

4 Experimental Setup

The experiments compare VX-One with calibrated classical and neural baselines across three public datasets, using both stratified and temporal splits where specified. Evaluation covers detection, attribution, calibration, and single-event latency.

Data

  • UNSW-NB15 contains 149,039 / 26,302 / 82,332 flows and nine attack families; CIC-IDS2017 uses 2.83M flows with stratified and by-day splits, and EMBER2024 uses 3.85M binaries with a temporal split.EMBER2024 includes 2.19M ground-truth-unknown training rows; CIC labels are audited and raw-label encoding artifacts are repaired.Source: §4 · p. 3

Baselines

  • Baselines are XGBoost, Random Forest, MLP, and Kitsune, trained with the same feature stage and comparable loss weighting and calibration protocols as VX-One.The planned zero-shot commercial System One API baseline was not executed that cycle.Source: §4 · p. 3

Metrics

  • The evaluation reports AUROC and FPR@95%TPR for detection, label-aware macro-F1 for attribution, calibration metrics including ECE and Brier, and single-thread batch-one latency.Latency uses 3,000 timed events, with model-exact hardware recorded for each result.Source: §4 · p. 3

Infrastructure

  • Training uses serverless L4 GPUs, while evaluation computes metrics locally from dumped raw logits; the infrastructure reports 60 passing unit tests.The tests include end-to-end smoke training.Source: §4 · p. 3

5 Results

VX-One leads detection under unsaturated and drifting conditions, while calibrated XGBoost leads attribution; unknown-family behavior, calibration, and cascade results depend strongly on the operating regime. The results also quantify costs and boundaries for typed decisions, including latency, payload opacity, and the unknown option.

5.1 E1/E3 — Detection, attribution, calibration

  • VX-One tops AUROC and FPR@95%TPR on UNSW-NB15 and temporal CIC, while calibrated XGBoost wins macro-F1 attribution on all three benchmarks by 4–8 points.On by-day CIC, tree FPR@95 is 1.0, while VX-One retains a partially usable operating point.Source: §5.1 · p. 3, Table 1 · p. 4

5.2 E2 — Unknown-attack honesty

  • Nine of nine UNSW held-out families have gate AUROC ≥0.85, but CIC recon falls below chance at 0.426 and the default abstention threshold transfers poorly.On CIC, unseen-family events are often assigned to normal with high confidence; per-event flow statistics cannot capture reconnaissance that requires temporal aggregation.Source: §5.2 · p. 4

5.3 E4 — Latency

  • VX-One’s typed decisions cost approximately 1 ms p50 on Small, supporting per-event inline use; API-level commercial latency claims are not directly comparable.The latency measurements use single events, batch size 1, and one CPU thread.Source: §5.3 · p. 5, Table 4 · p. 5

5.4 E5 — Two-phase cascade: how much of the line of fire is actually a gray zone

  • Under payload opacity, Suricata detects only ∼0.1% of daily attacks by volume, leaving a 99.9% gray zone; the estimated cascade retains recall ≥0.978.This is a payload-blind lower-bound setting, and combined cascade statistics rely on a documented independence assumption; Thursday precision is 0.552.Source: §5.4 · pp. 5–6, Table 5 · p. 6

5.5 E6 — Ablations

  • Ablations show regime-dependent trade-offs: removing the unknown option raises EMBER macro-F1 by 11.2 points, removing the CIC class tilt improves macro-F1 by 4.9, and zero dropout costs CIC macro-F1 by 14.7.Temperature scaling changes probabilities and improves ECE without changing argmax or AUROC; zero dropout instead gains 2.7 macro-F1 on EMBER.Source: §5.5 · p. 7

5.6 E7 — The commercial System One baseline, measured

  • On 2,021 stratified UNSW test rows, the zero-shot commercial gate is near chance and choice attribution is degenerate; adding an unknown option leaves performance unchanged or slightly worse.The paired unknown-option run lowers gate AUROC from 0.568 to 0.476 and macro-F1 from 0.069 to 0.066.Source: §5.6 · p. 7, Table 7 · p. 7

5.7 Discussion

  • The results bound VX-One’s advantage rather than establish a uniform win: typed decisions help detection robustness and attribution honesty under drift, while trees lead attribution on saturated splits.Honesty and accuracy are distinct under drift: a system can improve macro-F1 while falsely attributing a mutated family, whereas abstention incurs an accuracy cost.Source: §5.7 · p. 7

6 Limitations •

The authors bound the evidence to audited benchmark settings and qualify the cascade, drift, and vendor results rather than presenting them as universal guarantees.

6 Limitations •

  • LOAO measures generalization across benchmark families, not robustness to real-world novel attacks; the authors explicitly reject interpreting E2 as zero-day detection.Severity scores are rubric-derived rather than human-labeled, so cross-level claims above benign, low, high, and critical are approximate.Source: §6 · p. 8
  • The cascade estimate assumes independence between Phase 1 and Phase 2, an assumption that can be optimistic when Phase 1 misses the stealthiest subset.The measured per-day F1 and gray-zone table is distinct from that two-sample estimate.Source: §6 · p. 8
  • Only the four main-table runs were repeated with three seeds; the remaining tables are single-seed, and small CIC stratified head-to-head differences are borderline noise.Main-table ranking metrics varied by at most 0.02 points across seeds.Source: §6 · p. 8
  • Temporal drift breaks exchangeability, so the paper reports empirical conformal coverage rather than promising distribution-free coverage.The domain classifier certifies the total-variation gap only from below; failure to separate distributions does not establish exchangeability.Source: §6 · p. 8
  • The vendor-API comparison is a single zero-shot commercial deployment on one distribution, not an exhaustive comparison across models, prompts, or distributions.The supported conclusion is that the generic triad does not transfer zero-shot to this domain, not that fine-tuning cannot work.Source: §6 · p. 8

7 Conclusion

VX-One packages typed decisions for perimeter triage and evaluates their boundaries under unknown families, drift, and a signature-engine cascade. The paper reports measured advantages alongside negative results, not a uniform win.

7 Conclusion

  • The paper reports detection robustness and attribution honesty under unknown families and drift, alongside a measurable signature-engine gray zone and four negative results.Those negatives include calibration collapse under drift, non-portable abstention thresholds, regime-dependent class tilts, and an accuracy cost from the unknown option.Source: §7 · p. 8
  • VX-One is presented as the first published instantiation and public-data evaluation of the System One primitive triad for perimeter triage.The work evaluates the pipeline end-to-end, including a real signature engine in front of VX-One.Source: §7 · p. 8
  • The paper says its code, configurations, tests, and measured results accompany the work.The remaining next step identified is replacing the lower-bound cascade estimate with a payload-inclusive Phase-1 measurement.Source: §7 · pp. 8–9

A Formal statements

The formal and measured analyses distinguish single-head calibration from cross-head policy composition, and test how calibration, coverage, and ranking behave under drift.

A Formal statements

  • Single-head post-processing calibration error is bounded by the head's own deficiency plus the expected size of the transformation.The paper applies this result to de-tilting, temperature scaling, and thresholding.Source: A Formal statements · p. 9
  • CORN's product of true conditional probabilities gives the severity probability exactly, but its calibration bound concerns conditional deviations rather than marginal ECEs.The paper cautions that conditioning changes base rates and directly measures the drop gate's ECE.Source: A Formal statements · p. 9
  • For the drop gate, direct ECE was 0.026 on UNSW validation and 0.11 on temporal CIC test; its conditional-link ECEs did not determine the product's calibration.The paper therefore measures the composed gate directly rather than inferring it from its links.Source: A Formal statements · p. 9
  • A domain classifier's balanced error certifies only a lower bound on distributional total variation, so failure to distinguish domains certifies nothing about exchangeability.The stated bound is dTV ≥ 1 − 2 BER.Source: A Formal statements · p. 9
  • Across twelve EMBER test weeks, conformal coverage was 0.895–0.935, while malicious-side coverage fell to 0.884 in week 60 amid certified drift.The weighted variant shifted coverage by at most 0.022 without a consistent direction; the challenge set was excluded from these coverage statements.Source: A Formal statements · p. 9
  • With uniform misclassification cost and escalation cost λr, the rejection rule abstains when the largest class probability is below 1 − λr.The paper reports zero deviations from this abstraction under uniform costs, but asymmetric costs can require controller-side re-derivation.Source: A Formal statements · p. 9
  • On PortScan, delayed-label recalibration moved ECE from 0.553 to 0.526, while even oracle calibration left ECE at 0.445 because discrimination was weak.The choice head's confidence was more recalibratable: oracle choice ECE fell from 0.543 to 0.056, with macro-F1 unchanged by construction.Source: A Formal statements · p. 10
  • On CIC PortScan, retraining raised gate AUROC from 0.560 to 0.839 at two passes, but day-2 ECE stayed 0.553 versus 0.554 static.Four passes regressed AUROC to 0.760; the authors conclude fresh weights improve discrimination, not calibration.Source: A Formal statements · p. 10

Related papers

Loading 2610.30090v1…