PaperlayerView exact revision on arXiv

Source-linked AI summary

MechAudit-40: White-Box Auditing across 40 LLM Attack Mechanisms

Zhen Guo, Shanghao Shi, Shamim Yazdani, Ning Zhang, Reza Tourani

arXiv:2609.06612v1cs.CR

TL;DR

White-box LLM defenses are commonly evaluated on isolated attack families, leaving open whether representation shifts generalize to unseen mechanisms. MechAudit-40 evaluates that transfer and builds a zero-oracle runtime auditor; it sustains coverage across held-out mechanisms while its alerts remain evidence of exposure, not downstream compromise.

  • Problem

    White-box defenses have mostly been validated on isolated attack families, leaving generalization to unseen attack mechanisms unresolved.

    Source: §1 · p. 1

  • Method

    MechAudit-40 evaluates 40 mechanisms with matched clean–attack representations and grouped holdouts, then uses target-calibrated benign references for single-trace runtime auditing.

    Source: §1 · p. 2, §5.2 · p. 9

  • Results

    MECHAUDIT detects over 81% of held-out attack executions at a 0.70% false-positive rate and uniquely maintains target-averaged recall above 50% across all 40 mechanisms.

    Source: §8 · p. 13

  • Takeaways & Limitations

    Target-calibrated representation trajectories support cross-mechanism attack-exposure auditing, but do not establish downstream task compromise or parameter integrity.

    Source: §8 · p. 13, §6.3 · p. 13

    Unseen-mechanism generalization requires target-specific baseline calibration; at the uncalibrated threshold, recall drops to 5.0%.

    Source: §6.1 · p. 11

Abstract

from arXiv · show

While LLM attacks span prompt optimization, multi-turn context manipulation, retrieval poisoning, and model backdoors, white-box defenses are typically evaluated on isolated attack families. Consequently, whether heterogeneous attacks leave internal representation shifts that generalize to unseen threat mechanisms remains unknown. We present MechAudit-40, a systematic evaluation of 40 attack mechanisms across five open-weight model architectures. Threat-specific success criteria, 100,000 matched clean-attack representation pairs, predefined categories, and grouped holdouts isolate genuine attack-induced displacement from target scale, corpus bias, and data-leakage shortcuts. Across this testbed, attacks induce structured multi-depth trajectories rather than isolated layer spikes. While raw peaks are non-portable across architectures, target-calibrated profiles preserve transferable geometric signatures: under complete mechanism holdout, hidden states alone recover the threat category of unseen attacks with 82.5% accuracy. Guided by this finding, we design MechAudit, a runtime auditor that operates under strict zero-oracle constraints without requiring clean baseline traces or attack metadata. MechAudit detects 81.1% of held-out attack executions at a 0.70% false-positive rate and maintains 78.1% recall when an entire functional category is withheld. In matched comparisons, MechAudit is the only detector that avoids mechanism-level coverage collapse, maintaining over 50% recall across all 40 mechanisms. Internal representations thus support cross-mechanism attack-exposure auditing against calibrated benign references, but decouple from downstream task compromise and parameter integrity.

1 Introduction

The paper addresses whether white-box defenses can generalize beyond isolated attack families to unseen mechanisms. It builds a 40-mechanism testbed, finds transferable multi-depth signatures, and uses them to motivate a runtime auditor with explicit evidentiary boundaries.

1 Introduction

  • Existing white-box defenses are mostly evaluated on isolated attack families, leaving generalization to heterogeneous, unseen mechanisms unresolved.The paper frames the challenge as detecting representation anomalies relative to a calibrated benign reference distribution.Source: §1 · p. 1
  • Multi-depth hidden-state trajectories recover categories of unseen mechanisms under complete mechanism holdout, with signatures distributed across depth rather than isolated peak layers.Exact peak depths and activation magnitudes vary across architectures and attack formulations.Source: §1 · p. 2
  • MECHAUDIT sustains detection coverage across held-out threats, while specialized baselines fail to generalize beyond their training distributions.The paper reports this across complete mechanism- and category-level holdouts.Source: §1 · p. 2
  • The testbed spans 40 mechanisms across eight categories and five open-weight models, with over 190,000 judged executions and 100,000 matched representation pairs.Provenance-tracked train–test partitions support controlled generalization evaluation.Source: §1 · p. 2
  • MECHAUDIT inspects a single runtime trace against a calibrated benign reference without requiring clean traces or attack metadata at deployment.The framework is motivated by offline discovery from matched clean–attack pairs.Source: §1 · p. 2
  • MECHAUDIT’s alerts indicate empirical attack exposure, not downstream safety-alignment bypass or parameter tampering.The paper treats those outcomes as requiring separate validation.Source: §1 · p. 2

2 System and Threat Model

The threat model treats an LLM application as a pipeline with distinct attack surfaces and evaluates attacks at the interfaces they target. The registry and information regimes define what the auditor observes and what its alerts can establish.

2 System and Threat Model

  • The application model spans instructions, prompts, conversational context, retrieval, memory, parameters, and output actions, with evaluation at the targeted component interface.Attacks are evaluated under source-specific access capabilities and budgets.Source: §2 · p. 2
  • Offline discovery uses attack identities and matched clean–attack pairs, whereas runtime auditing uses one execution and a calibrated benign reference without attack labels or downstream feedback.The trusted computing base includes the serving runtime, activation extraction, trace metadata, trained auditor, benign references, and thresholds.Source: §2 · p. 3
  • The evaluation covers textual executions across open-weight models, excluding closed APIs, multimodal pipelines, and availability attacks.Each attack is evaluated in isolation rather than with cumulative adversary capabilities.Source: §2 · p. 3
  • The 40-mechanism registry assigns five mechanisms to each of eight categories and supports leave-one-category-out evaluation.Mechanisms meet reproducibility, bounded-access, and measurable-outcome criteria; the balanced testbed is not an estimate of real-world attack prevalence.Source: §2 · p. 3, Table 1 · p. 4
  • An MECHAUDIT alert indicates representation-level exposure relative to a calibrated benign baseline, not task compromise or checkpoint tampering.Attack success is measured with threat-specific criteria rather than a single completion metric.Source: Figure 1 · p. 3, §2 · p. 4

3 Measurement Design and Testbed

The measurement design combines a fully crossed behavioral testbed with matched clean–attack representations and grouped, leakage-resistant evaluation. Representation extraction measures displacement at aligned relative depths, while controls standardize targets and balance comparisons.

3 Measurement Design and Testbed

  • Execution traces, judgments, and hidden-state records remain separate artifacts linked by SHA-256 hashes to preserve provenance and prevent leakage across transfer evaluations.The design retains failed attack attempts in the empirical record.Source: §3 · p. 4

3.1 Behavioral Evaluation

  • The behavioral corpus crosses 40 mechanisms with five targets and contains 192,500 judged executions, retaining refusals, ill-formed outputs, and runtime faults.Threat-specific outcome criteria are recorded alongside refusal and attack-success measures.Source: §3.1 · pp. 4–5
  • The paired corpus contains 100,000 clean–attack representation pairs, matched on base tasks so displacement measures adversarial intervention rather than task differences.Failed attacks remain included; representations use final-prompt-token residual activations, cosine distance, and relative ℓ2 distance, with depth normalized across models.Source: §3.1 · p. 5, §3.2 · p. 5

3.2 Representation Extraction

  • Grouped holdouts isolate held-out mechanisms and categories before normalization or fitting, while target-specific benign references and per-cell macro-averaging control scale and imbalance.Significance testing uses balanced label permutations and confidence intervals bootstrap across cells.Source: §3.2 · pp. 5–6

4 Discovering Reusable Audit Traces

Across 40 mechanisms and five targets, attacks produce structured, multi-depth representation shifts whose calibrated profiles transfer across held-out mechanisms. Hidden-state profiles recover unseen threat categories, while displacement remains distinct from behavioral success.

4.1 Layer-Wise Profile Geometry

  • Attacks produce structured displacement across model depth, and cosine distance and relative-ℓ2 trajectories agree strongly (ρ = 0.96 across 200 attack–model cells).Figure 2 summarizes layer-wise displacement over 100,000 matched clean–attack pairs.Source: Figure 2 · p. 7, §4.1 · p. 7
  • Internal displacement does not track behavioral success: substantial shifts can accompany faithful attacks or parameter damage, while modest shifts can accompany failed prompt instantiations.The analysis treats displacement as evidence of representational exposure rather than downstream exploit completion.Source: §4.1 · p. 7
  • Raw peak depths are not portable: Llama and DeepSeek peak at normalized depths 0.46 and 0.62, while Qwen, Gemma, and GPT-OSS peak near 0.91–0.95.Cross-target raw peak-depth rank correlations are weak to moderate (ρ ∈ [0.03, 0.71]); reusable signatures instead require target-calibrated relative-depth trajectories.Source: §4.1 · p. 7, Figure 2 · p. 7
  • Four geometric descriptors summarize each distance trajectory, producing an eight-dimensional per-target state vector for signature discovery.The descriptors are mean displacement, peak magnitude, normalized peak depth, and distribution centroid for cosine and relative-ℓ2 profiles.Source: §4.1 · p. 7

4.2 Cross-Mechanism Transferability

  • Target calibration shifts cosine peak-depth variation from architecture-confounded raw profiles to mechanism-associated profiles: attack identity accounts for 78.6% of calibrated variation.Raw cosine peak depth is attributed 47.0% to model and 40.2% to attack; calibrated relative-ℓ2 variation is 69.7% mechanism-associated.Source: Table 4 · p. 8, §4.2 · p. 8
  • Calibrated profiles preserve structured but nonuniform cross-target agreement, with pairwise rank correlations ranging from ρ = 0.21 to ρ = 0.55.Figure 3 reports cross-target agreement and within-category variation.Source: Figure 3 · p. 8, §4.2 · p. 8
  • Related mechanisms redistribute representations across depth in patterns associated with their intervention surfaces; complete holdouts distinguish this geometry from generic anomaly magnitude.The profiles support discriminant validity, not causal localization; observed peaks are diagnostic trace locations rather than causal intervention targets.Source: §4.2 · p. 8
  • 82.5% accuracy: calibrated internal representation profiles recover unseen attack categories, significantly exceeding the 25.0% 95th-percentile permutation baseline (p < 0.001).Behavioral outcomes alone achieve 85.0% accuracy, and combining behavioral and state features also achieves 85.0%.Source: Figure 4 · p. 8, §4.2 · p. 8

5 MECHAUDIT Design

MECHAUDIT turns transferable multi-depth signatures into a target-calibrated auditor for individual runtime traces. It uses a shared detector design and benign references calibrated separately for each target, without requiring clean counterparts or attack metadata at deployment.

5 MECHAUDIT Design

  • MECHAUDIT operationalizes transferable multi-depth signatures as candidate-only runtime auditing against a pre-calibrated benign reference, without clean baselines or attack metadata at deployment.Offline discovery uses matched clean–attack pairs, but deployment inspects only the observed execution and target-model identifier.Source: §5 · p. 8, §5.3 · p. 10

5.1 Design Requirements

  • The design requires one detector across all 40 mechanisms, target-specific calibration, single-execution inference, and a sub-percent false-positive rate on challenging benign inputs.The benign controls include encoded texts, long-context passages, quoted untrusted content, and benign security discussions.Source: §5.1 · p. 8

5.2 State Representation

  • Each execution is mapped to 21 normalized relative-depth bins after projecting final-prompt-token residual activations across layers.A fixed Achlioptas random projection preserves coordinate geometry while standardizing depth across architectures.Source: §5.2 · p. 9
  • The state representation combines projected coordinate trajectories with seven structural profiles capturing activation scale, dispersion, and layer-to-layer transitions.Coordinate trajectories retain localized shifts; structural profiles summarize energy distribution and transitions across depth.Source: §5.2 · p. 9
  • Features are robustly standardized against a target-specific benign reference bank using feature medians and 10th–90th percentile ranges.The calibration is intended to isolate adversarial displacement from model-specific activation scales and does not require attack data.Source: §5.2 · p. 9

5.3 Hybrid Detection and Decision Policy

  • A hybrid two-gate detector combines an ExtraTrees classifier for known attack geometry with a benign-only Mahalanobis estimator for departures from the benign distribution.The second gate can detect an unfamiliar mechanism that diverges from the benign manifold even when it does not resemble observed training attacks.Source: §5.3 · p. 9
  • MECHAUDIT alerts when either gate rejects benign execution, allocating each gate half of the total false-positive budget.The gate thresholds are set using the corresponding quantiles of challenging benign validation controls.Source: §5.3 · p. 9

6 Audit Evaluation

The evaluation tests frozen, zero-oracle auditing across five architectures and 40 mechanisms, with target-specific calibration and complete mechanism and category holdouts. MechAudit transfers across unseen threats, maintains broader mechanism coverage than the baselines, but its alerts do not reliably predict downstream attack success.

6.1 Unseen-Mechanism Auditing

  • 81.1% of unseen-attack executions are detected at 0.70% FPR under LOMO; under LOCO, recall is 78.1% at 0.66% FPR, a 3.0-point gap.LOCO withholds the evaluated mechanism and its four category neighbors; the authors interpret the narrow gap as evidence of transfer beyond neighboring prompt templates.Source: §6.1 · p. 10
  • On calibrated targets, AUROC ranges from 0.989 to 0.997, recall from 69.3% to 92.0%, and FPR from 0.46% to 1.55%; transferring a source threshold to an uncalibrated target yields 5.0% recall.The findings support cross-architecture transfer on calibrated targets but make target-specific baseline calibration necessary.Source: §6.1 · pp. 10–11

6.2 Auditor Comparison

  • MECHAUDIT alone exceeds the 50% recall floor across all 40 mechanisms, with at least 50.4% target-averaged recall per mechanism and 63.8% recall in its weakest category under category holdout.All-depth linear and text n-gram detectors reach 90.7% and 92.7% aggregate LOMO recall but fall to 0.5% and 0.0% on their worst mechanisms.Source: Figure 6 · p. 11, §6.2 · p. 11
  • Under LOMO, PIShield-style and Sentinel-style baselines reach 32.1% and 28.1% recall, versus MECHAUDIT’s 81.1%; under LOCO, their recall is 30.0% and 26.0%, versus 78.1%.The comparison uses matched candidate records, grouped splits, benign controls, target-calibration sets, and a fixed 1% family-wise false-positive budget.Source: §6.2 · p. 11
  • Replacing projected profiles with scalar hand-crafted features lowers ExtraTrees recall from 78.4% to 56.9%, while adding the Mahalanobis gate raises recall from 78.4% to 81.1%.An oracle-routing baseline reaches 84.4% recall; cryptographic checks found no train–test overlap across 310 splits.Source: §6.2 · p. 11

6.3 Claim Boundary

  • Successful-versus-resisted state contrasts are sparse, inconsistent, and architecture-dependent; success prediction reaches only 0.512 macro AUROC within-model and 0.518 under joint model-plus-attack holdouts.Reaching 95% TPR requires 89.6% FPR, while naive record pooling inflates AUROC to 0.656 by learning cross-cell base rates.Source: §6.3 · p. 12, Figure 8 · p. 12, Figure 7 · p. 12
  • An MECHAUDIT alert is evidence of internal adversarial exposure, not an autonomous compromise verdict; the authors recommend containment, trace logging, or secondary outcome validation.Even an outcome probe with paired clean baselines remains near chance under grouped evaluation.Source: §6.3 · pp. 12–13

7 Related Work

Prior benchmarks and representation-based defenses cover distinct attack surfaces, but do not establish whether one auditor generalizes across unseen mechanisms and categories. The paper positions balanced benchmarking and rigorous holdouts as ways to test that transfer while distinguishing non-adaptive from adaptive threats.

Attack coverage and benchmarks

  • Existing benchmarks cover subsets of attack surfaces, but their security invariants differ, and external behavioral endpoints do not test transferable representation-level signatures across attack surfaces.MECHAUDIT-40 uses a balanced 40-mechanism registry with threat-specific ground-truth outcomes and denominator hygiene.Source: §7 · p. 13

White-box auditing and generalization

  • PIShield and Sentinel show representation-based detection within narrow threat models, but prior work does not establish transfer when a mechanism or its entire category is withheld.Random splits can reward lexical shortcuts or architecture-specific activation scales rather than generalizable representation geometry.Source: §7 · p. 13

Evaluation under distribution shift and adaptive threats

  • The evaluation combines grouped task-and-model partitions, cell-level macro-averaging, hash audits, and hard-negative calibration; adaptive evasion remains outside its non-adaptive threat setting.The authors distinguish baseline cross-mechanism generalization from adversaries using detector feedback or white-box oracle access.Source: §7 · p. 13

8 Conclusion

Across 40 mechanisms and five architectures, MechAudit-40 finds transferable target-calibrated representation signatures and develops a candidate-only auditor to detect exposure to held-out attacks. Its conclusion emphasizes mechanism-level coverage and distinguishes exposure alerts from downstream compromise or checkpoint tampering.

8 Conclusion

  • Heterogeneous attacks leave transferable structural signatures in target-calibrated geometric trajectories across model depth, rather than localized layer anomalies.Mechanism- and category-level holdouts support the claim that the patterns reflect threat structure rather than dataset or architectural scale artifacts.Source: §8 · p. 13
  • Under candidate-only, zero-oracle deployment, MECHAUDIT detects over 81% of held-out attacks at 0.70% FPR and maintains target-averaged recall above 50% across all 40 mechanisms.The conclusion contrasts that coverage with linear and lexical baselines that overfit dominant attacks and have out-of-distribution blind spots.Source: §8 · p. 13
  • An alert indicates internal representation-level exposure, not downstream task compromise or checkpoint tampering; emerging mechanisms should undergo complete holdout evaluation before entering training sets.The authors frame worst-case holdout coverage as a basis for auditing evolving threat surfaces without specialized detectors for each exploit.Source: §8 · p. 13

Ethical Considerations

The study identifies risks from consolidating attack techniques and describes safeguards for experiments and artifact release. Its experiments did not target production services, third-party accounts, or real users.

Ethical Considerations

  • Consolidating techniques can make harmful attacks easier to reproduce and risks exposing annotators or affecting external systems, but experiments used disposable, access-controlled resources and research identifiers.The study did not target production services, third-party accounts, or real users; its corpus excluded personal data, credentials, and user conversations.Source: Ethical Considerations · p. 14
  • Automated judgments reduce but do not eliminate annotation risk or error; manual validation uses content warnings, minimal context, opt-out, and disagreement resolution.Experiments preserve refusals, failed constructors, inapplicable cases, and infrastructure failures rather than retrying until success.Source: Ethical Considerations · p. 14
  • The public artifact provides identifiers, schemas, checksums, derived profiles, aggregates, and benign examples, while harmful requests, poisoned records, and modified checkpoints remain controlled.The release is intended to support verification without providing a turnkey attack collection.Source: Ethical Considerations · p. 14

Open Science

The appendices document the frozen attack registry, measurement artifacts, and diagnostics for behavioral outcomes, internal-state profiles, and candidate-only auditing. They show target-dependent representation geometry and mechanism-level variation while distinguishing descriptive evidence from causal or deployment claims.

Open Science

  • The review package provides checksums, claim-to-artifact indexing, registries, grouped splits, execution manifests, predictions, aggregate tables, and figure scripts for rebuilding results.Public materials contain non-sensitive metadata, derived measurements, benign examples, and deterministic analysis code.Source: Open Science · p. 14

A Frozen 40-Mechanism Attack Registry

  • The frozen registry balances eight primary categories with five mechanisms each, and its admission policy excludes benchmark suites, datasets, and prompt variants from mechanism slots.Alternates can replace only documented failures, and annotation proceeds with traces and study outcomes hidden until labels are frozen.Source: Table 6 · p. 19, §A.1 · p. 20, §A.2 · p. 20

C Target-Specific Internal-State Atlases

  • Within Qwen3-4B, profile shapes and peak depths vary even within categories, so raw layer indices or magnitudes cannot be transferred unchanged across architectures.The atlas displays row-standardized calibrated cosine-displacement profiles and summarizes peak depth, peak calibrated shift, and profile energy.Source: C Target-Specific Internal-State Atlases · p. 21, Figure 12 · p. 22

D.2 Detailed Cross-Target Interpretation

  • No model pair preserves architecture-invariant mechanism ordering; mean feature correlations range from 0.55 for Llama–Gemma to 0.21 for DeepSeek–GPT-OSS.Agreement involving GPT-OSS is weaker, including 0.24 for Llama–GPT-OSS.Source: §D.2 · p. 23, D Extended Cross-Model and Audit… · p. 23, Figure 13 · p. 23
  • A37 and A39 have the largest shared salience, but their abrupt shifts do not meet controls for causal claims; A38 has the largest model-specific component.The taxonomy representation also misclassifies several mechanisms, so category regularities do not make mechanisms within a category execution-equivalent.Source: §D.2 · p. 23

D.3 Complete Candidate-Only Auditor Results

  • Most attacks remain near the shared displacement range, while A37–A39 show the largest target-dependent departures and explain the C8 coverage challenge.Figure 16 summarizes within-target medians and interquartile ranges alongside cross-target profiles.Source: §D.3 · p. 23
  • The candidate-only auditor applies one detector form to all 40 mechanisms with target-specific benign calibration, reporting held-out-attack TPRs by mechanism and target.It receives no attack label, intervention-surface label, protected-outcome label, or matched-clean trace at test time.Source: §D.3 · p. 23, Figure 14 · p. 24, Table 7 · p. 24

D.4 Detector Adaptations and Ablations

  • Detector comparisons use identical candidate records, grouped splits, and target-specific calibration; the ablation changes one design choice at a time under a fixed protocol.The Table 9 caption defines minimum TPR and coverage as groups reaching at least 50% recall.Source: §D.4 · p. 23, Table 9 · p. 25

D.6 Extended Deployment Guidance and Limitations

The deployment guidance distinguishes attack-exposure evidence from protected-outcome success and checkpoint integrity. The measurement protocol specifies frozen targets, controlled execution records, threat-specific outcomes, and trace-quality checks.

D.6 Extended Deployment Guidance and Limitations

  • Richer state representations may recover discarded information, but any resulting detector still requires adaptive-evasion and distribution-shift evaluation.Least privilege, sandboxing, deterministic policy enforcement, and post-action verification remain necessary independently of the auditor.Source: §D.6 · p. 25
  • Five mechanisms per stratum cannot exhaust implementation diversity, and frozen snapshots provide reproducibility rather than timelessness.Operational use requires target-specific benign banks, workload-relevant hard negatives, and revalidation after serving changes; checkpoint or adapter integrity needs separate provenance validation and benign parameter-edit controls.Source: §D.6 · p. 26

E.1 Frozen Targets and Backends

  • The protocol pins model snapshots and decoding settings; hidden-state collection uses a direct Transformers forward pass because the vLLM backend does not expose hidden states.The collection exception is recorded separately from behavioral runs.Source: §E.1 · p. 26

E.2 Pre-Execution Gates

  • Before analysis, the registry, checkpoints, task groups, predicates, hard negatives, splits, attack budgets, trace definitions, rubrics, and analysis plan are frozen.Final-test outcomes cannot revise these objects; corrections require a versioned amendment and invalidate affected final runs.Source: §E.2 · p. 26

E.3 Protected-Execution Record

  • Each protected execution emits an immutable manifest row and trace, while hashes and structural metadata remain shareable when prompts or outputs are restricted.Figure 16 summarizes target-specific displacement distributions using within-target medians and interquartile ranges.Source: §E.3 · pp. 26–27, Figure 16 · p. 27

E.4 Condition Groups

  • Condition groups pair attacks with controls tailored to direct-input, dialogue, indirect-injection, persistent-state, and parameter interventions.When valid successful and unsuccessful candidates exist, both are sampled under the same base task and budget; behavioral denominators exclude construction and system failures.Source: §E.4 · pp. 27–28

E.5 Outcome Adjudication

  • Each security track has a dedicated outcome criterion, and no global success ratio pools their denominators.Semantic labels require documented independent judgments and blinded human audits; until validated, automated labels are measurement instruments rather than semantic ground truth.Source: §E.5 · pp. 28–29

E.6 Internal Trace Processing

  • Internal traces use final-input-token and protected-decoding states mapped to normalized depth, with whitening and clean-reference statistics fitted only on discovery training data.Quality-control failures remain in the ledger with rejection reasons, and exclusion rates are reported across categories, mechanisms, models, tasks, and conditions.Source: §E.6 · p. 29

E.7 Statistical Analysis

The statistical protocol treats mechanisms and base tasks as inferential units rather than treating generated prompts as independent attack methods. It requires balanced aggregation, hierarchical uncertainty estimates, and controlled confirmation for cross-mechanism claims.

E.7 Statistical Analysis

  • The inferential unit is the attack mechanism and base task, although protected executions are the descriptive unit.Category contrasts use equal-category macro weighting and bootstrap mechanisms within categories and tasks within mechanisms.Source: §E.7 · p. 29
  • Cross-mechanism signatures require held-out confirmation, cross-category coverage, mechanism consistency, model holdout, hard-negative specificity, and an aligned intervention.Layer-wise tests require false-discovery-rate control, and exploratory label-free analyses must precede disclosure of primary categories.Source: §E.7 · p. 29

F MECHAUDIT Development and Evaluation Protocol

  • The evidence ladder promotes behavioral coverage to exposure reports and calibrated profile geometry to mechanism-level triage, while withholding protected-outcome success claims absent supporting evidence.The auditor abstains when evidence supports attack exposure but not protected-outcome success.Source: F MECHAUDIT Development and Evaluation… · p. 29

F.1 Frozen Data Lifecycle

The evaluation pipeline separates offline paired-trace discovery, auditor development, and untouched grouped holdouts. Deployment scores one candidate execution against target-specific benign calibration and reports attack-exposure evidence, not proof of compromise.

F.1 Frozen Data Lifecycle

  • Three disjoint stages separate matched-trace discovery, auditor development, and final holdouts; the split audit found zero detected overlaps across 310 evaluated folds.Holdouts cover mechanisms, categories, and models, with grouped integrity checks on base IDs and input and feature hashes.Source: §F.1 · p. 29

F.2 Fast and Confirmatory Paths

  • Category and attack labels define grouped holdouts but are withheld from the candidate-only scorer, as is the intervention-surface label.A new target or serving revision requires a new benign bank and thresholds.Source: §F.1 · p. 29, §F.2 · p. 30
  • The runtime path scores one candidate execution using a shared projection and target-calibrated ExtraTrees and Mahalanobis gates, without clean traces or attack metadata.An alert is attack-exposure evidence, not mechanism attribution or proof that a protected violation succeeded; the paired clean–attack diagnostic is not a deployment detector.Source: §F.2 · p. 30

F.3 Evaluation Matrix

  • Completed regimes report per-mechanism, equal-category, worst-category, per-model, hard-negative, calibration, and hierarchical-interval results.Leave-one-category training has no expert for the withheld category and permits mechanism abstention.Source: §F.3 · p. 31, Table 13 · p. 31

F.4 Metrics and Thresholds

  • Detection comparisons use TPR at 1% FPR and AUPRC against the strongest development-selected baseline; unsupported resolutions or deployment metrics must not be inferred.The 0.1% FPR threshold is reportable only when benign test counts support that resolution.Source: §F.4 · p. 31

F.5 Causal Faithfulness and Adaptive Evaluation

  • Causal-localization claims require controlled edits and outcome comparisons, while adaptive-robustness claims require monitor-aware attacks and separate evasion-conditioned reporting.The reported non-adaptive evaluation alone does not establish adaptive robustness.Source: §F.5 · p. 31

F.6 Baseline and Ablation Contract

  • Baselines receive the same task splits and permitted inputs; completed comparisons include single-layer, complementary-layer, flat multi-layer, and lexical detectors.The oracle surface route is a nondeployable sensitivity analysis, not part of the primary implementation.Source: §F.6 · p. 31
Loading 2609.06612v1…